[OpenAFS] Re: [AFS3-std] Re: IBM will not re-license OpenAFS .xg files

[Jeffrey Altman]

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enigF7E08BC60E3CD414EE7CAFF3
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

[Adding openafs-info because this discussion is relevant to a broader
audience than simply the afs3-standardization group.  The full thread
can be found at

https://lists.openafs.org/pipermail/afs3-standardization/2012-August/0008=
98.html

for those not subscribed to afs3-standardization@openafs.org]

On 8/29/2012 2:48 PM, Kim Kimball wrote:
> I hear at least some extremes in this discussion. =20
>=20
> I hear that "we can't" ... "we must"=20
>=20
> Perhaps we can evaluate this:  "... or are desperately-needed functiona=
lity that can't afford to be
> blocked on standardization. "
>=20
> What is the desperately needed functionality, and for each such item wh=
at is the desperate need?

The list varies by the organization but a short list that requires
protocol changes includes:

 * A broad range of security weaknesses which are addressed by rxgk.
   These include:
    . no more DES
    . AES-256-SHA-1 (or stronger) and mandatory privacy
    . no cache poisoning
    . no unprotected callback channels
 * IPv6 addressing
 * read/write replication
 * alternate data stream/extended attributes
 * longer volume names
 * larger volume and file id namespaces
 * higher precision timestamps
 * 64-bit partition and volume quota sizes
 * removal of directory format limitations
 * uniform use of Unicode for directory entry names
 * per file ACLs
 * mandatory locking
 * byte range locking
 * improved cache coherency algorithms
 * ability to rapidly move partitions between file servers
 * machine accounts to replace IP ACLs
 * peer to peer cache sharing

There are broad range of other improvements that are implementation
specific and are therefore outside the realm of the AFS3 standardization
effort:

 * a long list of ubik changes
 * disconnected operations
 * performance, scalability, and stability improvements
 * standard library interfaces for administration that can
   be used by perl, ruby, python, etc.
 * ease of use improvements

And then there are non-feature specific building blocks such as:

 * xdr extensions for unions, new primitive types, etc.

This is not an exhaustive list.  I'm sure I have forgotten many things
that have been requested over the years.

Of those, the most pressing is probably the security work but each
organization has its own priorities and a lot of the work has mutual
dependencies.

> This begs questions: =20
>=20
> If OpenAFS could deprioritize some number of functionality related task=
s, would resources devoted to those tasks really be reallocated to standa=
rdization? =20

The resources of the gatekeepers are donated.  Up until quite recently
Stanford University donated a portion of Russ Allbery's time.  Your File
System Inc. (YFSI) has covered the expenses related to the salaries of
the other gatekeepers.  YFSI has (on its own volition) paid non-employee
members of the OpenAFS developer community to fix critical bugs that
have resulted in data loss, performance or instabilities.  The
gatekeepers manage the source tree, perform the majority of the code
reviews, write a significant amount of code, and act as bug fixers of
last resort.  They are also asked to perform architecture reviews, give
talks at conferences, and a variety of other tasks.

The rest of the contributors are equally independent of OpenAFS.  They
might be individuals that use OpenAFS at home, or employees of
organizations that deploy OpenAFS, or employees of companies that sell
support services to others.  The priorities of these individuals and
organizations are their own.

After the NJIT Best Practices Workshop it was decided that AFS3 protocol
standardization should be independent of OpenAFS.  There was a fear that
the dominance of OpenAFS would permit it to dictate the future direction
of the protocol simply by committing code.  OpenAFS agreed not to accept
code changes that altered the protocol unless those changes were backed
by the consensus of this independent group.

What resources does the AFS3 standardization group possess?  There are
two chair people that committed themselves to provide time to this group
when they accepted their nomination prior to election.  Beyond that,
this group does not have any specific resources.

> Can OpenAFS currently identify people who would gladly work on standard=
ization but are currently blocked on functionality tasks?

There are very few individuals that find writing documentation
(especially protocol documentation) an enjoyable hobby.  Protocol
documentation needs to be written to exacting standards and should be
verified by an attempt to use the documentation to implement the
protocol described.  The work is very tedious and time consuming.

There have been a total of 37 contributors to OpenAFS in the last twelve
months (down from a high of 54 over a 12 month period) and 10 in the
last 30 days.  Of those contributors, fewer than 10 contributed more
than 10 patches to code, documentation, packaging, or the web site over
the course of the project.  The number of individuals that have been
active in the AFS3 standardization group are even fewer.

We are talking about a very small pool of volunteers.  The majority of
these contributors do not work on AFS full time.

> IOW, is it possible to use Russ' candid observation to start a more pra=
gmatic conversation? =20
>
> I admit I haven't read all the posts in detail, and may have missed a m=
ore fact based discussion and if so I apologize.  And of course I'm not a=
ware of what everyone is doing, so am delightfully free from subtext.
>=20
> What do we need to know, factually?  What resources can OpenAFS count o=
n?  Does/Can OpenAFS agree on priorities?  Who's working on what right no=
w?  If tasks were reprioritized,  who would actually volunteer to work on=
 standards tasks?  Is it possible to list/name tasks/priorities/resources=
?
>
> To summarize ... It seems from my naive take on this conversation that =
it is worth some detailed analysis and that the current (explicitly?) agr=
eed priorities are not working.
>=20
> Kim

I believe that Russ' pessimism stems from his pragmatic observations of
the AFS3 ecosystem and his vast experience within a broad range of open
source organizations.  He understands the complexity of the required
functionality and can estimate the scope of the resources necessary to
implement them with and without protocol standardization or documentation=
=2E

I share Russ' pessimism.  There is a significant mismatch between the
desires of the end user community, the dreams of the developer community
and the funding models necessary to implement the dreams on behalf of
the end users.

For any given protocol standardization proposal, substantial expertise
and time is required to perform proper analysis and review let alone
write a document in the first place.  On top of that it is very
difficult to anticipate all of the needs of a protocol and its viability
without implementation experience.  To ensure that the documentation of
a protocol is correct, multiple independent implementations are required.=


Such a process works best when there are multiple competing
implementations of a protocol where each implementation has an
independent and stable funding source.  For AFS3, this is hardly the
case.  IBM AFS is completely shutdown as a product.  Funding for Arla
was halted by Stockholm University and KTH years ago.  David Howell's
kafs has not grown a development community of its own.  That leaves
OpenAFS as the only implementation and it does not have a stable funding
source.

Contributions to OpenAFS come from four sources:

 1. individuals that contribute relatively small changes to
    fix bugs, documentation, or provide support for revised
    operating system releases.

 2. end user organizations that either implement a feature
    in-house or more frequently contract with companies such
    as LinuxBox, Secure Endpoints, Sine Nomine or YFSI to
    perform the development.

 3. support companies that fix bugs or add new operating
    system support as part of on-going maintenance for their
    support clients.

 4. companies that are willing to invest in new features
    with plans to recoup the investment at a profit later on
    by selling products and services.

The fact is that the first three categories have rarely provided the
resources necessary to produce substantial change.  Individuals unless
they are independently wealthy simply do not have the time or resources.
 End user organizations are willing to fund small focused projects that
can be achieved on the order of months not years.  End user
organizations are unwilling to fund partial work that is reliant upon
other organizations to independently fund subsequent segments.

If the end user organization community wants major changes to the AFS3
protocols and to one or more AFS3 implementations, it is going to have
to back those change requests by substantial funding.  There is little
incentive for a company that sells support services to invest their
own financial resources in new protocol development, implementation, QA
testing, and documentation.  Support and development contracts are
professional services agreements which are backed by expensive labor.
Such contracts do not produce large profits and those contracts that are
purchased do not necessarily fund those that are performing the work.

That leaves the fourth category.  A private commercial entity to develop
a commercial for-profit product to meet the needs of the end user
community.  In 2007, Your File System, Inc. was founded to be such a
company.  YFSI has been working for five years to produce a next
generation file system as a successor to AFS that is backward compatible
with AFS3 standard clients and servers.

As Founder and lead investor in YFSI, it only makes sense for a company
to publicly release its intellectual property as open source or as a
protocol standard under the following circumstances:

 1. The company has a viable business model that is not dependent
    upon sales of product based on the intellectual property.

 2. The company is selling a product but there are substantial barriers
    to entry that prevent other companies from taking the released
    IP and using it to create competitive products that would reduce
    overall sales.  Competitive products that increase sales by growing
    the market are ok.

A company that gives the fruits of its investments away prior to
achieving product profitability will not be viable in the long term and
will not be serving the long term needs of the end user community.

YFSI wants to be a company that falls into the first category but that
cannot happen until there exists a next generation successor to AFS3 to
build its services upon.  It was my hope that after YFSI received SBIR
grant funding in 2008 to kickstart major new development that other
organizations would have provided technical assistance.  Unfortunately,
that did not occur.  Documents that were submitted to AFS3
standardization languished due to lack of review which was one
significant factor in dragging out implementation times.

In order to bring a product to market that meets the long term needs of
the end user community YFSI was forced to make the decision to
abandon the plan to build its next generation file system by extending
the AFS3 protocol.  As such it side stepped the paralysis that the AFS3
standardization process suffers from.  Since making that decision YFSI
has rapidly progressed towards completing version 1.0 of a product that:

 0. provides interoperability with existing AFS3 clients
    and AFS3 file servers
 1. improves performance and stability
 2. add numerous security enhancements using rxgk as a base
 3. makes frequently used volumes more accessible to end users
 4. permits read/write volume replicas to be used for performance
    enhancements and disaster recovery / business continuity planning
 5. extends the name space to support larger file spaces
 6. supports mobile computing platforms such as iOS.
      http://itunes.apple.com/us/app/iyfs/id491921617?mt=3D8

YFSI is currently evaluating beta sites and intends general availability
during the first half of 2013.

I am stating this here and now because it has a direct impact on the
availability of resources to the AFS3 standardization process.  Given
the status quo it will be very difficult for YFSI to make its resources
available to the AFS3 standardization process until after it has
established it product on the market.  As Harald pointed out in this
thread and others have at various times on openafs-info@openafs.org, if
certain critical features are not made available to the end user
community real soon now, large organizations that require that
functionality will have no choice but to move their IT services off of
AFS3 to another file system protocol.  Many already have and others are
known to be evaluating their options.

I believe the status quo must change if there is going to be a viable
future for OpenAFS as a product.  Putting on my hat as a former OpenAFS
Elder[1].  I do not want OpenAFS to be a transition product to another
offering.  In the 1990s IBM wanted AFS to be a stepping stone to DCE and
in 2005 the OpenAFS Elders purged the membership of individuals that
wanted OpenAFS to become a stepping stone to NFSv4.  It is 2012 and I do
not want to see OpenAFS simply become a stepping stone to a YFSI
commercial product.  However, OpenAFS as a community open source project
does not have sufficient financial support from the end user community
for it to meet the long term needs of the community.

I recently experienced an epiphany.  I realized that one of the most
significant problems facing OpenAFS is that the deployment of OpenAFS
does not undergo substantial review by senior management because
strategic infrastructure deployments undergo review as part of the
budget process.  If there is no substantial line item associated with
OpenAFS, then it will not be reviewed until such time as there is a
critical deficiency such as the continued dependency on DES combined
with an audit requirement.  This dichotomy in which OpenAFS is deployed
as a strategic dependency but it is not funded as such is the core
viability question for the community.

A typical university deployment of OpenAFS has dozens of critical IT
services that are built on top of the AFS infrastructure.  Each of those
services may or may not be reviewed on an annual basis as part of the
budget process.  The additional cost of providing those services with an
alternative to OpenAFS may total millions of dollars a year not to
mention the transition costs.  However, none of this is visible to
senior management.  The associated risks are ignored and as a result
funding is not available for improvements until a transition to
something new becomes inevitable.

The questions that I believe need to be answered by the OpenAFS
community are:

 1. Do you want an AFS-like solution deployed in your organizations
    over the course of the next decade?

 2. If yes to 1, how important is it that the solution be OpenAFS?

 3. If yes to 2, are you willing to contribute funding to a Foundation
    which will use it to appropriately staff the existing gatekeeper
    responsibilities and development of desired functionality or
    acquire functionality from others?

 4. Given the lack of multiple AFS3 implementations, does there
    still need to be a firewall between AFS3 standardization and
    OpenAFS?  OpenAFS could have a protocol documentation requirement
    which is less stringent than an independent AFS3 standardization
    requirement.

In the end an open source project on the scale of OpenAFS is only viable
if those that do the work have a means of supporting themselves.  If
working on OpenAFS is a hobby project, then it will get the attention
that a hobby project can receive and work on it will be deferred when
day to day responsibilities take higher priority.

YFSI is staffed by developers that believe in the AFS vision and are
committed to bringing a next generation product to market that fulfills
that vision.  YFSI started down the road to a commercial product in 2007
when it applied to the U.S. Department of Energy for a Small Business
Innovation Research grant which requires the development of a commercial
product.

YFSI is going to bring our product to market.  I see no other viable
path in the short term to bring the required functionality to fruition.
However, if the community can come together to properly fund an OpenAFS
Foundation, then I believe the Foundation can play a significant role in
ensuring the long term future of AFS-like systems.

Jeffrey Altman

[1] I submitted my resignation to the OpenAFS Elders in early August
2012 due to increasing conflicts of interest between my responsibilities
as YFSI CEO and that of OpenAFS Elder.  I am continuing to fulfill my
responsibilities as OpenAFS gatekeeper.


--------------enigF7E08BC60E3CD414EE7CAFF3
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)

iQEcBAEBAgAGBQJQP8wmAAoJENxm1CNJffh4iugIAKdBYf1tClYFr7i/8s0QScPj
s9n5M6K241k6dIn171mHBq/Vg/jiiYimWviEWcf+LVgIXZ/FsVi5nUKoirMtGVX+
JcbaHsExXlcC3EK5c/CDQSBjSke1AM2jTibnyjhgbXscqosjiOXcYdIvefydBYTS
2243g74XPmTvl9zWiIPebUGl/DF5wTzaApO8OiTucemmltN5LiFTYvNe5YK03Mdf
VdLxFnOHHuJMPzJ2F7eImltd8KIGUtsS0aQqPV5TrEWHvr+mpytR8r3HfoHHvTZU
TpRxiy2a88BibIU8uGCv2gFOtoDuQTDkZ/EeH4cnlgB3Z9/YwnT2NRuGeNp/0h0=
=yPUO
-----END PGP SIGNATURE-----

--------------enigF7E08BC60E3CD414EE7CAFF3--