[OpenAFS-devel] Java API for AFS Admin -- CORRECTION!

[Leo Luan]

Ted,

Thanks for your helpful comments on the proposed Java API for AFS Admin
(http://grand.central.org/twiki/bin/view/AFSLore/JavaAdminAPI).  Below are
point-by-point responses from us.

Leo

=================================

>Internal links for these attachments don't work.  I guess that is a
>problem with javaDoc.

Yes, Javadoc problem. This will be fixed for the next submission.

>Is Cell.getGroupNames significantly faster than just getting the list of
>group objects and asking each for its name?  And if so, is it useful
>very often?  Ditto for Cell.getInfoGroups and other similar methods
>throughout the API.  The getGroup and getGroupNames methods claim to
>return a list of currently cached groups, whereas getInfoGroups returns
>all groups in the cell.  What is the significance of this distinction?

As far as getting a list of names versus a list of objects, we did some
preliminary testing and it turns out that it is significantly faster, if
all you want are the names.  Getting the object requires some overhead in
JNI that should be avoided if necessary.  We are planning to do some more
formal testing on this, however, and obtaining some real numbers we can
maybe use in the documentation as justification. All the *Info* methods are
leftover from a testing stage of the code, and have since been taken out of
the documentation.

>In some cells there are a great many groups and users.  I think there
>are cells with upwards of 50K users.  Maybe using an iterator would be a
>better approach?

We've decided create Iterator objects for each type of list of objects that
can be obtained (but not iterators for lists of names, since this seems
unnecessary).  Thus, there would be for example a Cell.getUserIterator()
method, which would return an object that implements java.util.Iterator,
and could return User objects one at a time.

>In Cell.getKey, what is the key "name".  Would this normally be "afs"?
>The description of "name" says "the encrypted key String of the key to
>retrieve", which isn't clear.  Also the "server" argument mentions
>"partition", which might be a cut-and-paste error.  Would a kvno
>argument be useful for input (I see it is a property of the returned Key
>object).

We agree that the kvno would be a more useful indicator of a key (certainly
better than the octal form of the key), so maybe we'll change that.

>Maybe Cell.getKey, getPartition, getProcess would be better as Server
>object functions?  Generally, I would think it would better to push
>these functions further down in the object hierarchy.

Yes, we're going to push all of these getVolume, getProcess, etc. methods
down into the direct parent classes.  Originally they were in the Cell
class for efficiency purposes of an application we were building on top of
the API, but we have since separated our application from the API more
fully.

>There is some justification for Cell.getVolume, but it shouldn't require
>server and partition parameters.  But then the Volume.getPartition
>should return a list of all the partitions hosting this volume.
>Probably there should be two related classes: a VLDB volume and
>partition volume.  The latter would be a subclass of the former.  The
>partition volumes could be obtained from the VLDB volume by asking for
>the appropriate type.  The getReadonly method would return a list of
>(partition) volumes.

There is some VLDB support in the admin C library for iterating through
VLDB entries and zapping entries), so we may be adding it to the API.

>Does the Cell.getProcess "name" argument refer to bosserver instances?

Yes, it refers to the name of the server process that the "Process" object
represents.  If the name provided does not match an existing server process
(instance) name, then an exception is thrown.

>Cell.refreshTokenExpiration doesn't seem very useful.  The expiration
>time doesn't change.  Maybe you mean it reauthenticates to extend the
>expiration time using the (saved) password.

Good catch.  We've removed this from the API.

>Group.getCreator should return a User object instead of having separate
>getCreatorName and getCreatorUid methods.  Along the same lines, I'd
>eliminate getGroupsOwnedNames.  Generally, there are lots of similarly
>redundant methods throughout the API.  I'd favor simplifying the API by
>removing these extra methods.

The problem with getCreator returning a User object is that a Group could
actually be the creator (e.g. system:administrators is the creator of
system:administrators), and Java doesn't support covariant types.  The
underlying native library refers to the creator by name and id, and we
found this useful; and the user of the API could determine whether it's a
user or a group from the id and then construct the appropriate object with
name if needed.

>Group.getListAdd, for instance, returns a String, but the description
>says it returns one of Group.GROUP_xxx_ACCESS, which are declared to be
>static int.  I guess this is related to the getListAddHandle method, but
>the distinction isn't clear.

All the "handle" methods have been removed (or changed to a protected
status), and so things like getListAdd will once again return static ints.
This was another confusion related to efficiency issues in our application.

>The meaning of the Group.refresh method's "force" parameter is not
>clear.

The documentation for our caching model will definitely be cleared up in
the next release.  Basically, if "force" is true, the object will retrieve
all of its member fields (all statistics, lists, etc.), so they're a
reflection the current status of the underlying AFS system.  If "force" is
false, only ones that have been retrieved in the past will get updated to
the current status.  Ones that haven't been touched yet will not get
retrieved, thus making it more efficient.  The first time these untouched
things are accessed, they will be retrieved from AFS and cached.  However,
we've decided to just delete the refresh( force ) method from the external
public API and make the default behavior of refresh() act like
refresh(false), in order to make the caching mechanisms less cluttering for
the user.

>It is a little odd to have an ACL setting method in the Group and User
>objects, but perhaps not terribly so.  How can I remove an ACL entry:
>specify false for all the booleans?  Might it be handy to have a
>permission object which conveniently encodes various bits.  This would
>allow setACL to take fewer parameters and allow the Permission class to
>export static objects with useful names like "none", "read", "write" and
>"all".

We've taken all ACL-related methods out of the API.  We instead have them
in another, more file-centric application we've developed.

>Key.getVersion and Key.getVersionHandle seem redundant.  Surely it is
>easy to get the int from an Integer.  Ditto for lots of other *Handle
>methods in the API.

Again, no more handle methods exist.

>What happens if a key is created without a version number?  It doesn't
>look like there is a method for setting it later.

We think this is a misunderstanding of our constructors, which will
probably be cleared up with documentation.  For example, the Key
constructor does not actually create a key in AFS; it is merely an abstract
representation of a key.  So when you create a "blank" key (without a
version number), you are just making a blank container, which is filled in
later internally by the native library.  We've since made these "blank"
constructors protected in scope so they won't appear on the main API and
confuse everyone.

>Is it the case that the keyString input to Key.create is the same as
>what is returned by the getEncryptionKey method?  Is this a password
>which is passed through (one of the) StringToKey function(s) or is this
>the raw 56 bit/8 byte DES key?  Can you make provisions for support of
>the various types K5 encryption keys down the road?  Maybe there should
>be a key type attribute included here for future expansion.

We agree that k5 is an important direction and will definitely need to be
considered in the final version of the API, but we don't have any cells up
locally using anything other than kaserver.  This is certainly important,
though, since the community seems to be leaning very heavily toward K5
authentication -- we just haven't gotten there yet.

>The Server.getBinaryRestartDay and getBinaryRestartHandle are redundant,
>and the disparity doesn't exist for getBinaryRestartHour, Minute, etc.
>It seems a little silly to have so many little methods for accessing
>these many attributes.  Partly I guess this is a Java problem, but maybe
>there's a better way?  Return an array or hash of values?

Instead of having all the separate Hour, Minute, etc. methods, we've
decided to make something like an ExecutableTime class that will contain as
fields the hour, minute, etc. of the restart times.  This models more
closely the way the native library uses structs to represent executable
times, and makes things much cleaner for the API probably.

>Sometimes server log files are very large.  Is there a way to return a
>stream or a local file name, to avoid bundling an entire log file into a
>single huge string.

We have not yet found a way to stream data from JNI to Java, but that would
be very nice indeed.  This is something we're looking into.

>How do I set the name of a Server object that was created without one?
>Ditto for Process objects.

Again, these "blank" constructors have been effectively eliminated.

>Is it possible to get the output produced by the Server.salvage method
>subsequently, perhaps using something like getLog?  Ditto for the
>salvage method of Partition and Volume objects.

We haven't experimented with salvage too much; is the output useful enough
to provide this as a feature?  If so, it would probably be pretty easy to
do through getLog as you suggest; we'd just like to hear some arguments in
its favor.

>There should be a Server.getPartition by name, instead of only a method
>to return a list of all partitions.  This would be a proper replacement
>for the Cell method.  Ditto for getKeys and getProcesses.  Ditto for
>Partition.getVolumes.

Agreed.  These will be there.

>Server.getIPAddress returns a string.  Is this in dotted quad notation?
>What if a file server has multiple IP addresses.  Is the string typed in
>such a way that we can handle IPv6 addresses someday?

Yes, multiple IPs will be supported in the next version of the API.  The
string representation the IP addresses is just a normal Java String -- it
can represent anything it needs to. Do you think any special considerations
need to be made for IPv6?  It is dotted quad, there will be an example in
the next version.

>What is the format of the time string returned by getGeneralRestartTime?
>Maybe a sample string would be useful.

It will no longer return a string (cause parsing is bad), but the
ExecutableTime object I mentioned earlier.

>Server.getTotal{,Free,Used}Space seems like it should be a function of a
>Partition.  The description also mentions "server" and "partition" at
>different points, to further confuse the issue.  Eliminate the various
>getUsedPercentage methods and let the caller do the compuation himself?

We put these methods in the server for efficiency/caching purposes.  It's
definitely arguable as to whether or not they belong in an API, and we may
end up removing them from the public API.

>Does Server.getTotalQuota actually iterate through all the volumes, or
>does the server actively track this?  If the former, then maybe the
>caller should compute this number himself.

See above answer.  Another problem we came across with this is that Volumes
can have a quote of 0 to mean unlimited, and it doesn't make sense to add
this with the otehr quotas to get a total.

>I suppose it should be obvious, but maybe it is worth making clear that
>the constructors for Partition and Volume only instantiate object the
>underlying entity is created by the Volume.create method.  I guess there
>isn't a Partition.create method, though this would be useful in some
>scenarios, though it is out of the scope of existing AFS Admin tools.

Yes, exactly.  Creatig partitions/servers/cells is certainly out of scope,
but interesting for other projects perhaps.  New documentation should clear
this up. Same for Key/Process/User/Group constructors.

>Related to my earlier comments about two volume classes, is a Volume
>object localized to a server/partition?  I see that there is a moveTo
>method which takes a Partition (and implicitly a server), but then why
>does the constructor take a cell and server name?  How does one access
>VLDB functions like vos delentry or zap?

We've changed these constructors to only take the lowest level object (i.e.
Partition instead of Cell, Server, and Partition).  This was just an
oversight.  Again, we may be adding some VLDB functionality in the API.

>How do I set the id of a Volume?  This is needed for restore by id,
>isn't it?

Good point.  We were making the assumption that you'd only want to restore
a volume to having the VLDB assign an id automatically (assigning an id is
tricky business), but we'll probably add the option of passing in an id for
all those gungho administrators out there.

>It would be very nice if the User class would know how to manipulate
>users in a K5 environment as well.  Many of the user attributes for
>which there are lots of explicit methods, are idiosyncratic to the
>KaServer.  It would be nice to abstract some of this detail somehow, so
>that mapping to other authentication environments would not be too hard.
>Maybe a base class that contains only the basic user object behavior,
>and derrived classes that handle KaServer, KerberosVMIT, HeimdalKTH, or
>ActiveDirectory users.

As we mentioned earlier, we will be looking into what it means to be an
abstract AFS user and what it means to be a subclassed AFS user for future
versions of the API.  At the moment we are pretty inexperience in this
area.

>User.getEncryptionKey claims to return a string "in octal form".  What
>does this mean?

"Octal form" is described in the AFS documentation for "bos listkeys", but
basically it is eight three digit numbers preceded by backslashes, i.e:
"\123\123\123\123\123\123\123\123".  We will provide an example of this in
our documentation for anyone who might need to know.

>User.setPassword should also take a StringToKey type to better handle
>the present and future diversity of these functions.

Yes, future research.

>I noticed there is a User.equal, but, for instance, no Group or Volume
>equal methods except those inherited from the Object class.  Anything
>special going on for users?

This has been corrected.  Equal methods for everyone!

>Here are some random thoughts on caching.  Each of these objects has a
>welter of refresh functions.  It would be nice to have a better scheme
>for managing the caching of object attributes.  In particular, it would
>be desirable if there was a way to subsequently drop in a more clever
>caching mechanism without changing the users of this API.  It would make
>sense, I suppose to have a caching version of each object that
>sub-classes the base object but adds caching.  Perhaps this is how you
>have implemented it already.  From the stand-point of the API, the
>problem is that sometimes a user will have out-of-band reason to want to
>refresh or invalidate the cache.  As long as there is no explicit
>callback-like mechanism, then any polling interval is sometimes going to
>be too short and other times too long.
>
>It would be relatively easy to imagine adding a callback interface to
>the vldb, ptserver, bosserver, etc, which would keep a list of contacts
>interested changes.  Perhaps this could even be an add-on server that
>runs locally on the server and monitors the local file system for mtime
>changes to the database files.  Fine grained callbacks could be
>implemented by having this callback server make a (local) call to the
>database server to see if the specific object of interest had changed.
>
>An easier strategy would be to have variants of each function that
>allows the caller to specify an acceptable age for the information.
>Perhaps a better approach would be to have method for each object that
>sets the allowable latency (max age) and whether to keep it up to date
>in the background or whether to refetch it on demand if the cached value
>is stale.  Setting it to zero would cause all query functions to go
>straight to the server.  It could be adjusted up and down as needed by
>the API user.

We're actually thinking about our caching mechanisms in a very different
way: we want them to be as simple and out of the way as possible, and let
the application take care of when it wants the cache refreshed, etc.  As
far as adding callbacks, this would probably require changes to AFS itself,
which is not our intention.  We're not completely aware of what callbacks
AFS currently supports, but these features are not supported in the
administrative library we've built this API on, and we think it would make
things a bit more complicated than they need to be.

Thanks again for your comments, please let us know if you have further
comments or questions.