[OpenAFS] What filesystem?

[Jeffrey Altman]

This is a cryptographically signed message in MIME format.

--------------ms030408020903060102080101
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit

Christopher D. Clausen wrote:

> Well, at UIUC we have a common campus-wide domain so its a FEATURE that
> all shares are under it.  Seamless Kerberos logons between disparately
> managed file servers.  I don't think its possible to delegate control
> like that with AFS.

I believe that Rodney's point is that in most academic institutions one
can hardly assume that all student and faculty computers are members of
the domain.  At many schools there is a central Kerberos and Active
Directory infrastructure, but there are also many departmental Active
Directories that are not in any way connected to the centrally managed ones.

What you are describing as seemless Kerberos logons is the ability of
the Windows CIFS client to determine which server the user is attempting
to contact and to obtain a Kerberos service ticket for the CIFS service
on that server when the user has already obtained a TGT at logon.

This single sign-on capability is not provided by OpenAFS for Windows.
However, the new NetIDMgr with its AFS plug-in takes away much of the
inconvenience of supporting multiple cells.  One Kerberos realm (or
Active Directory domain) can be used as the authentication source for
multiple AFS cells.  Therefore, it is possible to give administrative
control over cells to different groups within the organization while
requiring the end users to perform only a single initial authentication.

For example, the ATHENA.MIT.EDU realm provides authentication to the
following cells in the public CellServDB: athena.mit.edu, dev.mit.edu,
net.mit.edu, sipb.mit.edu, and soap.mit.edu.  Using cross-realm
authentication the ATHENA.MIT.EDU principal can also be used to obtain
tokens for many other cells including andrew.cmu.edu, iastate.edu, etc.

With NetIDMgr and its AFS plug-in, the user can configure a list of
cells for which AFS tokens are to be obtained whenever the associated
Kerberos TGT is acquired.

NetIDMgr will replace afscreds.exe in a future release of OpenAFS
for Windows.

> I'm going to have to disagree with your statement of caching as well,
> unless I totally misunderstand how offline files work.  And, that is in
> fact another feature, offline access to remote files.  (AFS sort of has
> this on Windows, but only on Windows and using the same methods.)

The CIFS offline files mechanism is not exactly caching.  It provides
a mechanism by which the contents of a remote share can be locally
replicated at logon and/or logoff or upon user request similar to how
roaming profiles work.  This is a valuable functionality and I would
not claim that the AFS client on Windows has it.  The only reason it
would work with AFS on Windows is that the method of accessing AFS on
Windows is via the CIFS client which implements the "Client Side
Caching" functionality.

> As to sharing CIFS over the Internet, I certainly would do so but
> unfortunately the matter has been decided and the campus firewall blocks
> MS RPC and CIFS traffic.  However, the common refrain I keep hearing is
> "use the VPN client" to gain access to campus resources.  I am not
> really a fan of the Cisco VPN that UIUC offers, but there is nothing
> preventing me from running my own "Routing and Remote Access" server
> allowing access with the much friendlier native Windows client.  (One
> could argue that VPNed access provides a higher level of encryption than
> AFS does, although one can also use the VPN with AFS.)

Blocking the ports at the organizational boundary and using VPNs is the
recommendation of Microsoft.

>> AFS:  Yes, Yes, Yes, Yes, Yes, Yes
>> Dfs:  No, No, Sometimes, Maybe, Unsure, Not documented, etc. etc...
> 
> Again, depends on your needs.  After working with NTFS ACLs I find AFS
> ones limiting.  There appears to be no way to allow one to create mount
> points and not change ACLs.  

This is conscious decision to prevent random people with write privs
from being able to add a link to other space that has different ACL
requirements.  The notion is that adding a link to different ACL space
would be the same as making ACL changes.

> Why is that?  There isn't a easy way to
> have shared scratch space, preventing others from nuking not their own
> files.  There is no way to grant access to directory and NOT have that
> same access get inherited to newly created sub-directories.

In Windows, the concept of inheritance was added after the fact.  In
general, if you are locking down a directory, you want it to be locked
down.  If you want to give users individual space, an administrator
should create the directory and set the appropriate ACLs on it.

> Not needed to setup the equivalent of PTS is a feature IMHO as well. Why
> do I need an account to the filesystem?  Or for that matter, a token. 
> Not needed to have end-users worry about tokens can be a feature.  (Not
> being able to deny access to the filesystem by unloging is a non-feature.)

In other words, you would like to be able to use Windows SIDs as AFS IDs
and have a PTS RPC service as a front end to active directory.

> I'm going to have to say that AFS has some strange "undocumented
> features" as well.  Like how the owner of the root of a volume gets
> extra rights...  And the system:ptsviewers group... and the refresh time
> on IP-based ACLs...  (I could have missed all this in the pages of
> documentation though, and yes, I know its being worked on.)

At least the AFS3 file system protocols are documented in source code
and a few postscript files.   There is no complete documentation for
CIFS.  If there was, perhaps I could finish implementing the pieces
that the AFS SMB Server does not yet support.

> Well, not everyone here at UIUC can afford RAID setups.  I've worked at
> several departments where the file-server is just a desktop machine with
> a single hard drive.  Convincing them to spend money on an entire
> redundant server is easier than on expensive disk.  ("I got this 300GB
> USB drive for $200.  Why do you need $7000 for an array?")  We both know
> that RAID is good, and if you have the money use RAID with a
> replicatation partner.

What about software RAID?  At the very least you can implement that for
the same $200.

> RAID doesn't help if a processor or a power supply goes bad, or on OS
> upgrade freaks out or whatever.  As in, its not protection against
> SERVER downtime.  It protection against only disk failures.  (And being
> in a place that uses 3 year old Dell desktops as fileservers, this is a
> concern.)

No its not but frequently hardware failures are not day zero events.
Most of the time there are bad disk blocks, memory corruption errors,
blue screens that indicate that something is wrong before the hardware
finally goes.

The benefit of AFS is that you can migrate the volumes while they are
in use to another server so that you can perform the hardware
maintenance without disturbing your users.

> Only the user volumes are read-write?  All of my important data is in
> user volumes.  Read-only data is easy to keep online, just replicate it.
> The important stuff is what people are working on RIGHT NOW and if its
> down bad things happen.

The Windows replication model in many cases is only as good as backup
volumes or read-only clones that are taken with frequent snapshots.
This is especially true if the user is using client side caching.

> Warning: I am a Windows person.

So are Rodney and I.

> I'm not at an enterprise.  I'm at a University.  My systems at home are
> more advanced than most of the "production" services I find in use
> throughout campus.  (Not at the campus IT level, but at a department
> level: single fileserver, 100 clients max.)

I would really like to have the resources to get the AFS file servers
in a state they can be deployed on Windows Servers or even XP
workstations but I drift...

> Yes, this was annoying.  Had to create local partition "S:" (for
> software) and install things there, then copy into AFS and mount the
> path as S: on each client machine.  Same trick for Dfs installs as well.
> Almost everything works using this technique.  The problem appears to be
> mainly the installers, not the apps themselves.  Got to watch things
> that install as system services or explorer extensions though.  That
> causes PAIN.  Unless of course you ACL everything system:anyuser rl so
> that the system can read it.  Probably not legal to do in all
> instances... but that's another matter...
> 
> Another problem is 100BASE networks are at least 4 times slower than
> hard drives.  Who wants to wait for that?  And, I have 8GBs of software
> installed on my Windows machines.  I believe the Windows AFS cache size
> limit is about 1.3GBs.  Say all you want about bloated software, but its
> a fact of life.

1.3GB is a limitation of the Windows 32-bit process space.  Switch to
64-bit Windows and use a 1TB caches.

> Do you really run Visual Studio and AutoCAD directly out of AFS?

Um.  Yes.  and ProEngineer.  In fact, Rodney has more than 200
engineering applications running out of AFS.  I know because I use
his environment to help test each release of OAFW.

> I only have about 20 machines that I directly maintain.  Again, most of
> the data is in user volumes that people access from their own computers.

> And the vos release stuff is annoying as well.  Why should I need to
> write an app to handle allowing users to replicate volumes?  What about
> the user education issue?  "Oh the website isn't up to date b/c no one
> vos released it yet."

How is this different from "the website isn't up to date because no one
checked in and released the changes yet"?  That's an example from
Microsoft's FrontPage.   People have to be trained to use software.  Its
just a fact of life.

You have to give ACL permissions to the the end users to update the
web pages inside FrontPage.  Why wouldn't you have to grant ACLs to do
so for their web sites?

> Yes, having separate RO and RW paths is good in some cases.  Sometimes
> it isn't.

I'm not arguing that having RW replication is not a good thing.  Its
just really hard to get it right and provide the performance that most
people would prefer.  The fact that Windows allows users of the same
data on two different servers to see different versions of the same
file is a serious problem for many applications.   I am really
intrigued to know what Microsoft is doing in this case since all of
their locking mechanisms use a mandatory model and require data consistency.

> Oh I agree.  Your view of things and some other view may be different.
> I'm just supplying the reasons why someone may choose MS Dfs over
> OpenAFS.  (Having already done the research and picked AFS.)

If you have a Windows only environment with limited support staff by
all means use Dfs.  If you need to only allow your own users to access
it and are willing to perform all of the identity management then by
all means use Dfs.

> Being a Windows admin, I concur with that.  Why waste the fileserver
> serving out the same exact app everyday?  Why not utilize it for files
> that people actually need to share?  (I'm on a 10BASE network right now.
> I can assure you that there would be much time wasted waiting for Office
> to startup out of a network filesystem.)

and many Windows servers have fallen over because 100 users have all
logged in at the same time and started up the pre-configured set of
applications over the network.

> Well, we have no money for support of any kind and for us both Microsoft
> products and OpenAFS are free so its easy to try and compare.

you just pay in a different way.

> Most departments here at UIUC can easily afford $2000 for an additional
> fileserver for replication but cannot afford SAN gear, even low-cost
> iSCSI stuff.  In the end, it comes down to what you can do with the
> money you have.  Dfs is still better than just plain CIFS on a single
> fileserver.

Or what people think they can afford.

>> Like you said, Dfs is primarily an all Windows technology anyway.  If
>> you need anything heterogeneous then you must turn to AFS.
> 
> This is entirely true.  Again, some places are all Windows where Dfs
> would make sense.  I believe this is why you see more universities using
> AFS, they tend to allow more freedom in IT.  Corporations pick a
> standard and that's it.
> 
> <<CDC

Corporation A chooses 'a' and corporation B chooses 'b' and then A buy B
and now they have both or A needs to work with B and now they must
support both.  Not as different as the University as one might think.

Jeffrey Altman

--------------ms030408020903060102080101
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJXzCC
AwowggJzoAMCAQICAw7NrTANBgkqhkiG9w0BAQQFADBiMQswCQYDVQQGEwJaQTElMCMGA1UE
ChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNv
bmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwHhcNMDUwNTI3MTc0NzU3WhcNMDYwNTI3MTc0NzU3
WjBzMQ8wDQYDVQQEEwZBbHRtYW4xFTATBgNVBCoTDEplZmZyZXkgRXJpYzEcMBoGA1UEAxMT
SmVmZnJleSBFcmljIEFsdG1hbjErMCkGCSqGSIb3DQEJARYcamFsdG1hbkBzZWN1cmUtZW5k
cG9pbnRzLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKjPyrF+rdjOUSK/
bWwZHdx5p1+y6iiCd4vvYEVDxouYFp5C/fZEWm5n45ubBUbMSUI1MAZN6ooEoH09UTj6BXhM
S8B987ls81dKOIUphTF2jOzq8gsFmeA15yHMRAD20LqUWeLyvYk8FCNQw+dsKMMhX+WdsxOm
RY/1jPkJL6oN8kEwoUFkOX9/OfWWh6oFnV6faiEHUKDMFubsb9X0KVD8iIeR7Cxz7i4kXqRX
wMlp2fyoxcDIJrBaTY8nA++g3p34IkWt1a5po6g683nIgSnGpwYIwuJheBqSEZfLYWa+1KdD
6Sn27Ud94GqUvPVG5jC6zVC5EJ2aWuoAu+nNuV8CAwEAAaM5MDcwJwYDVR0RBCAwHoEcamFs
dG1hbkBzZWN1cmUtZW5kcG9pbnRzLmNvbTAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBAUA
A4GBADtvO//tjiAV6VJGtoNtrl34mB5jGyGTiotzw8riB6zz0GvY11bcWDmp6JKif+pVG+8L
IySDosbuva13qu2HwYUxBmWc7CoNd2k9kRlcrfbDUTTrGOZK8qyqNqT3gQZTAa9ZnUI0su9G
y/n2o5bQcaYdqR3htNrpvdLSPOWhILOXMIIDCjCCAnOgAwIBAgIDDs2tMA0GCSqGSIb3DQEB
BAUAMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBM
dGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQTAeFw0w
NTA1MjcxNzQ3NTdaFw0wNjA1MjcxNzQ3NTdaMHMxDzANBgNVBAQTBkFsdG1hbjEVMBMGA1UE
KhMMSmVmZnJleSBFcmljMRwwGgYDVQQDExNKZWZmcmV5IEVyaWMgQWx0bWFuMSswKQYJKoZI
hvcNAQkBFhxqYWx0bWFuQHNlY3VyZS1lbmRwb2ludHMuY29tMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAqM/KsX6t2M5RIr9tbBkd3HmnX7LqKIJ3i+9gRUPGi5gWnkL99kRa
bmfjm5sFRsxJQjUwBk3qigSgfT1ROPoFeExLwH3zuWzzV0o4hSmFMXaM7OryCwWZ4DXnIcxE
APbQupRZ4vK9iTwUI1DD52wowyFf5Z2zE6ZFj/WM+Qkvqg3yQTChQWQ5f3859ZaHqgWdXp9q
IQdQoMwW5uxv1fQpUPyIh5HsLHPuLiRepFfAyWnZ/KjFwMgmsFpNjycD76DenfgiRa3Vrmmj
qDrzeciBKcanBgjC4mF4GpIRl8thZr7Up0PpKfbtR33gapS89UbmMLrNULkQnZpa6gC76c25
XwIDAQABozkwNzAnBgNVHREEIDAegRxqYWx0bWFuQHNlY3VyZS1lbmRwb2ludHMuY29tMAwG
A1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEEBQADgYEAO287/+2OIBXpUka2g22uXfiYHmMbIZOK
i3PDyuIHrPPQa9jXVtxYOanokqJ/6lUb7wsjJIOixu69rXeq7YfBhTEGZZzsKg13aT2RGVyt
9sNRNOsY5kryrKo2pPeBBlMBr1mdQjSy70bL+fajltBxph2pHeG02um90tI85aEgs5cwggM/
MIICqKADAgECAgENMA0GCSqGSIb3DQEBBQUAMIHRMQswCQYDVQQGEwJaQTEVMBMGA1UECBMM
V2VzdGVybiBDYXBlMRIwEAYDVQQHEwlDYXBlIFRvd24xGjAYBgNVBAoTEVRoYXd0ZSBDb25z
dWx0aW5nMSgwJgYDVQQLEx9DZXJ0aWZpY2F0aW9uIFNlcnZpY2VzIERpdmlzaW9uMSQwIgYD
VQQDExtUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgQ0ExKzApBgkqhkiG9w0BCQEWHHBlcnNv
bmFsLWZyZWVtYWlsQHRoYXd0ZS5jb20wHhcNMDMwNzE3MDAwMDAwWhcNMTMwNzE2MjM1OTU5
WjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRk
LjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwgZ8wDQYJ
KoZIhvcNAQEBBQADgY0AMIGJAoGBAMSmPFVzVftOucqZWh5owHUEcJ3f6f+jHuy9zfVb8hp2
vX8MOmHyv1HOAdTlUAow1wJjWiyJFXCO3cnwK4Vaqj9xVsuvPAsH5/EfkTYkKhPPK9Xzgnc9
A74r/rsYPge/QIACZNenprufZdHFKlSFD0gEf6e20TxhBEAeZBlyYLf7AgMBAAGjgZQwgZEw
EgYDVR0TAQH/BAgwBgEB/wIBADBDBgNVHR8EPDA6MDigNqA0hjJodHRwOi8vY3JsLnRoYXd0
ZS5jb20vVGhhd3RlUGVyc29uYWxGcmVlbWFpbENBLmNybDALBgNVHQ8EBAMCAQYwKQYDVR0R
BCIwIKQeMBwxGjAYBgNVBAMTEVByaXZhdGVMYWJlbDItMTM4MA0GCSqGSIb3DQEBBQUAA4GB
AEiM0VCD6gsuzA2jZqxnD3+vrL7CF6FDlpSdf0whuPg2H6otnzYvwPQcUCCTcDz9reFhYsPZ
Ohl+hLGZGwDFGguCdJ4lUJRix9sncVcljd2pnDmOjCBPZV+V2vf3h9bGCE6u9uo05RAaWzVN
d+NWIXiC3CEZNd4ksdMdRv9dX2VPMYIDOzCCAzcCAQEwaTBiMQswCQYDVQQGEwJaQTElMCMG
A1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBl
cnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECAw7NrTAJBgUrDgMCGgUAoIIBpzAYBgkqhkiG
9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0wNjAzMDYyMjE4MTlaMCMGCSqG
SIb3DQEJBDEWBBQYZxlj/wljNB7LgByT85Gqa5YffTBSBgkqhkiG9w0BCQ8xRTBDMAoGCCqG
SIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG
9w0DAgIBKDB4BgkrBgEEAYI3EAQxazBpMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3
dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJl
ZW1haWwgSXNzdWluZyBDQQIDDs2tMHoGCyqGSIb3DQEJEAILMWugaTBiMQswCQYDVQQGEwJa
QTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhh
d3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECAw7NrTANBgkqhkiG9w0BAQEFAASC
AQAA/35RMGreqdrRw3dOfhzlQqFShUieg0r6o24Qy8XDV/AO0J7VYisYkNfh8Bw3yYr29NIU
TJAEQbJog/t3PJ8NNMEPIBCRxkGAxPEwdx5APfIpTY/3gmYXaF6+z6r0+bhJHUGD7QJ43ko6
9c+0R6UhLAR+5kbL+0+vWyEPmRNZDAAsXdTOBjVfcyco0jb8lrdTdFpy/cq0Uo8YdLODJYSr
3pdUIfNVejtKfZ8vbZLn7j/a2qoCMMIuV0b0/R4CuOD2GLBrPUyHvvxWh02BF+iYIjclIdFm
gLOjVrVmCDeHIVN7uiZbBzonP+3dF1RbUAZFmkH9gvOV5BBRvswvvKmZAAAAAAAA
--------------ms030408020903060102080101--