[OpenAFS] AFS design question: implementing AFS over a highly-distributed, low-bandwidth network

[Jeffrey Altman]

This is a cryptographically signed message in MIME format.

--------------ms090604030906000801050202
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Chaz Chandler wrote:

>> - if most of the users do mostly reading, and mostly smallish files, you
>> might get by using volume replication at each site, leaving the RW
>> volume wherever it is, and relying on some tricks to keep the user
>> confused during writes...i think, in general, the idea of moving RW
>> volumes all the time is a bad one...it will put big loads on your
>> fileservers if you have lots of users
> 
> For the most part, yes.  We try to keep the large stuff (databases) separately and have the db apps take care of synchronization in their own way.
> 
> I thought as much, but I'm getting very poor performance even for reading.  It seems that the clients don't do a good job of prioritizing the local fileserver first.  Some posts mention lower-numbered IP addresses getting priority, but I'm not sure how that would work for servers on different subnets.  Shouldn't the choice have something to do with lower latency?

In theory the answer is 'yes'.  Unfortunately, there is no logic in the
client to determine what the latency is being any given set of
connections.   The server preferences can be manually set using the
"fs setserverprefs" command

  http://www.openafs.org/pages/manpages/1/fs_setserverprefs.html

You can provide your users with scripts to run to prioritize the client
for their current location.

>> - check the top of your afs tree and make sure root.afs, root.cell and
>> probably most other volumes at the top of the tree contain only
>> mountpoints, and are all replicated, and don't change them much...in
>> other words, you want stability at the top of the tree, to reduce the
>> changes that clients have to pay attention to
> 
> These are all mounted R/O, right?  Should mostly-static volumes be mounted R/O and then only access the R/W vol through /afs/.domain/ for the occasional updates?  What about for the software repository scenario below?
> 
> The way I understand the volume access is that even if a volume is mounted R/W, its R/O vol is accessed until a write operation, in which case all further access (including reads) are from the R/W vol.  If so, then a user should ideally be able to use the local server for most reads.

There are two types of mount points:  normal and read-write.   If all of
the mount points are normal, then a readonly replica is preferred over a
read-write volume.   If there is no readonly replica, then the readwrite
is used.

Once the user crosses an explicit read-write mount point, then the
client will always prefer the read-write volume over the readonly.

> We haven't made any changes to the default cache options, so maybe this would be a good next step?

You will need to tune the cache for your users usage patterns.  At the
very least the size of the cache should be large enough to hold the
working set required by the applications being used.

>> how many users? what kinds of files and what kinds of usage? how much
>> sharing? these can make a difference in how you set things up, too. (i
>> worked on a cell once with clients in sweden, south america, and on the
>> european continent...one server at each site, whenever the south
>> american site became master db server, followed by transatlantic link
>> going down...the european users weren't happy....basically, as long as
>> you have good network connectivity between client systems and server
>> systems, there is no need for physical proximity...)
> 
> We've got about 5 users, so a pretty small setup.  They're mostly very non-technical, and expect it to Just Work (tm).
> 
> The files regular users are concerned with are mostly documents, images, and presentations.  Their heavy sharing is usually confined to a small set of small files (usually less that 1MB each); sets change as projects change (weekly/monthly).  Still, it takes about 1 minute to transfer 1MB across the VPN -- not insignificant.

What operating system?

Jeffrey Altman

--------------ms090604030906000801050202
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJeTCC
AxcwggKAoAMCAQICEDsE+kRcmomW1hYG6BoqhGEwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UE
BhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMT
I1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA4MDUzMDE5MTUyOVoX
DTA5MDUzMDE5MTUyOVowczEPMA0GA1UEBBMGQWx0bWFuMRUwEwYDVQQqEwxKZWZmcmV5IEVy
aWMxHDAaBgNVBAMTE0plZmZyZXkgRXJpYyBBbHRtYW4xKzApBgkqhkiG9w0BCQEWHGphbHRt
YW5Ac2VjdXJlLWVuZHBvaW50cy5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQCtf5bVJdYFtHIrV2XALpA5oaMu7FPYU7RP7vJhd8Cu9Kd9ud2crX2pHK4avuPaYb4Vg9qI
zPrPadePhJ3OWwNt1ZlUlpc5URnOfpg/I9iymZBUSnCFVLuIvoncacqyUlzqdYEF8XGEoEL6
6bj8uoCSX0D7ZjZiAS8993NvgiPYpf10acMyWQ4max+P7Wg9T03Nw2F6EsmP6gWxBRsekTXe
N6QjJdvaK0846lDqeBFoCEzIUMQXj2kiXVPCPEdxPc/L1sDMYf0GLaDIg8qyThpGd0X6DwfK
3RWcMy8DV7Q5Z+jSEdPn5X0l4anOTrjr3IwE57MC3bVs0EEpUODTzftnAgMBAAGjOTA3MCcG
A1UdEQQgMB6BHGphbHRtYW5Ac2VjdXJlLWVuZHBvaW50cy5jb20wDAYDVR0TAQH/BAIwADAN
BgkqhkiG9w0BAQUFAAOBgQA9kndmeLrdQOUbhNGGms/FnfDyraH4OjA4PIIMOCbGWK0YXczs
/Fqn4XkT70SG4s8v4Zg6TaAcJrZBVcZQXyzrhlF2Zev/g69zZMHQe+2r4i/3FBVKAtFCoea1
vgwJ5TfZYlKvt4D0Z4zexu9Y0VwCIR4plWjVD76zC2CGB/2fhjCCAxcwggKAoAMCAQICEDsE
+kRcmomW1hYG6BoqhGEwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoT
HFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25h
bCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA4MDUzMDE5MTUyOVoXDTA5MDUzMDE5MTUyOVow
czEPMA0GA1UEBBMGQWx0bWFuMRUwEwYDVQQqEwxKZWZmcmV5IEVyaWMxHDAaBgNVBAMTE0pl
ZmZyZXkgRXJpYyBBbHRtYW4xKzApBgkqhkiG9w0BCQEWHGphbHRtYW5Ac2VjdXJlLWVuZHBv
aW50cy5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCtf5bVJdYFtHIrV2XA
LpA5oaMu7FPYU7RP7vJhd8Cu9Kd9ud2crX2pHK4avuPaYb4Vg9qIzPrPadePhJ3OWwNt1ZlU
lpc5URnOfpg/I9iymZBUSnCFVLuIvoncacqyUlzqdYEF8XGEoEL66bj8uoCSX0D7ZjZiAS89
93NvgiPYpf10acMyWQ4max+P7Wg9T03Nw2F6EsmP6gWxBRsekTXeN6QjJdvaK0846lDqeBFo
CEzIUMQXj2kiXVPCPEdxPc/L1sDMYf0GLaDIg8qyThpGd0X6DwfK3RWcMy8DV7Q5Z+jSEdPn
5X0l4anOTrjr3IwE57MC3bVs0EEpUODTzftnAgMBAAGjOTA3MCcGA1UdEQQgMB6BHGphbHRt
YW5Ac2VjdXJlLWVuZHBvaW50cy5jb20wDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQUFAAOB
gQA9kndmeLrdQOUbhNGGms/FnfDyraH4OjA4PIIMOCbGWK0YXczs/Fqn4XkT70SG4s8v4Zg6
TaAcJrZBVcZQXyzrhlF2Zev/g69zZMHQe+2r4i/3FBVKAtFCoea1vgwJ5TfZYlKvt4D0Z4ze
xu9Y0VwCIR4plWjVD76zC2CGB/2fhjCCAz8wggKooAMCAQICAQ0wDQYJKoZIhvcNAQEFBQAw
gdExCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUg
VG93bjEaMBgGA1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNVBAsTH0NlcnRpZmljYXRp
b24gU2VydmljZXMgRGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFp
bCBDQTErMCkGCSqGSIb3DQEJARYccGVyc29uYWwtZnJlZW1haWxAdGhhd3RlLmNvbTAeFw0w
MzA3MTcwMDAwMDBaFw0xMzA3MTYyMzU5NTlaMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxU
aGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwg
RnJlZW1haWwgSXNzdWluZyBDQTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAxKY8VXNV
+065yplaHmjAdQRwnd/p/6Me7L3N9VvyGna9fww6YfK/Uc4B1OVQCjDXAmNaLIkVcI7dyfAr
hVqqP3FWy688Cwfn8R+RNiQqE88r1fOCdz0Dviv+uxg+B79AgAJk16emu59l0cUqVIUPSAR/
p7bRPGEEQB5kGXJgt/sCAwEAAaOBlDCBkTASBgNVHRMBAf8ECDAGAQH/AgEAMEMGA1UdHwQ8
MDowOKA2oDSGMmh0dHA6Ly9jcmwudGhhd3RlLmNvbS9UaGF3dGVQZXJzb25hbEZyZWVtYWls
Q0EuY3JsMAsGA1UdDwQEAwIBBjApBgNVHREEIjAgpB4wHDEaMBgGA1UEAxMRUHJpdmF0ZUxh
YmVsMi0xMzgwDQYJKoZIhvcNAQEFBQADgYEASIzRUIPqCy7MDaNmrGcPf6+svsIXoUOWlJ1/
TCG4+DYfqi2fNi/A9BxQIJNwPP2t4WFiw9k6GX6EsZkbAMUaC4J0niVQlGLH2ydxVyWN3amc
OY6MIE9lX5Xa9/eH1sYITq726jTlEBpbNU1341YheILcIRk13iSx0x1G/11fZU8xggNkMIID
YAIBATB2MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5
KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQQIQ
OwT6RFyaiZbWFgboGiqEYTAJBgUrDgMCGgUAoIIBwzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcN
AQcBMBwGCSqGSIb3DQEJBTEPFw0wOTAxMTUyMDM3NDJaMCMGCSqGSIb3DQEJBDEWBBT4y7/X
rLPBqAURjzoFM9TSnoPOdzBSBgkqhkiG9w0BCQ8xRTBDMAoGCCqGSIb3DQMHMA4GCCqGSIb3
DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDCBhQYJKwYB
BAGCNxAEMXgwdjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcg
KFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3Vpbmcg
Q0ECEDsE+kRcmomW1hYG6BoqhGEwgYcGCyqGSIb3DQEJEAILMXigdjBiMQswCQYDVQQGEwJa
QTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhh
d3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECEDsE+kRcmomW1hYG6BoqhGEwDQYJ
KoZIhvcNAQEBBQAEggEABKbNZ0NLcT5ALJXcNWsrJWI/EJAASugeDavlDNok+nJkYC2ID1BO
NABkm8p+7Ql6sKzHb6ayTpvIbchZHzruk1LXV444uo8RR73AXbTDY7PGUYjB7yuk1sfljosL
FjkKvGU2IH+H5xIvyBDMt7KN23KprPsTbRm/u1m97eIVsOHoX/lSu/nT7wEsn+0Kr/U3u6eo
KNhhnOA+RgVlbMWn2vozqBBJ+QmVVsyh6UhuU7rAuakgsP9GuxX9j0JMq5vwuHD6XF3ztI25
zb55RQ7bLwZ7dpGutpV5DPtApE3sNTFXrJQrxPFWr95cOr7CbXq1LS66AL7UCVLsYGWZyTJi
tgAAAAAAAA==
--------------ms090604030906000801050202--