[OpenAFS] OpenAFS on windows - profile in AFS, who uses it?
Rodney M. Dyer
rmdyer@uncc.edu
Sun, 10 Feb 2008 15:39:19 -0500
At 06:42 AM 2/10/2008, Lars Schimmer wrote:
>Biggest problems:
>
>1. slowness - Users reports of LONG loading times for 400-1000 MB roaming
>profiles, up to 20 minutes
Yeowie! I guess I should chime in at this point.
Our experience with roaming profiles in AFS.
We started using roaming profiles really early on, and are still using them
on XP. It works, but not without issues. Most of the issues are not
caused by AFS, but are caused by general profile problems that exist on any
Microsoft network. Roaming profiles, as positive as they are, are so badly
implemented by Microsoft that I just want to scream sometimes.
What is required to use roaming profiles in AFS?
1. A profile folder under the AFS users account.
So you might have something like:
/afs/cell/usr/joe/roaming_profile
(This folder should be secured so that only the user and the AFS
administrators have access!!)
2. You need to install the OpenAFS client with Logon Authentication "enabled".
3. You need to specify in the users account properties "Profile Path"
where the roaming profile is.
This can be done with a UNC path
"\\afs\cell\usr\joe\roaming_profile", or a
"drive:\dir\dir\dir\roaming_profile".
IMPORTANT! Using the "drive\:path" specification won't work without
a drive that "pre-exists" before the user logs on. Microsoft has no method
of mounting a drive to AFS before the user logs on. So you must do the
following...
a. Create a system boot script that mounts a "global drive" as user
SYSTEM.
b. Or, hack the "afslogon.dll" logon authentication provider to
"mount" a drive while the logon authentication is taking place.
c. Use the "Global Drive" mapping option of the AFS configuration
panel (this will be deprecated soon however).
All methods A ,B, and C are frowned upon because of the following reasons...
A global drive mounted as the SYSTEM user is not "officially" allowed
by Microsoft. In fact, there is documentation to the effect in the MSDN
that you should never mount a drive as SYSTEM. This makes sense for
various technical reasons, especially in Terminal Server environments. I
won't go into those reasons here.
However this being stated, it does in fact still work on XP for a
single user desktop, and there are various "positive" reasons for doing
so. I am being a devils advocate for the global drive until such time as
the AFS client has certain problems fixed that will enable me to use UNC
paths correctly. These problems I mention here about AFS have more to do
with "folder redirection" than roaming profiles. I will talk about folder
redirection later.
The "Global Drive" mapping option still exists on the
"afs_config.exe" panel as of the latest OpenAFS client version, but it will
be removed at some point in the future. The mapping option allows the AFS
service to setup the global drive whenever the service starts (since the
service runs as user SYSTEM).
We still use a global drive mounted to AFS at boot, and we have
regularly scheduled watchdog "task scripts" via the "at" command that check
and verify whether AFS is still working, and that the global drive still
exists. These scripts run every three hours, plus we perform the same
sanity checks when the user logs on with our logon scripting.
When a user is logged on, a global drive cannot be unmounted by the
user which is nice. (This assumes they aren't an administrator.)
If you mount a global drive in a system boot script, you might use
the following logic:
(In XP command shell BATCH language...)
::---------------------------------------------------------------------------
:: mountafs (subroutine)
:: Tries to mount the AFS global drive.
:: Returns 0 on success.
:: Returns 1 on failure.
::---------------------------------------------------------------------------
:mountafs
echo Mounting global AFS drive...
for /l %%i in (1,1,100) do (
net use n: \\afs\all /PERSISTENT:NO >> nul 2>>&1
if exist "%afs_test_folder%" (
echo done.
exit /b 0
)
sleep 3
echo .
)
echo error.
echo Unable to mount AFS global drive.
exit /b 1
You will notice that this script mounts AFS from the root. The
"afs_test_folder" environment variable contains a test path down your AFS
tree that the script can test for to verify that AFS is mounted. You would
call the above routine with:
call :mountafs || (
:: code for failure goes here...
)
Hacking the "afslogon.dll" to perform logon scripting for mounting global
drives is not something everyone is competent to do, and is generally
frowned upon by the OpenAFS develpers as well. Of course it is open
source, so you can do anything you want! ;)
Once you have a global drive mounted at startup (or at logon time), and you
have the users profile path configured in the account properties on
Windows, and you have "logon authentication" enabled on the AFS
configuration, then you are ready to try using roaming profiles. Now on to
the complications...
When you logon the first time Windows sees that you have no profile
"contents" in your AFS profile folder. Windows will then give you a
profile on the local box under "c:\documents and settings\username". When
you logout, Windows saves any changes made during your session to the AFS
profile folder. This is a "sync", only the files that have changed are
saved. The first time, all files will be saved.
Now, after the user logs out, you probably want to clear the local profile
folder right? So you set a active directory group policy to do so:
"Delete cached copies of roaming profiles" Enabled
Most of the time, this will work. But many times it just doesn't. THIS IS
THE NUMBER ONE BIGGEST PROBLEM THAT WE HAVE HAD WITH ROAMING PROFILES IN
OUR ENVIRONMENT. Windows tries to save the profile (which is does), then
"unload" the local profile. The "unloading" is very problematic. The main
reasons that profiles will not unload is because some application or
service still has an open handle or lock on a file or a registry setting
within the users profile. So Microsoft in their grand wisdom tried to come
up with a fix for this problem. It's called UPHClean, or "user profile
handle clean"'er. It's a system service that's supposed to "release" all
the locks and handles so the profile can be unloaded. However it only half
does it's job.
Why is this such a big problem you ask? Because if you have hundreds of
students/faculty/staff all logging in and out while moving from machine to
machine in a classroom/lab environment, then eventually you eat disk space
with all those "stranded" local profiles. Cleaning those stranded profiles
requires a "reboot" because Windows won't allow you to just delete them
while it has open handles or locks on them. And finding which file or
registry setting had the lock is annoying to say the least. We end up
performing a monthly reboot and clean within our regular
application/patching change managment system once a month.
Ok, off my rant now. I'm just voicing the fact that this is not an AFS
issue, while complaining about Microsofts profile system.
FOLDER REDIRECTION
Now, if you are going to going to use roaming profiles then you absolutely
should use "folder redirection". Here's why...
In the beginning, on NT 4.0, roaming profiles stored "a lot of
stuff". Profile sizes became huge as users began storing multi-megabyte
documents on their desktops. The "desktop" is a folder under the
profile. Microsoft, in their grand wisdom, also made the "My Documents",
and "Application Data" folders under the profile. So the situation you
have is that ever time the files change on the desktop, "my documents", or
"Application Data" folders, the files would be "sync'ed" back to the
network (in our case AFS).
Because of the size of the profiles, and the time it takes to copy data
back and forth across the network, many enterprise corporate Windows admins
began to complain to Microsoft. So, when Windows 2000 rolls out, Microsoft
makes a change to the profile system. Using the new "intelli-mirror"
services the "Desktop", "My Documents", and "Application Data" folders, as
well as other items in the profile, can be "redirected" to the network
directly instead. This means that if someone saves something on the
desktop, Windows will auto-magically save it out to the network. Folder
redirection is very nice. Now, profile sizes are small and light, and
logon times are short. With Folder Redirection enabled, profile sizes in
general should never be more than 10 to 20 meg (assuming you've also setup
a policy that cleans up IE cookies and the recent documents list).
Now for the part that made me scream when I began to setup "Folder
Redirection". First, folder redirection is a "group policy" setup on the
Active Directory server. No problem, we use AD to group all our XP
desktops under one administrative domain (and it is in a cross realm
(trust) relationship with our MIT KDC). But the group policy configuration
for "Folder Redirection" is a pain.
Let's look at the technical design first...
1. In our AFS cell, our user directories are stored as:
/afs/cell/usr/a/username
/afs/cell/usr/b/username
/afs/cell/usr/c/username
etc.
and some are:
/afs/cell/usr1/username
/afs/cell/usr2/username
/afs/cell/usr3/username
etc.
This was done for a few reasons. The biggest is that the history of the
AFS client was that it was "hard" (time consuming) to "stat" huge numbers
of items under a single directory. Because we had thousands of users we
needed to break them up into groups of smaller numbers under a single folder.
Now, the profiles are stored under...
/afs/cell/usr/a/username/xp_profile
So with Folder Redirection we needed another place to store the redirected
folders right? So let's create that new location...
/afs/cell/usr/a/username/pc/win_data/Desktop
/afs/cell/usr/a/username/pc/win_data/My Documents
/afs/cell/usr/a/username/pc/win_data/Application Data
We chose "Basic Redirection" and to "Redirect to the following location"
options in the folder redirection policy setup on the group policy config.
Now, the group policy setup allows the use of certain environment variables
you can use. From the doc...
"Folder Redirection and environment variables
The folder redirection client side extension is only able to process
two environment variables: %username% and %userprofile%. Other environment
variables such as %logonserver%, %homedrive% and %homepath% will not work
with folder redirection." ... From the "Windows XP, User Data and Settings
Management" document (2001,2002).
Ok the "%userprofile%" variable always points to the local machines profile
folder, so that's useless. So the only variable we have left is
"%username%". How am I supposed to setup folder redirection? I can't use:
"n:\cell\usr\a\%username%\pc\win_data\Desktop"
That won't work since the parent folders are different for every user.
There are a couple of ways around this. The first is simply to not setup
your users like we did. The second is to setup a "symlink"
tree. Something like...
/afs/cell/all_users/username1 -> /afs/cell/usr/a/username
/afs/cell/all_users/username2 -> /afs/cell/usr/b/username
/afs/cell/all_users/username3 -> /afs/cell/usr/c/username
etc.
The problem here is that you need to set this up during your automated
account creation practices. This may, or may not be any significant
problem in your IT shop.
There is a new "undocumented" third way that I might mention here. The
latest OpenAFS Windows client has the unique ability to access a users
volume directly by volume name. You must have the "freelance" option
enabled to do this, and it goes somthing like this...
\\afs\cell#user_volume\dir1\dir2\...
So in our case, we would put something like this in the group policy config
for folder redirection...
"Desktop" redirected to
"\\afs\uncc#user.volume\pc\win_data\Desktop"
"My Documents" redirected to
"\\afs\uncc#user.volume\pc\win_data\My Documents"
"Application Data" redirected to
"\\afs\uncc#user.volume\pc\win_data\Application Data"
This is nice, and I may switch over to it, but I don't as yet have the
latest client installed, and we haven't tested it enough.
In the meantime I will now describe the "dirty" interim solution I came up
with. I hacked the "afslogon.dll" to perform certain scripting when the
user logs on. The script runs as the user SYSTEM, and has the ability to
get a user token while being SYSTEM. This allows me to do a lot of things.
1. I grep the users home directory out of our password file.
2. I setup a "u:" drive that is mounted to the users home directory.
3. I do things like create and verify the following profile folders exist
before the logon takes place...
u:\xp_profile
u:\pc\win_data\Desktop
u:\pc\win_data\My Documents
u:\pc\win_data\Application Data
4. On the group policy configuration for folder redirection I just use the
folders from the "U:\" drive as they are.
Once the U: drive exists, the folder redirection works without issues. The
user is then logged on by Windows and once logged on, I have a script that
runs that unmounts the "U:\" drive as SYSTEM, then remounts it properly as
the user under their own account.
This is messy I know, but we've been doing it this way since 2002 just
fine, and for thousands of users. As time progresses, I'm trying to remove
more and more of the hacks that I've needed to put into place to solve
these annoying Windows networking issues as they deal with AFS.
There is one last way to deal with the "folder redirection" issue, that
still involves hacking, but does not use the new "direct to volume" AFS
syntax ( the \\afs\cell#user_volume\path ) that I outlined above. It
involves creating a "symlink" to the users home volume under the
"\\afs\all" mount. You can only do this with "freelance" enabled, and only
as administrator. So basically we would, during our hacked "afslogon.dll"
script, and in real time, during the logon, create the "symlink" for the
user logging on. So we have something like...
(Note: Unfortunately you need a drive mounted for this because you
can't create a symlink at the root of a UNC path using the existing symlink
command syntax.)
Mount a global drive to AFS: net use n: \\afs\all
CD to the global drive root: N:
cd N:\
Now create the link:
N:\>symlink make username /afs/cell/usr/a/userame
Now you can use the "\\afs\%username%\path" variable in the folder
redirection path on the AD group policy.
Problems with Roaming User Profiles and Folder Redirection
Using roaming profiles and folder redirection can be a royal pain.
1. Not all application vendors understand folder redirection so they
sometimes end up storing files under "c:\documents and
settings\username\Application Data" anyway. This is annoying, especially
if the application vendors application is a service. We've had a number of
applications that continue to store data in the local path, not the
redirected one.
2. A users profile has a folder under it called "Local Settings". THIS
FOLDER DOES NOT ROAM. This folder only exists during your session on the
local machine. When you logout, the data in that folder is considered
temporary for your session. Microsoft in further "grand wisdom" decided to
store valuable information in that folder that you really need to carry
around with you with the profile, but this data is "excluded by
default". Notable application data includes:
Microsoft Outlook email settings and PST files, etc..
Microsoft IE history, etc..
Microsoft Visual Studio .NET option settings,etc.
This creates an ugly situation that must be remedied or users will be angry
that not all their settings follow them around. I did something ugly to
fix this in scripting. Unfortunately it is again a hack around, so I won't
mention it here.
3. For some reason the folder redirection doesn't always work. This can
get really UGLY, as you end up with some settings and files in the profile,
and some in the redirected folders over time. The reasons are varied for
this situation, but mostly involve the group policy refesh from the AD
server not working.
4. Token issues. YOU MUST HAVE A VAILID TOKEN to save into
AFS...obviously. Yet Windows users are oblivious to the fact that the
token times-out. The model under windows is that once you logon, you stay
logged on. Microsoft, in their infinite wisdom, even refreshes its own
kerb ticket for you to prevent you from needing to understand
tickets. Unless you are running the Kfw package or NIM, you don't get that
with tokens. The uglyness happens when a user loses the token during a
session, then tries to logout. Windows can't save the profile. So the
profile becomes "stranded" on that machine. Also if you are using Folder
Redirection and you lose a token then the Explorer Shell doesn't like that
at all. Not everything will refresh properly even if you are able to get
your token back.
5. AFS client service issues (afsd_service.exe). The client service has
had a history of crashing from time to time, although recently it is much
more robust. When the service crashes you obviously lose access to AFS and
all the problems that occur are like issue #4, except worse.
6. Profiles saved under Windows contain unicode characters.
Profile and redirected folders don't currently work well with the AFS
client because it doesn't support unicode file names. This isn't a huge
issue, but sometimes this comes into play for our foreign students who save
web browser items in AFS, or IE does with the history, etc. Sometimes in
our environment the profile of a user will not save/load because the
filename has strange characters in it. Our help desk ends up fixing each
of these users by hand.
7. With the redirected folders (esp desktop), the explorer shell folder
refresh of newly created, or changing file icons may not work well, if at
all. This is also because of the lack of support for CIFS unicode (if you
are using UNC paths for the profile or redirected folder names).
I could go on, but i've just burned half my day typing this and I think
I've gone much further than I expected. Profiles do work with AFS, and if
properly cared for they work fine, and are no worse than any other Windows
networking setup. Recently however our profile folder issues have been
increasing more and more because of the inability of Windows to let go of
them at unload, so they can't be removed, which causes many problems for us.
I will try to answer any specific questions if you have any.
Enjoy,
Rodney
Rodney M. Dyer
Operations and Systems (Specialist)
Mosaic Computing Group
William States Lee College of Engineering
University of North Carolina at Charlotte
Email: rmdyer@uncc.edu
Web: http://www.coe.uncc.edu/~rmdyer
Phone: (704)687-3518
Help Desk Line: (704)687-3150
FAX: (704)687-2352
Office: Cameron Hall, Room 232