[OpenAFS] OpenAFS on windows - profile in AFS, who uses it?

Rodney M. Dyer rmdyer@uncc.edu
Sun, 10 Feb 2008 15:39:19 -0500

At 06:42 AM 2/10/2008, Lars Schimmer wrote:
>Biggest problems:
>1. slowness - Users reports of LONG loading times for 400-1000 MB roaming 
>profiles, up to 20 minutes

Yeowie!  I guess I should chime in at this point.

Our experience with roaming profiles in AFS.

We started using roaming profiles really early on, and are still using them 
on XP.  It works, but not without issues.  Most of the issues are not 
caused by AFS, but are caused by general profile problems that exist on any 
Microsoft network.  Roaming profiles, as positive as they are, are so badly 
implemented by Microsoft that I just want to scream sometimes.

What is required to use roaming profiles in AFS?

1.  A profile folder under the AFS users account.

      So you might have something like:


          (This folder should be secured so that only the user and the AFS 
administrators have access!!)

2.  You need to install the OpenAFS client with Logon Authentication "enabled".

3.  You need to specify in the users account properties "Profile Path" 
where the roaming profile is.

      This can be done with a UNC path 
"\\afs\cell\usr\joe\roaming_profile", or a 

      IMPORTANT!  Using the "drive\:path" specification won't work without 
a drive that "pre-exists" before the user logs on.  Microsoft has no method 
of mounting a drive to AFS before the user logs on.  So you must do the 

      a.  Create a system boot script that mounts a "global drive" as user 

      b.  Or, hack the "afslogon.dll" logon authentication provider to 
"mount" a drive while the logon authentication is taking place.

      c.  Use the "Global Drive" mapping option of the AFS configuration 
panel (this will be deprecated soon however).

All methods A ,B, and C are frowned upon because of the following reasons...

      A global drive mounted as the SYSTEM user is not "officially" allowed 
by Microsoft.  In fact, there is documentation to the effect in the MSDN 
that you should never mount a drive as SYSTEM.  This makes sense for 
various technical reasons, especially in Terminal Server environments.  I 
won't go into those reasons here.

      However this being stated, it does in fact still work on XP for a 
single user desktop, and there are various "positive" reasons for doing 
so.  I am being a devils advocate for the global drive until such time as 
the AFS client has certain problems fixed that will enable me to use UNC 
paths correctly.  These problems I mention here about AFS have more to do 
with "folder redirection" than roaming profiles.  I will talk about folder 
redirection later.

      The "Global Drive" mapping option still exists on the 
"afs_config.exe" panel as of the latest OpenAFS client version, but it will 
be removed at some point in the future.  The mapping option allows the AFS 
service to setup the global drive whenever the service starts (since the 
service runs as user SYSTEM).

      We still use a global drive mounted to AFS at boot, and we have 
regularly scheduled watchdog "task scripts" via the "at" command that check 
and verify whether AFS is still working, and that the global drive still 
exists.  These scripts run every three hours, plus we perform the same 
sanity checks when the user logs on with our logon scripting.

       When a user is logged on, a global drive cannot be unmounted by the 
user which is nice. (This assumes they aren't an administrator.)

      If you mount a global drive in a system boot script, you might use 
the following logic:

       (In XP command shell BATCH language...)

         :: mountafs (subroutine)
         :: Tries to mount the AFS global drive.
         :: Returns 0 on success.
         :: Returns 1 on failure.


                 echo Mounting global AFS drive...
                 for /l %%i in (1,1,100) do (
                         net use n: \\afs\all /PERSISTENT:NO >> nul 2>>&1
                         if exist "%afs_test_folder%" (
                                 echo done.
                                 exit /b 0
                         sleep 3
                         echo .

                 echo error.
                 echo Unable to mount AFS global drive.

                 exit /b 1

You will notice that this script mounts AFS from the root.  The 
"afs_test_folder" environment variable contains a test path down your AFS 
tree that the script can test for to verify that AFS is mounted.  You would 
call the above routine with:

         call :mountafs || (
                 :: code for failure goes here...

Hacking the "afslogon.dll" to perform logon scripting for mounting global 
drives is not something everyone is competent to do, and is generally 
frowned upon by the OpenAFS develpers as well.  Of course it is open 
source, so you can do anything you want!  ;)

Once you have a global drive mounted at startup (or at logon time), and you 
have the users profile path configured in the account properties on 
Windows, and you have "logon authentication" enabled on the AFS 
configuration, then you are ready to try using roaming profiles.  Now on to 
the complications...

When you logon the first time Windows sees that you have no profile 
"contents" in your AFS profile folder.  Windows will then give you a 
profile on the local box under "c:\documents and settings\username".  When 
you logout, Windows saves any changes made during your session to the AFS 
profile folder.  This is a "sync", only the files that have changed are 
saved.  The first time, all files will be saved.

Now, after the user logs out, you probably want to clear the local profile 
folder right?  So you set a active directory group policy to do so:

      "Delete cached copies of roaming profiles"            Enabled

Most of the time, this will work.  But many times it just doesn't.  THIS IS 
OUR ENVIRONMENT.  Windows tries to save the profile (which is does), then 
"unload" the local profile.  The "unloading" is very problematic.  The main 
reasons that profiles will not unload is because some application or 
service still has an open handle or lock on a file or a registry setting 
within the users profile.  So Microsoft in their grand wisdom tried to come 
up with a fix for this problem.  It's called UPHClean, or "user profile 
handle clean"'er.  It's a system service that's supposed to "release" all 
the locks and handles so the profile can be unloaded.  However it only half 
does it's job.

Why is this such a big problem you ask?  Because if you have hundreds of 
students/faculty/staff all logging in and out while moving from machine to 
machine in a classroom/lab environment, then eventually you eat disk space 
with all those "stranded" local profiles.  Cleaning those stranded profiles 
requires a "reboot" because Windows won't allow you to just delete them 
while it has open handles or locks on them.  And finding which file or 
registry setting had the lock is annoying to say the least.  We end up 
performing a monthly reboot and clean within our regular 
application/patching change managment system once a month.

Ok, off my rant now.  I'm just voicing the fact that this is not an AFS 
issue, while complaining about Microsofts profile system.


Now, if you are going to going to use roaming profiles then you absolutely 
should use "folder redirection".  Here's why...

In the beginning, on NT 4.0, roaming profiles stored "a lot of 
stuff".  Profile sizes became huge as users began storing multi-megabyte 
documents on their desktops.  The "desktop" is a folder under the 
profile.  Microsoft, in their grand wisdom, also made the "My Documents", 
and "Application Data" folders under the profile.  So the situation you 
have is that ever time the files change on the desktop, "my documents", or 
"Application Data" folders, the files would be "sync'ed" back to the 
network (in our case AFS).

Because of the size of the profiles, and the time it takes to copy data 
back and forth across the network, many enterprise corporate Windows admins 
began to complain to Microsoft.  So, when Windows 2000 rolls out, Microsoft 
makes a change to the profile system.  Using the new "intelli-mirror" 
services the "Desktop", "My Documents", and "Application Data" folders, as 
well as other items in the profile, can be "redirected" to the network 
directly instead.  This means that if someone saves something on the 
desktop, Windows will auto-magically save it out to the network.  Folder 
redirection is very nice.  Now, profile sizes are small and light, and 
logon times are short.  With Folder Redirection enabled, profile sizes in 
general should never be more than 10 to 20 meg (assuming you've also setup 
a policy that cleans up IE cookies and the recent documents list).

Now for the part that made me scream when I began to setup "Folder 
Redirection".  First, folder redirection is a "group policy" setup on the 
Active Directory server.  No problem, we use AD to group all our XP 
desktops under one administrative domain (and it is in a cross realm 
(trust) relationship with our MIT KDC).  But the group policy configuration 
for "Folder Redirection" is a pain.

Let's look at the technical design first...

1.  In our AFS cell, our user directories are stored as:


and some are:


This was done for a few reasons.  The biggest is that the history of the 
AFS client was that it was "hard" (time consuming) to "stat" huge numbers 
of items under a single directory.  Because we had thousands of users we 
needed to break them up into groups of smaller numbers under a single folder.

Now, the profiles are stored under...


So with Folder Redirection we needed another place to store the redirected 
folders right?  So let's create that new location...

      /afs/cell/usr/a/username/pc/win_data/My Documents
      /afs/cell/usr/a/username/pc/win_data/Application Data

We chose "Basic Redirection" and to "Redirect to the following location" 
options in the folder redirection policy setup on the group policy config.

Now, the group policy setup allows the use of certain environment variables 
you can use.  From the doc...

      "Folder Redirection and environment variables
      The folder redirection client side extension is only able to process 
two environment variables: %username% and %userprofile%. Other environment 
variables such as %logonserver%, %homedrive% and %homepath% will not work 
with folder redirection." ... From the "Windows XP, User Data and Settings 
Management" document (2001,2002).

Ok the "%userprofile%" variable always points to the local machines profile 
folder, so that's useless.  So the only variable we have left is 
"%username%".  How am I supposed to setup folder redirection?  I can't use:


That won't work since the parent folders are different for every user.

There are a couple of ways around this.  The first is simply to not setup 
your users like we did.  The second is to setup a "symlink" 
tree.  Something like...

      /afs/cell/all_users/username1 -> /afs/cell/usr/a/username
      /afs/cell/all_users/username2 -> /afs/cell/usr/b/username
      /afs/cell/all_users/username3 -> /afs/cell/usr/c/username

The problem here is that you need to set this up during your automated 
account creation practices.  This may, or may not be any significant 
problem in your IT shop.

There is a new "undocumented" third way that I might mention here.  The 
latest OpenAFS Windows client has the unique ability to access a users 
volume directly by volume name.  You must have the "freelance" option 
enabled to do this, and it goes somthing like this...


So in our case, we would put something like this in the group policy config 
for folder redirection...

               "Desktop"  redirected to 
          "My Documents"  redirected to 
"\\afs\uncc#user.volume\pc\win_data\My Documents"
      "Application Data"  redirected to 
"\\afs\uncc#user.volume\pc\win_data\Application Data"

This is nice, and I may switch over to it, but I don't as yet have the 
latest client installed, and we haven't tested it enough.

In the meantime I will now describe the "dirty" interim solution I came up 
with.  I hacked the "afslogon.dll" to perform certain scripting when the 
user logs on.  The script runs as the user SYSTEM, and has the ability to 
get a user token while being SYSTEM.  This allows me to do a lot of things.

1.  I grep the users home directory out of our password file.
2.  I setup a "u:" drive that is mounted to the users home directory.
3.  I do things like create and verify the following profile folders exist 
before the logon takes place...

      u:\pc\win_data\My Documents
      u:\pc\win_data\Application Data

4.  On the group policy configuration for folder redirection I just use the 
folders from the "U:\" drive as they are.

Once the U: drive exists, the folder redirection works without issues.  The 
user is then logged on by Windows and once logged on, I have a script that 
runs that unmounts the "U:\" drive as SYSTEM, then remounts it properly as 
the user under their own account.

This is messy I know, but we've been doing it this way since 2002 just 
fine, and for thousands of users.  As time progresses, I'm trying to remove 
more and more of the hacks that I've needed to put into place to solve 
these annoying Windows networking issues as they deal with AFS.

There is one last way to deal with the "folder redirection" issue, that 
still involves hacking, but does not use the new "direct to volume" AFS 
syntax ( the \\afs\cell#user_volume\path ) that I outlined above.  It 
involves creating a "symlink" to the users home volume under the 
"\\afs\all" mount.  You can only do this with "freelance" enabled, and only 
as administrator.  So basically we would, during our hacked "afslogon.dll" 
script, and in real time, during the logon, create the "symlink" for the 
user logging on.  So we have something like...

      (Note:  Unfortunately you need a drive mounted for this because you 
can't create a symlink at the root of a UNC path using the existing symlink 
command syntax.)

      Mount a global drive to AFS:  net use n: \\afs\all
      CD to the global drive root:  N:
                                    cd N:\
      Now create the link:

          N:\>symlink make username /afs/cell/usr/a/userame

Now you can use the "\\afs\%username%\path" variable in the folder 
redirection path on the AD group policy.

Problems with Roaming User Profiles and Folder Redirection

Using roaming profiles and folder redirection can be a royal pain.

1.  Not all application vendors understand folder redirection so they 
sometimes end up storing files under "c:\documents and 
settings\username\Application Data" anyway.  This is annoying, especially 
if the application vendors application is a service.  We've had a number of 
applications that continue to store data in the local path, not the 
redirected one.

2.  A users profile has a folder under it called "Local Settings".  THIS 
FOLDER DOES NOT ROAM.  This folder only exists during your session on the 
local machine.  When you logout, the data in that folder is considered 
temporary for your session.  Microsoft in further "grand wisdom" decided to 
store valuable information in that folder that you really need to carry 
around with you with the profile, but this data is "excluded by 
default".  Notable application data includes:

      Microsoft Outlook email settings and PST files, etc..
      Microsoft IE history, etc..
      Microsoft Visual Studio .NET option settings,etc.

This creates an ugly situation that must be remedied or users will be angry 
that not all their settings follow them around.  I did something ugly to 
fix this in scripting.  Unfortunately it is again a hack around, so I won't 
mention it here.

3.  For some reason the folder redirection doesn't always work.  This can 
get really UGLY, as you end up with some settings and files in the profile, 
and some in the redirected folders over time.  The reasons are varied for 
this situation, but mostly involve the group policy refesh from the AD 
server not working.

4.  Token issues.  YOU MUST HAVE A VAILID TOKEN to save into 
AFS...obviously.  Yet Windows users are oblivious to the fact that the 
token times-out.  The model under windows is that once you logon, you stay 
logged on.  Microsoft, in their infinite wisdom, even refreshes its own 
kerb ticket for you to prevent you from needing to understand 
tickets.  Unless you are running the Kfw package or NIM, you don't get that 
with tokens.  The uglyness happens when a user loses the token during a 
session, then tries to logout.  Windows can't save the profile.  So the 
profile becomes "stranded" on that machine.  Also if you are using Folder 
Redirection and you lose a token then the Explorer Shell doesn't like that 
at all.  Not everything will refresh properly even if you are able to get 
your token back.

5.  AFS client service issues (afsd_service.exe).  The client service has 
had a history of crashing from time to time, although recently it is much 
more robust.  When the service crashes you obviously lose access to AFS and 
all the problems that occur are like issue #4, except worse.

6.  Profiles saved under Windows contain unicode characters.

    Profile and redirected folders don't currently work well with the AFS 
client because it doesn't support unicode file names.  This isn't a huge 
issue, but sometimes this comes into play for our foreign students who save 
web browser items in AFS, or IE does with the history, etc.  Sometimes in 
our environment the profile of a user will not save/load because the 
filename has strange characters in it.  Our help desk ends up fixing each 
of these users by hand.

7.  With the redirected folders (esp desktop), the explorer shell folder 
refresh of newly created, or changing file icons may not work well, if at 
all.  This is also because of the lack of support for CIFS unicode (if you 
are using UNC paths for the profile or redirected folder names).

I could go on, but i've just burned half my day typing this and I think 
I've gone much further than I expected.  Profiles do work with AFS, and if 
properly cared for they work fine, and are no worse than any other Windows 
networking setup.  Recently however our profile folder issues have been 
increasing more and more because of the inability of Windows to let go of 
them at unload, so they can't be removed, which causes many problems for us.

I will try to answer any specific questions if you have any.



Rodney M. Dyer
Operations and Systems (Specialist)
Mosaic Computing Group
William States Lee College of Engineering
University of North Carolina at Charlotte
Email: rmdyer@uncc.edu
Web: http://www.coe.uncc.edu/~rmdyer
Phone: (704)687-3518
Help Desk Line: (704)687-3150
FAX: (704)687-2352
Office:  Cameron Hall, Room 232