[OpenAFS] Second-generation Volume Shadows at UMich

[Thomas Kula]

The following is a status update on where we are at with volume
shadows at UMich. Both Michael Garrison and I will be at the
Workshop starting Wednesday; feel free to hunt us down and ask
any questions you may have.


Second Generation Volume Shadows at the 
University of Michigan

= What are Shadows? =

Volume shadows are a mechanism that allow you to create
a copy of a volume on a different file server. These
volumes are identical to their source volume (as of
the time of creation) with the following exception:

 - The VolIsShadow bit is set
 - These volumes do not appear in the VLDB
 
= What are Shadows not? =

Volume shadows are *NOT* read-write replicas of volumes.
They are a static copy of a volume made at a single point
in time.

= What can you do with Shadows? =

- vos shadow <volid> -toserver <dest server> -topartition \
      <dest partition> [-fromserver <source server> \
      -frompartition <source partition> -incremental ]
      
  Creates a shadow of a volume to <dest server> on <dest 
  partition>. Optional flags allow you to choose the server to
  shadow from (put there because we could). If a shadow 
  already exists on <dest server>, <dest partition>, the 
  shadow is updated.
  
  The major change since generation one is making fromserver
  and frompartition options --- vos shadow will query the 
  VLDB for that information if it is not supplied.
  
  If you do not supply -incremental, a full copy will be 
  made, even if a destination shadow volume exists. If you
  supply -incremental, an incremental update is made (if no
  destination exist, a full copy will be made instead).
  
- vos listvol ... [-forceshadows -onlyshadows]

  The vos listvol command has been modified to *not* list volumes
  with the VolIsShadow bit set, *unless* -forceshadows is given.
  
  You can also get a list of only the shadows if -onlyshadows is
  given. This allows you do a shadow "refresh" operation; see more
  about this later.
  
- vos syncvldb ... [-forceshadows]
  
  vos syncvldb has been modified to *not* sync shadow volumes on a
  fileserver with the VLDB *unless* the -forceshadows flag is given.
  If -forceshadows is given, and shadow volumes are "promoted" and
  listed in the VLDB, the VolIsShadow flag is cleared.
  
- Additional changes:

  The fileserver allows only members of system:administrators access 
  to shadow volumes, all other users (regardless of volume ACLs) are
  returned EACCESS.
  
= Change from 'Generation One Shadows' =

Generation one was a "push" model: each source fileserver would have
to know where to shadow stuff to, and push out changes at an 
appropriate time. This generation is "pull" --- we will tell fileservers
to go refresh any shadows they may have. We went with pull because
we already handily have a database of the source volumes, the VLDB.
  
= Our planned usage of shadows =

Each volume will be shadowed to another fileserver (via several
mechanisms: our provisioning system will be changed to create a
shadow of the volume at creation time, our existing nightly sanity
check script will make sure each volume is shadowed to an
appropriate fileserver, etc.) In our operation, we will have 
dedicated "shadow servers" --- fileservers that have shadows.

Depending on how things work operationally, each fileserver will
be told every N minutes (determined by load, etc.) to refresh all
of its shadows. We're working on testing currently to see how 
frequently we can refresh shadows; we plan on doing it at least
once a day and ideally would run as frequently as we can and not
have an operational impact. I think having shadows no more than 
a few hours old would be our goal.

In the event of an extended fileserver outage (policy, still to
be determined), shadow copies of the volumes on the downed fileserver
will be "promoted", will show up in the VLDB and will be available
for use. If our shadowing goal is met, this means we can quickly 
(as long as the syncvol takes) return these volumes to service,
and have them be no more than a few hours old.

= Status of second generation shadows =

http://rt.central.org/rt/index.html?q=64267
Look for "volIsShadow patch; second generation"
Applies to stock 1.4.7.

Went into production on internal staff user volumes last
week. Code going into production on new fileservers we are
bringing online in the next few weeks, but no shadows will
be made of non-staff volumes until more testing is done.
Preliminary testing has shadow operations going fairly quickly,
but we want to do more testing varying volume size, number of
files, and size of deltas between shadow operations.

= What we'd like =

Feedback, consensus from other users (are we doing something sane?)
testing by other users.


--
Thomas L. Kula | tkula@umich.edu | 734.764.6531
University of Michigan - ITCS - UMCE
GPCC, IFS/IAA, Hostmaster