[OpenAFS-devel] Re: [OpenAFS] Volume root corruptions - anybody seen those?

[Hartmut Reuter]

Still too fast:

the correct patch is somewhat bigger and changes also the salvager 
because the salvager didn't ask with FSYNC_NEEDVOLUME.

Now the patch looks like that:

Small patch to make sure salvager and volserver cannot get the same
volume from the fileserver in parallel:



--- /afs/ipp/.cs/openafs/openafs-1.4.7-osd/src/vol/fssync.c 
2008-05-06 14:42:29.000000000 +0200
+++ ./fssync.c  2008-06-05 16:53:02.000000000 +0200
@@ -591,8 +591,23 @@
      case FSYNC_OFF:
      case FSYNC_NEEDVOLUME:{
             leaveonline = 0;
-           /* not already offline, we need to find a slot for newly 
offline volume */
             if (!v) {
+               /* not already offline by this handler */
+               if (command.command == FSYNC_NEEDVOLUME) {
+                   /* Check that no other handler has it offline */
+                   int j, found = 0;
+                   for (j=0; j<MAXHANDLERS; j++) {
+                       for (i = 0; i < MAXOFFLINEVOLUMES; i++) {
+                           if (OfflineVolumes[j][i].volumeID == 
command.volume)
+                               found = 1;
+                       }
+                   }
+                   if (found) {
+                       rc = FSYNC_DENIED;
+                       break;
+                   }
+               }
+               /* Find a slot for newly offline volume */
                 for (i = 0; i < MAXOFFLINEVOLUMES; i++) {
                     if (volumes[i].volumeID == 0) {
                         v = &volumes[i];
--- /afs/ipp/.cs/openafs/openafs-1.4.7-osd/src/vol/vol-salvage.c 
2008-05-06 12:59:46.000000000 +0200
+++ ./vol-salvage.c     2008-06-05 16:43:31.000000000 +0200
@@ -3720,8 +3720,8 @@
  void
  AskOffline(VolumeId volumeId)
  {
-    if (FSYNC_askfs(volumeId, NULL, FSYNC_OFF, FSYNC_SALVAGE) == 
FSYNC_DENIED) {
-       Log("AskOffline:  file server denied offline request; a general 
salvage is required.\n");
+    if (FSYNC_askfs(volumeId, NULL, Testing ? FSYNC_OFF : 
FSYNC_NEEDVOLUME, FSYNC_SALVAGE) == FSYNC_DENIED) {
+       Log("AskOffline:  file server denied offline request.\n");
         Abort("Salvage aborted\n");
      }
  }

Hartmut

Rainer Toebbicke wrote:
> Sorry, I started this in openafs-info a few days ago, but here's 
> probably the better place:
> 
> the problem: "salvager -part xxx -vol NNN" corrupts volumes
> (BTW, even with -nowrite!)
> 
> the scenario:
> 
> 1. the volserver takes a volume offline, the salvager takes the same 
> volume offline without knowing about the volserver. Possible, since the 
> partition lock only protects the VAttachVolume, and the "offline" works 
> (of course) even if the volume is already offline.
> 
> 2. the salvager and the volserver work on same data, first opportunity 
> for a thorough mess.
> 
> 3. assuming they didn't, e.g. salvager -nowrite, the salvager finishes 
> and puts the volume back online while the volserver still running. There 
> is no "reference count" on offline/online. The fileserver sees this, the 
> user jumps onto it, and we have a second opportunity to blow everything 
> apart.
> 
> 
> I see a number of alternatives,
> 
> 1. never use "salvager -vol",  always shut down the volserver before (a 
> challenge with bosserver restarting it), try gymnastics with "vos lock", 
> "vos offline", etc...  all looks pretty cumbersome.
> 
> 2. salvage a single volume within the volserver, triggered by a "vos 
> salvage" command; probably most logical, analogous to "vos zap" et al. I 
> haven't checked yet if the code can be shared without problems, looks 
> like some non-trivial Makefile and #ifdef gymnastics to get right. And 
> of course, it changes the documented commands, with changes in bos and vos.
> 
> 3. have the salvager create a transaction in the volserver, which takes 
> care of offline/online. Would have to care about deadlocks, as the 
> salvager doesn't just salvage the volume itself but the RW parent in 
> case of a RO or BK. And also with the (usual) case that the volume is so 
> damaged that it cannot be attached. But it's minimal intrusion 
> otherwise, no external changes.
> 
> 
> Before I go and implement 2. or 3., please give me your thoughts about 
> the scenario if you care (I may have missed something and would have to 
> look for another culprit - Derrick hinted at "-orphans attach" where I 
> did not understand everything yet), and whether 2. is preferable over 3.
> 
> 
> Cheers, Rainer
> 
> 


-- 
-----------------------------------------------------------------
Hartmut Reuter                  e-mail 		reuter@rzg.mpg.de
			   	phone 		 +49-89-3299-1328
			   	fax   		 +49-89-3299-1301
RZG (Rechenzentrum Garching)   	web    http://www.rzg.mpg.de/~hwr
Computing Center of the Max-Planck-Gesellschaft (MPG) and the
Institut fuer Plasmaphysik (IPP)
-----------------------------------------------------------------