Fwd: Re: [OpenAFS] openafs OSD

[Hartmut Reuter]

Sorry this mail should have gone also to the list!

-------- Original Message --------
Subject: Re: [OpenAFS] openafs OSD
Date: Thu, 30 May 2013 10:27:14 +0200
From: Hartmut Reuter <reuter@rzg.mpg.de>
To: Staffan Hämälä <sh@ltu.se>

Staffan,

as I have told you we are migrating from TSM-HSM to HPSS in order to get rid of
TSM-HSM. Five years ago we had a really bad experience with TSM-HSM running with
GPFS: a so called reconcile job had started in the background and had removed
millions of files from the TSM database. It never really has become clear what
caused this behaviour. We had much work to restore everything from dumps also
because this job ran for two or three days before it was detected, so new files
already migrated to tape were not in the restored database...

in 2007 I wrote an interface to dcache which can not be mounted as a file system
and only can be called by library calls. This same technique I used later when I
wrote the interface to HPSS which also works by calls to a shared library. Today
the rxosd loads a shared interface library when started with the appropriate
parameters which then itself contains the calls to the HPSS shared library. This
generic interface could be used also for other non-mountable archive systems
such as TSM. I was interested three years ago to do that for TSM, but never got
the necessary documentation about the TSM-library and so nothing happened.

A general problem with AFS-OSD is that I am the only person which knows the code
and which feels responsible to keep it running. I am 68 years old and now
retired, but still with good health and willing to work on it, but I think some
one else should start to do this work in order to make this project run longer
than other few years. I myself learnt these lessons 20 years ago when we had got
MR-AFS from PSC (Pittsburgh Supercomputer Center) and half a year later the
whole group of people who had worked on it left PSC. I became willing or not the
only developer of MR-AFS for the next 15 years until I replaced it by AFS-OSD.
There are many sites which principally are interested in AFS-OSD, but do not
deploy it because they don't see a long term support for it. Even my own site
RZG has planned a migration back to standard openafs for the next years unless
some one else appears to take the support over.

Hartmut

Staffan Hämälä wrote:
> Hi,
> 
> Yes, I'm intereseted in having a look. I'm primary checking out what can be done
> right now, and will start the real work on this after the summer.
> 
> I heard from Ragge (Anders Magnusson, a former collegue of me) that he had
> talked with you at the AFS conference a few years ago. Apparently, you were
> interested in using the TSM API to connect to TSM's archive function as a
> backend. Do you know if any work has been done on this?
> 
> If there isn't such a function today, would it be difficult to use TSM's archive
> function as backend?
> 
> As Harald says on the list, the archive function would be less complex. I had
> thought we would use TSM-HSM for this, but if there is a better option
> available, we should have a look at this as well. I would prefer to use TSM to
> handle the tapes, as our tape robot is currently connected only to TSM. But, the
> TS3500 is capable of connecting to several applications at the same time, so
> that's not really a requirement. But I would like to have all tapes handled by
> TSM. :-)
> 
> We currently use the archive function in TSM quite extensively. Our AFS montly /
> yearly backups are archived, as well as volumes for inactive users, etc.
> 
> I'm also interested to know why you are migrationg from TSM-HSM to HPSS as a
> backend?
> 
> /Staffan
> 
> On 2013-05-27 17:10, Hartmut Reuter wrote:
>> Hello Staffan,
>>
>> AFS-OSD is still alive and in use at two sites: RZG and PSI. At RZG we are using
>> it with HPSS and TSM-HSM as HSM back-ends. We are in the process to copy all
>> data from TSM-HSM over into HPSS and hope to get rid of TSM-HSM by the end of
>> this year. Presently our HPSS contains already > 8 million AFS-files with 729 TB
>> of data.
>>
>> RZG's cell "ipp-garching.mpg.de" is also in the process to migrate from AFS-OSD
>> 1.4 to AFS-OSD 1.6.
>>
>> The current source in git://github.com/hwr/openafs-osd.git is based on openafs
>> 1.6.2.
>>
>> Unlike HPSS TSM_HSM does not require special interface routines because you can
>> access the file-system directly by posix-calls. We are using GPFS along with
>> TSM-HSM, but I suppose you also use other file-systems as well.
>>
>> If you are interested have a look at the current version and feel free to ask
>> more.
>>
>> Hartmut Reuter
>>
>>
>>
>> Staffan Hämälä wrote:
>>> Hi,
>>>
>>> What's the current status of AFS-OSD? I've found a few presentations from 2009
>>> and 2010, but nothing more recent.
>>>
>>> I haven't found anything about OSD and openafs 1.6.
>>>
>>> Is there any, more recent, information on how to implement openafs/OSD to
>>> connect to TSM's HSM module?
>>>
>>> /Staffan
>>> LTU Sweden
>>> _______________________________________________
>>> OpenAFS-info mailing list
>>> OpenAFS-info@openafs.org
>>> https://lists.openafs.org/mailman/listinfo/openafs-info
>>
>>
> 


-- 
-----------------------------------------------------------------
Hartmut Reuter                  e-mail 		reuter@rzg.mpg.de
			   	phone 		 +49-89-3299-1328
			   	fax   		 +49-89-3299-1301
RZG (Rechenzentrum Garching)   	web    http://www.rzg.mpg.de/~hwr
Computing Center of the Max-Planck-Gesellschaft (MPG) and the
Institut fuer Plasmaphysik (IPP)
-----------------------------------------------------------------