[OpenAFS] tapeconfig capacity parameter still limited to 2TB?

Kristen J. Webb kwebb@teradactyl.com
Tue, 19 May 2015 18:07:46 -0600


Teradactyl is storing AFS data to tape on LTO and both Oracle and IBM enterprise 
series drives for clients globally.  We can perform this function on standalone 
drives or shared robotic tape libraries from virtually any manufacturer.  We 
remain committed to disk and tape backup of AFS.

We've seen 20TB of data in a single backup volume on a single LTO6 tape.  The 
bulk of the data was stored in variant call format (.vcf).  It was very 
compressible but it was not sparse (full of null bytes).  So the disk usage on 
the live filesystem was the expected 20TB.  On the backup server, the data 
consumed only about 1.4TB on disk (14:1 compression on ZFS with default 
compression enabled), and was written to tape at 298MB/sec, almost twice the 
160MB/sec uncompressed transfer rate for LTO-6.  The tape compression was about 
7.5:1 or about 3X the rated compression for LTO-6.  Only allowing 4TB/tape in 
this case results in about a 20% utilization of each tape.

The new LTO-6 drives and the road-map moving forward use a different compression 
algorithm  This is the LTO-DC vs. the older ALDC approach. Basically, LTO-DC 
does not apply the compression algorithm to uncompressible data.  This enables 
LTO-6 to often achieve higher compression than you might expect from LTO-5 and 
prievious tape technologies.

Teradactyl would be pleased to discuss this issue directly if you are interested 
and can arrange a meeting with CMU should you like a local point of contact.  We 
also plan to be in Pittsburgh as a returning sponsor for the North American AFS 
and Kerberos Best Practices Workshop in August.


Kris Webb

On 5/18/15 2:27 PM, Charles (Chas) Williams wrote:
> butc actually uses a signed int32 so the tape capacity could be doubled
> to 4TB fairly easily (I have misplaced the patch somewhere but I could
> find it).  That isn't quite 6.25TB but your compression ratio seems
> "optimistic" so it might be enough until the next generation of LTO.
> https://lists.openafs.org/pipermail/openafs-info/2013-May/039580.html
> On Mon, 2015-05-18 at 13:26 -0400, Jeffrey Altman wrote:
>> Russ,
>> All of the backup protocol messages are limited to sizes that max out at
>> 2TB (2^32 1KB blocks).
>> I am not aware of anyone that is actively maintaining the backup suite
>> shipped with OpenAFS.
>> I recommend that the University of Pittsburgh communicate with its
>> commercial support provider to request an estimate for designing,
>> implementing, and testing the necessary changes.  Changes to the backup
>> suite protocols do not require submission through a standardization
>> process (unlike other AFS3 RPC suites).
>> Jeffrey Altman
>> On 5/18/2015 12:48 PM, Howard Jr, Russell A wrote:
>>> Over the summer, we are upgrading our tape library from LTO-4 to LTO-6
>>> and would like to take advantage of the full capacity of the new tapes.
>>> The tapeconfig documentation indicates that the max tape size supported
>>> is 2TB.  Is this still the case?
>>> Our current drive is LTO-4 with a native uncompressed capacity of 800 GB
>>> and a compressed capacity of 1.6TB.  For our current backups, we
>>> specified the tape capacity in the tapeconfig file as 1600G.  This has
>>> allowed us to utilize the full capacity of the LTO-4 tapes.  Our new
>>> tape drive will have a native uncompressed capacity of 2.5TB and a
>>> compressed capacity of 6.25TB.
>>> If the 2TB limit is still in place, we would like to request that it be
>>> increased to at least 6.25TB (or larger to support future tape capacity
>>> increases).  Once our new tape drive arrives, we are willing to test any
>>> patches to help facilitate an increase in the tape size limit.
>>> --
>>> Russ Howard
>>> Systems/Programmer III
>>> Computer Science Department
>>> University of Pittsburgh
>>> 6209 Sennott Square
>>> 412-624-8834
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info

This message is NOT encrypted
Mr. Kristen J. Webb
Chief Technology Officer
Teradactyl LLC.
2450 Baylor Dr. S.E.
Albuquerque, New Mexico 87106
Phone: 1-505-338-6000
Email: kwebb@teradactyl.com
Web: http://www.teradactyl.com

Providers of Scalable Backup Solutions
    for Unique Data Environments

NOTICE TO RECIPIENTS: Any information contained in or attached to this message 
is intended solely for the use of the intended recipient(s). If you are not the 
intended recipient of this transmittal, you are hereby notified that you 
received this transmittal in error, and we request that you please delete and 
destroy all copies and attachments in your possession, notify the sender that 
you have received this communication in error, and note that any review or 
dissemination of, or the taking of any action in reliance on, this communication 
is expressly prohibited.

Regular internet e-mail transmission cannot be guaranteed to be secure or 
error-free. Therefore, we do not represent that this information is complete or 
accurate, and it should not be relied upon as such. If you prefer to communicate 
with Teradactyl LLC. using secure (i.e., encrypted and/or digitally signed) 
e-mail transmission, please notify the sender. Otherwise, you will be deemed to 
have consented to communicate with Teradactyl via regular internet e-mail 
transmission. Please note that Teradactyl reserves the right to intercept, 
monitor, and retain all e-mail messages (including secure e-mail messages) sent 
to or from its systems as permitted by applicable law.