[OpenAFS] tapeconfig capacity parameter still limited to 2TB?
Kristen J. Webb
kwebb@teradactyl.com
Tue, 19 May 2015 18:07:46 -0600
Russ,
Teradactyl is storing AFS data to tape on LTO and both Oracle and IBM enterprise
series drives for clients globally. We can perform this function on standalone
drives or shared robotic tape libraries from virtually any manufacturer. We
remain committed to disk and tape backup of AFS.
We've seen 20TB of data in a single backup volume on a single LTO6 tape. The
bulk of the data was stored in variant call format (.vcf). It was very
compressible but it was not sparse (full of null bytes). So the disk usage on
the live filesystem was the expected 20TB. On the backup server, the data
consumed only about 1.4TB on disk (14:1 compression on ZFS with default
compression enabled), and was written to tape at 298MB/sec, almost twice the
160MB/sec uncompressed transfer rate for LTO-6. The tape compression was about
7.5:1 or about 3X the rated compression for LTO-6. Only allowing 4TB/tape in
this case results in about a 20% utilization of each tape.
The new LTO-6 drives and the road-map moving forward use a different compression
algorithm This is the LTO-DC vs. the older ALDC approach. Basically, LTO-DC
does not apply the compression algorithm to uncompressible data. This enables
LTO-6 to often achieve higher compression than you might expect from LTO-5 and
prievious tape technologies.
Teradactyl would be pleased to discuss this issue directly if you are interested
and can arrange a meeting with CMU should you like a local point of contact. We
also plan to be in Pittsburgh as a returning sponsor for the North American AFS
and Kerberos Best Practices Workshop in August.
Sincerely,
Kris Webb
On 5/18/15 2:27 PM, Charles (Chas) Williams wrote:
> butc actually uses a signed int32 so the tape capacity could be doubled
> to 4TB fairly easily (I have misplaced the patch somewhere but I could
> find it). That isn't quite 6.25TB but your compression ratio seems
> "optimistic" so it might be enough until the next generation of LTO.
>
> https://lists.openafs.org/pipermail/openafs-info/2013-May/039580.html
>
> On Mon, 2015-05-18 at 13:26 -0400, Jeffrey Altman wrote:
>> Russ,
>>
>> All of the backup protocol messages are limited to sizes that max out at
>> 2TB (2^32 1KB blocks).
>>
>> I am not aware of anyone that is actively maintaining the backup suite
>> shipped with OpenAFS.
>>
>> I recommend that the University of Pittsburgh communicate with its
>> commercial support provider to request an estimate for designing,
>> implementing, and testing the necessary changes. Changes to the backup
>> suite protocols do not require submission through a standardization
>> process (unlike other AFS3 RPC suites).
>>
>> Jeffrey Altman
>>
>>
>>
>>
>> On 5/18/2015 12:48 PM, Howard Jr, Russell A wrote:
>>> Over the summer, we are upgrading our tape library from LTO-4 to LTO-6
>>> and would like to take advantage of the full capacity of the new tapes.
>>>
>>>
>>>
>>> The tapeconfig documentation indicates that the max tape size supported
>>> is 2TB. Is this still the case?
>>>
>>>
>>>
>>> Our current drive is LTO-4 with a native uncompressed capacity of 800 GB
>>> and a compressed capacity of 1.6TB. For our current backups, we
>>> specified the tape capacity in the tapeconfig file as 1600G. This has
>>> allowed us to utilize the full capacity of the LTO-4 tapes. Our new
>>> tape drive will have a native uncompressed capacity of 2.5TB and a
>>> compressed capacity of 6.25TB.
>>>
>>>
>>>
>>> If the 2TB limit is still in place, we would like to request that it be
>>> increased to at least 6.25TB (or larger to support future tape capacity
>>> increases). Once our new tape drive arrives, we are willing to test any
>>> patches to help facilitate an increase in the tape size limit.
>>>
>>>
>>>
>>> --
>>>
>>> Russ Howard
>>>
>>> Systems/Programmer III
>>>
>>> Computer Science Department
>>>
>>> University of Pittsburgh
>>>
>>> 6209 Sennott Square
>>>
>>> 412-624-8834
>>>
>>>
>>>
>>
>
>
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
>
--
This message is NOT encrypted
--------------------------------
Mr. Kristen J. Webb
Chief Technology Officer
Teradactyl LLC.
2450 Baylor Dr. S.E.
Albuquerque, New Mexico 87106
Phone: 1-505-338-6000
Email: kwebb@teradactyl.com
Web: http://www.teradactyl.com
Providers of Scalable Backup Solutions
for Unique Data Environments
--------------------------------
NOTICE TO RECIPIENTS: Any information contained in or attached to this message
is intended solely for the use of the intended recipient(s). If you are not the
intended recipient of this transmittal, you are hereby notified that you
received this transmittal in error, and we request that you please delete and
destroy all copies and attachments in your possession, notify the sender that
you have received this communication in error, and note that any review or
dissemination of, or the taking of any action in reliance on, this communication
is expressly prohibited.
Regular internet e-mail transmission cannot be guaranteed to be secure or
error-free. Therefore, we do not represent that this information is complete or
accurate, and it should not be relied upon as such. If you prefer to communicate
with Teradactyl LLC. using secure (i.e., encrypted and/or digitally signed)
e-mail transmission, please notify the sender. Otherwise, you will be deemed to
have consented to communicate with Teradactyl via regular internet e-mail
transmission. Please note that Teradactyl reserves the right to intercept,
monitor, and retain all e-mail messages (including secure e-mail messages) sent
to or from its systems as permitted by applicable law.