[AFS3-std] Compression support for AFS vos dump - Specification

Mon, 29 Oct 2007 19:42:09 -0400

This is a cryptographically signed message in MIME format.

--------------ms020102070602030407020004
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Peter Somogyi back almost three years ago proposed the following 
protocol enhancement for creating dump files containing compressed
data.  An implementation of this proposal is available for review 
in OpenAFS RT Ticket 17947.

Feedback is requested.  

Jeffrey Altman

Compression support for OpenAFS - specification

- additional param for "volserver" command line:
	-z [gzip:<level>|bz2:<level>]
	where level: [1..9|default]
	Example:
		/usr/afs/bin/volserver -z gzip:5

- new volserver must detect "not to be compressed" new flag in volume header:
	- VolumeDiskData.reserved1[0] & 1

- new "vos" setfields flag:
	- cmd line: compression {enabled,disabled}
	- "use" volintInfo.spare[2]&1 and volintInfo.spare[3]&1
	- modify VolSetInfo and ListOneVolume
	- modify "vos examine" to show this flag

- new volserver must provide 1 new RPC call in the interface:
	- RXAFS_GetCapabilities - same as on fs interface
	- 2 separate bits for zlib and bz2 support
	- new volserver must query this function (if exists) in VolForwardMultiple and VolForward
	- VolForwardMultiple: send compressed format only when ALL receivers support that

- new volserver must produce always compressed dump if the compression is turned on - at "vos dump"

- new dump format:
	volserver dumps:
		<0x01><DumpHeader>
		<0x02><VolumeHeader>
		<0x03><VNode1>
		[<0x06><length ::= 4><afs_uint32: flags]
		<0x05><unknown length::=0x80><CompressData>
		<0x04><DumpEndMagic>
		[EOF]

		where:
		- <CompressData> ::= <CompressHeader><CompressContent>
		- <CompressHeader> ::= <length><byte: CompressionType> //where length is sizeof(<CompressHeader>)-sizeof(<length>)
		- <CompressContent> ::= the zlib/bzip2 generated content
		- <VNode1> must be the first large vnode (error case if it doesn't exist):
			<VNode1> ::= <vnodeNumber><uniquifier>{<tag greater than 20><contents>}*
		- <length>: variable size length, 0x80=unknown (means that this section must be parsed to proceed)
			- If bit 7 is not set, bits 0-6 are the length
			- If bit 7 is set, bits 0-6 tell you how many bytes long the length is.
			The length follows immediately in MSB-first order.
			- The special value 0x80 means indefinite length.
			- 0xfe and 0xff would indicate a single-bit value, with the value stored in the low-order bit of the length.
		- after <0x06>, "extra flags" must be followed (tag with explicit length), containing one or more afs_uint32's
		(the first afs_uint32 will be transmitted into VolumeDiskData.reserved1[0] - for example the DONT_COMPRESS ::=0x01 flag)
		If there's no <0x06> tag, the volume's compression must be enabled.

	volserver parser:
		- unknown high-level tags treatment:
			- <0x06..0x60>: <length> awaited, and then <length> bytes must be skipped, write warning to VolserLog
			- <0x61..0x7a>: skip 4 bytes, write warning to VolserLog
			- the tag <0x7e> has no data by itself, but labels the next tag as "critical"
			- all of 0x7b-0x7f will be single-byte tags with no data (skip them if they're not "critical", otherwise error)
			- 0x80-0xff: error

		- <CompressData> mustn't appear more times in the dump. (It would cause more complex buffering implementation.)
		- no <CompressData> embedding supported

- include zlib and bzip2 (src/zlib and src/bzip2)
- "don't compress" flag must be transmitted only when the source volserver's compression is set, and each receiver volserver supports this flag (in case of "vos release"), or when "vos dump" was invoked

Peter Somogyi
2005-FEB-04

psomogyi@gamax.hu

--------------ms020102070602030407020004
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJeTCC
AxcwggKAoAMCAQICEALr5BE3U6n+HWCoLbyhohMwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UE
BhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMT
I1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA3MDUzMTA2MTM1N1oX
DTA4MDUzMDA2MTM1N1owczEPMA0GA1UEBBMGQWx0bWFuMRUwEwYDVQQqEwxKZWZmcmV5IEVy
aWMxHDAaBgNVBAMTE0plZmZyZXkgRXJpYyBBbHRtYW4xKzApBgkqhkiG9w0BCQEWHGphbHRt
YW5Ac2VjdXJlLWVuZHBvaW50cy5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQCsoz/0+s4Cn65n/3bU3shXw4y5u1uEMEsBOiqNU0PfIKGYQe95b1FKNbNAkctSdQT6GF5c
bhSnJPmb2OOb1frx64dlDgskaG561xa8XPA1aP8Cc+33dgsSLIxGEh97lyUYHEfWBC03KMCF
PKhZfcrGAXoVCrFBadnLAokQbUTFahVg/qQx2IT3wSj1sCIfV5UDuXcEKHCvRtEZIsSzu184
9Cj6I4nY5bt+r94kyDHM94MHYBJi+6tWLFRy2gkIB3HEPmxAiQrKljNpH9bOffiBLIAgmJ6d
1ZXepBXyexQbwOYvftpVlMEFHHQmdiwH3tj69hE78XvM5X9J+SbjbuNpAgMBAAGjOTA3MCcG
A1UdEQQgMB6BHGphbHRtYW5Ac2VjdXJlLWVuZHBvaW50cy5jb20wDAYDVR0TAQH/BAIwADAN
BgkqhkiG9w0BAQUFAAOBgQB8FShDN2Ig034Y5eyadiFDEtOvsIJ3Z2xV9aTL4u8xMlz1gZR1
AZAvCv+ZMMRRKWCsrG5tItV8DFPSfWAGMpInmMarA4f76JRLQEUhkRUg8GpkJM5ryk5EDakk
0oiBQcQD8A+UHwrcmaj3UWxQ9zCjDgU+1mY9nEQxZZyp4eeUfzCCAxcwggKAoAMCAQICEALr
5BE3U6n+HWCoLbyhohMwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoT
HFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25h
bCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA3MDUzMTA2MTM1N1oXDTA4MDUzMDA2MTM1N1ow
czEPMA0GA1UEBBMGQWx0bWFuMRUwEwYDVQQqEwxKZWZmcmV5IEVyaWMxHDAaBgNVBAMTE0pl
ZmZyZXkgRXJpYyBBbHRtYW4xKzApBgkqhkiG9w0BCQEWHGphbHRtYW5Ac2VjdXJlLWVuZHBv
aW50cy5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCsoz/0+s4Cn65n/3bU
3shXw4y5u1uEMEsBOiqNU0PfIKGYQe95b1FKNbNAkctSdQT6GF5cbhSnJPmb2OOb1frx64dl
DgskaG561xa8XPA1aP8Cc+33dgsSLIxGEh97lyUYHEfWBC03KMCFPKhZfcrGAXoVCrFBadnL
AokQbUTFahVg/qQx2IT3wSj1sCIfV5UDuXcEKHCvRtEZIsSzu1849Cj6I4nY5bt+r94kyDHM
94MHYBJi+6tWLFRy2gkIB3HEPmxAiQrKljNpH9bOffiBLIAgmJ6d1ZXepBXyexQbwOYvftpV
lMEFHHQmdiwH3tj69hE78XvM5X9J+SbjbuNpAgMBAAGjOTA3MCcGA1UdEQQgMB6BHGphbHRt
YW5Ac2VjdXJlLWVuZHBvaW50cy5jb20wDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQUFAAOB
gQB8FShDN2Ig034Y5eyadiFDEtOvsIJ3Z2xV9aTL4u8xMlz1gZR1AZAvCv+ZMMRRKWCsrG5t
ItV8DFPSfWAGMpInmMarA4f76JRLQEUhkRUg8GpkJM5ryk5EDakk0oiBQcQD8A+UHwrcmaj3
UWxQ9zCjDgU+1mY9nEQxZZyp4eeUfzCCAz8wggKooAMCAQICAQ0wDQYJKoZIhvcNAQEFBQAw
gdExCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUg
VG93bjEaMBgGA1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNVBAsTH0NlcnRpZmljYXRp
b24gU2VydmljZXMgRGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFp
bCBDQTErMCkGCSqGSIb3DQEJARYccGVyc29uYWwtZnJlZW1haWxAdGhhd3RlLmNvbTAeFw0w
MzA3MTcwMDAwMDBaFw0xMzA3MTYyMzU5NTlaMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxU
aGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwg
RnJlZW1haWwgSXNzdWluZyBDQTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAxKY8VXNV
+065yplaHmjAdQRwnd/p/6Me7L3N9VvyGna9fww6YfK/Uc4B1OVQCjDXAmNaLIkVcI7dyfAr
hVqqP3FWy688Cwfn8R+RNiQqE88r1fOCdz0Dviv+uxg+B79AgAJk16emu59l0cUqVIUPSAR/
p7bRPGEEQB5kGXJgt/sCAwEAAaOBlDCBkTASBgNVHRMBAf8ECDAGAQH/AgEAMEMGA1UdHwQ8
MDowOKA2oDSGMmh0dHA6Ly9jcmwudGhhd3RlLmNvbS9UaGF3dGVQZXJzb25hbEZyZWVtYWls
Q0EuY3JsMAsGA1UdDwQEAwIBBjApBgNVHREEIjAgpB4wHDEaMBgGA1UEAxMRUHJpdmF0ZUxh
YmVsMi0xMzgwDQYJKoZIhvcNAQEFBQADgYEASIzRUIPqCy7MDaNmrGcPf6+svsIXoUOWlJ1/
TCG4+DYfqi2fNi/A9BxQIJNwPP2t4WFiw9k6GX6EsZkbAMUaC4J0niVQlGLH2ydxVyWN3amc
OY6MIE9lX5Xa9/eH1sYITq726jTlEBpbNU1341YheILcIRk13iSx0x1G/11fZU8xggNkMIID
YAIBATB2MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5
KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQQIQ
AuvkETdTqf4dYKgtvKGiEzAJBgUrDgMCGgUAoIIBwzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcN
AQcBMBwGCSqGSIb3DQEJBTEPFw0wNzEwMjkyMzQyMDlaMCMGCSqGSIb3DQEJBDEWBBQXekfI
VbZ6flG6oAKrwcOPmrfkkTBSBgkqhkiG9w0BCQ8xRTBDMAoGCCqGSIb3DQMHMA4GCCqGSIb3
DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDCBhQYJKwYB
BAGCNxAEMXgwdjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcg
KFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3Vpbmcg
Q0ECEALr5BE3U6n+HWCoLbyhohMwgYcGCyqGSIb3DQEJEAILMXigdjBiMQswCQYDVQQGEwJa
QTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhh
d3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECEALr5BE3U6n+HWCoLbyhohMwDQYJ
KoZIhvcNAQEBBQAEggEAiT1cSFlqtS2QeApBglKwjtHTB/81dnqVX7o4qY3toWM5NK0hqDDv
I8UDz13m1A+6waoskmV4A9wjbgfF73eU5h7ScQeMsbR8nudlRafUIqpeiW0xfXudTPDvQybb
YTWTCgDBl2J7XnhPxyrcmlyd48+NH6Xkh/2vTSSKqUdiCHD/6LtmNXgERjVOCn7VbZvRyuRK
rcdlF+iJWp7AKBbDV0g/VPSx7IyoxFa5vh7FL9uUaA6Aw8K6HtHJexJcWv5SpS3sdp2Mhxg/
tD7L7ByjID1LITuHsy9qJlJIDaiOSCzEwXzJcUQms9jzTenjSqFhe8RP3l+ovFQ4yJh0cLS8
tAAAAAAAAA==
--------------ms020102070602030407020004--