[OpenAFS] Incremental backups

[Stephen Joyce]

On Fri, 25 Apr 2008, Kristen J. Webb wrote:

> Tom Fitzgerald wrote:
>> Can someone confirm that my assumptions about incremental AFS
>> backups are correct?
>> 
>> 1) Aside from needlessly increasing the size of the dump, there's
>>    no harm in setting the -time of an incremental dump to be
>>    substantially earlier than it needs to be.
> In my experience, you are correct.

I wouldn't go _substancially_ earlier, but I concur that you're correct. I 
assume that you're worried about the conversion of wall time in your 
timezone to epoch time wrt DST changes? I'm not sure about older versions, 
so I'll leave it to someone more familiar with AFS internals to answer, but 
I seem to recall that OpenAFS 1.4 and newer stores timestamps in UTC to 
prevent these issues.

>> 2) If I do a "vos backup X" and later a "vos dump X.backup", then
>>    later a incremental dump, the -time in the incremental dump
>>    should be the time of the first "vos backup" (or earlier), not
>>    the full "vos dump". 
> Yes, but before you do the incremental dump, you need to do another
> vos backup ;)

Yes. It's a common mistake to use the time of the dump, rather than the 
backup, as a reference. But as you've surmised, that can lead to missing 
changes. I'd personally recommend a "vos backup" immediately before your 
"vos dump". That way a "vos exam" of the backup volume will always reveal 
the last time the volume was actually dumped to your media.

> Depending on your incremental strategy there are a few more
> issues to consider.
>
> If your incrementals are partial (e.g. changes since the
> last incremental backup) then you need to ensure that the
> vos backup before each incremental actually works, or you
> may create a "hole" in your backup data.  We also ran into
> problems with time between running on the command line vs.
> cron.  Our software compares time stamps in the vos dump
> headers to avoid these types of problems.
>
> If your incrementals are cumulative (e.g. changes since
> the last full), then if any given vos backup fails, the
> next incremental will simply be another copy of the
> previous incremental.  You will miss that days changes.
> As long as the vos backup problem does not persist, your next
> backup should "catch up".

I think I've seen the "cumulative incrementals", which include all changes 
since the last full, described as "differentials" in the old AFS manuals. 
Kris' "partial incrementals", which include the changes since the last 
backup (whether a full, differential, or incremental), were just described 
as plain "incrementals."

While working on BackupPC4AFS <http://backuppc4afs.sourceforge.net>, I did 
quite a bit of testing; if your chain of backups includes a missing 
incremental, you'll lose those changes, but can still restore any newer 
incrementals that survive. However the changes on that incremental are, of 
course, gone unless they're in another dump too. In my opinion, a tower 
of hanoi backup rotation provides plenty of redundancy, unless you're very 
paranoid.

One other thing to be aware of is that giving the -time option to vos dump 
makes it look at the files' modification timestamps to determine what to 
back up. It's very possible to create a file with an old timestamp 
(using tar, touch, etc) so that it won't be caught by anything except a 
full backup. Mv'ing a file between volumes is an easy way to accidentally 
cause this. Just something to be aware of.

Cheers, Stephen
--
Stephen Joyce
Systems Administrator                                            P A N I C
Physics & Astronomy Department                         Physics & Astronomy
University of North Carolina at Chapel Hill         Network Infrastructure
voice: (919) 962-7214                                        and Computing
fax: (919) 962-0480                               http://www.panic.unc.edu

The purpose of the icons, the purpose of the entire OS X look and feel, is
to keep the customer happy during that critical period between the time of
sale and the time the check clears.