[OpenAFS] Symantec Support for AFS Backups
John Howard
jmhoward@gmail.com
Sun, 21 Jan 2007 15:42:45 -0500
------=_Part_178413_7205898.1169412165418
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Sounds like you got great support from Symantec. That is good to hear as
most are reporting less than stellar support since the Veritas to Symantec
switch.
We have patched our servers with MP6 recently but not our clients so we have
not seen this problem and I suspect we won't as we have decided to drop NBU
for AFS backups in favor of TiBS. The decision to drop NBU was driven by
the blurb in the 6.x release notes indicating that AFS backups would no
longer be supported in version 6.x and beyond.
john
On 1/20/07, Kathryn Hemness <kfhemness@ucdavis.edu> wrote:
>
> Hi John,
>
> The patch broke something in the client bpbkar binary.
> The backup would completely finish up to 99% done and then
> fail with the following error:
>
> 12/18/2006 10:45:16 master-server.ucdavis.edu client.ucdavis.edu File:
> nbe_cat_c_api.cpp Line: 1040: invalid input dataFunction: decode_data_field
> 12/18/2006 10:45:17 media-server.ucdavis.edu client.ucdavis.edu successfully
> wrote backup id reality.ucdavis.edu_1166466735, copy 1, fragment 3, 125472
> Kbytes at 5763.263 Kbytes/sec
> 12/18/2006 10:45:18 media-server.ucdavis.edu client.ucdavis.edu db_FLISTsend
> failed: file read failed (13)
>
> I replaced the client MP6 bpcd with the MP5 binary first, and the backup
> still failed.
> Then I replaced the bpbkar with the MP5 bpbkar and the problem went
> away. Reinstating
> the MP6 bpcd and keeping the MP5 bpbkar still caused failures.
>
> I get my NetBackup support through Sun but I didn't just open a call with
> Sun; I also
> went to the Symantec website and reported the problem. And, amazingly,
> Symantec actually
> responded even though I don't have a direct support contract with them.
>
> Symantec has sent me an engineering bpbkar binary for me to use until the
> next
> NB51 maintenance pack. This engineering binary works with the MP6 bpcd
> binary.
> I've tested both backups and restores with it.
>
>
> On Sat, 20 Jan 2007, John Howard wrote:
>
> > Date: Sat, 20 Jan 2007 14:43:12 -0500
> > From: John Howard <jmhoward@gmail.com>
> > To: Kathryn Hemness <kfhemness@ucdavis.edu>
> > Cc: openafs-info@openafs.org
> > Subject: Re: [OpenAFS] Symantec Support for AFS Backups
> >
> > Kathryn,
> >
> > How did MP6 break your AFS backups and was it the server patch or the
> client
> > patch that caused the issues. I'm just wondering if they (Symantec)
> tested
> > AFS backups against MP6, if they did and missed a bug that is acceptable
> but
> > if they didn't bother to test that is bad.
> >
> > john
> >
> >
> >
> > On 1/20/07, Kathryn Hemness <kfhemness@ucdavis.edu> wrote:
> > >
> > > Greetings,
> > >
> > > The NetBackup AFS backup capability has been built into the standard
> > > backup client/server binaries since NetBackup 4.5 (at least for
> Solaris,
> > > I don't know about for other operating systems).
> > >
> > > The basic (but not optimal) way it's used is to just create a an
> > > AFS backup policy with containing /vicep* partitions or volume
> > > names in the backup file list along with a CREATE_BACKUP_VOLUME
> > > and/or SKIP_SMALL_VOLUMES=X where X would be the size in Kbytes
> > > of volumes to skip.
> > >
> > > When the scheduled policy is launched, the AFS client system
> > > would run a listvol command which creates a file in
> > > /usr/openv/netbackup/listvol showing all existing volumes on the
> > > client. Then the CREATE_BACKUP_VOLUME would trigger the
> > > backup-volume-cloning of all of the RW volumes (including those
> > > volumes whose size is less that the SKIP directive.
> > >
> > > All meta information for the backup volume is retained, and
> > > when you wish to restore a file, you can't - instead you
> > > restore the entire volume (which is restored as R<vol name>.
> > > You can then create a mountpoint for the restored backup
> > > volume and allow your clients to get whatever files they wish from
> > > the backup volume. When your client is done with the restore,
> > > you can remove the mountpoint and the restored volume.
> > >
> > > This basic functionality works well when your AFS file storage
> > > is small. But when the AFS storage grew to hundreds of gigabytes
> > > and then to terabytes, the length of time required for the
> > > listvol and backup volume cloning caused backup timeouts and the
> > > backups were running 12+ hours. To complicate matters even more,
> > > NetBackup would only use a listvol listing younger than 3 hours.
> > >
> > > I've optimized my AFS backups by using cron to run a backupsys
> > > command at midnight daily (this backupsys command allows for
> > > there to be incremental and full backups of volumes); the
> > > backupsys command creates the backup clone volumes. Then,
> > > about 2 hours prior to the scheduled backup window opening,
> > > I have another cron script that runs the listvol. When
> > > the backup window opens, NetBackup uses the existing listvol
> > > file to backup the pre-existing backup clones. The total time
> > > of my largest AFS backup policy is now less than 8 hours
> > > (a significant improvement). Most of the time I am getting
> > > backup speeds of 4-9MB/sec to my backup sans.
> > >
> > > I've been extremely happy with the NetBackup capability which is
> > > why I want to find more than the handful of customers who use
> > > it for backing up their AFS storage. Symantec's perception that
> > > there is only a handful of organizations is based on their lack
> > > of support calls because the AFS capability is not a separately-priced
> > > agent. I've only had to call in their support because their most
> recent
> > > NB51 maintenance patch (MP6) broke the functionality. I find their
> logic
> > > flawed since satisfied users don't call for support.
> > >
> > > On Sat, 20 Jan 2007 openafs-info-request@openafs.org wrote:
> > > >
> > > > Message: 6
> > > > Date: Fri, 19 Jan 2007 16:31:46 -0500
> > > > From: Dave Botsch <botsch@cnf.cornell.edu>
> > > > To: openafs-info@openafs.org
> > > > Subject: Re: [OpenAFS] Symantec Support for AFS Backups
> > > >
> > > > I would certainly be interested in hearing more about how the
> backups
> > > and
> > > > restores work w. Symantec backup.
> > > >
> > > >
> > > > --
> > > > ********************************
> > > > David William Botsch
> > > > Programmer/Analyst
> > > > CNF Computing
> > > > botsch@cnf.cornell.edu
> > > > ********************************
> > > >
> > > > --__--__--
> > > >
> > > > Message: 7
> > > > Date: Fri, 19 Jan 2007 17:01:12 -0500
> > > > From: Mike Anderson <Mike.Anderson@nd.edu>
> > > > To: openafs-info@openafs.org
> > > > Subject: Re: [OpenAFS] Symantec Support for AFS Backups
> > > >
> > > >
> > > > I am very interested in hearing more details about how they work.
> > > >
> > > > Mike Anderson
> > > > OIT Operations & Engineering
> > > > University of Notre Dame
> > > > 320 IT Center
> > > > Notre Dame, IN 46556
> > > > Phone: 574-631-4966
> > > >
> > > >
> > > > Quoting Dave Botsch <botsch@cnf.cornell.edu>:
> > > >
> > > > > I would certainly be interested in hearing more about how the
> backups
> > > and
> > > > > restores work w. Symantec backup.
> > > > >
> > > > --__--__--
> > > >
> > > _______________________________________________
> > > OpenAFS-info mailing list
> > > OpenAFS-info@openafs.org
> > > https://lists.openafs.org/mailman/listinfo/openafs-info
> > >
> >
>
>
> --Kathy
>
> =======================================================================
> Kathryn Hemness kfhemness@ucdavis.edu
> System Administrator phone: 530.752.6547
> Campus Data Center & Client Services fax: 530.752.9154
>
------=_Part_178413_7205898.1169412165418
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Sounds like you got great support from Symantec. That is good to hear as most are reporting less than stellar support since the Veritas to Symantec switch. <br><br>We have patched our servers with MP6 recently but not our clients so we have not seen this problem and I suspect we won't as we have decided to drop NBU for AFS backups in favor of TiBS. The decision to drop NBU was driven by the blurb in the
6.x release notes indicating that AFS backups would no longer be supported in version 6.x and beyond. <br><br>john<br><br><div><span class="gmail_quote">On 1/20/07, <b class="gmail_sendername">Kathryn Hemness</b> <<a href="mailto:kfhemness@ucdavis.edu">
kfhemness@ucdavis.edu</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hi John,<br><br>The patch broke something in the client bpbkar binary.
<br>The backup would completely finish up to 99% done and then<br>fail with the following error:<br><br>12/18/2006 10:45:16 <a href="http://master-server.ucdavis.edu">master-server.ucdavis.edu</a> <a href="http://client.ucdavis.edu">
client.ucdavis.edu</a> File: nbe_cat_c_api.cpp Line: 1040: invalid input dataFunction: decode_data_field<br>12/18/2006 10:45:17 <a href="http://media-server.ucdavis.edu">media-server.ucdavis.edu</a> <a href="http://client.ucdavis.edu">
client.ucdavis.edu</a> successfully wrote backup id reality.ucdavis.edu_1166466735, copy 1, fragment 3, 125472 Kbytes at 5763.263 Kbytes/sec<br>12/18/2006 10:45:18 <a href="http://media-server.ucdavis.edu">media-server.ucdavis.edu
</a> <a href="http://client.ucdavis.edu">client.ucdavis.edu</a> db_FLISTsend failed: file read failed (13)<br><br>I replaced the client MP6 bpcd with the MP5 binary first, and the backup still failed.<br>Then I replaced the bpbkar with the MP5 bpbkar and the problem went away. Reinstating
<br>the MP6 bpcd and keeping the MP5 bpbkar still caused failures.<br><br>I get my NetBackup support through Sun but I didn't just open a call with Sun; I also<br>went to the Symantec website and reported the problem. And, amazingly, Symantec actually
<br>responded even though I don't have a direct support contract with them.<br><br>Symantec has sent me an engineering bpbkar binary for me to use until the next<br>NB51 maintenance pack. This engineering binary works with the MP6 bpcd binary.
<br>I've tested both backups and restores with it.<br><br><br>On Sat, 20 Jan 2007, John Howard wrote:<br><br>> Date: Sat, 20 Jan 2007 14:43:12 -0500<br>> From: John Howard <<a href="mailto:jmhoward@gmail.com">
jmhoward@gmail.com</a>><br>> To: Kathryn Hemness <<a href="mailto:kfhemness@ucdavis.edu">kfhemness@ucdavis.edu</a>><br>> Cc: <a href="mailto:openafs-info@openafs.org">openafs-info@openafs.org</a><br>> Subject: Re: [OpenAFS] Symantec Support for AFS Backups
<br>><br>> Kathryn,<br>><br>> How did MP6 break your AFS backups and was it the server patch or the client<br>> patch that caused the issues. I'm just wondering if they (Symantec) tested<br>> AFS backups against MP6, if they did and missed a bug that is acceptable but
<br>> if they didn't bother to test that is bad.<br>><br>> john<br>><br>><br>><br>> On 1/20/07, Kathryn Hemness <<a href="mailto:kfhemness@ucdavis.edu">kfhemness@ucdavis.edu</a>> wrote:<br>> >
<br>> > Greetings,<br>> ><br>> > The NetBackup AFS backup capability has been built into the standard<br>> > backup client/server binaries since NetBackup 4.5 (at least for Solaris,<br>> > I don't know about for other operating systems).
<br>> ><br>> > The basic (but not optimal) way it's used is to just create a an<br>> > AFS backup policy with containing /vicep* partitions or volume<br>> > names in the backup file list along with a CREATE_BACKUP_VOLUME
<br>> > and/or SKIP_SMALL_VOLUMES=X where X would be the size in Kbytes<br>> > of volumes to skip.<br>> ><br>> > When the scheduled policy is launched, the AFS client system<br>> > would run a listvol command which creates a file in
<br>> > /usr/openv/netbackup/listvol showing all existing volumes on the<br>> > client. Then the CREATE_BACKUP_VOLUME would trigger the<br>> > backup-volume-cloning of all of the RW volumes (including those
<br>> > volumes whose size is less that the SKIP directive.<br>> ><br>> > All meta information for the backup volume is retained, and<br>> > when you wish to restore a file, you can't - instead you
<br>> > restore the entire volume (which is restored as R<vol name>.<br>> > You can then create a mountpoint for the restored backup<br>> > volume and allow your clients to get whatever files they wish from
<br>> > the backup volume. When your client is done with the restore,<br>> > you can remove the mountpoint and the restored volume.<br>> ><br>> > This basic functionality works well when your AFS file storage
<br>> > is small. But when the AFS storage grew to hundreds of gigabytes<br>> > and then to terabytes, the length of time required for the<br>> > listvol and backup volume cloning caused backup timeouts and the
<br>> > backups were running 12+ hours. To complicate matters even more,<br>> > NetBackup would only use a listvol listing younger than 3 hours.<br>> ><br>> > I've optimized my AFS backups by using cron to run a backupsys
<br>> > command at midnight daily (this backupsys command allows for<br>> > there to be incremental and full backups of volumes); the<br>> > backupsys command creates the backup clone volumes. Then,<br>
> > about 2 hours prior to the scheduled backup window opening,<br>> > I have another cron script that runs the listvol. When<br>> > the backup window opens, NetBackup uses the existing listvol<br>> > file to backup the pre-existing backup clones. The total time
<br>> > of my largest AFS backup policy is now less than 8 hours<br>> > (a significant improvement). Most of the time I am getting<br>> > backup speeds of 4-9MB/sec to my backup sans.<br>> ><br>> > I've been extremely happy with the NetBackup capability which is
<br>> > why I want to find more than the handful of customers who use<br>> > it for backing up their AFS storage. Symantec's perception that<br>> > there is only a handful of organizations is based on their lack
<br>> > of support calls because the AFS capability is not a separately-priced<br>> > agent. I've only had to call in their support because their most recent<br>> > NB51 maintenance patch (MP6) broke the functionality. I find their logic
<br>> > flawed since satisfied users don't call for support.<br>> ><br>> > On Sat, 20 Jan 2007 <a href="mailto:openafs-info-request@openafs.org">openafs-info-request@openafs.org</a> wrote:<br>> > >
<br>> > > Message: 6<br>> > > Date: Fri, 19 Jan 2007 16:31:46 -0500<br>> > > From: Dave Botsch <<a href="mailto:botsch@cnf.cornell.edu">botsch@cnf.cornell.edu</a>><br>> > > To: <a href="mailto:openafs-info@openafs.org">
openafs-info@openafs.org</a><br>> > > Subject: Re: [OpenAFS] Symantec Support for AFS Backups<br>> > ><br>> > > I would certainly be interested in hearing more about how the backups<br>> > and
<br>> > > restores work w. Symantec backup.<br>> > ><br>> > ><br>> > > --<br>> > > ********************************<br>> > > David William Botsch<br>> > > Programmer/Analyst
<br>> > > CNF Computing<br>> > > <a href="mailto:botsch@cnf.cornell.edu">botsch@cnf.cornell.edu</a><br>> > > ********************************<br>> > ><br>> > > --__--__--<br>> > >
<br>> > > Message: 7<br>> > > Date: Fri, 19 Jan 2007 17:01:12 -0500<br>> > > From: Mike Anderson <<a href="mailto:Mike.Anderson@nd.edu">Mike.Anderson@nd.edu</a>><br>> > > To: <a href="mailto:openafs-info@openafs.org">
openafs-info@openafs.org</a><br>> > > Subject: Re: [OpenAFS] Symantec Support for AFS Backups<br>> > ><br>> > ><br>> > > I am very interested in hearing more details about how they work.
<br>> > ><br>> > > Mike Anderson<br>> > > OIT Operations & Engineering<br>> > > University of Notre Dame<br>> > > 320 IT Center<br>> > > Notre Dame, IN 46556<br>> > > Phone: 574-631-4966
<br>> > ><br>> > ><br>> > > Quoting Dave Botsch <<a href="mailto:botsch@cnf.cornell.edu">botsch@cnf.cornell.edu</a>>:<br>> > ><br>> > > > I would certainly be interested in hearing more about how the backups
<br>> > and<br>> > > > restores work w. Symantec backup.<br>> > > ><br>> > > --__--__--<br>> > ><br>> > _______________________________________________<br>> > OpenAFS-info mailing list
<br>> > <a href="mailto:OpenAFS-info@openafs.org">OpenAFS-info@openafs.org</a><br>> > <a href="https://lists.openafs.org/mailman/listinfo/openafs-info">https://lists.openafs.org/mailman/listinfo/openafs-info</a>
<br>> ><br>><br><br><br>--Kathy<br><br>=======================================================================<br>Kathryn Hemness <a href="mailto:kfhemness@ucdavis.edu">kfhemness@ucdavis.edu
</a><br>System Administrator phone: 530.752.6547<br>Campus Data Center & Client Services fax: 530.752.9154<br></blockquote></div><br>
------=_Part_178413_7205898.1169412165418--