[OpenAFS] Re: OpenAFS-info digest, Vol 1 #5205 - 9 msgs

[Derrick Brashear]

udebug talks to ubik servers. those are on ports 7002 and 7003 (and
7004 if you have the kaserver).

you udebug'd port 7001. nothing happened. it's also completely
irrelevant to any problem you might have.

On Wed, Feb 2, 2011 at 9:21 AM, sabah s. salih
<sabah.salih@hep.manchester.ac.uk> wrote:
> Dear Derrick,
> =A0 =A0 =A0 =A0 =A0Since this morning none of our users could see their h=
ome area in AFS.
>
> =A0Our database servers are
>
> =A0afs1.hep.man.ac.uk
> =A0afs2.hep.man.ac.uk
> =A0afs3.hep.man.ac.uk
>
> =A0and we have from afs5 - afs17 =A0our fileservers

look at the server logs on the db servers.
or
udebug the correct ports (you'll want 7003 for the vl service)
or
well, what does the client tell you? no /afs/help.man.ac.uk (or, if no
-dynroot on clients, no /afs)?
what's in the client's log? a server down message? for which server?

> =A0I can do
> =A0vos listvo afs1.hep.man.ac.uk =A0|more
> =A0.
> =A0.
> =A0.
> =A0.
> =A0vos listvo afs17.hep.manchester.ac.uk =A0|more
>
> =A0and I can list all users volumes without =A0any problems
> =A0They are all on line.
>
> =A0however none of us could type
> =A0ls
> =A0/afs/hep.man.ac.uk/u/sabah
> =A0.
> =A0.
> =A0.
> =A0.
> =A0.
>
> =A0Any Idea what goes wrong ?
>
> =A0Many Thanks, Sabah.
>
>
> From Sabah Salih
> The School of Physics and Astronomy,
> The University of Manchester,
> Schuster Laboratory,
> Brunswick Street,
> Manchester M13 9PL.
> Tel: +44 1612754171 or =A0x4171
> E-mail: sabah.salih@hep.manchester.ac.uk
> ________________________________________
> From: openafs-info-admin@openafs.org [openafs-info-admin@openafs.org] on =
behalf of openafs-info-request@openafs.org [openafs-info-request@openafs.or=
g]
> Sent: 02 February 2011 14:11
> To: openafs-info@openafs.org
> Subject: OpenAFS-info digest, Vol 1 #5205 - 9 msgs
>
> Send OpenAFS-info mailing list submissions to
> =A0 =A0 =A0 =A0openafs-info@openafs.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> =A0 =A0 =A0 =A0https://lists.openafs.org/mailman/listinfo/openafs-info
> or, via email, send a message with subject or body 'help' to
> =A0 =A0 =A0 =A0openafs-info-request@openafs.org
>
> You can reach the person managing the list at
> =A0 =A0 =A0 =A0openafs-info-admin@openafs.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of OpenAFS-info digest..."
>
>
> Today's Topics:
>
> =A0 1. Re: Re: Need volume state / fileserver / salvage knowledge (Jeff B=
laine)
> =A0 2. Re: Re: Need volume state / fileserver / salvage knowledge (Patric=
ia O'Reilly)
> =A0 3. Re: Need volume state / fileserver / salvage knowledge (Andrew Dea=
son)
> =A0 4. Re: Status of OS X 10.6 64bit kernel support ? (Chris Jones)
> =A0 5. Re: Status of OS X 10.6 64bit kernel support ? (Chris Jones)
> =A0 6. openafs on suse 10.2 with kernel 2.6.18.2-34-default (rosatrancoso=
)
> =A0 7. RE: OpenAFS-info digest, Vol 1 #5201 - 10 msgs (sabah s. salih)
> =A0 8. can not access to users home area (sabah s. salih)
> =A0 9. Re: can not access to users home area (Derrick Brashear)
>
> --__--__--
>
> Message: 1
> Date: Tue, 01 Feb 2011 14:39:18 -0500
> From: Jeff Blaine <jblaine@kickflop.net>
> To: openafs-info@openafs.org
> Subject: Re: [OpenAFS] Re: Need volume state / fileserver / salvage knowl=
edge
>
>>>> Wed Jan 26 12:28:13 2011: upclientetc exited on signal 15
>>>> Wed Jan 26 12:28:13 2011: upclientbin exited on signal 15
>>>> Wed Jan 26 12:28:24 2011: fs:vol exited on signal 15
>>>> Wed Jan 26 12:58:19 2011: bos shutdown: fileserver failed to shutdown =
within 1800 seconds
>>>> Wed Jan 26 12:58:37 2011: fs:file exited on signal 9
>
> Thanks for the replies.
>
> I can't at all fathom that our issue is one of existing
> client connections and callback break completion (timing out).
>
> =A0> Also, in this specific case, it may not be just that shutting down
> =A0> volumes took too long. 1.4.11 has known problems that can cause this
> =A0> (e.g. the host list gets a loop in it, and something spins forever
> =A0> trying to traverse the whole list).
>
> That's this, I think?:
>
> =A0 =A0 - Fixes to avoid issues cleaning up deleted hosts in
> =A0 =A0 =A0 the fileserver (126454)
>
> Let's assume this issue is what caused our problem. =A0I'm sort
> of at a loss as to how to approach OpenAFS versions. =A0On one
> hand, expectations of more effort to make it clear in the
> release notes what items could cause something like unclean
> server shutdowns (kind of a big deal, IMO) are not really
> justifiable. =A0It's open source, etc. =A0On the other hand,
> it's not acceptable to blindly upgrade to the latest stable
> release every time it comes out. =A0I understand that the most
> obvious take-away is just, "You got bit. =A0Move on.", but
> if anything can improve on our end, I'd like to do that.
>
> I welcome any suggestions for how others are approaching this.
>
> Jeff Blaine
>
> --__--__--
>
> Message: 2
> Date: Tue, 01 Feb 2011 12:04:08 -0800
> From: Patricia O'Reilly <oreilly@qualcomm.com>
> To: Jeff Blaine <jblaine@kickflop.net>
> Cc: openafs-info@openafs.org
> Subject: Re: [OpenAFS] Re: Need volume state / fileserver / salvage knowl=
edge
>
> From what you have described it sounds to me like you need the patch that=
 Andrew referenced earlier that allows you to configure an -offline-timeout=
 and -offline-shutdown-timeout option on your fileservers. We have has simi=
lar problems at our site and will be releasing that patch into production s=
hortly.
>
> --patty
>
> Jeff Blaine wrote:
>>>>> Wed Jan 26 12:28:13 2011: upclientetc exited on signal 15
>>>>> Wed Jan 26 12:28:13 2011: upclientbin exited on signal 15
>>>>> Wed Jan 26 12:28:24 2011: fs:vol exited on signal 15
>>>>> Wed Jan 26 12:58:19 2011: bos shutdown: fileserver failed to
>>>>> shutdown within 1800 seconds
>>>>> Wed Jan 26 12:58:37 2011: fs:file exited on signal 9
>>
>> Thanks for the replies.
>>
>> I can't at all fathom that our issue is one of existing
>> client connections and callback break completion (timing out).
>>
>>> Also, in this specific case, it may not be just that shutting down
>>> volumes took too long. 1.4.11 has known problems that can cause this
>>> (e.g. the host list gets a loop in it, and something spins forever
>>> trying to traverse the whole list).
>>
>> That's this, I think?:
>>
>> =A0 =A0 - Fixes to avoid issues cleaning up deleted hosts in
>> =A0 =A0 =A0 the fileserver (126454)
>>
>> Let's assume this issue is what caused our problem. =A0I'm sort
>> of at a loss as to how to approach OpenAFS versions. =A0On one
>> hand, expectations of more effort to make it clear in the
>> release notes what items could cause something like unclean
>> server shutdowns (kind of a big deal, IMO) are not really
>> justifiable. =A0It's open source, etc. =A0On the other hand,
>> it's not acceptable to blindly upgrade to the latest stable
>> release every time it comes out. =A0I understand that the most
>> obvious take-away is just, "You got bit. =A0Move on.", but
>> if anything can improve on our end, I'd like to do that.
>>
>> I welcome any suggestions for how others are approaching this.
>>
>> Jeff Blaine
>> _______________________________________________
>> OpenAFS-info mailing list
>> OpenAFS-info@openafs.org
>> https://lists.openafs.org/mailman/listinfo/openafs-info
>
> --__--__--
>
> Message: 3
> To: openafs-info@openafs.org
> From: Andrew Deason <adeason@sinenomine.net>
> Date: Tue, 1 Feb 2011 14:58:24 -0600
> Organization: Sine Nomine Associates
> Subject: [OpenAFS] Re: Need volume state / fileserver / salvage knowledge
>
> On Tue, 01 Feb 2011 12:04:08 -0800
> Patricia O'Reilly <oreilly@qualcomm.com> wrote:
>
>> From what you have described it sounds to me like you need the patch
>> that Andrew referenced earlier that allows you to configure an
>> -offline-timeout and -offline-shutdown-timeout option on your
>> fileservers. We have has similar problems at our site and will be
>> releasing that patch into production shortly.
>
> Maybe, maybe not. I think the most common cause of this is just having
> too many volumes that can be shut down in 30 minutes. Determining this
> is easy; if it happens every single time you shut down the fileserver,
> that's probably it. (But obviously that's not fun to do.)
>
> But it could also be the 1.4.11 host package bugs; I don't know, and I
> just noted that cause to illustrate that there are several possible
> reasons.
>
>> Jeff Blaine wrote:
>> >
>> > Thanks for the replies.
>> >
>> > I can't at all fathom that our issue is one of existing
>> > client connections and callback break completion (timing out).
>
> I'd only say that if you have pretty good control over all of your
> clients. It's possible to see some really bizarre behavior (from the
> fileserver's point of view) from old clients or clients on
> oddly-behaving networks or NATs.
>
>> >> Also, in this specific case, it may not be just that shutting down
>> >> volumes took too long. 1.4.11 has known problems that can cause this
>> >> (e.g. the host list gets a loop in it, and something spins forever
>> >> trying to traverse the whole list).
>> >
>> > That's this, I think?:
>> >
>> > =A0 =A0 - Fixes to avoid issues cleaning up deleted hosts in
>> > =A0 =A0 =A0 the fileserver (126454)
>
> There were a few issues; all of the ones known to cause problems are
> included in 1.4.12. I don't have references for all of them off the top
> of my head, but I can get them for you if you want.
>
>> > Let's assume this issue is what caused our problem. =A0I'm sort of at
>> > a loss as to how to approach OpenAFS versions. =A0On one hand,
>> > expectations of more effort to make it clear in the release notes
>> > what items could cause something like unclean server shutdowns (kind
>> > of a big deal, IMO) are not really justifiable.
>
> This wasn't an issue causing fileserver shutdowns to hang and get
> killed, it was a general fileserver stability issue; that hang (or
> crash, or however it manifested; I've seen both) could happen at any
> time.
>
> And doing something like that actually isn't that difficult for at least
> most of the issues I am involved with. I already generally know which
> versions are affected for the bigger issues, so just writing that down
> would not be that hard. (But going back through all of the changes
> between 1.4.Z and 1.4 head would be a lot of work at this point) But
> that's not true for all changes, and I think it may be prohibitively
> difficult if we had to include information like that with every single
> change to the stable branch.
>
> I'm not sure how useful it is, though. In the specific case of the host
> list issues, the only meaningful thing I can say is that "sometimes the
> fileserver crashes". It's not really possible for you to know how
> susceptible you are to it (unless you get hit by it), because the
> circumstances required to trigger the crash are rather complex, and they
> involve access patterns of clients that you generally cannot control or
> even detect.
>
>> > It's open source, etc. =A0On the other hand, it's not acceptable to
>> > blindly upgrade to the latest stable release every time it comes
>> > out. I understand that the most obvious take-away is just, "You got
>> > bit. Move on.", but if anything can improve on our end, I'd like to
>> > do that.
>
> Perhaps not right when it comes out, but it can be a good idea to move
> towards them, depending on how you do your risk/change management.
> Waiting a bit after each stable release for production machines makes
> sense, to see if unknown issues crop up, but if there are significant
> issues, you will hear about it if you are paying attention (probably in
> the form of a new release, fixing the issue).
>
> 1.4.12 was released almost a year ago, and I don't think there are any
> significant problems besides the issues that caused 1.4.14 to be
> released. There are some smaller issues here and there that sometimes
> get hit, but there's no fix on the 1.4.x branch for 1.4.12 that would
> cause me to recommend rolling back to pre-1.4.12 if you had upgraded a
> machine to 1.4.12.
>
> --
> Andrew Deason
> adeason@sinenomine.net
>
>
> --__--__--
>
> Message: 4
> From: Chris Jones <christopher.rob.jones@cern.ch>
> Date: Tue, 1 Feb 2011 22:57:40 +0000
> Cc: <openafs-info@openafs.org>
> To: Brandon S Allbery KF8NH <allbery.b@gmail.com>
> Subject: Re: [OpenAFS] Status of OS X 10.6 64bit kernel support ?
>
>
> On 1 Feb 2011, at 5:31am, Brandon S Allbery KF8NH wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>=3D20
>> On 2/1/11 00:26 , Derrick Brashear wrote:
>>> On Fri, Jan 21, 2011 at 9:41 AM, Chris Jones =3D
> <jonesc@hep.phy.cam.ac.uk> wrote:
>>>> I need to run the 64bit kernel, as have updated my MacBook Pro to =3D
> 8GB ram,
>>>> and I need to run the 64 bit kernel to properly support this. AFS is =
=3D
> so slow
>>=3D20
>> Coming in a bit late: =A0is the 64-bit kernel really necessary? =A0I've =
=3D
> been
>> running an iMac with 16GB using the 32-bit kernel; no doubt there's =3D
> some
>> memory wastage due to 1MB pages but not really noticeable in practice.
>
> Interesting side question. I was lead to believe my late 2008 MBPro =3D
> (which according Apple doesn't support more than 4GB) needs the 64 bit =
=3D
> kernel to run with more. I haven't tried though.
>
> That said, I don't think if it is or is not necessary is really that =3D
> relevant. Running the 64 bit kernel has other performance benefits, so =
=3D
> is desirable. Also I believe the latest Mac Pros now use the 64 bit =3D
> kernel by default... The 64 bit kernel is the way of the future, so I =3D
> think it best OpenAFS works out any niggles it has with it as soon as it =
=3D
> can ... Just my opinion ...
>
> cheers Chris=3D
>
> --__--__--
>
> Message: 5
> From: Chris Jones <christopher.rob.jones@cern.ch>
> Date: Tue, 1 Feb 2011 23:01:16 +0000
> To: openafs Info <openafs-info@openafs.org>
> Subject: Re: [OpenAFS] Status of OS X 10.6 64bit kernel support ?
>
>
>> Also I believe the latest Mac Pros now use the 64 bit kernel by =3D
> default...
>
> See
>
> http://support.apple.com/kb/ht3770
>
> The 64 bit kernel is clearly the way things are going, so needs proper =
=3D
> support.
>
> Chris=3D
>
> --__--__--
>
> Message: 6
> From: rosatrancoso <rosatrancoso@gmail.com>
> Date: Wed, 2 Feb 2011 13:21:24 +0000
> To: openafs-info@openafs.org
> Subject: [OpenAFS] openafs on suse 10.2 with kernel 2.6.18.2-34-default
>
> --90e6ba6e82a8c6b03e049b4c8721
> Content-Type: text/plain; charset=3DISO-8859-1
>
> Hello!
>
> I am trying to install openafs in machines with:
>
> cat /etc/SuSE-release: openSUSE 10.2 (X86-64)
> uname -r : 2.6.18.2-34-default
>
> I need to install openafs-client (which needs openafs) and openafs-krb5-m=
it.
>
> Everything installs OK but when I try to activate the service, it says th=
at
> openafs-client is not installed.
> Why?
> Do I need to install openafs-kmp-default? The problem with this is that t=
he
> available version of this rpm is not compatible with my kernel (2.6.18
> versus
> http://download.opensuse.org/repositories/filesystems/SLE_10/x86_64/opena=
fs-kmp-default-1.4.12.1_2.6.16.46_0.12-11.1.x86_64.rpm)
> I don't want to update the kernel as my machine has very important servic=
es
> running,..
>
> So, what can I do? Compile openafs-kernel-source? Are there any other rpm=
's?
>
> Thank you
> Rosa
>
> --90e6ba6e82a8c6b03e049b4c8721
> Content-Type: text/html; charset=3DISO-8859-1
> Content-Transfer-Encoding: quoted-printable
>
> Hello!<br><br>I am trying to install openafs in machines with:<br><br>cat=
 /=3D
> etc/SuSE-release: openSUSE 10.2 (X86-64)<br>uname -r : 2.6.18.2-34-defaul=
t<=3D
> br><br>I need to install openafs-client (which needs openafs) and openafs=
-k=3D
> rb5-mit. <br>
>
>
> Everything installs OK but when I try to activate the service, it says th=
at=3D
> =A0openafs-client is not installed. <br>Why?<br>Do I need to install open=
afs-=3D
> kmp-default? The problem with this is that the available version of this =
rp=3D
> m is not compatible with my kernel (2.6.18 versus <a href=3D3D"http://dow=
nloa=3D
> d.opensuse.org/repositories/filesystems/SLE_10/x86_64/openafs-kmp-default=
-1=3D
> .4.12.1_2.6.16.46_0.12-11.1.x86_64.rpm" target=3D3D"_blank">http://downlo=
ad.o=3D
> pensuse.org/repositories/filesystems/SLE_10/x86_64/openafs-kmp-default-1.=
4.=3D
> 12.1_2.6.16.46_0.12-11.1.x86_64.rpm</a>) I don&#39;t want to update the k=
er=3D
> nel as my machine has very important services running,..<br>
>
>
> <br>So, what can I do? Compile openafs-kernel-source? Are there any other=
 r=3D
> pm&#39;s?<br><br>Thank you<br>Rosa<br>
>
> --90e6ba6e82a8c6b03e049b4c8721--
>
> --__--__--
>
> Message: 7
> From: "sabah s. salih" <sabah.salih@hep.manchester.ac.uk>
> To: "openafs-info@openafs.org" <openafs-info@openafs.org>
> Date: Wed, 2 Feb 2011 13:59:03 +0000
> Subject: [OpenAFS] RE: OpenAFS-info digest, Vol 1 #5201 - 10 msgs
>
> Dear All,=3D0A=3D
> =A0 =A0 =A0 =A0 =A0 =A0Does anyone know how to fix cache manager problem?=
=3D0A=3D
> =A0=3D0A=3D
> =A0When I do the following=3D0A=3D
> =3D0A=3D
> udebug afs1.hep.man.ac.uk 7001=3D0A=3D
> udebug afs2.hep.man.ac.uk 7001 =3D0A=3D
> udebug afs3.hep.man.ac.uk 7001=3D0A=3D
> =3D0A=3D
> =A0I do not get an answer=3D0A=3D
> =3D0A=3D
> =A0Many Thanks, Sabah.=3D0A=3D
> =3D0A=3D
> =3D0A=3D
> From Sabah Salih=3D0A=3D
> The School of Physics and Astronomy,=3D0A=3D
> The University of Manchester,=3D0A=3D
> Schuster Laboratory,=3D0A=3D
> Brunswick Street,=3D0A=3D
> Manchester M13 9PL.=3D0A=3D
> Tel: +44 1612754171 or =A0x4171=3D0A=3D
> E-mail: sabah.salih@hep.manchester.ac.uk=3D
>
> --__--__--
>
> Message: 8
> From: "sabah s. salih" <sabah.salih@hep.manchester.ac.uk>
> To: "openafs-info@openafs.org" <openafs-info@openafs.org>
> Date: Wed, 2 Feb 2011 14:07:10 +0000
> Subject: [OpenAFS] can not access to users home area
>
> =A0Dear All,=3D0A=3D
> =A0 =A0 =A0 =A0 =A0 =A0Does anyone know how to fix cache manager problem?=
=3D0A=3D
> =3D0A=3D
> =A0When I do the following=3D0A=3D
> =3D0A=3D
> udebug afs1.hep.man.ac.uk 7001=3D0A=3D
> udebug afs2.hep.man.ac.uk 7001=3D0A=3D
> udebug afs3.hep.man.ac.uk 7001=3D0A=3D
> =3D0A=3D
> =A0I do not get an answer=3D0A=3D
> =3D0A=3D
> =A0RX debug give an answer=3D0A=3D
> =3D0A=3D
> rxdebug afs1.hep.man.ac.uk 7001=3D0A=3D
> Trying 194.36.2.3 (port 7001):=3D0A=3D
> Free packets: 130, packet reclaims: 0, calls: 0, used FDs: 64=3D0A=3D
> not waiting for packets.=3D0A=3D
> 0 calls waiting for a thread=3D0A=3D
> 1 threads are idle=3D0A=3D
> Done.=3D0A=3D
> =3D0A=3D
> rxdebug afs2.hep.man.ac.uk 7001=3D0A=3D
> Trying 194.36.2.4 (port 7001):=3D0A=3D
> Free packets: 130, packet reclaims: 0, calls: 569, used FDs: 64=3D0A=3D
> not waiting for packets.=3D0A=3D
> 0 calls waiting for a thread=3D0A=3D
> 1 threads are idle=3D0A=3D
> Done.=3D0A=3D
> =3D0A=3D
> rxdebug afs3.hep.man.ac.uk 7001=3D0A=3D
> Trying 194.36.2.5 (port 7001):=3D0A=3D
> Free packets: 130, packet reclaims: 0, calls: 187, used FDs: 64=3D0A=3D
> not waiting for packets.=3D0A=3D
> 0 calls waiting for a thread=3D0A=3D
> 1 threads are idle=3D0A=3D
> Done.=3D0A=3D
> =3D0A=3D
> =3D0A=3D
> =3D0A=3D
> =A0=3D0A=3D
> .=3D0A=3D
> .=3D0A=3D
> .=3D0A=3D
> .=3D0A=3D
> =3D0A=3D
> =3D0A=3D
> =A0Many Thanks, Sabah.=3D0A=3D
> =3D0A=3D
> =3D0A=3D
> From Sabah Salih=3D0A=3D
> The School of Physics and Astronomy,=3D0A=3D
> The University of Manchester,=3D0A=3D
> Schuster Laboratory,=3D0A=3D
> Brunswick Street,=3D0A=3D
> Manchester M13 9PL.=3D0A=3D
> Tel: +44 1612754171 or =A0x4171=3D0A=3D
> E-mail: sabah.salih@hep.manchester.ac.uk=3D0A=3D
>
> --__--__--
>
> Message: 9
> Date: Wed, 2 Feb 2011 09:09:03 -0500
> From: Derrick Brashear <shadow@gmail.com>
> To: "sabah s. salih" <sabah.salih@hep.manchester.ac.uk>
> Cc: "openafs-info@openafs.org" <openafs-info@openafs.org>
> Subject: Re: [OpenAFS] can not access to users home area
>
> On Wed, Feb 2, 2011 at 9:07 AM, sabah s. salih
> <sabah.salih@hep.manchester.ac.uk> wrote:
>> =3DA0Dear All,
>> =3DA0 =3DA0 =3DA0 =3DA0 =3DA0 =3DA0Does anyone know how to fix cache man=
ager problem?
>>
>> =3DA0When I do the following
>>
>> udebug afs1.hep.man.ac.uk 7001
>> udebug afs2.hep.man.ac.uk 7001
>> udebug afs3.hep.man.ac.uk 7001
>>
>> =3DA0I do not get an answer
>
> perhaps you mean cmdebug?
>
>> =3DA0RX debug give an answer
>>
>> rxdebug afs1.hep.man.ac.uk 7001
>> Trying 194.36.2.3 (port 7001):
>> Free packets: 130, packet reclaims: 0, calls: 0, used FDs: 64
>> not waiting for packets.
>> 0 calls waiting for a thread
>> 1 threads are idle
>> Done.
>>
>> rxdebug afs2.hep.man.ac.uk 7001
>> Trying 194.36.2.4 (port 7001):
>> Free packets: 130, packet reclaims: 0, calls: 569, used FDs: 64
>> not waiting for packets.
>> 0 calls waiting for a thread
>> 1 threads are idle
>> Done.
>>
>> rxdebug afs3.hep.man.ac.uk 7001
>> Trying 194.36.2.5 (port 7001):
>> Free packets: 130, packet reclaims: 0, calls: 187, used FDs: 64
>> not waiting for packets.
>> 0 calls waiting for a thread
>> 1 threads are idle
>> Done.
>
> what problem are you trying to solve?
>
>
> --__--__--
>
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
>
>
> End of OpenAFS-info Digest
>



--=20
Derrick