[OpenAFS-devel] linux45: smoke test failed
Joe Gorse
jhgorse@gmail.com
Fri, 15 Jul 2016 18:35:35 -0400
--001a1142cbd6de51b80537b43d51
Content-Type: text/plain; charset=UTF-8
Stephan,
Picking this back up, I am having difficulties repeating it consistently.
Debian 8.4, kernel 4.4.15, OpenAFS master
f14d263a73f0be75e4de92f62e836fb2e55680dd. I see the gerrit for reverting on
master is not in yet, so that's not it. Tried increasing the frequency of
afs_ShakeLooseVCaches.
A smaller git repo (e.g. openafs-robotest) never seems to trip the CWD bug.
Test method:
[vagrant@openafs-debian-dev:/afs/.robotest/test] $ mkdir g; cd g; git clone
git://gerrit.openafs.org/openafs.git;sleep 180;git log
Cloning into 'openafs'...
remote: Counting objects: 192945, done.
remote: Compressing objects: 100% (46882/46882), done.
remote: Total 192945 (delta 159381), reused 177218 (delta 145040)
Receiving objects: 100% (192945/192945), 71.80 MiB | 7.31 MiB/s, done.
Resolving deltas: 100% (159381/159381), done.
Checking connectivity... done.
Checking out files: 100% (5563/5563), done.
fatal: Unable to read current working directory: No such file or directory
[vagrant@openafs-debian-dev:/afs/.robotest/test/g] 3m31s 128 $ date
Fri Jul 15 21:39:16 UTC 2016
[vagrant@openafs-debian-dev:/afs/.robotest/test/g] $
I don't see anything obvious in the log files, though I am not sure what I
would be looking for.
FileLog:
==> /usr/afs/logs/FileLog <==
Fri Jul 15 21:37:43 2016 SAFS_GiveUpCallBacks (Noffids=32)
Fri Jul 15 21:37:43 2016 SAFS_GiveUpCallBacks (Noffids=32)
Fri Jul 15 21:37:43 2016 SAFS_GiveUpCallBacks (Noffids=32)
Fri Jul 15 21:37:43 2016 SAFS_GiveUpCallBacks (Noffids=17)
Fri Jul 15 21:38:51 2016 SRXAFS_FetchData, Fid = 536870915.1.1
Fri Jul 15 21:38:51 2016 SAFS_FetchStatus, Fid = 536870915.4.3, Host
10.0.2.15:7001, Id 1
Fri Jul 15 21:38:51 2016 SRXAFS_FetchData, Fid = 536870915.4.3
Fri Jul 15 21:38:51 2016 SRXAFS_FetchData, Fid = 536870918.1.1
Fri Jul 15 21:38:51 2016 SRXAFS_FetchData, Fid = 536870918.3.12879
Fri Jul 15 21:39:33 2016 SAFS_GiveUpCallBacks (Noffids=2)
So how do we repeat this more reliably?
Cheers,
Joe
On Sat, Jun 18, 2016 at 2:27 PM, Stephan Wiesand <stephan.wiesand@desy.de>
wrote:
> Joe,
>
> thanks for the feedback.
>
> On Jun 17, 2016, at 23:39 , Joe Gorse wrote:
> > FWIW, I am able to reproduce a "cwd" message for "git log" command on
> > Fedora 23, 4.5.6-200.fc23.x86_64. "git log" reads:
> >
> > fatal: Unable to read current working directory: No such file or
> directory
> >
> > Though it should read:
> >
> > fatal: Not a git repository (or any of the parent directories): .git
>
> Exactly the problem I'm seeing.
> >
> > However, I am not having any trouble with the git checkout. It seems to
> > consistently work on Fedora 23. Even the "git checkout
> > openafs-stable-1_6_18". Perhaps try on 4.5.6 on Fedora?
>
> That's where I started, see the first message in this thread. And in all
> cases, the git checkout actually works. I'm just using it to trigger the
> cwd problem - there are probably many other ways. Note that also "git log"
> is just one way of exposing the problem
>
> > Though I have seen some more of this issue on Debian 8 with kernel 4.6.0.
> > Three of three tests failed to checkout the openafs tree on this system.
> I
> > will test some other kernels on this system later and note anything
> > interesting.
>
> Sounds even considerably worse :-(
>
> Any errors logged? Does the client actually have some variant of gerrit
> 12228 applied?
>
> Cheers,
> Stephan
>
> > Cheers,
> > Joe
> >
> > On Fri, Jun 17, 2016 at 11:30 AM, Stephan Wiesand <
> stephan.wiesand@desy.de>
> > wrote:
> >
> >>
> >> On Jun 17, 2016, at 04:45 , Benjamin Kaduk wrote:
> >>
> >>> On Thu, 16 Jun 2016, Stephan Wiesand wrote:
> >>>
> >>>> I smoke tested what was planned to be OpenAFS 1.6.18.1, as discussed
> in
> >> yesterday's release team meeting, on a Fedora 23 x86_64 VM with kernel
> >> 4.5.6-200 today. The result was disappointing:
> >>>>
> >>>> git clone git://gerrit.openafs.org/openafs.git
> >>>
> >>> Is the pwd the root of a volume?
> >>
> >> No, everything happens at least one level below.
> >>
> >>>> cd openafs
> >>>> git log
> >>>> # scrolled through a few dozen changes, took a couple of seconds
> >>>> git checkout openafs-stable-1_6_18
> >>>>
> >>>> At this point I got the following error:
> >>>>
> >>>> fatal: Unable to read current working directory: No such file or
> >> directory
> >>>>
> >>>> A "cd; cd -" cures this for a while, and there's no apparent data
> >> corruption. I'm still worried. The problem isn't 100% reproducible, but
> it
> >> doesn't take too may tries checking out random tags or branches.
> >>>>
> >>>> This was plain 1.6.18 + gerrit 12300 12301 12302 12274.
> >>>>
> >>>> Cache is on ext4, no separate partition, default size as set by our
> RPM
> >> (I think 100MB, but I don't have access to the VM right now to check).
> >>>>
> >>>> The small cache size may contribute to the problem. But I found no
> >> errors logged anywhere, and this shouldn't happen no matter how small
> the
> >> cache is.
> >>>
> >>> Please check if the cmdebug output is empty (I expect it is, but it is
> >>> good to check).
> >>
> >> It is empty.
> >>
> >>>> NB we have a user report of exactly this problem happening frequently
> >> while just editing files in a local git repo in AFS space. The data is a
> >> bit sketchy, but it's probably Ubuntu 14.04 with its current default
> kernel
> >> and the openafs packages from Anders' ppa. I'll try to get us more data.
> >>>>
> >>>>
> >>>> Any thoughts? For the time being I'm considering this a showstopper
> for
> >>>> 1.6.18.1, and it looks like we're not quite there yet regarding Linux
> >>>> 4.5, let alone 4.6 or the 4.7 due in a few weeks :-(
> >>>
> >>> Can you run the same test on a 4.4 kernel for comparison?
> >>
> >> I tried under the last F22 kernel, 4.4.6-200.fc22. And ok, it's not 4.5
> >> specific, though it seems to happen more frequently with 4.5.2 than with
> >> 4.4.6.
> >>
> >> By chance I found a pretty reliable reproducer:
> >>
> >> cd /vol/ume/root
> >> mkdir g; cd g
> >> git clone git://gerrit.openafs.org/openafs.git; sleep 180; git
> log
> >>
> >> Note indeed no "cd openafs". Of course this should complain about the
> cwd
> >> not being a git repo. But most of the time it will complain about the
> cwd
> >> issue instead.
> >>
> >> I'm planning to verify that plain 1.6.18 behaves the same on 4.4.6, and
> if
> >> it does I'll proceed with the 1.6.18.1 release.
> >>
> >> I couldn't reproduce this with any EL clients, but those have larger
> >> caches (it's indeed 100 MB on that Fedora VM), so there's more to test.
> >> Help welcome...
> >>
> >>
> >> _______________________________________________
> >> OpenAFS-devel mailing list
> >> OpenAFS-devel@openafs.org
> >> https://lists.openafs.org/mailman/listinfo/openafs-devel
> >>
> >
> >
> >
> > --
> > Joe Gorse
> >
> > C: 440-552-0730
> > LI: Joe Gorse <http://www.linkedin.com/pub/joe-gorse/7/12/397>
>
> --
> Stephan Wiesand
> DESY -DV-
> Platanenenallee 6
> 15738 Zeuthen, Germany
>
>
--
Joe Gorse
C: 440-552-0730
LI: Joe Gorse <http://www.linkedin.com/pub/joe-gorse/7/12/397>
--001a1142cbd6de51b80537b43d51
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Stephan,<div><br></div><div>Picking this back up, I am hav=
ing difficulties repeating it consistently. Debian 8.4, kernel 4.4.15, Open=
AFS master f14d263a73f0be75e4de92f62e836fb2e55680dd. I see the gerrit for r=
everting on master is not in yet, so that's not it. Tried increasing th=
e frequency of afs_ShakeLooseVCaches.</div><div><br></div><div>A smaller gi=
t repo (e.g. openafs-robotest) never seems to trip the CWD bug.</div><div><=
br></div><div><div></div>Test method:<br><div><div class=3D"" style=3D"disp=
lay:block;padding:0.5em;color:rgb(51,51,51);font-family:monospace;backgroun=
d:rgb(248,248,248)"><div>[vagrant@openafs-debian-dev:/afs/.robotest/<span c=
lass=3D"" style=3D"color:rgb(57,115,0)">test</span>] $ mkdir g; <span class=
=3D"" style=3D"color:rgb(57,115,0)">cd</span> g; git <span class=3D"" style=
=3D"color:rgb(57,115,0)">clone</span> git://<a href=3D"http://gerrit.openaf=
s.org/openafs.git;sleep">gerrit.openafs.org/openafs.git;sleep</a> 180;git <=
span class=3D"" style=3D"color:rgb(57,115,0)">log</span></div><div>Cloning =
into <span class=3D"" style=3D"color:rgb(136,0,0)">'openafs'</span>=
...</div><div>remote: Counting objects: 192945, done.</div><div>remote: Com=
pressing objects: 100% (46882/46882), done.</div><div>remote: Total 192945 =
(delta 159381), reused 177218 (delta 145040)</div><div>Receiving objects: 1=
00% (192945/192945), 71.80 MiB | 7.31 MiB/s, done.</div><div>Resolving delt=
as: 100% (159381/159381), done.</div><div>Checking connectivity... done.</d=
iv><div>Checking out files: 100% (5563/5563), done.</div><div>fatal: Unable=
to <span class=3D"" style=3D"color:rgb(57,115,0)">read</span> current work=
ing directory: No such file or directory</div><div>[vagrant@openafs-debian-=
dev:/afs/.robotest/<span class=3D"" style=3D"color:rgb(57,115,0)">test</spa=
n>/g] 3m31s 128 $ date</div><div>Fri Jul 15 21:39:16 UTC 2016</div><div>[va=
grant@openafs-debian-dev:/afs/.robotest/<span class=3D"" style=3D"color:rgb=
(57,115,0)">test</span>/g] $</div></div></div></div><div><br></div><div>I d=
on't see anything obvious in the log files, though I am not sure what I=
would be looking for.</div><div>FileLog:</div><div><div><div class=3D"hljs=
xml" style=3D"display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,=
51);font-family:monospace;background:rgb(248,248,248)"><div>=3D=3D> /usr=
/afs/logs/FileLog <span class=3D"hljs-tag"><<span class=3D"hljs-name" st=
yle=3D"font-weight:bold">=3D=3D</span></span></div><div><span class=3D"hljs=
-tag"><span class=3D"hljs-name" style=3D"font-weight:bold">Fri</span> <span=
class=3D"hljs-attr">Jul</span> <span class=3D"hljs-attr">15</span> <span c=
lass=3D"hljs-attr">21:37:43</span> <span class=3D"hljs-attr">2016</span> <s=
pan class=3D"hljs-attr">SAFS_GiveUpCallBacks</span> (<span class=3D"hljs-at=
tr">Noffids</span>=3D<span class=3D"hljs-string" style=3D"color:rgb(136,0,0=
)">32)</span></span></div><div><span class=3D"hljs-tag"><span class=3D"hljs=
-string" style=3D"color:rgb(136,0,0)">Fri</span> <span class=3D"hljs-attr">=
Jul</span> <span class=3D"hljs-attr">15</span> <span class=3D"hljs-attr">21=
:37:43</span> <span class=3D"hljs-attr">2016</span> <span class=3D"hljs-att=
r">SAFS_GiveUpCallBacks</span> (<span class=3D"hljs-attr">Noffids</span>=3D=
<span class=3D"hljs-string" style=3D"color:rgb(136,0,0)">32)</span></span><=
/div><div><span class=3D"hljs-tag"><span class=3D"hljs-string" style=3D"col=
or:rgb(136,0,0)">Fri</span> <span class=3D"hljs-attr">Jul</span> <span clas=
s=3D"hljs-attr">15</span> <span class=3D"hljs-attr">21:37:43</span> <span c=
lass=3D"hljs-attr">2016</span> <span class=3D"hljs-attr">SAFS_GiveUpCallBac=
ks</span> (<span class=3D"hljs-attr">Noffids</span>=3D<span class=3D"hljs-s=
tring" style=3D"color:rgb(136,0,0)">32)</span></span></div><div><span class=
=3D"hljs-tag"><span class=3D"hljs-string" style=3D"color:rgb(136,0,0)">Fri<=
/span> <span class=3D"hljs-attr">Jul</span> <span class=3D"hljs-attr">15</s=
pan> <span class=3D"hljs-attr">21:37:43</span> <span class=3D"hljs-attr">20=
16</span> <span class=3D"hljs-attr">SAFS_GiveUpCallBacks</span> (<span clas=
s=3D"hljs-attr">Noffids</span>=3D<span class=3D"hljs-string" style=3D"color=
:rgb(136,0,0)">17)</span></span></div><div><span class=3D"hljs-tag"><span c=
lass=3D"hljs-string" style=3D"color:rgb(136,0,0)">Fri</span> <span class=3D=
"hljs-attr">Jul</span> <span class=3D"hljs-attr">15</span> <span class=3D"h=
ljs-attr">21:38:51</span> <span class=3D"hljs-attr">2016</span> <span class=
=3D"hljs-attr">SRXAFS_FetchData</span>, <span class=3D"hljs-attr">Fid</span=
> =3D <span class=3D"hljs-string" style=3D"color:rgb(136,0,0)">536870915.1.=
1</span></span></div><div><span class=3D"hljs-tag"><span class=3D"hljs-stri=
ng" style=3D"color:rgb(136,0,0)">Fri</span> <span class=3D"hljs-attr">Jul</=
span> <span class=3D"hljs-attr">15</span> <span class=3D"hljs-attr">21:38:5=
1</span> <span class=3D"hljs-attr">2016</span> <span class=3D"hljs-attr">SA=
FS_FetchStatus</span>, =C2=A0<span class=3D"hljs-attr">Fid</span> =3D <span=
class=3D"hljs-string" style=3D"color:rgb(136,0,0)">536870915.4.3,</span> <=
span class=3D"hljs-attr">Host</span> <span class=3D"hljs-attr"><a href=3D"h=
ttp://10.0.2.15:7001">10.0.2.15:7001</a></span>, <span class=3D"hljs-attr">=
Id</span> <span class=3D"hljs-attr">1</span></span></div><div><span class=
=3D"hljs-tag"><span class=3D"hljs-attr">Fri</span> <span class=3D"hljs-attr=
">Jul</span> <span class=3D"hljs-attr">15</span> <span class=3D"hljs-attr">=
21:38:51</span> <span class=3D"hljs-attr">2016</span> <span class=3D"hljs-a=
ttr">SRXAFS_FetchData</span>, <span class=3D"hljs-attr">Fid</span> =3D <spa=
n class=3D"hljs-string" style=3D"color:rgb(136,0,0)">536870915.4.3</span></=
span></div><div><span class=3D"hljs-tag"><span class=3D"hljs-string" style=
=3D"color:rgb(136,0,0)">Fri</span> <span class=3D"hljs-attr">Jul</span> <sp=
an class=3D"hljs-attr">15</span> <span class=3D"hljs-attr">21:38:51</span> =
<span class=3D"hljs-attr">2016</span> <span class=3D"hljs-attr">SRXAFS_Fetc=
hData</span>, <span class=3D"hljs-attr">Fid</span> =3D <span class=3D"hljs-=
string" style=3D"color:rgb(136,0,0)">536870918.1.1</span></span></div><div>=
<span class=3D"hljs-tag"><span class=3D"hljs-string" style=3D"color:rgb(136=
,0,0)">Fri</span> <span class=3D"hljs-attr">Jul</span> <span class=3D"hljs-=
attr">15</span> <span class=3D"hljs-attr">21:38:51</span> <span class=3D"hl=
js-attr">2016</span> <span class=3D"hljs-attr">SRXAFS_FetchData</span>, <sp=
an class=3D"hljs-attr">Fid</span> =3D <span class=3D"hljs-string" style=3D"=
color:rgb(136,0,0)">536870918.3.12879</span></span></div><div><span class=
=3D"hljs-tag"><span class=3D"hljs-string" style=3D"color:rgb(136,0,0)">Fri<=
/span> <span class=3D"hljs-attr">Jul</span> <span class=3D"hljs-attr">15</s=
pan> <span class=3D"hljs-attr">21:39:33</span> <span class=3D"hljs-attr">20=
16</span> <span class=3D"hljs-attr">SAFS_GiveUpCallBacks</span> (<span clas=
s=3D"hljs-attr">Noffids</span>=3D<span class=3D"hljs-string" style=3D"color=
:rgb(136,0,0)">2)</span></span></div></div></div><br><div></div></div><div>=
So how do we repeat this more reliably?</div><div><br></div><div>Cheers,</d=
iv><div>Joe</div><div><br></div>
</div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Sat, Jun=
18, 2016 at 2:27 PM, Stephan Wiesand <span dir=3D"ltr"><<a href=3D"mail=
to:stephan.wiesand@desy.de" target=3D"_blank">stephan.wiesand@desy.de</a>&g=
t;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0=
.8ex;border-left:1px #ccc solid;padding-left:1ex">Joe,<br>
<br>
thanks for the feedback.<br>
<span class=3D""><br>
On Jun 17, 2016, at 23:39 , Joe Gorse wrote:<br>
> FWIW, I am able to reproduce a "cwd" message for "git l=
og" command on<br>
> Fedora 23, 4.5.6-200.fc23.x86_64. "git log" reads:<br>
><br>
> fatal: Unable to read current working directory: No such file or direc=
tory<br>
><br>
> Though it should read:<br>
><br>
> fatal: Not a git repository (or any of the parent directories): .git<b=
r>
<br>
</span>Exactly the problem I'm seeing.<br>
<span class=3D"">><br>
> However, I am not having any trouble with the git checkout. It seems t=
o<br>
> consistently work on Fedora 23. Even the "git checkout<br>
> openafs-stable-1_6_18". Perhaps try on 4.5.6 on Fedora?<br>
<br>
</span>That's where I started, see the first message in this thread. An=
d in all cases, the git checkout actually works. I'm just using it to t=
rigger the cwd problem - there are probably many other ways. Note that also=
"git log" is just one way of exposing the problem<br>
<span class=3D""><br>
> Though I have seen some more of this issue on Debian 8 with kernel 4.6=
.0.<br>
> Three of three tests failed to checkout the openafs tree on this syste=
m. I<br>
> will test some other kernels on this system later and note anything<br=
>
> interesting.<br>
<br>
</span>Sounds even considerably worse :-(<br>
<br>
Any errors logged? Does the client actually have some variant of gerrit 122=
28 applied?<br>
<br>
Cheers,<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Stephan<br>
<div><div class=3D"h5"><br>
> Cheers,<br>
> Joe<br>
><br>
> On Fri, Jun 17, 2016 at 11:30 AM, Stephan Wiesand <<a href=3D"mailt=
o:stephan.wiesand@desy.de">stephan.wiesand@desy.de</a>><br>
> wrote:<br>
><br>
>><br>
>> On Jun 17, 2016, at 04:45 , Benjamin Kaduk wrote:<br>
>><br>
>>> On Thu, 16 Jun 2016, Stephan Wiesand wrote:<br>
>>><br>
>>>> I smoke tested what was planned to be OpenAFS 1.6.18.1, as=
discussed in<br>
>> yesterday's release team meeting, on a Fedora 23 x86_64 VM wit=
h kernel<br>
>> 4.5.6-200 today. The result was disappointing:<br>
>>>><br>
>>>> git clone git://<a href=3D"http://gerrit.openafs.org/opena=
fs.git" rel=3D"noreferrer" target=3D"_blank">gerrit.openafs.org/openafs.git=
</a><br>
>>><br>
>>> Is the pwd the root of a volume?<br>
>><br>
>> No, everything happens at least one level below.<br>
>><br>
>>>> cd openafs<br>
>>>> git log<br>
>>>> # scrolled through a few dozen changes, took a couple of s=
econds<br>
>>>> git checkout openafs-stable-1_6_18<br>
>>>><br>
>>>> At this point I got the following error:<br>
>>>><br>
>>>> fatal: Unable to read current working directory: No such f=
ile or<br>
>> directory<br>
>>>><br>
>>>> A "cd; cd -" cures this for a while, and there&#=
39;s no apparent data<br>
>> corruption. I'm still worried. The problem isn't 100% repr=
oducible, but it<br>
>> doesn't take too may tries checking out random tags or branche=
s.<br>
>>>><br>
>>>> This was plain 1.6.18 + gerrit 12300 12301 12302 12274.<br=
>
>>>><br>
>>>> Cache is on ext4, no separate partition, default size as s=
et by our RPM<br>
>> (I think 100MB, but I don't have access to the VM right now to=
check).<br>
>>>><br>
>>>> The small cache size may contribute to the problem. But I =
found no<br>
>> errors logged anywhere, and this shouldn't happen no matter ho=
w small the<br>
>> cache is.<br>
>>><br>
>>> Please check if the cmdebug output is empty (I expect it is, b=
ut it is<br>
>>> good to check).<br>
>><br>
>> It is empty.<br>
>><br>
>>>> NB we have a user report of exactly this problem happening=
frequently<br>
>> while just editing files in a local git repo in AFS space. The dat=
a is a<br>
>> bit sketchy, but it's probably Ubuntu 14.04 with its current d=
efault kernel<br>
>> and the openafs packages from Anders' ppa. I'll try to get=
us more data.<br>
>>>><br>
>>>><br>
>>>> Any thoughts? For the time being I'm considering this =
a showstopper for<br>
>>>> 1.6.18.1, and it looks like we're not quite there yet =
regarding Linux<br>
>>>> 4.5, let alone 4.6 or the 4.7 due in a few weeks :-(<br>
>>><br>
>>> Can you run the same test on a 4.4 kernel for comparison?<br>
>><br>
>> I tried under the last F22 kernel, 4.4.6-200.fc22. And ok, it'=
s not 4.5<br>
>> specific, though it seems to happen more frequently with 4.5.2 tha=
n with<br>
>> 4.4.6.<br>
>><br>
>> By chance I found a pretty reliable reproducer:<br>
>><br>
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 cd /vol/ume/root<br>
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 mkdir g; cd g<br>
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 git clone git://<a href=3D"http://gerri=
t.openafs.org/openafs.git" rel=3D"noreferrer" target=3D"_blank">gerrit.open=
afs.org/openafs.git</a>; sleep 180; git log<br>
>><br>
>> Note indeed no "cd openafs". Of course this should compl=
ain about the cwd<br>
>> not being a git repo. But most of the time it will complain about =
the cwd<br>
>> issue instead.<br>
>><br>
>> I'm planning to verify that plain 1.6.18 behaves the same on 4=
.4.6, and if<br>
>> it does I'll proceed with the 1.6.18.1 release.<br>
>><br>
>> I couldn't reproduce this with any EL clients, but those have =
larger<br>
>> caches (it's indeed 100 MB on that Fedora VM), so there's =
more to test.<br>
>> Help welcome...<br>
>><br>
>><br>
>> _______________________________________________<br>
>> OpenAFS-devel mailing list<br>
>> <a href=3D"mailto:OpenAFS-devel@openafs.org">OpenAFS-devel@openafs=
.org</a><br>
>> <a href=3D"https://lists.openafs.org/mailman/listinfo/openafs-deve=
l" rel=3D"noreferrer" target=3D"_blank">https://lists.openafs.org/mailman/l=
istinfo/openafs-devel</a><br>
>><br>
><br>
><br>
><br>
> --<br>
> Joe Gorse<br>
><br>
> C: <a href=3D"tel:440-552-0730" value=3D"+14405520730">440-552-0730</a=
><br>
</div></div>> LI: Joe Gorse <<a href=3D"http://www.linkedin.com/pub/j=
oe-gorse/7/12/397" rel=3D"noreferrer" target=3D"_blank">http://www.linkedin=
.com/pub/joe-gorse/7/12/397</a>><br>
<span class=3D"HOEnZb"><font color=3D"#888888"><br>
--<br>
Stephan Wiesand<br>
DESY -DV-<br>
Platanenenallee 6<br>
15738 Zeuthen, Germany<br>
<br>
</font></span></blockquote></div><br><br clear=3D"all"><div><br></div>-- <b=
r><div class=3D"gmail_signature" data-smartmail=3D"gmail_signature"><div di=
r=3D"ltr"><div>Joe Gorse<div><font face=3D"Calibri" size=3D"1"><br></font><=
/div><div><font face=3D"Calibri" size=3D"1"><div style=3D"font-family:arial=
;font-size:small"><span style=3D"font-family:arial;background-color:rgb(255=
,255,255)"><font size=3D"1">C:=C2=A0</font></span><span style=3D"background=
-color:rgb(255,255,255);font-size:x-small"><a value=3D"+14402344531"><font =
color=3D"#0000cc" face=3D"Helvetica">440-552-0730</font></a></span></div><d=
iv style=3D"font-family:arial;font-size:small"><span style=3D"font-family:a=
rial;background-color:rgb(255,255,255)"><font size=3D"1">LI: <a href=3D"htt=
p://www.linkedin.com/pub/joe-gorse/7/12/397" target=3D"_blank">Joe Gorse</a=
></font></span></div></font></div></div></div></div>
</div>
--001a1142cbd6de51b80537b43d51--