[OpenAFS] Re: Ubik trouble
Timothy Balcer
timothy@telmate.com
Sun, 12 Jan 2014 16:20:29 -0800
--001a11c2f58e0dace404efcf0b62
Content-Type: text/plain; charset=ISO-8859-1
Oh FYI, when I said ' some udp traffic was affected', I didn't mean
blocked. I meant mangled. Some packets were being partially truncated, so
looking at the headers only didn't give me any data. This was very non
obvious, until I had a hint of the problem.
Best,
On Sun, Jan 12, 2014 at 4:17 PM, Timothy Balcer <timothy@telmate.com> wrote:
> Wow. Just as you send it out....you find the problem. Always the way :-)
>
> This turned out to be a subtle network problem. There was a change in the
> cross colo VPN link by our provider, and while it didn't affect the
> majority of our traffic, apparently -some- udp traffic was affected, due to
> some encapsulation changes.
>
> I discovered this when I decided to bypass the network mesh entirely, just
> to 'try it', and things started sync'ing again.
>
> I dug deeper, and there was indeed a subtle network fault.
>
> So.. there you go.. if anyone else is beating their heads over a Ubik
> problem like this, try bypassing parts of your network to see if there is a
> network fault hiding amongst the trees.
>
> Best,
>
>
> On Sun, Jan 12, 2014 at 3:54 PM, Timothy Balcer <timothy@telmate.com>wrote:
>
>> Hey folks. Odd problem. I have gone over many things, and I am stumped. I
>> am tempted to just destroy and rebuild my errant vlserver, but I'd like to
>> know what's going on and I know I am missing something :-)
>>
>> I have three vlservers, one in each of my colos. Lets call them A
>> (10.33.10.43), B(10.36.10.7) and C(10.38.10.7)
>>
>> A and B sync with each other
>> B and C sync with each other (or try to) but fail
>> A and C do not.
>>
>> They do know about each other, as a udebug -long shows that. However
>> between A and C I see this in the long form:
>>
>> Server (10.38.10.7): (db 0.0)
>> last vote never rcvd
>> last beacon never sent
>> dbcurrent=0, up=0 beaconSince=0
>>
>> Same is true for C to A.
>>
>> The database version is completely off between A and C.
>>
>> This is the complete udebug from server C, which is the one that is
>> acting up:
>>
>> Host's addresses are: 10.38.10.7
>> Host's 10.38.10.7 time is Sun Jan 12 23:04:13 2014
>> Local time is Sun Jan 12 23:04:16 2014 (time differential 3 secs)
>> Last yes vote for 10.38.10.7 was 2 secs ago (not sync site);
>> Last vote started 2 secs ago (at Sun Jan 12 23:04:14 2014)
>> Local db version is 1388991001.15777
>> I am not sync site
>> Lowest host 10.38.10.7 was set 2 secs ago
>> Sync host 0.0.0.0 was set 1389567853 secs ago
>> The last trans I handled was 0.46
>> Sync site's db version is 1388991001.15777
>> 0 locked pages, 0 of them for write
>>
>> Server (10.36.10.7): (db 0.0)
>> last vote rcvd 2 secs ago (at Sun Jan 12 23:04:14 2014),
>> last beacon sent 2 secs ago (at Sun Jan 12 23:04:14 2014), last vote
>> was no
>> dbcurrent=0, up=1 beaconSince=1
>>
>> Server (10.33.10.43): (db 0.0)
>> last vote never rcvd
>> last beacon never sent
>> dbcurrent=0, up=0 beaconSince=0
>>
>> So it thinks it is not the sync site, but 0.0.0.0 is. In reality, server
>> A is the sync site.
>>
>> I have run tcpdump to make absolutely certain there are packets running
>> between the two hosts on all AFS ports, and there are.
>>
>> KeyFiles are fine, and checked with md5sum.
>>
>> Set debug to 25 on the affected vlserver (10.38.10.7, Server C) and got
>> this:
>>
>> Sun Jan 12 23:19:24 2014 Using 10.38.10.7 as my primary address
>> Sun Jan 12 23:19:40 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:19:40 2014 Starting AFS vlserver 4
>> (/usr/lib/openafs/vlserver -rxbind -d 25)
>> Sun Jan 12 23:19:40 2014 no vote from 10.36.10.7
>> @(#) OpenAFS 1.6.1-1+ubuntu0.2-debian built 2013-07-24
>> 12 23:19:40 2014 Received beacon type 0 from host 10.38.10.7
>> Sun Jan 12 23:19:44 2014 recovery running in state 0
>> Sun Jan 12 23:19:55 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:19:55 2014 no vote from 10.36.10.7
>> Sun Jan 12 23:19:55 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:19:55 2014 Received beacon type 0 from host 10.38.10.7
>> Sun Jan 12 23:20:10 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:20:10 2014 no vote from 10.36.10.7
>> Sun Jan 12 23:20:10 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:20:10 2014 Received beacon type 0 from host 10.38.10.7
>> Sun Jan 12 23:20:25 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:20:25 2014 no vote from 10.36.10.7
>> Sun Jan 12 23:20:25 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:20:25 2014 Received beacon type 0 from host 10.38.10.7
>> Sun Jan 12 23:20:40 2014 ubik:server 10.33.10.43 still down
>> Sun Jan 12 23:20:40 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:20:40 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:20:40 2014 no vote from 10.36.10.7
>> Sun Jan 12 23:20:40 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:20:40 2014 Received beacon type 0 from host 10.38.10.7
>> Sun Jan 12 23:20:40 2014 Ubik: vote 'yes' for 10.38.10.7 (NOT in quorum)
>> Sun Jan 12 23:20:44 2014 recovery running in state 0
>> Sun Jan 12 23:20:44 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:20:48 2014 recovery running in state 0
>> Sun Jan 12 23:20:48 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:20:52 2014 recovery running in state 0
>> Sun Jan 12 23:20:52 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:20:55 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:20:55 2014 no vote from 10.36.10.7
>> Sun Jan 12 23:20:55 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:20:55 2014 Received beacon type 0 from host 10.38.10.7
>> Sun Jan 12 23:20:56 2014 recovery running in state 0
>> Sun Jan 12 23:20:56 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:21:00 2014 recovery running in state 0
>> Sun Jan 12 23:21:00 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:21:04 2014 recovery running in state 0
>> Sun Jan 12 23:21:04 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:21:07 2014 allbetter checking
>> Sun Jan 12 23:21:07 2014 allbetter: returning 1
>> Sun Jan 12 23:21:07 2014 allbetter checking
>> Sun Jan 12 23:21:07 2014 allbetter: returning 1
>> Sun Jan 12 23:21:07 2014 allbetter checking
>> Sun Jan 12 23:21:07 2014 allbetter: returning 1
>> Sun Jan 12 23:21:07 2014 allbetter checking
>> Sun Jan 12 23:21:07 2014 allbetter: returning 1
>> Sun Jan 12 23:21:07 2014 allbetter checking
>> Sun Jan 12 23:21:07 2014 allbetter: returning 1
>> Sun Jan 12 23:21:07 2014 GetVolumeByID 536870913 (2) 10.38.10.83 noauth
>> Sun Jan 12 23:21:07 2014 allbetter checking
>> Sun Jan 12 23:21:07 2014 allbetter: returning 1
>> Sun Jan 12 23:21:07 2014 allbetter checking
>> Sun Jan 12 23:21:07 2014 allbetter: returning 1
>> Sun Jan 12 23:21:07 2014 allbetter checking
>> Sun Jan 12 23:21:07 2014 allbetter: returning 1
>> Sun Jan 12 23:21:07 2014 allbetter checking
>> Sun Jan 12 23:21:07 2014 allbetter: returning 1
>> Sun Jan 12 23:21:07 2014 allbetter checking
>> Sun Jan 12 23:21:07 2014 allbetter: returning 1
>> Sun Jan 12 23:21:07 2014 allbetter checking
>> Sun Jan 12 23:21:07 2014 allbetter: returning 1
>> Sun Jan 12 23:21:07 2014 GetVolumeByID 536870913 (2) 10.38.10.83 noauth
>> Sun Jan 12 23:21:07 2014 allbetter checking
>> Sun Jan 12 23:21:07 2014 allbetter: returning 1
>> Sun Jan 12 23:21:07 2014 allbetter checking
>> Sun Jan 12 23:21:07 2014 allbetter: returning 1
>> Sun Jan 12 23:21:07 2014 allbetter checking
>> Sun Jan 12 23:21:07 2014 allbetter: returning 1
>> Sun Jan 12 23:21:07 2014 allbetter checking
>> Sun Jan 12 23:21:07 2014 allbetter: returning 1
>> Sun Jan 12 23:21:07 2014 allbetter checking
>> Sun Jan 12 23:21:07 2014 allbetter: returning 1
>> Sun Jan 12 23:21:08 2014 recovery running in state 0
>> Sun Jan 12 23:21:08 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:21:10 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:21:10 2014 no vote from 10.36.10.7
>> Sun Jan 12 23:21:10 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:21:10 2014 Received beacon type 0 from host 10.38.10.7
>> Sun Jan 12 23:21:12 2014 recovery running in state 0
>> Sun Jan 12 23:21:25 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:21:25 2014 no vote from 10.36.10.7
>> Sun Jan 12 23:21:25 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:21:25 2014 Received beacon type 0 from host 10.38.10.7
>> Sun Jan 12 23:21:29 2014 allbetter checking
>> Sun Jan 12 23:21:29 2014 allbetter: returning 1
>> Sun Jan 12 23:21:29 2014 GetVolumeByName <snipped>
>> Sun Jan 12 23:21:29 2014 allbetter checking
>> Sun Jan 12 23:21:29 2014 allbetter: returning 1
>> Sun Jan 12 23:21:29 2014 allbetter checking
>> Sun Jan 12 23:21:29 2014 allbetter: returning 1
>> Sun Jan 12 23:21:29 2014 allbetter checking
>> Sun Jan 12 23:21:29 2014 allbetter: returning 1
>> Sun Jan 12 23:21:30 2014 allbetter checking
>> Sun Jan 12 23:21:30 2014 allbetter: returning 1
>> Sun Jan 12 23:21:30 2014 GetVolumeByName *<snipped>*
>> Sun Jan 12 23:21:30 2014 allbetter checking
>> Sun Jan 12 23:21:30 2014 allbetter: returning 1
>> Sun Jan 12 23:21:30 2014 allbetter checking
>> Sun Jan 12 23:21:30 2014 allbetter: returning 1
>> Sun Jan 12 23:21:30 2014 allbetter checking
>> Sun Jan 12 23:21:30 2014 allbetter: returning 1
>> Sun Jan 12 23:21:30 2014 allbetter checking
>> Sun Jan 12 23:21:30 2014 allbetter: returning 1
>> Sun Jan 12 23:21:30 2014 allbetter checking
>> Sun Jan 12 23:21:30 2014 allbetter: returning 1
>> Sun Jan 12 23:21:30 2014 allbetter checking
>> Sun Jan 12 23:21:30 2014 allbetter: returning 1
>> Sun Jan 12 23:21:30 2014 GetVolumeByName *<snipped>*
>> Sun Jan 12 23:21:30 2014 allbetter checking
>> Sun Jan 12 23:21:30 2014 allbetter: returning 1
>> Sun Jan 12 23:21:30 2014 allbetter checking
>> Sun Jan 12 23:21:30 2014 allbetter: returning 1
>> Sun Jan 12 23:21:30 2014 allbetter checking
>> Sun Jan 12 23:21:30 2014 allbetter: returning 1
>> Sun Jan 12 23:21:30 2014 allbetter checking
>> Sun Jan 12 23:21:30 2014 allbetter: returning 1
>> Sun Jan 12 23:21:30 2014 allbetter checking
>> Sun Jan 12 23:21:30 2014 allbetter: returning 1
>> Sun Jan 12 23:21:30 2014 allbetter checking
>> Sun Jan 12 23:21:30 2014 allbetter: returning 1
>> Sun Jan 12 23:21:30 2014 GetVolumeByName *<snipped>*
>> Sun Jan 12 23:21:30 2014 allbetter checking
>> Sun Jan 12 23:21:30 2014 allbetter: returning 1
>> Sun Jan 12 23:21:30 2014 allbetter checking
>> Sun Jan 12 23:21:30 2014 allbetter: returning 1
>> Sun Jan 12 23:21:30 2014 allbetter checking
>> Sun Jan 12 23:21:30 2014 allbetter: returning 1
>> Sun Jan 12 23:21:40 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:21:40 2014 no vote from 10.36.10.7
>> Sun Jan 12 23:21:40 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:21:40 2014 Received beacon type 0 from host 10.38.10.7
>> Sun Jan 12 23:21:55 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:21:55 2014 no vote from 10.36.10.7
>> Sun Jan 12 23:21:55 2014 beacon: amSyncSite is 0
>> Sun Jan 12 23:21:55 2014 Received beacon type 0 from host 10.38.10.7
>>
>> Any help is appreciated! I have already upgraded and rebooted all three
>> servers, lowest to highest, to no avail. I also attempted moving the data
>> files out of the way and starting the affected vlserver, but the same
>> symptoms remain.
>>
>> --
>> Timothy Balcer / IT Services
>> Telmate / San Francisco, CA
>> Direct / (415) 300-4313
>> Customer Service / (800) 205-5510
>>
>
>
>
> --
> Timothy Balcer / IT Services
> Telmate / San Francisco, CA
> Direct / (415) 300-4313
> Customer Service / (800) 205-5510
>
--
Timothy Balcer / IT Services
Telmate / San Francisco, CA
Direct / (415) 300-4313
Customer Service / (800) 205-5510
--001a11c2f58e0dace404efcf0b62
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Oh FYI, when I said ' some udp traffic was affected=
9;, I didn't mean blocked. I meant mangled. Some packets were being par=
tially truncated, so looking at the headers only didn't give me any dat=
a. This was very non obvious, until I had a hint of the problem.<br>
<br>Best,<br></div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_q=
uote">On Sun, Jan 12, 2014 at 4:17 PM, Timothy Balcer <span dir=3D"ltr"><=
;<a href=3D"mailto:timothy@telmate.com" target=3D"_blank">timothy@telmate.c=
om</a>></span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div><div><div><div><div>Wo=
w. Just as you send it out....you find the problem. Always the way :-)<br><=
br>
</div>This turned out to be a subtle network problem. There was a change in=
the cross colo VPN link by our provider, and while it didn't affect th=
e majority of our traffic, apparently -some- udp traffic was affected, due =
to some encapsulation changes.<br>
<br></div>I discovered this when I decided to bypass the network mesh entir=
ely, just to 'try it', and things started sync'ing again.<br><b=
r></div>I dug deeper, and there was indeed a subtle network fault.<br>
<br>
</div>So.. there you go.. if anyone else is beating their heads over a Ubik=
problem like this, try bypassing parts of your network to see if there is =
a network fault hiding amongst the trees.<br><br></div>Best,<br></div>
<div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_extra">
<br><br><div class=3D"gmail_quote">On Sun, Jan 12, 2014 at 3:54 PM, Timothy=
Balcer <span dir=3D"ltr"><<a href=3D"mailto:timothy@telmate.com" target=
=3D"_blank">timothy@telmate.com</a>></span> wrote:<br><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex">
<div dir=3D"ltr"><div><div><div><div><div><div><div><div>Hey folks. Odd pro=
blem. I have gone over many things, and I am stumped. I am tempted to just =
destroy and rebuild my errant vlserver, but I'd like to know what's=
going on and I know I am missing something :-)<br>
<br></div>I have three vlservers, one in each of my colos. Lets call them A=
(10.33.10.43), B(10.36.10.7) and C(10.38.10.7)<br><br></div>A and B sync w=
ith each other<br></div>B and C sync with each other (or try to) but fail<b=
r>
</div>A and C do not.<br><br></div>They do know about each other, as a udeb=
ug -long shows that. However between A and C I see this in the long form:<b=
r><br>Server (10.38.10.7): (db 0.0)<br>=A0=A0=A0 last vote never rcvd<br>=
=A0=A0=A0 last beacon never sent<br>
=A0=A0=A0 dbcurrent=3D0, up=3D0 beaconSince=3D0<br><br></div>Same is true f=
or C to A.<br><br></div>The database version is completely off between A an=
d C.<br><br></div><div>This is the complete udebug from server C, which is =
the one that is acting up:<br>
<br><font face=3D"courier new,monospace">Host's addresses are: 10.38.10=
.7<br>Host's 10.38.10.7 time is Sun Jan 12 23:04:13 2014<br>Local time =
is Sun Jan 12 23:04:16 2014 (time differential 3 secs)<br>Last yes vote for=
10.38.10.7 was 2 secs ago (not sync site);<br>
Last vote started 2 secs ago (at Sun Jan 12 23:04:14 2014)<br>Local db vers=
ion is 1388991001.15777<br>I am not sync site<br>Lowest host 10.38.10.7 was=
set 2 secs ago<br>Sync host 0.0.0.0 was set 1389567853 secs ago<br>The las=
t trans I handled was 0.46<br>
Sync site's db version is 1388991001.15777<br>0 locked pages, 0 of them=
for write<br><br>Server (10.36.10.7): (db 0.0)<br>=A0=A0=A0 last vote rcvd=
2 secs ago (at Sun Jan 12 23:04:14 2014),<br>=A0=A0=A0 last beacon sent 2 =
secs ago (at Sun Jan 12 23:04:14 2014), last vote was no<br>
=A0=A0=A0 dbcurrent=3D0, up=3D1 beaconSince=3D1<br><br>Server (10.33.10.43)=
: (db 0.0)<br>=A0=A0=A0 last vote never rcvd<br>=A0=A0=A0 last beacon never=
sent<br>=A0=A0=A0 dbcurrent=3D0, up=3D0 beaconSince=3D0</font><br></div><d=
iv><br></div>So it thinks it is not the sync site, but 0.0.0.0 is. In reali=
ty, server A is the sync site.<br>
<br>I have run tcpdump to make absolutely certain there are packets running=
between the two hosts on all AFS ports, and there are<font face=3D"courier=
new,monospace">.</font><br><div><div><div><div><div><div><div><div><div>
<div><br></div><div>KeyFiles are fine, and checked with md5sum.<br><br></di=
v><div>Set debug to 25 on the affected vlserver (10.38.10.7, Server C) and =
got this:<br><span style=3D"font-family:courier new,monospace"><br>Sun Jan =
12 23:19:24 2014 Using 10.38.10.7 as my primary address<br>
Sun Jan 12 23:19:40 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:19:40 201=
4 Starting AFS vlserver 4 (/usr/lib/openafs/vlserver -rxbind -d 25)<br>Sun =
Jan 12 23:19:40 2014 no vote from 10.36.10.7<br>@(#) OpenAFS 1.6.1-1+ubuntu=
0.2-debian built=A0 2013-07-24<br>
12 23:19:40 2014 Received beacon type 0 from host 10.38.10.7<br>Sun Jan 12 =
23:19:44 2014 recovery running in state 0<br>Sun Jan 12 23:19:55 2014 beaco=
n: amSyncSite is 0<br>Sun Jan 12 23:19:55 2014 no vote from 10.36.10.7<br>
Sun Jan 12 23:19:55 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:19:55 201=
4 Received beacon type 0 from host 10.38.10.7<br>Sun Jan 12 23:20:10 2014 b=
eacon: amSyncSite is 0<br>Sun Jan 12 23:20:10 2014 no vote from 10.36.10.7<=
br>
Sun Jan 12 23:20:10 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:10 201=
4 Received beacon type 0 from host 10.38.10.7<br>Sun Jan 12 23:20:25 2014 b=
eacon: amSyncSite is 0<br>Sun Jan 12 23:20:25 2014 no vote from 10.36.10.7<=
br>
Sun Jan 12 23:20:25 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:25 201=
4 Received beacon type 0 from host 10.38.10.7<br>Sun Jan 12 23:20:40 2014 u=
bik:server 10.33.10.43 still down<br>Sun Jan 12 23:20:40 2014 beacon: amSyn=
cSite is 0<br>
Sun Jan 12 23:20:40 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:40 201=
4 no vote from 10.36.10.7<br>Sun Jan 12 23:20:40 2014 beacon: amSyncSite is=
0<br>Sun Jan 12 23:20:40 2014 Received beacon type 0 from host 10.38.10.7<=
br>
Sun Jan 12 23:20:40 2014 Ubik: vote 'yes' for 10.38.10.7 (NOT in qu=
orum)<br>Sun Jan 12 23:20:44 2014 recovery running in state 0<br>Sun Jan 12=
23:20:44 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:48 2014 recovery=
running in state 0<br>
Sun Jan 12 23:20:48 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:52 201=
4 recovery running in state 0<br>Sun Jan 12 23:20:52 2014 beacon: amSyncSit=
e is 0<br>Sun Jan 12 23:20:55 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23=
:20:55 2014 no vote from 10.36.10.7<br>
Sun Jan 12 23:20:55 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:55 201=
4 Received beacon type 0 from host 10.38.10.7<br>Sun Jan 12 23:20:56 2014 r=
ecovery running in state 0<br>Sun Jan 12 23:20:56 2014 beacon: amSyncSite i=
s 0<br>
Sun Jan 12 23:21:00 2014 recovery running in state 0<br>Sun Jan 12 23:21:00=
2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:21:04 2014 recovery running =
in state 0<br>Sun Jan 12 23:21:04 2014 beacon: amSyncSite is 0<br>Sun Jan 1=
2 23:21:07 2014 allbetter checking<br>
Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014=
allbetter checking<br>Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>S=
un Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 allb=
etter: returning 1<br>
Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 all=
better: returning 1<br>Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun J=
an 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014 GetV=
olumeByID 536870913 (2) 10.38.10.83 noauth<br>
Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 all=
better: returning 1<br>Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun J=
an 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014 allb=
etter checking<br>
Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014=
allbetter checking<br>Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>S=
un Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 allb=
etter: returning 1<br>
Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 all=
better: returning 1<br>Sun Jan 12 23:21:07 2014 GetVolumeByID 536870913 (2)=
10.38.10.83 noauth<br>Sun Jan 12 23:21:07 2014 allbetter checking<br>
Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>
Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 all=
better: returning 1<br>Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun J=
an 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014 allb=
etter checking<br>
Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014=
allbetter checking<br>Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>S=
un Jan 12 23:21:08 2014 recovery running in state 0<br>Sun Jan 12 23:21:08 =
2014 beacon: amSyncSite is 0<br>
Sun Jan 12 23:21:10 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:21:10 201=
4 no vote from 10.36.10.7<br>Sun Jan 12 23:21:10 2014 beacon: amSyncSite is=
0<br>Sun Jan 12 23:21:10 2014 Received beacon type 0 from host 10.38.10.7<=
br>
Sun Jan 12 23:21:12 2014 recovery running in state 0<br>Sun Jan 12 23:21:25=
2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:21:25 2014 no vote from 10.3=
6.10.7<br>Sun Jan 12 23:21:25 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23=
:21:25 2014 Received beacon type 0 from host 10.38.10.7<br>
Sun Jan 12 23:21:29 2014 allbetter checking<br>Sun Jan 12 23:21:29 2014 all=
better: returning 1<br>Sun Jan 12 23:21:29 2014 GetVolumeByName <snipped=
><br>Sun Jan 12 23:21:29 2014 allbetter checking<br>Sun Jan 12 23:21:29 =
2014 allbetter: returning 1<br>
Sun Jan 12 23:21:29 2014 allbetter checking<br>Sun Jan 12 23:21:29 2014 all=
better: returning 1<br>Sun Jan 12 23:21:29 2014 allbetter checking<br>Sun J=
an 12 23:21:29 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014 allb=
etter checking<br>
Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014=
GetVolumeByName <i><snipped></i><br>Sun Jan 12 23:21:30 2014 allbett=
er checking<br>Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 1=
2 23:21:30 2014 allbetter checking<br>
Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014=
allbetter checking<br>Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>S=
un Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 allb=
etter: returning 1<br>
Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 all=
better: returning 1<br>Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun J=
an 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014 GetV=
olumeByName <i><snipped></i><br>
Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 all=
better: returning 1<br>Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun J=
an 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014 allb=
etter checking<br>
Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014=
allbetter checking<br>Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>S=
un Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 allb=
etter: returning 1<br>
Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 all=
better: returning 1<br>Sun Jan 12 23:21:30 2014 GetVolumeByName <i><snip=
ped></i><br>Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23=
:21:30 2014 allbetter: returning 1<br>
Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 all=
better: returning 1<br>Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun J=
an 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:40 2014 beac=
on: amSyncSite is 0<br>
Sun Jan 12 23:21:40 2014 no vote from 10.36.10.7<br>Sun Jan 12 23:21:40 201=
4 beacon: amSyncSite is 0<br>Sun Jan 12 23:21:40 2014 Received beacon type =
0 from host 10.38.10.7<br>Sun Jan 12 23:21:55 2014 beacon: amSyncSite is 0<=
br>
Sun Jan 12 23:21:55 2014 no vote from 10.36.10.7<br>Sun Jan 12 23:21:55 201=
4 beacon: amSyncSite is 0<br>Sun Jan 12 23:21:55 2014 Received beacon type =
0 from host 10.38.10.7</span><br></div><div><br></div><div>Any help is appr=
eciated! I have already upgraded and rebooted all three servers, lowest to =
highest, to no avail. I also attempted moving the data files out of the way=
and starting the affected vlserver, but the same symptoms remain.<span><fo=
nt color=3D"#888888"><br>
<br></font></span></div><span><font color=3D"#888888"><div>-- <br><span sty=
le=3D"border-collapse:collapse;color:rgb(102,102,102);font-family:verdana,s=
ans-serif;font-size:x-small">Timothy Balcer / IT Services<br>
Telmate / San Francisco, CA<br>Direct / </span><span style=3D"border-collap=
se:collapse;font-family:verdana,sans-serif;font-size:x-small"><font color=
=3D"#1155cc"><a href=3D"tel:%28415%29%20300-4313" value=3D"+14153004313" ta=
rget=3D"_blank">(415) 300-4313</a></font><br>
<font color=3D"#666666">Customer Service /=A0</font><a value=3D"+1800205551=
0" style=3D"color:rgb(17,85,204)">(800) 205-5510</a></span>
</div></font></span></div></div></div></div></div></div></div></div></div><=
/div>
</blockquote></div><br><br clear=3D"all"><br>-- <br><span style=3D"border-c=
ollapse:collapse;color:rgb(102,102,102);font-family:verdana,sans-serif;font=
-size:x-small">Timothy Balcer / IT Services<br>Telmate / San Francisco, CA<=
br>
Direct / </span><span style=3D"border-collapse:collapse;font-family:verdana=
,sans-serif;font-size:x-small"><font color=3D"#1155cc"><a href=3D"tel:%2841=
5%29%20300-4313" value=3D"+14153004313" target=3D"_blank">(415) 300-4313</a=
></font><br>
<font color=3D"#666666">Customer Service /=A0</font><a value=3D"+1800205551=
0" style=3D"color:rgb(17,85,204)">(800) 205-5510</a></span>
</div>
</div></div></blockquote></div><br><br clear=3D"all"><br>-- <br><span style=
=3D"border-collapse:collapse;color:rgb(102,102,102);font-family:verdana,san=
s-serif;font-size:x-small">Timothy Balcer / IT Services<br>Telmate / San Fr=
ancisco, CA<br>
Direct / </span><span style=3D"border-collapse:collapse;font-family:verdana=
,sans-serif;font-size:x-small"><font color=3D"#1155cc">(415) 300-4313</font=
><br><font color=3D"#666666">Customer Service /=A0</font><a value=3D"+18002=
055510" style=3D"color:rgb(17,85,204)">(800) 205-5510</a></span>
</div>
--001a11c2f58e0dace404efcf0b62--