[OpenAFS] Re: Ubik trouble
Timothy Balcer
timothy@telmate.com
Sun, 12 Jan 2014 16:17:28 -0800
--047d7b6783f84b780e04efcf00c9
Content-Type: text/plain; charset=ISO-8859-1
Wow. Just as you send it out....you find the problem. Always the way :-)
This turned out to be a subtle network problem. There was a change in the
cross colo VPN link by our provider, and while it didn't affect the
majority of our traffic, apparently -some- udp traffic was affected, due to
some encapsulation changes.
I discovered this when I decided to bypass the network mesh entirely, just
to 'try it', and things started sync'ing again.
I dug deeper, and there was indeed a subtle network fault.
So.. there you go.. if anyone else is beating their heads over a Ubik
problem like this, try bypassing parts of your network to see if there is a
network fault hiding amongst the trees.
Best,
On Sun, Jan 12, 2014 at 3:54 PM, Timothy Balcer <timothy@telmate.com> wrote:
> Hey folks. Odd problem. I have gone over many things, and I am stumped. I
> am tempted to just destroy and rebuild my errant vlserver, but I'd like to
> know what's going on and I know I am missing something :-)
>
> I have three vlservers, one in each of my colos. Lets call them A
> (10.33.10.43), B(10.36.10.7) and C(10.38.10.7)
>
> A and B sync with each other
> B and C sync with each other (or try to) but fail
> A and C do not.
>
> They do know about each other, as a udebug -long shows that. However
> between A and C I see this in the long form:
>
> Server (10.38.10.7): (db 0.0)
> last vote never rcvd
> last beacon never sent
> dbcurrent=0, up=0 beaconSince=0
>
> Same is true for C to A.
>
> The database version is completely off between A and C.
>
> This is the complete udebug from server C, which is the one that is acting
> up:
>
> Host's addresses are: 10.38.10.7
> Host's 10.38.10.7 time is Sun Jan 12 23:04:13 2014
> Local time is Sun Jan 12 23:04:16 2014 (time differential 3 secs)
> Last yes vote for 10.38.10.7 was 2 secs ago (not sync site);
> Last vote started 2 secs ago (at Sun Jan 12 23:04:14 2014)
> Local db version is 1388991001.15777
> I am not sync site
> Lowest host 10.38.10.7 was set 2 secs ago
> Sync host 0.0.0.0 was set 1389567853 secs ago
> The last trans I handled was 0.46
> Sync site's db version is 1388991001.15777
> 0 locked pages, 0 of them for write
>
> Server (10.36.10.7): (db 0.0)
> last vote rcvd 2 secs ago (at Sun Jan 12 23:04:14 2014),
> last beacon sent 2 secs ago (at Sun Jan 12 23:04:14 2014), last vote
> was no
> dbcurrent=0, up=1 beaconSince=1
>
> Server (10.33.10.43): (db 0.0)
> last vote never rcvd
> last beacon never sent
> dbcurrent=0, up=0 beaconSince=0
>
> So it thinks it is not the sync site, but 0.0.0.0 is. In reality, server A
> is the sync site.
>
> I have run tcpdump to make absolutely certain there are packets running
> between the two hosts on all AFS ports, and there are.
>
> KeyFiles are fine, and checked with md5sum.
>
> Set debug to 25 on the affected vlserver (10.38.10.7, Server C) and got
> this:
>
> Sun Jan 12 23:19:24 2014 Using 10.38.10.7 as my primary address
> Sun Jan 12 23:19:40 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:19:40 2014 Starting AFS vlserver 4
> (/usr/lib/openafs/vlserver -rxbind -d 25)
> Sun Jan 12 23:19:40 2014 no vote from 10.36.10.7
> @(#) OpenAFS 1.6.1-1+ubuntu0.2-debian built 2013-07-24
> 12 23:19:40 2014 Received beacon type 0 from host 10.38.10.7
> Sun Jan 12 23:19:44 2014 recovery running in state 0
> Sun Jan 12 23:19:55 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:19:55 2014 no vote from 10.36.10.7
> Sun Jan 12 23:19:55 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:19:55 2014 Received beacon type 0 from host 10.38.10.7
> Sun Jan 12 23:20:10 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:20:10 2014 no vote from 10.36.10.7
> Sun Jan 12 23:20:10 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:20:10 2014 Received beacon type 0 from host 10.38.10.7
> Sun Jan 12 23:20:25 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:20:25 2014 no vote from 10.36.10.7
> Sun Jan 12 23:20:25 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:20:25 2014 Received beacon type 0 from host 10.38.10.7
> Sun Jan 12 23:20:40 2014 ubik:server 10.33.10.43 still down
> Sun Jan 12 23:20:40 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:20:40 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:20:40 2014 no vote from 10.36.10.7
> Sun Jan 12 23:20:40 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:20:40 2014 Received beacon type 0 from host 10.38.10.7
> Sun Jan 12 23:20:40 2014 Ubik: vote 'yes' for 10.38.10.7 (NOT in quorum)
> Sun Jan 12 23:20:44 2014 recovery running in state 0
> Sun Jan 12 23:20:44 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:20:48 2014 recovery running in state 0
> Sun Jan 12 23:20:48 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:20:52 2014 recovery running in state 0
> Sun Jan 12 23:20:52 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:20:55 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:20:55 2014 no vote from 10.36.10.7
> Sun Jan 12 23:20:55 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:20:55 2014 Received beacon type 0 from host 10.38.10.7
> Sun Jan 12 23:20:56 2014 recovery running in state 0
> Sun Jan 12 23:20:56 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:21:00 2014 recovery running in state 0
> Sun Jan 12 23:21:00 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:21:04 2014 recovery running in state 0
> Sun Jan 12 23:21:04 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:21:07 2014 allbetter checking
> Sun Jan 12 23:21:07 2014 allbetter: returning 1
> Sun Jan 12 23:21:07 2014 allbetter checking
> Sun Jan 12 23:21:07 2014 allbetter: returning 1
> Sun Jan 12 23:21:07 2014 allbetter checking
> Sun Jan 12 23:21:07 2014 allbetter: returning 1
> Sun Jan 12 23:21:07 2014 allbetter checking
> Sun Jan 12 23:21:07 2014 allbetter: returning 1
> Sun Jan 12 23:21:07 2014 allbetter checking
> Sun Jan 12 23:21:07 2014 allbetter: returning 1
> Sun Jan 12 23:21:07 2014 GetVolumeByID 536870913 (2) 10.38.10.83 noauth
> Sun Jan 12 23:21:07 2014 allbetter checking
> Sun Jan 12 23:21:07 2014 allbetter: returning 1
> Sun Jan 12 23:21:07 2014 allbetter checking
> Sun Jan 12 23:21:07 2014 allbetter: returning 1
> Sun Jan 12 23:21:07 2014 allbetter checking
> Sun Jan 12 23:21:07 2014 allbetter: returning 1
> Sun Jan 12 23:21:07 2014 allbetter checking
> Sun Jan 12 23:21:07 2014 allbetter: returning 1
> Sun Jan 12 23:21:07 2014 allbetter checking
> Sun Jan 12 23:21:07 2014 allbetter: returning 1
> Sun Jan 12 23:21:07 2014 allbetter checking
> Sun Jan 12 23:21:07 2014 allbetter: returning 1
> Sun Jan 12 23:21:07 2014 GetVolumeByID 536870913 (2) 10.38.10.83 noauth
> Sun Jan 12 23:21:07 2014 allbetter checking
> Sun Jan 12 23:21:07 2014 allbetter: returning 1
> Sun Jan 12 23:21:07 2014 allbetter checking
> Sun Jan 12 23:21:07 2014 allbetter: returning 1
> Sun Jan 12 23:21:07 2014 allbetter checking
> Sun Jan 12 23:21:07 2014 allbetter: returning 1
> Sun Jan 12 23:21:07 2014 allbetter checking
> Sun Jan 12 23:21:07 2014 allbetter: returning 1
> Sun Jan 12 23:21:07 2014 allbetter checking
> Sun Jan 12 23:21:07 2014 allbetter: returning 1
> Sun Jan 12 23:21:08 2014 recovery running in state 0
> Sun Jan 12 23:21:08 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:21:10 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:21:10 2014 no vote from 10.36.10.7
> Sun Jan 12 23:21:10 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:21:10 2014 Received beacon type 0 from host 10.38.10.7
> Sun Jan 12 23:21:12 2014 recovery running in state 0
> Sun Jan 12 23:21:25 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:21:25 2014 no vote from 10.36.10.7
> Sun Jan 12 23:21:25 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:21:25 2014 Received beacon type 0 from host 10.38.10.7
> Sun Jan 12 23:21:29 2014 allbetter checking
> Sun Jan 12 23:21:29 2014 allbetter: returning 1
> Sun Jan 12 23:21:29 2014 GetVolumeByName <snipped>
> Sun Jan 12 23:21:29 2014 allbetter checking
> Sun Jan 12 23:21:29 2014 allbetter: returning 1
> Sun Jan 12 23:21:29 2014 allbetter checking
> Sun Jan 12 23:21:29 2014 allbetter: returning 1
> Sun Jan 12 23:21:29 2014 allbetter checking
> Sun Jan 12 23:21:29 2014 allbetter: returning 1
> Sun Jan 12 23:21:30 2014 allbetter checking
> Sun Jan 12 23:21:30 2014 allbetter: returning 1
> Sun Jan 12 23:21:30 2014 GetVolumeByName *<snipped>*
> Sun Jan 12 23:21:30 2014 allbetter checking
> Sun Jan 12 23:21:30 2014 allbetter: returning 1
> Sun Jan 12 23:21:30 2014 allbetter checking
> Sun Jan 12 23:21:30 2014 allbetter: returning 1
> Sun Jan 12 23:21:30 2014 allbetter checking
> Sun Jan 12 23:21:30 2014 allbetter: returning 1
> Sun Jan 12 23:21:30 2014 allbetter checking
> Sun Jan 12 23:21:30 2014 allbetter: returning 1
> Sun Jan 12 23:21:30 2014 allbetter checking
> Sun Jan 12 23:21:30 2014 allbetter: returning 1
> Sun Jan 12 23:21:30 2014 allbetter checking
> Sun Jan 12 23:21:30 2014 allbetter: returning 1
> Sun Jan 12 23:21:30 2014 GetVolumeByName *<snipped>*
> Sun Jan 12 23:21:30 2014 allbetter checking
> Sun Jan 12 23:21:30 2014 allbetter: returning 1
> Sun Jan 12 23:21:30 2014 allbetter checking
> Sun Jan 12 23:21:30 2014 allbetter: returning 1
> Sun Jan 12 23:21:30 2014 allbetter checking
> Sun Jan 12 23:21:30 2014 allbetter: returning 1
> Sun Jan 12 23:21:30 2014 allbetter checking
> Sun Jan 12 23:21:30 2014 allbetter: returning 1
> Sun Jan 12 23:21:30 2014 allbetter checking
> Sun Jan 12 23:21:30 2014 allbetter: returning 1
> Sun Jan 12 23:21:30 2014 allbetter checking
> Sun Jan 12 23:21:30 2014 allbetter: returning 1
> Sun Jan 12 23:21:30 2014 GetVolumeByName *<snipped>*
> Sun Jan 12 23:21:30 2014 allbetter checking
> Sun Jan 12 23:21:30 2014 allbetter: returning 1
> Sun Jan 12 23:21:30 2014 allbetter checking
> Sun Jan 12 23:21:30 2014 allbetter: returning 1
> Sun Jan 12 23:21:30 2014 allbetter checking
> Sun Jan 12 23:21:30 2014 allbetter: returning 1
> Sun Jan 12 23:21:40 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:21:40 2014 no vote from 10.36.10.7
> Sun Jan 12 23:21:40 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:21:40 2014 Received beacon type 0 from host 10.38.10.7
> Sun Jan 12 23:21:55 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:21:55 2014 no vote from 10.36.10.7
> Sun Jan 12 23:21:55 2014 beacon: amSyncSite is 0
> Sun Jan 12 23:21:55 2014 Received beacon type 0 from host 10.38.10.7
>
> Any help is appreciated! I have already upgraded and rebooted all three
> servers, lowest to highest, to no avail. I also attempted moving the data
> files out of the way and starting the affected vlserver, but the same
> symptoms remain.
>
> --
> Timothy Balcer / IT Services
> Telmate / San Francisco, CA
> Direct / (415) 300-4313
> Customer Service / (800) 205-5510
>
--
Timothy Balcer / IT Services
Telmate / San Francisco, CA
Direct / (415) 300-4313
Customer Service / (800) 205-5510
--047d7b6783f84b780e04efcf00c9
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div><div><div><div><div>Wow. Just as you send it out....y=
ou find the problem. Always the way :-)<br><br></div>This turned out to be =
a subtle network problem. There was a change in the cross colo VPN link by =
our provider, and while it didn't affect the majority of our traffic, a=
pparently -some- udp traffic was affected, due to some encapsulation change=
s.<br>
<br></div>I discovered this when I decided to bypass the network mesh entir=
ely, just to 'try it', and things started sync'ing again.<br><b=
r></div>I dug deeper, and there was indeed a subtle network fault.<br><br>
</div>So.. there you go.. if anyone else is beating their heads over a Ubik=
problem like this, try bypassing parts of your network to see if there is =
a network fault hiding amongst the trees.<br><br></div>Best,<br></div><div =
class=3D"gmail_extra">
<br><br><div class=3D"gmail_quote">On Sun, Jan 12, 2014 at 3:54 PM, Timothy=
Balcer <span dir=3D"ltr"><<a href=3D"mailto:timothy@telmate.com" target=
=3D"_blank">timothy@telmate.com</a>></span> wrote:<br><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex">
<div dir=3D"ltr"><div><div><div><div><div><div><div><div>Hey folks. Odd pro=
blem. I have gone over many things, and I am stumped. I am tempted to just =
destroy and rebuild my errant vlserver, but I'd like to know what's=
going on and I know I am missing something :-)<br>
<br></div>I have three vlservers, one in each of my colos. Lets call them A=
(10.33.10.43), B(10.36.10.7) and C(10.38.10.7)<br><br></div>A and B sync w=
ith each other<br></div>B and C sync with each other (or try to) but fail<b=
r>
</div>A and C do not.<br><br></div>They do know about each other, as a udeb=
ug -long shows that. However between A and C I see this in the long form:<b=
r><br>Server (10.38.10.7): (db 0.0)<br>=A0=A0=A0 last vote never rcvd<br>=
=A0=A0=A0 last beacon never sent<br>
=A0=A0=A0 dbcurrent=3D0, up=3D0 beaconSince=3D0<br><br></div>Same is true f=
or C to A.<br><br></div>The database version is completely off between A an=
d C.<br><br></div><div>This is the complete udebug from server C, which is =
the one that is acting up:<br>
<br><font face=3D"courier new,monospace">Host's addresses are: 10.38.10=
.7<br>Host's 10.38.10.7 time is Sun Jan 12 23:04:13 2014<br>Local time =
is Sun Jan 12 23:04:16 2014 (time differential 3 secs)<br>Last yes vote for=
10.38.10.7 was 2 secs ago (not sync site);<br>
Last vote started 2 secs ago (at Sun Jan 12 23:04:14 2014)<br>Local db vers=
ion is 1388991001.15777<br>I am not sync site<br>Lowest host 10.38.10.7 was=
set 2 secs ago<br>Sync host 0.0.0.0 was set 1389567853 secs ago<br>The las=
t trans I handled was 0.46<br>
Sync site's db version is 1388991001.15777<br>0 locked pages, 0 of them=
for write<br><br>Server (10.36.10.7): (db 0.0)<br>=A0=A0=A0 last vote rcvd=
2 secs ago (at Sun Jan 12 23:04:14 2014),<br>=A0=A0=A0 last beacon sent 2 =
secs ago (at Sun Jan 12 23:04:14 2014), last vote was no<br>
=A0=A0=A0 dbcurrent=3D0, up=3D1 beaconSince=3D1<br><br>Server (10.33.10.43)=
: (db 0.0)<br>=A0=A0=A0 last vote never rcvd<br>=A0=A0=A0 last beacon never=
sent<br>=A0=A0=A0 dbcurrent=3D0, up=3D0 beaconSince=3D0</font><br></div><d=
iv><br></div>So it thinks it is not the sync site, but 0.0.0.0 is. In reali=
ty, server A is the sync site.<br>
<br>I have run tcpdump to make absolutely certain there are packets running=
between the two hosts on all AFS ports, and there are<font face=3D"courier=
new,monospace">.</font><br><div><div><div><div><div><div><div><div><div>
<div><br></div><div>KeyFiles are fine, and checked with md5sum.<br><br></di=
v><div>Set debug to 25 on the affected vlserver (10.38.10.7, Server C) and =
got this:<br><span style=3D"font-family:courier new,monospace"><br>Sun Jan =
12 23:19:24 2014 Using 10.38.10.7 as my primary address<br>
Sun Jan 12 23:19:40 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:19:40 201=
4 Starting AFS vlserver 4 (/usr/lib/openafs/vlserver -rxbind -d 25)<br>Sun =
Jan 12 23:19:40 2014 no vote from 10.36.10.7<br>@(#) OpenAFS 1.6.1-1+ubuntu=
0.2-debian built=A0 2013-07-24<br>
12 23:19:40 2014 Received beacon type 0 from host 10.38.10.7<br>Sun Jan 12 =
23:19:44 2014 recovery running in state 0<br>Sun Jan 12 23:19:55 2014 beaco=
n: amSyncSite is 0<br>Sun Jan 12 23:19:55 2014 no vote from 10.36.10.7<br>
Sun Jan 12 23:19:55 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:19:55 201=
4 Received beacon type 0 from host 10.38.10.7<br>Sun Jan 12 23:20:10 2014 b=
eacon: amSyncSite is 0<br>Sun Jan 12 23:20:10 2014 no vote from 10.36.10.7<=
br>
Sun Jan 12 23:20:10 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:10 201=
4 Received beacon type 0 from host 10.38.10.7<br>Sun Jan 12 23:20:25 2014 b=
eacon: amSyncSite is 0<br>Sun Jan 12 23:20:25 2014 no vote from 10.36.10.7<=
br>
Sun Jan 12 23:20:25 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:25 201=
4 Received beacon type 0 from host 10.38.10.7<br>Sun Jan 12 23:20:40 2014 u=
bik:server 10.33.10.43 still down<br>Sun Jan 12 23:20:40 2014 beacon: amSyn=
cSite is 0<br>
Sun Jan 12 23:20:40 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:40 201=
4 no vote from 10.36.10.7<br>Sun Jan 12 23:20:40 2014 beacon: amSyncSite is=
0<br>Sun Jan 12 23:20:40 2014 Received beacon type 0 from host 10.38.10.7<=
br>
Sun Jan 12 23:20:40 2014 Ubik: vote 'yes' for 10.38.10.7 (NOT in qu=
orum)<br>Sun Jan 12 23:20:44 2014 recovery running in state 0<br>Sun Jan 12=
23:20:44 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:48 2014 recovery=
running in state 0<br>
Sun Jan 12 23:20:48 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:52 201=
4 recovery running in state 0<br>Sun Jan 12 23:20:52 2014 beacon: amSyncSit=
e is 0<br>Sun Jan 12 23:20:55 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23=
:20:55 2014 no vote from 10.36.10.7<br>
Sun Jan 12 23:20:55 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:55 201=
4 Received beacon type 0 from host 10.38.10.7<br>Sun Jan 12 23:20:56 2014 r=
ecovery running in state 0<br>Sun Jan 12 23:20:56 2014 beacon: amSyncSite i=
s 0<br>
Sun Jan 12 23:21:00 2014 recovery running in state 0<br>Sun Jan 12 23:21:00=
2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:21:04 2014 recovery running =
in state 0<br>Sun Jan 12 23:21:04 2014 beacon: amSyncSite is 0<br>Sun Jan 1=
2 23:21:07 2014 allbetter checking<br>
Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014=
allbetter checking<br>Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>S=
un Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 allb=
etter: returning 1<br>
Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 all=
better: returning 1<br>Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun J=
an 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014 GetV=
olumeByID 536870913 (2) 10.38.10.83 noauth<br>
Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 all=
better: returning 1<br>Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun J=
an 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014 allb=
etter checking<br>
Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014=
allbetter checking<br>Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>S=
un Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 allb=
etter: returning 1<br>
Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 all=
better: returning 1<br>Sun Jan 12 23:21:07 2014 GetVolumeByID 536870913 (2)=
10.38.10.83 noauth<br>Sun Jan 12 23:21:07 2014 allbetter checking<br>
Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>
Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 all=
better: returning 1<br>Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun J=
an 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014 allb=
etter checking<br>
Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014=
allbetter checking<br>Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>S=
un Jan 12 23:21:08 2014 recovery running in state 0<br>Sun Jan 12 23:21:08 =
2014 beacon: amSyncSite is 0<br>
Sun Jan 12 23:21:10 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:21:10 201=
4 no vote from 10.36.10.7<br>Sun Jan 12 23:21:10 2014 beacon: amSyncSite is=
0<br>Sun Jan 12 23:21:10 2014 Received beacon type 0 from host 10.38.10.7<=
br>
Sun Jan 12 23:21:12 2014 recovery running in state 0<br>Sun Jan 12 23:21:25=
2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:21:25 2014 no vote from 10.3=
6.10.7<br>Sun Jan 12 23:21:25 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23=
:21:25 2014 Received beacon type 0 from host 10.38.10.7<br>
Sun Jan 12 23:21:29 2014 allbetter checking<br>Sun Jan 12 23:21:29 2014 all=
better: returning 1<br>Sun Jan 12 23:21:29 2014 GetVolumeByName <snipped=
><br>Sun Jan 12 23:21:29 2014 allbetter checking<br>Sun Jan 12 23:21:29 =
2014 allbetter: returning 1<br>
Sun Jan 12 23:21:29 2014 allbetter checking<br>Sun Jan 12 23:21:29 2014 all=
better: returning 1<br>Sun Jan 12 23:21:29 2014 allbetter checking<br>Sun J=
an 12 23:21:29 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014 allb=
etter checking<br>
Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014=
GetVolumeByName <i><snipped></i><br>Sun Jan 12 23:21:30 2014 allbett=
er checking<br>Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 1=
2 23:21:30 2014 allbetter checking<br>
Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014=
allbetter checking<br>Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>S=
un Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 allb=
etter: returning 1<br>
Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 all=
better: returning 1<br>Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun J=
an 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014 GetV=
olumeByName <i><snipped></i><br>
Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 all=
better: returning 1<br>Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun J=
an 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014 allb=
etter checking<br>
Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014=
allbetter checking<br>Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>S=
un Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 allb=
etter: returning 1<br>
Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 all=
better: returning 1<br>Sun Jan 12 23:21:30 2014 GetVolumeByName <i><snip=
ped></i><br>Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23=
:21:30 2014 allbetter: returning 1<br>
Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 all=
better: returning 1<br>Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun J=
an 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:40 2014 beac=
on: amSyncSite is 0<br>
Sun Jan 12 23:21:40 2014 no vote from 10.36.10.7<br>Sun Jan 12 23:21:40 201=
4 beacon: amSyncSite is 0<br>Sun Jan 12 23:21:40 2014 Received beacon type =
0 from host 10.38.10.7<br>Sun Jan 12 23:21:55 2014 beacon: amSyncSite is 0<=
br>
Sun Jan 12 23:21:55 2014 no vote from 10.36.10.7<br>Sun Jan 12 23:21:55 201=
4 beacon: amSyncSite is 0<br>Sun Jan 12 23:21:55 2014 Received beacon type =
0 from host 10.38.10.7</span><br></div><div><br></div><div>Any help is appr=
eciated! I have already upgraded and rebooted all three servers, lowest to =
highest, to no avail. I also attempted moving the data files out of the way=
and starting the affected vlserver, but the same symptoms remain.<span cla=
ss=3D"HOEnZb"><font color=3D"#888888"><br>
<br></font></span></div><span class=3D"HOEnZb"><font color=3D"#888888"><div=
>-- <br><span style=3D"border-collapse:collapse;color:rgb(102,102,102);font=
-family:verdana,sans-serif;font-size:x-small">Timothy Balcer / IT Services<=
br>
Telmate / San Francisco, CA<br>Direct / </span><span style=3D"border-collap=
se:collapse;font-family:verdana,sans-serif;font-size:x-small"><font color=
=3D"#1155cc"><a href=3D"tel:%28415%29%20300-4313" value=3D"+14153004313" ta=
rget=3D"_blank">(415) 300-4313</a></font><br>
<font color=3D"#666666">Customer Service /=A0</font><a value=3D"+1800205551=
0" style=3D"color:rgb(17,85,204)">(800) 205-5510</a></span>
</div></font></span></div></div></div></div></div></div></div></div></div><=
/div>
</blockquote></div><br><br clear=3D"all"><br>-- <br><span style=3D"border-c=
ollapse:collapse;color:rgb(102,102,102);font-family:verdana,sans-serif;font=
-size:x-small">Timothy Balcer / IT Services<br>Telmate / San Francisco, CA<=
br>
Direct / </span><span style=3D"border-collapse:collapse;font-family:verdana=
,sans-serif;font-size:x-small"><font color=3D"#1155cc">(415) 300-4313</font=
><br><font color=3D"#666666">Customer Service /=A0</font><a value=3D"+18002=
055510" style=3D"color:rgb(17,85,204)">(800) 205-5510</a></span>
</div>
--047d7b6783f84b780e04efcf00c9--