[OpenAFS] Ubik trouble
Timothy Balcer
timothy@telmate.com
Sun, 12 Jan 2014 15:54:58 -0800
--001a11348964d310bf04efceafe9
Content-Type: text/plain; charset=ISO-8859-1
Hey folks. Odd problem. I have gone over many things, and I am stumped. I
am tempted to just destroy and rebuild my errant vlserver, but I'd like to
know what's going on and I know I am missing something :-)
I have three vlservers, one in each of my colos. Lets call them A
(10.33.10.43), B(10.36.10.7) and C(10.38.10.7)
A and B sync with each other
B and C sync with each other (or try to) but fail
A and C do not.
They do know about each other, as a udebug -long shows that. However
between A and C I see this in the long form:
Server (10.38.10.7): (db 0.0)
last vote never rcvd
last beacon never sent
dbcurrent=0, up=0 beaconSince=0
Same is true for C to A.
The database version is completely off between A and C.
This is the complete udebug from server C, which is the one that is acting
up:
Host's addresses are: 10.38.10.7
Host's 10.38.10.7 time is Sun Jan 12 23:04:13 2014
Local time is Sun Jan 12 23:04:16 2014 (time differential 3 secs)
Last yes vote for 10.38.10.7 was 2 secs ago (not sync site);
Last vote started 2 secs ago (at Sun Jan 12 23:04:14 2014)
Local db version is 1388991001.15777
I am not sync site
Lowest host 10.38.10.7 was set 2 secs ago
Sync host 0.0.0.0 was set 1389567853 secs ago
The last trans I handled was 0.46
Sync site's db version is 1388991001.15777
0 locked pages, 0 of them for write
Server (10.36.10.7): (db 0.0)
last vote rcvd 2 secs ago (at Sun Jan 12 23:04:14 2014),
last beacon sent 2 secs ago (at Sun Jan 12 23:04:14 2014), last vote
was no
dbcurrent=0, up=1 beaconSince=1
Server (10.33.10.43): (db 0.0)
last vote never rcvd
last beacon never sent
dbcurrent=0, up=0 beaconSince=0
So it thinks it is not the sync site, but 0.0.0.0 is. In reality, server A
is the sync site.
I have run tcpdump to make absolutely certain there are packets running
between the two hosts on all AFS ports, and there are.
KeyFiles are fine, and checked with md5sum.
Set debug to 25 on the affected vlserver (10.38.10.7, Server C) and got
this:
Sun Jan 12 23:19:24 2014 Using 10.38.10.7 as my primary address
Sun Jan 12 23:19:40 2014 beacon: amSyncSite is 0
Sun Jan 12 23:19:40 2014 Starting AFS vlserver 4 (/usr/lib/openafs/vlserver
-rxbind -d 25)
Sun Jan 12 23:19:40 2014 no vote from 10.36.10.7
@(#) OpenAFS 1.6.1-1+ubuntu0.2-debian built 2013-07-24
12 23:19:40 2014 Received beacon type 0 from host 10.38.10.7
Sun Jan 12 23:19:44 2014 recovery running in state 0
Sun Jan 12 23:19:55 2014 beacon: amSyncSite is 0
Sun Jan 12 23:19:55 2014 no vote from 10.36.10.7
Sun Jan 12 23:19:55 2014 beacon: amSyncSite is 0
Sun Jan 12 23:19:55 2014 Received beacon type 0 from host 10.38.10.7
Sun Jan 12 23:20:10 2014 beacon: amSyncSite is 0
Sun Jan 12 23:20:10 2014 no vote from 10.36.10.7
Sun Jan 12 23:20:10 2014 beacon: amSyncSite is 0
Sun Jan 12 23:20:10 2014 Received beacon type 0 from host 10.38.10.7
Sun Jan 12 23:20:25 2014 beacon: amSyncSite is 0
Sun Jan 12 23:20:25 2014 no vote from 10.36.10.7
Sun Jan 12 23:20:25 2014 beacon: amSyncSite is 0
Sun Jan 12 23:20:25 2014 Received beacon type 0 from host 10.38.10.7
Sun Jan 12 23:20:40 2014 ubik:server 10.33.10.43 still down
Sun Jan 12 23:20:40 2014 beacon: amSyncSite is 0
Sun Jan 12 23:20:40 2014 beacon: amSyncSite is 0
Sun Jan 12 23:20:40 2014 no vote from 10.36.10.7
Sun Jan 12 23:20:40 2014 beacon: amSyncSite is 0
Sun Jan 12 23:20:40 2014 Received beacon type 0 from host 10.38.10.7
Sun Jan 12 23:20:40 2014 Ubik: vote 'yes' for 10.38.10.7 (NOT in quorum)
Sun Jan 12 23:20:44 2014 recovery running in state 0
Sun Jan 12 23:20:44 2014 beacon: amSyncSite is 0
Sun Jan 12 23:20:48 2014 recovery running in state 0
Sun Jan 12 23:20:48 2014 beacon: amSyncSite is 0
Sun Jan 12 23:20:52 2014 recovery running in state 0
Sun Jan 12 23:20:52 2014 beacon: amSyncSite is 0
Sun Jan 12 23:20:55 2014 beacon: amSyncSite is 0
Sun Jan 12 23:20:55 2014 no vote from 10.36.10.7
Sun Jan 12 23:20:55 2014 beacon: amSyncSite is 0
Sun Jan 12 23:20:55 2014 Received beacon type 0 from host 10.38.10.7
Sun Jan 12 23:20:56 2014 recovery running in state 0
Sun Jan 12 23:20:56 2014 beacon: amSyncSite is 0
Sun Jan 12 23:21:00 2014 recovery running in state 0
Sun Jan 12 23:21:00 2014 beacon: amSyncSite is 0
Sun Jan 12 23:21:04 2014 recovery running in state 0
Sun Jan 12 23:21:04 2014 beacon: amSyncSite is 0
Sun Jan 12 23:21:07 2014 allbetter checking
Sun Jan 12 23:21:07 2014 allbetter: returning 1
Sun Jan 12 23:21:07 2014 allbetter checking
Sun Jan 12 23:21:07 2014 allbetter: returning 1
Sun Jan 12 23:21:07 2014 allbetter checking
Sun Jan 12 23:21:07 2014 allbetter: returning 1
Sun Jan 12 23:21:07 2014 allbetter checking
Sun Jan 12 23:21:07 2014 allbetter: returning 1
Sun Jan 12 23:21:07 2014 allbetter checking
Sun Jan 12 23:21:07 2014 allbetter: returning 1
Sun Jan 12 23:21:07 2014 GetVolumeByID 536870913 (2) 10.38.10.83 noauth
Sun Jan 12 23:21:07 2014 allbetter checking
Sun Jan 12 23:21:07 2014 allbetter: returning 1
Sun Jan 12 23:21:07 2014 allbetter checking
Sun Jan 12 23:21:07 2014 allbetter: returning 1
Sun Jan 12 23:21:07 2014 allbetter checking
Sun Jan 12 23:21:07 2014 allbetter: returning 1
Sun Jan 12 23:21:07 2014 allbetter checking
Sun Jan 12 23:21:07 2014 allbetter: returning 1
Sun Jan 12 23:21:07 2014 allbetter checking
Sun Jan 12 23:21:07 2014 allbetter: returning 1
Sun Jan 12 23:21:07 2014 allbetter checking
Sun Jan 12 23:21:07 2014 allbetter: returning 1
Sun Jan 12 23:21:07 2014 GetVolumeByID 536870913 (2) 10.38.10.83 noauth
Sun Jan 12 23:21:07 2014 allbetter checking
Sun Jan 12 23:21:07 2014 allbetter: returning 1
Sun Jan 12 23:21:07 2014 allbetter checking
Sun Jan 12 23:21:07 2014 allbetter: returning 1
Sun Jan 12 23:21:07 2014 allbetter checking
Sun Jan 12 23:21:07 2014 allbetter: returning 1
Sun Jan 12 23:21:07 2014 allbetter checking
Sun Jan 12 23:21:07 2014 allbetter: returning 1
Sun Jan 12 23:21:07 2014 allbetter checking
Sun Jan 12 23:21:07 2014 allbetter: returning 1
Sun Jan 12 23:21:08 2014 recovery running in state 0
Sun Jan 12 23:21:08 2014 beacon: amSyncSite is 0
Sun Jan 12 23:21:10 2014 beacon: amSyncSite is 0
Sun Jan 12 23:21:10 2014 no vote from 10.36.10.7
Sun Jan 12 23:21:10 2014 beacon: amSyncSite is 0
Sun Jan 12 23:21:10 2014 Received beacon type 0 from host 10.38.10.7
Sun Jan 12 23:21:12 2014 recovery running in state 0
Sun Jan 12 23:21:25 2014 beacon: amSyncSite is 0
Sun Jan 12 23:21:25 2014 no vote from 10.36.10.7
Sun Jan 12 23:21:25 2014 beacon: amSyncSite is 0
Sun Jan 12 23:21:25 2014 Received beacon type 0 from host 10.38.10.7
Sun Jan 12 23:21:29 2014 allbetter checking
Sun Jan 12 23:21:29 2014 allbetter: returning 1
Sun Jan 12 23:21:29 2014 GetVolumeByName <snipped>
Sun Jan 12 23:21:29 2014 allbetter checking
Sun Jan 12 23:21:29 2014 allbetter: returning 1
Sun Jan 12 23:21:29 2014 allbetter checking
Sun Jan 12 23:21:29 2014 allbetter: returning 1
Sun Jan 12 23:21:29 2014 allbetter checking
Sun Jan 12 23:21:29 2014 allbetter: returning 1
Sun Jan 12 23:21:30 2014 allbetter checking
Sun Jan 12 23:21:30 2014 allbetter: returning 1
Sun Jan 12 23:21:30 2014 GetVolumeByName *<snipped>*
Sun Jan 12 23:21:30 2014 allbetter checking
Sun Jan 12 23:21:30 2014 allbetter: returning 1
Sun Jan 12 23:21:30 2014 allbetter checking
Sun Jan 12 23:21:30 2014 allbetter: returning 1
Sun Jan 12 23:21:30 2014 allbetter checking
Sun Jan 12 23:21:30 2014 allbetter: returning 1
Sun Jan 12 23:21:30 2014 allbetter checking
Sun Jan 12 23:21:30 2014 allbetter: returning 1
Sun Jan 12 23:21:30 2014 allbetter checking
Sun Jan 12 23:21:30 2014 allbetter: returning 1
Sun Jan 12 23:21:30 2014 allbetter checking
Sun Jan 12 23:21:30 2014 allbetter: returning 1
Sun Jan 12 23:21:30 2014 GetVolumeByName *<snipped>*
Sun Jan 12 23:21:30 2014 allbetter checking
Sun Jan 12 23:21:30 2014 allbetter: returning 1
Sun Jan 12 23:21:30 2014 allbetter checking
Sun Jan 12 23:21:30 2014 allbetter: returning 1
Sun Jan 12 23:21:30 2014 allbetter checking
Sun Jan 12 23:21:30 2014 allbetter: returning 1
Sun Jan 12 23:21:30 2014 allbetter checking
Sun Jan 12 23:21:30 2014 allbetter: returning 1
Sun Jan 12 23:21:30 2014 allbetter checking
Sun Jan 12 23:21:30 2014 allbetter: returning 1
Sun Jan 12 23:21:30 2014 allbetter checking
Sun Jan 12 23:21:30 2014 allbetter: returning 1
Sun Jan 12 23:21:30 2014 GetVolumeByName *<snipped>*
Sun Jan 12 23:21:30 2014 allbetter checking
Sun Jan 12 23:21:30 2014 allbetter: returning 1
Sun Jan 12 23:21:30 2014 allbetter checking
Sun Jan 12 23:21:30 2014 allbetter: returning 1
Sun Jan 12 23:21:30 2014 allbetter checking
Sun Jan 12 23:21:30 2014 allbetter: returning 1
Sun Jan 12 23:21:40 2014 beacon: amSyncSite is 0
Sun Jan 12 23:21:40 2014 no vote from 10.36.10.7
Sun Jan 12 23:21:40 2014 beacon: amSyncSite is 0
Sun Jan 12 23:21:40 2014 Received beacon type 0 from host 10.38.10.7
Sun Jan 12 23:21:55 2014 beacon: amSyncSite is 0
Sun Jan 12 23:21:55 2014 no vote from 10.36.10.7
Sun Jan 12 23:21:55 2014 beacon: amSyncSite is 0
Sun Jan 12 23:21:55 2014 Received beacon type 0 from host 10.38.10.7
Any help is appreciated! I have already upgraded and rebooted all three
servers, lowest to highest, to no avail. I also attempted moving the data
files out of the way and starting the affected vlserver, but the same
symptoms remain.
--
Timothy Balcer / IT Services
Telmate / San Francisco, CA
Direct / (415) 300-4313
Customer Service / (800) 205-5510
--001a11348964d310bf04efceafe9
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div><div><div><div><div><div><div><div>Hey folks. Odd pro=
blem. I have gone over many things, and I am stumped. I am tempted to just =
destroy and rebuild my errant vlserver, but I'd like to know what's=
going on and I know I am missing something :-)<br>
<br></div>I have three vlservers, one in each of my colos. Lets call them A=
(10.33.10.43), B(10.36.10.7) and C(10.38.10.7)<br><br></div>A and B sync w=
ith each other<br></div>B and C sync with each other (or try to) but fail<b=
r>
</div>A and C do not.<br><br></div>They do know about each other, as a udeb=
ug -long shows that. However between A and C I see this in the long form:<b=
r><br>Server (10.38.10.7): (db 0.0)<br>=A0=A0=A0 last vote never rcvd<br>=
=A0=A0=A0 last beacon never sent<br>
=A0=A0=A0 dbcurrent=3D0, up=3D0 beaconSince=3D0<br><br></div>Same is true f=
or C to A.<br><br></div>The database version is completely off between A an=
d C.<br><br></div><div>This is the complete udebug from server C, which is =
the one that is acting up:<br>
<br><font face=3D"courier new,monospace">Host's addresses are: 10.38.10=
.7<br>Host's 10.38.10.7 time is Sun Jan 12 23:04:13 2014<br>Local time =
is Sun Jan 12 23:04:16 2014 (time differential 3 secs)<br>Last yes vote for=
10.38.10.7 was 2 secs ago (not sync site);<br>
Last vote started 2 secs ago (at Sun Jan 12 23:04:14 2014)<br>Local db vers=
ion is 1388991001.15777<br>I am not sync site<br>Lowest host 10.38.10.7 was=
set 2 secs ago<br>Sync host 0.0.0.0 was set 1389567853 secs ago<br>The las=
t trans I handled was 0.46<br>
Sync site's db version is 1388991001.15777<br>0 locked pages, 0 of them=
for write<br><br>Server (10.36.10.7): (db 0.0)<br>=A0=A0=A0 last vote rcvd=
2 secs ago (at Sun Jan 12 23:04:14 2014),<br>=A0=A0=A0 last beacon sent 2 =
secs ago (at Sun Jan 12 23:04:14 2014), last vote was no<br>
=A0=A0=A0 dbcurrent=3D0, up=3D1 beaconSince=3D1<br><br>Server (10.33.10.43)=
: (db 0.0)<br>=A0=A0=A0 last vote never rcvd<br>=A0=A0=A0 last beacon never=
sent<br>=A0=A0=A0 dbcurrent=3D0, up=3D0 beaconSince=3D0</font><br></div><d=
iv><br></div>So it thinks it is not the sync site, but 0.0.0.0 is. In reali=
ty, server A is the sync site.<br>
<br>I have run tcpdump to make absolutely certain there are packets running=
between the two hosts on all AFS ports, and there are<font face=3D"courier=
new,monospace">.</font><br><div><div><div><div><div><div><div><div><div>
<div><br></div><div>KeyFiles are fine, and checked with md5sum.<br><br></di=
v><div>Set debug to 25 on the affected vlserver (10.38.10.7, Server C) and =
got this:<br><span style=3D"font-family:courier new,monospace"><br>Sun Jan =
12 23:19:24 2014 Using 10.38.10.7 as my primary address<br>
Sun Jan 12 23:19:40 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:19:40 201=
4 Starting AFS vlserver 4 (/usr/lib/openafs/vlserver -rxbind -d 25)<br>Sun =
Jan 12 23:19:40 2014 no vote from 10.36.10.7<br>@(#) OpenAFS 1.6.1-1+ubuntu=
0.2-debian built=A0 2013-07-24<br>
12 23:19:40 2014 Received beacon type 0 from host 10.38.10.7<br>Sun Jan 12 =
23:19:44 2014 recovery running in state 0<br>Sun Jan 12 23:19:55 2014 beaco=
n: amSyncSite is 0<br>Sun Jan 12 23:19:55 2014 no vote from 10.36.10.7<br>
Sun Jan 12 23:19:55 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:19:55 201=
4 Received beacon type 0 from host 10.38.10.7<br>Sun Jan 12 23:20:10 2014 b=
eacon: amSyncSite is 0<br>Sun Jan 12 23:20:10 2014 no vote from 10.36.10.7<=
br>
Sun Jan 12 23:20:10 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:10 201=
4 Received beacon type 0 from host 10.38.10.7<br>Sun Jan 12 23:20:25 2014 b=
eacon: amSyncSite is 0<br>Sun Jan 12 23:20:25 2014 no vote from 10.36.10.7<=
br>
Sun Jan 12 23:20:25 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:25 201=
4 Received beacon type 0 from host 10.38.10.7<br>Sun Jan 12 23:20:40 2014 u=
bik:server 10.33.10.43 still down<br>Sun Jan 12 23:20:40 2014 beacon: amSyn=
cSite is 0<br>
Sun Jan 12 23:20:40 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:40 201=
4 no vote from 10.36.10.7<br>Sun Jan 12 23:20:40 2014 beacon: amSyncSite is=
0<br>Sun Jan 12 23:20:40 2014 Received beacon type 0 from host 10.38.10.7<=
br>
Sun Jan 12 23:20:40 2014 Ubik: vote 'yes' for 10.38.10.7 (NOT in qu=
orum)<br>Sun Jan 12 23:20:44 2014 recovery running in state 0<br>Sun Jan 12=
23:20:44 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:48 2014 recovery=
running in state 0<br>
Sun Jan 12 23:20:48 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:52 201=
4 recovery running in state 0<br>Sun Jan 12 23:20:52 2014 beacon: amSyncSit=
e is 0<br>Sun Jan 12 23:20:55 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23=
:20:55 2014 no vote from 10.36.10.7<br>
Sun Jan 12 23:20:55 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:20:55 201=
4 Received beacon type 0 from host 10.38.10.7<br>Sun Jan 12 23:20:56 2014 r=
ecovery running in state 0<br>Sun Jan 12 23:20:56 2014 beacon: amSyncSite i=
s 0<br>
Sun Jan 12 23:21:00 2014 recovery running in state 0<br>Sun Jan 12 23:21:00=
2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:21:04 2014 recovery running =
in state 0<br>Sun Jan 12 23:21:04 2014 beacon: amSyncSite is 0<br>Sun Jan 1=
2 23:21:07 2014 allbetter checking<br>
Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014=
allbetter checking<br>Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>S=
un Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 allb=
etter: returning 1<br>
Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 all=
better: returning 1<br>Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun J=
an 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014 GetV=
olumeByID 536870913 (2) 10.38.10.83 noauth<br>
Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 all=
better: returning 1<br>Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun J=
an 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014 allb=
etter checking<br>
Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014=
allbetter checking<br>Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>S=
un Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 allb=
etter: returning 1<br>
Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 all=
better: returning 1<br>Sun Jan 12 23:21:07 2014 GetVolumeByID 536870913 (2)=
10.38.10.83 noauth<br>Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun J=
an 12 23:21:07 2014 allbetter: returning 1<br>
Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun Jan 12 23:21:07 2014 all=
better: returning 1<br>Sun Jan 12 23:21:07 2014 allbetter checking<br>Sun J=
an 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014 allb=
etter checking<br>
Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>Sun Jan 12 23:21:07 2014=
allbetter checking<br>Sun Jan 12 23:21:07 2014 allbetter: returning 1<br>S=
un Jan 12 23:21:08 2014 recovery running in state 0<br>Sun Jan 12 23:21:08 =
2014 beacon: amSyncSite is 0<br>
Sun Jan 12 23:21:10 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:21:10 201=
4 no vote from 10.36.10.7<br>Sun Jan 12 23:21:10 2014 beacon: amSyncSite is=
0<br>Sun Jan 12 23:21:10 2014 Received beacon type 0 from host 10.38.10.7<=
br>
Sun Jan 12 23:21:12 2014 recovery running in state 0<br>Sun Jan 12 23:21:25=
2014 beacon: amSyncSite is 0<br>Sun Jan 12 23:21:25 2014 no vote from 10.3=
6.10.7<br>Sun Jan 12 23:21:25 2014 beacon: amSyncSite is 0<br>Sun Jan 12 23=
:21:25 2014 Received beacon type 0 from host 10.38.10.7<br>
Sun Jan 12 23:21:29 2014 allbetter checking<br>Sun Jan 12 23:21:29 2014 all=
better: returning 1<br>Sun Jan 12 23:21:29 2014 GetVolumeByName <snipped=
><br>Sun Jan 12 23:21:29 2014 allbetter checking<br>Sun Jan 12 23:21:29 =
2014 allbetter: returning 1<br>
Sun Jan 12 23:21:29 2014 allbetter checking<br>Sun Jan 12 23:21:29 2014 all=
better: returning 1<br>Sun Jan 12 23:21:29 2014 allbetter checking<br>Sun J=
an 12 23:21:29 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014 allb=
etter checking<br>
Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014=
GetVolumeByName <i><snipped></i><br>Sun Jan 12 23:21:30 2014 allbett=
er checking<br>Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 1=
2 23:21:30 2014 allbetter checking<br>
Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014=
allbetter checking<br>Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>S=
un Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 allb=
etter: returning 1<br>
Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 all=
better: returning 1<br>Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun J=
an 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014 GetV=
olumeByName <i><snipped></i><br>
Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 all=
better: returning 1<br>Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun J=
an 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014 allb=
etter checking<br>
Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:30 2014=
allbetter checking<br>Sun Jan 12 23:21:30 2014 allbetter: returning 1<br>S=
un Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 allb=
etter: returning 1<br>
Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 all=
better: returning 1<br>Sun Jan 12 23:21:30 2014 GetVolumeByName <i><snip=
ped></i><br>Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23=
:21:30 2014 allbetter: returning 1<br>
Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun Jan 12 23:21:30 2014 all=
better: returning 1<br>Sun Jan 12 23:21:30 2014 allbetter checking<br>Sun J=
an 12 23:21:30 2014 allbetter: returning 1<br>Sun Jan 12 23:21:40 2014 beac=
on: amSyncSite is 0<br>
Sun Jan 12 23:21:40 2014 no vote from 10.36.10.7<br>Sun Jan 12 23:21:40 201=
4 beacon: amSyncSite is 0<br>Sun Jan 12 23:21:40 2014 Received beacon type =
0 from host 10.38.10.7<br>Sun Jan 12 23:21:55 2014 beacon: amSyncSite is 0<=
br>
Sun Jan 12 23:21:55 2014 no vote from 10.36.10.7<br>Sun Jan 12 23:21:55 201=
4 beacon: amSyncSite is 0<br>Sun Jan 12 23:21:55 2014 Received beacon type =
0 from host 10.38.10.7</span><br></div><div><br></div><div>Any help is appr=
eciated! I have already upgraded and rebooted all three servers, lowest to =
highest, to no avail. I also attempted moving the data files out of the way=
and starting the affected vlserver, but the same symptoms remain.<br>
<br></div><div>-- <br><span style=3D"border-collapse:collapse;color:rgb(102=
,102,102);font-family:verdana,sans-serif;font-size:x-small">Timothy Balcer =
/ IT Services<br>Telmate / San Francisco, CA<br>Direct / </span><span style=
=3D"border-collapse:collapse;font-family:verdana,sans-serif;font-size:x-sma=
ll"><font color=3D"#1155cc">(415) 300-4313</font><br>
<font color=3D"#666666">Customer Service /=A0</font><a value=3D"+1800205551=
0" style=3D"color:rgb(17,85,204)">(800) 205-5510</a></span>
</div></div></div></div></div></div></div></div></div></div></div>
--001a11348964d310bf04efceafe9--