[OpenAFS] Problems on AFS Unix clients after AFS fileserver moves

Derek Atkins warlord@MIT.EDU
Tue, 9 Aug 2005 13:50:51 -0400


Hmm... Did you "vos addsite" without "release" to the priddy server?

-derek

Quoting Rich Sudlow <rich@nd.edu>:

> Rich Sudlow wrote:
>> Derrick J Brashear wrote:
>>
>>> On Tue, 9 Aug 2005, Rich Sudlow wrote:
>>>
>>>>>> [root@xeon028 root]# ls -l /afs/nd.edu/common/custom
>>>>>> ls: /afs/nd.edu/common/custom/hpcc: Connection timed out
>
> Now I'm even getting:
>
> xeon028{root}14: cd /afs/.nd.edu
> /afs/.nd.edu: Connection timed out.
>
> I should note that over the last couple years we've also been
> having problems with our root.cell releasing consitently without 
> connection timeout problems.
>
> But everything looks ok now
>
>
> [root@xeon028 custom]# vos ex root.cell -v
> Fetching VLDB entry for 536889194 .. done
> Getting volume listing from the server bubba.helios.nd.edu .. done
> root.cell                         536889194 RW      37345 K  On-line
>     bubba.helios.nd.edu /vicepa
>     RWrite  536889194 ROnly  536889195 Backup  536889196
>     MaxQuota      50000 K
>     Creation    Tue Jul  2 08:46:36 1991
>     Last Update Tue Aug  9 08:43:05 2005
>     438949 accesses in the past day (i.e., vnode references)
>
>     RWrite: 536889194     ROnly: 536889195     Backup: 536889196
>     number of sites -> 8
>        server bubba.helios.nd.edu partition /vicepa RW Site
>        server priddy.helios.nd.edu partition /vicepa RO Site
>        server bubba.helios.nd.edu partition /vicepa RO Site
>        server nevada.helios.nd.edu partition /vicepb RO Site
>        server emdall.helios.nd.edu partition /vicepb RO Site
>        server widmark.helios.nd.edu partition /vicepb RO Site
>        server yaya.helios.nd.edu partition /vicepa RO Site
>        server planet10.helios.nd.edu partition /vicepa RO Site
> [root@xeon028 custom]# vos ex root.cell.readonly
> Could not fetch the information about volume 536889195 from the server
> : No such device
> Volume does not exist on server priddy.helios.nd.edu as indicated by the VLDB
>
> root.cell.readonly                536889195 RO      37345 K  On-line
>     bubba.helios.nd.edu /vicepa
>     RWrite  536889194 ROnly  536889195 Backup  536889196
>     MaxQuota      50000 K
>     Creation    Tue Aug  9 08:59:11 2005
>     Last Update Tue Aug  9 08:59:11 2005
>     71583 accesses in the past day (i.e., vnode references)
>
> root.cell.readonly                536889195 RO      37345 K  On-line
>     nevada.helios.nd.edu /vicepb
>     RWrite  536889194 ROnly          0 Backup          0
>     MaxQuota      50000 K
>     Creation    Tue Aug  9 08:59:11 2005
>     Last Update Tue Aug  9 08:59:11 2005
>     5236 accesses in the past day (i.e., vnode references)
>
> root.cell.readonly                536889195 RO      37345 K  On-line
>     emdall.helios.nd.edu /vicepb
>     RWrite  536889194 ROnly          0 Backup          0
>     MaxQuota      50000 K
>     Creation    Tue Aug  9 08:59:11 2005
>     Last Update Tue Aug  9 08:59:11 2005
>     621076 accesses in the past day (i.e., vnode references)
>
> root.cell.readonly                536889195 RO      37345 K  On-line
>     widmark.helios.nd.edu /vicepb
>     RWrite  536889194 ROnly          0 Backup          0
>     MaxQuota      50000 K
>     Creation    Tue Aug  9 08:59:11 2005
>     Last Update Tue Aug  9 08:59:11 2005
>     47125 accesses in the past day (i.e., vnode references)
>
> root.cell.readonly                536889195 RO      37345 K  On-line
>     yaya.helios.nd.edu /vicepa
>     RWrite  536889194 ROnly          0 Backup          0
>     MaxQuota      50000 K
>     Creation    Tue Aug  9 08:59:11 2005
>     Last Update Tue Aug  9 08:59:11 2005
>     171534 accesses in the past day (i.e., vnode references)
>
> root.cell.readonly                536889195 RO      37345 K  On-line
>     planet10.helios.nd.edu /vicepa
>     RWrite  536889194 ROnly          0 Backup          0
>     MaxQuota      50000 K
>     Creation    Tue Aug  9 08:59:11 2005
>     Last Update Tue Aug  9 08:59:11 2005
>     53916 accesses in the past day (i.e., vnode references)
>
>     RWrite: 536889194     ROnly: 536889195     Backup: 536889196
>     number of sites -> 8
>        server bubba.helios.nd.edu partition /vicepa RW Site
>        server priddy.helios.nd.edu partition /vicepa RO Site
>        server bubba.helios.nd.edu partition /vicepa RO Site
>        server nevada.helios.nd.edu partition /vicepb RO Site
>        server emdall.helios.nd.edu partition /vicepb RO Site
>        server widmark.helios.nd.edu partition /vicepb RO Site
>        server yaya.helios.nd.edu partition /vicepa RO Site
>        server planet10.helios.nd.edu partition /vicepa RO Site
> [root@xeon028 custom]#
>
> Rich
>
>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> i assume if you fs flushmount hpcc it gets better?
>>>>
>>>>
>>>>
>>>> I wish ;-)
>>>
>>>
>>>
>>> ok, so what happens network-wise after you say fs flushv in common, 
>>> then ls again. (tcpdump -vv -s 1500 port 7000 and watch the network 
>>> traffic)
>>
>>
>> fs flushvolume /afs/nd.edu /afs/nd.edu/common 
>> /afs/nd.edu/common/custom /afs/nd.edu/common/custom/hpcc
>>
>> fs flushmount /afs/nd.edu /afs/nd.edu/common 
>> /afs/nd.edu/common/custom /afs/nd.edu/common/custom/hpcc
>>
>> while using  tcpdump -vv -s 1500 -i eth1 port 7000
>>
>> 12:27:09.030372 xeon028.afs3-callback > 
>> emdall.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx data cid 
>> ceccc7b4 call# 42 seq 1 ser 173 <client-init>,<last-pckt> fs call 
>> fetch-status fid 536870919/2/22297 (44) (DF) (ttl 64, id 57995, len 
>> 72)
>> 12:27:09.031116 emdall.helios.nd.edu.afs3-fileserver > 
>> xeon028.afs3-callback: [udp sum ok]  rx data cid ceccc7b4 call# 42 
>> seq 1 ser 236 <last-pckt> fs reply fetch-status (148) (DF) (ttl 252, 
>> id 21104, len 176)
>> 12:27:09.031615 xeon028.afs3-callback > 
>> emdall.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx ack cid 
>> ceccc7b4 call# 42 seq 0 ser 174 <client-init>,<slow-start> fir 2 0n 
>> (65) (DF) (ttl 64, id 57996, len 93)
>> 12:27:09.031788 xeon028.afs3-callback > 
>> emdall.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx data cid 
>> ceccc7b4 call# 43 seq 1 ser 175 <client-init>,<last-pckt> fs call 
>> fetch-status fid 536889195/4/3 (44) (DF) (ttl 64, id 57997, len 72)
>> 12:27:09.032313 emdall.helios.nd.edu.afs3-fileserver > 
>> xeon028.afs3-callback: [udp sum ok]  rx data cid ceccc7b4 call# 43 
>> seq 1 ser 237 <last-pckt> fs reply fetch-status (148) (DF) (ttl 252, 
>> id 21105, len 176)
>> 12:27:09.032510 xeon028.afs3-callback > 
>> emdall.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx ack cid 
>> ceccc7b4 call# 43 seq 0 ser 176 <client-init>,<slow-start> fir 2 0n 
>> (65) (DF) (ttl 64, id 57998, len 93)
>> 12:27:09.032749 xeon028.afs3-callback > 
>> priddy.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx data cid 
>> ceccc80c call# 30 seq 1 ser 62 <client-init>,<last-pckt> fs call 
>> fetch-status fid 536872982/34/31616 (44) (DF) (ttl 64, id 57999, len 
>> 72)
>> 12:27:09.033861 priddy.helios.nd.edu.afs3-fileserver > 
>> xeon028.afs3-callback: [udp sum ok]  rx data cid ceccc80c call# 30 
>> seq 1 ser 35 <last-pckt> fs reply fetch-status (148) (DF) (ttl 252, 
>> id 58352, len 176)
>> 12:27:09.034041 xeon028.afs3-callback > 
>> priddy.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx ack cid 
>> ceccc80c call# 30 seq 0 ser 63 <client-init>,<slow-start> fir 2 0n 
>> (65) (DF) (ttl 64, id 58000, len 93)
>> 12:27:12.370168 xeon028.afs3-callback > 
>> emdall.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx data cid 
>> ceccc7b4 call# 44 seq 1 ser 177 <client-init>,<last-pckt> fs call 
>> fetch-status fid 536870919/2/22297 (44) (DF) (ttl 64, id 58001, len 
>> 72)
>> 12:27:12.370715 emdall.helios.nd.edu.afs3-fileserver > 
>> xeon028.afs3-callback: [udp sum ok]  rx data cid ceccc7b4 call# 44 
>> seq 1 ser 238 <last-pckt> fs reply fetch-status (148) (DF) (ttl 252, 
>> id 21106, len 176)
>> 12:27:12.370877 xeon028.afs3-callback > 
>> emdall.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx ack cid 
>> ceccc7b4 call# 44 seq 0 ser 178 <client-init>,<slow-start> fir 2 0n 
>> (65) (DF) (ttl 64, id 58002, len 93)
>> 12:27:12.371005 xeon028.afs3-callback > 
>> emdall.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx data cid 
>> ceccc7b4 call# 45 seq 1 ser 179 <client-init>,<last-pckt> fs call 
>> fetch-status fid 536889195/4/3 (44) (DF) (ttl 64, id 58003, len 72)
>> 12:27:12.371649 emdall.helios.nd.edu.afs3-fileserver > 
>> xeon028.afs3-callback: [udp sum ok]  rx data cid ceccc7b4 call# 45 
>> seq 1 ser 239 <last-pckt> fs reply fetch-status (148) (DF) (ttl 252, 
>> id 21107, len 176)
>> 12:27:12.371746 xeon028.afs3-callback > 
>> emdall.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx ack cid 
>> ceccc7b4 call# 45 seq 0 ser 180 <client-init>,<slow-start> fir 2 0n 
>> (65) (DF) (ttl 64, id 58004, len 93)
>> 12:27:12.371873 xeon028.afs3-callback > 
>> priddy.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx data cid 
>> ceccc80c call# 31 seq 1 ser 64 <client-init>,<last-pckt> fs call 
>> fetch-status fid 536872982/34/31616 (44) (DF) (ttl 64, id 58005, len 
>> 72)
>> 12:27:12.372945 priddy.helios.nd.edu.afs3-fileserver > 
>> xeon028.afs3-callback: [udp sum ok]  rx data cid ceccc80c call# 31 
>> seq 1 ser 36 <last-pckt> fs reply fetch-status (148) (DF) (ttl 252, 
>> id 58353, len 176)
>> 12:27:12.373028 xeon028.afs3-callback > 
>> priddy.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx ack cid 
>> ceccc80c call# 31 seq 0 ser 65 <client-init>,<slow-start> fir 2 0n 
>> (65) (DF) (ttl 64, id 58006, len 93)
>> 12:27:12.373152 xeon028.afs3-callback > 
>> bert.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx data cid 
>> ceccc7b0 call# 89 seq 1 ser 219 <client-init>,<last-pckt> fs call 
>> fetch-status fid 1918991272/966/11826 (44) (DF) (ttl 64, id 58007, 
>> len 72)
>> 12:27:12.374128 bert.helios.nd.edu.afs3-fileserver > 
>> xeon028.afs3-callback: [udp sum ok]  rx ack cid ceccc7b0 call# 89 
>> seq 0 ser 131 <req-ack>,<slow-start> fir 1 0p* (66) (DF) (ttl 252, 
>> id 3509, len 94)
>> 12:27:12.374180 xeon028.afs3-callback > 
>> bert.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx ack cid 
>> ceccc7b0 call# 89 seq 0 ser 220 <client-init>,<slow-start> fir 1 
>> 131r (65) (DF) (ttl 64, id 58008, len 93)
>> 12:27:12.374732 bert.helios.nd.edu.afs3-fileserver > 
>> xeon028.afs3-callback: [udp sum ok]  rx data cid ceccc7b0 call# 89 
>> seq 1 ser 132 <last-pckt> fs reply fetch-status (148) (DF) (ttl 252, 
>> id 3510, len 176)
>> 12:27:12.374846 xeon028.afs3-callback > 
>> bert.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx ack cid 
>> ceccc7b0 call# 89 seq 0 ser 221 <client-init>,<slow-start> fir 2 0n 
>> (65) (DF) (ttl 64, id 58009, len 93)
>> 12:27:12.374927 xeon028.afs3-callback > 
>> bert.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx data cid 
>> ceccc7b0 call# 90 seq 1 ser 222 <client-init>,<last-pckt> fs call 
>> fetch-data fid 1918991272/966/11826 offset 0 length 17 (52) (DF) 
>> (ttl 64, id 58010, len 80)
>> 12:27:12.375727 bert.helios.nd.edu.afs3-fileserver > 
>> xeon028.afs3-callback: [udp sum ok]  rx data cid ceccc7b0 call# 90 
>> seq 1 ser 133 <last-pckt> fs reply fetch-data (169) (DF) (ttl 252, 
>> id 3511, len 197)
>> 12:27:12.375830 xeon028.afs3-callback > 
>> bert.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx ack cid 
>> ceccc7b0 call# 90 seq 0 ser 223 <client-init>,<slow-start> fir 2 0n 
>> (65) (DF) (ttl 64, id 58011, len 93)
>>
>> 26 packets received by filter
>>
>> I also saw some attempts to contact reno when running tcpdump too.
>>
>> 12:22:00.717951 xeon028.afs3-callback > 
>> reno.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx data cid 
>> ceccc87c call# 1 seq 1 ser 1 <client-init>,<last-pckt> fs call 
>> get-time (32) (DF) (ttl 64, id 52214, len 60)
>> 12:22:01.507927 xeon028.afs3-callback > 
>> reno.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx data cid 
>> ceccc87c call# 1 seq 1 ser 2 <client-init>,<req-ack>,<last-pckt> fs 
>> call get-time (32) (DF) (ttl 64, id 52215, len 60)
>> 12:22:02.527991 xeon028.afs3-callback > 
>> reno.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx data cid 
>> ceccc87c call# 1 seq 1 ser 3 <client-init>,<req-ack>,<last-pckt> fs 
>> call get-time (32) (DF) (ttl 64, id 52216, len 60)
>> 12:22:04.058099 xeon028.afs3-callback > 
>> reno.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx data cid 
>> ceccc87c call# 1 seq 1 ser 4 <client-init>,<req-ack>,<last-pckt> fs 
>> call get-time (32) (DF) (ttl 64, id 52217, len 60)
>> 12:22:06.098225 xeon028.afs3-callback > 
>> reno.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx ack cid 
>> ceccc87c call# 1 seq 0 ser 5 <client-init>,<req-ack>,<slow-start> 
>> fir 1 0p (65) (DF) (ttl 64, id 52218, len 93)
>> 12:22:06.608256 xeon028.afs3-callback > 
>> reno.helios.nd.edu.afs3-fileserver: [udp sum ok]  rx data cid 
>> ceccc87c call# 1 seq 1 ser 6 <client-init>,<req-ack>,<last-pckt> fs 
>> call get-time (32) (DF) (ttl 64, id 52219, len 60)
>>
>>
>> Rich
>>
>>
>>>
>>>
>>> _______________________________________________
>>> OpenAFS-info mailing list
>>> OpenAFS-info@openafs.org
>>> https://lists.openafs.org/mailman/listinfo/openafs-info
>>
>>
>>
>
>
> -- 
> Rich Sudlow
> University of Notre Dame
> Office of Information Technologies
> 321 Information Technologies Center
> PO Box 539
> Notre Dame, IN 46556-0539
>
> (574) 631-7258 office phone
> (574) 631-9283 office fax
>
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
>



-- 
       Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
       Member, MIT Student Information Processing Board  (SIPB)
       URL: http://web.mit.edu/warlord/    PP-ASEL-IA     N1NWH
       warlord@MIT.EDU                        PGP key available