[OpenAFS] Unexpected no space left on device error
Jeffrey Altman
jaltman@auristor.com
Wed, 14 Nov 2018 05:23:24 +0100
--Apple-Mail-1E822D2F-F4DB-44A8-ABEC-2F7E3E773A09
Content-Type: text/plain;
charset=us-ascii
Content-Transfer-Encoding: quoted-printable
I'm placing a beer on the directory being full. For extra credit I will gue=
ss that the directory is full as a result of abandoned silly rename files. Y=
ou should try salvaging the volume with the rebuild directories option.
Jeffrey Altman
> On Nov 14, 2018, at 4:36 AM, Benjamin Kaduk <kaduk@mit.edu> wrote:
>=20
>> On Tue, Nov 13, 2018 at 08:46:28PM -0500, Theo Ouzhinski wrote:
>> Hi all,
>>=20
>> Sorry for my previous incorrectly formatted email.
>> Recently, I've seen an uptick in "no space left on device" errors for
>> some of the home directories I administer.
>>=20
>> For example,
>>=20
>> matsumoto <USERNAME> # touch a
>> touch: cannot touch 'a': No space left on device
>>=20
>> We are not even close to filling up the cache (located at
>> /var/cache/openafs) on this client machine.
>>=20
>> matsumoto ~ # fs getcacheparms
>> AFS using 10314 of the cache's available 10000000 1K byte blocks.
>> matsumoto ~ # df -h
>> Filesystem Size Used Avail Use% Mounted on
>> ....
>> /dev/mapper/vgwrkstn-root 456G 17G 417G 4% /
>> ....
>> AFS 2.0T 0 2.0T 0% /afs
>>=20
>>=20
>> Nor is this home directory or any other problematic home directory close
>> to their quota.
>>=20
>> matsumoto <USERNAME> # fs lq
>> Volume Name Quota Used %Used Partition
>> <VOLUME NAME> 4194304 194403 5% 37%
>>=20
>> According to previous posts on this list, many issues can be attributed
>> to high inode usage. However, this is not the case on our machines.
>>=20
>> Here is sample output from one of our OpenAFS servers, which is similar
>> to all of the four other ones.
>>=20
>> openafs1 ~ # df -i
>> Filesystem Inodes IUsed IFree IUse% Mounted on
>> udev 1903816 413 1903403 1% /dev
>> tmpfs 1911210 551 1910659 1% /run
>> /dev/vda1 1905008 154821 1750187 9% /
>> tmpfs 1911210 1 1911209 1% /dev/shm
>> tmpfs 1911210 5 1911205 1% /run/lock
>> tmpfs 1911210 17 1911193 1% /sys/fs/cgroup
>> /dev/vdb 19660800 3461203 16199597 18% /vicepa
>> /dev/vdc 19660800 1505958 18154842 8% /vicepb
>> tmpfs 1911210 4 1911206 1% /run/user/0
>> AFS 2147483647 0 2147483647 0% /afs
>>=20
>>=20
>> We are running the latest HWE kernel (4.15.0-38-generic) for Ubuntu
>> 16.04 (which is the OS for both server and client machines). We are
>> running on the clients, the following versions:
>>=20
>> openafs-client/xenial,now 1.8.2-0ppa2~ubuntu16.04.1 amd64 [installed]
>> openafs-krb5/xenial,now 1.8.2-0ppa2~ubuntu16.04.1 amd64 [installed]
>> openafs-modules-dkms/xenial,xenial,now 1.8.2-0ppa2~ubuntu16.04.1 all
>> [installed]
>>=20
>> and on the servers, the following versions:
>>=20
>> openafs-client/xenial,now 1.6.15-1ubuntu1 amd64 [installed]
>> openafs-dbserver/xenial,now 1.6.15-1ubuntu1 amd64 [installed]
>> openafs-fileserver/xenial,now 1.6.15-1ubuntu1 amd64 [installed]
>> openafs-krb5/xenial,now 1.6.15-1ubuntu1 amd64 [installed]
>> openafs-modules-dkms/xenial,xenial,now 1.6.15-1ubuntu1 all [installed]
>=20
> (Off-topic, but that looks to be missing some security fixes.)
>=20
>> What could be the problem? Is there something I missed?
>=20
> It's not really ringing a bell off the top of my head, no.
>=20
> That said, there's a number of potential ways to get ENOSPC, so it would b=
e
> good to get more data, like an strace of the failing touch, and maybe a
> packet capture (port 7000) during the touch, both from a clean cache and
> potentially a second attempt.
>=20
> -Ben
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
--Apple-Mail-1E822D2F-F4DB-44A8-ABEC-2F7E3E773A09
Content-Type: application/pkcs7-signature;
name=smime.p7s
Content-Disposition: attachment;
filename=smime.p7s
Content-Transfer-Encoding: base64
MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCBdYw
ggXSMIIEuqADAgECAhBAAWbTGehnfUuu91hYwM5DMA0GCSqGSIb3DQEBCwUAMDoxCzAJBgNVBAYT
AlVTMRIwEAYDVQQKEwlJZGVuVHJ1c3QxFzAVBgNVBAMTDlRydXN0SUQgQ0EgQTEyMB4XDTE4MTEw
MjA2MjYyMloXDTE5MTEwMjA2MjYyMlowcDEvMC0GCgmSJomT8ixkAQETH0EwMTQyN0UwMDAwMDE2
NkQzMTlFODFBMDAwMDdBN0IxGTAXBgNVBAMTEEplZmZyZXkgRSBBbHRtYW4xFTATBgNVBAoTDEF1
cmlTdG9yIEluYzELMAkGA1UEBhMCVVMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDq
EYwjLORE23Gc8m7YgKqbGzWn/fmVGtoZkBNwOEYlrFOu84PbEhV4sxQrChhPyXVW2jquV2rg2/5d
sVC8RO+RwlXuAkUvR9KhWJLu6GJXwUnZr83wtEzJ8nqpTHj6W+3velLwWx7qhADyrMnKN0bTYh+5
M9HWt2We4qYi6i1/ejgKtM0arWYxVx6Iwb4xZpilMDNqV15Dwuunnkq4vNEByIT81zDoClqylMxx
KJpvc3tqC66+BHHM5RxF+z36Pt8fb3Q54VrytxXFm+kVSclKGaWgjq5SqV4tR0FWv6OnMY8tAx1Y
rljfvgxW5npZgBbo+YVoYEfUrz77WIYQyzn7AgMBAAGjggKcMIICmDAOBgNVHQ8BAf8EBAMCBPAw
gYQGCCsGAQUFBwEBBHgwdjAwBggrBgEFBQcwAYYkaHR0cDovL2NvbW1lcmNpYWwub2NzcC5pZGVu
dHJ1c3QuY29tMEIGCCsGAQUFBzAChjZodHRwOi8vdmFsaWRhdGlvbi5pZGVudHJ1c3QuY29tL2Nl
cnRzL3RydXN0aWRjYWExMi5wN2MwHwYDVR0jBBgwFoAUpHPa72k1inXMoBl7CDL4a4nkQuwwCQYD
VR0TBAIwADCCASsGA1UdIASCASIwggEeMIIBGgYLYIZIAYb5LwAGAgEwggEJMEoGCCsGAQUFBwIB
Fj5odHRwczovL3NlY3VyZS5pZGVudHJ1c3QuY29tL2NlcnRpZmljYXRlcy9wb2xpY3kvdHMvaW5k
ZXguaHRtbDCBugYIKwYBBQUHAgIwga0agapUaGlzIFRydXN0SUQgQ2VydGlmaWNhdGUgaGFzIGJl
ZW4gaXNzdWVkIGluIGFjY29yZGFuY2Ugd2l0aCBJZGVuVHJ1c3QncyBUcnVzdElEIENlcnRpZmlj
YXRlIFBvbGljeSBmb3VuZCBhdCBodHRwczovL3NlY3VyZS5pZGVudHJ1c3QuY29tL2NlcnRpZmlj
YXRlcy9wb2xpY3kvdHMvaW5kZXguaHRtbDBFBgNVHR8EPjA8MDqgOKA2hjRodHRwOi8vdmFsaWRh
dGlvbi5pZGVudHJ1c3QuY29tL2NybC90cnVzdGlkY2FhMTIuY3JsMB8GA1UdEQQYMBaBFGphbHRt
YW5AYXVyaXN0b3IuY29tMB0GA1UdDgQWBBQevV8IqWfIUNkQqAugGhxR938z+jAdBgNVHSUEFjAU
BggrBgEFBQcDAgYIKwYBBQUHAwQwDQYJKoZIhvcNAQELBQADggEBAKsUkshF6tfL43itTIVy9vjY
qqPErG9n8kX5FlRYbtIVlWIYTxQpeqtDpUPur1jfBiNY+xT+9PayO2+XxXu9ZEykCz5T4+3q7s5t
5RLsHu1dxYcMnAgfUqb13mhZxY8PVPE4PTHSvZLjPZ6Nt7j0tXjddZJqjDhr7neNpmYgQWSe+oaI
xbUqQ34rVW/hDimv9Y2DnCXL0LopCfABQDK9HDzmsuXdbVH6LUpS6ncge9kQEh1QIGuwqEv2tHCW
eauWM6h3BOXj3dlfbJEawUYz2hvc3nSXpscFlCN5tGAyUAE8QbKnH1ha/zZVrJY1EglFhnDho34l
Wl35t7pE5NP4kscxggKmMIICogIBATBOMDoxCzAJBgNVBAYTAlVTMRIwEAYDVQQKEwlJZGVuVHJ1
c3QxFzAVBgNVBAMTDlRydXN0SUQgQ0EgQTEyAhBAAWbTGehnfUuu91hYwM5DMA0GCWCGSAFlAwQC
AQUAoIIBKTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xODExMTQw
NDIzMjRaMC8GCSqGSIb3DQEJBDEiBCBF1bSUHeRA1SUKZT5K3SM44ZgG7jixz5xCTDvSmPdqJTBd
BgkrBgEEAYI3EAQxUDBOMDoxCzAJBgNVBAYTAlVTMRIwEAYDVQQKEwlJZGVuVHJ1c3QxFzAVBgNV
BAMTDlRydXN0SUQgQ0EgQTEyAhBAAWbTGehnfUuu91hYwM5DMF8GCyqGSIb3DQEJEAILMVCgTjA6
MQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MRcwFQYDVQQDEw5UcnVzdElEIENBIEEx
MgIQQAFm0xnoZ31LrvdYWMDOQzANBgkqhkiG9w0BAQEFAASCAQCXW85WxGkt0cM0mFWjM7YqF9r1
okbu0NX4Lki35a3sPCiXfGfyhQEnoPahktL8qT2jnP/fQxov6cmgGCKwWVLvA/OYNaNKNhA+Q5tN
TSPPWX/BskXqSAVmmyQHk9ZTtBLrBrRsbpSXU5VZ/tvaLixJzarN8Q1+kyN1RFRrwGDiwhQ1EHom
hts7pWGmAiGsDqS2ua0EPKk8u/QZ/du9NCHdmp+ZZqr58DfxAp+pzbga6mwYFA2+rzaFpJwMiiKV
+4N8cM5TEllBKHMu+stmlo09ICRqTZMRqUgrmRNOR7C1Vykc1oDeFiihq9ckoD5hY5HBB79xPft3
0H1DSvlkH6fEAAAAAAAA
--Apple-Mail-1E822D2F-F4DB-44A8-ABEC-2F7E3E773A09--