[OpenAFS-devel] afs_osi_Sleep and afs_osi_Wakeup on Linux
Srikanth Vishwanathan
vsrikanth@in.ibm.com
Wed, 5 Jun 2002 10:47:35 -0400
> if this were the deadlock it would seem to imply that we were
> waiting for afs to return memory so that vmalloc would be
> able to finish. since afs isnt very good about returning
> memory i cant see this as being the case. do anyone know
> why GLOCK is dropped? what was the deadlock condition?
Well, we have seen a deadlock problem in this area in the past.
But that was in osi_linux_alloc() and the semaphore was
afs_linux_alloc_sem. The code used to call linux_alloc with this
semaphore held. The deadlock was like this:
One thread was stuck in vmalloc (disk sleep) trying to allocate
memory. Another thread was trying to free memory but it could
not because afs_linux_alloc_sem was taken.
I guess we drop GLOCK because there is a possibility that alloc
might sleep in vmalloc under heavy load and we don't want to
lock up the system for so long, which is probably a good idea.
> another thing about the allocator. is it wise to choose
> such a large max size when choosing to use kmalloc? for
I guess the max size was chosen simply because that's the
maximum memory that kmalloc can allocate.
> more efficient. however, if your region is greater than
> a page size (and you dont really need it to be contiguous)
> you should probably be using vmalloc. i know afs allocates
Someone needs to try this out. Like you said, AFS only allocates
and never frees memory. Is is possible that vmallocing too
much memory could result in a problem ?
chas williams
<chas@cmf.nrl.nav To: Srikanth Vishwanathan/India/IBM@IBMIN
y.mil> cc: "openafs-devel" <openafs-devel@openafs.org>
Subject: Re: [OpenAFS-devel] afs_osi_Sleep and afs_osi_Wakeup on Linux
06/05/2002 09:35
AM
Please respond to
chas williams
In message <OFEDF28ED3.7100E0A7-ON85256BCE.0051C9F7@in.ibm.com>,"Srikanth
Vishwanathan" writes:
>I don't know is this is possible. But we don't drop GLOCK on other
>OSes. I guess we need to drop GLOCK because we call vmalloc() and
>schedule() in a loop in linux_alloc().
if this were the deadlock it would seem to imply that we were
waiting for afs to return memory so that vmalloc would be
able to finish. since afs isnt very good about returning
memory i cant see this as being the case. do anyone know
why GLOCK is dropped? what was the deadlock condition?
another thing about the allocator. is it wise to choose
such a large max size when choosing to use kmalloc? for
anything less than a pagesize, kmalloc is going to be
more efficient. however, if your region is greater than
a page size (and you dont really need it to be contiguous)
you should probably be using vmalloc. i know afs allocates
some 20000 chunks. as an example look at this machine
running stock openafs:
size-131072(DMA) 0 0 131072 0 0 32
size-131072 2 2 131072 2 2 32
size-65536(DMA) 0 0 65536 0 0 16
size-65536 2 2 65536 2 2 16
size-32768(DMA) 0 0 32768 0 0 8
size-32768 4 4 32768 4 4 8
size-16384(DMA) 0 0 16384 0 0 4
size-16384 1 1 16384 1 1 4
size-8192(DMA) 0 0 8192 0 0 2
size-8192 6 6 8192 6 6 2
changing the kmalloc max size to 4096 yeilds what i would
consider better results:
size-131072(DMA) 0 0 131072 0 0 32
size-131072 0 0 131072 0 0 32
size-65536(DMA) 0 0 65536 0 0 16
size-65536 0 0 65536 0 0 16
size-32768(DMA) 0 0 32768 0 0 8
size-32768 0 0 32768 0 0 8
size-16384(DMA) 0 0 16384 0 0 4
size-16384 0 0 16384 0 0 4
size-8192(DMA) 0 0 8192 0 0 2
size-8192 2 3 8192 2 3 2
note how much the memory footprint has improved.