[OpenAFS] Performance issue with "many" volumes in a single /vicep?

[Derrick Brashear]

On Thu, Mar 18, 2010 at 6:43 AM, Jeffrey Altman
<jaltman@secure-endpoints.com> wrote:
> On 3/17/2010 7:41 PM, Derrick Brashear wrote:
>> On Wed, Mar 17, 2010 at 12:50 PM, Steve Simmons <scs@umich.edu> wrote:
>>> We've been seeing issues for a while that seem to relate to the number =
of volumes in a single vice partition. The numbers and data are inexact bec=
ause there are so many damned possible parameters that affect performance, =
but it appears that somewhere between 10,000 and 14,000 volumes performance=
 falls off significantly. That 40% difference in volume count results in 2x=
 to 3x falloffs for performance in issues that affect the /vicep as a whole=
 - backupsys, nightly dumps, vos listvol, etc.
>>>
>>> My initial inclination is to say it's a linux issue with directory sear=
ches, but before pursuing this much further I'd be interested in hearing fr=
om anyone who's running 14,000 or move volumes in a single vicep. No, I'm n=
ot counting .backup volumes in there, so 14,000 volumes means 28,000 entrie=
s in the directory.
>>
>> Another possibility: there's a hash table which is taking the bulk of
>> that that you then search linearly.
>
> In the 1.4 series, the volume hash table size is just 128 which
> would produce (assuming even distributions) average hash chains of
> 160 to 220 volumes per bucket given the number of volumes you
> describe. =A0This is quite long.
>
> In the 1.5 series, the volume hash table size defaults to 256
> which would be an average hash bucket chain length of 80 to 108.
>
> I would say that for the number of volumes that you are using
> that you would want the hash table to be no smaller than 4096
> which would bring the average hash chain length below 7 per bucket.
>
> In the 1.5 series the size of the hash table can be configured
> at run-time with the -vhashsize value. =A0In 1.4 you can modify
> the definition of VOLUME_HASH_TABLE_SIZE in src/vol/volume.c
> and rebuild.

If that's the problem. Of course, given that it's easy to check it's
worth doing so.



--=20
Derrick