[OpenAFS-devel] better vcache/dcache locking

[Nickolai Zeldovich]

I've been trying to add better (OS-initiated) prefetch support
to AFS, in particular on Solaris, and so far it seems to hurt
more than it helps.  As far as I can tell, the problem is that
the vcache locks double as dcache locks for dcache entries under
that vcache.  This only allows one data fetch at a time for a
given file.  The prefetch request causes the daemons to lock the
vcache and prefetch pages ahead, while preventing the process from
getting pages it needs now.

A possible solution seems to be adding rwlock's to dcache entries
themselves.  They would protect the struct dcache as well as the
underlying cache file contents.  It looks like there's already an
afs_rwlock_t in struct dcache, although it's only used in a couple
of places, and even there it's #ifdef'ed out.

The technical report Jim Rees referenced here a few weeks ago[1]
also finds the same locking problem and points out they were able
to achieve 26-40% speedup by avoiding it, presumably in the case
of sequential reads.  (I'm not entirely certain what the difference
between those two numbers in the report is.)

Since this is likely to be a fairly involved modification, I'd
appreciate any thoughts or comments people might have about making
such a change..

[1]  http://www.citi.umich.edu/techreports/reports/citi-tr-01-3.ps.gz

-- kolya