[OpenAFS-devel] Re: Testing (and a call for volunteers)

[Andrew Deason]

On Sun, 16 Sep 2012 10:11:47 +0100
Simon Wilkinson <sxw@your-file-system.com> wrote:

> Of course, none of this will cover the cases that Jeffrey and David
> have been talking about. Many of the bugs that we encounter in
> production with AFS are incredibly complex - they rely on timing or
> load issues that just can't be simulated in a single machine
> environment. Other tests require particular versions of clients and
> servers, or on servers being modified to exhibit particular buggy
> behaviour.

I just want to mention... if anyone knows of ways that anyone else deals
with this, or has ideas on this, I'd like to know what they are. For
pretty much any bug I've ever figured out, I know how to reproduce it
reliably, but often one of the steps is "insert a sleep at line X" (or
more generally, "code modification"). So even though I know exactly how
to do it, I don't have any automation for it. Even if we had all of the
great modularization and stuff with clearly-defined APIs between package
boundaries and unit tests etc etc, I would imagine you still get bugs
like that. 

I've wondered about regression tests or something dealing with bugs like
that... I initially wrote that off as impossible, but I'm not really
sure. At least in theory you _could_ have something like:

    writelock(&foo->lock);
    TESTPOINT_ID1234();
    dosomething(&foo);
    writeunlock(&foo->lock);

And you need to rebuild stuff for the "testing version", where that
TESTPOINT symbol would be #defined to sleep, or wait for some external
signaling mechanism or... something.

But even that seems difficult to maintain, and makes it easy to have
bugs in the tests. Not to mention that it would probably make the code
harder to read, and it requires building stuff multiple times (this is
probably not feasible with a runtime test, at least for things in a hot
codepath). There's some stuff in 'vos' kinda like this that I assume
nobody has used in the past decade, and even that involved human
interaction.

Anyway, I was just wondering and thinking aloud, if anyone else had some
thoughts in the area.

-- 
Andrew Deason
adeason@sinenomine.net