[OpenAFS] buildbot and packages

[Troy Benjegerdes]

Sometimes I think we get hung up on 'good testing' vs having *something*.

The last time I worked for someone else, it was writing test code for Cray's
supercomputer systems. You don't get much more complex than a machine
with 30,000 cores in which 'acceptable' performance is defined as 'pushing
the system to the point right before it collapses into an unusable heap',
and it's got to run a workload of hundreds of thousands of the world's most
complex and numerically sensitive computational codes.

And I'd hazard a guess that 3/4 of the system problems were with the filesystem
(Lustre most often). I've also heard a pretty good argument that the reason 
Cray went bankrupt a couple of times is they over-tested. If you did get a
machine back in the YMP days, it was very well tested, but the price showed 
it, and clusters ate their market.


Maybe we don't have money.. But how many users of AFS are there. I'm not talking
companies, I'm talking people.. specifically, bored college students. How many
people have used AFS at a major university, and might help us out doing manual
testing if we give them a framework?

To paraphrase the .. well.. chief cat herder .. of the most widely deployed
operating system ever (Linux), 
"With enough QA testers, all bugs are shallow"

On Fri, Sep 14, 2012 at 04:42:37PM -0500, David Boyes wrote:
> > In this case I think you are low-balling the estimate.  To do it right it isn't
> > sufficient to test one build against itself.  You need to test new clients
> > against a range of old servers and vice versa in a constrained environment.
> > It is necessary to be able to identify when a change has an adverse
> > performance impact as well as accuracy.  There is a need to be able to
> > introduce intentional errors at various points in the protocol.  Just the
> > hardware costs are mid 5 digits and the software development is
> > significantly more than that.
> 
>  I agree --  if you were starting from scratch, you're probably right. 
> 
> But, a) I wasn't starting from scratch, so the additional equipment for adding the AFS framework stuff was about what I quoted, and b) I was discussing our tooling and test setup, not the general case. 
> We reused existing tooling in a number of places, and layered the AFS component onto that. We do this kind of thing for other software, so we had a decent baseline to start from. 
> 
> Solid QA infrastructure -- especially for complex systems -- isn't simple or cheap; there we agree wholeheartedly. 
> 
> 
> 
> :??