[OpenAFS] buildbot and packages

Simon Wilkinson sxw@your-file-system.com
Mon, 17 Sep 2012 20:20:09 +0100

On 17 Sep 2012, at 17:25, David Boyes wrote:
>>  'make check' on a single machine will never give you useful testing =
results other than to find packaging or "smoke test" errors, which =
aren't all that helpful overall.=20

I agree with you with regards to crowd sourced testing, but I just =
wanted to call out this statement, as I think make check can actually be =
hugely helpful.

In the past, we used to have the problem that changes would get =
committed to master that would cause the build to fail. So, if you were =
working on new stuff, and had just pulled in a recent upstream to work =
on, you often ended up having to spend lots of time just getting the =
tree to work again on your chosen platform. Buildbot has pretty much =
eradicated this problem - I can be sure that git fetch will give me a =
tree that works.

So now, the new problem is that changes in master may well be unstable. =
Instead of working on building your shiny new feature, developers end up =
having to track down why a particular binary segmentation faults and so =
on. Applying 'make check' as a commit requirement, and improving its =
coverage will hugely help with this problem. I see it more as a =
developer tool than a general QA solution.

What its not going to do is replace large scale test harnesses and =
detailed test plans. We still need the ability to generate large numbers =
of clients hammering a server to expose particular problems. Over the =
1.6 cycle, the majority of this work was done for OpenAFS by volunteers =
at European HPC sites, who used down time on their clusters to generate =
particular types of load against our fileservers. This led to a number =
of long standing bugs in both the demand attach fileserver, and the RX =
stack itself, being identified and fixed.