[OpenAFS] buildbot and packages

Jeffrey Altman jaltman@your-file-system.com
Fri, 14 Sep 2012 17:22:36 -0400

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 9/14/2012 12:12 PM, David Boyes wrote:
>>> All this talk about 'reliable code for our users' is total BS until
>>> 'make check' actually does some realisitic functionality tests.
>>> If you can't write an automated test for a feature, they I would
>>> request we consider disabling that feature.
> I'm not sure this is a realistic goal in a single machine environment. =
For a realistic testing environment, you need at least 10 system images (=
and the ability to create a lot more would be very desirable for some of =
the subtler bugs), and the ability to control time and replay multiple st=
reams of events in a repeatable way. It involves a separate Kerberos infr=
astructure, and a lot of other moving parts, plus a lot of scripting to b=
uild the environment , run the test, and then reset the environment for t=
he next run. You also need different types of systems, different OS level=
s, etc which complicate the test even further.
> =20
> It's not impossible, but I can say that it cost a fair amount (in the m=
id-to-high 5 digit range) to build that environment here.=20


In this case I think you are low-balling the estimate.  To do it right
it isn't sufficient to test one build against itself.  You need to test
new clients against a range of old servers and vice versa in a
constrained environment.  It is necessary to be able to identify when a
change has an adverse performance impact as well as accuracy.  There is
a need to be able to introduce intentional errors at various points in
the protocol.  Just the hardware costs are mid 5 digits and the software
development is significantly more than that.

Jeffrey Altman

Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

Version: GnuPG v1.4.9 (MingW32)