[OpenAFS] buildbot and packages

[Jeffrey Altman]

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig6DDB93938A7729D300AA0A00
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 9/14/2012 12:12 PM, David Boyes wrote:
>>> All this talk about 'reliable code for our users' is total BS until
>>> 'make check' actually does some realisitic functionality tests.
>>> If you can't write an automated test for a feature, they I would
>>> request we consider disabling that feature.
>=20
> I'm not sure this is a realistic goal in a single machine environment. =
For a realistic testing environment, you need at least 10 system images (=
and the ability to create a lot more would be very desirable for some of =
the subtler bugs), and the ability to control time and replay multiple st=
reams of events in a repeatable way. It involves a separate Kerberos infr=
astructure, and a lot of other moving parts, plus a lot of scripting to b=
uild the environment , run the test, and then reset the environment for t=
he next run. You also need different types of systems, different OS level=
s, etc which complicate the test even further.
> =20
> It's not impossible, but I can say that it cost a fair amount (in the m=
id-to-high 5 digit range) to build that environment here.=20

David,

In this case I think you are low-balling the estimate.  To do it right
it isn't sufficient to test one build against itself.  You need to test
new clients against a range of old servers and vice versa in a
constrained environment.  It is necessary to be able to identify when a
change has an adverse performance impact as well as accuracy.  There is
a need to be able to introduce intentional errors at various points in
the protocol.  Just the hardware costs are mid 5 digits and the software
development is significantly more than that.

Jeffrey Altman



--------------enig6DDB93938A7729D300AA0A00
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)

iQEcBAEBAgAGBQJQU6AfAAoJENxm1CNJffh4WBcH/0VRDx/Vdxj4Q77gM8oDVZKS
SAjhFUnhhJnG/DIH0frmQr7vmlOMFJkNHhFASTqVcWdHc0mCVm70bSJfSvpau7Ij
kvbkV65V+zBD4pNfm4wLGXlJko+/m8rdJI455ZlGls33W0iBODEPo9mxxtdASWdd
mOWlaUo/4zFjzQtixvHbq7SL+FpaQAvMLH/R4Vii6nzCzEI5kAFWrzyGxYSoDSiY
MyGc7aS2QtAVuyNAgh+oNtcaC13yQda4nkysZWD5+WQtMRdSTHQmeVnApWKY7itm
8psRaiqRhYoDaEJdaQPKvmv0Yq/yZezd1DZl3ld59QglgM2fIB0W3sGE+jwyCGc=
=CFcK
-----END PGP SIGNATURE-----

--------------enig6DDB93938A7729D300AA0A00--