[OpenAFS] When to publish security advisories?
Sat, 16 Apr 2011 21:48:31 -0500
> Sadly, the issue is not related to build system complexity.
That's what I said. It's a packaging issue (eg automated test and deploy).
The build system in place is not particularly amenable to making the
packaging process easier. The fact that people are having to do it by hand
is IMHO a design flaw in the build and packaging process.
> When a new release happens, it relies on a number of volunteers who
> build and test the source on their respective platforms. We generally
> try to ensure that we have good build and test coverage before making a
> release, which all takes time. For RedHat, for example, a complete set
> of builds for a new release can take nearly a week to complete. Whilst
> we could automate portions of this build process for all of the systems
> we have buildbots for, we can't automate the testing process and this
> stage remains important, if only to pick up bad builds.
Can't is a pretty big word. Haven't is more accurate.
I think that can be done, although it will require some changes in how
development is done. What it means is that some thought has to be put into
three things: a test framework, unit tests, and (since we're dealing with
kernel code running in supervisor mode) an overall deadman monitor for the
test machine to determine if the machine has hung based on some thing the
test did. A good test framework that can be incorporated into the build
system is a positive argument for looking at how the build is done.
The second item (unit tests) requires the discipline to generate them with
each patch. That can be hard, but it's by no means impossible. If a test
can be done by hand, then it's not impossible to automate it. You need to
have a way to create a known test scenario, simulate a sequence of events,
and know the expected outcomes. Functional testing is relatively
straightforward; load testing more difficult in that it requires multiple
drivers to get realistic tests.
The third item is well understood (easy to borrow from the embedded
systems world). We use a 2514 as a console serial heartbeat monitor from a
build system master controller to verify during the testing phase for
other tools. The STONITH code from 'cluster' is also helpful to construct
If you're interested in using our test harness, let's talk.=20