Testing

[Eric S. Raymond]

Hal Murray <hmurray at megapathdsl.net>:
> There was some sort of document on the release process.  Anybody remember 
> where it lives?

It's in devel/pre-release.txt.  I think you wrote it.

> In the short term, I think we have 2 options.  One is to make a bug fix 
> release with only a few lines of code changed.  That doesn't require much 
> testing.  The other is to release the current HEAD.  I think that deserves 
> serious testing.  Handwave, week.

Agreed. I like the second option (release HEAD) because of the lots of
little ntpq bugs we fixed after Hans Meyer reported them.

> > What does a "serious testing phase" look like? When we do a documentation
> > pass, how do we know when we're done? 
> 
> My view of a testing phase is either of two options.  One is to allow plenty 
> of time and ship when the timer runs out unless the fan is covered with brown 
> stuff.  That requires good judgment on the project managers part to pick the 
> duration of testing.  The other is to ship after you have gone N days without 
> finding anything interesting.  Again, somebody has to pick N.

We've more or less been doing your first option.  I'm OK with that continuing.
Can you think of a good reason to switch to the second one?

> My comment about a documentation pass was expecting at least one set of 
> eyeballs to look at each of the documents in our collection.  More than one 
> would be better.  It can be one person looking at everything or split over 
> several people.

I have a different plan.  I always write doc patches as part of my
change commits; my discipline is to prevent code and docs from getting out
of sync in the first place.

I'm not opposed to giving the docs an occasional separate once-over, I
just don't want that to substitute for the above. And there's the
how-do-you-know-you're done problem. What's the *acceptance criterion*
for a pass over the documents?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.