My pre-1.0 wishlist

[Daniel Franke]

On 6/4/16, Eric S. Raymond <esr at thyrsus.com> wrote:

> We have three major possible pathways to this kind of test coverage.  One
> is Mozilla rr, the other is some kind of serious simplification attack
> on the hairball in the network code, and the third is somebody managing to
> grok the hairball in its full, present, hideous glory.

We don't need anything nearly as clever as Mozilla rr or TESTFRAME to
achieve the kind of automation I'm asking for. You want to test things
at the granularity of system calls (rr) or some abstraction that goes
deeper into the code (TESTFRAME). I want to test things at the
*user-visible* level. Automate the process of supplying configuration
files that exercise a variety of functionality, running them on real
hardware and real networks, and monitoring the results with ntpq. That
is, take the sort of testing that we're already doing by hand and make
it systematic and automatic.

> Better collaboration with NTP.org...well, that would sure be nice and
> I'd be willing, but I fear pigs will achieve escape velocity before we
> get any actual cooperation from them.  I won't be more specific on a
> list with publicly-accessible archives.
>
> Do you really think it's realistic to block 1.0 on that?  I don't.

If it can't be done then it can't be done, but I don't think we've
exhausted our options. I'll follow up off-list.

> I have a little list, I have a little list...
>
> Unfortunately you're entering another political minefield here.  At
> least two features I want to remove would cause a political shitstorm
> if word got out that we were thinking of diking them out *before* I
> have solid white papers to show they're nugatory.

Okay, if some of the featurectomies are going to be controversial no
matter what, then we don't have to block 1.0 on those. But let's do
the rest of them so that we're not (justifiably) accused of
gratuitously breaking people's setups post-1.0.

> I agree with the intent to revisit using something Phabricator-like.
> I disagree that that should be coupled with 1.0.  Stability ought
> to be measured by er, *stability*, not by whether we've gone through
> that process.

We're using two different senses of "stable". You're using it to mean
"free of crashes". I'm using it way Debian uses it: a version is
stable when it ceases to require frequent patching, and if a
particular, serious issue comes up (usually but not always
security-related) then the maintainers will provide an update that
fixes that issue and leaves everything else untouched. A corollary to
something being stable in this sense is that we can afford high
overhead for making changes, because we don't have to make very many
of them.