My pre-1.0 wishlist

[Achim Gratz]

Eric S. Raymond writes:
> I don't believe you've thought the problem all of the way through.  All plans
> to test in a normal environment founder on one simple problem: you can't
> beat behavioral replication out of *this* software without spoofing the
> clock it sees.

You're actually trying to ascertain statistical validity of the
implementation, so this isn't absolutely necessary.  This misses certain
classes of errors that finally end up in the correct result anyway, but
that can be mitigated by unit-testing the parts.  In fact I'd prefer to
check e.g. the loop dynamics with unit tests rather than trying to come
up with input traces that exercise them in vivo.

> Unless you set up behavioral replicability (that is, an environment in
> which a known sequence of clock readings, I/O events, and other
> syscalls leads to another known sequence, or at least correct
> recognition teatures of same like ntpq -p showing what you expect) you
> don't have testing - because you don't know what output features
> discriminate between success and failure pf the test.
>
> Follow out the implications of that and I think you'll find there
> isn't any solution less "clever" than TESTFRAME or
> rr-with-clock-spoofing.  In fact I'm somewhat doubtful that rr will be
> strong enough.  Hoping to be wrong about that.
>
> Believe me, I'd love to get away with something much simpler, like
> gpsfake in GPSD that spoofs the daemon's I/O environment and feeds it
> data streams from the outside. But that won't work because you have to
> spoof the clock, too.  And socket opens -  your test environment needs
> to be able to pretend to be the entire net so TCP/IP traffic with the
> daemon under test can be controlled.

If you really want to do full behavioral testing like that, then I'd
rather go for complete mocking/scaffolding of the environment (including
kernel calls) to produce the required determinism to make the results
meaningful.  This way you can do a trace replay faster than real-time
and ensure that the input to ntpd is identical between runs.  There's
still the problem of coming up with traces that meaningfully exercise
the code.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds