My pre-1.0 wishlist

[Achim Gratz]

Eric S. Raymond writes:
[…]
>    We say that test performance is *stable* when replay mode is
>    idempotent - that is, replaying an event-capture log produces an exact
>    copy of itself by duplicating the same output events.
>
>    When test performance is stable, we know two things: (1) we have
>    successfully captured all inputs of the system, and (2) the code
>    has experienced no functional regressions since the event capture.

Nope, you will just know that the output matches for that exact set of
inputs, which is a much weaker assurance than the one you (should) seek.
It doesn't tell you anything about how much of the code you have
exercised and whether the computations that produced the output were
correct.  Even determining the test coverage at that abstraction level
is going to be impossible.  You will further run into the problem that
setting up some specific internal state from the outside or triggering a
specific state transition can require extremely large and complex input
sequences that are hard to impossible to construct.

Your view of ntpd as a state machine seems more fruitful if you pare it
down from encompassing ntpd as a whole to something more manageable,
which is more likely to resemble a set of interconnected state machines.
The boundary betrween these machines are likely a good guide to figure
out any refactoring you might want to do.  There's a protocol engine
that processes NTP packets for instance and another part of ntpd really
is just a bunch of filters.  They share a process space today, but
that's not necessarily the optimum way of dealing with their
communication, which is only a subset of the complete state on both
sides of that boundary.

>    We can regression-test the code by capturing logs of production
>    behavior and replaying them.  We can also hand-craft tests to probe
>    edge cases.  To support the latter case, it is highly desirable that the
>    event-capture format be a text stream in an eyeball-friendly,
>    readily-editable format.

So you're envisioning a DSL for specifying NTP packets on the network and
events on all sorts of interfaces?  I can see why you wish you'd had
that, but that's a whole new set of problems to solve.

>    == Implementation ==
>
>    ntpd needs two new switches: capture and replay.  The capture switch
>    says: to log all event calls to an event capture file in addition to
>    their normal behaviors.  This includes both read events (such as
>    refclock inputs) and write events (such as adjtimex calls).
[…]

No, ntpd should not have or do any of that IMHO.  This must be done in
some wrapper that calls ntpd or even a VM specifically designed to do
all that recording and replay.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves