Testing

[Eric S. Raymond]

Hal Murray <hmurray at megapathdsl.net>:
> I agree that the core FSM is complicated, but how often do we change it?

A lot of configuration options - even things like minsane - effectively change
the FSM.

Sure, you can think of the config as part of the input state - this
isn't a code mutation. But it also means you can only ever test very
tiny parts of the input-state space, with no way to know when a config
change might produce a boojum and tyically no way to have real
confidence about how a test relates to behavior under any change in
configuration at all (I note an exception below). This is a kind
of brittleness that GPSD and reposurgeon don't have,

>                            It would be great to be able to run regression 
> tests after adding NTS.  Yes, we would have to add new test cases in order to 
> test NTS, but all the old tests should keep on working.

NTS is, I think, special. In a good way that other forms of auth share. 
There's a kind of decomposability about it - you can say with reasonable
confidence that once you're past a certain fairly early stage in the
packet-processing pipeline nothing about auth matters any more.

So yes, that's a corner of the testing problem that can probably be
bitten off.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>