Testing

[Hal Murray]

esr at thyrsus.com said:
> A lot of configuration options - even things like minsane - effectively
> change the FSM. 

Right.  But as you said, that's a configuration option.

> Sure, you can think of the config as part of the input state - this isn't a
> code mutation. But it also means you can only ever test very tiny parts of
> the input-state space, with no way to know when a config change might produce
> a boojum and tyically no way to have real confidence about how a test relates
> to behavior under any change in ...

Sure, but we can't currently test any of the state space.  I'd be very happy if we could test parts of the state space that are known to be interesting.

Your writeup focuses on code mutations rather than state space.  (Or maybe I didn't read what you intended.)

How do code mutations interact with the state space?  Do you have an example of a code mutation that would change things and that we wouldn't want to know about?

I expect changes in the logging would be the most common problem.  In most cases, I'd expect it would be a simple eyeball check on a diff and poke a button to accept the new version.  Did TESTFRAME have separate logging for the gettime call used for logging?

--------

The "known to be interesting" phrase gets back to my query that started this thread.  I'm looking for a way to test corner cases.  Would TESTFRAME would have done that?

If we don't like TESTFRAME, what else can we do?

Can we look for patterns in the log file from a live run?    This has the advantage of not requiring any changes to ntpd.

It gets complicated with lost packets and widely variable timing on the big bad internet.  A local net might be stable enough.

Suppose that works.  Can we describe the required configuration so that tests that start on my environment can be run on other environments?  I'm thinking of things like $BOB is a local stratum 2 server, $TED is local stratum 3.  ...

-----------

I'd like a way to test various OSes and hardware platforms.  I can think of two interesting areas.

One is the basic timekeeping - ntp_adjtime and friends.  Does normal ntpd work in some simple configuration.

The other is OpenSSL and friends.  That library gets a lot of attention, but it's large and we could be hitting a corner that doesn't get tested.  Various OSes/distros are running different versions and different patch levels and our code could have bugs in my attempts to dance around API changes.
 

-- 
These are my opinions.  I hate spam.