Testing

[Hal Murray]

Sorry for the delay.  I got distracted on other things.

While I think of it, did TESTFRAME dump floating point numbers in ASCII or 
hex?  If ASCII, there are likely to be round off errors when you read them 
back in.  Were there any floating point numbers that were read back in?  (as 
compared to loopstats where they are written out)


> How do you tell that any given capture represents correct operation? What
> check do you apply to the captured I/O history to verify that the sync
> algorithms were functioning as intended when the cature was taken?  *That's*
> the hard part - not the mechanics of TESTFRAME  itself, which is just
> tooling. 

OK, maybe we are getting someplace.

Is this a half full vs half empty problem?  I'm willing to assume that it is 
working correctly unless we get hints otherwise.

I'm trusting the guy who collects and contributes the sample data to verify 
that things are working correctly by looking at ntpq printout and/or graphs 
from log files and/or carefully reading ntpd.log and/or maybe other sources.  
This isn't a proof, but it's what we are currently using - the best we can do 
today.

In this context, there are two types of log files.  loopstats is probably best 
checked graphically.  ntpd.log may have single events or several related 
events that indicate that something happened.


> (You still don't know how to compose captures to trigger specied corner
> cases, but there's no point in worrying about that problem until you have
> your check procedure.)

Maybe "corner case" isn't the right term for the things I've been thinking 
about.  There are lots of cases that are reasonably easy to setup, but since 
there are a lot of them it would be nice if we could automate the procedure so 
I didn't have to go through the whole list every time we do a release.  
Testing at commit/push time is just a bonus.  Examples:
  Does "pool" work?  Do all the forms of crypto work?  All OSes/distros?  Does 
a single server work?  Does local clock work?  Can 2 truechimers outvote 1 
falseticker?

There may be actual obscure corner cases that are hard to generate.  I'd be 
willing to add logging for them so we can tell if an operational system hits 
them.

Do you remember anything about performance when collecting data?  I assume it 
would be reasonable to collect data on a client.  How much would it slow down 
a server?  How much disk space would a test require?

----------

Things are likely to break if we change a constant deep in the system or add a 
sanity check for an obscure case.  We haven't done that in a long time, so I'm 
not too worried about that problem.  Breaking tests may be a feature rather 
than a bug.  It will make us think twice about making that change.  If we 
decide the change is worthwhile, then we have to start the collection over.



-- 
These are my opinions.  I hate spam.