Testing

[Hal Murray]

esr at thyrsus.com said:
> https://blog.ntpsec.org/2017/02/22/testframe-the-epic-failure.html
> Read that and think about it for a while.  This is a very hard problem.  I
> hit it and bounced.

Thanks.

>From the blog page:
> In effect, the entire logic of the sync algorithms is a gigantic free
> parameter with no real equivalent in the simple, straight-line data
> transformations of gpsd, and only a weak analogy with the somewhat more
> complex but variable-free ones of reposurgeon. 

> Under these assumptions, there is some mutation rate threshold above which
> attempting deterministic replay simply stops being useful at all, because the
> gains from it are exceeded by the complexity costs of updating tests. GPSD
> and reposurgeon are well below that threshold; I now think ntpd is above it. 

I think I understand the ideas, but it's not making sense.

I agree that the core FSM is complicated, but how often do we change it?  I 
can't think of any changes, aka the mutation rate is close to zero.  Things 
like NTS are on the periphery.  It would be great to be able to run regression 
tests after adding NTS.  Yes, we would have to add new test cases in order to 
test NTS, but all the old tests should keep on working.


-- 
These are my opinions.  I hate spam.