Testing ntpd and/or timing from gpsd

[Dan Drown]

Quoting "Eric S. Raymond" <esr at thyrsus.com>:
> Hal Murray <hmurray at megapathdsl.net>:
>> You can use a Pi with GPS to monitor other systems, but the Ethernet on USB
>> that limits the resolution of what you can measure.  It's probably good
>> enough for watching sites on the Internet and/or your ISP connection.  It's
>> probably not good enough if you are interested in the difference  
>> due to using
>> the kernel PLL.
>
> I'm planning to use the test farm to address that question.  The idea
> is: configure two Pi 3s identically, except that one ntpd build has
> KERNEL_PLL disabled. Probably the easiest way to arrange this will also
> be closest to the real-world case: benchmark refclock 20 against
> 28 being fed by gpsd.
>
> Run them in paallel for a couple days after they stabilize, collecting
> *mumble* statistics.  Compare statistics to see if we can see evidence
> of any accuracy gain above the noise floor.
>
> The main reason I haven't tried this already (other than being swamped with
> other tasks) is that I don't know what the value of *mumble* ought to be.
> What statistics should we collect?  What figure(s) of merit do we look
> for after the test run?
>
> I need a pretty detailed answer to this question before we can move forward
> on this front.  Please consider it carefully and bring the full weight of
> your expertise to bear.

I've found the built-in measurement to be useful.  I have this configuration:

statsdir /var/log/ntpstats/
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable

The local clock offset histogram (comes from loopstats data) has been  
the most useful to me to determine the quality of the clock  
synchronization.  For example, the offset histogram of my beaglebone  
black: https://dan.drown.org/bbb/offset-histogram.png

If there is only one number reported, I'd say it should be something  
along the lines of: "after ntpsec stabilized, 99% of all local clock  
offsets were within +/- X us" (or 99.9% for a tighter tolerance)

However, looking the data "over time" is useful to see the impact of  
various events.  They are helpful to debug conditions and measure  
changes.  It has been useful for: GNSS module loses lock and doesn't  
come back till someone hits the reset button, GNSS module goes crazy  
and starts giving >1ms offset in the PPS, changing the cpufreq  
settings improve the network latency and jitter but increase the  
temperature, etc.