From hmurray at megapathdsl.net  Mon Aug  1 00:13:23 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 31 Jul 2016 17:13:23 -0700
Subject: Removing the worst cruft
In-Reply-To: Message from Mark Atwood <fallenpegasus@gmail.com> of "Sun,
 31 Jul 2016 23:38:46 -0000."
 <CANW5CYXQwFEg0OBdT-mgqG5MF4t0+kwT8X7KKDUkL3nAzhQJgg@mail.gmail.com>
Message-ID: <20160801001323.F0BA9406057@ip-64-139-1-69.sjc.megapath.net>

> Can the palisade/trimble driver be replaced with a parse driver?

I doubt it, but I'm far from familiar with the parse driver.

Based on Eric's previous comments, the parse driver handles devices that 
provide the time in an easy to parse format.  TSIP might fit that if all goes 
well.

But there are many variations of TSIP.  One covers reversing the normal PPS 
operation.  Instead of needing kernel support to time stamp the PPS pulse, 
you send it a pulse by flapping one of the modem control signals and it tells 
you the time that happened.

My vote would be to not rock that boat.  There are more important things to 
work on.


-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Mon Aug  1 03:13:25 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sun, 31 Jul 2016 23:13:25 -0400
Subject: Removing the worst cruft
In-Reply-To: <20160801001323.F0BA9406057@ip-64-139-1-69.sjc.megapath.net>
References: <fallenpegasus@gmail.com>
 <CANW5CYXQwFEg0OBdT-mgqG5MF4t0+kwT8X7KKDUkL3nAzhQJgg@mail.gmail.com>
 <20160801001323.F0BA9406057@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160801031325.GA18480@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> > Can the palisade/trimble driver be replaced with a parse driver?
> 
> I doubt it, but I'm far from familiar with the parse driver.
> 
> Based on Eric's previous comments, the parse driver handles devices that 
> provide the time in an easy to parse format.  TSIP might fit that if all goes 
> well.
> 
> But there are many variations of TSIP.  One covers reversing the normal PPS 
> operation.  Instead of needing kernel support to time stamp the PPS pulse, 
> you send it a pulse by flapping one of the modem control signals and it tells 
> you the time that happened.
> 
> My vote would be to not rock that boat.  There are more important things to 
> work on.

I concur with this.

While I like the idea of replacing as many as possible of the
remaining legacy drives with modes of the generic driver, an absolute
requirement for this is that we be able to live-test with the equipment.
Obviously we're not set up for that yet.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From fallenpegasus at gmail.com  Mon Aug  1 03:14:32 2016
From: fallenpegasus at gmail.com (Mark Atwood)
Date: Mon, 01 Aug 2016 03:14:32 +0000
Subject: Removing the worst cruft
In-Reply-To: <20160801031325.GA18480@thyrsus.com>
References: <fallenpegasus@gmail.com>
 <CANW5CYXQwFEg0OBdT-mgqG5MF4t0+kwT8X7KKDUkL3nAzhQJgg@mail.gmail.com>
 <20160801001323.F0BA9406057@ip-64-139-1-69.sjc.megapath.net>
 <20160801031325.GA18480@thyrsus.com>
Message-ID: <CANW5CYUfsYhF3eU-FdcOek4EAyiDFxORiJFhR8rh6vSvZ63-hA@mail.gmail.com>

Good point.

On Sun, Jul 31, 2016, 8:13 PM Eric S. Raymond <esr at thyrsus.com> wrote:

> Hal Murray <hmurray at megapathdsl.net>:
> > > Can the palisade/trimble driver be replaced with a parse driver?
> >
> > I doubt it, but I'm far from familiar with the parse driver.
> >
> > Based on Eric's previous comments, the parse driver handles devices that
> > provide the time in an easy to parse format.  TSIP might fit that if all
> goes
> > well.
> >
> > But there are many variations of TSIP.  One covers reversing the normal
> PPS
> > operation.  Instead of needing kernel support to time stamp the PPS
> pulse,
> > you send it a pulse by flapping one of the modem control signals and it
> tells
> > you the time that happened.
> >
> > My vote would be to not rock that boat.  There are more important things
> to
> > work on.
>
> I concur with this.
>
> While I like the idea of replacing as many as possible of the
> remaining legacy drives with modes of the generic driver, an absolute
> requirement for this is that we be able to live-test with the equipment.
> Obviously we're not set up for that yet.
> --
>                 <a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160801/9d66f121/attachment.html>

From hmurray at megapathdsl.net  Mon Aug  1 09:40:11 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Mon, 01 Aug 2016 02:40:11 -0700
Subject: Kernel PLL graphs
Message-ID: <20160801094011.D771A406057@ip-64-139-1-69.sjc.megapath.net>


There are two parts to PPS processing in the kernel.  RFC 2783 describes an 
API for capturing time stamps.  RFC 1589 describes a PLL that lives in the 
kernel.

Most Linux distros don't support RFC 1589.  The code is in the kernel, but it doesn't work with the shipped kernels.  It requires !NO_HZ, but most distros prefer NO_HZ.

I pulled over the sources and built my own kernel.

Here are the before and after graphs:
  http://users.megapathdsl.net/~hmurray/ntpsec/PPS-kernel.png
The data is from two separate days so this isn't a clean comparison.  I don't know what that machine was doing on either day.

Here is a zoom in on the Kernel PLL day.
  http://users.megapathdsl.net/~hmurray/ntpsec/PPS-kernel2.png
Note that the peak offset is less than a microsecond.

------------

We should see if we can get similar results on a Raspberry Pi.  I haven't tried building an ARM kernel.

I think we should be able to run the PLL code outside the kernel.  The PPS time stamp is key.  The PLL calculations don't need to be run in the kernel.  They need to be run soon after the PPS, but not interrupt level immediately.  The API has an option to wakeup on PPS.  I don't know if it is implemented on Linux.

The no-PLL test was run at the default maxpoll of 6.  I should try faster.  I also need a standard test load.

I remember various FreeBSD-is-better type comments from many years ago.  I don't know if the PLL was working in Linux at the time.  I should setup a test case.


-- 
These are my opinions.  I hate spam.




From gem at rellim.com  Mon Aug  1 22:22:00 2016
From: gem at rellim.com (Gary E. Miller)
Date: Mon, 1 Aug 2016 15:22:00 -0700
Subject: Kernel PLL graphs
In-Reply-To: <20160801094011.D771A406057@ip-64-139-1-69.sjc.megapath.net>
References: <20160801094011.D771A406057@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160801152200.69d752c5@spidey.rellim.com>

Yo Hal!

On Mon, 01 Aug 2016 02:40:11 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> Here are the before and after graphs:
>   http://users.megapathdsl.net/~hmurray/ntpsec/PPS-kernel.png
> The data is from two separate days so this isn't a clean comparison.
> I don't know what that machine was doing on either day.

I would have expected your non-kernel PLL to be 30x or 100x better.

Even on a RasPi2 I get much better, -400 ?Sec to +800?Sec offset on the
PPS:
    https://pi2.rellim.com/day/#PPS

Are you sure your kernel PPS (RFC 2783) time stamping is working?

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: remote-peerstats.127.127.28.1.png
Type: image/png
Size: 15365 bytes
Desc: not available
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160801/714d2688/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160801/714d2688/attachment.bin>

From gem at rellim.com  Tue Aug  2 02:40:18 2016
From: gem at rellim.com (Gary E. Miller)
Date: Mon, 1 Aug 2016 19:40:18 -0700
Subject: =?UTF-8?B?4pyYZHJpZnQ=?= file
Message-ID: <20160801194018.17e7a3e6@spidey.rellim.com>

Yo All!

Eric asked me to write up why I thought the chrony drift file handling
is better than NTPsec's handling.

1.  On startup chronyd checks the time stamp on the drift file.
    if the timestamp > sysclock, the sysclock is set to the timestamp

    This is a nice sanity check on the system clock.

2.  ntpd stores the frequency ppm offset in the driftfile.
    chronyd stores the frequency ppm offset and the 'skew' (estimated accuracy 
    of the existing frequency value).

    Knowing the 'skew' at startup allows chrony to better reject bad
    reclock input.
    
I can see that saving the 'skew' is a nice touch, but I suspect much the
good chronyd startup behavior is explained elsewhere.

In a related topic, it would be nice (maybe an option) for ntpd to hold
off logging the initial aweful data until after the -g option has
set the system clock.  And a bit longer, so the wonky startup data is
masked.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160801/f105a5ee/attachment.bin>

From gem at rellim.com  Tue Aug  2 03:40:27 2016
From: gem at rellim.com (Gary E. Miller)
Date: Mon, 1 Aug 2016 20:40:27 -0700
Subject: =?UTF-8?B?4pyYbnRwdml6?=
Message-ID: <20160801204027.0e9d6518@spidey.rellim.com>

Yo Eric!

No rush on these, but i was playing with stats today, so I looked at
ntpstats.  Looking good, but not ready for me to try to use live.

1. not installed by default

2. no help:
	# ./ntpviz  -h
	option -h not recognized

3. not finding my Liberation fonts:

	# ./ntpviz  -d /var/log/ntpstats
	warning: liberation truetype fonts not found

chrony-graph finds them just fine, they are in:
    /usr/share/fonts/liberation-fonts/

And I have the Sans-regular:

    /usr/share/fonts/liberation-fonts/LiberationSans-Regular.ttf

3. not working options:

	# ./ntpviz  -d /var/log/ntpstats --clock-offset
	option --clock-offset not recognized

	# ./ntpviz  -d /var/log/ntpstats --clock-jitter
	option --clock-jitter not recognized

	# ./ntpviz  -d /var/log/ntpstats --clock-stability > tmp.plot
	option --clock-stability not recognized

4. --peer-jitters, works.  
   Missing 99%, 90%, etc. lines and values. 
   Why '--peer-jitters" when it only takes one argment, and can only be used
   once?

5. --peer-jitters, works.  
   Missing 50% line and value.  I'd also like the 90% and 5% lines/values.
   Why '--peer-offsets" when it only takes one argment, and can only be used
   once?

6. error handling needs work:

    # ./ntpviz  -d /var/log/ntpstats --peer-offsets 204.17.205.8  > tmp.plot
    Traceback (most recent call last):
      File "./ntpviz", line 107, in <module>
	plot = stats.peer_offsets_gnuplot(show_peer_offsets)
      File "/u1/src/NTP/ntpsec/ntpstats/ntpstats.py", line 225, in peer_offsets_gnuplot
	return self.peerstats_gnuplot(peerlist, 4, "Peer clock offset")
      File "/u1/src/NTP/ntpsec/ntpstats/ntpstats.py", line 221, in peerstats_gnuplot
	stderr.write("No such peer as %s" % key)
    NameError: global name 'stderr' is not defined

7. "-n name" seems to do nothing.

8. looks good: -all-peer-offsets

9. looks good: -all-peer-jitters

10. -s broken:

    # ./ntpviz  -d /var/log/ntpstats --peer-offsets 204.17.205.1  -s 1470021644 > tmp.plot
    Traceback (most recent call last):
      File "./ntpviz", line 48, in <module>
	starttime = iso_to_unix(val)
      File "/u1/src/NTP/ntpsec/ntpstats/ntpstats.py", line 282, in iso_to_unix
	return calendar.timelocal(time.strptime(tv, "%Y-%m-%dT%H:%M:%S"))
    AttributeError: 'module' object has no attribute 'timelocal'

11. "-e endtime" would be nice.
    Then I could do a nice week or month plot without having to run
    exactly after the period end.
   
12. seems to do nothing by default:

    # ./ntpviz
    # 

13. not sure what to do about html...

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160801/af6cb12b/attachment.bin>

From fallenpegasus at gmail.com  Tue Aug  2 19:34:16 2016
From: fallenpegasus at gmail.com (Mark Atwood)
Date: Tue, 02 Aug 2016 19:34:16 +0000
Subject: =?UTF-8?B?UmU6IOKcmGRyaWZ0IGZpbGU=?=
In-Reply-To: <20160801194018.17e7a3e6@spidey.rellim.com>
References: <20160801194018.17e7a3e6@spidey.rellim.com>
Message-ID: <CANW5CYW+3OU5Xxs2SG1ijJ2DCLq3vv_6aZe53S=T_dt-Gf3G5g@mail.gmail.com>

If we make this change, framing it as "it's how chronyd has been doing it
for the past N years" makes it a much easier sell.

Especially if we can make the file format the same.

What principled objections would the hardcore time nerds have?   We do have
to keep their needs firmly in mind.

..m

On Mon, Aug 1, 2016 at 7:40 PM Gary E. Miller <gem at rellim.com> wrote:

> Yo All!
>
> Eric asked me to write up why I thought the chrony drift file handling
> is better than NTPsec's handling.
>
> 1.  On startup chronyd checks the time stamp on the drift file.
>     if the timestamp > sysclock, the sysclock is set to the timestamp
>
>     This is a nice sanity check on the system clock.
>
> 2.  ntpd stores the frequency ppm offset in the driftfile.
>     chronyd stores the frequency ppm offset and the 'skew' (estimated
> accuracy
>     of the existing frequency value).
>
>     Knowing the 'skew' at startup allows chrony to better reject bad
>     reclock input.
>
> I can see that saving the 'skew' is a nice touch, but I suspect much the
> good chronyd startup behavior is explained elsewhere.
>
> In a related topic, it would be nice (maybe an option) for ntpd to hold
> off logging the initial aweful data until after the -g option has
> set the system clock.  And a bit longer, so the wonky startup data is
> masked.
>
> RGDS
> GARY
> ---------------------------------------------------------------------------
> Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
>         gem at rellim.com  Tel:+1 541 382 8588
> _______________________________________________
> devel mailing list
> devel at ntpsec.org
> http://lists.ntpsec.org/mailman/listinfo/devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160802/9acd0e2c/attachment.html>

From gem at rellim.com  Tue Aug  2 19:53:09 2016
From: gem at rellim.com (Gary E. Miller)
Date: Tue, 2 Aug 2016 12:53:09 -0700
Subject: =?UTF-8?B?4pyYZHJpZnQ=?= file
In-Reply-To: <CANW5CYW+3OU5Xxs2SG1ijJ2DCLq3vv_6aZe53S=T_dt-Gf3G5g@mail.gmail.com>
References: <20160801194018.17e7a3e6@spidey.rellim.com>
 <CANW5CYW+3OU5Xxs2SG1ijJ2DCLq3vv_6aZe53S=T_dt-Gf3G5g@mail.gmail.com>
Message-ID: <20160802125309.4480bc3f@spidey.rellim.com>

Yo Mark!

On Tue, 02 Aug 2016 19:34:16 +0000
Mark Atwood <fallenpegasus at gmail.com> wrote:

> If we make this change, framing it as "it's how chronyd has been
> doing it for the past N years" makes it a much easier sell.

chronyd setting sysclock to driftfile time is only since Agu 2014.  In
Oct 2015 they made it optiona with their -s option, musta been some
problems for some people.  With the non-default -s option it is backward
compatible and people can use it if they want.

A lot of distros, like Gentoo, preceed ntpd with a shell script to 
set a worst case sysclock from a file touched on shutdown.  So the -s is
just internalizing common practice.

Gentoo calls it /etc/init.d/swclock.

> Especially if we can make the file format the same.

ntpd puts just the freq ppm offset on one line in the driftfile.
chronyd puts the freq ppm and the freq skew on one line in the file.

It would be easy to change ntpd to accept the second parameter, and
always write the second parameter.  Then maybe only use the second
parameter with a non-default option.

Only if people like it and gain confidence would it then become 
Best Practice, or even default.

> What principled objections would the hardcore time nerds have?   We
> do have to keep their needs firmly in mind.

I can't see how they can object if the two features are non-default.

Saving the freq skew for restart could improve startup performance,
but how much is TBD.  With the option we can test it.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160802/a2c18372/attachment.bin>

From gem at rellim.com  Tue Aug  2 20:00:02 2016
From: gem at rellim.com (Gary E. Miller)
Date: Tue, 2 Aug 2016 13:00:02 -0700
Subject: =?UTF-8?B?4pyYZHJpZnQ=?= file
In-Reply-To: <20160802125309.4480bc3f@spidey.rellim.com>
References: <20160801194018.17e7a3e6@spidey.rellim.com>
 <CANW5CYW+3OU5Xxs2SG1ijJ2DCLq3vv_6aZe53S=T_dt-Gf3G5g@mail.gmail.com>
 <20160802125309.4480bc3f@spidey.rellim.com>
Message-ID: <20160802130002.0ed11a48@spidey.rellim.com>

Yo Mark!

On Tue, 02 Aug 2016 19:34:16 +0000
Mark Atwood <fallenpegasus at gmail.com> wrote:

> If we make this change, framing it as "it's how chronyd has been
> doing it for the past N years" makes it a much easier sell.  

Forgot an important part.  chronyd has saved the freq ppm and freq skew in
the driftfile since Jan 2006.  Over 10 years ago.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160802/333fed46/attachment.bin>

From hmurray at megapathdsl.net  Wed Aug  3 10:13:47 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Wed, 03 Aug 2016 03:13:47 -0700
Subject: driftMime-Version: 1.0
Message-ID: <20160803101347.3ADDD406060@ip-64-139-1-69.sjc.megapath.net>


gem at rellim.com said:
> 1.  On startup chronyd checks the time stamp on the drift file.
>     if the timestamp > sysclock, the sysclock is set to the timestamp 

I vote that we don't do anything, not even make it optional behind a command 
line switch.

We have more important things to do.

The OS should be doing that sort of thing, probably using the root directory.
Why stop with the drift file?  Should we check the log files too?

It's the sort of code that is hard to test and likely to have subtle problems.

I think it's a good item to put on the what-do-customers-want list.



> 2.  ntpd stores the frequency ppm offset in the driftfile. 
>     chronyd stores the frequency ppm offset and the 'skew'
> (estimated accuracy of the existing frequency value). 

> I can see that saving the 'skew' is a nice touch, but I suspect much the
> good chronyd startup behavior is explained elsewhere. 

I'm not sure that ntpd has a parameter equivalent to skew.

Again, I vote that we don't do anything now.  The current startup stuff is 
broken.  There is no point in working on things like this until we understand 
and fix the current problems.


gem at rellim.com said:
> In a related topic, it would be nice (maybe an option) for ntpd to hold off
> logging the initial aweful data until after the -g option has set the system
> clock.  And a bit longer, so the wonky startup data is masked. 

But that is when you really really want the logging.

I might agree to put it someplace other than the normal place.


-- 
These are my opinions.  I hate spam.




From Matthew.Selsky at twosigma.com  Wed Aug  3 18:49:44 2016
From: Matthew.Selsky at twosigma.com (Matthew Selsky)
Date: Wed, 3 Aug 2016 14:49:44 -0400
Subject: Kernel PLL graphs
In-Reply-To: <20160801094011.D771A406057@ip-64-139-1-69.sjc.megapath.net>
References: <20160801094011.D771A406057@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160803184944.GE25969@twosigma.com>

On Mon, Aug 01, 2016 at 02:40:11AM -0700, Hal Murray wrote:
> 
> There are two parts to PPS processing in the kernel.  RFC 2783 describes an 
> API for capturing time stamps.  RFC 1589 describes a PLL that lives in the 
> kernel.
> 
> Most Linux distros don't support RFC 1589.  The code is in the kernel, but it doesn't work with the shipped kernels.  It requires !NO_HZ, but most distros prefer NO_HZ.
> 
> I pulled over the sources and built my own kernel.
> 
> Here are the before and after graphs:
>   http://users.megapathdsl.net/~hmurray/ntpsec/PPS-kernel.png
> The data is from two separate days so this isn't a clean comparison.  I don't know what that machine was doing on either day.
> 
> Here is a zoom in on the Kernel PLL day.
>   http://users.megapathdsl.net/~hmurray/ntpsec/PPS-kernel2.png
> Note that the peak offset is less than a microsecond.
> 
> ------------
> 
> We should see if we can get similar results on a Raspberry Pi.  I haven't tried building an ARM kernel.
> 
> I think we should be able to run the PLL code outside the kernel.  The PPS time stamp is key.  The PLL calculations don't need to be run in the kernel.  They need to be run soon after the PPS, but not interrupt level immediately.  The API has an option to wakeup on PPS.  I don't know if it is implemented on Linux.
> 
> The no-PLL test was run at the default maxpoll of 6.  I should try faster.  I also need a standard test load.
> 
> I remember various FreeBSD-is-better type comments from many years ago.  I don't know if the PLL was working in Linux at the time.  I should setup a test case.

Hey Hal,

I'm using maxpoll of 1 on my stratum 1 servers.  And I have !NO_HZ set.  My offsets stay belong 1 microsecond as reported by ntpq.  If we switched the units to nanoseconds, that might be interesting.

I don't have !NO_HZ set on my stratum 2 servers, but I'm looking at the ramifications of that.

I'm curious what your results are.

Cheers,
-Matt

From gem at rellim.com  Wed Aug  3 19:26:09 2016
From: gem at rellim.com (Gary E. Miller)
Date: Wed, 3 Aug 2016 12:26:09 -0700
Subject: driftMime-Version: 1.0
In-Reply-To: <20160803101347.3ADDD406060@ip-64-139-1-69.sjc.megapath.net>
References: <20160803101347.3ADDD406060@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160803122609.2847616f@spidey.rellim.com>

Yo Hal!

On Wed, 03 Aug 2016 03:13:47 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> gem at rellim.com said:
> > 1.  On startup chronyd checks the time stamp on the drift file.
> >     if the timestamp > sysclock, the sysclock is set to the
> > timestamp   

> We have more important things to do.

A few lines of code, faster to code than to argue about it.  Better
yet, steal tthe chronyd code for it.
 
> The OS should be doing that sort of thing, probably using the root
> directory. Why stop with the drift file?  Should we check the log
> files too?

We can't trust the OS to do the rightt thing about time.  The OS
trusts ntpd to do tthe right thing, but it does not.

> It's the sort of code that is hard to test and likely to have subtle
> problems.

Really?  Just 'touch /var/log/ntppstats/driftfile' to a time in the
future, start nttpd, and then check the system time.

> I think it's a good item to put on the what-do-customers-want list.

I assure you Rasi uuers really want this.  The lack of an RTC causes
real problems.  Every time I reboot a RasPi I curse ntpd's poor
startup behavior.  Almost enough to go back to chronyd.

> > 2.  ntpd stores the frequency ppm offset in the driftfile. 
> >     chronyd stores the frequency ppm offset and the 'skew'
> > (estimated accuracy of the existing frequency value).   
> 
> > I can see that saving the 'skew' is a nice touch, but I suspect
> > much the good chronyd startup behavior is explained elsewhere.   
> 
> I'm not sure that ntpd has a parameter equivalent to skew.

I think this is what ntpd calls 'RMS Jitter'.  I'm not clear on the ntpd
internals, but I do know it has methods for clock selection, and they fail
badly on startup. ntpd does not need to exactly mirror what chronyd does.
But it clearly needs better start up behavior, and since chronyd starts
up muuch better than ntpd that is a good place to start looking.

> Again, I vote that we don't do anything now.  The current startup
> stuff is broken.  There is no point in working on things like this
> until we understand and fix the current problems.

Clearly chronyd understands the problem, so looking at that code is
ppart of the path to unuderstanding and fixing the problem.


> gem at rellim.com said:
> > In a related topic, it would be nice (maybe an option) for ntpd to
> > hold off logging the initial aweful data until after the -g option
> > has set the system clock.  And a bit longer, so the wonky startup
> > data is masked.   
> 
> But that is when you really really want the logging.

Sometimes, but then it takes a week for my graphs to be readable
again...

> I might agree to put it someplace other than the normal place.

Works for me, or maybe a switch.  Or maybe fix the startup problems.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160803/f7a323c6/attachment.bin>

From gem at rellim.com  Wed Aug  3 19:45:53 2016
From: gem at rellim.com (Gary E. Miller)
Date: Wed, 3 Aug 2016 12:45:53 -0700
Subject: Kernel PLL graphs
In-Reply-To: <20160803184944.GE25969@twosigma.com>
References: <20160801094011.D771A406057@ip-64-139-1-69.sjc.megapath.net>
 <20160803184944.GE25969@twosigma.com>
Message-ID: <20160803124553.3cd9d755@spidey.rellim.com>

Yo Matthew!

On Wed, 3 Aug 2016 14:49:44 -0400
Matthew Selsky <Matthew.Selsky at twosigma.com> wrote:

> I'm using maxpoll of 1 on my stratum 1 servers.  And I have !NO_HZ
> set.  My offsets stay belong 1 microsecond as reported by ntpq.  If
> we switched the units to nanoseconds, that might be interesting.

chronyc reports to the nanoSec.  ntpd pputs nanoSec in the peerstats
log file.

> I don't have !NO_HZ set on my stratum 2 servers, but I'm looking at
> the ramifications of that.

I found no huge difference, but a reduuction in the odd wild hare
measurement.  Noticeable on chrony-graph.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160803/1eb2307a/attachment.bin>

From gem at rellim.com  Wed Aug  3 20:35:26 2016
From: gem at rellim.com (Gary E. Miller)
Date: Wed, 3 Aug 2016 13:35:26 -0700
Subject: =?UTF-8?B?4pyYYnVpbGQ=?= with Python 3.4 broken.
Message-ID: <20160803133526.04adf280@spidey.rellim.com>

Yo All!

I just tried to compile ntpsec on a new gentoo install.  It failed.

pi ntpsec # ../Do-config-ntpsec 
../Do-config-ntpsec: line 3: cd: ntpsec: No such file or directory
Waf: The wscript in '/usr/local/src/NTP/ntpsec' is unreadable
Traceback (most recent call last):
  File "/usr/local/src/NTP/ntpsec/.waf3-1.8.20-c859ca7dc3693011756f4edf45c36626/waflib/Scripting.py", line 104, in waf_entry_point
    set_main_module(os.path.normpath(os.path.join(Context.run_dir,Context.WSCRIPT_FILE)))
  File "/usr/local/src/NTP/ntpsec/.waf3-1.8.20-c859ca7dc3693011756f4edf45c36626/waflib/Scripting.py", line 129, in set_main_module
    Context.g_module=Context.load_module(file_path)
  File "/usr/local/src/NTP/ntpsec/.waf3-1.8.20-c859ca7dc3693011756f4edf45c36626/waflib/Context.py", line 354, in load_module
    try:exec(compile(code,path,'exec'),module.__dict__)
  File "/usr/local/src/NTP/ntpsec/wscript", line 11, in <module>
    from pylib.configure import cmd_configure
  File "/usr/local/src/NTP/ntpsec/pylib/configure.py", line 154
    print "ID    Description"
                            ^
SyntaxError: Missing parentheses in call to 'print'
Waf: The wscript in '/usr/local/src/NTP/ntpsec' is unreadable
Traceback (most recent call last):
  File "/usr/local/src/NTP/ntpsec/.waf3-1.8.20-c859ca7dc3693011756f4edf45c36626/waflib/Scripting.py", line 104, in waf_entry_point
    set_main_module(os.path.normpath(os.path.join(Context.run_dir,Context.WSCRIPT_FILE)))
  File "/usr/local/src/NTP/ntpsec/.waf3-1.8.20-c859ca7dc3693011756f4edf45c36626/waflib/Scripting.py", line 129, in set_main_module
    Context.g_module=Context.load_module(file_path)
  File "/usr/local/src/NTP/ntpsec/.waf3-1.8.20-c859ca7dc3693011756f4edf45c36626/waflib/Context.py", line 354, in load_module
    try:exec(compile(code,path,'exec'),module.__dict__)
  File "/usr/local/src/NTP/ntpsec/wscript", line 11, in <module>
    from pylib.configure import cmd_configure
  File "/usr/local/src/NTP/ntpsec/pylib/configure.py", line 154
    print "ID    Description"
                            ^
SyntaxError: Missing parentheses in call to 'print'

Ah, Gentoo now uses Python 3.4 by default.  Switching back to 2.7 and
things are better.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160803/0cd2ac14/attachment.bin>

From verm at darkbeer.org  Wed Aug  3 20:40:29 2016
From: verm at darkbeer.org (Amar Takhar)
Date: Wed, 3 Aug 2016 20:40:29 +0000
Subject: ???build with Python 3.4 broken.
In-Reply-To: <20160803133526.04adf280@spidey.rellim.com>
References: <20160803133526.04adf280@spidey.rellim.com>
Message-ID: <20160803204029.GA49838@darkbeer.org>

On 2016-08-03 13:35 -0700, Gary E. Miller wrote:
> Yo All!
> 
> I just tried to compile ntpsec on a new gentoo install.  It failed.

There is a ticket for this I will try to get my patch in soon.


Amar.

From hmurray at megapathdsl.net  Wed Aug  3 21:24:01 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Wed, 03 Aug 2016 14:24:01 -0700
Subject: Kernel PLL graphs
In-Reply-To: Message from Matthew Selsky <Matthew.Selsky@twosigma.com>
 of "Wed, 03 Aug 2016 14:49:44 EDT." <20160803184944.GE25969@twosigma.com>
Message-ID: <20160803212401.75CE5406057@ip-64-139-1-69.sjc.megapath.net>


Matthew.Selsky at twosigma.com said:
> I'm using maxpoll of 1 on my stratum 1 servers.  And I have !NO_HZ set.  My
> offsets stay belong 1 microsecond as reported by ntpq.  If we switched the
> units to nanoseconds, that might be interesting.

Time to make sure I've got the right number of negatives...  "I have !NO_HZ 
set" means you have unset NO_HZ which probably means you had to build your 
own kernel.

Do you have flag3 turned on?  If so, the kernel does all the work and maxpoll 
is essentially ignored.

I though there was a min to maxpoll so I'm a bit surprised you could set it 
to 1.

> I don't have !NO_HZ set on my stratum 2 servers, but I'm looking at the
> ramifications of that.

At least for the effect I'm discussing, it only matters if you have a PPS.

> I'm curious what your results are. 



-- 
These are my opinions.  I hate spam.




From Matthew.Selsky at twosigma.com  Wed Aug  3 21:49:41 2016
From: Matthew.Selsky at twosigma.com (Matthew Selsky)
Date: Wed, 3 Aug 2016 17:49:41 -0400
Subject: Kernel PLL graphs
In-Reply-To: <20160803212401.75CE5406057@ip-64-139-1-69.sjc.megapath.net>
References: <Matthew.Selsky@twosigma.com> <20160803184944.GE25969@twosigma.com>
 <20160803212401.75CE5406057@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160803214940.GF25969@twosigma.com>

On Wed, Aug 03, 2016 at 02:24:01PM -0700, Hal Murray wrote:

> Time to make sure I've got the right number of negatives...  "I have !NO_HZ 
> set" means you have unset NO_HZ which probably means you had to build your 
> own kernel.

We build our own kernels and we boot our stratum 1 clocks with "nohz=off"

> Do you have flag3 turned on?  If so, the kernel does all the work and maxpoll 
> is essentially ignored.

We do not have flag3 turned on.  ntpd is reading the GPS via shared memory.  The GPS comes with a lightweight daemon that reads /dev/refclock0 and stuffs the time in shared memory.

> I though there was a min to maxpoll so I'm a bit surprised you could set it 
> to 1.

We had a local patch to change the minimum value of minpoll/maxpoll to 1.  Eric recently committed a similar patch upstream. https://gitlab.com/NTPsec/ntpsec/commit/a3047c7a375877436d422e04a138aace7ce1bd06

> At least for the effect I'm discussing, it only matters if you have a PPS.

We're using PCIe GPS receiver cards.


Cheers,
-Matt


From hmurray at megapathdsl.net  Wed Aug  3 22:15:42 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Wed, 03 Aug 2016 15:15:42 -0700
Subject: driftMime-Version: 1.0
In-Reply-To: Message from "Gary E. Miller" <gem@rellim.com>
 of "Wed, 03 Aug 2016 12:26:09 PDT." <20160803122609.2847616f@spidey.rellim.com>
Message-ID: <20160803221542.7434B406057@ip-64-139-1-69.sjc.megapath.net>


gem at rellim.com said:
> A few lines of code, faster to code than to argue about it.  Better yet,
> steal tthe chronyd code for it. 

I could make a similar argument.  In this case, I think it doesn't apply.

I can kludge up a one time test.  We can't test all the strange cases.  Yes, 
the risk is low, but we need to focus on important things.

We need to get Eric working on TESTFRAME.  I think it's long past time to 
stop getting distracted by things like this.  I already spend too much time 
sorting out minor bugs introduced by changes.



-- 
These are my opinions.  I hate spam.




From gem at rellim.com  Wed Aug  3 22:23:37 2016
From: gem at rellim.com (Gary E. Miller)
Date: Wed, 3 Aug 2016 15:23:37 -0700
Subject: driftMime-Version: 1.0
In-Reply-To: <20160803221542.7434B406057@ip-64-139-1-69.sjc.megapath.net>
References: <20160803122609.2847616f@spidey.rellim.com>
 <20160803221542.7434B406057@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160803152337.63be27f7@spidey.rellim.com>

Yo Hal!

On Wed, 03 Aug 2016 15:15:42 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> gem at rellim.com said:
> > A few lines of code, faster to code than to argue about it.  Better
> > yet, steal tthe chronyd code for it.   
> 
> I could make a similar argument.  In this case, I think it doesn't
> apply.
> 
> I can kludge up a one time test.  We can't test all the strange
> cases.  Yes, the risk is low, but we need to focus on important
> things.

Anything that might improve startup will be a big win.  Saving a couple
more state variables could be that win.

> We need to get Eric working on TESTFRAME.  I think it's long past
> time to stop getting distracted by things like this.  I already spend
> too much time sorting out minor bugs introduced by changes.

Yes, Eric needs to focus on TESTFRAME, once he finishes the ntpviz.
The ntpviz at least tells you the build is working.  Something lacking
now.  chrony-graph has picked up a lot of subtle things for me.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160803/8794cf3d/attachment.bin>

From hmurray at megapathdsl.net  Wed Aug  3 22:54:19 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Wed, 03 Aug 2016 15:54:19 -0700
Subject: driftMime-Version: 1.0
In-Reply-To: Message from "Gary E. Miller" <gem@rellim.com>
 of "Wed, 03 Aug 2016 15:23:37 PDT." <20160803152337.63be27f7@spidey.rellim.com>
Message-ID: <20160803225419.AF369406057@ip-64-139-1-69.sjc.megapath.net>


gem at rellim.com said:
> Yes, Eric needs to focus on TESTFRAME, once he finishes the ntpviz. The
> ntpviz at least tells you the build is working.  Something lacking now.
> chrony-graph has picked up a lot of subtle things for me. 

I'd put ntpviz behind TESTFRAME.  We'll have plenty of time to work on 
visualization while collecting interesting test cases to feed to TESTFRAME.

-- 
These are my opinions.  I hate spam.




From gem at rellim.com  Wed Aug  3 22:58:19 2016
From: gem at rellim.com (Gary E. Miller)
Date: Wed, 3 Aug 2016 15:58:19 -0700
Subject: driftMime-Version: 1.0
In-Reply-To: <20160803225419.AF369406057@ip-64-139-1-69.sjc.megapath.net>
References: <20160803152337.63be27f7@spidey.rellim.com>
 <20160803225419.AF369406057@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160803155819.44887045@spidey.rellim.com>

Yo Hal!

On Wed, 03 Aug 2016 15:54:19 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> gem at rellim.com said:
> > Yes, Eric needs to focus on TESTFRAME, once he finishes the ntpviz.
> > The ntpviz at least tells you the build is working.  Something
> > lacking now. chrony-graph has picked up a lot of subtle things for
> > me.   
> 
> I'd put ntpviz behind TESTFRAME.  We'll have plenty of time to work
> on visualization while collecting interesting test cases to feed to
> TESTFRAME.

Except ntpviz only took a few days, and likely only needs another day
or two.  Just needs some tweaks to the graphs, and a harness to create
all the graphs automagically from the ntp.conf.  Plus maybe an html
generator.

Bsaically the hard part is copied from chrony-graph, and a bit more to do.

TESTFRAME might take the rest of the year.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160803/c304ac41/attachment.bin>

From hmurray at megapathdsl.net  Wed Aug  3 23:23:44 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Wed, 03 Aug 2016 16:23:44 -0700
Subject: driftMime-Version: 1.0
In-Reply-To: Message from "Gary E. Miller" <gem@rellim.com>
 of "Wed, 03 Aug 2016 15:58:19 PDT." <20160803155819.44887045@spidey.rellim.com>
Message-ID: <20160803232345.09426406057@ip-64-139-1-69.sjc.megapath.net>


gem at rellim.com said:
> Except ntpviz only took a few days, and likely only needs another day or
> two.  Just needs some tweaks to the graphs, and a harness to create all the
> graphs automagically from the ntp.conf.  Plus maybe an html generator. 

Plus all the other stuff that you toss in once that gets working well enough 
to see what else you want.

> TESTFRAME might take the rest of the year.

It will take even longer if Eric keeps working on other things.

We need to get Eric focused on TESTFRAME.



-- 
These are my opinions.  I hate spam.




From gem at rellim.com  Wed Aug  3 23:52:31 2016
From: gem at rellim.com (Gary E. Miller)
Date: Wed, 3 Aug 2016 16:52:31 -0700
Subject: driftMime-Version: 1.0
In-Reply-To: <20160803232345.09426406057@ip-64-139-1-69.sjc.megapath.net>
References: <20160803155819.44887045@spidey.rellim.com>
 <20160803232345.09426406057@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160803165231.18debae8@spidey.rellim.com>

Yo Hal!

On Wed, 03 Aug 2016 16:23:44 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> gem at rellim.com said:
> > Except ntpviz only took a few days, and likely only needs another
> > day or two.  Just needs some tweaks to the graphs, and a harness to
> > create all the graphs automagically from the ntp.conf.  Plus maybe
> > an html generator.   
> 
> Plus all the other stuff that you toss in once that gets working well
> enough to see what else you want.

I've been using chrony-graph for months now.  When ntpviz mostly duplicates
that, I'll be happy for the mear term.

> > TESTFRAME might take the rest of the year.  
> 
> It will take even longer if Eric keeps working on other things.
> 
> We need to get Eric focused on TESTFRAME.

I suspect that Eric has not dived into TESTFRAME since he does not have 
a good mental model he likes yet.  That is the hard part, and the part
I think he is unsure on.

If it were me, I'd consider shimming a few syscalls: adjtime(), send(),
recv(), clock_gettime().  Maybe at the runtime linker level, maybe
at the libntpd_lib.a level.

Then run a good ntpd in record mode.  With that saved data, a test
ntpd could be run in playback mode.

To keep the playback time small, maybe also a way to save internal state
to a file, and reload that state in a test ntpd.  A nice state dump
could also be a good debugging tool.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160803/85bd803e/attachment.bin>

From christian.ehrhardt at canonical.com  Tue Aug  9 15:10:16 2016
From: christian.ehrhardt at canonical.com (Christian Ehrhardt)
Date: Tue, 9 Aug 2016 17:10:16 +0200
Subject: Discussion about PR: WIP: Snapify ntpsec
Message-ID: <CAATJJ0JG+9gGgRWU0_m2C0D0OAwRUVK_t=_B5vdUpQ21M8UB9w@mail.gmail.com>

Hi,
I wanted to give the ML a ping as well about this, so that not only the
Pull Request is existing.
Eventually one here might chime in as well.

There is a prototype to snap ntpsec at
https://gitlab.com/NTPsec/ntpsec/merge_requests/49

I'll quote my PR text here and hope for a great discussion:

"Hi, on one hand I worked on packaging ntp (classic) recently and on the
other hand I worked a bit with snapcraft (=> http://snapcraft.io/). I
really think ntpsec would be a perfect candidate to exploit snap packaging.

Please consider this an RFC for now - following the spirit of NTPsec
contribution policy "Before starting significant work, please propose it
and discuss it first" I'll also write to the ML linking to this branch. But
also did I not just want to mention snapcraft and run away - instead I
thought to provide a prototype that can be tested, but discuss motivation,
tech and details before doing some more heavy lifting work.

My current example is meant for a daily build, but this can easily be
changed to whatever you prefer. Snapcraft could - for example - build from
a stable branch of your tree automatically or whatever else you want.

Benefits of exploiting snap(craft) in ntpsec (in my opinion):

   - for security it is often important to be able to push fixes fast to
   consumers, snaps are great for that as it somewhat cut's out the
   distributions as a gatekeeper of a release process
   - ntpsec isn't packaged in distributions yet, an upload to the snapstore
   would make you instantly available on multiple distributions
   - faster development iteration cycles, which is especially useful for
   new (or newly forked) projects
   - and of course all the benefits listed at http://snapcraft.io/

Limitations:

   - this doesn't use any of the great snap isolation features yet (still
   using --devmode to get the prototype fast). Implementing those will need a
   few new interfaces and that effort should be spent after the discussion
   (but on the good side, you haven't lost anything - just not gained all of
   the snap isolation features yet).
   - currently there is no snapcraft plugin for waf, so I provided one (but
   I also started to push it to snapcraft already so it can be dropped from
   ntpsec in a bit)

I'm looking forward and hope that the security improvements of ntpsec and
those of snap's for packaging will one day stack up to be even better
together. Let's discuss.

Kind Regards Christian

P.S. FYI - I'm soon going to vaction - so please don't wonder if there is
kind of no-response between 13th and 23rd August. OTOH this gives everyone
more time to play and experiment with it."



-- 
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160809/416f812c/attachment.html>

From esr at thyrsus.com  Tue Aug  9 15:38:55 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Tue, 9 Aug 2016 11:38:55 -0400
Subject: Discussion about PR: WIP: Snapify ntpsec
In-Reply-To: <CAATJJ0JG+9gGgRWU0_m2C0D0OAwRUVK_t=_B5vdUpQ21M8UB9w@mail.gmail.com>
References: <CAATJJ0JG+9gGgRWU0_m2C0D0OAwRUVK_t=_B5vdUpQ21M8UB9w@mail.gmail.com>
Message-ID: <20160809153855.GA30444@thyrsus.com>

Christian Ehrhardt <christian.ehrhardt at canonical.com>:
> I'm looking forward and hope that the security improvements of ntpsec and
> those of snap's for packaging will one day stack up to be even better
> together. Let's discuss.

This looks like good work to me.  I'm encouraging Christian to go forward.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From fallenpegasus at gmail.com  Tue Aug  9 15:58:47 2016
From: fallenpegasus at gmail.com (Mark Atwood)
Date: Tue, 09 Aug 2016 15:58:47 +0000
Subject: Discussion about PR: WIP: Snapify ntpsec
In-Reply-To: <CAATJJ0JG+9gGgRWU0_m2C0D0OAwRUVK_t=_B5vdUpQ21M8UB9w@mail.gmail.com>
References: <CAATJJ0JG+9gGgRWU0_m2C0D0OAwRUVK_t=_B5vdUpQ21M8UB9w@mail.gmail.com>
Message-ID: <CANW5CYWNoK7zd9qzUCvV3kVSYEKVVdz3gx57PK+iC7O9=7-c_w@mail.gmail.com>

This looks great, Christian.

Is there anything we need to do to have our buildbot system test it?

..m

On Tue, Aug 9, 2016 at 8:10 AM Christian Ehrhardt <
christian.ehrhardt at canonical.com> wrote:

> Hi,
> I wanted to give the ML a ping as well about this, so that not only the
> Pull Request is existing.
> Eventually one here might chime in as well.
>
> There is a prototype to snap ntpsec at
> https://gitlab.com/NTPsec/ntpsec/merge_requests/49
>
> I'll quote my PR text here and hope for a great discussion:
>
> "Hi, on one hand I worked on packaging ntp (classic) recently and on the
> other hand I worked a bit with snapcraft (=> http://snapcraft.io/). I
> really think ntpsec would be a perfect candidate to exploit snap packaging.
>
> Please consider this an RFC for now - following the spirit of NTPsec
> contribution policy "Before starting significant work, please propose it
> and discuss it first" I'll also write to the ML linking to this branch. But
> also did I not just want to mention snapcraft and run away - instead I
> thought to provide a prototype that can be tested, but discuss motivation,
> tech and details before doing some more heavy lifting work.
>
> My current example is meant for a daily build, but this can easily be
> changed to whatever you prefer. Snapcraft could - for example - build from
> a stable branch of your tree automatically or whatever else you want.
>
> Benefits of exploiting snap(craft) in ntpsec (in my opinion):
>
>    - for security it is often important to be able to push fixes fast to
>    consumers, snaps are great for that as it somewhat cut's out the
>    distributions as a gatekeeper of a release process
>    - ntpsec isn't packaged in distributions yet, an upload to the
>    snapstore would make you instantly available on multiple distributions
>    - faster development iteration cycles, which is especially useful for
>    new (or newly forked) projects
>    - and of course all the benefits listed at http://snapcraft.io/
>
> Limitations:
>
>    - this doesn't use any of the great snap isolation features yet (still
>    using --devmode to get the prototype fast). Implementing those will need a
>    few new interfaces and that effort should be spent after the discussion
>    (but on the good side, you haven't lost anything - just not gained all of
>    the snap isolation features yet).
>    - currently there is no snapcraft plugin for waf, so I provided one
>    (but I also started to push it to snapcraft already so it can be dropped
>    from ntpsec in a bit)
>
> I'm looking forward and hope that the security improvements of ntpsec and
> those of snap's for packaging will one day stack up to be even better
> together. Let's discuss.
>
> Kind Regards Christian
>
> P.S. FYI - I'm soon going to vaction - so please don't wonder if there is
> kind of no-response between 13th and 23rd August. OTOH this gives everyone
> more time to play and experiment with it."
>
>
>
> --
> Christian Ehrhardt
> Software Engineer, Ubuntu Server
> Canonical Ltd
> _______________________________________________
> devel mailing list
> devel at ntpsec.org
> http://lists.ntpsec.org/mailman/listinfo/devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160809/ed68faa3/attachment.html>

From verm at darkbeer.org  Tue Aug  9 16:02:14 2016
From: verm at darkbeer.org (Amar Takhar)
Date: Tue, 9 Aug 2016 16:02:14 +0000
Subject: Discussion about PR: WIP: Snapify ntpsec
In-Reply-To: <CANW5CYWNoK7zd9qzUCvV3kVSYEKVVdz3gx57PK+iC7O9=7-c_w@mail.gmail.com>
References: <CAATJJ0JG+9gGgRWU0_m2C0D0OAwRUVK_t=_B5vdUpQ21M8UB9w@mail.gmail.com>
 <CANW5CYWNoK7zd9qzUCvV3kVSYEKVVdz3gx57PK+iC7O9=7-c_w@mail.gmail.com>
Message-ID: <20160809160214.GA16897@darkbeer.org>

On 2016-08-09 15:58 +0000, Mark Atwood wrote:
> This looks great, Christian.
> 
> Is there anything we need to do to have our buildbot system test it?

I will take a look at this later on and let you know.


Amar.

From christian.ehrhardt at canonical.com  Wed Aug 10 08:27:11 2016
From: christian.ehrhardt at canonical.com (Christian Ehrhardt)
Date: Wed, 10 Aug 2016 10:27:11 +0200
Subject: Discussion about PR: WIP: Snapify ntpsec
In-Reply-To: <20160809160214.GA16897@darkbeer.org>
References: <CAATJJ0JG+9gGgRWU0_m2C0D0OAwRUVK_t=_B5vdUpQ21M8UB9w@mail.gmail.com>
 <CANW5CYWNoK7zd9qzUCvV3kVSYEKVVdz3gx57PK+iC7O9=7-c_w@mail.gmail.com>
 <20160809160214.GA16897@darkbeer.org>
Message-ID: <CAATJJ0JYUhNv3tx=69iV6c4o8aO3Rserv6-SPbCUBAC_nDg+Gg@mail.gmail.com>

Thank you all so much for your positive feedback and support on this.
I'll try to poke on it again then when back from vacation in about two
weeks.

My personal plan would be:
- get the devmode version rebased and make sure it has no obvious faults
via more tests and discussions
- get the PR accepted into ntpsec
+ here you can start to spin off a buildbot-driven-auto-upload if you want
that
- work step by step on all the pieces needed to make it run in strict
confinement keeping you in the loop (that will be a long term effort)

Kind Regards
Christian


On Tue, Aug 9, 2016 at 6:02 PM, Amar Takhar <verm at darkbeer.org> wrote:

> On 2016-08-09 15:58 +0000, Mark Atwood wrote:
> > This looks great, Christian.
> >
> > Is there anything we need to do to have our buildbot system test it?
>
> I will take a look at this later on and let you know.
>
>
> Amar.
> _______________________________________________
> devel mailing list
> devel at ntpsec.org
> http://lists.ntpsec.org/mailman/listinfo/devel
>



-- 
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160810/3824ff51/attachment.html>

From esr at thyrsus.com  Thu Aug 11 12:11:05 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Thu, 11 Aug 2016 08:11:05 -0400 (EDT)
Subject: Need modification to waf build recipe
Message-ID: <20160811121105.C8C9513A0E7D@snark.thyrsus.com>

This is a heads-up to Amar.

As part of the effort to rewrite ntpq in Python, I have renamed the old
pylib/ directory to wafhelpers/ and created a new pylib/ that contains
an installable Python module.  (The rename is to maintain parallelism
with perllib).

On install, I need the build recipe to copy this module directory to
rootspace where Python programs can import it under the name 'ntp'.  I
looked in the waf book to try to do this myself but have not found a
recipe.

Eventually this will replace perllib as we translate all the old
Perl scripts to Python.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

"Those who make peaceful revolution impossible 
will make violent revolution inevitable."
	-- John F. Kennedy

From verm at darkbeer.org  Thu Aug 11 12:46:46 2016
From: verm at darkbeer.org (Amar Takhar)
Date: Thu, 11 Aug 2016 12:46:46 +0000
Subject: Need modification to waf build recipe
In-Reply-To: <20160811121105.C8C9513A0E7D@snark.thyrsus.com>
References: <20160811121105.C8C9513A0E7D@snark.thyrsus.com>
Message-ID: <20160811124646.GA70530@darkbeer.org>

On 2016-08-11 08:11 -0400, Eric S. Raymond wrote:
> This is a heads-up to Amar.
> 
> As part of the effort to rewrite ntpq in Python, I have renamed the old
> pylib/ directory to wafhelpers/ and created a new pylib/ that contains
> an installable Python module.  (The rename is to maintain parallelism
> with perllib).

wafhelpers is a terrible name since they aren't helpers it's the entire build.  
Please call it 'wafbuild'.  Some projects do use it as re-usable modules but 
that is not how ours is written.


> On install, I need the build recipe to copy this module directory to
> rootspace where Python programs can import it under the name 'ntp'.  I
> looked in the waf book to try to do this myself but have not found a
> recipe.

There is built-in support in waf to do this It's been a long time since I uhsed 
it I will take a look.


> Eventually this will replace perllib as we translate all the old
> Perl scripts to Python.

Great!


Amar.

From esr at thyrsus.com  Thu Aug 11 13:04:28 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Thu, 11 Aug 2016 09:04:28 -0400
Subject: Need modification to waf build recipe
In-Reply-To: <20160811124646.GA70530@darkbeer.org>
References: <20160811121105.C8C9513A0E7D@snark.thyrsus.com>
 <20160811124646.GA70530@darkbeer.org>
Message-ID: <20160811130428.GA2513@thyrsus.com>

Amar Takhar <verm at darkbeer.org>:
> On 2016-08-11 08:11 -0400, Eric S. Raymond wrote:
> > This is a heads-up to Amar.
> > 
> > As part of the effort to rewrite ntpq in Python, I have renamed the old
> > pylib/ directory to wafhelpers/ and created a new pylib/ that contains
> > an installable Python module.  (The rename is to maintain parallelism
> > with perllib).
> 
> wafhelpers is a terrible name since they aren't helpers it's the entire build.

How is it the entire build?  There are lots of wscript files not in there...

> Please call it 'wafbuild'.  Some projects do use it as re-usable modules but 
> that is not how ours is written.

'wafbuild' was in fact my first choice.  Can't use it; there's a collision with
something named 'wafbuild' that waf is using internally.

I don't really care what it's named.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From verm at darkbeer.org  Thu Aug 11 13:26:49 2016
From: verm at darkbeer.org (Amar Takhar)
Date: Thu, 11 Aug 2016 13:26:49 +0000
Subject: Need modification to waf build recipe
In-Reply-To: <20160811130428.GA2513@thyrsus.com>
References: <20160811121105.C8C9513A0E7D@snark.thyrsus.com>
 <20160811124646.GA70530@darkbeer.org>
 <20160811130428.GA2513@thyrsus.com>
Message-ID: <20160811132649.GA71311@darkbeer.org>

On 2016-08-11 09:04 -0400, Eric S. Raymond wrote:
> How is it the entire build?  There are lots of wscript files not in there...

That's the source designation files, all the logic is in the former 'pylib' dir.


> 'wafbuild' was in fact my first choice.  Can't use it; there's a collision with
> something named 'wafbuild' that waf is using internally.
> 
> I don't really care what it's named.


I'll figure something out.  More notice would have been nice.


Amar.

From fallenpegasus at gmail.com  Thu Aug 11 21:28:56 2016
From: fallenpegasus at gmail.com (Mark Atwood)
Date: Thu, 11 Aug 2016 21:28:56 +0000
Subject: Next point release
Message-ID: <CANW5CYWNGKcf9tVYidhouMhgUMzokuOo0uVcz3nL1OY4zKzOiA@mail.gmail.com>

It is time for another point release.

I've received enough private communications from people who are
successfuilly running lab machines on tip, and people who are successfully
running lab machines on the previous point release, and that my work with
the CII is illuminating the need for proof of motion, that I have decided
to make another point release.

I understand the objection that we do not yet have a formalized release
criteria system, nor do we yet have a formalized checklist or automated
release process.  As nobody wise is yet running us in load production, that
is not yet an issue.

This point release will not have Daniel's state machine patch merged in,
because it is a minor risk I don't want to take for the point release just
yet.


Other than that, everyone please chime in:
Daniel
Eric
Amar
Hal
everyone else?

Please get your low risk bug fixes in in the next couple of days.

Unless there is a credible stop reason, I will issue the tag this coming
Tuesday morning.

Thank you everyone!

..m
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160811/f23bbeeb/attachment.html>

From verm at darkbeer.org  Thu Aug 11 21:33:25 2016
From: verm at darkbeer.org (Amar Takhar)
Date: Thu, 11 Aug 2016 21:33:25 +0000
Subject: Next point release
In-Reply-To: <CANW5CYWNGKcf9tVYidhouMhgUMzokuOo0uVcz3nL1OY4zKzOiA@mail.gmail.com>
References: <CANW5CYWNGKcf9tVYidhouMhgUMzokuOo0uVcz3nL1OY4zKzOiA@mail.gmail.com>
Message-ID: <20160811213325.GA81884@darkbeer.org>

On 2016-08-11 21:28 +0000, Mark Atwood wrote:
> It is time for another point release.
> 
> I've received enough private communications from people who are successfuilly
> running lab machines on tip, and people who are successfully running lab
> machines on the previous point release, and that my work with the CII is
> illuminating the need for proof of motion, that I have decided to make another
> point release.

Can you quantify what 'lab machine' means here?  Are we talking scientific 
laboratories running science equipment or general purpose lab workstations?


> I understand the objection that we do not yet have a formalized release
> criteria system, nor do we yet have a formalized checklist or automated release
> process.?? As nobody wise is yet running us in load production, that is not yet
> an issue.

That's not really a reason to now have a formalised process if we're going to do 
it in the future we should do it now.  It lets us have a chance at getting 
organised under it when it comes time for it to be a hard set of rules to follow 
it will be easier.


<snip>
> Please get your low risk bug fixes in in the next couple of days.
> 
> Unless there is a credible stop reason, I will issue the tag this coming
> Tuesday morning.

Tuesday should be fine.  There are some build changes I need to make to handle 
the new Python module I will do that this weekend.


Amar.

From gem at rellim.com  Thu Aug 11 21:34:53 2016
From: gem at rellim.com (Gary E. Miller)
Date: Thu, 11 Aug 2016 14:34:53 -0700
Subject: Next point release
In-Reply-To: <CANW5CYWNGKcf9tVYidhouMhgUMzokuOo0uVcz3nL1OY4zKzOiA@mail.gmail.com>
References: <CANW5CYWNGKcf9tVYidhouMhgUMzokuOo0uVcz3nL1OY4zKzOiA@mail.gmail.com>
Message-ID: <20160811143453.7761bfc0@spidey.rellim.com>

Yo Mark!

On Thu, 11 Aug 2016 21:28:56 +0000
Mark Atwood <fallenpegasus at gmail.com> wrote:

> It is time for another point release.

Not yet, unless something is urgent.  In the middle of August no one
will see it.  It will get lost in the haze of summer.  People are out
and about, or watching the Olympics.

If you wait until most Universities are back in session then all the
script kiddies will jump on it.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160811/d569486c/attachment.bin>

From esr at thyrsus.com  Thu Aug 11 22:28:40 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Thu, 11 Aug 2016 18:28:40 -0400
Subject: Next point release
In-Reply-To: <CANW5CYWNGKcf9tVYidhouMhgUMzokuOo0uVcz3nL1OY4zKzOiA@mail.gmail.com>
References: <CANW5CYWNGKcf9tVYidhouMhgUMzokuOo0uVcz3nL1OY4zKzOiA@mail.gmail.com>
Message-ID: <20160811222840.GB12690@thyrsus.com>

Mark Atwood <fallenpegasus at gmail.com>:
> It is time for another point release.

No blockers.  I was expecting this and have been cleaning up various
minor things on the issues list.

> This point release will not have Daniel's state machine patch merged in,
> because it is a minor risk I don't want to take for the point release just
> yet.

Good plan. I have every confidence in Daniel, but rushing the new code
into production would be a needless risk.  I'd like to have it
simmering on the test farm machines for a month or so before we ship
it.

> Please get your low risk bug fixes in in the next couple of days.

I should have ntpviz ready to ship this weekend, so we'll have a nice
tasty feature to brag about in the release notes.

> Unless there is a credible stop reason, I will issue the tag this coming
> Tuesday morning.

Hal, I think that tells us when we dive into TESTFRAME - right after that.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From hmurray at megapathdsl.net  Thu Aug 11 23:05:36 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Thu, 11 Aug 2016 16:05:36 -0700
Subject: Using first server to respond, Issue #68
Message-ID: <20160811230536.E60D1406076@ip-64-139-1-69.sjc.megapath.net>


I think I have found the problem.

minsane defaults to 1 so ntpd is "happy" as soon as a server gets past the 
individual server filtering.

I don't see any simple fix.  Even if minsane is set (via ntp.conf) to 
something larger, there is still the possibility that a majority of the first 
N servers are not the ones you want.

We might be able to work out something based on how many servers you are 
using, but that gets tangled up with how many servers to use and why.  (If 
you haven't been around for a while, that's an endless topic.)

We could add a warning.  Does anybody look at them?  It won't break anything, 
so I guess I'll give it a try.


-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Fri Aug 12 03:10:22 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Thu, 11 Aug 2016 20:10:22 -0700
Subject: Next point release
Message-ID: <20160812031022.C662B406076@ip-64-139-1-69.sjc.megapath.net>


I don't know of any problems that need fixing.


fallenpegasus at gmail.com said:
> I understand the objection that we do not yet have a formalized release
> criteria system, nor do we yet have a formalized checklist or automated
> release process.  As nobody wise is yet running us in load production, that
> is not yet an issue. 

Please take notes on what you do for the release.  If you send them to me, 
I'll try to turn them into some sort of document.

I'm not looking for anything ultra-detailed, just enough reminders that a 
step won't fall through the cracks.  Chunks that can be copy/pasted would be 
good.  If the purpose isn't obvious, then I'd like a few words to describe 
why/what.


-- 
These are my opinions.  I hate spam.




From fallenpegasus at gmail.com  Fri Aug 12 15:10:13 2016
From: fallenpegasus at gmail.com (Mark Atwood)
Date: Fri, 12 Aug 2016 15:10:13 +0000
Subject: Using first server to respond, Issue #68
In-Reply-To: <20160811230536.E60D1406076@ip-64-139-1-69.sjc.megapath.net>
References: <20160811230536.E60D1406076@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <CANW5CYUFJqaHEvNN31AtN7DyfB3JA60bupJbfR-L0-J8M50QcQ@mail.gmail.com>

Start with the warning, while we think of a solution.

Thank you, Hal.

On Thu, Aug 11, 2016 at 4:05 PM Hal Murray <hmurray at megapathdsl.net> wrote:

>
> I think I have found the problem.
>
> minsane defaults to 1 so ntpd is "happy" as soon as a server gets past the
> individual server filtering.
>
> I don't see any simple fix.  Even if minsane is set (via ntp.conf) to
> something larger, there is still the possibility that a majority of the
> first
> N servers are not the ones you want.
>
> We might be able to work out something based on how many servers you are
> using, but that gets tangled up with how many servers to use and why.  (If
> you haven't been around for a while, that's an endless topic.)
>
> We could add a warning.  Does anybody look at them?  It won't break
> anything,
> so I guess I'll give it a try.
>
>
> --
> These are my opinions.  I hate spam.
>
>
>
> _______________________________________________
> devel mailing list
> devel at ntpsec.org
> http://lists.ntpsec.org/mailman/listinfo/devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160812/dbde7bd2/attachment.html>

From hmurray at megapathdsl.net  Sun Aug 14 18:07:47 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 14 Aug 2016 11:07:47 -0700
Subject: Why did you remove that "unnecessary" link?
In-Reply-To: Message from esr@thyrsus.com (Eric S. Raymond) of "Sun,
 14 Aug 2016 13:04:47 EDT." <20160814170447.754FE13A0E7D@snark.thyrsus.com>
Message-ID: <20160814180747.A206B406078@ip-64-139-1-69.sjc.megapath.net>


esr at thyrsus.com said:
> You broke my test setup.  The purpose of that link was to allow the
> neighboring pyntpq script to import ntp.packet even when the corresponding
> Python module under pylib hasn't been installed in rootspace.  Which it
> never is, yet - the waf production to do that hasn't landed.

> I guess I'll add a README so this doesn't happen again. 

Sorry.  I thought I was helping by removing some cruft.  It built and tested 
without that link.

Can you add a test that will catch it?

That opens up the whole mess of testing with local libraries vs installed 
libraries.  I remember lots of "fun" in that area for gpsd.


-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Sun Aug 14 18:22:16 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sun, 14 Aug 2016 14:22:16 -0400
Subject: Why did you remove that "unnecessary" link?
In-Reply-To: <20160814180747.A206B406078@ip-64-139-1-69.sjc.megapath.net>
References: <20160814170447.754FE13A0E7D@snark.thyrsus.com>
 <20160814180747.A206B406078@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160814182216.GA9614@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> Can you add a test that will catch it?

Yes, maybe a pretty trivial one.  But we don't really have the larger
check framework for it to live inside yet; that gets into some waf issues
rgat I need to chase down in my copious free time.

> That opens up the whole mess of testing with local libraries vs installed 
> libraries.  I remember lots of "fun" in that area for gpsd.

You misspelled "I'd rather have root canal than go through that
again."  Yeesh...  On NTPsec, I'm planning to stay strictly away from
building shared libraries and Python extensions written in C for that
exact reason.  They were a fun stunt in GPSD but not worth the
downstream complexity cost.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From hmurray at megapathdsl.net  Sun Aug 14 22:49:15 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 14 Aug 2016 15:49:15 -0700
Subject: Possible abuse from fetching the leap second file
Message-ID: <20160814224915.BC72A406070@ip-64-139-1-69.sjc.megapath.net>

Matt Selsky is working on Pythonizing the script that grabs a new leap second 
file.  The idea is to run a cron job that keeps it up to date.  That opens an 
interesting can of worms.

As a general rule, you shouldn't use a resource on a system that you don't 
own without permission from the owner.  Informed consent might be a better 
term.  A system open to occasional  downloads by a human might not be willing 
to support automated fetches from many many systems.

This case is doubly nasty in two ways.

First, the load will normally be light but then go up sharply 60 days before 
the file expires.  (The doc mentions a crontab, but I can't find specifics.)  
That could easily turn into a DDoS.

Second, the URL from NIST is unreliable[1] and the IEFT clone is out of date. 
 It's not obvious that NIST is expecting to support non US clients or that 
either NIST or IEFT is prepared to support high volumes of automated fetches.

The clean solution is for us to provide the server(s), or at least the DNS so 
we can provide the servers tomorrow.  That commits us to long term support, 
but since we have control of everything we can fix it if something goes wrong.

Does anybody know how many downloads/hour a cloud server can suppor?  I'm 
interested in this simple case, just downloading a small file, no fancy 
database processing.  Are there special web server packages designed for this 
case?

How many clients are we expecting to run this code?

Another approach might be to get the time zone people to distribute the leap 
second file too.  That seems to get updated often enough.

---------

1] The current URL is ftp://time.nist.gov/pub/leap-seconds.list
DNS for time.nist.gov is setup for time, not ftp.  It rotates through all 
their public NTP servers and many of them don't support ftp.


Matt:  The current code has an option to restart ntpd.  The current ntpd will 
check for a new leap file on SIGHUP but that will kill ntp-classic.

Please see if you can find a simple way to spread the load.  We can reduce 
the load on the servers by a factor of 30 if you can spread that out over a 
month.


-- 
These are my opinions.  I hate spam.




From dan-ntp at drown.org  Mon Aug 15 00:15:32 2016
From: dan-ntp at drown.org (Dan Drown)
Date: Sun, 14 Aug 2016 19:15:32 -0500
Subject: Possible abuse from fetching the leap second file
Message-ID: <20160814191532.Horde.4LVSOxSUn-r6y6wqTb9LOCq@mail.drown.org>


Quoting Hal Murray <hmurray at megapathdsl.net>:
> Matt Selsky is working on Pythonizing the script that grabs a new leap second
> file.  The idea is to run a cron job that keeps it up to date.  That opens an
> interesting can of worms.
>
> As a general rule, you shouldn't use a resource on a system that you don't
> own without permission from the owner.  Informed consent might be a better
> term.  A system open to occasional  downloads by a human might not be willing
> to support automated fetches from many many systems.
>
> This case is doubly nasty in two ways.
>
> First, the load will normally be light but then go up sharply 60 days before
> the file expires.  (The doc mentions a crontab, but I can't find specifics.)
> That could easily turn into a DDoS.

I agree that it's impolite to automate this.  What's ok for 100  
servers to do isn't ok for 1 million.

> Second, the URL from NIST is unreliable[1] and the IEFT clone is out of date.
> It's not obvious that NIST is expecting to support non US clients or that
> either NIST or IEFT is prepared to support high volumes of automated fetches.
>
> The clean solution is for us to provide the server(s), or at least the DNS so
> we can provide the servers tomorrow.  That commits us to long term support,
> but since we have control of everything we can fix it if something  
> goes wrong.
>
> Does anybody know how many downloads/hour a cloud server can suppor?  I'm
> interested in this simple case, just downloading a small file, no fancy
> database processing.  Are there special web server packages designed for this
> case?

There are a few webservers designed for high connection count static  
file serving - lighttpd, nginx are two examples

I'd guess downloads/hour would be mainly limited on the packets per  
second side of things (especially on a cloud server, which are usually  
bad at high packets per second rates).

Starting with 100k packets per second and 21 packets to complete a  
http GET for the leapsecond file.  This gives a rate of 4,761 requests  
per second completed (and 409Mbit/s rate).  After an hour, that's 17  
million requests completed (182.4 Gbyte out, ~$16 in EC2).

Looking at it in a different way, let's take a theoretical cloud  
server that includes 4TB/month transfer.  That plan would cover around  
372 million requests for the leapsecond file over a month (at an  
average rate of around 143 requests/second).

This is also a thing that would be easy to mirror.  You'd want to  
distribute a gpg external signature with the file (updated every 6  
months?), so end users could be confident the leapsecond file wasn't  
messed with by a mirror.

All those numbers were with HTTP overhead, HTTPS overhead reduces  
these numbers by around 33%.

> How many clients are we expecting to run this code?
>
> Another approach might be to get the time zone people to distribute the leap
> second file too.  That seems to get updated often enough.

I'm using chrony's feature to read the leapsecond from the timezone files:
https://chrony.tuxfamily.org/manual.html#leapsectz-directive

I like this because the leapsecond updates come with regular OS  
updates.  Doesn't look like Ubuntu or Fedora have the Dec 31, 2016  
leap second yet, though.

[2015]
$ TZ=right/UTC date -d 'Jun 30 2015 23:59:60'
Tue Jun 30 23:59:60 UTC 2015
[2016]
$ TZ=right/UTC date -d 'Dec 31 2016 23:59:60'
date: invalid date ?Dec 31 2016 23:59:60?


> 1] The current URL is ftp://time.nist.gov/pub/leap-seconds.list
> DNS for time.nist.gov is setup for time, not ftp.  It rotates through all
> their public NTP servers and many of them don't support ftp.
>
>
> Matt:  The current code has an option to restart ntpd.  The current ntpd will
> check for a new leap file on SIGHUP but that will kill ntp-classic.
>
> Please see if you can find a simple way to spread the load.  We can reduce
> the load on the servers by a factor of 30 if you can spread that out over a
> month.



From esr at thyrsus.com  Mon Aug 15 01:28:01 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sun, 14 Aug 2016 21:28:01 -0400
Subject: Possible abuse from fetching the leap second file
In-Reply-To: <20160814224915.BC72A406070@ip-64-139-1-69.sjc.megapath.net>
References: <20160814224915.BC72A406070@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160815012801.GA16465@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> Matt Selsky is working on Pythonizing the script that grabs a new leap second 
> file.  The idea is to run a cron job that keeps it up to date.  That opens an 
> interesting can of worms.
> 
> As a general rule, you shouldn't use a resource on a system that you don't 
> own without permission from the owner.  Informed consent might be a better 
> term.  A system open to occasional  downloads by a human might not be willing 
> to support automated fetches from many many systems.

While I accept this as a general principle, is there anything about the
new ntpleapfetch that inflicts a heavier load than the old ntpleapfetch
has been causing for decades with the tolerance of NIST and USNO?

If not, then I think we get to mutter "customary usage" and move on.

I will also note that the GPSD build process has actually been doing
something very like ntpleapfetch (to get the current leap-second so
it can be compiled into the build) for about a decade.  I didn't see
it as a potential problem when I wrote it, and nobody associated with
the targeted servers has ever complained to me.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From hmurray at megapathdsl.net  Mon Aug 15 06:02:08 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 14 Aug 2016 23:02:08 -0700
Subject: Possible abuse from fetching the leap second file
In-Reply-To: Message from "Eric S. Raymond" <esr@thyrsus.com>
 of "Sun, 14 Aug 2016 21:28:01 EDT." <20160815012801.GA16465@thyrsus.com>
Message-ID: <20160815060208.3D77A406070@ip-64-139-1-69.sjc.megapath.net>


esr at thyrsus.com said:
> While I accept this as a general principle, is there anything about the new
> ntpleapfetch that inflicts a heavier load than the old ntpleapfetch has been
> causing for decades with the tolerance of NIST and USNO? 

The old stuff has poor publicity.  None of the major distros/OSes come setup 
to run it from a cron job.  As long as you don't change that we won't have 
any problems.

The problem will happen if somebody improves our documentation enough so that 
somebody notices, and that seems reasonably likely.




> I will also note that the GPSD build process has actually been doing
> something very like ntpleapfetch (to get the current leap-second so it can
> be compiled into the build) for about a decade.  I didn't see it as a
> potential problem when I wrote it, and nobody associated with the targeted
> servers has ever complained to me. 

Two things: 

One, that only happens when building from source.  All the systems running 
gpsd as provided by their distro aren't doing that.

Two, it's not synchronized so that a zillion systems all try to do it 60 days 
before the file expires.


-- 
These are my opinions.  I hate spam.




From kurt at roeckx.be  Mon Aug 15 09:29:28 2016
From: kurt at roeckx.be (Kurt Roeckx)
Date: Mon, 15 Aug 2016 11:29:28 +0200
Subject: Possible abuse from fetching the leap second file
In-Reply-To: <20160814224915.BC72A406070@ip-64-139-1-69.sjc.megapath.net>
References: <20160814224915.BC72A406070@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160815092928.GA1814@roeckx.be>

On Sun, Aug 14, 2016 at 03:49:15PM -0700, Hal Murray wrote:
> Matt Selsky is working on Pythonizing the script that grabs a new leap second 
> file.  The idea is to run a cron job that keeps it up to date.  That opens an 
> interesting can of worms.

I've been pondering about using ntp-classic's script for that in
Debian, but decided not to.  I also don't know if there is any
other Linux distribution that automates this.

However, I already get a /usr/share/zoneinfo/leap-seconds.list
from the tzdata package that updates the timezone information and
I should probably look into using that file to update things.
Those files get regular updates.  The tz at iana.org list has been
notified, the git repository updated, there just hans't been a
release since.

There has been a proposal by Poul-Henning Kamp to use an DNS
A-record for it, see:
http://phk.freebsd.dk/time/20151122.html

I don't know if this went anywhere.


Kurt


From esr at thyrsus.com  Mon Aug 15 12:37:14 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Mon, 15 Aug 2016 08:37:14 -0400
Subject: Possible abuse from fetching the leap second file
In-Reply-To: <20160815060208.3D77A406070@ip-64-139-1-69.sjc.megapath.net>
References: <esr@thyrsus.com> <20160815012801.GA16465@thyrsus.com>
 <20160815060208.3D77A406070@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160815123714.GA11020@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> 
> esr at thyrsus.com said:
> > While I accept this as a general principle, is there anything about the new
> > ntpleapfetch that inflicts a heavier load than the old ntpleapfetch has been
> > causing for decades with the tolerance of NIST and USNO? 
> 
> The old stuff has poor publicity.  None of the major distros/OSes come setup 
> to run it from a cron job.  As long as you don't change that we won't have 
> any problems.
> 
> The problem will happen if somebody improves our documentation enough so that 
> somebody notices, and that seems reasonably likely.

I've thought about this some more, and now I am in doubt that the
general principle (don't use other peoples' resources without their
permission) applies here.  I think we need to apply what tort law
would call a reasonable-person test.

Some kinds of public-facing offer of a resource clearly constitute an
implied invitation to download it as needed.  Consider, for example, a web page.

I think the NIST/IERS public offer of an authoritative leap-second resource
constitutes the same sort of invitation.  If you disagree, ask yourself
if your evaluation would change if that data were in HTML and accessed
through port 80, or accessed by anonymous FTP. Surely the mechanics
of how it's downloaded are irrelevant to the ethics of the situation!

That said, I think we do have a duty in this case, which is to implement some
load-spreading so that the process doesn't hit those servers harder than it has
to.  A random delay on the fetch would be polite.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From kurt at roeckx.be  Mon Aug 15 13:11:07 2016
From: kurt at roeckx.be (Kurt Roeckx)
Date: Mon, 15 Aug 2016 15:11:07 +0200
Subject: Possible abuse from fetching the leap second file
In-Reply-To: <20160815123714.GA11020@thyrsus.com>
References: <esr@thyrsus.com> <20160815012801.GA16465@thyrsus.com>
 <20160815060208.3D77A406070@ip-64-139-1-69.sjc.megapath.net>
 <20160815123714.GA11020@thyrsus.com>
Message-ID: <20160815131107.vsj5ozlh3dlb5ip4@roeckx.be>

On Mon, Aug 15, 2016 at 08:37:14AM -0400, Eric S. Raymond wrote:
> Hal Murray <hmurray at megapathdsl.net>:
> > 
> > esr at thyrsus.com said:
> > > While I accept this as a general principle, is there anything about the new
> > > ntpleapfetch that inflicts a heavier load than the old ntpleapfetch has been
> > > causing for decades with the tolerance of NIST and USNO? 
> > 
> > The old stuff has poor publicity.  None of the major distros/OSes come setup 
> > to run it from a cron job.  As long as you don't change that we won't have 
> > any problems.
> > 
> > The problem will happen if somebody improves our documentation enough so that 
> > somebody notices, and that seems reasonably likely.
> 
> I've thought about this some more, and now I am in doubt that the
> general principle (don't use other peoples' resources without their
> permission) applies here.  I think we need to apply what tort law
> would call a reasonable-person test.
> 
> Some kinds of public-facing offer of a resource clearly constitute an
> implied invitation to download it as needed.  Consider, for example, a web page.
> 
> I think the NIST/IERS public offer of an authoritative leap-second resource
> constitutes the same sort of invitation.  If you disagree, ask yourself
> if your evaluation would change if that data were in HTML and accessed
> through port 80, or accessed by anonymous FTP. Surely the mechanics
> of how it's downloaded are irrelevant to the ethics of the situation!

That doesn't mean you should go and have millions of clients
change from not checking it to downloading it without at least
warning them about it.  It doesn't help anybody if we overload the
servers.


Kurt


From esr at thyrsus.com  Mon Aug 15 14:29:25 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Mon, 15 Aug 2016 10:29:25 -0400
Subject: Possible abuse from fetching the leap second file
In-Reply-To: <20160815131107.vsj5ozlh3dlb5ip4@roeckx.be>
References: <esr@thyrsus.com> <20160815012801.GA16465@thyrsus.com>
 <20160815060208.3D77A406070@ip-64-139-1-69.sjc.megapath.net>
 <20160815123714.GA11020@thyrsus.com>
 <20160815131107.vsj5ozlh3dlb5ip4@roeckx.be>
Message-ID: <20160815142925.GA13318@thyrsus.com>

Kurt Roeckx <kurt at roeckx.be>:
> That doesn't mean you should go and have millions of clients
> change from not checking it to downloading it without at least
> warning them about it.  It doesn't help anybody if we overload the
> servers.

Fair enough.  Can we identify the responsible persons to communicate with?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From fallenpegasus at gmail.com  Mon Aug 15 14:58:07 2016
From: fallenpegasus at gmail.com (Mark Atwood)
Date: Mon, 15 Aug 2016 14:58:07 +0000
Subject: Possible abuse from fetching the leap second file
In-Reply-To: <20160814224915.BC72A406070@ip-64-139-1-69.sjc.megapath.net>
References: <20160814224915.BC72A406070@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <CANW5CYXnqC_mjdyo-1-ydZ6NtZevG3iURPZJH7UL4LA62ijjrw@mail.gmail.com>

The long term, I like the DNS for solutions to this kind of problem.  But,
under what name?

Other solutions are putting it in AWS & Cloudfront, and in their
equivalents at AZR and at GCS.  To take that route, I would want to arrange
that Amazon, Microsoft, and Google donate that capacity.   The those 3
cloud CDNs could handle that load.  But, that will take negotation time,
and programming time we don't have right now.

An even faster to implement solution would be to put it in github.com.   We
could do that today, and it would cost us nothing, and github on their
backend smoothly pours very high demand raw pages into the the assorted
worldwide cloud providers and into the CDNs. Plus it versions the data, and
they have wellknown TLS certs.

Let's do that!  Hal, others, do you happen to have copies of all the past
leap files, so we can synthesize a git history for it?

..m



On Sun, Aug 14, 2016 at 3:49 PM Hal Murray <hmurray at megapathdsl.net> wrote:

> Matt Selsky is working on Pythonizing the script that grabs a new leap
> second
> file.  The idea is to run a cron job that keeps it up to date.  That opens
> an
> interesting can of worms.
>
> As a general rule, you shouldn't use a resource on a system that you don't
> own without permission from the owner.  Informed consent might be a better
> term.  A system open to occasional  downloads by a human might not be
> willing
> to support automated fetches from many many systems.
>
> This case is doubly nasty in two ways.
>
> First, the load will normally be light but then go up sharply 60 days
> before
> the file expires.  (The doc mentions a crontab, but I can't find
> specifics.)
> That could easily turn into a DDoS.
>
> Second, the URL from NIST is unreliable[1] and the IEFT clone is out of
> date.
>  It's not obvious that NIST is expecting to support non US clients or that
> either NIST or IEFT is prepared to support high volumes of automated
> fetches.
>
> The clean solution is for us to provide the server(s), or at least the DNS
> so
> we can provide the servers tomorrow.  That commits us to long term support,
> but since we have control of everything we can fix it if something goes
> wrong.
>
> Does anybody know how many downloads/hour a cloud server can suppor?  I'm
> interested in this simple case, just downloading a small file, no fancy
> database processing.  Are there special web server packages designed for
> this
> case?
>
> How many clients are we expecting to run this code?
>
> Another approach might be to get the time zone people to distribute the
> leap
> second file too.  That seems to get updated often enough.
>
> ---------
>
> 1] The current URL is ftp://time.nist.gov/pub/leap-seconds.list
> DNS for time.nist.gov is setup for time, not ftp.  It rotates through all
> their public NTP servers and many of them don't support ftp.
>
>
> Matt:  The current code has an option to restart ntpd.  The current ntpd
> will
> check for a new leap file on SIGHUP but that will kill ntp-classic.
>
> Please see if you can find a simple way to spread the load.  We can reduce
> the load on the servers by a factor of 30 if you can spread that out over a
> month.
>
>
> --
> These are my opinions.  I hate spam.
>
>
>
> _______________________________________________
> devel mailing list
> devel at ntpsec.org
> http://lists.ntpsec.org/mailman/listinfo/devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160815/0d89fe21/attachment.html>

From esr at thyrsus.com  Mon Aug 15 15:14:10 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Mon, 15 Aug 2016 11:14:10 -0400
Subject: Possible abuse from fetching the leap second file
In-Reply-To: <CANW5CYXnqC_mjdyo-1-ydZ6NtZevG3iURPZJH7UL4LA62ijjrw@mail.gmail.com>
References: <20160814224915.BC72A406070@ip-64-139-1-69.sjc.megapath.net>
 <CANW5CYXnqC_mjdyo-1-ydZ6NtZevG3iURPZJH7UL4LA62ijjrw@mail.gmail.com>
Message-ID: <20160815151410.GA14218@thyrsus.com>

Mark Atwood <fallenpegasus at gmail.com>:
> Let's do that!  Hal, others, do you happen to have copies of all the past
> leap files, so we can synthesize a git history for it?

We don't need all the past leap files. The data is only ever modified by
appending a line.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From kurt at roeckx.be  Mon Aug 15 15:44:36 2016
From: kurt at roeckx.be (Kurt Roeckx)
Date: Mon, 15 Aug 2016 17:44:36 +0200
Subject: Possible abuse from fetching the leap second file
In-Reply-To: <20160815151410.GA14218@thyrsus.com>
References: <20160814224915.BC72A406070@ip-64-139-1-69.sjc.megapath.net>
 <CANW5CYXnqC_mjdyo-1-ydZ6NtZevG3iURPZJH7UL4LA62ijjrw@mail.gmail.com>
 <20160815151410.GA14218@thyrsus.com>
Message-ID: <20160815154436.jvsgmgdzbljkqict@roeckx.be>

On Mon, Aug 15, 2016 at 11:14:10AM -0400, Eric S. Raymond wrote:
> Mark Atwood <fallenpegasus at gmail.com>:
> > Let's do that!  Hal, others, do you happen to have copies of all the past
> > leap files, so we can synthesize a git history for it?
> 
> We don't need all the past leap files. The data is only ever modified by
> appending a line.

There is also at least a timestamp until when it's valid updated.
It also contains a hash sum.  They have actually changed the text
over the years.

You can at least get a short history at:
https://github.com/eggert/tz/commits/master/leap-seconds.list

If you want to contact the NIST people, there is actually a
contact address in the file.


Kurt


From fallenpegasus at gmail.com  Mon Aug 15 17:17:06 2016
From: fallenpegasus at gmail.com (Mark Atwood)
Date: Mon, 15 Aug 2016 17:17:06 +0000
Subject: Possible abuse from fetching the leap second file
In-Reply-To: <20160815151410.GA14218@thyrsus.com>
References: <20160814224915.BC72A406070@ip-64-139-1-69.sjc.megapath.net>
 <CANW5CYXnqC_mjdyo-1-ydZ6NtZevG3iURPZJH7UL4LA62ijjrw@mail.gmail.com>
 <20160815151410.GA14218@thyrsus.com>
Message-ID: <CANW5CYX36xBrVxExOEfM=1cEQGT3ffjpKo892x2JRDXCGQDoJA@mail.gmail.com>

On Mon, Aug 15, 2016 at 8:14 AM Eric S. Raymond <esr at thyrsus.com> wrote:

> We don't need all the past leap files. The data is only ever modified by
> appending a line.
>
>
I'm a crazy competitionist.  Git resources need histories.

OTOH, it looks like the eggert/tz repo already exists. On the other other
hand, it doesn't look like it's got the historical history.

..m
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160815/dc915a05/attachment.html>

From hmurray at megapathdsl.net  Mon Aug 15 20:26:31 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Mon, 15 Aug 2016 13:26:31 -0700
Subject: Possible abuse from fetching the leap second file
In-Reply-To: Message from Kurt Roeckx <kurt@roeckx.be>
 of "Mon, 15 Aug 2016 11:29:28 +0200." <20160815092928.GA1814@roeckx.be>
Message-ID: <20160815202631.3F337406077@ip-64-139-1-69.sjc.megapath.net>


kurt at roeckx.be said:
> However, I already get a /usr/share/zoneinfo/leap-seconds.list from the
> tzdata package that updates the timezone information and I should probably
> look into using that file to update things. Those files get regular updates.
>  The tz at iana.org list has been notified, the git repository updated, there
> just hans't been a release since. 

Thanks.  I didn't know it was getting distributed.  That looks like the way 
to go.

You don't need to "update things".  Just point ntp.conf at that file.

I don't see that file in Fedora yet.  Time to see if I can help get it there.


-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Mon Aug 15 20:40:31 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Mon, 15 Aug 2016 13:40:31 -0700
Subject: Possible abuse from fetching the leap second file
In-Reply-To: Message from "Eric S. Raymond" <esr@thyrsus.com>
 of "Mon, 15 Aug 2016 08:37:14 EDT." <20160815123714.GA11020@thyrsus.com>
Message-ID: <20160815204032.03169406077@ip-64-139-1-69.sjc.megapath.net>


esr at thyrsus.com said:
> I've thought about this some more, and now I am in doubt that the general
> principle (don't use other peoples' resources without their permission)
> applies here.  I think we need to apply what tort law would call a
> reasonable-person test.

> Some kinds of public-facing offer of a resource clearly constitute an
> implied invitation to download it as needed.  Consider, for example, a web
> page.

I think you are just plain wrong.  The invitation to download a web page has 
some expectations of reasonable use.  What's reasonable for an individual 
trying to answer a question using google and following links is no longer 
reasonable if it gets automated so that every box running ntp tries to 
download a file on the same day.

I'm not a lawyer etc.  Consider a public drinking fountain.  That's not an 
invitation to fill up your tanker truck to water your lawn.

Besides, as Kurt Roeckx has pointed out, there is a much much better way.  
Just use the copy that gets distributed by the time zone package.  It's 
already there on Debian and Ubuntu and Raspberian.




-- 
These are my opinions.  I hate spam.




From Matthew.Selsky at twosigma.com  Mon Aug 15 20:50:22 2016
From: Matthew.Selsky at twosigma.com (Matthew Selsky)
Date: Mon, 15 Aug 2016 16:50:22 -0400
Subject: Possible abuse from fetching the leap second file
In-Reply-To: <20160815204032.03169406077@ip-64-139-1-69.sjc.megapath.net>
References: <esr@thyrsus.com> <20160815123714.GA11020@thyrsus.com>
 <20160815204032.03169406077@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160815205022.GR25969@twosigma.com>

On Mon, Aug 15, 2016 at 01:40:31PM -0700, Hal Murray wrote:

> Besides, as Kurt Roeckx has pointed out, there is a much much better way.  
> Just use the copy that gets distributed by the time zone package.  It's 
> already there on Debian and Ubuntu and Raspberian.

This file is only on some distributions:
Jessie has the file: https://packages.debian.org/jessie/all/tzdata/filelist
Wheezy does not have the file: https://packages.debian.org/wheezy/all/tzdata/filelist


Cheers,
-Matt

From kurt at roeckx.be  Mon Aug 15 20:59:16 2016
From: kurt at roeckx.be (Kurt Roeckx)
Date: Mon, 15 Aug 2016 22:59:16 +0200
Subject: Possible abuse from fetching the leap second file
In-Reply-To: <20160815202631.3F337406077@ip-64-139-1-69.sjc.megapath.net>
References: <kurt@roeckx.be> <20160815092928.GA1814@roeckx.be>
 <20160815202631.3F337406077@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160815205915.kgldgucwzpciryvs@roeckx.be>

On Mon, Aug 15, 2016 at 01:26:31PM -0700, Hal Murray wrote:
> 
> kurt at roeckx.be said:
> > However, I already get a /usr/share/zoneinfo/leap-seconds.list from the
> > tzdata package that updates the timezone information and I should probably
> > look into using that file to update things. Those files get regular updates.
> >  The tz at iana.org list has been notified, the git repository updated, there
> > just hans't been a release since. 
> 
> Thanks.  I didn't know it was getting distributed.  That looks like the way 
> to go.
> 
> You don't need to "update things".  Just point ntp.conf at that file.

With update things I was just thinking that I would need to add a
trigger when tzdata is update so I can either reload the config file
or restart ntpd, depending on what's needed to get it to check the
file.

In any case, I will never use a cron job in Debian for this, I
will use the one provided by tzdata.


Kurt


From hmurray at megapathdsl.net  Mon Aug 15 21:03:23 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Mon, 15 Aug 2016 14:03:23 -0700
Subject: Possible abuse from fetching the leap second file
In-Reply-To: Message from Mark Atwood <fallenpegasus@gmail.com> of "Mon,
 15 Aug 2016 14:58:07 -0000."
 <CANW5CYXnqC_mjdyo-1-ydZ6NtZevG3iURPZJH7UL4LA62ijjrw@mail.gmail.com>
Message-ID: <20160815210323.785EF406077@ip-64-139-1-69.sjc.megapath.net>


fallenpegasus at gmail.com said:
> The long term, I like the DNS for solutions to this kind of problem.  But,
> under what name?

I think piggybacking on the time zone database is a better solution.

There are 2 parts to the problem.  One is when the next leap second is 
scheduled.  The other is the whole chain of leap seconds.  The kernel needs 
the first which what ntpd tells it.  The second is needed by time conversion 
routines.

The current DNS implementation is a demo.  If you want to use it for real, 
you will have to setup an infrastructure to run highly reliable servers that 
are likely to be targets for script kiddies.


fallenpegasus at gmail.com said:
> Let's do that!  Hal, others, do you happen to have copies of all the past
> leap files, so we can synthesize a git history for it? 

I have some, not all.  There is a good chance you could get them all from 
NIST.


-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Mon Aug 15 21:09:07 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Mon, 15 Aug 2016 14:09:07 -0700
Subject: Possible abuse from fetching the leap second file
In-Reply-To: Message from Kurt Roeckx <kurt@roeckx.be> of "Mon,
 15 Aug 2016 22:59:16 +0200." <20160815205915.kgldgucwzpciryvs@roeckx.be>
Message-ID: <20160815210908.0687C406077@ip-64-139-1-69.sjc.megapath.net>


kurt at roeckx.be said:
> You don't need to "update things".  Just point ntp.conf at that file.
> With update things I was just thinking that I would need to add a trigger
> when tzdata is update so I can either reload the config file or restart
> ntpd, depending on what's needed to get it to check the file.

If you send SIGHUP to ntpd it will reload a new file.  You can see that in syslog or the ntpd log file.

If you do nothing, ntpd checks once a day so it will pick up the new version within 24 hours.

SIGHUP will kill ntp classic so doing nothing seems like the conservative approach.  I think it's good enough.


-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Mon Aug 15 21:17:48 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Mon, 15 Aug 2016 17:17:48 -0400
Subject: Possible abuse from fetching the leap second file
In-Reply-To: <20160815210323.785EF406077@ip-64-139-1-69.sjc.megapath.net>
References: <fallenpegasus@gmail.com>
 <CANW5CYXnqC_mjdyo-1-ydZ6NtZevG3iURPZJH7UL4LA62ijjrw@mail.gmail.com>
 <20160815210323.785EF406077@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160815211748.GC19860@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> 
> fallenpegasus at gmail.com said:
> > The long term, I like the DNS for solutions to this kind of problem.  But,
> > under what name?
> 
> I think piggybacking on the time zone database is a better solution.

I concur.  Anything that touches DNS administration tends to get messy.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From hmurray at megapathdsl.net  Mon Aug 15 21:20:23 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Mon, 15 Aug 2016 14:20:23 -0700
Subject: Possible abuse from fetching the leap second file
In-Reply-To: Message from Matthew Selsky <Matthew.Selsky@twosigma.com>
 of "Mon, 15 Aug 2016 16:50:22 EDT." <20160815205022.GR25969@twosigma.com>
Message-ID: <20160815212023.EAC37406077@ip-64-139-1-69.sjc.megapath.net>


Matthew.Selsky at twosigma.com said:
> This file is only on some distributions: Jessie has the file: https://
> packages.debian.org/jessie/all/tzdata/filelist Wheezy does not have the
> file: https://packages.debian.org/wheezy/all/tzdata/filelist

I was assuming that it would be good enough if the file was available on the 
latest supported release.  That's assuming that future releases will continue 
to support it.

The current code works without a leap file.  It uses a local refclock or a 
majority vote of the servers it is using.

-- 
These are my opinions.  I hate spam.




From gem at rellim.com  Mon Aug 15 21:41:26 2016
From: gem at rellim.com (Gary E. Miller)
Date: Mon, 15 Aug 2016 14:41:26 -0700
Subject: Possible abuse from fetching the leap second file
In-Reply-To: <20160815210908.0687C406077@ip-64-139-1-69.sjc.megapath.net>
References: <20160815205915.kgldgucwzpciryvs@roeckx.be>
 <20160815210908.0687C406077@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160815144126.321370e4@spidey.rellim.com>

Yo Hal!

On Mon, 15 Aug 2016 14:09:07 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> kurt at roeckx.be said:
> > You don't need to "update things".  Just point ntp.conf at that
> > file. With update things I was just thinking that I would need to
> > add a trigger when tzdata is update so I can either reload the
> > config file or restart ntpd, depending on what's needed to get it
> > to check the file.  
> 
> If you send SIGHUP to ntpd it will reload a new file.  You can see
> that in syslog or the ntpd log file.

I just pulled git.  I tried -HUP, this is all that happened:

    08-15T14:39:18 ntpd[19189]: Saw SIGHUP
    08-15T14:39:18 ntpd[19189]: reopen_logfile: same length, ignored

What I need ntpd to do on a -HUP is reread ntp.conf.

> SIGHUP will kill ntp classic so doing nothing seems like the
> conservative approach.  I think it's good enough.

Yes.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160815/c47a013a/attachment.bin>

From esr at thyrsus.com  Thu Aug 18 14:16:24 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Thu, 18 Aug 2016 10:16:24 -0400
Subject: stats directory is gone - now, how is logfile pruning done?
Message-ID: <20160818141624.GA6606@thyrsus.com>

Because ntpviz is now working and documented, I removed a pretty large volume
of crufty ancient scripts for postprocessing statfiles from util/ and
util/stats this morning.  I will be astonished if anyone misses them.  I
merged the documentation under util/stats into docs/, where it should
have been in the first place.

Being able to do this was a pretty major victory for the
maintainability of the codebase, cutting down the number of languages
we have to worry about by 2 - awk and S.  Eventually Perl will also be
banished, leaving Python and sh as the only scripting languages in place.
At which point I will be looking hard at getting rid of sh.

This does, however, leave me with a question: How are we doing pruning
of statfiles at this point?  I seem to recall someone who is not me
working on this.  The facility, whatever it is, needs to be documented.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From hmurray at megapathdsl.net  Fri Aug 19 00:27:14 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Thu, 18 Aug 2016 17:27:14 -0700
Subject: stats directory is gone - now, how is logfile pruning done?
In-Reply-To: Message from "Eric S. Raymond" <esr@thyrsus.com>
 of "Thu, 18 Aug 2016 10:16:24 EDT." <20160818141624.GA6606@thyrsus.com>
Message-ID: <20160819002714.DA217406077@ip-64-139-1-69.sjc.megapath.net>


esr at thyrsus.com said:
> This does, however, leave me with a question: How are we doing pruning of
> statfiles at this point?  I seem to recall someone who is not me working on
> this.  The facility, whatever it is, needs to be documented. 

Whatever you do, make sure it's easy to get the no-prune option.

I suspect that's going to be a distro option and whatever you do will just be 
an example.  Most people won't care about how well their clock is working as 
long at it works well enough so they don't have to pay any attention to it.

Debian sets things up so that /etc/cron.daily/ntp
compresses things and only keeps the last week.
I don't know why they did it that way rather than letting logrotate do it.


-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Fri Aug 19 03:01:15 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Thu, 18 Aug 2016 23:01:15 -0400
Subject: stats directory is gone - now, how is logfile pruning done?
In-Reply-To: <20160819002714.DA217406077@ip-64-139-1-69.sjc.megapath.net>
References: <esr@thyrsus.com> <20160818141624.GA6606@thyrsus.com>
 <20160819002714.DA217406077@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160819030115.GA15248@thyrsus.com>

Heads up, Mark!  Policy sanity check requested.

Hal Murray <hmurray at megapathdsl.net>:
> 
> esr at thyrsus.com said:
> > This does, however, leave me with a question: How are we doing pruning of
> > statfiles at this point?  I seem to recall someone who is not me working on
> > this.  The facility, whatever it is, needs to be documented. 
> 
> Whatever you do, make sure it's easy to get the no-prune option.
> 
> I suspect that's going to be a distro option and whatever you do will just be 
> an example.  Most people won't care about how well their clock is working as 
> long at it works well enough so they don't have to pay any attention to it.
> 
> Debian sets things up so that /etc/cron.daily/ntp
> compresses things and only keeps the last week.
> I don't know why they did it that way rather than letting logrotate do it.

OK, you've told me the important thing: the NTP suite is not historically
expected to do statfile pruning itself.

That's fine, then.  My decision: We won't try taking the job over from
the distros - I'm quite happy to let this be somebody else's problem.

OTOH. if someone pushes a well-documented and neatly-packaged solution
upstream to us (like, say, a logrotate recipe) we'll keep it in etc/
as an option for distro packagers.

If Mark has any larger-context reason to overrule this it won't bother
me any.  But I doubt he will.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From esr at thyrsus.com  Fri Aug 19 14:41:39 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Fri, 19 Aug 2016 10:41:39 -0400 (EDT)
Subject: Python module installation
Message-ID: <20160819144139.5C41813A0C5B@snark.thyrsus.com>

This is heads-up to Amar.

I need the waf recipe modified so that the contents of pylib is
installed as a Python module named 'ntp'.

(Right now this is for the packet module.  I intend to move
the ntpstats.py used by ntpviz there as well.  Later there may
well be an sntp client module added so we can replace ntpdig.)

I've created a wscript file in that directory that is properly
called by the top-level wscript.  What I don't know is how to write
the build production. Presently it's a stub.
--

From verm at darkbeer.org  Fri Aug 19 15:22:43 2016
From: verm at darkbeer.org (Amar Takhar)
Date: Fri, 19 Aug 2016 15:22:43 +0000
Subject: Python module installation
In-Reply-To: <20160819144139.5C41813A0C5B@snark.thyrsus.com>
References: <20160819144139.5C41813A0C5B@snark.thyrsus.com>
Message-ID: <20160819152243.GA6925@darkbeer.org>

On 2016-08-19 10:41 -0400, Eric S. Raymond wrote:
> This is heads-up to Amar.
> 
> I need the waf recipe modified so that the contents of pylib is
> installed as a Python module named 'ntp'.
> 
> (Right now this is for the packet module.  I intend to move
> the ntpstats.py used by ntpviz there as well.  Later there may
> well be an sntp client module added so we can replace ntpdig.)
> 
> I've created a wscript file in that directory that is properly
> called by the top-level wscript.  What I don't know is how to write
> the build production. Presently it's a stub.

OK, I'm busy with my day job right now but I can look at this tonight / 
tomorrow.

I did start something to get this installing properly it won't take long to 
finish it.


Amar.

From fallenpegasus at gmail.com  Fri Aug 19 16:52:13 2016
From: fallenpegasus at gmail.com (Mark Atwood)
Date: Fri, 19 Aug 2016 16:52:13 +0000
Subject: stats directory is gone - now, how is logfile pruning done?
In-Reply-To: <20160819030115.GA15248@thyrsus.com>
References: <esr@thyrsus.com> <20160818141624.GA6606@thyrsus.com>
 <20160819002714.DA217406077@ip-64-139-1-69.sjc.megapath.net>
 <20160819030115.GA15248@thyrsus.com>
Message-ID: <CANW5CYWyv0DMVnJYOaoOBR51DwMt-J_D2h+BHemaUyBS3=wN+Q@mail.gmail.com>

I have no overrule on this point.  Pray continue.

And welcome back, Hal.

On Thu, Aug 18, 2016 at 8:01 PM Eric S. Raymond <esr at thyrsus.com> wrote:

> Heads up, Mark!  Policy sanity check requested.
>
> Hal Murray <hmurray at megapathdsl.net>:
> >
> > esr at thyrsus.com said:
> > > This does, however, leave me with a question: How are we doing pruning
> of
> > > statfiles at this point?  I seem to recall someone who is not me
> working on
> > > this.  The facility, whatever it is, needs to be documented.
> >
> > Whatever you do, make sure it's easy to get the no-prune option.
> >
> > I suspect that's going to be a distro option and whatever you do will
> just be
> > an example.  Most people won't care about how well their clock is
> working as
> > long at it works well enough so they don't have to pay any attention to
> it.
> >
> > Debian sets things up so that /etc/cron.daily/ntp
> > compresses things and only keeps the last week.
> > I don't know why they did it that way rather than letting logrotate do
> it.
>
> OK, you've told me the important thing: the NTP suite is not historically
> expected to do statfile pruning itself.
>
> That's fine, then.  My decision: We won't try taking the job over from
> the distros - I'm quite happy to let this be somebody else's problem.
>
> OTOH. if someone pushes a well-documented and neatly-packaged solution
> upstream to us (like, say, a logrotate recipe) we'll keep it in etc/
> as an option for distro packagers.
>
> If Mark has any larger-context reason to overrule this it won't bother
> me any.  But I doubt he will.
> --
>                 <a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
> _______________________________________________
> devel mailing list
> devel at ntpsec.org
> http://lists.ntpsec.org/mailman/listinfo/devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160819/7130b142/attachment.html>

From esr at thyrsus.com  Fri Aug 19 19:03:22 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Fri, 19 Aug 2016 15:03:22 -0400
Subject: Python module installation
In-Reply-To: <20160819152243.GA6925@darkbeer.org>
References: <20160819144139.5C41813A0C5B@snark.thyrsus.com>
 <20160819152243.GA6925@darkbeer.org>
Message-ID: <20160819190322.GA27369@thyrsus.com>

Amar Takhar <verm at darkbeer.org>:
> OK, I'm busy with my day job right now but I can look at this tonight / 
> tomorrow.
> 
> I did start something to get this installing properly it won't take long to 
> finish it.

I've solved it already. Got some help from the waf author.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From esr at thyrsus.com  Sun Aug 21 20:15:46 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sun, 21 Aug 2016 16:15:46 -0400
Subject: Linux Journal article on NTPsec
Message-ID: <20160821201546.GA14610@thyrsus.com>

This will be published in Linux Journal, probably in October, possibly
as the cover story. They asked me for cover concepts: I suggested either
Salvador Dali's "The Persistence of Memory" (the one with the melting clocks)
or the famous silent-film image of Harold Lloyd hanging from the hands
of a tower clock.

Mark has already reviewed it and suggested one very minor change. There's
still time for corrections.  Daniel, if you have harder numbers for
how we've dodged CVEs, that 'graph could be updated.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
-------------- next part --------------
= NTPsec - a secure, hardened NTP implementation =

// This file is marked up in asciidoc

== Introduction ==

Network time synchronization - aligning your computer's clock to the
same Universal Coordinated Time (UTC) that everyone else is using - is
both necessary and a hard problem.  Many Internet protocols rely on
being able to exchange UTC timestamps accurate to small tolerances,
but the clock crystal in your computer drifts (its frequency varies by
temperature) so it needs occasional adjustments.

That's where life gets complicated.  Sure, you can get another
computer to tell you what time it thinks it is, but if you don't know
how long that packet took to get to you the report isn't very useful.
On top of that, its clock might be broken. Or lying.

To get anywhere, you need to exchange packets with several computers
that allow you to compare your notion of UTC with theirs, estimate
network delays, apply statistical cluster analysis to the resulting
inputs to get a plausible approximation of real UTC, and then adjust
your local clock to it. Generally speaking you can get sustained
accuracy to on the close order of 10 milliseconds this way, though
asymmetrical routing delays can make it much worse if you're in a bad
neighborhood of the Internet.

The protocol for doing this is called "NTP", Network Time Protocol,
and the original implementation was written near the dawn of Internet
time by an eccentric genius named Dave Mills.  Legend has it that
Dr. Mills was the person who got a kid named Vint Cerf interested in
this ARPANET thing. Whether that's true or not, for decades Mills was
*the* go-to guy for computers and high-precision time measurement.

Eventually, though, Dave Mills semi-retired, then retired completely.
His implementation (which we now call "NTP Classic") was left in the
hands of the Network Time Foundation and Harlan Stenn, the man
Information Week feted as "Father Time" in 2015 <<FT>>. Unfortunately,
on NTF's watch some serious problems accumulated. By that year the
codebase was already more than a quarter-century old, and techniques
that had been state-of-the-art when it was first built were showing
their age.  The code had become rigid and difficult to modify, a
problem exacerbated by the fact that very few people actually
understood the Byzantine time-synchronization algorithms at its core.

Among the real-world symptoms of these problems were serious security
issues.  That same year of 2015, infosec researchers becan to realize
that NTP Classic installations were being routinely used as DDoS
amplifiers - ways for crackers to packet-lash target sites by remote
control.  NTF, which had complained for years of being underbudgeted
and understaffed, seemed unable to fix these bugs.

This is intended to be a technical article, so I'm going to pass
lightly over the political and fundraising complications that
ensued. There was, alas, a certain amount of drama.  When the dust
finally settled, a very reluctant fork of the Mills implementation had
been performed in early June 2015 and named 'NTPsec', I had been funded
on an effectively full-time basis by the Linux Foundation to be the
NTPsec's architect/tech-lead, and we had both the nucleus of a capable
development team and some serious challenges.

This much about the drama I will say because it is technically
relevant: One of NTF's major problems was that though NTP Classic was
nominally under an open-source license, NTF retained pre-open-source
habits of mind.  Development was closed and secretive, technically and
socially isolated by NTF's determination to keep using the BitKeeper
version-control system.  One of our mandates from the Linux Foundation
was to fix this, and one of our first serious challenges was simply
moving the code history to git.

This is never trivial for a codebase as large and old as NTP Classic,
and it's especially problematic when the old version-control system is
proprietary with code you can't touch.  I ended up having to heavily
revise Andrew Tridgell's sourecepuller utility - yes, the same code
that triggered Linus Torvald's famous public break with BitKeeper back
in '05 - to do part of the work.  The rest was tedious and difficult
hand-patching with reposurgeon <<RS>>. A year later in May 2016 - far
too late to be helpful - BitKeeper went open-source.

== Strategy and challenges ==

Getting a clean history conversion to git took ten weeks and,
gruelling as that was, it was only the beginning. I had a problem: I was
expected to harden and secure the NTP code, but came in knowing very
little about time service and even less about security engineering.
I'd picked up a few clues about the former from my work leading GPSD
<<GPSD>>, which is widely used for time service. About the latter, I
had some basics about how to harden code - because when you get right
down to it, *that* kind of security engineering is a special case of
reliability engineering, which I *do* understand.  But I had no
experience at "adversarial mindset", the kind of active defense that
good infosec people do, nor any instinct for it.

A way forward came to me when I remembered a famous quote by
C. A. R. Hoare: "There are two ways of constructing a software design:
One way is to make it so simple that there are obviously no
deficiencies, and the other way is to make it so complicated that
there are no obvious deficiencies."  A slightly different angle on
this was the perhaps better-known aphorism by St.-Exup&eacute;ry that
I was to adopt as NTPsec's motto: "Perfection is achieved, not when
there is nothing more to add, but when there is nothing left to take
away."

In the language of modern infosec, Hoare was talking about reducing
attack surface, global complexity, and the scope for unintended
interactions leading to exploitable holes. This was bracing, because
it suggested that maybe I didn't actually need to learn to think like
an infosec specialist or a time service expert. If I could refactor,
cut, and simplify the NTP Classic codebase enough, maybe all those
domain-specific problems would come out in the wash. And if not, then
at least taking the pure software-engineering approach I was
comfortable with might buy me enough time to learn the domain-specific
things I needed to know.

I went all-in on this strategy. It drove my argument for one of the
very first decisions we made, which was to code to a fully modern API
- pure POSIX and C99. This was only partly a move for ensuring portability;
mainly I wanted a principled reason (one we could give potential users
and allies) for ditching all the cruft in the codebase from the
big-iron Unix era.

And there was a *lot* of that.  The code was snarled with portability
#ifdefs and shims for a dozen ancient Unix systems: SunOS, AT&T System
V, HP-UX, UNICOS, DEC OSF/1, Dynix, AIX, and others more obscure. All
relics from the days before API standardization really took hold. The
NTP Classic people were too terrified of offending their legacy
customers to remove any of this stuff, but I knew something they
apparently didn't.  Back around 2006 I had done a cruft-removal pass
over GPSD, pulling it up to pretty strict POSIX conformance - and
nobody from GPSD's highly varied userbase ever said boo about it or
told me they missed the ancient portability shims at all. Thus, what I
had in my pocket was nine years of subsequent GPSD field experience
telling me that the standards people had won their game without most
Unix systems programmers actually capturing all the implications of
that victory.

So I decrufted the NTP code *ruthlessly*. Sometimes I had to fight my
own reflexes in order to to do it.  I too have long been part of the
culture that says "Oh, leave in that old portability shim, you never
know, there just *might* still be a VAX running ISC/5 out there, and
it's not doing any harm."

But when your principal concern is reducing complexity and attack
surface, that thinking is wrong. No individual piece of obsolete code
costs very much, but in a codebase as aged as NTP Classic the
cumulative burden on readability and maintainability becomes massive
and paralyzing.  You have to be hard about this; it all has to go, or
exceptions will pile up on you and you'll never achieve the mission
objective.

I'm emphasizing this point because I think much of what landed NTP
Classic in trouble was not want of skill but a continuing failure of
what one might call surgical courage - the kind of confidence and
determination it takes to make that first incision, knowing that
you're likely to have to make a bloody mess on the way to fixing
what's actually wrong.  Software systems architects working on legacy
infrastructure code need this quality almost as much as surgeons do.

The same applies to superannuated features.  The NTP Classic codebase
was full of dead ends, false starts, failed experiments, drivers for
obsolete clock hardware, and other code that might have been a good
idea once but had long outlived the assumptions behind it.  Mode 7
control messages.  Interleave mode.  Autokey.  An SNMP daemon that was
never conformant to the published standard and never finished. Half a
dozen other smaller warts. Some of these (Mode 7 handling and Autokey
especially) were major attractors for security defects.

As with the port shims, these lingered in the NTP Classic codebase
not because they couldn't have been removed, but because NTF cherished
compatibility back to the year zero and had an allergic reaction to
the thought of removing any features at all. 

Then there were the incidental problems, the largest of which was
Classic's build system.  It was a huge, crumbling, buggy,
poorly-documented pile of autoconf macrology. One of the things that
jumped out at me when I studied NTF's part of the code history was
that in recent years they seemed to spend as much or more effort
fighting defects in their build system as they did modifying code.

But there was one amazingly good thing about the NTP Classic code:
that despite all these problems it *still worked*.  It wheezed and
clanked and was rife with incidental security holes, but it did the
job it was supposed to do. When all was said and done and all the
problems admitted, Dave Mills had been a brilliant systems architect
and, even groaning under the weight of decades of unfortunate
accretions, NTP Classic still functioned.

Thus, the big bet on Hoare's advice at the heart of our technical
strategy unpacked to two assumptions: (a) that beneath the cruft and
barnacles the NTP Classic codebase was fundamentally sound, and (b)
that it would be practically possible to clean it up without breaking
that soundness.

Neither assumption was trivial.  This could have been the a priori
*right* bet on the odds and still failed because the Dread God Finagle
and his mad prophet Murphy micturated in our soup. Or, the code left
after we scraped off the barnacles could actually turn out to be
unsound, fundamentally flawed.

Nevertheless, the success of the team and the project at its declared
objectives was riding on these premises. Through 2015 and early 2016
that was a constant worry in the back of my mind.  *What if I was
wrong?* What I was like the drunk in that old joke, looking for his
keys under the streetlamp when he's dropped then two darkened streets
over because "Offisher, this is where I can see".

The final verdict is not quite in on that question; as I write, NTPsec
is still in beta.  But, as we shall see, there are now (in August
2016) solid indications that the project is on the right track.

== Stripping down, cleaning up ==

One of our team's earliest victories after getting the code history
moved to git was throwing out the autoconf build recipe and replacing
it with one written in a new-school build engine called waf (also used
by Samba and RTEMS). Builds became *much* faster and more reliable.
Just as importantly, this made the the build recipe an order of magnitude
smaller so it could be comprehended as a whole and maintained.

Another early focus was cleaning up and updating the NTP documentation.
We did this before most of the code modifications because the research
required to get it done was an excellent way way to build knowledge about
what was actually going on in the codebase.

These moves began a virtuous cycle. With the build recipe no longer a
buggy and opaque mess, the code could be modified more rapidly and
with more confidence. Each bit of cruft removal lowered the total
complexity of the codebase, making the next one slightly easier.

Testing was pretty ad-hoc at first. Around May 2016, for reasons not
originally related to NTPsec, I became interested in Raspberry Pis.
Then it occurred to me that they would make an excellent way to run
long-term stability tests on NTPsec builds.  Thus it came to be that
the windowsill above my home-office desk is now home to six headless
Raspberry Pis, all equipped with on-board GPSes, all running stability
and correctness tests on NTPsec 24/7. Just as good as a conventional
rack full of servers, but far less bulky and expensive!

We got a lot done over our first eighteen months.  The headline number
that shows just how much was the change in the codebase's total size.
We went from 227KLOC to 88KLOC, cutting the total line count by almost
a factor of three.

Dramatic as that sounds, it actually understates the attack-surface
reduction we achieved, because complexity was not evenly distributed
in the codebase.  The worst technical debt, and the security holes,
tended to lurk in the obsolete and semi-obsolete code that hadn't
gotten any developer attention in a long time.  NTP Classic was not
exceptional in this; I've seen the same pattern in other large, old
codebases I've worked on.

Another important measure was systematically hunting down and replacing
all unsafe C function calls with equivalents that can provably not cause
buffer overruns.  I'll quote from NTPsec's hacking guide:

------------------------------------------------------------------------
* strcpy, strncpy, strcat:  Use strlcpy and strlcat instead.
* sprintf, vsprintf: use snprintf and vsnprintf instead.
* In scanf and friends, the %s format without length limit is banned.
* strtok: use strtok_r() or unroll this into the obvious loop.
* gets: Use fgets instead. 
* gmtime(), localtime(), asctime(), ctime(): use the reentrant *_r variants.
* tmpnam() - use mkstemp() or tmpfile() instead.
* dirname() - the Linux version is re-entrant but this property is not portable.
------------------------------------------------------------------------

This formalized an approach I?d used successfully on GPSD ? instead of
fixing defects and security holes after the fact, constrain your code
so that it *cannot have* entire classes of defects.

The experienced C programmers out there are are thinking "What about
wild-pointer and wild-index problems?" And it?s true that the achtung
verboten above will not prevent those kinds of overruns. That's why another
prong of the strategy was systematic use of static code analyzers like
Coverity, which actually is pretty good at picking up the defects that
cause that sort of thing. Not 100% perfect, C will always allow you to
shoot yourself in the foot, but I knew from prior success with GPSD
that the combination of careful coding with automatic defect scanning
can reduce your bug load a very great deal.

To help defect scanners do a better job, we enriched the type
information in the code.  The largest single change of this kind was
changing int variables to C99 bools everywhere they were being used as
booleans.

Little things also mattered, like fixing all compiler warnings. I
thought it was shockingly sloppy that the NTP Classic maintainers
hadn?t done this. The pattern detectors behind those warnings are
there because they often point at real defects. Also, voluminous
warnings make it too easy to miss actual errors that break your
build. And you never want to break your build, because later on that
will make bisection testing more difficult.

An early sign that this systematic defect-prevention approach was
working was the extremely low rate of bugs we detected by testing as
having been introduced during our cleanup. In the first fourteen
months we averaged less than one iatrogenic C bug every ninety days.

I would have had a lot of trouble believing that if GPSD hadn't posted
a defect frequency nearly as low over the previous five years.  A
major lesson from both projects is that applying best practices in
coding and testing really works.  I pushed this point back in 2012 in
my essay on GPSD for 'The Architecture of Open Source, Volume 2" <<AOS2>>;
what NTPsec shows is that GPSD is not a fluke.

I think this is one of the most important takeaways from both
projects.  We really don't have to settle for what have historically
been considered "normal" defect rates in C code.  Modern tools and
practices can go a very long way towards driving those defect rates
towards zero.  It's no longer even very difficult to do the right
thing; what's too often missing is a grasp of the possibility and the
determination to pursue it.

And here's the real payoff. Early in 2016, CVEs (security alerts)
started issuing against NTP Classic that NTPsec dodged because we had
already cut out their attack surface before we knew there was a bug!  This
actually became a regular thing, with the percentage of dodged bullets
increasing over time.  Somewhere, Hoare and St.-Exup&eacute;ry might
be smiling.

The cleanup isn't done yet.  We're testing a major refactoring and
simplification of the central protocol machine for processing NTP
packets. We believe this has already revealed a significant number of
potential security defects nobody ever had a clue about before. Every
one of these will be another dodged bullet attributable to getting our 
practice and strategic direction right.

== Features? What features? ==

I have yet to mention new features because NTPsec doesn't have many;
that's not where our energy has been going.  But here's one that
came directly out of the cleanup work...

When NTP was originally written, computer clocks only delivered
microsecond precision. Now they deliver nanosecond precision (though
not all of that precision is accurate). By changing some internal
representations we have made NTPsec able to use the full precision of
modern clocks when stepping them, which can result in a factor 10 or more
of accuracy improvement with real hardware such as GPSDOs and
dedicated time radios.

Fixing this was about a four-line patch. It might have been noticed
sooner if the code hadn't been using an uneasy mixture of microsecond and
nanosecond precision for historical reasons. As it is, anything short 
of the kind of systematic API-usage update we were doing would have
been quite unlikely to spot the problem.

A longstanding pain point we've begun to address is the
nigh-impenetrable syntax of the ntp.conf file. We've already
implemented a new syntax for declaring reference clocks that is far
easier to understand than the old.  We have more work planned towards
making composing NTP configurations less of a black art.

The diagnostic tools shipped with NTP Classic were messy,
undocumented, and archaic.  We have a new tool, ntpviz, which gives
time-server operators a graphical and much more informative view of
what's been going on in the server logfiles.  This will assist in
understanding and mitigating various sources of inaccuracy.

== Where we go from here ==

We don't think our 1.0 release is far in the future - in fact, given
normal publication delays, it might well have shipped by the time you
read this. An early-adopter contingent - including at least one
high-frequency-trading company for which accurate time is
business-critical - is already happily using NTPsec for production.

There remains much work to be done after 1.0.  We're cooperating
closely with IETF to develop a replacement for Autokey public-key
authentication that actually works. We want to move as much of the C
code as possible outside ntpd itself to Python in order to reduce
long-term maintainance load.  There's a possibility that the core
daemon itself might be split in two to separate the TCP/IP parts from
the handling of local reference clocks, drastically reducing global
complexity.

Beyond that, we're gaining insight into the core time-synchronization
algorithms and suspect there are real possibilities for improvement in
those. Better statistical filtering that's sensitive to measurements
of network weather and topology looks possible.

It's an adventure, and we welcome anyone who'd like to join in.
NTP is vital infrastructure, and keeping it healthy over a time-frame
of decades will need a large, flourishing community.  You can learn
more about how to take part at our project website <<NTPSEC>>>.

== References ==

[bibliography]

[[[FT]]] http://www.informationweek.com/it-life/ntps-fate-hinges-on-father-time/d/d-id/1319432[NTP's Fate Hinges On "Father Time"]

[[[RS]]] http://www.catb.org/esr/reposurgeon/[reposurgeon]

[[[GPSD]]] http://catb.org/gpsd/[GPSD]

[[[AOS2]]] http://www.aosabook.org/en/gpsd.html[GPSD in AOS2]

[[[NTPSEC]]] https://www.ntpsec.org/[Welcome to NTPsec]

From hmurray at megapathdsl.net  Sun Aug 21 23:17:33 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 21 Aug 2016 16:17:33 -0700
Subject: Big picture
Message-ID: <20160821231733.32369406061@ip-64-139-1-69.sjc.megapath.net>


Look ahead a year or two.  Assume the NTPWG has agreed upon a good crypto 
layer and that we are happy with our code.  What happens next?

Somebody is going to need to deploy enough well run NTP servers.

My straw man is that it will be roughly parallel to DNS.  We will need a 
diverse collection of stratum 1 servers parallel to the DNS root servers.  
Then ISPs will have to run servers for their clients similar to the way 
many/most of them run caching DNS servers.

Are any of the people you talk to thinking about that area?


-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Mon Aug 22 01:07:51 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sun, 21 Aug 2016 21:07:51 -0400
Subject: Big picture
In-Reply-To: <20160821231733.32369406061@ip-64-139-1-69.sjc.megapath.net>
References: <20160821231733.32369406061@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160822010751.GC15767@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> Look ahead a year or two.  Assume the NTPWG has agreed upon a good crypto 
> layer and that we are happy with our code.  What happens next?

Er, most likely I hand off to a successor who doesn't need to be a
forensic systems architect (I'm grooming Daniel Franke for this) and
go looking for another big codebase that badly needs to be fixed.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From hmurray at megapathdsl.net  Mon Aug 22 05:36:06 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 21 Aug 2016 22:36:06 -0700
Subject: Wish list - hack to monitor a server
Message-ID: <20160822053606.14F16406061@ip-64-139-1-69.sjc.megapath.net>


I'm thinking of something that will monitor a NTP server and send 
mail/SMS/whatever when it finds troubles.

I can see two types of trouble.  One is when the server isn't responding.  
The other is that one or more of the upstream servers or refclocks it is 
using stops responding.

There are lots of possible variations/options/parameters.

This should probably wait until Eric gets the python library ready for use 
under ntpq.  I mention it now because I thought of it and/or it might provide 
input to the requirements for that library.


-- 
These are my opinions.  I hate spam.




From gem at rellim.com  Mon Aug 22 18:31:27 2016
From: gem at rellim.com (Gary E. Miller)
Date: Mon, 22 Aug 2016 11:31:27 -0700
Subject: Wish list - hack to monitor a server
In-Reply-To: <20160822053606.14F16406061@ip-64-139-1-69.sjc.megapath.net>
References: <20160822053606.14F16406061@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160822113127.0ad32300@spidey.rellim.com>

Yo Hal!

On Sun, 21 Aug 2016 22:36:06 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> I'm thinking of something that will monitor a NTP server and send 
> mail/SMS/whatever when it finds troubles.

I use Icinga2 (nagios fork) for that.  NTPsec prolly just needs to
document that existing usage.

> I can see two types of trouble.  One is when the server isn't
> responding. The other is that one or more of the upstream servers or
> refclocks it is using stops responding.

I can see us including a nagio/icinga2 plugin for that, but the
existing plugins have worked great for me for over a decade.

Here are the existing plugins:

/usr/lib64/nagios/plugins/check_ntp_peer
/usr/lib64/nagios/plugins/check_ntp_time

> There are lots of possible variations/options/parameters.

Yeah.  Here are the existing options.
spidey ntpsec # /usr/lib64/nagios/plugins/check_ntp_peer

Usage:
 check_ntp_peer -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-W <warn>] [-C <crit>]
       [-j <warn>] [-k <crit>] [-v verbose]

spidey ntpsec # /usr/lib64/nagios/plugins/check_ntp_time

Usage:
 check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
/usr/lib64/nagios/plugins/check_ntp_time

> This should probably wait until Eric gets the python library ready
> for use under ntpq.  I mention it now because I thought of it and/or
> it might provide input to the requirements for that library.

Seems to me that's reinventing what is already a very sturday, well
trusted and well deployed wheel.

New bug:

    https://gitlab.com/NTPsec/ntpsec/issues/101

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160822/b30dc913/attachment.bin>

From gem at rellim.com  Mon Aug 22 21:50:15 2016
From: gem at rellim.com (Gary E. Miller)
Date: Mon, 22 Aug 2016 14:50:15 -0700
Subject: =?UTF-8?B?4pyYbnRwdml6?= on missing ntpstats, bug #102
Message-ID: <20160822145015.11a7b80f@spidey.rellim.com>

Yo Hal!

Check out bug #102: https://gitlab.com/NTPsec/ntpsec/issues/102

I think I fixed your crashes on bad ntpstats directory and missing
peerstas and cputemp.  Please tests.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160822/9a3c0d27/attachment.bin>

From hmurray at megapathdsl.net  Tue Aug 23 03:34:41 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Mon, 22 Aug 2016 20:34:41 -0700
Subject: Wish list - hack to monitor a server
In-Reply-To: Message from "Gary E. Miller" <gem@rellim.com>
 of "Mon, 22 Aug 2016 11:31:27 PDT." <20160822113127.0ad32300@spidey.rellim.com>
Message-ID: <20160823033441.A22DB406061@ip-64-139-1-69.sjc.megapath.net>


gem at rellim.com said:
> I use Icinga2 (nagios fork) for that.  NTPsec prolly just needs to document
> that existing usage. 

I think it's more complicated than that.

Somebody needs to look at their source code and see what sort of API they are 
using.  Then we need to document that API so we don't "fix" something and 
break their code in the process.

(or help them clean things up if they have done something we don't want to support)


-- 
These are my opinions.  I hate spam.




From gem at rellim.com  Tue Aug 23 04:38:23 2016
From: gem at rellim.com (Gary E. Miller)
Date: Mon, 22 Aug 2016 21:38:23 -0700
Subject: Wish list - hack to monitor a server
In-Reply-To: <20160823033441.A22DB406061@ip-64-139-1-69.sjc.megapath.net>
References: <20160822113127.0ad32300@spidey.rellim.com>
 <20160823033441.A22DB406061@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160822213823.65589710@spidey.rellim.com>

Yo Hal!

On Mon, 22 Aug 2016 20:34:41 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> gem at rellim.com said:
> > I use Icinga2 (nagios fork) for that.  NTPsec prolly just needs to
> > document that existing usage.   
> 
> I think it's more complicated than that.
> 
> Somebody needs to look at their source code and see what sort of API
> they are using.  Then we need to document that API so we don't "fix"
> something and break their code in the process.

No API.  You run a commmand line program, it returns: OK, Warning or
Error.

> (or help them clean things up if they have done something we don't
> want to support)

I've been using it for over a decade, I can't think of anything I'd change.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160822/687e6db9/attachment.bin>

From hmurray at megapathdsl.net  Tue Aug 23 05:19:16 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Mon, 22 Aug 2016 22:19:16 -0700
Subject: Wish list - hack to monitor a server
In-Reply-To: Message from "Gary E. Miller" <gem@rellim.com>
 of "Mon, 22 Aug 2016 21:38:23 PDT." <20160822213823.65589710@spidey.rellim.com>
Message-ID: <20160823051916.DD534406061@ip-64-139-1-69.sjc.megapath.net>


gem at rellim.com said:
> No API.  You run a commmand line program, it returns: OK, Warning or Error.

API may not be the right term, but they have to be using some interface.

Their program talks to our stuff somehow.  That's the interface I was 
referring to.  They may be using ntpq and parsing the printout which makes 
the interface at the ntpq printout level.  They may be using mode 6 packets 
(the same as ntpq uses), if so, that's the interface.

I think the mode 6 packets are documented in an RFC.  It's way out of date by 
now.


-- 
These are my opinions.  I hate spam.




From gem at rellim.com  Tue Aug 23 05:32:44 2016
From: gem at rellim.com (Gary E. Miller)
Date: Mon, 22 Aug 2016 22:32:44 -0700
Subject: Wish list - hack to monitor a server
In-Reply-To: <20160823051916.DD534406061@ip-64-139-1-69.sjc.megapath.net>
References: <20160822213823.65589710@spidey.rellim.com>
 <20160823051916.DD534406061@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160822223244.674c0f23@spidey.rellim.com>

Yo Hal!

On Mon, 22 Aug 2016 22:19:16 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> gem at rellim.com said:
> > No API.  You run a commmand line program, it returns: OK, Warning
> > or Error.  
> 
> API may not be the right term, but they have to be using some
> interface.

OK.

> Their program talks to our stuff somehow.

Sure, it just make an ordinary time request like any other NTP client.

Which in the end is what we care about, we want to know that an ordinary
client would see a good and ordinary response.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160822/92412e1e/attachment.bin>

From hmurray at megapathdsl.net  Tue Aug 23 06:18:12 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Mon, 22 Aug 2016 23:18:12 -0700
Subject: Wish list - hack to monitor a server
In-Reply-To: Message from "Gary E. Miller" <gem@rellim.com>
 of "Mon, 22 Aug 2016 22:32:44 PDT." <20160822223244.674c0f23@spidey.rellim.com>
Message-ID: <20160823061812.60DAB406061@ip-64-139-1-69.sjc.megapath.net>


> Sure, it just make an ordinary time request like any other NTP client.

That's only half of what I was looking for.

The other half is "Are the servers that system is using working right?"  Or 
rather does the server I want to monitor think they are working.  The latter 
covers refclocks and the network path from server X to server Y which may be 
different from monitoring station to server Y.


-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Tue Aug 23 08:53:35 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Tue, 23 Aug 2016 01:53:35 -0700
Subject: Comments about ntpviz
Message-ID: <20160823085335.3092E406057@ip-64-139-1-69.sjc.megapath.net>


I've been playing with graphs for a long time.   Here is a dump of what I've 
done and/or some comments about the current ntpviz.  Maybe something will be 
useful.

I have a script that uses rsync to collect the log files from various 
machines.

Mostly, I plot 2 days, yesterday and as much of today as is available.

I have a script that sets up links from
  <name>-loop to <name>/loopstats-<today> and
  <name>-loop-1 to <name>/loopstats-<yesterday>
It has a cousin that sets the links up to target and day-before-target.

There are more kludgy scripts that split <name>-peer into <name>
-peer-<ipaddrss>

Then I use gnuplot to scan through a batch of graphs.  I have a top level 
file full of things like:
  reset; load "xxx.gp"
  pause -1
where all the xxx.gp are setup by hand.  They don't change very often.  
Occasionally I have to comment out one of the load/pause pairs because the 
system generating the data is broken for all the time in the graph and that 
kills gnuplot.

A lot of the graphs I'm interested in need two scales, one that shows 
everything and another that ignores the outliers and shows the main stuff.  
For those, I add something like:
  set yscale [xx:xx]; replot
  pause -1

I have a few graphs that are long running, months.  For those, I typically 
use different colors on odd/even days, and another pair of colors for 
alternate months.  Here are a couple of samples:
  http://users.megapathdsl.net/~hmurray/time-nuts/line/Calif-60Hz-2015-Jul.png
  http://users.megapathdsl.net/~hmurray/time-nuts/line/Calif-60Hz-2014-2015.pn
g

Some of the long running graphs need pre-processing.  I usually cache as much 
of that as is convenient.

-------

Some of gnuplots colors are hard to see, at least on my setup with my eyes.  
Yellow is the worst case.

Some graphs look better (to my eye) in points mode rather than lines.  You 
have to be careful to pick a dot that is big enough to see.  Test to gnuplot 
will show them along the right side.  I use the solid ones.

If a server is down for a while, the long straight line linking over that gap 
looks ugly to me.

------

Using microseconds for everything is hard to read for long delays typical of 
network links.  In bufferbloat cases, I see delays of several seconds.  I've 
seen delays of 10s of seconds.  I can't count all those 0s quickly.  I think 
milliseconds would be better.  (Looks like Gary tweaked things while I was 
typing this in.)

The time labels on the X axis are hard to read on 2 day graphs.  The hours 
that I want are hidden between day and minutes.  (I use 0-48, but I don't 
have to worry about fractional day offsets.)

-------

You can monitor other servers by using noselect.  It sends packets and writes 
rawstats and peerstats, then ignores the answer.  They show up in ntpq -peers 
with a blank in col 1.

The round trip time is interesting, especially if you have a bufferbloated 
DLS link.

There is a lot of interesting info in rawstats.  Some/much of it gets 
filtered out on the way to peerstats.  It probably tells you more about your 
network than your local ntp server.

If you assume that both your local clock and the server's clock are accurate, 
you can get the network transit delay in each direction.  If you plot that 
you can match steps in the round trip time with steps in a particular 
direction.

There is interesting data in clockstats, at least for some drivers.  One of 
the important ones is the count for the PPS driver.  It's real interesting if 
your PPS pulse isn't wide enough to work all the time.

If you have a busy server, there is interesting stuff in usestats.
 


-- 
These are my opinions.  I hate spam.




From Stromeko at nexgo.de  Tue Aug 23 16:04:26 2016
From: Stromeko at nexgo.de (Achim Gratz)
Date: Tue, 23 Aug 2016 18:04:26 +0200
Subject: Comments about ntpviz
References: <20160823085335.3092E406057@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <87oa4jv5dh.fsf@Rainer.invalid>

Hal Murray writes:
> Using microseconds for everything is hard to read for long delays typical of 
> network links.  In bufferbloat cases, I see delays of several seconds.  I've 
> seen delays of 10s of seconds.  I can't count all those 0s quickly.  I think 
> milliseconds would be better.  (Looks like Gary tweaked things while I was 
> typing this in.)

What I do is scale the delat-times with an asinh, which makes it linear
around zero and logarithmic further out, plus it automatically accepts
zero and negative input.  Depending on the the base level of noise you'd
need to change the sclaing constant that determines where the "knee"
going from logarithmic to linear is.  If your preprocessing gives you
the MAD or sigma, then you can use that to compute that constant.

I monitor the PPS signal via ppswatch.  That gives you a file that grows
enormously, but you'll have all the data independently of ntp recording
it.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves


From gem at rellim.com  Tue Aug 23 17:28:26 2016
From: gem at rellim.com (Gary E. Miller)
Date: Tue, 23 Aug 2016 10:28:26 -0700
Subject: Comments about ntpviz
In-Reply-To: <20160823085335.3092E406057@ip-64-139-1-69.sjc.megapath.net>
References: <20160823085335.3092E406057@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160823102826.51655f9c@spidey.rellim.com>

Yo Hal!

On Tue, 23 Aug 2016 01:53:35 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> I have a script that uses rsync to collect the log files from various 
> machines.

Eric is working on something like that.

> Mostly, I plot 2 days, yesterday and as much of today as is available.

So two graphs, one for today and yesterday?  ntpviz can do that.

> There are more kludgy scripts that split <name>-peer into <name>
> -peer-<ipaddrss>

ntpviz does that.

> A lot of the graphs I'm interested in need two scales, one that shows 
> everything and another that ignores the outliers and shows the main
> stuff. For those, I add something like:
>   set yscale [xx:xx]; replot
>   pause -1

Yeah, I do see that is handy now and then.

> I have a few graphs that are long running, months.  For those, I
> typically use different colors on odd/even days, and another pair of
> colors for alternate months.  Here are a couple of samples:

Looks like a holiday tree, I find it hard to ficus on the red/green pairs.

> Some of gnuplots colors are hard to see, at least on my setup with my
> eyes. Yellow is the worst case.

Yeah, if you have any specificc suggestions on how ntpviz can fix that.
But please no red/green barber polls...

> If a server is down for a while, the long straight line linking over
> that gap looks ugly to me.

Yeah, a down server is ugly.  It should be shown ugly.

> Using microseconds for everything is hard to read for long delays
> typical of network links.  In bufferbloat cases, I see delays of
> several seconds.  I've seen delays of 10s of seconds.  I can't count
> all those 0s quickly.  I think milliseconds would be better.  (Looks
> like Gary tweaked things while I was typing this in.)

Yup, I'm working on autoscaling.  A lot more to do.  I'm thinking of
just clipping the outliers ( < 0.5%, > 99.5% ) with a text message of
the worst ones

> The round trip time is interesting, especially if you have a
> bufferbloated DLS link.

Yes.  Dan Drown had those.  They are in the work queue.

> If you assume that both your local clock and the server's clock are
> accurate, you can get the network transit delay in each direction.
> If you plot that you can match steps in the round trip time with
> steps in a particular direction.

An interessting idea, but NTPsec needs to focus on core ntpd needs
until those are nailed down.

> There is interesting data in clockstats, at least for some drivers.
> One of the important ones is the count for the PPS driver.  It's real
> interesting if your PPS pulse isn't wide enough to work all the time.

Yeah, been looking at that.  Since ntpd is undersampling the PPS it can
be either good, or real bad.  I'm tempted to get Eric to fix the bug
first, but maybe I'll need to data so he sees the bug first.

> If you have a busy server, there is interesting stuff in usestats.

Such as?

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160823/ed50615a/attachment.bin>

From hmurray at megapathdsl.net  Tue Aug 23 20:38:54 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Tue, 23 Aug 2016 13:38:54 -0700
Subject: Comments about ntpviz
In-Reply-To: Message from "Gary E. Miller" <gem@rellim.com>
 of "Tue, 23 Aug 2016 10:28:26 PDT." <20160823102826.51655f9c@spidey.rellim.com>
Message-ID: <20160823203854.7972540605C@ip-64-139-1-69.sjc.megapath.net>


The part I forgot to mention is that if I was going to do something to make 
my setup better, I would try to make a wrapper to drive gnuplot - a little 
window off to the side so I could poke Next rather than typing return at the 
gnuplot command window.  The main thing I want is a way to backup a few 
graphs.  The second step would be a simple way to adjust vertical and/or 
horizontal scaling (aka zoom in/out) to cover the cases where I didn't 
anticipate I would want one and/or didn't get it right.

---------

> So two graphs, one for today and yesterday?  ntpviz can do that.

No, one graph, 48 hours wide, yesterday on the left and today on the right.  
The part of today that hasn't happened yet (or hasn't been copied over to the 
local system) is empty.

Remember, I'm using pre-edited gnuplot commands.  There is no place in my 
work flow to tell gnuplot how wide the graph should be.

I often set the graph to be 8 extra hours wide so the key has a place to go 
without trashing part of the graph I might be interested in.


[Need two different scales.]
>>   set yscale [xx:xx]; replot
>>   pause -1
> Yeah, I do see that is handy now and then. 

It happens enough for me that I did something about it.


> Looks like a holiday tree, I find it hard to ficus on the red/green pairs.

The usual nasty case is boundaries between between red/blue areas.  Your eye 
has lots of chromatic aberration.  It can't focus on different colors at the 
same time.  Red and blue are the farthest apart.


[using crappy colors]
> Yeah, if you have any specificc suggestions on how ntpviz can fix that. But
> please no red/green barber polls... 

The only fix I know about is to manually pick the set of colors you are 
willing to use and then tell gnuplot which color to use for each chunk of 
data.  If you let it assign things automatically, it will rotate through all 
of them and that includes the crappy ones.  Yellow is 5 so you get 4 decent 
ones without any effort.  Somewhere around 4 different data sets many graphs 
get too busy so this is only a problem occasionally.  I have one plot where 
lots of lines work well.  It's the temperature for various machines and 
places in the room.  They don't overlap much all track well so you can see 
any interesting differences.

> Yeah, a down server is ugly.  It should be shown ugly.

I have some systems that are normally off to save power.  For me, the gap is 
enough to attract my attention.

I had one setup where the script to start ntpd appended 2 blank lines to all 
the stats files.  gnuplot treats that as a request to not draw the line 
connecting the points on either side.


> Yup, I'm working on autoscaling.  A lot more to do.  I'm thinking of just
> clipping the outliers ( < 0.5%, > 99.5% ) with a text message of the worst
> ones 

I want to be able to see the outliers.  At least optionally default on.

I think you will have troubles trying to automate that.  It will get mixed up 
with scaling.


>> If you have a busy server, there is interesting stuff in usestats.
> Such as?

The reason I added it was to get the memory usage.

sysstats has packet counts.  I think usestats has wakeup counts so you should 
be able to figure out how many packets arrived while you were already working 
on something.

There is lots of room for improvement in this area.  I thin it's all on the 
geeky end.


[rawstats/round trip time]
> An interessting idea, but NTPsec needs to focus on core ntpd needs until
> those are nailed down.

The round trip time is important if you are trying to understand what 
happened to a server out on the big bad internet.  It's probably less 
interesting for debugging your local LAN.  It might be helpful if you are 
using WiFi.

A reasonably common quirk is that the offset of a server will jump, say 10 
ms.  If the round trip time jumps at the same time, the problem is most 
likely a routing shift rather than a glitch at the remote server.


> Yeah, been looking at that.  Since ntpd is undersampling the PPS it can be
> either good, or real bad.  I'm tempted to get Eric to fix the bug first, but
> maybe I'll need to data so he sees the bug first. 

Response in a new thread.



-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Tue Aug 23 23:13:04 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Tue, 23 Aug 2016 16:13:04 -0700
Subject: PPS undersampling
Message-ID: <20160823231304.82BB5406057@ip-64-139-1-69.sjc.megapath.net>


> Yeah, been looking at that.  Since ntpd is undersampling the PPS it can be
> either good, or real bad.  I'm tempted to get Eric to fix the bug first, but
> maybe I'll need to data so he sees the bug first. 

I don't know of any ntpd bug in this area.  Do you have a test case?  Or data 
from a buggy run?

There are/were glitches in some of the gpsd utilities.  I fixed one a long 
time ago and I think you fixed one recently.  But they are using a different 
mechanism.

Your mention of Nyquist on IRC a few days ago makes me think that you don't 
understand this area or more likely are just grabbing a word in a related 
area for a similar problem.

Nyquist refers to analog signals.  If your signal has a bandwidth of X Hz, 
you need 2*X samples per second in order to be able to reconstruct the signal.

The PPS is not analog.  We are not trying to reconstruct a signal.  We are 
trying to pass information across clock domains.  The equivalent of the 
Nyquist rule is that you have to grab the data before it gets updated with 
new data.  There is no factor of 2 in there.  Just "fast enough".

The gpsd code I fixed was doing something like:
  while true {
    if sampleready() then dosomething()
    sleep 1 second
  }
That is polling at slightly slower than 1 second.  If the PPS goes off every 
second, that approach will occasionally miss a sample.  But the occasional 
will be very small (unless your scheduler is very busy or your CPU is slow or 
your dosomething does a lot of work).  Unless you look carefully it will be 
hard to notice.  (But geeks do look carefully.)

The code in ntpd doesn't work that way.  It runs off a timer that signals and 
resets itself.  That signal goes off every second, the same rate as the PPS.

If you are polling at a fixed rate from an unsynchronized clock, the "fast 
enough" has to include the jitter on the data source and the jitter on the 
scheduling clock.  If the PPS signal has 1 ms peak-peak of jitter, you would 
have to poll every 999 ms.  If your scheduler adds 1 ms of jitter, you have 
to ask it to run you every 998 ms.

But the ntpd clock is not unsynchronized.  It's running at exactly 1 second 
per second in the long term.

ntpd doesn't do anything special about when it starts the timer.  So if you 
are unlucky and you start ntpd so that it is near a PPS pulse when it sets 
the timer, you might miss occasional samples.  It will be tough to reproduce 
but there should be evidence in clockstats.

If we decide to fix this, the fix is not to sample twice as fast, but to make 
sure the timer doesn't go off too close to the top of a second.  In the 
meantime, we should collect data on the PPS jitter and/or scheduler jitter so 
we can make an informed estimate of what "too close" will be.

Actually, any fix should wait for a bigger cleanup in this area.  We should 
get rid of the current every-second timer unless some refclock needs it.  
(battery power)  The PPS API has an optional wait/wakeup option.  We should 
use that if available.  The timer stuff also covers when to transmit a 
packet, but we should peek ahead and set the timeout on the select for as 
long as possible.  But all that should wait until after TESTFRAME so we can 
test it.




-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Tue Aug 23 23:32:55 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Tue, 23 Aug 2016 19:32:55 -0400
Subject: PPS undersampling
In-Reply-To: <20160823231304.82BB5406057@ip-64-139-1-69.sjc.megapath.net>
References: <20160823231304.82BB5406057@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160823233255.GB16539@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
>                                                                We should 
> get rid of the current every-second timer unless some refclock needs it.  
> (battery power)  The PPS API has an optional wait/wakeup option.  We should 
> use that if available.  The timer stuff also covers when to transmit a 
> packet, but we should peek ahead and set the timeout on the select for as 
> long as possible.  But all that should wait until after TESTFRAME so we can 
> test it.

Agreed.  This has been on my long-term to-do list for some time.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From gem at rellim.com  Tue Aug 23 23:43:32 2016
From: gem at rellim.com (Gary E. Miller)
Date: Tue, 23 Aug 2016 16:43:32 -0700
Subject: PPS undersampling
In-Reply-To: <20160823233255.GB16539@thyrsus.com>
References: <20160823231304.82BB5406057@ip-64-139-1-69.sjc.megapath.net>
 <20160823233255.GB16539@thyrsus.com>
Message-ID: <20160823164332.4d8e370f@spidey.rellim.com>

Yo Eric!

On Tue, 23 Aug 2016 19:32:55 -0400
"Eric S. Raymond" <esr at thyrsus.com> wrote:

> Hal Murray <hmurray at megapathdsl.net>:
> >                                                                We
> > should get rid of the current every-second timer unless some
> > refclock needs it. (battery power)  The PPS API has an optional
> > wait/wakeup option.  We should use that if available.  The timer
> > stuff also covers when to transmit a packet, but we should peek
> > ahead and set the timeout on the select for as long as possible.
> > But all that should wait until after TESTFRAME so we can test it.  
> 
> Agreed.  This has been on my long-term to-do list for some time.

Sadly, PPS with the current algorithm should bump up to sampling every
one-half second.  Some even want 10 Hz or even 100 Hz.

Let us leave it on the long-term to-do list and not hash over it now.
The subject is complicated and contentious.  Way too many moving parts
in a core chunk of the code to touch without TESTFRAME.  I also expect
TESTFRAME will smoke out some of these issues and also have its own
different timer loop needs.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160823/96de3745/attachment.bin>

From gem at rellim.com  Wed Aug 24 00:06:16 2016
From: gem at rellim.com (Gary E. Miller)
Date: Tue, 23 Aug 2016 17:06:16 -0700
Subject: PPS undersampling
In-Reply-To: <20160823231304.82BB5406057@ip-64-139-1-69.sjc.megapath.net>
References: <20160823231304.82BB5406057@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160823170616.0d1bf9df@spidey.rellim.com>

Yo Hal!

On Tue, 23 Aug 2016 16:13:04 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> > Yeah, been looking at that.  Since ntpd is undersampling the PPS it
> > can be either good, or real bad.  I'm tempted to get Eric to fix
> > the bug first, but maybe I'll need to data so he sees the bug
> > first.   
> 
> I don't know of any ntpd bug in this area.  Do you have a test case?
> Or data from a buggy run?

Not a formal bug, but a bug none the less.  Easy to reproduce: reboot
your host, with an intentional off swclock.  Then look at how many PPS
sample you get per poll period.  At the perverse end ntpd can miss
one in 12 samples.

I have proposed this simple test many times.  I have run it myself many
times.  Have you done that yet?  I have also proposed other simple tests.
For some reason no one ever comes back with data to show my tests show
no problem.

Misunderstandings in this area cause people to break the PPS code in
gpsd every few years.  I keep having to revert the changes and show
people the data that says their changes are bad

Having been there, done that, so many time in the PPS code I am not
prepared to take two weeks of my time to explain it again until
Eric is prepared to take 4 weeks of his time to fix it.

> There are/were glitches in some of the gpsd utilities.  I fixed one a
> long time ago and I think you fixed one recently.  But they are using
> a different mechanism.

Yes, totally unrelated.

> Your mention of Nyquist on IRC a few days ago makes me think that you
> don't understand this area or more likely are just grabbing a word in
> a related area for a similar problem.

Nyquist was a key part of my EE degree.  I worked for and consulted to
many test equipment companies for many years.  I reverse engineered
a lot of HP gear.  So yes, I'll put my knowledge of Nyquist up with
anyone's.

BUT, I do not expect anyone to care about the previous paragraph.  When
the timme is right, when NTPsec is prepared to understand and accept
improvements, I will dust off my test cses for one and all to see.

In tthe end the data will speak for itself.

> Nyquist refers to analog signals.  If your signal has a bandwidth of
> X Hz, you need 2*X samples per second in order to be able to
> reconstruct the signal.

Mostly correct, except for the word 'analog', and some gross over
simplification.

FWIW, note that Wikipedia agree with me:

https://en.wikipedia.org/wiki/Nyquist%E2%80%93Shannon_sampling_theorem

You would do well to read and undertand that article, as a start.

I actually spent time in the dark ages creating UARTs out of TTL.  If
you are up for archeology I suggested you look at how that is done, and
how Nyquist-Shannon theory was such an important part.

> The gpsd code I fixed was doing something like:
>   while true {
>     if sampleready() then dosomething()
>     sleep 1 second
>   }

No, that is the gpsd code that broke it.  Since fixed.

> The code in ntpd doesn't work that way.  It runs off a timer that
> signals and resets itself.  That signal goes off every second, the
> same rate as the PPS.

Which is proveably wrong.

> But the ntpd clock is not unsynchronized.  It's running at exactly 1
> second per second in the long term.

And the long term is not the problem.  The problem is the positively 
aweful startup performance of ntpd.

> Actually, any fix should wait for a bigger cleanup in this area.


And in this we 100% agree.  When the time comes I have a number of
experiments for one and all to run that show the problems.


> We
> should get rid of the current every-second timer unless some refclock
> needs it.

Yes, but what drivers really need what is a huge discussion.  That is
for the long term todo list.

> But all that
> should wait until after TESTFRAME so we can test it.


And on that we also agree, so no further discussion needed until after
TESTFRAME.

BTW, did you do the soldering ron test I asked you to do?


RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160823/176f3c82/attachment.bin>

From esr at thyrsus.com  Wed Aug 24 00:07:21 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Tue, 23 Aug 2016 20:07:21 -0400
Subject: PPS undersampling
In-Reply-To: <20160823164332.4d8e370f@spidey.rellim.com>
References: <20160823231304.82BB5406057@ip-64-139-1-69.sjc.megapath.net>
 <20160823233255.GB16539@thyrsus.com>
 <20160823164332.4d8e370f@spidey.rellim.com>
Message-ID: <20160824000721.GD16539@thyrsus.com>

Gary E. Miller <gem at rellim.com>:
> Sadly, PPS with the current algorithm should bump up to sampling every
> one-half second.  Some even want 10 Hz or even 100 Hz.
> 
> Let us leave it on the long-term to-do list and not hash over it now.
> The subject is complicated and contentious.  Way too many moving parts
> in a core chunk of the code to touch without TESTFRAME.  I also expect
> TESTFRAME will smoke out some of these issues and also have its own
> different timer loop needs.

Hm. I think I get it - and you've just added a pretty powerful reason to
eventually pull the refclocks into a separate daemon by telling me I need to
get ntpd itself out of the PPS-watching business entirely in order to get
rid of that timer.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 811 bytes
Desc: Digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160823/127dacc2/attachment.bin>

From gem at rellim.com  Wed Aug 24 00:16:12 2016
From: gem at rellim.com (Gary E. Miller)
Date: Tue, 23 Aug 2016 17:16:12 -0700
Subject: PPS undersampling
In-Reply-To: <20160824000721.GD16539@thyrsus.com>
References: <20160823231304.82BB5406057@ip-64-139-1-69.sjc.megapath.net>
 <20160823233255.GB16539@thyrsus.com>
 <20160823164332.4d8e370f@spidey.rellim.com>
 <20160824000721.GD16539@thyrsus.com>
Message-ID: <20160823171612.57bf480c@spidey.rellim.com>

Yo Eric!

On Tue, 23 Aug 2016 20:07:21 -0400
"Eric S. Raymond" <esr at thyrsus.com> wrote:

> Gary E. Miller <gem at rellim.com>:
> > Sadly, PPS with the current algorithm should bump up to sampling
> > every one-half second.  Some even want 10 Hz or even 100 Hz.
> > 
> > Let us leave it on the long-term to-do list and not hash over it
> > now. The subject is complicated and contentious.  Way too many
> > moving parts in a core chunk of the code to touch without
> > TESTFRAME.  I also expect TESTFRAME will smoke out some of these
> > issues and also have its own different timer loop needs.  
> 
> Hm. I think I get it - and you've just added a pretty powerful reason
> to eventually pull the refclocks into a separate daemon by telling me
> I need to get ntpd itself out of the PPS-watching business entirely
> in order to get rid of that timer.

Sadly that does not solve the problem.  You still have the problem of 
how to watch the PPS watcher.  :-)

There will be many ways to skin this cat, but first we need to fatten up
this cat, obtain the other ingredients of the stew, and boil the water.

So let us leave the fun stuff of ripping up the algorithms until after
TESTFRAME.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160823/79515670/attachment.bin>

From esr at thyrsus.com  Wed Aug 24 12:30:01 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Wed, 24 Aug 2016 08:30:01 -0400
Subject: pool vs nopeer
In-Reply-To: <20160824063646.161B6406057@ip-64-139-1-69.sjc.megapath.net>
References: <20160824063646.161B6406057@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160824123001.GA20769@thyrsus.com>

Copying Mark and devel, everyone should know about the task dependencies.

Hal Murray <hmurray at megapathdsl.net>:
> I think [the fact that the pool keyword can't combine with nopeer is] a bug.
> 
> It should be reasonable to find, but may take a while to learn that chunk of 
> code.

You are in a maze of twisty little dependencies, all different.

We should change the client-start page and the Stratum-1-Microserver-HOWTO
example to use 'pool', but that should wait on the pool/nopeer bug being
fixed.

Fixing the pool/nopeer bug will have to be done twice unless we land
Daniel's protocol-machine refactoring.  Might have to be done *three* times
if his redesign of the access-control options requires it.

Daniel's protocol-machine refactoring is a big, potentially
incompatible change.  Which, now that 0.9.4 is shipped, we should land
anyway.  But verifying rigorously that it doesn't introduce a
regression will require TESTFRAME.

Conclusion: TESTFRAME just went from "we need it someday" to critical-path.
Well, we were preparing for another swing at it anyway.

Mark, I think this dependency chain defines our targets for 0.9.5.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From christian.ehrhardt at canonical.com  Wed Aug 24 14:29:31 2016
From: christian.ehrhardt at canonical.com (Christian Ehrhardt)
Date: Wed, 24 Aug 2016 16:29:31 +0200
Subject: Which interfaces are important today to define a time-hardware snap
 interface
Message-ID: <CAATJJ0LGx_5HFRZ4fZekzfJ8WcUeHgjNAuwv4KVyqGb-fJd3Kw@mail.gmail.com>

Hi,
I'm back from PTO and continued on the snap packaging of ntpsec.
Background details on interfaces:
https://github.com/snapcore/snapd/blob/master/docs/interfaces.md

Based on grep -Hrn '\/dev\/' | awk '{gsub(".*/dev/", "/dev/"); print $1}' |
sort |uniq
I tried to summarize what an "time-hardware" interface for snappy might be.

TL;DR - think of it as a list of things allowed to be accessed

I streamlined the list a bit for readability.
I removed some on my own like /dev/kmem being the opposite of isolation :-)
But please let me know if I forgot something.

/dev/acts%d+
/dev/cuaa%d+
/dev/device%d+
/dev/dumbclock%d+
/dev/gps%d+
/dev/gpspps%d+
/dev/hpgps%d+
/dev/icom
/dev/jjy%d+
/dev/neoclock4x-%d+
/dev/oncore.pps.%d+
/dev/oncore.serial.%d+
/dev/palisade%d+
/dev/pps%d+
/dev/refclock-%d+
/dev/spectracom%d+
/dev/trimble%d+
/dev/true%d"
/dev/zyfer%d+

- Do we also need like /dev/ttyS%d+ ?
- I hope you could suggest if we could drop some of these by being no more
important today or in general?

To be honest - I personally only ever "met" /dev/pps, but then i'm no NTP
expert so please advise what is would be reasonable to define such an
interface.
If I look at the usually used apparmor profile it only lists /dev/pps as
well, but provides an option to add own devices - so there might be some of
the list above still important.

FWIW - if in doubt I'd prefer to better keep the list short.



-- 
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160824/5c78f925/attachment.html>

From esr at thyrsus.com  Wed Aug 24 15:36:17 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Wed, 24 Aug 2016 11:36:17 -0400
Subject: Which interfaces are important today to define a time-hardware
 snap interface
In-Reply-To: <CAATJJ0LGx_5HFRZ4fZekzfJ8WcUeHgjNAuwv4KVyqGb-fJd3Kw@mail.gmail.com>
References: <CAATJJ0LGx_5HFRZ4fZekzfJ8WcUeHgjNAuwv4KVyqGb-fJd3Kw@mail.gmail.com>
Message-ID: <20160824153617.GA22335@thyrsus.com>

Christian Ehrhardt <christian.ehrhardt at canonical.com>:
> Hi,
> I'm back from PTO and continued on the snap packaging of ntpsec.
> Background details on interfaces:
> https://github.com/snapcore/snapd/blob/master/docs/interfaces.md
> 
> Based on grep -Hrn '\/dev\/' | awk '{gsub(".*/dev/", "/dev/"); print $1}' |
> sort |uniq
> I tried to summarize what an "time-hardware" interface for snappy might be.
> 
> TL;DR - think of it as a list of things allowed to be accessed
> 
> I streamlined the list a bit for readability.
> I removed some on my own like /dev/kmem being the opposite of isolation :-)
> But please let me know if I forgot something.
> 
> /dev/acts%d+
> /dev/cuaa%d+
> /dev/device%d+
> /dev/dumbclock%d+
> /dev/gps%d+
> /dev/gpspps%d+
> /dev/hpgps%d+
> /dev/icom
> /dev/jjy%d+
> /dev/neoclock4x-%d+
> /dev/oncore.pps.%d+
> /dev/oncore.serial.%d+
> /dev/palisade%d+
> /dev/pps%d+
> /dev/refclock-%d+
> /dev/spectracom%d+
> /dev/trimble%d+
> /dev/true%d"
> /dev/zyfer%d+
> 
> - Do we also need like /dev/ttyS%d+ ?

For full generality we need /dev/ttyS%d+ and /dev/ttyUSB%d+.  These might
be referenced by a refclock path option.

You also need to add /dev/refclockpps-%d+, the generic driver requires it.

> - I hope you could suggest if we could drop some of these by being no more
> important today or in general?

You can certainly drop /dev/icom, that was used only by the
now-removed audio drivers.  Also /dev/palisade%d+, that was the old
name for /dev/trimble%d+.
 
> FWIW - if in doubt I'd prefer to better keep the list short.

So would I, but we can't if the snapified version is going to
support refclocks.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From gem at rellim.com  Wed Aug 24 17:26:56 2016
From: gem at rellim.com (Gary E. Miller)
Date: Wed, 24 Aug 2016 10:26:56 -0700
Subject: Which interfaces are important today to define a time-hardware
 snap interface
In-Reply-To: <CAATJJ0LGx_5HFRZ4fZekzfJ8WcUeHgjNAuwv4KVyqGb-fJd3Kw@mail.gmail.com>
References: <CAATJJ0LGx_5HFRZ4fZekzfJ8WcUeHgjNAuwv4KVyqGb-fJd3Kw@mail.gmail.com>
Message-ID: <20160824102656.67e1f35b@spidey.rellim.com>

Yo Christian!

On Wed, 24 Aug 2016 16:29:31 +0200
Christian Ehrhardt <christian.ehrhardt at canonical.com> wrote:

> I tried to summarize what an "time-hardware" interface for snappy
> - Do we also need like /dev/ttyS%d+ ?

A lot of procedures on the internet refer to /dev/ttyS%d, /dev/ttyUSB%d,
and /dev/ttyAMA%d

So for generatlity those would be nice.

> To be honest - I personally only ever "met" /dev/pps, but then i'm no
> NTP expert so please advise what is would be reasonable to define
> such an interface.

/dev/pps%d is often autogenerated when needed by gpsd or ldattach.  So
many NTPP users never know it is there.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160824/be7531d6/attachment.bin>

From gem at rellim.com  Wed Aug 24 18:27:51 2016
From: gem at rellim.com (Gary E. Miller)
Date: Wed, 24 Aug 2016 11:27:51 -0700
Subject: pool vs nopeer
In-Reply-To: <20160824123001.GA20769@thyrsus.com>
References: <20160824063646.161B6406057@ip-64-139-1-69.sjc.megapath.net>
 <20160824123001.GA20769@thyrsus.com>
Message-ID: <20160824112751.537cf34c@spidey.rellim.com>

Yo Eric!

On Wed, 24 Aug 2016 08:30:01 -0400
"Eric S. Raymond" <esr at thyrsus.com> wrote:

> Conclusion: TESTFRAME just went from "we need it someday" to
> critical-path. Well, we were preparing for another swing at it anyway.

+1.  And now that you are getting a gut feel for what ntpd does from
the ntpviz and Pi projects, you are ready for it.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160824/870b3434/attachment.bin>

From hmurray at megapathdsl.net  Wed Aug 24 22:22:50 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Wed, 24 Aug 2016 15:22:50 -0700
Subject: pool vs nopeer
In-Reply-To: Message from "Eric S. Raymond" <esr@thyrsus.com>
 of "Wed, 24 Aug 2016 08:30:01 EDT." <20160824123001.GA20769@thyrsus.com>
Message-ID: <20160824222250.D758840605C@ip-64-139-1-69.sjc.megapath.net>


esr at thyrsus.com said:
> Fixing the pool/nopeer bug will have to be done twice unless we land
> Daniel's protocol-machine refactoring.  Might have to be done *three* times
> if his redesign of the access-control options requires it.

That's the easy part.  First we have to understand the code and find the bug.


-- 
These are my opinions.  I hate spam.




From dfoxfranke at gmail.com  Wed Aug 24 23:04:51 2016
From: dfoxfranke at gmail.com (Daniel Franke)
Date: Wed, 24 Aug 2016 19:04:51 -0400
Subject: pool vs nopeer
In-Reply-To: <20160824222250.D758840605C@ip-64-139-1-69.sjc.megapath.net>
References: <esr@thyrsus.com> <20160824123001.GA20769@thyrsus.com>
 <20160824222250.D758840605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <CAJm83bC_jGohoSewX7Np00JkOYn=Eqdc1nAhUZ254jf99upMdw@mail.gmail.com>

On 8/24/16, Hal Murray <hmurray at megapathdsl.net> wrote:
> That's the easy part.  First we have to understand the code and find the
> bug.

It should already be fixed in my branch. If you wanted to unfix it, at
ntp_proto.c:415 change MDF_ACAST to (MDF_ACAST | MDF_POOL).

From esr at thyrsus.com  Thu Aug 25 00:08:08 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Wed, 24 Aug 2016 20:08:08 -0400
Subject: pool vs nopeer
In-Reply-To: <20160824222250.D758840605C@ip-64-139-1-69.sjc.megapath.net>
References: <esr@thyrsus.com> <20160824123001.GA20769@thyrsus.com>
 <20160824222250.D758840605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160825000808.GA25912@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> 
> esr at thyrsus.com said:
> > Fixing the pool/nopeer bug will have to be done twice unless we land
> > Daniel's protocol-machine refactoring.  Might have to be done *three* times
> > if his redesign of the access-control options requires it.
> 
> That's the easy part.  First we have to understand the code and find the bug.

Right.  Worst case, understand *three different* stretches of code.  That's
why I think landing dfranke's refactor first is the right way to do things.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From esr at thyrsus.com  Thu Aug 25 00:09:37 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Wed, 24 Aug 2016 20:09:37 -0400
Subject: pool vs nopeer
In-Reply-To: <CAJm83bC_jGohoSewX7Np00JkOYn=Eqdc1nAhUZ254jf99upMdw@mail.gmail.com>
References: <esr@thyrsus.com> <20160824123001.GA20769@thyrsus.com>
 <20160824222250.D758840605C@ip-64-139-1-69.sjc.megapath.net>
 <CAJm83bC_jGohoSewX7Np00JkOYn=Eqdc1nAhUZ254jf99upMdw@mail.gmail.com>
Message-ID: <20160825000937.GB25912@thyrsus.com>

Daniel Franke <dfoxfranke at gmail.com>:
> On 8/24/16, Hal Murray <hmurray at megapathdsl.net> wrote:
> > That's the easy part.  First we have to understand the code and find the
> > bug.
> 
> It should already be fixed in my branch. If you wanted to unfix it, at
> ntp_proto.c:415 change MDF_ACAST to (MDF_ACAST | MDF_POOL).

OK, that's good news.

Do you know how to backport that fix?  It's not a big deal if you don't,
but would decouple some tasks.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From dfoxfranke at gmail.com  Thu Aug 25 00:16:24 2016
From: dfoxfranke at gmail.com (Daniel Franke)
Date: Wed, 24 Aug 2016 20:16:24 -0400
Subject: pool vs nopeer
In-Reply-To: <20160825000937.GB25912@thyrsus.com>
References: <esr@thyrsus.com> <20160824123001.GA20769@thyrsus.com>
 <20160824222250.D758840605C@ip-64-139-1-69.sjc.megapath.net>
 <CAJm83bC_jGohoSewX7Np00JkOYn=Eqdc1nAhUZ254jf99upMdw@mail.gmail.com>
 <20160825000937.GB25912@thyrsus.com>
Message-ID: <CAJm83bA+we=69Y-55bRun437dEwNKv7MJD5wkB8HW2PwgYVN1Q@mail.gmail.com>

Not straightforwardly, but the change you originally proposed months ago
when you first observed this bug was correct AFAICT.

On Aug 24, 2016 8:10 PM, "Eric S. Raymond" <esr at thyrsus.com> wrote:

> Daniel Franke <dfoxfranke at gmail.com>:
> > On 8/24/16, Hal Murray <hmurray at megapathdsl.net> wrote:
> > > That's the easy part.  First we have to understand the code and find
> the
> > > bug.
> >
> > It should already be fixed in my branch. If you wanted to unfix it, at
> > ntp_proto.c:415 change MDF_ACAST to (MDF_ACAST | MDF_POOL).
>
> OK, that's good news.
>
> Do you know how to backport that fix?  It's not a big deal if you don't,
> but would decouple some tasks.
> --
>                 <a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160824/9d569c22/attachment.html>

From esr at thyrsus.com  Thu Aug 25 04:19:46 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Thu, 25 Aug 2016 00:19:46 -0400
Subject: Stratum one autonomy and assumptions about GPS
Message-ID: <20160825041946.GA26984@thyrsus.com>

This was going to be a note to just Hal originally, but it will do the
rest of the team no harm to know more about the scenarios and
assumptions driving some of my design choices.

Hal objected (off list) to me drawing a conclusion from today's
offset multiplot that check servers aren't necessary when you have
a local GPS - a Stratum 1 really can run autonomously. He said,
correctly of course, that the check servers aren't there to improve
time accuracy when the GPS has sat lock, but to backstop the GPS when
it flakes out.

I shall now discuss three interlocking reasons this possibility does
not loom as large in my mind as it does in Hal's.

1. GPS outage length and frequencies are decreasing

I've been watching GPS technology evolve since I took the maintainer's
baton of GPSD in 2004.  A very clear trend is that GPS reliability has
gone way up. Weak-signal sensitivity keeps improving, the number of
channels in the engines has gone from 1 or 2 to 12 and (with newer
devices) up in to 18-24 range.

It's now pretty normal for state-of-market chips like the ublox-8 or
MTK3339 to be able to operate *continuously* indoors - I know this
because I have a row of six of them on the Official Windowsill of Mad
Science and can see their blinkenlights in my upper peripheral vision when
I look at my main monitor.  And this is with tall trees right outside.

If your expectations are formed by experience with older hardware,
you're going to overestimate the frequency and mean duration of
outages significantly.  And things will get better - in fact, are
still getting better fast enough that improvements are visible on
roughly 6-month timescales.

(Case in point: Gary is playing with a recent chip that does *centimeter*
accuracy in *real time* - not postprocessed.)

2. The autonomy scenarios I think about are not hobbyist-budget productions

The user story I have in mind when I think about supporting autonomous
Stratum 1 is not the Official Windowsill of Mad Science. It's a big
data center, an oil tanker, or a military base - an organization that
wants to firewall out NTP packets and can afford to put a radio mast (or
three) on a roof.

Thus we get to assume good skyview from the get-go.  We also get to assume
high-end GPS hardware and extended antennas and other good things. It's not
like paying for quality in this space is expensive - by the standards of
anyone who can out up a radio mast it's actually trivial.

3. There's a lower bound below which outages don't matter; we may be there.

Any given fixed accuracy target for deviation from UTC, combined with a maximum
crystal drift rate, defines a longest tolerable GPS outage. 

When I think about this, I consider 10ms total deviation from UTC as
the target for WAN service.  Let's say your local clock drifts by 3ms
per hour.  Then you can tolerate a GPS outage of a bit over 3 hours
before you will start shipping bad time.  I have *never* seen a GPS
outage that long, even with older hardware.

My observed worst case these days is about 20 minutes, and that is *rare* -
usually only when cold-booting.  For that to be intolerably long, the local
clock would have to drift by 30ms per hour.  Actually I seldom see outages
longer than 5 minutes, which implies a critical drift of 120ms/hr.  And
this is with my consumer-grade hardware, indoors of a windowsill with
trees outside.

We may already be at a technological place where GPS outages don't bust the
tolerable-error budget, even with cheap hardware. If we aren't, we'll
probably be there soon.  One of my medium-term agenda items is to measure
and see.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From esr at thyrsus.com  Thu Aug 25 09:05:57 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Thu, 25 Aug 2016 05:05:57 -0400
Subject: Internal Clock Drift Estimation in Computer Clusters
Message-ID: <20160825090557.GA28887@thyrsus.com>

ABSTRACT: Most computers have several high-resolution timing sources,
from the programmable interrupt timer to the cycle counter. Yet, even
at a precision of one cycle in ten millions, clocks may drift
significantly in a single second at a clock frequency of several
GHz. When tracing the low-level system events in computer clusters,
such as packet sending or reception, each computer system records its
own events using an internal clock. In order to properly understand
the global system behavior and performance, as reported by the events
recorded on each computer, it is important to estimate precisely the
clock differences and drift between the different computers in the
system. This article studies the clock precision and stability of
several computer systems, with different architectures. It also
studies the typical network delay characteristics, since time
synchronization algorithms rely on the exchange of network packets and
are dependent on the symmetry of the delays. A very precise clock,
based on the atomic time provided by the GPS satellite network, was
used as a reference to measure clock drifts and network delays. The
results obtained are of immediate use to all applications which depend
on computer clocks or network time synchronization accuracy.

http://www.hindawi.com/journals/jcnc/2008/583162/
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From hmurray at megapathdsl.net  Thu Aug 25 09:29:54 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Thu, 25 Aug 2016 02:29:54 -0700
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: Message from "Eric S. Raymond" <esr@thyrsus.com>
 of "Thu, 25 Aug 2016 00:19:46 EDT." <20160825041946.GA26984@thyrsus.com>
Message-ID: <20160825092955.051A9406060@ip-64-139-1-69.sjc.megapath.net>


There is lots of good stuff there.

But...

It sounds like you are trying to rationalize running without any network 
servers.

Even if you do collect lots of data, you can't extrapolate very far from a 
sample of one.

There are many ways to screw up.  Even if you get a lot of data, there will 
be more things to consider.

Examples:

Firmware will have bugs.  OSes will have bugs.  Software will have bugs.
  I have a unit that occasionally gets off by a second with gpsd.
  (It's the one that starts sending NMEA just before the top of a second.)

The GPS operators may screw up.
A trucker with a jammer may park in front of your house.
Rain/snow may land on your trees.
Somebody will try to use their wonderful modern unit farther from the window 
or deeper in an urban canyon.

Your 10ms error budget may be fine for some applications but others may want 
better.

---------

Actually, for short outages, it may be better to ignore network servers and 
just coast with the current drift setting.  That would be a good experiment.  
That probably depends on the unit dying cleanly rather than sliding off in 
some direction before it gives up.





-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Thu Aug 25 10:00:50 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Thu, 25 Aug 2016 03:00:50 -0700
Subject: Internal Clock Drift Estimation in Computer Clusters
In-Reply-To: Message from "Eric S. Raymond" <esr@thyrsus.com>
 of "Thu, 25 Aug 2016 05:05:57 EDT." <20160825090557.GA28887@thyrsus.com>
Message-ID: <20160825100050.17066406060@ip-64-139-1-69.sjc.megapath.net>


That's from 2008.

Was there anything in there that you thought was particularly interesting or 
surprising?

-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Thu Aug 25 10:03:19 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Thu, 25 Aug 2016 06:03:19 -0400
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: <20160825092955.051A9406060@ip-64-139-1-69.sjc.megapath.net>
References: <esr@thyrsus.com> <20160825041946.GA26984@thyrsus.com>
 <20160825092955.051A9406060@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160825100319.GA29235@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> It sounds like you are trying to rationalize running without any network 
> servers.

One of my agenda items for a while has been to develop a protocol for
places that want to firewall out NTP for security reasons - that is, a
set of practices and error estimates based on documented assumptions.

It's no problem for *me* to add check servers to my config, but if
you're - say - an overseas military base in a country where Internet
is dodgy or might be shut down occasionally by the local government,
that's a different matter.

I believe, given NTPsec's mission, that we want to have something
contructive to say about that use case.

> Even if you do collect lots of data, you can't extrapolate very far from a 
> sample of one.

True.  But we can use experiments to get a feel for the scale of the
problem and the magnitude of different parts of the error budget.  Up
to now there has been very little documentation of this sort of thing
and a lot of handwaving. I want to fix that.

> There are many ways to screw up.  Even if you get a lot of data, there will 
> be more things to consider.
> 
> Examples:
> 
> Firmware will have bugs.  OSes will have bugs.  Software will have bugs.
>   I have a unit that occasionally gets off by a second with gpsd.
>   (It's the one that starts sending NMEA just before the top of a second.)

Right.  This sort of thing you hedge against with tell-me-three-times.
(Remember that the user story assumes a big hardware budget.)

> The GPS operators may screw up.
> A trucker with a jammer may park in front of your house.
> Rain/snow may land on your trees.
> Somebody will try to use their wonderful modern unit farther from the window 
> or deeper in an urban canyon.

I don't think I need to care about specific causality much.  The overriding
question is what the statistical distribution of outage lengths is.

(Rain/snow on your trees.  Why is this special? Does it increase multipath
reflections or something?)

> Your 10ms error budget may be fine for some applications but others may want 
> better.

In general they're going to need to go with PTP over a LAN. I'm focused on
Stratum 1 for WAN time service.

> Actually, for short outages, it may be better to ignore network servers and 
> just coast with the current drift setting.  That would be a good experiment.  
> That probably depends on the unit dying cleanly rather than sliding off in 
> some direction before it gives up.

That's one of a large number of experiments I'd like to run.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From dfoxfranke at gmail.com  Thu Aug 25 11:00:58 2016
From: dfoxfranke at gmail.com (Daniel Franke)
Date: Thu, 25 Aug 2016 07:00:58 -0400
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: <20160825100319.GA29235@thyrsus.com>
References: <esr@thyrsus.com> <20160825041946.GA26984@thyrsus.com>
 <20160825092955.051A9406060@ip-64-139-1-69.sjc.megapath.net>
 <20160825100319.GA29235@thyrsus.com>
Message-ID: <CAJm83bBZEoMC7G=HFgNPkFdv9XDUm8-JvTEktJtK4HgXMHqEfg@mail.gmail.com>

On 8/25/16, Eric S. Raymond <esr at thyrsus.com> wrote:
> (Remember that the user story assumes a big hardware budget.)

A big hardware budget should have room for a rubidium GPSDO,
in which case you can get away with freerunning for a lot longer.

From dan-ntp at drown.org  Thu Aug 25 15:23:50 2016
From: dan-ntp at drown.org (Dan Drown)
Date: Thu, 25 Aug 2016 10:23:50 -0500
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: <20160825101549.Horde.n3e0uFvL5xJ-fPfTT7gC_at@mail.drown.org>
References: <20160825041946.GA26984@thyrsus.com>
 <20160825101549.Horde.n3e0uFvL5xJ-fPfTT7gC_at@mail.drown.org>
Message-ID: <20160825102350.Horde.X9OWWeThDAj1vqp7_rq_Psa@mail.drown.org>

Quoting "Eric S. Raymond" <esr at thyrsus.com>:
:
> Hal objected (off list) to me drawing a conclusion from today's
> offset multiplot that check servers aren't necessary when you have
> a local GPS - a Stratum 1 really can run autonomously. He said,
> correctly of course, that the check servers aren't there to improve
> time accuracy when the GPS has sat lock, but to backstop the GPS when
> it flakes out.
:
> We may already be at a technological place where GPS outages don't bust the
> tolerable-error budget, even with cheap hardware. If we aren't, we'll
> probably be there soon.  One of my medium-term agenda items is to measure
> and see.

I had setup a test along these lines a week ago:
https://blog.dan.drown.org/gps-pps-drift-when-it-has-no-signal/

The GPS module I'm using still outputs PPS even when it loses lock.
That won't be true for every GPS module.

My results were a long term average of 250us-500us lost per day
(~3-6ppb).  This surprised me because of how low it was, I assume I
got lucky with the long term average temperature of the GPS module.

ntpsec finally rejected the PPS at a 2ms offset.  I think I could
change that with the "tos mindist" setting (which was at its default
of 1ms), but I haven't verified that.

Of course, this data doesn't apply to all GPS outage situations.  But
it does suggest that holdover within 30ms is easy for the "can't hear
the satellites for a few hours" case.



From esr at thyrsus.com  Thu Aug 25 16:05:47 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Thu, 25 Aug 2016 12:05:47 -0400
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: <CAJm83bBZEoMC7G=HFgNPkFdv9XDUm8-JvTEktJtK4HgXMHqEfg@mail.gmail.com>
References: <esr@thyrsus.com> <20160825041946.GA26984@thyrsus.com>
 <20160825092955.051A9406060@ip-64-139-1-69.sjc.megapath.net>
 <20160825100319.GA29235@thyrsus.com>
 <CAJm83bBZEoMC7G=HFgNPkFdv9XDUm8-JvTEktJtK4HgXMHqEfg@mail.gmail.com>
Message-ID: <20160825160547.GA31039@thyrsus.com>

Daniel Franke <dfoxfranke at gmail.com>:
> On 8/25/16, Eric S. Raymond <esr at thyrsus.com> wrote:
> > (Remember that the user story assumes a big hardware budget.)
> 
> A big hardware budget should have room for a rubidium GPSDO,
> in which case you can get away with freerunning for a lot longer.

Yes, that did occur to me.  Or, a TCxO like the one in the $300 ublox-6
eval kit I have - not as good holdover as a GPSDO but way better than
an uncompensated GPS.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From esr at thyrsus.com  Thu Aug 25 16:06:41 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Thu, 25 Aug 2016 12:06:41 -0400
Subject: Internal Clock Drift Estimation in Computer Clusters
In-Reply-To: <20160825100050.17066406060@ip-64-139-1-69.sjc.megapath.net>
References: <esr@thyrsus.com> <20160825090557.GA28887@thyrsus.com>
 <20160825100050.17066406060@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160825160641.GB31039@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> That's from 2008.
> 
> Was there anything in there that you thought was particularly interesting or 
> surprising?

The actual numbers.  I've seen the logic before.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From esr at thyrsus.com  Thu Aug 25 16:11:26 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Thu, 25 Aug 2016 12:11:26 -0400
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: <20160825102350.Horde.X9OWWeThDAj1vqp7_rq_Psa@mail.drown.org>
References: <20160825041946.GA26984@thyrsus.com>
 <20160825101549.Horde.n3e0uFvL5xJ-fPfTT7gC_at@mail.drown.org>
 <20160825102350.Horde.X9OWWeThDAj1vqp7_rq_Psa@mail.drown.org>
Message-ID: <20160825161126.GC31039@thyrsus.com>

Dan Drown <dan-ntp at drown.org>:
> >We may already be at a technological place where GPS outages don't bust the
> >tolerable-error budget, even with cheap hardware. If we aren't, we'll
> >probably be there soon.  One of my medium-term agenda items is to measure
> >and see.
> 
> I had setup a test along these lines a week ago:
> https://blog.dan.drown.org/gps-pps-drift-when-it-has-no-signal/
> 
> The GPS module I'm using still outputs PPS even when it loses lock.
> That won't be true for every GPS module.
> 
> My results were a long term average of 250us-500us lost per day
> (~3-6ppb).  This surprised me because of how low it was, I assume I
> got lucky with the long term average temperature of the GPS module.
> 
> ntpsec finally rejected the PPS at a 2ms offset.  I think I could
> change that with the "tos mindist" setting (which was at its default
> of 1ms), but I haven't verified that.
> 
> Of course, this data doesn't apply to all GPS outage situations.  But
> it does suggest that holdover within 30ms is easy for the "can't hear
> the satellites for a few hours" case.

Very interesting.

I have a USB thermometer on order, they're cheap.  Might I suggest you
get one and repeat this experiment, actually plotting your temperature
variation?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From Stromeko at nexgo.de  Thu Aug 25 18:33:36 2016
From: Stromeko at nexgo.de (Achim Gratz)
Date: Thu, 25 Aug 2016 20:33:36 +0200
Subject: Stratum one autonomy and assumptions about GPS
References: <20160825041946.GA26984@thyrsus.com>
 <20160825101549.Horde.n3e0uFvL5xJ-fPfTT7gC_at@mail.drown.org>
 <20160825102350.Horde.X9OWWeThDAj1vqp7_rq_Psa@mail.drown.org>
Message-ID: <8760qozojj.fsf@Rainer.invalid>

Dan Drown writes:
> The GPS module I'm using still outputs PPS even when it loses lock.
> That won't be true for every GPS module.

I also have a NavSpark mini attached at the moment and I was surprised
it kept outputting the PPS when losing 3D lock (it stops blinking the
LED).  It seems semi-documented to keep the PPS on if it at least sees
one satellite, although the corresponding settings can't be read or
altered (that seems resereved for the timing firmware, on the mini I
just get an NACK response).  Losing 3D lock happens about once or twice
a day for me at the moment, probably because of a somewhat too agressive
mask settings.  It usually lasts less than half an hour with negligible
effect on ntp.

> My results were a long term average of 250us-500us lost per day
> (~3-6ppb).  This surprised me because of how low it was, I assume I
> got lucky with the long term average temperature of the GPS module.

Recently I've had an extended loss of lock and it was in fact drifting
by about 1ms/hour.  It may have been the result of some sort of power
fault, since I've needed to reboot the Pi and powercycle the mini to
recover.

> Of course, this data doesn't apply to all GPS outage situations.  But
> it does suggest that holdover within 30ms is easy for the "can't hear
> the satellites for a few hours" case.

You might want to play with the elevation and CNR mask settings and
check what happens with just one or two satellites considered in view.
I think the degradation there is actually larger than when it is in
complete holdover.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra


From Stromeko at nexgo.de  Thu Aug 25 18:51:43 2016
From: Stromeko at nexgo.de (Achim Gratz)
Date: Thu, 25 Aug 2016 20:51:43 +0200
Subject: Stratum one autonomy and assumptions about GPS
References: <20160825041946.GA26984@thyrsus.com>
Message-ID: <871t1cznpc.fsf@Rainer.invalid>

Eric S. Raymond writes:
> I shall now discuss three interlocking reasons this possibility does
> not loom as large in my mind as it does in Hal's.
[?]
> When I think about this, I consider 10ms total deviation from UTC as
> the target for WAN service.  Let's say your local clock drifts by 3ms
> per hour.  Then you can tolerate a GPS outage of a bit over 3 hours
> before you will start shipping bad time.  I have *never* seen a GPS
> outage that long, even with older hardware.

The MTTR doesn't tell you anbything about the tail of that distribution,
which is more likely caused by other unlikely events, like some squirrel
gnawing through the antenna cable, a cat taking it'?s hunting instinct
out on the GPS puck or (since we're talking about long-running services
on hardware that's really not designed for that as the rasPi) a memory
corruption getting to some sensitive spot.

> My observed worst case these days is about 20 minutes, and that is *rare* -
> usually only when cold-booting.  For that to be intolerably long, the local
> clock would have to drift by 30ms per hour.  Actually I seldom see outages
> longer than 5 minutes, which implies a critical drift of 120ms/hr.  And
> this is with my consumer-grade hardware, indoors of a windowsill with
> trees outside.

I've been running my rasPi 2 more or less non-stop for about a year, save
for the reboots when a new kernel gets installed.  It has crashed or
otherwise shown signs of memory corruption three times in that period.
The B+ hasn't had any such problems, so my guess is these are related to
the PSU.  The point is, you'll have to expect that some portion of these
systems will be run unsupervised for quite some time, so they need to
look after themselves.

> We may already be at a technological place where GPS outages don't bust the
> tolerable-error budget, even with cheap hardware. If we aren't, we'll
> probably be there soon.  One of my medium-term agenda items is to measure
> and see.

Forget about GPS outages.  If you want to connect these time servers to
the net, they should have some form of sanity checking that's
independent from their primary time source and take them off the net as
soon as something looks fishy.  I frequently see .GPS. falsetickers in
NTPpool (actually the last month I haven't seen them, so maybe they've
been kicked out) that are/were 50ms and more out, which tells me they
have either not bothered to determine the fudge factors or run without
PPS.  You don't want to see that kind of stratum-1 clock on a larger
scale.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada


From esr at thyrsus.com  Thu Aug 25 19:37:32 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Thu, 25 Aug 2016 15:37:32 -0400
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: <871t1cznpc.fsf@Rainer.invalid>
References: <20160825041946.GA26984@thyrsus.com>
 <871t1cznpc.fsf@Rainer.invalid>
Message-ID: <20160825193732.GA4168@thyrsus.com>

Achim Gratz <Stromeko at nexgo.de>:
> Forget about GPS outages.  If you want to connect these time servers to
> the net, they should have some form of sanity checking that's
> independent from their primary time source and take them off the net as
> soon as something looks fishy.

What kind of sanity checking do you recommend?

>                             I frequently see .GPS. falsetickers in
> NTPpool (actually the last month I haven't seen them, so maybe they've
> been kicked out) that are/were 50ms and more out, which tells me they
> have either not bothered to determine the fudge factors or run without
> PPS.  You don't want to see that kind of stratum-1 clock on a larger
> scale.

My recipe/HOWTO does emphasize that you need 1PPS for Stratum 1 service.
Beyond that I don't know there's much I can do to head off operator error.

How do you fudge a 1PPS source?  What do you have that you consider
more accurate?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From Stromeko at nexgo.de  Thu Aug 25 20:47:26 2016
From: Stromeko at nexgo.de (Achim Gratz)
Date: Thu, 25 Aug 2016 22:47:26 +0200
Subject: Stratum one autonomy and assumptions about GPS
References: <20160825041946.GA26984@thyrsus.com>
 <871t1cznpc.fsf@Rainer.invalid> <20160825193732.GA4168@thyrsus.com>
Message-ID: <87wpj4y3s1.fsf@Rainer.invalid>

Eric S. Raymond writes:
> Achim Gratz <Stromeko at nexgo.de>:
>> Forget about GPS outages.  If you want to connect these time servers to
>> the net, they should have some form of sanity checking that's
>> independent from their primary time source and take them off the net as
>> soon as something looks fishy.
>
> What kind of sanity checking do you recommend?

I'd think that keeping two or three known good servers as noselect would
go a long way in that scenario.  Could even be a pool statement,
although personally I'd opt for at least five servers in that case.  If
more than one of those servers are more than 10ms either way from your
time, something is going on that shouldn't be.  If it's other stratum-1
sources then you should likely see below 1ms (2ms tops) unless the
network totally flakes out.  Since ntpd doesn't do anything with
noselect sources the actual logic of taking the stratum-1 offline in
case of problems would need to be delegated to some monitor process.

The message offset relative to the PPS is recorded somewhere in ntpd
(you can set up the extended statistics to spit it out), so if that
starts drifting from the expected value (fudge2) the clock should become
suspect.  The same goes for sudden changes in the local frequency
offset.  Again would need to be an external process that does the health
monitoring.

Another thing would be to have more than one refclock, but ntpd
currently doesn't support that very well.  Part of that is that you
seemingly can't have more than one pps-gpio for whatever reason.  The
other part is that you can't tell ntpd which clock you expect to be more
accurate and how to switch from one to the other.  That would need new
code to support it from within ntpd I think.

As I said before, I'm planning to use a microcontroller to integrate
GPS, DCF77 and maybe an OCXO into a time box.  I'm currently looking at
a controller that has Ethernet w/ PTP support (hardware timestamps), so
it could be a completely autonomous dual-protocol timeserver.  But a
less capable controller might just produce PPS and a serial stream and
connect to a rasPi or other server for a resilient stratum-1.  That
would use the Uni Erlangen format, like the Meinberg clocks.

> My recipe/HOWTO does emphasize that you need 1PPS for Stratum 1 service.
> Beyond that I don't know there's much I can do to head off operator error.

There's no way to prevent stupidity and malice, yes.  But since the goal
is (IIUC) to get more stratum-1 servers onto the net, some education
how not to shoot yourself into the foot is in order.

> How do you fudge a 1PPS source?  What do you have that you consider
> more accurate?

You are only talking about an aligned PPS source, not frequency PPS,
right?

The NMEA driver in PPS mode has noth a PPS offset (fudge1) as well as an
NMEA message offset (fudge2) parameter.  I haven't used the ATOM/prefer
combination in a while, but IIRC, ATOM can be fudged as well.  If the
PPS is done via pps-gpio the offset should be below a few milliseconds.
I can't be sure of my own offset better than about 2?4ms due to the
asymmetry of the DSL line, but to get the same time as the PTB servers I
need to fudge about 1.7ms with the navSpark mini.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada


From hmurray at megapathdsl.net  Thu Aug 25 21:50:19 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Thu, 25 Aug 2016 14:50:19 -0700
Subject: Internal Clock Drift Estimation in Computer Clusters
In-Reply-To: Message from "Eric S. Raymond" <esr@thyrsus.com>
 of "Thu, 25 Aug 2016 12:06:41 EDT." <20160825160641.GB31039@thyrsus.com>
Message-ID: <20160825215019.7EB25406060@ip-64-139-1-69.sjc.megapath.net>

>> That's from 2008.
>> Was there anything in there that you thought was particularly interesting 
or
>> surprising?

> The actual numbers.  I've seen the logic before.

That was a long time ago.  Any numbers are likely to be misleading.  Best to 
collect your own.  How about "Trust, but verify."

PC have two crystals, one for the CPU and one for the RTC/TOY clock.  In the 
old days of that paper, Linux used an interrupt from the RTC clock for 
timekeeping.  The TSC was used to interpolate between ticks but the main 
timekeeping use the RTC crystal.  Current Linux timekeeping uses the TSC 
which is derived from the CPU crystal.

The RTC crystal probably has a much better temp coefficient than the CPU 
crystal and/or it is probably farther from the CPU so gets less temperature 
change under load.

There is another worm in this can.  On most PCs, the CPU clock is "spread" to 
dance under the FCC EMI rules.  I don't have any data on the temperature 
coefficient of that stuff.  (SoC boards like the Raspberry Pi typically don't 
do that.)

They used 100 megabit ethernet.  Anything more advanced than a Raspberry Pi 
has gigabit these days.




-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Thu Aug 25 22:30:25 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Thu, 25 Aug 2016 15:30:25 -0700
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: Message from "Eric S. Raymond" <esr@thyrsus.com>
 of "Thu, 25 Aug 2016 12:11:26 EDT." <20160825161126.GC31039@thyrsus.com>
Message-ID: <20160825223025.152CC406060@ip-64-139-1-69.sjc.megapath.net>


esr at thyrsus.com said:
> I have a USB thermometer on order, they're cheap.  Might I suggest you get
> one and repeat this experiment, actually plotting your temperature
> variation? 

Most CPU chips include a temperature sensor.

For Linux PCs, you need the coretemp module loaded.  The lm-sensors package 
does that.

You can get it by either parsing the output of the sensors command or by 
looking in things like
  /sys/devices/platform/coretemp.0/hwmon/hwmon0/temp2_input

-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Thu Aug 25 23:14:05 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Thu, 25 Aug 2016 16:14:05 -0700
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: Message from "Eric S. Raymond" <esr@thyrsus.com>
 of "Thu, 25 Aug 2016 06:03:19 EDT." <20160825100319.GA29235@thyrsus.com>
Message-ID: <20160825231405.88CCF406060@ip-64-139-1-69.sjc.megapath.net>


esr at thyrsus.com said:
> One of my agenda items for a while has been to develop a protocol for places
> that want to firewall out NTP for security reasons - that is, a set of
> practices and error estimates based on documented assumptions. 

Background data is good.  I think you will get into trouble if you try to 
turn that into useful recommendations.  There are too many assumptions and 
there will be huge differences between places.

> Right.  This sort of thing you hedge against with tell-me-three-times.
> (Remember that the user story assumes a big hardware budget.) 

If you have a big budget, you buy a does-it-all box with GPSDO and NTP 
server, and pay a defense contractor to integrate things.
  http://spectracom.com/products-services/precision-timing

> I don't think I need to care about specific causality much.  The overriding
> question is what the statistical distribution of outage lengths is.

The time pattern depends upon the mechanism.  If your GPS unit fades out 
several times a day you can collect useful statistics.  If your GPS screws up 
due to a 1024 week rollover it's hard to get enough data to work with.

> (Rain/snow on your trees.  Why is this special? Does it increase multipath
> reflections or something?) 

Water attenuates GPS signals.  So wet trees/roofs attenuate more than dry.


>> Your 10ms error budget may be fine for some applications
>> but others may want better. 
> In general they're going to need to go with PTP over a LAN. I'm focused on
> Stratum 1 for WAN time service. 

You can easily get better than 10 ms over a LAN.

With a bit of care, you can do it on a WAN.  An overloaded last-mile link 
will make that harder.

----------

I think you are mixing two uses for good background data.  Earlier you said 
your target was a firewall blocking all NTP traffic to the WAN.  Now you are 
back to running a WAN server.

My bottom line is that collecting data is good but it's going to be hard to 
turn it into useful recommendations.

When you are collecting data, please be sure to get the next layer of 
details: hardware, OS, software, versions...  It's probably best to save the 
log files too.  That part is easy if you save everything.  Then you just need 
to know the date/time when you collected the data.


-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Fri Aug 26 03:23:05 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Thu, 25 Aug 2016 23:23:05 -0400
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: <20160825223025.152CC406060@ip-64-139-1-69.sjc.megapath.net>
References: <esr@thyrsus.com> <20160825161126.GC31039@thyrsus.com>
 <20160825223025.152CC406060@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160826032305.GA9428@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> 
> esr at thyrsus.com said:
> > I have a USB thermometer on order, they're cheap.  Might I suggest you get
> > one and repeat this experiment, actually plotting your temperature
> > variation? 
> 
> Most CPU chips include a temperature sensor.
> 
> For Linux PCs, you need the coretemp module loaded.  The lm-sensors package 
> does that.
> 
> You can get it by either parsing the output of the sensors command or by 
> looking in things like
>   /sys/devices/platform/coretemp.0/hwmon/hwmon0/temp2_input

Is the CPU temp going to be a good approximation of what the clock sees,
though?  I can think of several reasons to doubt it.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From dan-ntp at drown.org  Fri Aug 26 04:05:29 2016
From: dan-ntp at drown.org (Dan Drown)
Date: Thu, 25 Aug 2016 23:05:29 -0500
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: <20160826032305.GA9428@thyrsus.com>
References: <esr@thyrsus.com> <20160825161126.GC31039@thyrsus.com>
 <20160825223025.152CC406060@ip-64-139-1-69.sjc.megapath.net>
 <20160826032305.GA9428@thyrsus.com>
Message-ID: <20160825230529.Horde.2JsllT1FgCyDj4BtMBb_kBD@mail.drown.org>

Quoting "Eric S. Raymond" <esr at thyrsus.com>:
> Is the CPU temp going to be a good approximation of what the clock sees,
> though?  I can think of several reasons to doubt it.

It's better than nothing, but it does have two limitations:

1. the CPU heats itself at a different rate than the surrounding  
board, so it can be much hotter than the oscillator
2. the in-CPU temperature sensors tend to be low-ish resolution

For example, see the first two graphs:
https://blog.dan.drown.org/beaglebone-black-ntpgps-server-temperature-compensation/


From esr at thyrsus.com  Fri Aug 26 04:11:14 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Fri, 26 Aug 2016 00:11:14 -0400
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: <20160825230529.Horde.2JsllT1FgCyDj4BtMBb_kBD@mail.drown.org>
References: <esr@thyrsus.com> <20160825161126.GC31039@thyrsus.com>
 <20160825223025.152CC406060@ip-64-139-1-69.sjc.megapath.net>
 <20160826032305.GA9428@thyrsus.com>
 <20160825230529.Horde.2JsllT1FgCyDj4BtMBb_kBD@mail.drown.org>
Message-ID: <20160826041114.GA10021@thyrsus.com>

Dan Drown <dan-ntp at drown.org>:
> Quoting "Eric S. Raymond" <esr at thyrsus.com>:
> >Is the CPU temp going to be a good approximation of what the clock sees,
> >though?  I can think of several reasons to doubt it.
> 
> It's better than nothing, but it does have two limitations:
> 
> 1. the CPU heats itself at a different rate than the surrounding board, so
> it can be much hotter than the oscillator
> 2. the in-CPU temperature sensors tend to be low-ish resolution

That's about what I figured.  It's why I followed Gar's recommendation to get
a room-temperature sensor.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From Stromeko at nexgo.de  Fri Aug 26 05:58:27 2016
From: Stromeko at nexgo.de (Achim Gratz)
Date: Fri, 26 Aug 2016 07:58:27 +0200
Subject: Stratum one autonomy and assumptions about GPS
References: <esr@thyrsus.com>
 <20160825223025.152CC406060@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <87oa4gt6kc.fsf@Rainer.invalid>

Hal Murray writes:
> Most CPU chips include a temperature sensor.

The temperature sensor for the CPU is close to useless unless you have
an SoC with the XO integrated.  It tells you a lot about the average
load of the CPU, but not much else.

> For Linux PCs, you need the coretemp module loaded.  The lm-sensors package 
> does that.

The motherboard temp sensor usually is located near the Southbridge if
there is still one or near the PoL converters.  Both these locations
don't necessarily measure a temperature that tells you anything useful
about the XO, either.  There are systems with more temp sensors, if
there's one that measures the cooling fan intake then that would be
useful I guess.

If we're talking about the rasPi, there's a not too expensive
temperature sensor from SiLabs (Si7053) with an I?C interface that gives
you one sample per second and has 14bit resolution.  The XO on the rasPi
is on the underside of the board, but it should be possible to glue that
sensor onto the XO with some thermal compound and get thin wires around
the board to the GPIO header with most cases.  You'd have to be careful
not to short something of course.  That'd give you the actual crystal
temperature, which might enable forward temp compensation (there's a
blog post somewhere from someone who's done that with x86 hardware, I
can't find that link quickly enough).  It seems that at least some rasPi
actually use an TCXO that is pretty good, so I'm not sure how much of an
improvement that would be for the typical indoor environment with
limited temperature swings.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds


From esr at thyrsus.com  Fri Aug 26 15:02:51 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Fri, 26 Aug 2016 11:02:51 -0400
Subject: Removing interleaved mode
In-Reply-To: <20160706222717.16FAE406061@ip-64-139-1-69.sjc.megapath.net>
References: <dfoxfranke@gmail.com>
 <CAJm83bBJhhqB3BiLK0=OFCURuQOvkq=mix4RybkRbahXhOAXWQ@mail.gmail.com>
 <20160706222717.16FAE406061@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160826150251.GA1022@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> 
> dfoxfranke at gmail.com said:
> > With Eric's permission, I have removed support for interleaved mode in my
> > proto-refactor branch. Here is its commit-message eulogy: 
> 
> Seems fine with me.  I've never used it.
> 
> We should test things to make sure nothing strange happens.  I think that 
> requires 4 systems: 2 new and 2 old.  (I guess it could be done with 2 if you 
> are willing to run the tests serially.)
> 
> This might be a good candidate for Pis with GPS HATs.  It might take the 
> kernel PLL to notice the difference.

I'm not clear how we would tell if anything were broken. Or are you just
suggesting looking for a timing difference?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From esr at thyrsus.com  Fri Aug 26 15:20:15 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Fri, 26 Aug 2016 11:20:15 -0400
Subject: Possible cleanup
In-Reply-To: <20160724084437.CEE36406063@ip-64-139-1-69.sjc.megapath.net>
References: <20160724084437.CEE36406063@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160826152015.GA1355@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> 
> There is a SAVE_ERRNO macro that wraps around some code to preserve errno.  
> It's only used in a few places.  The first place I saw was calling msyslog.  
> That would make sense if following code did something that depended on the 
> error, but I checked all 4 cases and they never looked at errno.  They are 
> all in ntp_refclock.
> 
> Looks like we can rip it out.  But that's not the sort of code that's easy to 
> test so we should be careful.
> 
> SAVE_ERRNO uses a socket_errno() macro to get errno.  That looks like it's a 
> hook to work with windows.  The answer gets put back into errno.
> 
> Ahhh.  the SAVE_ERRNO macro isn't being used to save errno but rather to copy 
> the windows error over to errno where %m in msyslog can get it.
> 
> I wonder if we can push that into msyslog.  Looks like it's already there.
> 
> So either we can rip it out, or we have to fix all the other places that use 
> %m.
> 
> (Or I haven't analyzed things correctly.)

This macro is only used in the Windows port code.  I think the answer is going
to be that we drop all that crap and rely on the Windows emulation environment
to supply adjtime(2) as will as the POSIX interfaces.

Mark knows a Microsoft guy who he thinks can get this done.  That conversation
is on his to-do list.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From esr at thyrsus.com  Fri Aug 26 16:39:26 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Fri, 26 Aug 2016 12:39:26 -0400
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: <20160825231405.88CCF406060@ip-64-139-1-69.sjc.megapath.net>
References: <esr@thyrsus.com> <20160825100319.GA29235@thyrsus.com>
 <20160825231405.88CCF406060@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160826163926.GA1456@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> 
> esr at thyrsus.com said:
> > One of my agenda items for a while has been to develop a protocol for places
> > that want to firewall out NTP for security reasons - that is, a set of
> > practices and error estimates based on documented assumptions. 
> 
> Background data is good.  I think you will get into trouble if you try to 
> turn that into useful recommendations.  There are too many assumptions and 
> there will be huge differences between places.

The way you get around that problem is to document your assumptions, then
tell the users *how to test whether their environment matches those
assumptions.*

This is one reason I've been willing to put a lot of effort into
ntpviz.  It doesn't yet give us statistics about GPS outage-time
distribution, but we're not very far from where it will be able to do
so.  Once we have that kind of visibility, we can reasonably make
claims about what kind of preconditions are needed to run autonomously
with GPS and then, with ntpviz, give users a way to *test* for those
preconditions.

Certainty isn't achievable, but we can say things like: Keep watching
your GPS until outage times have a distribution (or a worst case) you
think you can model. Once you have that, you can make inferences about
how often you expect an outage to drive time accuracy outside a
specified range, and use that that to evaluate your relative risk
between time inaccuracy due to natural disruptions vs. cyberattack.
That will tell you if you should try to run autonomously.

Part of my strategic push behind ntpviz (besides making our funding
authority happy with pretty pictures) was to get us from a place where
analytics like that seem like too hard to tackle to a place where
they're natural, relatively easy extensions of a code framework
already in place.

Now that we have that framework in place, I'm certain you and Gary
have the domain knowledge required to make it tell us what we need to
know to make confidence estimates for running autonomous. But first
you have to get used to the idea that this is possible, that there
really is a rational statistical-estimation model you can apply!

Concrete direction: we should get to where we can plot a histogram of
outage lengths over a timespan of months and curve-fit it to a Poisson
distribution (if you need an explanation of why Poisson is appropriate
I can supply one). Then, by setting a maximum allowable skew and modeling
the drift envelope, we should be able to put a line on that chart that
expresses how often outages will bust that maximum skew.  This is a
fairly simple problem in applied statistics; people analyzing noisy
experimental data have to do similar things all the time.

> > Right.  This sort of thing you hedge against with tell-me-three-times.
> > (Remember that the user story assumes a big hardware budget.) 
> 
> If you have a big budget, you buy a does-it-all box with GPSDO and NTP 
> server, and pay a defense contractor to integrate things.
>   http://spectracom.com/products-services/precision-timing

True.  Either way, this is not a risk category that *we* need to design
around.

> > I don't think I need to care about specific causality much.  The overriding
> > question is what the statistical distribution of outage lengths is.
> 
> The time pattern depends upon the mechanism.  If your GPS unit fades out 
> several times a day you can collect useful statistics.  If your GPS screws up 
> due to a 1024 week rollover it's hard to get enough data to work with.

I think this was a bad example to use. We know exactly when the
rollovers are coming, they're rate, and we can have a human watching
to mitigate problems when they happen.  The real problem is large
*random* outages - that we don't necessarily know what the tail of the
distribution looks like.

> > (Rain/snow on your trees.  Why is this special? Does it increase multipath
> > reflections or something?) 
> 
> Water attenuates GPS signals.  So wet trees/roofs attenuate more than dry.

Noted.  Is this effect significant enough that SNR drops below
critical in severe weather?  If so, that would change the risk
profile a lot.

> I think you are mixing two uses for good background data.  Earlier you said 
> your target was a firewall blocking all NTP traffic to the WAN.  Now you are 
> back to running a WAN server.

You're right.  I talked about two different cases and didn't draw the link
between them.  My tacit assumptions are as follows:

(a) Users have come to expect WAN time-sync accuracy on the close order of
    10ms from existing NTP and network infrastructure.

(b) That expectation conditions the way people design for soft realtime even
    in off-net applications.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From esr at thyrsus.com  Sat Aug 27 08:42:29 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sat, 27 Aug 2016 04:42:29 -0400 (EDT)
Subject: OK, *now* you can bikeshed!
Message-ID: <20160827084229.BCA7C13A625C@snark.thyrsus.com>

Mark: Heads up, PR issue.

In the dark and backward abysm of time near the beginning of this
project, I designed us a logo. It's the one used at
www.ntpsec.org. Here it is:

https://www.ntpsec.org/clocktower128.png

Historical note: The thought in my mind when I made this was that
I wanted an image that unified time with security.  Thus, a clock
tower crossed with a fortification tower. The step from that to "chess
rook with a clockface" was almost instant.  Then I went looking for
two pieces of free clip-art I could composit with the GIMP, and found
them.

More recently, Mark Atwood's sister very kindly designed us a
shaded/sculpted version of the same concept. It's the one used on
our GitLab pages. Here it is:

https://gitlab.com/uploads/group/avatar/303316/rsz_1ntpseclogo.png

The question I lay before you is: should we drop one of these?  If
not, what are the appropriate usage conditions for deploying either
rather than the other?

Now I will say an obvious thing and an unobvious thing.  The obvious
thing is that the second is much prettier.  The unobvious thing is
that I have zero ego investment in keeping my original in use.

However, the brutalist #1 is not brutal merely because I am a poor
excuse for a visual artist who couldn't compose anything like #2 to
save my life.  I've read stuff about optimizing logo design - had to
choose a logo for an organization more than once (and the rather
successful OSI logo was one of my choices).

Simple is good.  Fewer colors and shades makes a design easier and
less expensive to replicate at different sizes and in different media.
#1 can readily be printed on clothing; #2 is much trickier, you'd
basically have to print it on an applique and bond that to the cloth.

So the actual question here is what we want to optimize for. If it's
visual appeal on the Web and in other media where the shading renders
well, absolutely #2.  If we want consistency and recognizability
across all media (including low-rez stuff like silkscreening) #1 is
a better choice.

We could use both.  But if we're going to do that, we need a
deployment theory.  Also (and I sold open source to the mainstream
partly by taking issues like this seriously, so don't snort) there's
a brand dilution problem when you use two logos that aren't very close
variants of each other.  Strong brands have unitary visual identities.

I don't have a particular answer to this question that I want to sell.
Supposing I did, it wouldn't be my place to make the decision. The
individual with responsibility is Mark, our PM and external-relations
person.  But it would be nice if the project had consensus about symbolic
stuff like this, too.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Hoplophobia (n.): The irrational fear of weapons, correctly described by 
Freud as "a sign of emotional and sexual immaturity".  Hoplophobia, like
homophobia, is a displacement symptom; hoplophobes fear their own
"forbidden" feelings and urges to commit violence.  This would be
harmless, except that they project these feelings onto others.  The
sequelae of this neurosis include irrational and dangerous behaviors
such as passing "gun-control" laws and trashing the Constitution.

From hmurray at megapathdsl.net  Sat Aug 27 09:36:40 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sat, 27 Aug 2016 02:36:40 -0700
Subject: Removing interleaved mode
In-Reply-To: Message from "Eric S. Raymond" <esr@thyrsus.com>
 of "Fri, 26 Aug 2016 11:02:51 EDT." <20160826150251.GA1022@thyrsus.com>
Message-ID: <20160827093640.8CEA6406074@ip-64-139-1-69.sjc.megapath.net>


esr at thyrsus.com said:
> I'm not clear how we would tell if anything were broken. Or are you just
> suggesting looking for a timing difference? 

Mostly I was suggesting that we needed to test it to make sure that it seemed 
to work rather than worrying about how well it worked.

After that, it would be nice to see how well it works.  I think that would 
take 2 systems with PPS on both.  That's 2 per test.  I think we could test 
old and new with 4 systems: 2 new, 2 old, and exchange packets in all 12 
combinations.

My suggestion of using Pi-s is probably bogus.  The Ethernet on USB will add 
enough noise to confuse things.



-- 
These are my opinions.  I hate spam.




From ghane0 at gmail.com  Sat Aug 27 09:54:02 2016
From: ghane0 at gmail.com (Sanjeev Gupta)
Date: Sat, 27 Aug 2016 17:54:02 +0800
Subject: OK, *now* you can bikeshed!
In-Reply-To: <20160827084229.BCA7C13A625C@snark.thyrsus.com>
References: <20160827084229.BCA7C13A625C@snark.thyrsus.com>
Message-ID: <CAHZk5We_XXqUZ04mtAtU9T396g750BBrjMWyX7yYS0MiuOjjkQ@mail.gmail.com>

On Sat, Aug 27, 2016 at 4:42 PM, Eric S. Raymond <esr at thyrsus.com> wrote:

> The question I lay before you is: should we drop one of these?  If
> not, what are the appropriate usage conditions for deploying either
> rather than the other?
>

I have only one observation: the time shown is 3'oclock, and that looks
like the letter 'L', not a clockface.

10:20, anyone?

-- 
Sanjeev Gupta
+65 98551208     http://www.linkedin.com/in/ghane
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160827/24ca1ce1/attachment.html>

From esr at thyrsus.com  Sat Aug 27 10:12:17 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sat, 27 Aug 2016 06:12:17 -0400
Subject: OK, *now* you can bikeshed!
In-Reply-To: <CAHZk5We_XXqUZ04mtAtU9T396g750BBrjMWyX7yYS0MiuOjjkQ@mail.gmail.com>
References: <20160827084229.BCA7C13A625C@snark.thyrsus.com>
 <CAHZk5We_XXqUZ04mtAtU9T396g750BBrjMWyX7yYS0MiuOjjkQ@mail.gmail.com>
Message-ID: <20160827101217.GA17328@thyrsus.com>

Sanjeev Gupta <ghane0 at gmail.com>:
> I have only one observation: the time shown is 3'oclock, and that looks
> like the letter 'L', not a clockface.
> 
> 10:20, anyone?

Orthogonal issue.  Once we've decided which style to use, that can be hacked.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From dtpoirot at gmail.com  Sat Aug 27 11:35:25 2016
From: dtpoirot at gmail.com (Daniel Poirot)
Date: Sat, 27 Aug 2016 06:35:25 -0500
Subject: OK, *now* you can bikeshed!
In-Reply-To: <20160827101217.GA17328@thyrsus.com>
References: <20160827084229.BCA7C13A625C@snark.thyrsus.com>
 <CAHZk5We_XXqUZ04mtAtU9T396g750BBrjMWyX7yYS0MiuOjjkQ@mail.gmail.com>
 <20160827101217.GA17328@thyrsus.com>
Message-ID: <1BFB91AF-7E6D-4F65-8067-AA6F4B5F130A@gmail.com>

What?  

10:10 - universal clock time:

http://mobile.nytimes.com/2008/11/28/business/media/28adco.html?_r=0&referer=http://petapixel.com/2013/06/27/why-photographs-of-watches-and-clocks-show-the-time-1010/


> On Aug 27, 2016, at 5:12 AM, Eric S. Raymond <esr at thyrsus.com> wrote:
> 
> Sanjeev Gupta <ghane0 at gmail.com>:
>> I have only one observation: the time shown is 3'oclock, and that looks
>> like the letter 'L', not a clockface.
>> 
>> 10:20, anyone?
> 
> Orthogonal issue.  Once we've decided which style to use, that can be hacked.
> -- 
>        <a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
> _______________________________________________
> devel mailing list
> devel at ntpsec.org
> http://lists.ntpsec.org/mailman/listinfo/devel

From Stromeko at nexgo.de  Sun Aug 28 18:25:27 2016
From: Stromeko at nexgo.de (Achim Gratz)
Date: Sun, 28 Aug 2016 20:25:27 +0200
Subject: Stratum one autonomy and assumptions about GPS
References: <esr@thyrsus.com>
 <20160825223025.152CC406060@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <87k2f0agyw.fsf@Rainer.invalid>

Hal Murray writes:
> esr at thyrsus.com said:
>> I have a USB thermometer on order, they're cheap.  Might I suggest you get
>> one and repeat this experiment, actually plotting your temperature
>> variation? 
>
> Most CPU chips include a temperature sensor.
>
> For Linux PCs, you need the coretemp module loaded.  The lm-sensors package 
> does that.
>
> You can get it by either parsing the output of the sensors command or by 
> looking in things like
>   /sys/devices/platform/coretemp.0/hwmon/hwmon0/temp2_input

I've checked the loopstat data from last week and both my rasPi get
steered less than +-1ppm from their baseline of -6ppm and -9ppm with a
temperature swing of about 4K over the day.  I can see when the sun was
hitting the roof and when I've had open windows, so I'm pretty certain
this is real.  You'd need a thermometer that does at least have
repeatability down to 0.25K to get something useful out of the
temperature data.

If you can get repeatability down to 10mK, then it'd probably be
possible to forward-compensate the temperature effects to below 5ppb,
giving you holdover capability for around 5 hours.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada


From Stromeko at nexgo.de  Sun Aug 28 18:48:12 2016
From: Stromeko at nexgo.de (Achim Gratz)
Date: Sun, 28 Aug 2016 20:48:12 +0200
Subject: Stratum one autonomy and assumptions about GPS
References: <esr@thyrsus.com>
 <20160825223025.152CC406060@ip-64-139-1-69.sjc.megapath.net>
 <87oa4gt6kc.fsf@Rainer.invalid>
Message-ID: <87fupoafwz.fsf@Rainer.invalid>

Achim Gratz writes:
> That'd give you the actual crystal
> temperature, which might enable forward temp compensation (there's a
> blog post somewhere from someone who's done that with x86 hardware, I
> can't find that link quickly enough).

This was the article I was referring to, plus the one it linked to:

http://www.wraith.sf.ca.us/ntp/
https://www.ijs.si/time/temp-compensation/


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables


From esr at thyrsus.com  Mon Aug 29 19:03:40 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Mon, 29 Aug 2016 15:03:40 -0400
Subject: Use of pool servers reveals unacceptable crash rate in async DNS
In-Reply-To: <20160627011103.6CF3F406057@ip-64-139-1-69.sjc.megapath.net>
References: <esr@thyrsus.com> <20160626131126.GD9051@thyrsus.com>
 <20160627011103.6CF3F406057@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160829190340.GA11892@thyrsus.com>

Processing old mail...

Hal Murray <hmurray at megapathdsl.net>:
> > I believe you're right that these platforms don't have it.  The question is,
> > how important is that fact?  Is the performance hit from synchronous DNS
> > really a showstopper?  I don't know the answer. 
> 
> There are two cases I know of where ntpd does a DNS lookup after it gets 
> started.
> 
> One is the try again when DNS for the normal server case doesn't work during 
> initialization.  It will try again occasionally until it gets an answer. 
> (which might be negative)
> 
> The main one is the pool code trying for a new server.  I think we should be 
> extending this rather than dropping it.  There are several possibles in this 
> area.  The main one would be to verify that a server you are using is still 
> in the pool.  (There isn't a way to do that yet - the pool doesn't have any 
> DNS support for that.)  The other would be to try replacing the poorest 
> server rather than only replacing dead servers.
> 
> DNS lookups can take a LONG time.  I think I've seen 40 seconds on a failing 
> case.
> 
> If we get the recv time stamp from the OS, I think the DNS delays won't 
> introduce any lies on the normal path.  We could test that by putting a sleep 
> in the main loop.  (There is a filter to reject packets that take too long, 
> but I think that's time-in-flight and excludes time sitting on the server.)
> 
> There are two cases I can think of where a pause in ntpd would cause 
> troubles.  One is that it would mess up refclocks.  The other is that packets 
> will get dropped if too many of them arrive.
> 
> I think that means we could use the pool command on a system without 
> refclocks.  That covers end nodes and maybe lightly loaded servers.
> 
> -------
> 
> It's worth checking out the input buffering side of things.  There may be 
> some code there that we don't need.  I think there is a pool of buffers.  
> Where can a buffer sit other than on the free queue.   Why do we need a pool?

The project has more important priorities than chasing this down.  But: I have
edited this text, adding a few details I have learned since, into a new
section for the internals tour (devel/tour.txt).  That will give somebody
a better-than-nothing place to start if we ever again try something like
the cAres replacement.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From gem at rellim.com  Tue Aug 30 03:43:49 2016
From: gem at rellim.com (Gary E. Miller)
Date: Mon, 29 Aug 2016 20:43:49 -0700
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: <20160825041946.GA26984@thyrsus.com>
References: <20160825041946.GA26984@thyrsus.com>
Message-ID: <20160829204349.6fba8ec5@spidey.rellim.com>

Yo Eric!

On Thu, 25 Aug 2016 00:19:46 -0400
"Eric S. Raymond" <esr at thyrsus.com> wrote:

> This was going to be a note to just Hal originally, but it will do the
> rest of the team no harm to know more about the scenarios and
> assumptions driving some of my design choices.
> 
> Hal objected (off list) to me drawing a conclusion from today's
> offset multiplot that check servers aren't necessary when you have
> a local GPS - a Stratum 1 really can run autonomously. He said,
> correctly of course, that the check servers aren't there to improve
> time accuracy when the GPS has sat lock, but to backstop the GPS when
> it flakes out.
> 
> I shall now discuss three interlocking reasons this possibility does
> not loom as large in my mind as it does in Hal's.
> 
> 1. GPS outage length and frequencies are decreasing

Don't care.  If you need your NTP to work, you need to know it is working.
Otherwise failure are not noticed.

> 2. The autonomy scenarios I think about are not hobbyist-budget
> productions

Yeah, and the big biys REALLY need to know their NTP is right.

> 3. There's a lower bound below which outages don't matter; we may be
> there.

I don't agree.  I monitor all my services 24x7, and I do get NTP
problems in my logs.

> Any given fixed accuracy target for deviation from UTC, combined with
> a maximum crystal drift rate, defines a longest tolerable GPS outage. 

Not the majority failure mode.

> We may already be at a technological place where GPS outages don't
> bust the tolerable-error budget, even with cheap hardware. If we
> aren't, we'll probably be there soon. 

We can't define a single tolerable error budget.  We can provide some
ranges of options for the user.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160829/ae201e37/attachment.bin>

From gem at rellim.com  Tue Aug 30 03:47:31 2016
From: gem at rellim.com (Gary E. Miller)
Date: Mon, 29 Aug 2016 20:47:31 -0700
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: <20160825102350.Horde.X9OWWeThDAj1vqp7_rq_Psa@mail.drown.org>
References: <20160825041946.GA26984@thyrsus.com>
 <20160825101549.Horde.n3e0uFvL5xJ-fPfTT7gC_at@mail.drown.org>
 <20160825102350.Horde.X9OWWeThDAj1vqp7_rq_Psa@mail.drown.org>
Message-ID: <20160829204731.44a202fe@spidey.rellim.com>

Yo Dan!

On Thu, 25 Aug 2016 10:23:50 -0500
Dan Drown <dan-ntp at drown.org> wrote:

> I had setup a test along these lines a week ago:
> https://blog.dan.drown.org/gps-pps-drift-when-it-has-no-signal/

Yeah, and I've been watching it.

> My results were a long term average of 250us-500us lost per day
> (~3-6ppb).  This surprised me because of how low it was, I assume I
> got lucky with the long term average temperature of the GPS module.

I do not believe your data.  After two weeks your GPS is still outputting
sat used data.  The Ephemeris timed out long ago, and the Almanac almost
gone too.  I think you still have a weak signal and since you GPS already
had a lock it is seeing something.

> Of course, this data doesn't apply to all GPS outage situations.

Nor does it reflect any test I ever ran.  But I admit to never testing
high end gear like Trimble gear.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160829/eeced3f1/attachment.bin>

From gem at rellim.com  Tue Aug 30 03:49:17 2016
From: gem at rellim.com (Gary E. Miller)
Date: Mon, 29 Aug 2016 20:49:17 -0700
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: <8760qozojj.fsf@Rainer.invalid>
References: <20160825041946.GA26984@thyrsus.com>
 <20160825101549.Horde.n3e0uFvL5xJ-fPfTT7gC_at@mail.drown.org>
 <20160825102350.Horde.X9OWWeThDAj1vqp7_rq_Psa@mail.drown.org>
 <8760qozojj.fsf@Rainer.invalid>
Message-ID: <20160829204917.6fe2499d@spidey.rellim.com>

Yo Achim!

On Thu, 25 Aug 2016 20:33:36 +0200
Achim Gratz <Stromeko at nexgo.de> wrote:

> Dan Drown writes:
> > The GPS module I'm using still outputs PPS even when it loses lock.
> > That won't be true for every GPS module.  
> 
> I also have a NavSpark mini attached at the moment and I was surprised
> it kept outputting the PPS when losing 3D lock (it stops blinking the
> LED).

Most cheaper GPS always output PPS, even when they have no idea
where they are or what time it is.  Garmin documents this, as do others.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160829/9dc3efd8/attachment.bin>

From gem at rellim.com  Tue Aug 30 03:51:30 2016
From: gem at rellim.com (Gary E. Miller)
Date: Mon, 29 Aug 2016 20:51:30 -0700
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: <20160825223025.152CC406060@ip-64-139-1-69.sjc.megapath.net>
References: <20160825161126.GC31039@thyrsus.com>
 <20160825223025.152CC406060@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160829205130.58cb0327@spidey.rellim.com>

Yo Hal!

On Thu, 25 Aug 2016 15:30:25 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> esr at thyrsus.com said:
> > I have a USB thermometer on order, they're cheap.  Might I suggest
> > you get one and repeat this experiment, actually plotting your
> > temperature variation?   
> 
> Most CPU chips include a temperature sensor.

Which I have found does not correlate with any of my NTP data.

I now have a lot of data, just need to finish up the ntpviz temp
module.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160829/47fb3c5b/attachment.bin>

From gem at rellim.com  Tue Aug 30 04:00:24 2016
From: gem at rellim.com (Gary E. Miller)
Date: Mon, 29 Aug 2016 21:00:24 -0700
Subject: Comments about ntpviz
In-Reply-To: <20160823203854.7972540605C@ip-64-139-1-69.sjc.megapath.net>
References: <20160823102826.51655f9c@spidey.rellim.com>
 <20160823203854.7972540605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160829210024.75782d27@spidey.rellim.com>

Yo Hal!

On Tue, 23 Aug 2016 13:38:54 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> The part I forgot to mention is that if I was going to do something
> to make my setup better, I would try to make a wrapper to drive
> gnuplot - a little window off to the side so I could poke Next rather
> than typing return at the gnuplot command window.  The main thing I
> want is a way to backup a few graphs.  

If you want to backup your graphs, just add a loggerd script to gzip
them to somewhere on a schedule.

> The second step would be a
> simple way to adjust vertical and/or horizontal scaling (aka zoom
> in/out) to cover the cases where I didn't anticipate I would want one
> and/or didn't get it right.

ntpviz is a cron job thing.  No one has come up with a good idea yet.
It could just generate a 90% plot and a 100% plot.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160829/dbf57bbe/attachment.bin>

From esr at thyrsus.com  Tue Aug 30 04:08:17 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Tue, 30 Aug 2016 00:08:17 -0400
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: <20160829204349.6fba8ec5@spidey.rellim.com>
References: <20160825041946.GA26984@thyrsus.com>
 <20160829204349.6fba8ec5@spidey.rellim.com>
Message-ID: <20160830040817.GA20612@thyrsus.com>

Gary E. Miller <gem at rellim.com>:
> > 1. GPS outage length and frequencies are decreasing
> 
> Don't care.  If you need your NTP to work, you need to know it is working.
> Otherwise failure are not noticed.

OK, the test for "know it is working" is: you have lock, or you had lock
less than x seconds ago where x is a worst-case of your drift model to
whatever confidence interfal you want to fix.

> > 3. There's a lower bound below which outages don't matter; we may be
> > there.
> 
> I don't agree.  I monitor all my services 24x7, and I do get NTP
> problems in my logs.

And you also said in recent mail that you don't work with the kind of
hardware a serious autonomy-seeker would use.  So *your* NTP problems
are not determinative, though they could be useful input data for
improving error-estimation techniques.

> > Any given fixed accuracy target for deviation from UTC, combined with
> > a maximum crystal drift rate, defines a longest tolerable GPS outage. 
> 
> Not the majority failure mode.

That's an interesting statement.  What *is*, in your experience, the
dominant failure mode.

> > We may already be at a technological place where GPS outages don't
> > bust the tolerable-error budget, even with cheap hardware. If we
> > aren't, we'll probably be there soon. 
> 
> We can't define a single tolerable error budget.  We can provide some
> ranges of options for the user.

And that's exactly what I've been pushing towards - to develop some
statistical modeling on the basis of which we can make estimates to whatever
confidence bound the user wants to set as a parameter.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 811 bytes
Desc: not available
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160830/ac57ed2a/attachment.bin>

From gem at rellim.com  Tue Aug 30 04:29:59 2016
From: gem at rellim.com (Gary E. Miller)
Date: Mon, 29 Aug 2016 21:29:59 -0700
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: <20160830040817.GA20612@thyrsus.com>
References: <20160825041946.GA26984@thyrsus.com>
 <20160829204349.6fba8ec5@spidey.rellim.com>
 <20160830040817.GA20612@thyrsus.com>
Message-ID: <20160829212959.36231dd3@spidey.rellim.com>

Yo Eric!

On Tue, 30 Aug 2016 00:08:17 -0400
"Eric S. Raymond" <esr at thyrsus.com> wrote:

> Gary E. Miller <gem at rellim.com>:
> > > 1. GPS outage length and frequencies are decreasing  
> > 
> > Don't care.  If you need your NTP to work, you need to know it is
> > working. Otherwise failure are not noticed.  
> 
> OK, the test for "know it is working" is: you have lock, or you had
> lock less than x seconds ago where x is a worst-case of your drift
> model to whatever confidence interfal you want to fix.

Not THE test, A test.  Your test would catch few of the failure modes I see.

> > I don't agree.  I monitor all my services 24x7, and I do get NTP
> > problems in my logs.  
> 
> And you also said in recent mail that you don't work with the kind of
> hardware a serious autonomy-seeker would use.  So *your* NTP problems
> are not determinative, though they could be useful input data for
> improving error-estimation techniques.

Correct.  There is NO determinative solution, because their is NO
determinative use case.

I suspect we could probably come up with about 5 use cases, each
with VERY different needs.  A guy that needs good time to keep his
OAUTH/DNSSEC working has very different time stability needs than
someone doing audio work, or harder still video work.

And if that guy doing OAUTH or DNSSEC just needs it to shop Amazon his
uptime needs are very different than someone running a ton of DNSSEC zones.

Yet they are all perfect condidates for a GPS/NTP solution.

> > Not the majority failure mode.  
> 
> That's an interesting statement.  What *is*, in your experience, the
> dominant failure mode.

The GPS goes nuts and start spitting out crazy time.  Or no time at all.

As previously discussed, there are many causes to get those two effects.

> > > We may already be at a technological place where GPS outages don't
> > > bust the tolerable-error budget, even with cheap hardware. If we
> > > aren't, we'll probably be there soon.   
> > 
> > We can't define a single tolerable error budget.  We can provide
> > some ranges of options for the user.  
> 
> And that's exactly what I've been pushing towards - to develop some
> statistical modeling on the basis of which we can make estimates to
> whatever confidence bound the user wants to set as a parameter.

A good friend of mine was a former statistician.  He gave it up and
became an engineer.  He said you only use statistitics until you start
to understand the problem.  Then other sciences take over.  He hated
starting to get a handle on the problem, then some specialist would
solve it.

So, when you fall back on stats, it is because you don't really know what
is happening.  And yes, clearly we can not even agree on failure modes, so
we needs more stats, and that just leads us to a few interesting problesm to
colve.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160829/7872cbe2/attachment.bin>

From esr at thyrsus.com  Tue Aug 30 10:12:05 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Tue, 30 Aug 2016 06:12:05 -0400
Subject: Removing interleaved mode
In-Reply-To: <20160827093640.8CEA6406074@ip-64-139-1-69.sjc.megapath.net>
References: <esr@thyrsus.com> <20160826150251.GA1022@thyrsus.com>
 <20160827093640.8CEA6406074@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160830101205.GA25667@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> 
> esr at thyrsus.com said:
> > I'm not clear how we would tell if anything were broken. Or are you just
> > suggesting looking for a timing difference? 
> 
> Mostly I was suggesting that we needed to test it to make sure that it seemed 
> to work rather than worrying about how well it worked.
> 
> After that, it would be nice to see how well it works.  I think that would 
> take 2 systems with PPS on both.  That's 2 per test.  I think we could test 
> old and new with 4 systems: 2 new, 2 old, and exchange packets in all 12 
> combinations.
> 
> My suggestion of using Pi-s is probably bogus.  The Ethernet on USB will add 
> enough noise to confuse things.

Given the nature of the bugs Daniel found, I don't think this effort can really
be justified.  Here is his comment in full.

    Remove interleaved mode
    
    Interleaved mode was an invention intended to improve timekeeping
    precision in symmetric and broadcast mode. The problem it intended to
    solve is that transmit timestamps have to be written before the packet
    is sent, but right *after* the packet is sent, better information
    becomes available because you know exactly when the packet made it
    through the kernel and out onto the wire. So, the basic idea of
    interleaved mode was to dump that better value into the *next* packet,
    and have the peer follow along with that, always one packet behind.
    
    This is a problem that PTP is clearly better suited to solving, but
    interleaved mode still seems at least reasonable in theory. However,
    there are two big problems.
    
    First, interleaved mode adds a great deal of complexity to NTP's state
    machine. This led to at least one terrible vulnerability (CVE-2016-1548)
    which took two tries to fix (CVE-2016-4956), and probably indirectly
    led to a few others.
    
    Second, the implementation was flawed. "Drivestamps" were collected
    simply by calling get_systime() immediately after sendpkt() returned.
    However, on modern kernels, send() returns immediately unless the
    network buffer is full. So the timestamp that NTP was collecting had
    nothing to do with the time the packet actually went out, and was not
    any more accurate than the transmit timestamp obtained in basic mode.
    
    If interleaved mode ever provided a timekeeping improvement, there are
    two possible explanations for why. One possibility is that the Solaris
    boxen that Dave Mills tested it on had a simpler kernel networking
    stack, so the timestamp he was collecting was something closer to a
    true drivestamp. Another possibility is the presence of a simple bug:
    before the recent refactor of receive(), in every mode except
    interleaved mode, NTP was storing a transmit timestamp where a receive
    timestamp belonged. Interleaved mode may have been improving
    performance just by dodging this buggy code.

-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From Stromeko at nexgo.de  Tue Aug 30 16:01:38 2016
From: Stromeko at nexgo.de (Achim Gratz)
Date: Tue, 30 Aug 2016 18:01:38 +0200
Subject: Stratum one autonomy and assumptions about GPS
References: <20160825041946.GA26984@thyrsus.com>
 <20160825101549.Horde.n3e0uFvL5xJ-fPfTT7gC_at@mail.drown.org>
 <20160825102350.Horde.X9OWWeThDAj1vqp7_rq_Psa@mail.drown.org>
 <8760qozojj.fsf@Rainer.invalid>
 <20160829204917.6fe2499d@spidey.rellim.com>
Message-ID: <87oa4ab5zx.fsf@Rainer.invalid>

Gary E. Miller writes:
> Most cheaper GPS always output PPS, even when they have no idea
> where they are or what time it is.  Garmin documents this, as do others.

Well, the NavSpark mini does not output a PPS until it aquires a 3D
lock, as documented.  The configuration program has options to either
keep the PPS indefinitely on, stop after losing 2D lock or stop after
losing 3D lock.  Unfortunately both the query and the configuration
commands are NACKed by the mini, so I don't know what the setting
actually is and the documentation is inconclusive.  From reading some
blog post (that I can't find again right now) my impression was it
should be stopping the PPS, but I have not seen that happen.  It seems
to output PPS at least through periods of 2D lock.  I have not yet
managed to blind the GPS antenna completely, though, so I can't know
what happens when it loses all signal.  I'm not too keen on prying that
U.FL connector lose while it's operating.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada


From hmurray at megapathdsl.net  Wed Aug 31 03:41:30 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Tue, 30 Aug 2016 20:41:30 -0700
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: Message from "Gary E. Miller" <gem@rellim.com>
 of "Mon, 29 Aug 2016 20:51:30 PDT." <20160829205130.58cb0327@spidey.rellim.com>
Message-ID: <20160831034130.75CAF406061@ip-64-139-1-69.sjc.megapath.net>


gem at rellim.com said:
>> Most CPU chips include a temperature sensor.
> Which I have found does not correlate with any of my NTP data. 

I have a Raspberry Pi sitting on a window sill.  It gets a couple of hours of 
late afternoon sun.  There is a big temperature spike.  Drift roughly follows 
it.

I wouldn't call it great, but if you were interested in understanding this 
area, it's enough to strongly suggest getting a temperature probe on the 
crystal.

-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Wed Aug 31 05:51:14 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Tue, 30 Aug 2016 22:51:14 -0700
Subject: Stratum one autonomy and assumptions about GPS
In-Reply-To: Message from Achim Gratz <Stromeko@nexgo.de>
 of "Tue, 30 Aug 2016 18:01:38 +0200." <87oa4ab5zx.fsf@Rainer.invalid>
Message-ID: <20160831055114.BF48A406061@ip-64-139-1-69.sjc.megapath.net>


Stromeko at nexgo.de said:
> I have not yet managed to blind the GPS antenna completely, though, so I
> can't know what happens when it loses all signal. 

Have you tried putting it under a metal pie tin?

Or inside a metal box? (being careful not to mangle the cord)

-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Wed Aug 31 05:56:43 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Tue, 30 Aug 2016 22:56:43 -0700
Subject: Comments about ntpviz
In-Reply-To: Message from "Gary E. Miller" <gem@rellim.com>
 of "Mon, 29 Aug 2016 21:00:24 PDT." <20160829210024.75782d27@spidey.rellim.com>
Message-ID: <20160831055643.49D43406074@ip-64-139-1-69.sjc.megapath.net>


gem at rellim.com said:
> If you want to backup your graphs, just add a loggerd script to gzip them to
> somewhere on a schedule. 

Wrong binding for "backup".

The context was the way I'm currently looking at graphs.  I feed things like 
this to gnuplot:
  reset; load "foo1.gp"
  pause -1 
  reset; load "foo2.gp"
  pause -1
  reset; load "foo3.gp"
  pause -1
  ...

What I meant by backup was look at the graph I was looking at a few seconds 
ago.  The above only allows me to go forward to the next graph.

Some sort of wrapper to drive gnuplot should be simple.  It just hasn't 
gotten to the top of my list yet.


-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Wed Aug 31 06:24:26 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Tue, 30 Aug 2016 23:24:26 -0700
Subject: PPS undersampling
In-Reply-To: Message from "Eric S. Raymond" <esr@thyrsus.com>
 of "Tue, 23 Aug 2016 20:07:21 EDT." <20160824000721.GD16539@thyrsus.com>
Message-ID: <20160831062426.CBB7E406061@ip-64-139-1-69.sjc.megapath.net>


> Hm. I think I get it - and you've just added a pretty powerful reason to
> eventually pull the refclocks into a separate daemon by telling me I need to
> get ntpd itself out of the PPS-watching business entirely in order to get
> rid of that timer. 

The reason I want to get rid of the every-second timer is to save power when 
operating on battery with no refclocks.

If you have a PPS, you have to take an interrupt every second.  I'm not a 
wizard on that area.  It would be interesting to know if you can get useful 
interrupt responses when starting from a power-save mode.  Similarly, can you 
wakeup in time to get data from a serial port without dropping any 
characters?  It may be that power-save just won't happen if you are using 
refclocks.

Moving the PPS processing out of ntpd doesn't change any of that.


-- 
These are my opinions.  I hate spam.




From gem at rellim.com  Wed Aug 31 06:33:09 2016
From: gem at rellim.com (Gary E. Miller)
Date: Tue, 30 Aug 2016 23:33:09 -0700
Subject: PPS undersampling
In-Reply-To: <20160831062426.CBB7E406061@ip-64-139-1-69.sjc.megapath.net>
References: <20160824000721.GD16539@thyrsus.com>
 <20160831062426.CBB7E406061@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160830233309.007f377e@spidey.rellim.com>

Yo Hal!

On Tue, 30 Aug 2016 23:24:26 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> > Hm. I think I get it - and you've just added a pretty powerful
> > reason to eventually pull the refclocks into a separate daemon by
> > telling me I need to get ntpd itself out of the PPS-watching
> > business entirely in order to get rid of that timer.   
> 
> The reason I want to get rid of the every-second timer is to save
> power when operating on battery with no refclocks.

Saving power is good, but I suspect the extra power is minimal.  I
hace USB power meters, so we can measure this.

> If you have a PPS, you have to take an interrupt every second.

No, every 500 millisecond.

>  I'm
> not a wizard on that area.  It would be interesting to know if you
> can get useful interrupt responses when starting from a power-save
> mode.  Similarly, can you wakeup in time to get data from a serial
> port without dropping any characters?  It may be that power-save just
> won't happen if you are using refclocks.

Easy to test, when we have a knob to twist.  It would depend a lot on the
mode the CPU is in.  I always set governor=performance, so I would see no
change in power.

> Moving the PPS processing out of ntpd doesn't change any of that.

True, whether it is on gpsd or ntpd it should be similar power.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160830/de354f9b/attachment.bin>

From gem at rellim.com  Wed Aug 31 06:35:50 2016
From: gem at rellim.com (Gary E. Miller)
Date: Tue, 30 Aug 2016 23:35:50 -0700
Subject: Comments about ntpviz
In-Reply-To: <20160831055643.49D43406074@ip-64-139-1-69.sjc.megapath.net>
References: <20160829210024.75782d27@spidey.rellim.com>
 <20160831055643.49D43406074@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160830233550.1ccefecc@spidey.rellim.com>

Yo Hal!

On Tue, 30 Aug 2016 22:56:43 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> gem at rellim.com said:
> > If you want to backup your graphs, just add a loggerd script to
> > gzip them to somewhere on a schedule.   
> 
> Wrong binding for "backup".
> 
> The context was the way I'm currently looking at graphs.  I feed
> things like this to gnuplot:
>   reset; load "foo1.gp"
>   pause -1 
>   reset; load "foo2.gp"
>   pause -1
>   reset; load "foo3.gp"
>   pause -1
>   ...

I'm noty sure how this appies to ntpviz???

> What I meant by backup was look at the graph I was looking at a few
> seconds ago.  The above only allows me to go forward to the next
> graph.

Since the graphs are every 30 mins or hour, I'm not sure what you are
trying for?

> Some sort of wrapper to drive gnuplot should be simple.  It just
> hasn't gotten to the top of my list yet.

But we already have chrony-graph and ntpviz to frive gnuplot, So I'm
confused what you want?

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160830/9cf7741d/attachment.bin>

From hmurray at megapathdsl.net  Wed Aug 31 07:38:58 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Wed, 31 Aug 2016 00:38:58 -0700
Subject: Comments about ntpviz
In-Reply-To: Message from "Gary E. Miller" <gem@rellim.com>
 of "Tue, 30 Aug 2016 23:35:50 PDT." <20160830233550.1ccefecc@spidey.rellim.com>
Message-ID: <20160831073858.960E9406057@ip-64-139-1-69.sjc.megapath.net>

> I'm noty sure how this appies to ntpviz???

I was trying to describe what I'm doing currently and what I like and don't 
like about it.

The current ntpviz makes a bunch of graphs as files and puts a web page 
wrapper around it.  I'd like to avoid that and put the output of gnuplot 
directly on the screen.  Currently, I'm doing that with gnuplot's pause.  
That only goes forward.  I'd like to go backwards.  (That's what I meant by 
backup.  Maybe I should have said back up or go back a few graphs.)

I'd also like a bunch of knobs so I can tweak the graph I'm looking at.  Most 
likely zoom in or out.

I assume all that would be reasonably simple after I knew how to do it, but 
it hasn't gotten to the top of my list yet.


> Since the graphs are every 30 mins or hour, I'm not sure what you are trying
> for? 

I'm not interested in automatically making graphs for somebody else to look 
at.  I'm only interested in "now", when I want to look at things.

A frequent thing that happens is that I see something interesting on a graph 
and then want to look more carefully at a graph I was looking at a few 
seconds ago.


-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Wed Aug 31 08:19:49 2016
From: hmurray at megapathdsl.net (Hal Murray)
Date: Wed, 31 Aug 2016 01:19:49 -0700
Subject: PPS undersampling
In-Reply-To: Message from "Gary E. Miller" <gem@rellim.com>
 of "Tue, 30 Aug 2016 23:33:09 PDT." <20160830233309.007f377e@spidey.rellim.com>
Message-ID: <20160831081949.60C06406061@ip-64-139-1-69.sjc.megapath.net>


gem at rellim.com said:
> Saving power is good, but I suspect the extra power is minimal.  I hace USB
> power meters, so we can measure this. 

It depends on the details.  You can save a lot of power by turning stuff off. 
 The more you turn off, the more power you save but the longer it takes to 
get running again.  I think most CPUs these days have an instruction that is 
roughly "halt and wait for an interrupt".  That lets the CPU turn off the 
clock for the instruction unit so it doesn't have to burn power running a 
tight loop.  As an experiment, you could write some code that goes into a 
tight loop and see if that takes more power than not running anything.

The next step is things like putting the memory into self refresh mode or 
turning off the disk controller.

USB is ugly to power down since there aren't any interrupt wires.  It polls 
for "interrupt" requests, so you can't power that off if you want a wakeup 
from an interrupt on a USB device.

In the extreme, you can power off the clock generation circuit.  You need 
something like a push button to wake up.  (or timer based on a low power 
clock)  If the clock is off, the only power is leakage which can be very low 
if you use a silicon process targeted at low power rather than high 
performance.


I plugged a PC into an Ethernet Hub the other day.  While powered off, the 
hub said 100 megabits.  I assume that was the wake-via-LAN stuff and it was 
running at 100 vs 1000 to save power.


Your USB power meter should work for a Raspberry Pi or BeagleBone.  I assume 
you have a wall-power meter for a laptop or desktop.

-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Wed Aug 31 08:20:07 2016
From: esr at thyrsus.com (Eric S. Raymond)
Date: Wed, 31 Aug 2016 04:20:07 -0400
Subject: Comments about ntpviz
In-Reply-To: <20160831073858.960E9406057@ip-64-139-1-69.sjc.megapath.net>
References: <gem@rellim.com> <20160830233550.1ccefecc@spidey.rellim.com>
 <20160831073858.960E9406057@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160831082007.GA22551@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> I'm not interested in automatically making graphs for somebody else to look 
> at.  I'm only interested in "now", when I want to look at things.

For each possible graph, there is a command-line option that generates
only that graph.  And the default directory is /var/log/ntpstats.  So
you can say, e.g., this:

ntpviz --local-offset -g | display -

and get an instant report.  (The -g tells it to generate PNG; without it you
get the GNUPLOT.)
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

From gem at rellim.com  Wed Aug 31 15:50:36 2016
From: gem at rellim.com (Gary E. Miller)
Date: Wed, 31 Aug 2016 08:50:36 -0700
Subject: PPS undersampling
In-Reply-To: <20160831081949.60C06406061@ip-64-139-1-69.sjc.megapath.net>
References: <20160830233309.007f377e@spidey.rellim.com>
 <20160831081949.60C06406061@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160831085036.7d716826@spidey.rellim.com>

Yo Hal!

On Wed, 31 Aug 2016 01:19:49 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> gem at rellim.com said:
> > Saving power is good, but I suspect the extra power is minimal.  I
> > hace USB power meters, so we can measure this.   
> 
> It depends on the details.  You can save a lot of power by turning
> stuff off. The more you turn off, the more power you save but the
> longer it takes to get running again.  I think most CPUs these days
> have an instruction that is roughly "halt and wait for an
> interrupt".  That lets the CPU turn off the clock for the instruction
> unit so it doesn't have to burn power running a tight loop.  As an
> experiment, you could write some code that goes into a tight loop and
> see if that takes more power than not running anything.

All I care about is how it affects ntpd.  As long as ntpd does a usleep()
the OS will do the right thing.

> The next step is things like putting the memory into self refresh
> mode or turning off the disk controller.

Turning those off is a deep sleep.  That will not work for ntpd.

> USB is ugly to power down since there aren't any interrupt wires.

Yup, not gonna happen.

> In the extreme, you can power off the clock generation circuit.

And then there is no point in running ntpd.

> Your USB power meter should work for a Raspberry Pi or BeagleBone.  I
> assume you have a wall-power meter for a laptop or desktop.

Yeah, but they all average over a good part of a second.  Nothing
ntpd does will move the (virtual) needle.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160831/ee59d6bf/attachment.bin>

From gem at rellim.com  Wed Aug 31 15:55:48 2016
From: gem at rellim.com (Gary E. Miller)
Date: Wed, 31 Aug 2016 08:55:48 -0700
Subject: Comments about ntpviz
In-Reply-To: <20160831073858.960E9406057@ip-64-139-1-69.sjc.megapath.net>
References: <20160830233550.1ccefecc@spidey.rellim.com>
 <20160831073858.960E9406057@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20160831085548.61af9cca@spidey.rellim.com>

Yo Hal!

On Wed, 31 Aug 2016 00:38:58 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> > I'm noty sure how this appies to ntpviz???  
> 
> I was trying to describe what I'm doing currently and what I like and
> don't like about it.
> 
> The current ntpviz makes a bunch of graphs as files and puts a web
> page wrapper around it.

Yes.

> I'd like to avoid that and put the output of
> gnuplot directly on the screen.

So ignore the html and just look at the png output graphs with the
png viewer of your choice.  You could multipane, 3D wheel, carosel, etc.

Or a simple Python script with some GUI buttons.

> I'd also like a bunch of knobs so I can tweak the graph I'm looking
> at.  Most likely zoom in or out.

> I assume all that would be reasonably simple after I knew how to do
> it, but it hasn't gotten to the top of my list yet.

Pretty much any pnng viewer can do that.  Granted the png resolution may be
too low, but that is an easy fix.

> > Since the graphs are every 30 mins or hour, I'm not sure what you
> > are trying for?   
> 
> I'm not interested in automatically making graphs for somebody else
> to look at.  I'm only interested in "now", when I want to look at
> things.
> 
> A frequent thing that happens is that I see something interesting on
> a graph and then want to look more carefully at a graph I was looking
> at a few seconds ago.

ntpviz only takes 20 seconds to run on a Pi3 for me, ant then I have
all the graphs.  Or just ask ntpviz for the one graph you want, prolly
about a second,

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ntpsec.org/pipermail/devel/attachments/20160831/6249c33a/attachment.bin>