From hmurray at megapathdsl.net  Sun Oct  1 07:46:41 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 01 Oct 2017 00:46:41 -0700
Subject: What's the story on day/year from refclocks?
Message-ID: <20171001074641.6419840605C@ip-64-139-1-69.sjc.megapath.net>

I have 2 devices with GPS wrap around glitches.

One is a HP/Lucent box.  It gets the date from the text string.  Fudging 
works.

The other is NMEA.  It gets the right time and the current date, ignoring the 
date in the text.

58027 27891.670 127.127.20.0 $GPZDA,074452,15,02,1998,+00,00  384 64 0 0 64 0

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+glypnod         192.168.1.33     2 u   14  128  377    0.312    0.011   0.021
-shuksan         .PPS.            1 u   34  128  377    0.309   -0.012   0.014
+mon             .PPS.            1 u  107  128  377    0.312   -0.173   0.079
-tom             .PPS.            1 u  115  128  377    0.367   -0.039   0.054
*cent            .PPS.            1 u   67  128  377    0.436   -0.044   0.022
 NMEA(0)         .GPS.            0 l   30   64  377    0.000  265.910   5.460

I haven't tuned the fudge yet - just got it in the right ballpark.


-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Sun Oct  1 17:01:18 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sun, 1 Oct 2017 13:01:18 -0400
Subject: What's the story on day/year from refclocks?
In-Reply-To: <20171001074641.6419840605C@ip-64-139-1-69.sjc.megapath.net>
References: <20171001074641.6419840605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20171001170118.GB24550@thyrsus.com>

Hal Murray via devel <devel at ntpsec.org>:
> I have 2 devices with GPS wrap around glitches.
> 
> One is a HP/Lucent box.  It gets the date from the text string.  Fudging 
> works.
> 
> The other is NMEA.  It gets the right time and the current date, ignoring the 
> date in the text.

There's magic code in the NMEA driver that computes the year from
other dare components based on some calendrical trick I don't
understand. A comment notes that it will fail in 2399.

Under consideration for 1.0, once I *do* undertstand it: making all
refclock drivers use this.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.



From hmurray at megapathdsl.net  Sun Oct  1 18:28:24 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 01 Oct 2017 11:28:24 -0700
Subject: What's the story on day/year from refclocks?
In-Reply-To: Message from "Eric S. Raymond via devel" <devel@ntpsec.org>
 of "Sun, 01 Oct 2017 13:01:18 EDT." <20171001170118.GB24550@thyrsus.com>
Message-ID: <20171001182824.DE19640605C@ip-64-139-1-69.sjc.megapath.net>


devel at ntpsec.org said:
> There's magic code in the NMEA driver that computes the year from other dare
> components based on some calendrical trick I don't understand. A comment
> notes that it will fail in 2399.

> Under consideration for 1.0, once I *do* undertstand it: making all refclock
> drivers use this. 

That sounds like the sort of magic that could do the wrong thing without 
warning.  Maybe we should have a flag to enable it and/or why would we want 
to add that to other drivers?

NMEA may be special since some of the date formats only have 2 digits of 
year.  I think the standard century rollover fixup based on the build date 
would be better than something that is hard to explain.

Another possibility would be to have a per-driver GPS flag and apply the 
normal GPS rollover fixup based on the build date.  That is at least 
reasonable to describe.  But it only works for 20 years.


-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Sun Oct  1 19:46:46 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sun,  1 Oct 2017 15:46:46 -0400 (EDT)
Subject: Resolution of the library-path mess
Message-ID: <20171001194646.0B03513A0206@snark.thyrsus.com>

I've spent the last several days researching all the various proposals
for how we handle locating the Python libraries, and trying to fully implement
several of them.

My conclusion is that Mark called this one right on the 28th:

>My inclination is to keep [Fred's] patch, document the lack of FHS compliance,
>and roadmap a fix to get_python_lib, possibly by nudging the WAF or python
>communities to write it.

More specifically, the only information we have about where we
can put Python modules is the return from the distutils funcrion
get_python_lib().

My attempt to deduce a parallel 'local' location for that fails because
it is not in fact a rule on all our target platforms that if /usr/lib/X/Y
exists then /usr/local/lib/X/Y does as well (however, see below for
what we can do if this is true).

Messing with the default sys.path in any way (notably through .pth files)
is not a good idea either, because, as Fed Wright notes:

>One of the this I've discovered in looking at this stuff is that having
>code that relies on looking at sys.path would be fragile, for at least two
>reasons:
>
>1) Some directories get added by installs of other packages, but relying
>on any such directories would break if the relevant packages were removed.
>
>2) Directories that don't exist don't get added.  But the install process
>creates directories as needed.  So if one came up with a directory that
>Python knows about but doesn't currently exist, it would appear from
>looking at sys.path at configure time that Python doesn't know about it,
>even though it might work perfectly well after the install.

I'm also opposed to it because changing the default load path
could end up causing misbehavior in *other packages*.  Not good
to have sharp elbows like that.

However there is good news for Gary and anybody else who is really worried
about FHS compilance.  While we cannot *guarantee* being able to do that, it
was trivial to change the installation code so that if the FHS-compliant
/usr/$PREFIX directory corresponding to the return value of python_lib()
*already exists* (and can therefore be presumed to be scanned by the
Python library loader) it is used.

On most systems this will do the right thing.

I have implemented and tested this solution, along with full
documentation and packaging guidance.

We are now at T + 3 days and need to get a move on, so I'm limiting
the comment and objections period on this to end at close of business
Monday (2 oct).  If we don't have consensus on a clearly better
solution by then, we go to freeze and testing for a week, aiming to
ship on the 9th.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

If gun laws in fact worked, the sponsors of this type of legislation
should have no difficulty drawing upon long lists of examples of
criminal acts reduced by such legislation. That they cannot do so
after a century and a half of trying -- that they must sweep under the
rug the southern attempts at gun control in the 1870-1910 period, the
northeastern attempts in the 1920-1939 period, the attempts at both
Federal and State levels in 1965-1976 -- establishes the repeated,
complete and inevitable failure of gun laws to control serious crime.
        -- Senator Orrin Hatch, in a 1982 Senate Report

From esr at thyrsus.com  Sun Oct  1 20:37:09 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sun, 1 Oct 2017 16:37:09 -0400
Subject: What's the story on day/year from refclocks?
In-Reply-To: <20171001182824.DE19640605C@ip-64-139-1-69.sjc.megapath.net>
References: <devel@ntpsec.org> <20171001170118.GB24550@thyrsus.com>
 <20171001182824.DE19640605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20171001203624.GA31169@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> 
> devel at ntpsec.org said:
> > There's magic code in the NMEA driver that computes the year from other dare
> > components based on some calendrical trick I don't understand. A comment
> > notes that it will fail in 2399.
> 
> > Under consideration for 1.0, once I *do* undertstand it: making all refclock
> > drivers use this. 
> 
> That sounds like the sort of magic that could do the wrong thing without 
> warning.  Maybe we should have a flag to enable it and/or why would we want 
> to add that to other drivers?

We might want to make available it to other drivers because they too
return only 2-digit dates.  This is true of the Symmetricom type 2,
for example.

I'm opposed to adding flags until an actual need is demonstrated.  If we're
going to add test and documentation complexity, it should be because we
know we have to do it.

> NMEA may be special since some of the date formats only have 2 digits of 
> year.

As noted, that's not just an NMEA problem.

>           I think the standard century rollover fixup based on the build date 
> would be better than something that is hard to explain.

Better still, we could write a common century-rollover check that does both
things and complains if they don't match.

> Another possibility would be to have a per-driver GPS flag and apply the 
> normal GPS rollover fixup based on the build date.  That is at least 
> reasonable to describe.  But it only works for 20 years.

This area is worth a serious look once we ship 1.0.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.



From hmurray at megapathdsl.net  Sun Oct  1 23:29:19 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 01 Oct 2017 16:29:19 -0700
Subject: What's the story on day/year from refclocks?
In-Reply-To: Message from "Eric S. Raymond via devel" <devel@ntpsec.org>
 of "Sun, 01 Oct 2017 16:37:09 EDT." <20171001203624.GA31169@thyrsus.com>
Message-ID: <20171001232919.2476C40605C@ip-64-139-1-69.sjc.megapath.net>


>>> There's magic code in the NMEA driver
>> Maybe we should have a flag..
> I'm opposed to adding flags until an actual need is demonstrated.  If we're
> going to add test and documentation complexity, it should be because we know
> we have to do it. 

I agree with the philosophy, but I'll withhold final judgment until somebody 
figures out what is actually going on.

Fixups based on the build date are (relatively) easy to understand and 
explain.

Fixups based on file system dates work fine as long as everything is working 
fine.  But what happens if, somehow, the file system date jumps to the 
future?  Explaining that won't be fun.  What happens if the box sits on the 
shelf for several years?

Beware of hidden state.  The HP driver worked until restart after the GPS 
rollover.  (I remember getting confused until I looked at the clockstats.)  
That was before you fixed it to take advantage of 4 digit years.  Now it 
correctly requires a 20 year fudge.  (I haven't figured out how to set the 
GPS date on that box.)


-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Mon Oct  2 00:35:30 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sun, 1 Oct 2017 20:35:30 -0400
Subject: What's the story on day/year from refclocks?
In-Reply-To: <20171001232919.2476C40605C@ip-64-139-1-69.sjc.megapath.net>
References: <devel@ntpsec.org> <20171001203624.GA31169@thyrsus.com>
 <20171001232919.2476C40605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20171002003530.GA2044@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> 
> >>> There's magic code in the NMEA driver
> >> Maybe we should have a flag..
> > I'm opposed to adding flags until an actual need is demonstrated.  If we're
> > going to add test and documentation complexity, it should be because we know
> > we have to do it. 
> 
> I agree with the philosophy, but I'll withhold final judgment until somebody 
> figures out what is actually going on.

...which is definitely on my list of things to untangle after 1.0.
For a very practical reason; if the code in NMEA that deduces a
4-digit year can be used by other drivers, more of them can drive ntpd
running with no network peers.

> Fixups based on the build date are (relatively) easy to understand and 
> explain.
> 
> Fixups based on file system dates work fine as long as everything is working 
> fine.  But what happens if, somehow, the file system date jumps to the 
> future?  Explaining that won't be fun.  What happens if the box sits on the 
> shelf for several years?

None of the wraparound kluges depend on filesystem dates.  The hacks for
disambiguating two-digit years work in one of two ways:

1. In the NMEA driver, calendrical black magic that's supposed to work
   until 2399.

2. Elsewhere, the year is deduced from the receipt time of the last
   incoming clock sample.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.



From gem at rellim.com  Mon Oct  2 01:25:15 2017
From: gem at rellim.com (Gary E. Miller)
Date: Sun, 1 Oct 2017 18:25:15 -0700
Subject: Resolution of the library-path mess
In-Reply-To: <20171001194646.0B03513A0206@snark.thyrsus.com>
References: <20171001194646.0B03513A0206@snark.thyrsus.com>
Message-ID: <20171001182422.78b4da19@spidey.rellim.com>

Yo Eric!

On Sun,  1 Oct 2017 15:46:46 -0400 (EDT)
"Eric S. Raymond via devel" <devel at ntpsec.org> wrote:

> However there is good news for Gary and anybody else who is really
> worried about FHS compilance.  While we cannot *guarantee* being able
> to do that, it was trivial to change the installation code so that if
> the FHS-compliant /usr/$PREFIX directory corresponding to the return
> value of python_lib() *already exists* (and can therefore be presumed
> to be scanned by the Python library loader) it is used.

I not so much concerned about FHS compliance as in not stepping on
system installed copies of NTPsec.  This is a small problem for gpsd,
and as systems install their own copies of NTPsec this will become
a bigger problem for NTPsec.

If you install in /usr/$PREFIX, and I assume $PREFIX is "lib/pyrhonXX",
then you will be installing on the system copy.

How do you plan that a local NTPsec install from source does not
overwite an NTPsec install from the native OS repositories?

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can?t measure it, you can?t improve it." - Lord Kelvin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171001/b6364541/attachment.bin>

From gem at rellim.com  Mon Oct  2 01:46:23 2017
From: gem at rellim.com (Gary E. Miller)
Date: Sun, 1 Oct 2017 18:46:23 -0700
Subject: Fw: [Git][NTPsec/ntpsec][master] Mostly resolve and document FHS
 conformance issue.
Message-ID: <20171001184623.2f0826be@spidey.rellim.com>

Yo Eric!


+Due to a limitation of the Python distutils library, if you install
+from the source distribution with prefix set to a value other than
+/usr (in particular, if it's the default value /usr/local), that
+prefix will be honored *only if the corresponding Python library
+directory already exists*.

I do not understand the logic here?  

What is wrong with creating /usr/local/XXX ??

If I tell NTPsec to install in /usr/local/ I fully expect to find it in
/usr/local/ after install.

Conversely, if you do install intp /usr/local/XXX, then then user needs
to do the PYTHONPATH drill, which got removed from the doc.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can?t measure it, you can?t improve it." - Lord Kelvin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171001/89a4e6e3/attachment.html>
-------------- next part --------------
_______________________________________________
vc mailing list
vc at ntpsec.org
http://lists.ntpsec.org/mailman/listinfo/vc
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171001/89a4e6e3/attachment.bin>

From esr at thyrsus.com  Mon Oct  2 01:48:20 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sun, 1 Oct 2017 21:48:20 -0400
Subject: Resolution of the library-path mess
In-Reply-To: <20171001182422.78b4da19@spidey.rellim.com>
References: <20171001194646.0B03513A0206@snark.thyrsus.com>
 <20171001182422.78b4da19@spidey.rellim.com>
Message-ID: <20171002014820.GA3947@thyrsus.com>

Gary E. Miller via devel <devel at ntpsec.org>:
> How do you plan that a local NTPsec install from source does not
> overwite an NTPsec install from the native OS repositories?

That now will never happen if the /usr/local/lb/python-X.Y directory exists;
the install logic will notice that.

I'll add language to INSTALL that one should make sure this directory
exists so as not to step on any distribution copy of the ntp library.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 811 bytes
Desc: not available
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171001/a7b4d1f7/attachment.bin>

From hmurray at megapathdsl.net  Mon Oct  2 02:04:20 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 01 Oct 2017 19:04:20 -0700
Subject: Resolution of the library-path mess
Message-ID: <20171002020420.7847640605C@ip-64-139-1-69.sjc.megapath.net>


> I'll add language to INSTALL that one should make sure this directory exists
> so as not to step on any distribution copy of the ntp library. 

That doesn't sound strong enough.  It's too easy to accidentally screw up.


-- 
These are my opinions.  I hate spam.




From fw at fwright.net  Mon Oct  2 02:32:06 2017
From: fw at fwright.net (Fred Wright)
Date: Sun, 1 Oct 2017 19:32:06 -0700 (PDT)
Subject: Resolution of the library-path mess
In-Reply-To: <20171002014820.GA3947@thyrsus.com>
References: <20171001194646.0B03513A0206@snark.thyrsus.com>
 <20171001182422.78b4da19@spidey.rellim.com>
 <20171002014820.GA3947@thyrsus.com>
Message-ID: <Pine.LNX.4.58L1X.1710011918430.1188@bolt.sonic.net>


On Sun, 1 Oct 2017, Eric S. Raymond via devel wrote:

> Gary E. Miller via devel <devel at ntpsec.org>:
> > How do you plan that a local NTPsec install from source does not
> > overwite an NTPsec install from the native OS repositories?
>
> That now will never happen if the /usr/local/lb/python-X.Y directory exists;
> the install logic will notice that.

That of course assumes that if the directory exists, it's in sys.path.
Perhaps that's a reasonable assumption, though it's hard to be sure.

> I'll add language to INSTALL that one should make sure this directory
> exists so as not to step on any distribution copy of the ntp library.

Looking at:

+Thw function get_python_lib() in the Python distutils library is the
+only reliable way to know where in fact the ntp Python librarty can
+installed; it normally returns sometyhing under /usr/lib.

Aside from the typos, the last statement only seems to be true of Linux;
*BSD returns something under /usr/local/lib, and OSX uses something
totally different.  So get_python_lib() has no FHS issues outside Linux.

Note that both the original and updated versions of massage() replace
BSD's /usr/local/lib with /usr/local/local/lib, which is certainly
undesirable, though I suppose the existence test will probably fail on the
result.  It might be better to test for /usr/<not local>/ before munging
anything.

Fred Wright

From hmurray at megapathdsl.net  Mon Oct  2 03:01:40 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 01 Oct 2017 20:01:40 -0700
Subject: Resolution of the library-path mess
In-Reply-To: Message from "Eric S. Raymond via devel" <devel@ntpsec.org>
 of "Sun,
 01 Oct 2017 15:46:46 EDT." <20171001194646.0B03513A0206@snark.thyrsus.com>
Message-ID: <20171002030140.90EF540605C@ip-64-139-1-69.sjc.megapath.net>


> On most systems this will do the right thing.

Which systems don't work right and/or what are we supposed to do there?

> I have implemented and tested this solution, along with full documentation
> and packaging guidance. 

I think the documentation needs more work in this area.  A separate file may 
be appropriate.

Where does FHS think we should install out code?  Why?

Are there 2 cases?  Developer mode and packaging for system distribution?

---------

Comments on packaging/packaging.txt

> == ntp.conf installation =
> The reason this is so is that NTPsec does not yet have an authorized
> pool group of its own. This may change in the future.

Even if we have a pool group, I don't think we should smash an existing ntp.conf

I think a warning message would be better than installing a default.

> Python librarty
typo - no "t" in library





-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Mon Oct  2 03:26:53 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 01 Oct 2017 20:26:53 -0700
Subject: What is the best sentence to use with NMEA?
Message-ID: <20171002032653.205F740605C@ip-64-139-1-69.sjc.megapath.net>


GPZDA doesn't have a valid/bogus flag
GPRMC only has 2 digit years.

I have a NMEA device that is in the old era.
   $GPRMC,030328,A,3726.0814,N,12212.2610,W,000.3,346.3,160298,015.5,E*60

I should be able to calculate the offset, but I haven't got my head around it 
yet.  It's some mix of year truncation and GPS epoch.

If it did the GPS fixup right, it would be off by 619315200 seconds. ntpq -p 
says it is off by 59184000.


-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Mon Oct  2 03:43:20 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sun, 1 Oct 2017 23:43:20 -0400
Subject: Resolution of the library-path mess
In-Reply-To: <Pine.LNX.4.58L1X.1710011918430.1188@bolt.sonic.net>
References: <20171001194646.0B03513A0206@snark.thyrsus.com>
 <20171001182422.78b4da19@spidey.rellim.com>
 <20171002014820.GA3947@thyrsus.com>
 <Pine.LNX.4.58L1X.1710011918430.1188@bolt.sonic.net>
Message-ID: <20171002034320.GA6850@thyrsus.com>

Fred Wright via devel <devel at ntpsec.org>:
> 
> On Sun, 1 Oct 2017, Eric S. Raymond via devel wrote:
> 
> > Gary E. Miller via devel <devel at ntpsec.org>:
> > > How do you plan that a local NTPsec install from source does not
> > > overwite an NTPsec install from the native OS repositories?
> >
> > That now will never happen if the /usr/local/lb/python-X.Y directory exists;
> > the install logic will notice that.
> 
> That of course assumes that if the directory exists, it's in sys.path.
> Perhaps that's a reasonable assumption, though it's hard to be sure.

Duh.  I should have added that check.  Doing it now.

> > I'll add language to INSTALL that one should make sure this directory
> > exists so as not to step on any distribution copy of the ntp library.
> 
> Looking at:
> 
> +Thw function get_python_lib() in the Python distutils library is the
> +only reliable way to know where in fact the ntp Python librarty can
> +installed; it normally returns sometyhing under /usr/lib.
> 
> Aside from the typos, the last statement only seems to be true of Linux;
> *BSD returns something under /usr/local/lib, and OSX uses something
> totally different.  So get_python_lib() has no FHS issues outside Linux.

I've notes that in the documentation.

> Note that both the original and updated versions of massage() replace
> BSD's /usr/local/lib with /usr/local/local/lib, which is certainly
> undesirable, though I suppose the existence test will probably fail on the
> result.  It might be better to test for /usr/<not local>/ before munging
> anything.

Hm, I just realized that what I should be checking there isn't /usr
but sys.prefix - which will normally be /usr under Linux, but might be
/usr/local under *BSD.  That should make the transformation into a no-op
in the case you just pointed out.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.



From esr at thyrsus.com  Mon Oct  2 03:48:20 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sun, 1 Oct 2017 23:48:20 -0400
Subject: Resolution of the library-path mess
In-Reply-To: <20171002020420.7847640605C@ip-64-139-1-69.sjc.megapath.net>
References: <20171002020420.7847640605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20171002034820.GA7250@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> 
> > I'll add language to INSTALL that one should make sure this directory exists
> > so as not to step on any distribution copy of the ntp library. 
> 
> That doesn't sound strong enough.  It's too easy to accidentally screw up.

What would your recommend instead?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.



From hmurray at megapathdsl.net  Mon Oct  2 03:58:27 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 01 Oct 2017 20:58:27 -0700
Subject: Resolution of the library-path mess
In-Reply-To: Message from "Eric S. Raymond via devel" <devel@ntpsec.org>
 of "Sun, 01 Oct 2017 23:48:20 EDT." <20171002034820.GA7250@thyrsus.com>
Message-ID: <20171002035827.E58C840605C@ip-64-139-1-69.sjc.megapath.net>

>>> I'll add language to INSTALL that one should make sure this directory 
exists
>>> so as not to step on any distribution copy of the ntp library.  
>> That doesn't sound strong enough.  It's too easy to accidentally screw up.

> What would your recommend instead?

I'm not sure.  This whole mess is mostly over my head.

How about making waf install do the check?  I think that means adding an 
option to bypass the check in case that's what you really want.  Or maybe 
just explicitly telling it where is good enough if that directory exists.



-- 
These are my opinions.  I hate spam.




From daniele at grinta.net  Mon Oct  2 03:59:16 2017
From: daniele at grinta.net (Daniele Nicolodi)
Date: Sun, 1 Oct 2017 21:59:16 -0600
Subject: Resolution of the library-path mess
In-Reply-To: <20171001194646.0B03513A0206@snark.thyrsus.com>
References: <20171001194646.0B03513A0206@snark.thyrsus.com>
Message-ID: <3e140269-b95e-73ea-bb90-58efbf8bbd36@grinta.net>

Hello,

I may be suggesting the obvious, but has anyone contacted the folks on
the Python Distutils mailing list
https://www.python.org/community/sigs/current/distutils-sig/ for their
advise?  I'm surprised NTPSec it the first project facing those problems.

Cheers,
Daniele

From esr at thyrsus.com  Mon Oct  2 04:14:16 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Mon, 2 Oct 2017 00:14:16 -0400
Subject: Resolution of the library-path mess
In-Reply-To: <20171002030140.90EF540605C@ip-64-139-1-69.sjc.megapath.net>
References: <devel@ntpsec.org> <20171001194646.0B03513A0206@snark.thyrsus.com>
 <20171002030140.90EF540605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20171002041416.GB7250@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> 
> > On most systems this will do the right thing.
> 
> Which systems don't work right and/or what are we supposed to do there?

Under my Ubuntu system, /usr/local/lib/python-X.Y already exists.  Under
some others, including Genooo, it doesn't.  In the latter case, this
directory needs to be manually created so that

1. Installations frpm source don't step on  /usr/lib/python-X.Y

2. The layout is FHS-compliant.

> I think the documentation needs more work in this area.  A separate file may 
> be appropriate.

I'm a lumper rather than a splitter in cases like this. Separtate files
have an unfortunate tendency to never get found.

> Where does FHS think we should install out code?  Why?
> 
> Are there 2 cases?  Developer mode and packaging for system distribution?

There are two cases. User-installed code and binaries are supposed to
go under /usr/local/X; stuff from packages goes under /usr/X, where X
= {bin, lib}.

> Comments on packaging/packaging.txt
> 
> > == ntp.conf installation =
> > The reason this is so is that NTPsec does not yet have an authorized
> > pool group of its own. This may change in the future.
> 
> Even if we have a pool group, I don't think we should smash an existing
> ntp.conf

Agreed.

> I think a warning message would be better than installing a default.

To be revisited when we have a pool group.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.



From esr at thyrsus.com  Mon Oct  2 04:26:40 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Mon, 2 Oct 2017 00:26:40 -0400
Subject: Resolution of the library-path mess
In-Reply-To: <20171002035827.E58C840605C@ip-64-139-1-69.sjc.megapath.net>
References: <devel@ntpsec.org> <20171002034820.GA7250@thyrsus.com>
 <20171002035827.E58C840605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20171002042640.GA7921@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> >>> I'll add language to INSTALL that one should make sure this directory 
> exists
> >>> so as not to step on any distribution copy of the ntp library.  
> >> That doesn't sound strong enough.  It's too easy to accidentally screw up.
> 
> > What would your recommend instead?
> 
> I'm not sure.  This whole mess is mostly over my head.
> 
> How about making waf install do the check?  I think that means adding an 
> option to bypass the check in case that's what you really want.  Or maybe 
> just explicitly telling it where is good enough if that directory exists.

As usual, I thing adding an option is a bad substitute for figuring out
what the right thing is and doing it.

It would be easy enough to throw a warning if the desired /usr/local directory
doesn't exist.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.



From esr at thyrsus.com  Mon Oct  2 04:27:23 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Mon,  2 Oct 2017 00:27:23 -0400 (EDT)
Subject: Draft blog post for review
Message-ID: <20171002042723.7C25D13A0206@snark.thyrsus.com>

Those of you with access to the project blog repo might want to review

_drafts/after-first-ship.ad

I want to have it all ready for shipping day on the 9th.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

In recent years it has been suggested that the Second Amendment
protects the "collective" right of states to maintain militias, while
it does not protect the right of "the people" to keep and bear arms.
If anyone entertained this notion in the period during which the
Constitution and the Bill of Rights were debated and ratified, it
remains one of the most closely guarded secrets of the eighteenth
century, for no known writing surviving from the period between 1787
and 1791 states such a thesis.
        -- Stephen P. Halbrook, "That Every Man Be Armed", 1984

From esr at thyrsus.com  Mon Oct  2 04:28:34 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Mon, 2 Oct 2017 00:28:34 -0400
Subject: Resolution of the library-path mess
In-Reply-To: <3e140269-b95e-73ea-bb90-58efbf8bbd36@grinta.net>
References: <20171001194646.0B03513A0206@snark.thyrsus.com>
 <3e140269-b95e-73ea-bb90-58efbf8bbd36@grinta.net>
Message-ID: <20171002042834.GB7921@thyrsus.com>

Daniele Nicolodi via devel <devel at ntpsec.org>:
> Hello,
> 
> I may be suggesting the obvious, but has anyone contacted the folks on
> the Python Distutils mailing list
> https://www.python.org/community/sigs/current/distutils-sig/ for their
> advise?  I'm surprised NTPSec it the first project facing those problems.
> 
> Cheers,
> Daniele

One of the things still on the todo list *is* filing an upstream bug
about this.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.



From hmurray at megapathdsl.net  Mon Oct  2 05:21:01 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 01 Oct 2017 22:21:01 -0700
Subject: Resolution of the library-path mess
In-Reply-To: Message from "Eric S. Raymond" <esr@thyrsus.com>
 of "Mon, 02 Oct 2017 00:26:40 EDT." <20171002042640.GA7921@thyrsus.com>
Message-ID: <20171002052101.C958640605C@ip-64-139-1-69.sjc.megapath.net>


> As usual, I thing adding an option is a bad substitute for figuring out what
> the right thing is and doing it.

Then create the directory.

> It would be easy enough to throw a warning if the desired /usr/local
> directory doesn't exist. 

Warnings are easily lost in the noise.  So either create the directory or 
treat it as an error and bail.

Even if you see a warning, it's too late.  You have already trashed the 
system directories.




-- 
These are my opinions.  I hate spam.




From gem at rellim.com  Mon Oct  2 06:07:04 2017
From: gem at rellim.com (Gary E. Miller)
Date: Sun, 1 Oct 2017 23:07:04 -0700
Subject: Resolution of the library-path mess
In-Reply-To: <Pine.LNX.4.58L1X.1710011918430.1188@bolt.sonic.net>
References: <20171001194646.0B03513A0206@snark.thyrsus.com>
 <20171001182422.78b4da19@spidey.rellim.com>
 <20171002014820.GA3947@thyrsus.com>
 <Pine.LNX.4.58L1X.1710011918430.1188@bolt.sonic.net>
Message-ID: <20171001230704.1af9d1c4@spidey.rellim.com>

Yo Fred!

On Sun, 1 Oct 2017 19:32:06 -0700 (PDT)
Fred Wright via devel <devel at ntpsec.org> wrote:

> On Sun, 1 Oct 2017, Eric S. Raymond via devel wrote:
> 
> > Gary E. Miller via devel <devel at ntpsec.org>:  
> > > How do you plan that a local NTPsec install from source does not
> > > overwite an NTPsec install from the native OS repositories?  
> >
> > That now will never happen if the /usr/local/lb/python-X.Y
> > directory exists; the install logic will notice that.  
> 
> That of course assumes that if the directory exists, it's in sys.path.
> Perhaps that's a reasonable assumption, though it's hard to be sure.

I would bet that /usr/local//lib/python-X.Y is NOT in the sys.path
no matter what.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can?t measure it, you can?t improve it." - Lord Kelvin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171001/0eda3566/attachment.bin>

From daniele at grinta.net  Mon Oct  2 06:14:55 2017
From: daniele at grinta.net (Daniele Nicolodi)
Date: Mon, 2 Oct 2017 00:14:55 -0600
Subject: Resolution of the library-path mess
In-Reply-To: <20171002042834.GB7921@thyrsus.com>
References: <20171001194646.0B03513A0206@snark.thyrsus.com>
 <3e140269-b95e-73ea-bb90-58efbf8bbd36@grinta.net>
 <20171002042834.GB7921@thyrsus.com>
Message-ID: <21e73d60-181f-1fc1-8456-63f198e2a5d3@grinta.net>

On 01/10/17 22:28, Eric S. Raymond wrote:
> Daniele Nicolodi via devel <devel at ntpsec.org>:
>> Hello,
>>
>> I may be suggesting the obvious, but has anyone contacted the folks on
>> the Python Distutils mailing list
>> https://www.python.org/community/sigs/current/distutils-sig/ for their
>> advise?  I'm surprised NTPSec it the first project facing those problems.
>>
>> Cheers,
>> Daniele
> 
> One of the things still on the todo list *is* filing an upstream bug
> about this.

I'm not sure a bug report is the way to go.  Despite that, contacting
the Python folks may help in deciding how to handle this.  I think
obeying the principle of less surprise and do what other projects do
would be the best option here.

Thinking a bit more about it, I think that autotools projects usually
have a --with-python=<path to python interpreter> configure option that
tells the install machinery which python interpreter the admin wishes to
use to run the script or for which interpreter the modules should be
available. Given that, the python install path is fixed.


Cheers,
Daniele

From hmurray at megapathdsl.net  Mon Oct  2 06:28:58 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 01 Oct 2017 23:28:58 -0700
Subject: Resolution of the library-path mess
Message-ID: <20171002062858.5F22840605C@ip-64-139-1-69.sjc.megapath.net>


> I would bet that /usr/local//lib/python-X.Y is NOT in the sys.path no matter
> what. 

So are we back to PYTHONPATH?


-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Mon Oct  2 06:31:33 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 01 Oct 2017 23:31:33 -0700
Subject: Resolution of the library-path mess
In-Reply-To: Message from "Eric S. Raymond via devel" <devel@ntpsec.org>
 of "Mon, 02 Oct 2017 00:14:16 EDT." <20171002041416.GB7250@thyrsus.com>
Message-ID: <20171002063133.E993840605C@ip-64-139-1-69.sjc.megapath.net>


> There are two cases. User-installed code and binaries are supposed to go
> under /usr/local/X; stuff from packages goes under /usr/X, where X = {bin,
> lib}. 

I think that means our normal devel-mode install is "user-installed code".   
I think that needs to go in our documentation.  (along with a quick summary 
of FHS: user-installed goes in local, system packages go in non-local.  A 
reference to FHS would be good.)

Do we want a system-code install mode?

Does install have a single where-to-put-it parameter?  Or do we need several, 
one for ntpd (sbin), one for python code (bin) and one for python libraries?


-------

Do we need a way to check that we are using the right library?

I think that means we need the version string or time stamp in both the 
library and the code that uses the library with a simple sanity check at the 
top of the uses code.


-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Mon Oct  2 06:52:26 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 01 Oct 2017 23:52:26 -0700
Subject: NMEA: Strange GPS warp
Message-ID: <20171002065226.3A32540605C@ip-64-139-1-69.sjc.megapath.net>

 1 Oct 23:48:57 ntpd[16245]: NMEA(0) Changed GPS epoch warp to -4096 weeks

Note that there is not TAG: on the front of that message, or this one:
 1 Oct 23:48:56 ntpd[16245]: NMEA(0) serial /dev/cuau0 open at 4800 bps


-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Mon Oct  2 11:00:41 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Mon, 2 Oct 2017 07:00:41 -0400
Subject: Resolution of the library-path mess
In-Reply-To: <20171002063133.E993840605C@ip-64-139-1-69.sjc.megapath.net>
References: <devel@ntpsec.org> <20171002041416.GB7250@thyrsus.com>
 <20171002063133.E993840605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20171002110041.GB14210@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> 
> > There are two cases. User-installed code and binaries are supposed to go
> > under /usr/local/X; stuff from packages goes under /usr/X, where X = {bin,
> > lib}. 
> 
> I think that means our normal devel-mode install is "user-installed code".   

Correct.

> I think that needs to go in our documentation.  (along with a quick summary 
> of FHS: user-installed goes in local, system packages go in non-local.  A 
> reference to FHS would be good.)

I've done some documentation polishing to address this.

> Do we want a system-code install mode?

That's what you get if you specify --prefix=/usr, as opposed to the default
prefix of /usr/local.

> Does install have a single where-to-put-it parameter?  Or do we need several, 
> one for ntpd (sbin), one for python code (bin) and one for python libraries?

There is just one.  That is what --prefix is.  There is a separate option
--destdir that chabges the root of the *entire* installation hierchy.
The default of --destdir is /.

User installs under /usr/local - and both these options - are well-known
conventions established decades ago by autoconf.  Then later inherited by
waf and FHS.  They're pretty much universal across build systems.

> Do we need a way to check that we are using the right library?
> 
> I think that means we need the version string or time stamp in both the 
> library and the code that uses the library with a simple sanity check at the 
> top of the uses code.

It's not normal practice to do this in Python. GPSD never has.

I think the reason is that when you do have such a mismatch, the Python
stack trace it generally produces is likely to be more informative than
a version-mismatch message.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.



From ianbruene at gmail.com  Mon Oct  2 13:50:46 2017
From: ianbruene at gmail.com (Ian Bruene)
Date: Mon, 2 Oct 2017 08:50:46 -0500
Subject: Nailing an elusive unicode bug, and a heads up
Message-ID: <8b4aa36d-836e-315d-3081-99c91bca8fb0@gmail.com>


Several months ago when I added the unit display feature there was a bug 
that caused ntpq to crash with a unicode error for no apparent reason. 
The crash was never replicated, but I added some debugging statements in 
hope of catching it. Someday.

In the last week during the release delay Gary and I stumbled on a 
little known feature of Python where the encoding of the standard 
streams can be changed by an environment variable. Tests unearthed a 
number of unicode bugs due to this in both ntpq and ntpmon, ntpq was 
fixed to detect via sys.stdout.encoding if it was talking over UTF-8 and 
if not to use a wrapper to force UTF-8 output. ntpmon unfortunately 
could not be fixed in this way; it uses the curses library which 
interfered with this solution. ntpmon now downgrades to a non-unicode 
version if it encounters this problem.

At the end of last week I got another bug report for ntpq from jasonium 
which matched the old never replicated bug. Today I got the full 
traceback from him and discovered that sys.stdout.encoding sometimes 
lies, sometimes claiming that it is UTF-8 when it is not. ntpq now 
always forces UTF-8 regardless of what python claims it is.

Be advised: sys.stdwhatever.encoding can be fake news.

-- 
In the end; what separates a Man, from a Slave? Money? Power?
No. A Man Chooses, a Slave Obeys. -- Andrew Ryan


From hmurray at megapathdsl.net  Mon Oct  2 18:48:41 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Mon, 02 Oct 2017 11:48:41 -0700
Subject: Resolution of the library-path mess
In-Reply-To: Message from "Eric S. Raymond via devel" <devel@ntpsec.org>
 of "Mon, 02 Oct 2017 07:00:41 EDT." <20171002110041.GB14210@thyrsus.com>
Message-ID: <20171002184841.D941C40605C@ip-64-139-1-69.sjc.megapath.net>

>> Do we need a way to check that we are using the right library?

>> I think that means we need the version string or time stamp in both the 
>> library and the code that uses the library with a simple sanity check at 
the
>> top of the uses code.

> It's not normal practice to do this in Python. GPSD never has.

> I think the reason is that when you do have such a mismatch, the Python
> stack trace it generally produces is likely to be more informative than
> a version-mismatch message.

Given the confusion we have had in this area, it seemed like a simple and low 
cost way to avoid what might turn into a long chase.  Sure, if you get a 
messy backtrace you can probably figure out that you have the wrong library.  
But what if it doesn't crash and just gives the wrong answer?

-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Mon Oct  2 19:30:32 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Mon, 2 Oct 2017 15:30:32 -0400
Subject: Resolution of the library-path mess
In-Reply-To: <20171002184841.D941C40605C@ip-64-139-1-69.sjc.megapath.net>
References: <devel@ntpsec.org> <20171002110041.GB14210@thyrsus.com>
 <20171002184841.D941C40605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20171002193032.GA23355@thyrsus.com>

Hal Murray <hmurray at megapathdsl.net>:
> >> Do we need a way to check that we are using the right library?
> 
> >> I think that means we need the version string or time stamp in both the 
> >> library and the code that uses the library with a simple sanity check at 
> the
> >> top of the uses code.
> 
> > It's not normal practice to do this in Python. GPSD never has.
> 
> > I think the reason is that when you do have such a mismatch, the Python
> > stack trace it generally produces is likely to be more informative than
> > a version-mismatch message.
> 
> Given the confusion we have had in this area, it seemed like a simple and low 
> cost way to avoid what might turn into a long chase.  Sure, if you get a 
> messy backtrace you can probably figure out that you have the wrong library.  
> But what if it doesn't crash and just gives the wrong answer?

It is rather unusual to be able to address that effectively with a
version check.  I say this as a person who has written more than his
share of service libraries and clients for them.

Yes, in theory one would think this is a common case (error discovered without
API change).  But thinking back I can only remember one time *I* ever did this -
that was when I discovered that the error-bounds computation in GPSD had been
slightly wrong for years because of a duplicated variable in a nested loop.

I bumped the library version, sure, but it didn't matter.  As with our
Python libraries, libgpsd is never updated separately from its client code.

I deduce that my experience is representative from the fact that Python
library version checks are not routine in other peoples' code.  They're
not nonexistent - GTK clients commonly do this - so people do know it
is possible.

So, think through the cases. If a library is always shipped coupled with
its client code, you can't get the kind of skew a version check would
notice.  And...how is a third-party client to know when it should bump
the floor to check for?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.



From esr at thyrsus.com  Tue Oct  3 09:32:00 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Tue,  3 Oct 2017 05:32:00 -0400 (EDT)
Subject: The freeze is on
Message-ID: <20171003093200.54A1013A0206@snark.thyrsus.com>

It appears from recent traffic that we have a resiolution of the
library-path mess that is, if not perfect, at least acceptable to
all.  It's good that PYTHONPATH is gone.

Thus, the freeze is on again. Expected ship date is Monday, October
9th on or slightly before 5PM Eastern.

>From this point, bug fixes only. There are presently just four non-RFE
issues on the tracker.

My next work item is to bring the Microserver HOWTO up to date.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Question with boldness even the existence of a God; because, if there
be one, he must more approve the homage of reason, than that of
blindfolded fear.... Do not be frightened from this inquiry from any
fear of its consequences. If it ends in the belief that there is no
God, you will find incitements to virtue in the comfort and
pleasantness you feel in its exercise...
	-- Thomas Jefferson, in a 1787 letter to his nephew

From Stromeko at nexgo.de  Tue Oct  3 09:57:39 2017
From: Stromeko at nexgo.de (Achim Gratz)
Date: Tue, 03 Oct 2017 11:57:39 +0200
Subject: Timestamp in build
Message-ID: <87efqka6zw.fsf@Rainer.invalid>


The timestamp compiled into ntpd and shown in the ntpq variables output
has a different timezone depending on conditions that I can't determine:

Raspbian(jessie):
version="ntpd ntpsec-0.9.7+1468 2017-10-02T08:25:48-0500"

Linaro(stretch)
version="ntpd ntpsec-0.9.7+1468 2017-10-02T13:25:48Z"

The git show command on both machines uses the "-0500" time zone
designation, so that's not where the difference comes from.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra


From hmurray at megapathdsl.net  Wed Oct  4 20:43:58 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Wed, 04 Oct 2017 13:43:58 -0700
Subject: The freeze is on
In-Reply-To: Message from "Eric S. Raymond via devel" <devel@ntpsec.org>
 of "Tue,
 03 Oct 2017 05:32:00 EDT." <20171003093200.54A1013A0206@snark.thyrsus.com>
Message-ID: <20171004204358.630A740605C@ip-64-139-1-69.sjc.megapath.net>


> It appears from recent traffic that we have a resiolution of the
> library-path mess that is, if not perfect, at least acceptable to all.  It's
> good that PYTHONPATH is gone. 

Is there a clear statement someplace of what is supposed to happen?

On Fedora, the python library stuff is getting stored in /usr/lib (no local). 
 Is that correct?

Is anybody else testing on Fedora?

(The electrical power to my toys is on the fritz so I can't verify things and/or won't be able to do as much testing as I'd like.)  I'm pretty sure I saw our libraries in both /usr/lib/ and /usr/local/lib/ but I don't remember where and/or I think that system is offline now.


> Thus, the freeze is on again. Expected ship date is Monday, October 9th on
> or slightly before 5PM Eastern. 

I assume that applies to code and that it is OK to improve the documentation.


I suggest that you bump the current version number (0.9.9?) so we can all be sure we are testing the to-be-released code.  And make a tarball so we can test that too.

How about cloning devel/pre-release.txt and annotating it with what we have tested?  That would serve as documentation post-release as well as letting us know where to focus additional work.


-- 
These are my opinions.  I hate spam.




From fw at fwright.net  Thu Oct  5 21:10:24 2017
From: fw at fwright.net (Fred Wright)
Date: Thu, 5 Oct 2017 14:10:24 -0700 (PDT)
Subject: Resolution of the library-path mess
In-Reply-To: <20171002063133.E993840605C@ip-64-139-1-69.sjc.megapath.net>
References: <20171002063133.E993840605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <Pine.LNX.4.58L1X.1710051338060.29004@bolt.sonic.net>


On Sun, 1 Oct 2017, Eric S. Raymond wrote:
> Fred Wright via devel <devel at ntpsec.org>:
> > On Sun, 1 Oct 2017, Eric S. Raymond via devel wrote:
> > > Gary E. Miller via devel <devel at ntpsec.org>:
> > > > How do you plan that a local NTPsec install from source does not
> > > > overwite an NTPsec install from the native OS repositories?
> > >
> > > That now will never happen if the /usr/local/lb/python-X.Y directory exists;
> > > the install logic will notice that.
> >
> > That of course assumes that if the directory exists, it's in sys.path.
> > Perhaps that's a reasonable assumption, though it's hard to be sure.
>
> Duh.  I should have added that check.  Doing it now.

Sorry for the lateness, but I realized that the current code still has a
bug (as well as a couple of deficiencies of a more-or-less cosmetic
nature).  It's currently checking sys.path in the *running* Python, but it
needs to be checking it in the *target* Python.  Specifying a different
Python otherwise works, so this should respect the difference.

One ofthe "cosmetic" issues is that the directory existence check is
superfluous.  The case table looks like this:

Exists?		In sys.path?	Usable?
-------		------------	-------
No     		No          	Unknown
No     		Yes		Yes, but this can't happen (*)
Yes    		No		No
Yes    		Yes		Yes

* site.py's sys.path setup filters nonexistent directories

Since the install procedure would create the directory if needed, testing
for its preexistence is at best useless.

> Hm, I just realized that what I should be checking there isn't /usr
> but sys.prefix - which will normally be /usr under Linux, but might be
> /usr/local under *BSD.  That should make the transformation into a no-op
> in the case you just pointed out.

The other "cosmetic" issue is that there's no need to "manually" construct
'localized' at all, since that's what get_python_lib() with the prefix
supplied already does.  The only reason to avoid the value that waf comes
up with is that it might not be in sys.path, but *conditionally* using
that value is fine, and simpler.

Thus, the corrected and simplified logic becomes:

-------------------------------------------------------------------------
If the path isn't supplied explicitly and the value obtained by waf isn't
in the target Python's sys.path, then replace it with the target Python's
prefixless get_python_lib() result.
-------------------------------------------------------------------------

I'll make (and test) a version that does that.

On Sun, 1 Oct 2017, Hal Murray via devel wrote:

> > It would be easy enough to throw a warning if the desired /usr/local
> > directory doesn't exist.
>
> Warnings are easily lost in the noise.  So either create the directory or
> treat it as an error and bail.

There are two issues with just "creating the directory":

1) There's no guarantee that Python will actually use it.

2) Creating the directory requires root, and it would be undesirable to
require that 'configure' run as root.

Some improvement in this area is needed, but probably post-1.0.

On Mon, 2 Oct 2017, Daniele Nicolodi via devel wrote:

> On 01/10/17 22:28, Eric S. Raymond wrote:
> > Daniele Nicolodi via devel <devel at ntpsec.org>:
> >> Hello,
> >>
> >> I may be suggesting the obvious, but has anyone contacted the folks on
> >> the Python Distutils mailing list
> >> https://www.python.org/community/sigs/current/distutils-sig/ for their
> >> advise?  I'm surprised NTPSec it the first project facing those problems.
> >>
> >> Cheers,
> >> Daniele
> >
> > One of the things still on the todo list *is* filing an upstream bug
> > about this.
>
> I'm not sure a bug report is the way to go.  Despite that, contacting
> the Python folks may help in deciding how to handle this.  I think
> obeying the principle of less surprise and do what other projects do
> would be the best option here.
>
> Thinking a bit more about it, I think that autotools projects usually
> have a --with-python=<path to python interpreter> configure option that
> tells the install machinery which python interpreter the admin wishes to
> use to run the script or for which interpreter the modules should be
> available. Given that, the python install path is fixed.

That's not the issue.  It's already possibly to specify which Python to
target, and that works (modulo the bug noted above).  But there's no
reliable way to map that to a library directory which is both recognized
by Python and properly prefixed.

The whole concept of including Python-based programs in a base install may
be too new for anyone to have considered all the ramifications.  That's
not an excuse for stuff that's just broken, but it may explain why nobody
has been screaming about it.

On Sun, 1 Oct 2017, Hal Murray via devel wrote:

> > I would bet that /usr/local//lib/python-X.Y is NOT in the sys.path no matter
> > what.

You'd lose that bet on Debian and Ubuntu (at least the versions I have
here), provided that the directory exists.

> So are we back to PYTHONPATH?

No, since it now only uses that directory when it *is* in sys.path and
therefor doesn't have to be added via PYTHONPATH.

On Sun, 1 Oct 2017, Hal Murray via devel wrote:

> Does install have a single where-to-put-it parameter?  Or do we need several,
> one for ntpd (sbin), one for python code (bin) and one for python libraries?

That's what PREFIX is for (as the nominal parent of all those places);
it's just that Python doesn't always allow one to honor it and still have
something that actually works.

Fred Wright

From gem at rellim.com  Fri Oct  6 00:23:46 2017
From: gem at rellim.com (Gary E. Miller)
Date: Thu, 5 Oct 2017 17:23:46 -0700
Subject: Resolution of the library-path mess
In-Reply-To: <Pine.LNX.4.58L1X.1710051338060.29004@bolt.sonic.net>
References: <20171002063133.E993840605C@ip-64-139-1-69.sjc.megapath.net>
 <Pine.LNX.4.58L1X.1710051338060.29004@bolt.sonic.net>
Message-ID: <20171005172346.155d8a88@spidey.rellim.com>

Yo Fred!

On Thu, 5 Oct 2017 14:10:24 -0700 (PDT)
Fred Wright via devel <devel at ntpsec.org> wrote:

> Exists?		In sys.path?	Usable?
> -------		------------	-------
> No     		No          	Unknown
> No     		Yes		Yes, but this can't happen (*)
> Yes    		No		No
> Yes    		Yes		Yes

Uh, I disagree on the 3rd case (Yes, No).  I install, then add my
dir to PYTHONPATH.

I guess we could document that: for people that want to be HFS
conformant, add PYTHONPATH before doing the install.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can?t measure it, you can?t improve it." - Lord Kelvin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171005/4455de31/attachment.bin>

From daniele at grinta.net  Fri Oct  6 03:20:36 2017
From: daniele at grinta.net (Daniele Nicolodi)
Date: Thu, 5 Oct 2017 21:20:36 -0600
Subject: Resolution of the library-path mess
In-Reply-To: <20171005172346.155d8a88@spidey.rellim.com>
References: <20171002063133.E993840605C@ip-64-139-1-69.sjc.megapath.net>
 <Pine.LNX.4.58L1X.1710051338060.29004@bolt.sonic.net>
 <20171005172346.155d8a88@spidey.rellim.com>
Message-ID: <f61d0c13-50b4-709d-89b7-a7d6e077fba0@grinta.net>

On 10/5/17 6:23 PM, Gary E. Miller via devel wrote:
> Yo Fred!
> 
> On Thu, 5 Oct 2017 14:10:24 -0700 (PDT)
> Fred Wright via devel <devel at ntpsec.org> wrote:
> 
>> Exists?		In sys.path?	Usable?
>> -------		------------	-------
>> No     		No          	Unknown
>> No     		Yes		Yes, but this can't happen (*)
>> Yes    		No		No
>> Yes    		Yes		Yes
> 
> Uh, I disagree on the 3rd case (Yes, No).  I install, then add my
> dir to PYTHONPATH.
> 
> I guess we could document that: for people that want to be HFS
> conformant, add PYTHONPATH before doing the install.

I alredy suggested that, but it has been ignored. Let's try again.

If you want to install in /usr/local or wherever else:

python3 -m venv --system-site-packages --without-pip /usr/local/
waf configure --python=/usr/local/bin/python3

and all should work (assuming that waf does the right thing when a
python interpreter is specified at configure time, ie replace the right
executable on the shebang of the installed scripts).

Cheers,
Daniele

From hmurray at megapathdsl.net  Fri Oct  6 04:51:32 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Thu, 05 Oct 2017 21:51:32 -0700
Subject: Resolution of the library-path mess
In-Reply-To: Message from Fred Wright via devel <devel@ntpsec.org> of "Thu,
 05 Oct 2017 14:10:24 PDT."
 <Pine.LNX.4.58L1X.1710051338060.29004@bolt.sonic.net>
Message-ID: <20171006045133.05F0740605C@ip-64-139-1-69.sjc.megapath.net>

>> Warnings are easily lost in the noise.  So either create the
>> directory or treat it as an error and bail.

> There are two issues with just "creating the directory":
> 1) There's no guarantee that Python will actually use it.
> 2) Creating the directory requires root, and it would be
> undesirable to require that 'configure' run as root.

The actual create can be deferred until the install.

I don't understand the comment about "actually use it"?  Is that any 
different from the case where the directory did exist?

-- 
These are my opinions.  I hate spam.




From daniele at grinta.net  Fri Oct  6 05:39:51 2017
From: daniele at grinta.net (Daniele Nicolodi)
Date: Thu, 5 Oct 2017 23:39:51 -0600
Subject: Resolution of the library-path mess
In-Reply-To: <20171006045133.05F0740605C@ip-64-139-1-69.sjc.megapath.net>
References: <20171006045133.05F0740605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <c522a638-a0ce-f0b7-df9f-04dd42f03aee@grinta.net>

On 05/10/17 22:51, Hal Murray via devel wrote:
>>> Warnings are easily lost in the noise.  So either create the
>>> directory or treat it as an error and bail.
> 
>> There are two issues with just "creating the directory":
>> 1) There's no guarantee that Python will actually use it.
>> 2) Creating the directory requires root, and it would be
>> undesirable to require that 'configure' run as root.
> 
> The actual create can be deferred until the install.
> 
> I don't understand the comment about "actually use it"?  Is that any 
> different from the case where the directory did exist?

I will repeat myself once more.

If you want a clean way to install Python modules outside the system
path (for example in /usr/local/, the default prefix directory for
NTPSec) you create a python virtual environment in /usr/local

python3 -m venv --system-site-packages --without-pip /usr/local/

and then tell waf to install the Python modules there:

waf configure --python=/usr/local/bin/python3

clean and effective.

Creating a virtual environment you (virtually) install a new interpreter
there (a few symlinks).  The reason why Python does not include
/usr/local folders in his search path by default is: what if you want to
install an alternative Pythion _interpreter_ there?  That would be a
nice recipe for troubles.

Cheers,
Daniele

From esr at thyrsus.com  Fri Oct  6 11:52:47 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Fri,  6 Oct 2017 07:52:47 -0400 (EDT)
Subject: Testing float-valued functions
Message-ID: <20171006115247.C7BD413A03F2@snark.thyrsus.com>

This is a training assignment for Ian Bruene.  I'm describing the
background in public because this is an instance of a generic error
other programmers are likely to run into, and all need to guard
against it.

As it presently exists, the pylib/packet.py function def
ntp_to_posix(t) is incorrect.  Here is how it now reads:

    @staticmethod
    def ntp_to_posix(t):
        "Scale from NTP time to POSIX time"
        # Note: assumes we're in the same NTP era as the transmitter...
        return (t >> 32) - SyncPacket.UNIX_EPOCH

The problem with this code is that it throws away the low 32 bits of
information in t, the fractional-second part.  Writing it this way was
a coding error on my part which produced this bug:

https://gitlab.com/NTPsec/ntpsec/issues/402

The fractional-seconds part of ntpdig's report was simply missing
because ntp_to_posix() had discarded it.

I fixed this on 2 Oct with the commit "Address GitLab issue #402:
ntpdig: no fraction of seconds", which changed the function to return
a float, as follows:

    def ntp_to_posix(t):
        "Scale from NTP time to POSIX time"
        # Do not do the obvious thing with a shift, the low 32 bits
        # need to become decimal fractional seconds.
        # Note: assumes we're in the same NTP era as the transmitter...
        return (t / (2**32)) - SyncPacket.UNIX_EPOCH

This code is functionally correct, but unmasked an issue wth the test
code for ntp_to_posix() and posix_to_ntp().  Here is that code:

    def test_ntp_to_posix(self):
        f = self.target.ntp_to_posix
        # Test the Timeless Void Before Existence
        self.assertEqual(f(-1), -2208988801)  # NTP can see the Timeless Void
        # Test beginning of NTP epoch
        self.assertEqual(f(0), -2208988800)
        # Test just before the Age of Unix
        self.assertEqual(f(2208988799 * 2**32), -1)
        # Test the beginning of the Age of Unix
        self.assertEqual(f(2208988800 * 2**32), 0)
        # Test the End of Time
        self.assertEqual(f(0xFFFFFFFF00000000), 2085978495)
        # Test the Timeless Void Beyond Existence
        self.assertEqual(f(0x10000000000000000), 2085978496)  # It doesn't clip

    def test_posix_to_ntp(self):
        f = self.target.posix_to_ntp
        # Test the Timeless Void Before Existence
        self.assertEqual(f(-2208988801), -0x100000000)  # It doesn't clip
        # Test beginning of NTP epoch
        self.assertEqual(f(-2208988800), 0)
        # Test just before the Age of Unix
        self.assertEqual(f(-1), 2208988799 * 2**32)
        # Test the beginning of the Age of Unix
        self.assertEqual(f(0), 2208988800 * 2**32)
        # Test the End of Time
        self.assertEqual(f(2085978495), 0xFFFFFFFF00000000)
        # Test the Timeless Void Beyond Existence
        self.assertEqual(f(2085978496), 0x10000000000000000)  # It doesn't clip

The specific problem we began to see was that the assertion

        self.assertEqual(f(-1), -2208988801

began to fail.  But not everywhere - on 18 of 20 platforms in GitLab CI.

In fact, all this test code is subtly wrong, and it is just blind luck
nothing went sproing sooner.  Any of those assertions could have gone
toes-up at any time. The tests in posixize() are wrong, too. 

The problem here is that float representation is not exact, and the
"same" float calculation on different platforms may yield slightly
different results.  Thus comparing floats to floats is flaky,
and comparing ints to floats is flaky.  This is true no matter
what language you are using; it's an intrinsic property of the
float data representation.

This is why the unit-test framework has an assertAlmostEqual(x, y)
function - it's there exactly to deal with float fuzziness.

The flakiness of floats is a trap lying in wait for even experienced
programmers - it's easy to forget that functions are float-valued
rather than int. Little blame attaches to a novice getting caught,
but one should learn from it.

Ian, your assignment is to fix this and verify the fix.  Please do
this as immediately as possible.

A question for you to ponder: should you replace all assertEqual calls
or only those on which we have seen failures?  If the latter, what
else do you need to do?

Be prepared to discuss the tradeoffs and justify your answer.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Probably fewer than 2% of handguns and well under 1% of all guns will
ever be involved in a violent crime. Thus, the problem of criminal gun
violence is concentrated within a very small subset of gun owners,
indicating that gun control aimed at the general population faces a
serious needle-in-the-haystack problem.
	-- Gary Kleck, "Point Blank: Handgun Violence In America"

From esr at thyrsus.com  Fri Oct  6 13:32:06 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Fri,  6 Oct 2017 09:32:06 -0400 (EDT)
Subject: Library-path glitch, again
Message-ID: <20171006133206.9E10813A03F2@snark.thyrsus.com>

Fred Wright writes:
>Sorry for the lateness, but I realized that the current code still has a
>bug (as well as a couple of deficiencies of a more-or-less cosmetic
>nature).  It's currently checking sys.path in the *running* Python, but it
>needs to be checking it in the *target* Python.  Specifying a different
>Python otherwise works, so this should respect the difference.
>
>One ofthe "cosmetic" issues is that the directory existence check is
>superfluous.  The case table looks like this:
>
>Exists?         In sys.path?    Usable?
>-------         ------------    -------
>No              No              Unknown
>No              Yes             Yes, but this can't happen (*)
>Yes             No              No
>Yes             Yes             Yes
>
>* site.py's sys.path setup filters nonexistent directories
>
>Since the install procedure would create the directory if needed, testing
>for its preexistence is at best useless.
>
>The other "cosmetic" issue is that there's no need to "manually" construct
>'localized' at all, since that's what get_python_lib() with the prefix
>supplied already does.  The only reason to avoid the value that waf comes
>up with is that it might not be in sys.path, but *conditionally* using
>that value is fine, and simpler.

But Gary replies:

>Uh, I disagree on the 3rd case (Yes, No).  I install, then add my
>dir to PYTHONPATH.
>
>I guess we could document that: for people that want to be HFS
>conformant, add PYTHONPATH before doing the install.

Then Daniele Nicolodi says:

>I alredy suggested that, but it has been ignored. Let's try again.
>
>If you want to install in /usr/local or wherever else:
>
>python3 -m venv --system-site-packages --without-pip /usr/local/
>waf configure --python=/usr/local/bin/python3
>
>and all should work (assuming that waf does the right thing when a
>python interpreter is specified at configure time, ie replace the right
>executable on the shebang of the installed scripts).

Fred, your timing is awful and you shouldn't have submitted multiple
changes as a single MR.  Three days ago I might have taken a single,
simple change that fixed the one bug.  Now I can't justify doing that
- especially when Gary has raised a question about your analysis.

This is not a time for cosmetic fixes.

Daniele, you weren't ignored.  I read and processed what you said,
but it's way too late in the cycle to redo the build around a technique
(virtual environments) that the senior devs don't have experience with.

Unless somebody explains a *release-blocking* bug in the build to me,
we're going to ship the recipe as-is and futz with it after 1.0. It's
time to get the scientists away from them thar rocket and shoot it.

The one patch I would like to see is to devel/TODO explaining the
source/target bug.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

The Constitution is not neutral. It was designed to take the
government off the backs of the people.
	-- Justice William O. Douglas 

From ianbruene at gmail.com  Fri Oct  6 13:59:31 2017
From: ianbruene at gmail.com (Ian Bruene)
Date: Fri, 6 Oct 2017 08:59:31 -0500
Subject: Testing float-valued functions
In-Reply-To: <20171006115247.C7BD413A03F2@snark.thyrsus.com>
References: <20171006115247.C7BD413A03F2@snark.thyrsus.com>
Message-ID: <9a7d6b34-0160-2250-7c2f-f54a5768cd02@gmail.com>



On 10/06/2017 06:52 AM, Eric S. Raymond wrote:
> In fact, all this test code is subtly wrong, and it is just blind luck
> nothing went sproing sooner.  Any of those assertions could have gone
> toes-up at any time. The tests in posixize() are wrong, too.
>
> The problem here is that float representation is not exact, and the
> "same" float calculation on different platforms may yield slightly
> different results.  Thus comparing floats to floats is flaky,
> and comparing ints to floats is flaky.  This is true no matter
> what language you are using; it's an intrinsic property of the
> float data representation.
>
> [...]
> Ian, your assignment is to fix this and verify the fix.  Please do
> this as immediately as possible.

I changed ntp_to_posix back to what it should be, and fixed what tests I 
can fix immediately.

> A question for you to ponder: should you replace all assertEqual calls
> or only those on which we have seen failures?  If the latter, what
> else do you need to do?
>
> Be prepared to discuss the tradeoffs and justify your answer.

For ntp_to_posix and posixize yes, all of the float tests should be 
changed. Without changing them J. Random System might throw a fit at any 
time. For ntp_to_posix it is more complicated; it takes a float, so 
representation issues will arise. However it returns an integer, so a 
simple assertAlmostEqual() won't cut it. This will probably require some 
custom test code.

Today I will do a run through the tests to find other float-returning 
testees that need adjustment.

A related issue is that I am unsure of the value of the Timeless Void 
tests. Those are testing values outside of what NTP represents, and in 
theory are the same kind of unnecessary test as force feeding a list to 
a function that expects a bool. On the other hand one of those 
"unnecessary" tests was the one that caught the bug.

-- 
In the end; what separates a Man, from a Slave? Money? Power?
No. A Man Chooses, a Slave Obeys. -- Andrew Ryan


From esr at thyrsus.com  Fri Oct  6 15:15:00 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Fri, 6 Oct 2017 11:15:00 -0400
Subject: Testing float-valued functions
In-Reply-To: <9a7d6b34-0160-2250-7c2f-f54a5768cd02@gmail.com>
References: <20171006115247.C7BD413A03F2@snark.thyrsus.com>
 <9a7d6b34-0160-2250-7c2f-f54a5768cd02@gmail.com>
Message-ID: <20171006151500.GA19905@thyrsus.com>

Ian Bruene via devel <devel at ntpsec.org>:
> I changed ntp_to_posix back to what it should be, and fixed what tests I can
> fix immediately.

Good.

> >A question for you to ponder: should you replace all assertEqual calls
> >or only those on which we have seen failures?  If the latter, what
> >else do you need to do?
> >
> >Be prepared to discuss the tradeoffs and justify your answer.
> 
> For ntp_to_posix and posixize yes, all of the float tests should be changed.
> Without changing them J. Random System might throw a fit at any time.

That is one correct answer.

But: not quite any time. In practice, you have two windows of vulnerability:

(1) When you add a new platform to supported ports.

(2) Compiler upgrades that change how FP code is generated.  And, actually,
we might not see one of those again, now that on-chip FP hardware conforming
to IEEE754 is ubiquitous.

Thus, the *other* correct answer would be to change the tests only as
needed, but document in the test code that this is what you are doing
and why.  Then count on the tests to break constructively on new
platforms.

Neither answer is wrong, they just have different tradeoffs.  

> For ntp_to_posix it is more complicated; it takes a float, so representation
> issues will arise. However it returns an integer, so a simple
> assertAlmostEqual() won't cut it. This will probably require some custom
> test code.

This is why the minimal-change alternative is worth considering at this
point in our release cycle.  It means you don't have to do that research
project.

> Today I will do a run through the tests to find other float-returning
> testees that need adjustment.

And, of course, you'll stay aware of the issue in the future. So this
is all good.

> A related issue is that I am unsure of the value of the Timeless Void tests.
> Those are testing values outside of what NTP represents, and in theory are
> the same kind of unnecessary test as force feeding a list to a function that
> expects a bool. On the other hand one of those "unnecessary" tests was the
> one that caught the bug.

True, but keep firmly in mind that this was a bug in *tests*, not a bug in
code.  You may indeed have ventured into over-testing here - it's rare, but it
does happen.

My advice is to keep the out-of-domain tests, but mark them as low-value
and disposable if passing them would incur additional complexity costs.

But exercise your judgment.  You own the Python code, so I'm only
advising you rather than issuing a diktat.  Part of your training is
making these decisions under conditions where you're allowed to call
things wrong and observe the consequences, provided doing so does not
significantly threaten our code quality.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.



From daniele at grinta.net  Fri Oct  6 15:32:12 2017
From: daniele at grinta.net (Daniele Nicolodi)
Date: Fri, 6 Oct 2017 09:32:12 -0600
Subject: Library-path glitch, again
In-Reply-To: <20171006133206.9E10813A03F2@snark.thyrsus.com>
References: <20171006133206.9E10813A03F2@snark.thyrsus.com>
Message-ID: <209b3fd4-c946-09e1-1b5e-98a955f7ecad@grinta.net>

On 06/10/17 07:32, Eric S. Raymond via devel wrote:
> Daniele, you weren't ignored.  I read and processed what you said,
> but it's way too late in the cycle to redo the build around a technique
> (virtual environments) that the senior devs don't have experience with.

I don't understand this statement.  There is nothing that needs to
change in the buid process (if it works for installing in the default
python module path).  Creating the virtual env is something that the
administrator needs to do (what if I have a Python interpreter installed
in /usr/local?).  I definitely don't want the NTPSec install do anything
else than copy NTPSec files in the right place.

Cheers,
Daniele

From ianbruene at gmail.com  Fri Oct  6 16:50:30 2017
From: ianbruene at gmail.com (Ian Bruene)
Date: Fri, 6 Oct 2017 11:50:30 -0500
Subject: Testing float-valued functions
In-Reply-To: <20171006151500.GA19905@thyrsus.com>
References: <20171006115247.C7BD413A03F2@snark.thyrsus.com>
 <9a7d6b34-0160-2250-7c2f-f54a5768cd02@gmail.com>
 <20171006151500.GA19905@thyrsus.com>
Message-ID: <5258b2f6-da19-8db0-07aa-eceff71d691a@gmail.com>



On 10/06/2017 10:15 AM, Eric S. Raymond wrote:

> This is why the minimal-change alternative is worth considering at this
> point in our release cycle.  It means you don't have to do that research
> project.

Done, functions that can be simply changed to assertAlmostEqual have 
been, problematic ones have comments documenting the potential problem.

> My advice is to keep the out-of-domain tests, but mark them as low-value
> and disposable if passing them would incur additional complexity costs.

Done.

-- 
In the end; what separates a Man, from a Slave? Money? Power?
No. A Man Chooses, a Slave Obeys. -- Andrew Ryan


From gem at rellim.com  Fri Oct  6 18:50:03 2017
From: gem at rellim.com (Gary E. Miller)
Date: Fri, 6 Oct 2017 11:50:03 -0700
Subject: Library-path glitch, again
In-Reply-To: <20171006133206.9E10813A03F2@snark.thyrsus.com>
References: <20171006133206.9E10813A03F2@snark.thyrsus.com>
Message-ID: <20171006115003.062c51db@spidey.rellim.com>

Yo Eric!

On Fri,  6 Oct 2017 09:32:06 -0400 (EDT)
"Eric S. Raymond via devel" <devel at ntpsec.org> wrote:

> Fred, your timing is awful and you shouldn't have submitted multiple
> changes as a single MR.  Three days ago I might have taken a single,
> simple change that fixed the one bug.  Now I can't justify doing that
> - especially when Gary has raised a question about your analysis.

Uh, I have been rasing this issue for weeks...

> This is not a time for cosmetic fixes.

Agreed, but the issue is not cosmetic.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can?t measure it, you can?t improve it." - Lord Kelvin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171006/499ec24e/attachment.bin>

From gem at rellim.com  Fri Oct  6 19:46:52 2017
From: gem at rellim.com (Gary E. Miller)
Date: Fri, 6 Oct 2017 12:46:52 -0700
Subject: Resolution of the library-path mess
In-Reply-To: <c522a638-a0ce-f0b7-df9f-04dd42f03aee@grinta.net>
References: <20171006045133.05F0740605C@ip-64-139-1-69.sjc.megapath.net>
 <c522a638-a0ce-f0b7-df9f-04dd42f03aee@grinta.net>
Message-ID: <20171006124635.048ab4b7@spidey.rellim.com>

Yo Daniele!

On Thu, 5 Oct 2017 23:39:51 -0600
Daniele Nicolodi via devel <devel at ntpsec.org> wrote:

> The reason why Python does not
> include /usr/local folders in his search path by default is: what if
> you want to install an alternative Pythion _interpreter_ there? 

Uh, no.  Each interpreter makes it's own path under /usr/local

Easy to see:

$ ls /usr/local/lib/py* -ld
drwxr-xr-x 2 root root 4096 Nov  4  2016 /usr/local/lib/pylib
drwxr-xr-x 3 root root 4096 Jan 23  2017 /usr/local/lib/python2.7
drwxr-xr-x 3 root root 4096 Nov 21  2016 /usr/local/lib/python3.4



RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can?t measure it, you can?t improve it." - Lord Kelvin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171006/84479ff7/attachment.bin>

From fw at fwright.net  Fri Oct  6 21:00:43 2017
From: fw at fwright.net (Fred Wright)
Date: Fri, 6 Oct 2017 14:00:43 -0700 (PDT)
Subject: Library-path glitch, again
In-Reply-To: <209b3fd4-c946-09e1-1b5e-98a955f7ecad@grinta.net>
References: <20171006133206.9E10813A03F2@snark.thyrsus.com>
 <209b3fd4-c946-09e1-1b5e-98a955f7ecad@grinta.net>
Message-ID: <Pine.LNX.4.58L1X.1710061340110.8069@bolt.sonic.net>


On Fri, 6 Oct 2017, Eric S. Raymond via devel wrote:
> Fred Wright writes:
> >Sorry for the lateness, but I realized that the current code still has a
> >bug (as well as a couple of deficiencies of a more-or-less cosmetic
> >nature).  It's currently checking sys.path in the *running* Python, but it
> >needs to be checking it in the *target* Python.  Specifying a different
> >Python otherwise works, so this should respect the difference.
> >
> >One ofthe "cosmetic" issues is that the directory existence check is
> >superfluous.  The case table looks like this:
> >
> >Exists?         In sys.path?    Usable?
> >-------         ------------    -------
> >No              No              Unknown
> >No              Yes             Yes, but this can't happen (*)
> >Yes             No              No
> >Yes             Yes             Yes
> >
> >* site.py's sys.path setup filters nonexistent directories
> >
> >Since the install procedure would create the directory if needed, testing
> >for its preexistence is at best useless.
> >
> >The other "cosmetic" issue is that there's no need to "manually" construct
> >'localized' at all, since that's what get_python_lib() with the prefix
> >supplied already does.  The only reason to avoid the value that waf comes
> >up with is that it might not be in sys.path, but *conditionally* using
> >that value is fine, and simpler.
>
> But Gary replies:
>
> >Uh, I disagree on the 3rd case (Yes, No).  I install, then add my
> >dir to PYTHONPATH.
> >
> >I guess we could document that: for people that want to be HFS
> >conformant, add PYTHONPATH before doing the install.

> Fred, your timing is awful and you shouldn't have submitted multiple
> changes as a single MR.  Three days ago I might have taken a single,
> simple change that fixed the one bug.  Now I can't justify doing that
> - especially when Gary has raised a question about your analysis.

The table above simply shows the known behavior of site.py in a clearer
way.  Gary's "question" is based on wanting to use PYTHONPATH to allow
directories not in the default sys.path, even though the undesirability of
that was supposedly a settled question.  PYTHONPATH seems to be some sort
of zombie that refuses to stay dead. :-)  My proposed change doesn't make
any policy change at all; it simply implements the current policy in a
simpler and more correct manner.

That being said, I don't think it's a big problem to wait until after 1.0,
especially since it's not the only broken thing about installing for a
non-default Python.

I combined the fixes since they interact at the coding level.  I posted
the updated logic and reasoning behind it here; the change simply
implements that logic.  The hard part was figuring out that getting
sys.path through waf's get_python_variables() is hopeless, and going
around it.  At least *my* code doesn't apply eval() to externally obtained
data, which is more than I can say for waf. :-)

> The one patch I would like to see is to devel/TODO explaining the
> source/target bug.

As things are now, it should simply be documented that installing for a
non-default Python version doesn't work.  The bug being addressed here
only has the effect of making it less likely to come up with an
FHS-compliant location in that case, while the lack of appropriate
shebanging would make that case not work at all.

Fred Wright

From fw at fwright.net  Fri Oct  6 21:07:52 2017
From: fw at fwright.net (Fred Wright)
Date: Fri, 6 Oct 2017 14:07:52 -0700 (PDT)
Subject: Resolution of the library-path mess
In-Reply-To: <20171006045133.05F0740605C@ip-64-139-1-69.sjc.megapath.net>
References: <20171006045133.05F0740605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <Pine.LNX.4.58L1X.1710061401070.8069@bolt.sonic.net>


On Thu, 5 Oct 2017, Hal Murray wrote:

> >> Warnings are easily lost in the noise.  So either create the
> >> directory or treat it as an error and bail.
>
> > There are two issues with just "creating the directory":
> > 1) There's no guarantee that Python will actually use it.
> > 2) Creating the directory requires root, and it would be
> > undesirable to require that 'configure' run as root.
>
> The actual create can be deferred until the install.

Actually, it can't - see below.

> I don't understand the comment about "actually use it"?  Is that any
> different from the case where the directory did exist?

Yes.  It's about whether the directory in question is one of those that
Python considers for automatic inclusion in its sys.path.  Since Python
filters nonexistent directories out of that setup, looking at sys.path at
configure time can only work if the directory in question already exists
at configure time.  Install time is too late.

The overarching question is:  Why does Python have to make this so damn
difficult?

Fred Wright

From daniele at grinta.net  Fri Oct  6 22:57:13 2017
From: daniele at grinta.net (Daniele Nicolodi)
Date: Fri, 6 Oct 2017 16:57:13 -0600
Subject: Resolution of the library-path mess
In-Reply-To: <20171006152123.4ddfb086@spidey.rellim.com>
References: <20171006045133.05F0740605C@ip-64-139-1-69.sjc.megapath.net>
 <c522a638-a0ce-f0b7-df9f-04dd42f03aee@grinta.net>
 <20171006124635.048ab4b7@spidey.rellim.com>
 <d6ef3525-d496-c4be-909a-4ee42b40c5f5@grinta.net>
 <20171006152123.4ddfb086@spidey.rellim.com>
Message-ID: <93eca881-c653-c3fa-8b72-eae8da404e35@grinta.net>

On 10/6/17 4:21 PM, Gary E. Miller wrote:
> Yo Daniele!
> 
> On Fri, 6 Oct 2017 15:59:05 -0600
> Daniele Nicolodi <daniele at grinta.net> wrote:
> 
>> On 10/6/17 1:46 PM, Gary E. Miller via devel wrote:
>>> Yo Daniele!
>>>
>>> On Thu, 5 Oct 2017 23:39:51 -0600
>>> Daniele Nicolodi via devel <devel at ntpsec.org> wrote:
>>>   
>>>> The reason why Python does not
>>>> include /usr/local folders in his search path by default is: what
>>>> if you want to install an alternative Pythion _interpreter_
>>>> there?   
>>>
>>> Uh, no.  Each interpreter makes it's own path under /usr/local
>>>
>>> Easy to see:
>>>
>>> $ ls /usr/local/lib/py* -ld
>>> drwxr-xr-x 2 root root 4096 Nov  4  2016 /usr/local/lib/pylib
>>> drwxr-xr-x 3 root root 4096 Jan 23  2017 /usr/local/lib/python2.7
>>> drwxr-xr-x 3 root root 4096 Nov 21  2016 /usr/local/lib/python3.4  
>>
>> Is this serious?  Are you confusing different interpreters with
>> interpreter minor version?
> 
> Nothing to confuse. Clearly separate.
> 
>> What if I want to have 2.7.1 and 2.7.2, for example?
> 
> Gentoo can easily do that as well.  Pretty common on gentoo.

How? It is clearly not possible with the scheme you described before.

It is not clear to me how an user component installing files in /usr is
seen as a violation of policy, but a system component installing or
picking up files into /urs/local is fine.

Cheers,
Daniele

From gem at rellim.com  Fri Oct  6 23:43:48 2017
From: gem at rellim.com (Gary E. Miller)
Date: Fri, 6 Oct 2017 16:43:48 -0700
Subject: Resolution of the library-path mess
In-Reply-To: <93eca881-c653-c3fa-8b72-eae8da404e35@grinta.net>
References: <20171006045133.05F0740605C@ip-64-139-1-69.sjc.megapath.net>
 <c522a638-a0ce-f0b7-df9f-04dd42f03aee@grinta.net>
 <20171006124635.048ab4b7@spidey.rellim.com>
 <d6ef3525-d496-c4be-909a-4ee42b40c5f5@grinta.net>
 <20171006152123.4ddfb086@spidey.rellim.com>
 <93eca881-c653-c3fa-8b72-eae8da404e35@grinta.net>
Message-ID: <20171006164348.6cfbc0b4@spidey.rellim.com>

Yo Daniele!

On Fri, 6 Oct 2017 16:57:13 -0600
Daniele Nicolodi <daniele at grinta.net> wrote:

> >> What if I want to have 2.7.1 and 2.7.2, for example?  
> > 
> > Gentoo can easily do that as well.  Pretty common on gentoo.  
> 
> How? It is clearly not possible with the scheme you described before.

Not gonna worry about.  Not an issue here.

> It is not clear to me how an user component installing files in /usr
> is seen as a violation of policy, but a system component installing or
> picking up files into /urs/local is fine.

Uh, no.  The problem here is a user installed component installing
into, and protentially over-writing, system installed companaents.

I am not aware of any system installed component that looks in
/usr/local without specific user instructions to do so.

So I'm not seeing where you are going with this?

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can?t measure it, you can?t improve it." - Lord Kelvin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171006/a7c95d2e/attachment.bin>

From daniele at grinta.net  Sat Oct  7 00:37:53 2017
From: daniele at grinta.net (Daniele Nicolodi)
Date: Fri, 6 Oct 2017 18:37:53 -0600
Subject: Resolution of the library-path mess
In-Reply-To: <20171006164348.6cfbc0b4@spidey.rellim.com>
References: <20171006045133.05F0740605C@ip-64-139-1-69.sjc.megapath.net>
 <c522a638-a0ce-f0b7-df9f-04dd42f03aee@grinta.net>
 <20171006124635.048ab4b7@spidey.rellim.com>
 <d6ef3525-d496-c4be-909a-4ee42b40c5f5@grinta.net>
 <20171006152123.4ddfb086@spidey.rellim.com>
 <93eca881-c653-c3fa-8b72-eae8da404e35@grinta.net>
 <20171006164348.6cfbc0b4@spidey.rellim.com>
Message-ID: <398e8220-9756-2d3d-a33d-f7c74ecafd11@grinta.net>

On 06/10/17 17:43, Gary E. Miller via devel wrote:
> Yo Daniele!
> 
> On Fri, 6 Oct 2017 16:57:13 -0600
> Daniele Nicolodi <daniele at grinta.net> wrote:
> 
>>>> What if I want to have 2.7.1 and 2.7.2, for example?  
>>>
>>> Gentoo can easily do that as well.  Pretty common on gentoo.  
>>
>> How? It is clearly not possible with the scheme you described before.
> 
> Not gonna worry about.  Not an issue here.
> 
>> It is not clear to me how an user component installing files in /usr
>> is seen as a violation of policy, but a system component installing or
>> picking up files into /urs/local is fine.
> 
> Uh, no.  The problem here is a user installed component installing
> into, and protentially over-writing, system installed companaents.
> 
> I am not aware of any system installed component that looks in
> /usr/local without specific user instructions to do so.
> 
> So I'm not seeing where you are going with this?

This exchange started with me stating the reason why python does not
look for modules outside its installation path, or why a python
interpreter installed in /usr/bin and loading modules from /usr/lib/
does not have (unless so explicitly instructed) dirs in /usr/local/lib/
in its module paths.

The reason is that a python interpreter cannot assume that any other
path, outside the one in which it has been installed belongs to him.

You replied that this is not true, because the path are versioned with
the python major and minor version number. That's works as soon as you
want to install two python versions that share the same major and minor
version number.

Debian solves the problem of installing user components in the system
path creating a dist-libraries along the site-libraries path in the
python module folder.

The python way to avoid the clash is to use virtual environments, which
I suggested several times now, but it seems that it is a too complex of
a concept to be picked up.  Please go on debating your /usr vs
/usr/local and PYTHONPATH issues.

Cheers,
Daniele

From gem at rellim.com  Sat Oct  7 00:47:42 2017
From: gem at rellim.com (Gary E. Miller)
Date: Fri, 6 Oct 2017 17:47:42 -0700
Subject: Resolution of the library-path mess
In-Reply-To: <398e8220-9756-2d3d-a33d-f7c74ecafd11@grinta.net>
References: <20171006045133.05F0740605C@ip-64-139-1-69.sjc.megapath.net>
 <c522a638-a0ce-f0b7-df9f-04dd42f03aee@grinta.net>
 <20171006124635.048ab4b7@spidey.rellim.com>
 <d6ef3525-d496-c4be-909a-4ee42b40c5f5@grinta.net>
 <20171006152123.4ddfb086@spidey.rellim.com>
 <93eca881-c653-c3fa-8b72-eae8da404e35@grinta.net>
 <20171006164348.6cfbc0b4@spidey.rellim.com>
 <398e8220-9756-2d3d-a33d-f7c74ecafd11@grinta.net>
Message-ID: <20171006174742.4b7d0161@spidey.rellim.com>

Yo Daniele!

On Fri, 6 Oct 2017 18:37:53 -0600
Daniele Nicolodi via devel <devel at ntpsec.org> wrote:

> The reason is that a python interpreter cannot assume that any other
> path, outside the one in which it has been installed belongs to him.

Uh, no.  In fact, nothing outside of the base python 'belongs' to
the installed python.  

> You replied that this is not true,

Yup.

> because the path are versioned with
> the python major and minor version number.

Uh, no.

> Debian solves the problem of installing user components in the system
> path creating a dist-libraries along the site-libraries path in the
> python module folder.

Uh, yeah.  Pretty much universal.  Except you left out the sort of
details we are discussing.  There are MANY site-linraries in MANY
python module folders, all used by the SAME python interpreter.  This
discussion is about which of those is correct, and whick is not.

> The python way to avoid the clash is to use virtual environments,

Oh, funny.  When did we veer into comedy?  Let's not.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can?t measure it, you can?t improve it." - Lord Kelvin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171006/a63feecc/attachment.bin>

From daniele at grinta.net  Sat Oct  7 06:35:58 2017
From: daniele at grinta.net (Daniele Nicolodi)
Date: Sat, 7 Oct 2017 00:35:58 -0600
Subject: Resolution of the library-path mess
In-Reply-To: <20171006174742.4b7d0161@spidey.rellim.com>
References: <20171006045133.05F0740605C@ip-64-139-1-69.sjc.megapath.net>
 <c522a638-a0ce-f0b7-df9f-04dd42f03aee@grinta.net>
 <20171006124635.048ab4b7@spidey.rellim.com>
 <d6ef3525-d496-c4be-909a-4ee42b40c5f5@grinta.net>
 <20171006152123.4ddfb086@spidey.rellim.com>
 <93eca881-c653-c3fa-8b72-eae8da404e35@grinta.net>
 <20171006164348.6cfbc0b4@spidey.rellim.com>
 <398e8220-9756-2d3d-a33d-f7c74ecafd11@grinta.net>
 <20171006174742.4b7d0161@spidey.rellim.com>
Message-ID: <12b7188d-8f15-ec00-3ed4-676da621465a@grinta.net>

On 06/10/17 18:47, Gary E. Miller via devel wrote:

>> Debian solves the problem of installing user components in the system
>> path creating a dist-libraries along the site-libraries path in the
>> python module folder.
> 
> Uh, yeah.  Pretty much universal.  Except you left out the sort of
> details we are discussing.  There are MANY site-linraries in MANY
> python module folders, all used by the SAME python interpreter.  This
> discussion is about which of those is correct, and whick is not.

This is true only for patched up python installations. Stock python does
not do that, that is not documented, and is absolutely not standard.
Please point me to the official python documentation that states that a
python interpreter installed in $prefix/bin/ looks for modules outside
$prefix.  Here is the documentation I read:

https://docs.python.org/3/library/site.html#module-site

>> The python way to avoid the clash is to use virtual environments,
> 
> Oh, funny.  When did we veer into comedy?  Let's not.

I think you don't understand what a python virtual environment is.

https://docs.python.org/3/library/venv.html

But you and your mates may be too smart to think there is may be
something you can learn.

And this is going to be my last contribution to this mailing list.

Cheers,
Dan

From gem at rellim.com  Sat Oct  7 08:08:14 2017
From: gem at rellim.com (Gary E. Miller)
Date: Sat, 7 Oct 2017 01:08:14 -0700
Subject: Fw: Resolution of the library-path mess
Message-ID: <20171007010814.15136dbb@spidey.rellim.com>

Yo Daniele!

On Sat, 7 Oct 2017 00:32:33 -0600
Daniele Nicolodi <daniele at grinta.net> wrote:

> On 06/10/17 18:47, Gary E. Miller via devel wrote:
>   
> >> Debian solves the problem of installing user components in the
> >> system path creating a dist-libraries along the site-libraries
> >> path in the python module folder.    
> > 
> > Uh, yeah.  Pretty much universal.  Except you left out the sort of
> > details we are discussing.  There are MANY site-linraries in MANY
> > python module folders, all used by the SAME python interpreter.
> > This discussion is about which of those is correct, and whick is
> > not.    
> 
> This is true only for patched up python installations. Stock python
> does not do that, that is not documented, and is absolutely not
> standard. Please point me to the official python documentation that
> states that a python interpreter installed in $prefix/bin/ looks for
> modules outside $prefix.  Here is the documentation I read:  

Uh, I can't, and I never suggested it did.  That is why you need
PYTHONPATH.

> > Oh, funny.  When did we veer into comedy?  Let's not.    
> 
> I think you don't understand what a python virtual environment is.  

I understand it, I just assert that it is way more uncommon than you
think.

> But you and your mates may be too smart to think there is may be
> something you can learn.  

My mates?  Who would that be?

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can?t measure it, you can?t improve it." - Lord Kelvin



RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can?t measure it, you can?t improve it." - Lord Kelvin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171007/f8636350/attachment.bin>

From esr at thyrsus.com  Sat Oct  7 08:23:18 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sat, 7 Oct 2017 04:23:18 -0400
Subject: Resolution of the library-path mess
In-Reply-To: <12b7188d-8f15-ec00-3ed4-676da621465a@grinta.net>
References: <20171006045133.05F0740605C@ip-64-139-1-69.sjc.megapath.net>
 <c522a638-a0ce-f0b7-df9f-04dd42f03aee@grinta.net>
 <20171006124635.048ab4b7@spidey.rellim.com>
 <d6ef3525-d496-c4be-909a-4ee42b40c5f5@grinta.net>
 <20171006152123.4ddfb086@spidey.rellim.com>
 <93eca881-c653-c3fa-8b72-eae8da404e35@grinta.net>
 <20171006164348.6cfbc0b4@spidey.rellim.com>
 <398e8220-9756-2d3d-a33d-f7c74ecafd11@grinta.net>
 <20171006174742.4b7d0161@spidey.rellim.com>
 <12b7188d-8f15-ec00-3ed4-676da621465a@grinta.net>
Message-ID: <20171007082318.GA15251@thyrsus.com>

Daniele Nicolodi via devel <devel at ntpsec.org>:
> But you and your mates may be too smart to think there is may be
> something you can learn.
> 
> And this is going to be my last contribution to this mailing list.

Whoa.  I wasn't outright rejecting your idea for post-1.0; in my book,
"smart" implies being willing to learn from anyone.

Can you point me at a reference on virtual environments?  I can't read it
before we shiop, but I will after.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.



From hmurray at megapathdsl.net  Sat Oct  7 09:17:31 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sat, 07 Oct 2017 02:17:31 -0700
Subject: Resolution of the library-path mess
In-Reply-To: Message from "Eric S. Raymond via devel" <devel@ntpsec.org>
 of "Sat, 07 Oct 2017 04:23:18 EDT." <20171007082318.GA15251@thyrsus.com>
Message-ID: <20171007091731.1AE2C406060@ip-64-139-1-69.sjc.megapath.net>


If I have PYTHONPATH defined as /usr/local/lib/python2.7/site-packages
Then the python libs get installed in /usr/local/lib/...
If I unset PYTHONPATH, they get installed to /usr/lib/...

Having a PYTHONPATH setup could be leftover from when ntpsec needed it, or it 
could be setup because some other package needs it.

That's on a Fedora system.


-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Sat Oct  7 09:21:36 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sat, 07 Oct 2017 02:21:36 -0700
Subject: Where to python libraries get installed?
Message-ID: <20171007092136.CA843406060@ip-64-139-1-69.sjc.megapath.net>

waf configure prints out:
  PREFIX                        :  /usr/local 

Should it also printout where it will install the python libraries?


-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Sat Oct  7 10:01:22 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sat, 7 Oct 2017 06:01:22 -0400
Subject: Where to python libraries get installed?
In-Reply-To: <20171007092136.CA843406060@ip-64-139-1-69.sjc.megapath.net>
References: <20171007092136.CA843406060@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20171007100122.GA17010@thyrsus.com>

Hal Murray via devel <devel at ntpsec.org>:
> waf configure prints out:
>   PREFIX                        :  /usr/local 
> 
> Should it also printout where it will install the python libraries?

Good idea.  I'd take that patch.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.



From fw at fwright.net  Sat Oct  7 19:15:27 2017
From: fw at fwright.net (Fred Wright)
Date: Sat, 7 Oct 2017 12:15:27 -0700 (PDT)
Subject: Resolution of the library-path mess
In-Reply-To: <20171007091731.1AE2C406060@ip-64-139-1-69.sjc.megapath.net>
References: <20171007091731.1AE2C406060@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <Pine.LNX.4.58L1X.1710071152350.3534@bolt.sonic.net>


On Sat, 7 Oct 2017, Hal Murray via devel wrote:

> If I have PYTHONPATH defined as /usr/local/lib/python2.7/site-packages
> Then the python libs get installed in /usr/local/lib/...
> If I unset PYTHONPATH, they get installed to /usr/lib/...

Doh! (on Eric's behalf)  That would be an unfortunate property of the
current sys.path check.  Having PYTHONPATH set means that that directory
will get added to sys.path by Python, which in turn means that the
sys.path check in FixConfig.massage() will think that it's OK to use, even
though the purpose of the check is supposed to be to verify that the
location is usable *without* PYTHONPATH.

The fix would be to have waf remove PYTHONPATH from the internal
environment, so that its Python invocations wouldn't see it, but that
would only be effective if my fix to use the target Python for the
sys.path check were also applied.  Within the waf scripts, it's too late
to fix the sys.path setup for the Python already running, and any attempt
to do it retroactively would be messy and fragile.

> Having a PYTHONPATH setup could be leftover from when ntpsec needed it, or it
> could be setup because some other package needs it.

Most likely the former, since reasonable packages don't require
PYTHONPATH. :-)

Fortunately, the effect of this bug is limited to people who've already
set up PYTHONPATH based on earlier recommendations.

> That's on a Fedora system.

Makes sense.  Fedora is one of the systems where Python doesn't include a
"local" directory in the sys.path setup by default.  Ditto for CentOS.
Ubuntu/Debian seems better behaved (i.e., the /usr/local/lib/... location
will be used if it exists, regardless of PYTHONPATH).

Fred Wright

From fw at fwright.net  Sat Oct  7 19:16:37 2017
From: fw at fwright.net (Fred Wright)
Date: Sat, 7 Oct 2017 12:16:37 -0700 (PDT)
Subject: Where to python libraries get installed?
In-Reply-To: <20171007100122.GA17010@thyrsus.com>
References: <20171007092136.CA843406060@ip-64-139-1-69.sjc.megapath.net>
 <20171007100122.GA17010@thyrsus.com>
Message-ID: <Pine.LNX.4.58L1X.1710071215420.3534@bolt.sonic.net>


On Sat, 7 Oct 2017, Eric S. Raymond via devel wrote:

> Hal Murray via devel <devel at ntpsec.org>:
> > waf configure prints out:
> >   PREFIX                        :  /usr/local
> >
> > Should it also printout where it will install the python libraries?
>
> Good idea.  I'd take that patch.

It's already there.  Look at PYTHONDIR towards the end.

Fred Wright

From gem at rellim.com  Sat Oct  7 19:20:41 2017
From: gem at rellim.com (Gary E. Miller)
Date: Sat, 7 Oct 2017 12:20:41 -0700
Subject: Resolution of the library-path mess
In-Reply-To: <Pine.LNX.4.58L1X.1710071152350.3534@bolt.sonic.net>
References: <20171007091731.1AE2C406060@ip-64-139-1-69.sjc.megapath.net>
 <Pine.LNX.4.58L1X.1710071152350.3534@bolt.sonic.net>
Message-ID: <20171007122041.7f60399f@spidey.rellim.com>

Yo Fred!

On Sat, 7 Oct 2017 12:15:27 -0700 (PDT)
Fred Wright via devel <devel at ntpsec.org> wrote:

> > Having a PYTHONPATH setup could be leftover from when ntpsec needed
> > it, or it could be setup because some other package needs it.  
> 
> Most likely the former, since reasonable packages don't require
> PYTHONPATH. :-)

Duh.  That because 'packages' are system things, and thus, byt the FHS, 
go in the system directories.  We are very much talking about things
that are NOT packages, but instead user installed code.

> Makes sense.  Fedora is one of the systems where Python doesn't
> include a "local" directory in the sys.path setup by default.

Of course, no system following the FHS is supposed to have/usr/local
in the sys,path.  That is the FHS again.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can?t measure it, you can?t improve it." - Lord Kelvin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171007/2c1df52a/attachment.bin>

From fw at fwright.net  Sat Oct  7 19:36:41 2017
From: fw at fwright.net (Fred Wright)
Date: Sat, 7 Oct 2017 12:36:41 -0700 (PDT)
Subject: Resolution of the library-path mess
In-Reply-To: <20171007122041.7f60399f@spidey.rellim.com>
References: <20171007091731.1AE2C406060@ip-64-139-1-69.sjc.megapath.net>
 <Pine.LNX.4.58L1X.1710071152350.3534@bolt.sonic.net>
 <20171007122041.7f60399f@spidey.rellim.com>
Message-ID: <Pine.LNX.4.58L1X.1710071234080.3534@bolt.sonic.net>


On Sat, 7 Oct 2017, Gary E. Miller via devel wrote:
> On Sat, 7 Oct 2017 12:15:27 -0700 (PDT)
> Fred Wright via devel <devel at ntpsec.org> wrote:
>
>
> > Makes sense.  Fedora is one of the systems where Python doesn't
> > include a "local" directory in the sys.path setup by default.
>
> Of course, no system following the FHS is supposed to have/usr/local
> in the sys,path.  That is the FHS again.

Sigh.  The "sys" in "sys.path" is the name of the Python module containing
a variety of variables and functions, including the search path for
imports.  It has nothing to do with "system".

Fred Wright

From gem at rellim.com  Sat Oct  7 19:43:10 2017
From: gem at rellim.com (Gary E. Miller)
Date: Sat, 7 Oct 2017 12:43:10 -0700
Subject: Resolution of the library-path mess
In-Reply-To: <Pine.LNX.4.58L1X.1710071234080.3534@bolt.sonic.net>
References: <20171007091731.1AE2C406060@ip-64-139-1-69.sjc.megapath.net>
 <Pine.LNX.4.58L1X.1710071152350.3534@bolt.sonic.net>
 <20171007122041.7f60399f@spidey.rellim.com>
 <Pine.LNX.4.58L1X.1710071234080.3534@bolt.sonic.net>
Message-ID: <20171007124310.46b10910@spidey.rellim.com>

Yo Fred!

On Sat, 7 Oct 2017 12:36:41 -0700 (PDT)
Fred Wright via devel <devel at ntpsec.org> wrote:

> On Sat, 7 Oct 2017, Gary E. Miller via devel wrote:
> > On Sat, 7 Oct 2017 12:15:27 -0700 (PDT)
> > Fred Wright via devel <devel at ntpsec.org> wrote:
> >
> >  
> > > Makes sense.  Fedora is one of the systems where Python doesn't
> > > include a "local" directory in the sys.path setup by default.  
> >
> > Of course, no system following the FHS is supposed to have/usr/local
> > in the sys,path.  That is the FHS again.  
> 
> Sigh.  The "sys" in "sys.path" is the name of the Python module
> containing a variety of variables and functions, including the search
> path for imports.  It has nothing to do with "system".

100% agreed.  Never said otherwise.  Please re-read my comments after
pondering this:

By 'system' I am using the definition that is in the FHS.  Consider it a
synonym for what some would call a 'distribution' (Fedroa, Ubuntu, etc.).

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can?t measure it, you can?t improve it." - Lord Kelvin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171007/99b83b47/attachment.bin>

From hmurray at megapathdsl.net  Sat Oct  7 23:30:12 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sat, 07 Oct 2017 16:30:12 -0700
Subject: Resolution of the library-path mess
In-Reply-To: Message from Fred Wright via devel <devel@ntpsec.org> of "Sat,
 07 Oct 2017 12:15:27 PDT."
 <Pine.LNX.4.58L1X.1710071152350.3534@bolt.sonic.net>
Message-ID: <20171007233012.419DC40605C@ip-64-139-1-69.sjc.megapath.net>


>> Having a PYTHONPATH setup could be leftover from when
>> ntpsec needed it, or it could be setup because some other
>> package needs it.

> Most likely the former, since reasonable packages don't 
> require PYTHONPATH.   :-) 

It seems reasonably likely to me that anybody geeky enough to be installing a 
local version of one package is reasonably likely to be using local versions 
of other packages.  Or at least likely enough that we should make sure that 
any fix does work when PYTHONPATH is set.

My notes say I wanted it for GNU Radio.  I don't know if that is still 
correct.


-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Sat Oct  7 23:35:35 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sat, 07 Oct 2017 16:35:35 -0700
Subject: Where to python libraries get installed?
In-Reply-To: Message from Fred Wright via devel <devel@ntpsec.org> of "Sat,
 07 Oct 2017 12:16:37 PDT."
 <Pine.LNX.4.58L1X.1710071215420.3534@bolt.sonic.net>
Message-ID: <20171007233535.3676B40605C@ip-64-139-1-69.sjc.megapath.net>

>>> Should it also printout where it will install the python libraries?
>> Good idea.  I'd take that patch.

> It's already there.  Look at PYTHONDIR towards the end.

Thanks.

Is that used for anything other that storing python libraries?

What is PYTHONARCHDIR used for?


-- 
These are my opinions.  I hate spam.




From fw at fwright.net  Sun Oct  8 00:09:42 2017
From: fw at fwright.net (Fred Wright)
Date: Sat, 7 Oct 2017 17:09:42 -0700 (PDT)
Subject: Where to python libraries get installed?
In-Reply-To: <20171007233535.3676B40605C@ip-64-139-1-69.sjc.megapath.net>
References: <20171007233535.3676B40605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <Pine.LNX.4.58L1X.1710071700570.26135@bolt.sonic.net>


On Sat, 7 Oct 2017, Hal Murray wrote:

> >>> Should it also printout where it will install the python libraries?
> >> Good idea.  I'd take that patch.
>
> > It's already there.  Look at PYTHONDIR towards the end.
>
> Thanks.
>
> Is that used for anything other that storing python libraries?

No, if "storing" includes the code that may clean up old library installs.

> What is PYTHONARCHDIR used for?

The difference is in the value of the 'plat_specific' argument to
get_python_lib().  The intended distinction is between
architecture-independent pure Python code and architecture-dependent
compiled extensions, in case the Python implementation wants to keep them
in different places.  So in theory, PYTHONARCHDIR should be the install
location for the C extension, but doing that doesn't seem to work on
platforms where the two are actually different, so currently PYTHONARCHDIR
is unused.  However, anyone overriding the default(s) should set both
appropriately, in case that gets sorted out in the future.

Fred Wright

From hmurray at megapathdsl.net  Sun Oct  8 02:10:14 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sat, 07 Oct 2017 19:10:14 -0700
Subject: Where to python libraries get installed?
In-Reply-To: Message from Fred Wright via devel <devel@ntpsec.org> of "Sat,
 07 Oct 2017 17:09:42 PDT."
 <Pine.LNX.4.58L1X.1710071700570.26135@bolt.sonic.net>
Message-ID: <20171008021015.0587240605C@ip-64-139-1-69.sjc.megapath.net>

> No, if "storing" includes the code that may clean up old library installs.

Ahh...  Thanks.  That brings up a question I've been wondering about for ages.

What sort of cleanup does waf install do?  Is that documented anyplace?


-- 
These are my opinions.  I hate spam.




From fw at fwright.net  Sun Oct  8 03:14:48 2017
From: fw at fwright.net (Fred Wright)
Date: Sat, 7 Oct 2017 20:14:48 -0700 (PDT)
Subject: Where to python libraries get installed?
In-Reply-To: <20171008021015.0587240605C@ip-64-139-1-69.sjc.megapath.net>
References: <20171008021015.0587240605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <Pine.LNX.4.58L1X.1710072006240.28535@bolt.sonic.net>


On Sat, 7 Oct 2017, Hal Murray wrote:

> > No, if "storing" includes the code that may clean up old library installs.
>
> Ahh...  Thanks.  That brings up a question I've been wondering about for ages.
>
> What sort of cleanup does waf install do?  Is that documented anyplace?

I don't think install cleans up anything at all.  Uninstall removes files
that would be installed by the *current* code, which of course may leave
stuff around if the location or set of files changes.  There's no general
solution to this, but I added some temporary code to clean up the old
(prior to FixConfig) Python library install location when it differs from
the current location.  That's in a separate commit so that it can be
reverted later.  And it was never designed to handle more than two
possibilities. :-)

In some cases, a packaging system may keep track of what was installed and
properly remove it at uninstall time, but waf itself has no database to
track what was installed by previous versions of the scripts.

Fred Wright

From hmurray at megapathdsl.net  Sun Oct  8 05:13:02 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sat, 07 Oct 2017 22:13:02 -0700
Subject: Where to python libraries get installed?
In-Reply-To: Message from Fred Wright via devel <devel@ntpsec.org> of "Sat,
 07 Oct 2017 20:14:48 PDT."
 <Pine.LNX.4.58L1X.1710072006240.28535@bolt.sonic.net>
Message-ID: <20171008051302.460DE40605C@ip-64-139-1-69.sjc.megapath.net>


>> What sort of cleanup does waf install do?  Is that documented anyplace?
> I don't think install cleans up anything at all.  Uninstall removes files
> that would be installed by the *current* code, which of course may leave
> stuff around if the location or set of files changes. 

Thanks.

There is one trick for uninstalling old stuff - back up to before the commit 
that changed things.

Doing it by hand is probably faster/simpler with today's collection of files 
that get installed.


-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Sun Oct  8 08:06:13 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 08 Oct 2017 01:06:13 -0700
Subject: Fedora: FHS vs PYTHONPATH
Message-ID: <20171008080613.38F5640605C@ip-64-139-1-69.sjc.megapath.net>

My reading is that Fedora doesn't support /usr/local/,
at least not the way we want/expect it to.

So we have 2 options:
  Install in /usr/ (no local) and break FHS
  Install in /usr/local/ and require PYTHONPATH to use it.


-- 
These are my opinions.  I hate spam.




From gem at rellim.com  Sun Oct  8 19:53:37 2017
From: gem at rellim.com (Gary E. Miller)
Date: Sun, 8 Oct 2017 12:53:37 -0700
Subject: Fedora: FHS vs PYTHONPATH
In-Reply-To: <20171008080613.38F5640605C@ip-64-139-1-69.sjc.megapath.net>
References: <20171008080613.38F5640605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20171008125337.5cab77f3@spidey.rellim.com>

Yo Hal!

On Sun, 08 Oct 2017 01:06:13 -0700
Hal Murray via devel <devel at ntpsec.org> wrote:

> My reading is that Fedora doesn't support /usr/local/,
> at least not the way we want/expect it to.

Lost me.  What is expected/wanted?  Don't you expect Fedora to respect
the 40+ years of UNIX tradition and standards to led to the current FHS?

> So we have 2 options:
>   Install in /usr/ (no local) and break FHS
>   Install in /usr/local/ and require PYTHONPATH to use it.

Yup.  But when the distros that have been waiting for NTPsec 1.0 start
shipping NTPsec, it will be more than just ther FHS that gets broken.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can?t measure it, you can?t improve it." - Lord Kelvin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171008/d847c5c4/attachment.bin>

From ianbruene at gmail.com  Mon Oct  9 02:16:55 2017
From: ianbruene at gmail.com (Ian Bruene)
Date: Sun, 8 Oct 2017 21:16:55 -0500
Subject: Attn: Anyone familiar with 32 vs 64 bit time_t issues, or who can
 make policy decisions.
Message-ID: <4b07378f-525e-8b52-4b65-47e9c4c7ff0b@gmail.com>


Please take a look at bug #404 at the earliest opportunity. I hate to 
raise such an alarm over a test bug but we are running out of hours 
before release.

Relevant problem is my final comment on the bug thread.

-- 
In the end; what separates a Man, from a Slave? Money? Power?
No. A Man Chooses, a Slave Obeys. -- Andrew Ryan


From hmurray at megapathdsl.net  Mon Oct  9 04:32:26 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 08 Oct 2017 21:32:26 -0700
Subject: Attn: Anyone familiar with 32 vs 64 bit time_t issues, or who 
 can make policy decisions.
In-Reply-To: Message from Ian Bruene via devel <devel@ntpsec.org> of "Sun,
 08 Oct 2017 21:16:55 CDT." <4b07378f-525e-8b52-4b65-47e9c4c7ff0b@gmail.com>
Message-ID: <20171009043226.B4ED340605C@ip-64-139-1-69.sjc.megapath.net>

> python -m discover -s build/main/tests/pylib
/usr/bin/python: No module named discover

What is discover and/or why aren't those tests run as part of the normal 
tests run by waf check?

----------

> @esr I believe that the problem here is caused by differing
> time_t sizes as handled by lfp_stamp_to_tspec.

32 bit time_t should work for time up to 31 bits.  Are you testing bigger 
times?  Maybe NTP Rollover?

Can you bypass the screwup tests if time_t is only 32 bits?

------
There is no reference to lfp_stamp_to_tspec from within tests/

I haven't found the code that isn't working.  I assume it's burried under a 
several layers.  Can you make a simpler test?


-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Mon Oct  9 06:01:49 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 08 Oct 2017 23:01:49 -0700
Subject: Fedora: FHS vs PYTHONPATH
Message-ID: <20171009060149.E571040605C@ip-64-139-1-69.sjc.megapath.net>


>> My reading is that Fedora doesn't support /usr/local/,
>> at least not the way we want/expect it to.
> Lost me.  What is expected/wanted?  Don't you expect Fedora to respect the
> 40+ years of UNIX tradition and standards to led to the current FHS? 

Their default python setup doesn't look for libraries in /usr/local/

>> So we have 2 options:
>>   Install in /usr/ (no local) and break FHS
>>   Install in /usr/local/ and require PYTHONPATH to use it.

> Yup.  But when the distros that have been waiting for NTPsec 1.0 start
> shipping NTPsec, it will be more than just ther FHS that gets broken.

Is that a suggestion to go with PYTHONPATH?

Should we add a trap to waf configure and/or install to check that 
/usr/local/whatever is on the python search path?


-- 
These are my opinions.  I hate spam.




From gem at rellim.com  Mon Oct  9 19:39:43 2017
From: gem at rellim.com (Gary E. Miller)
Date: Mon, 9 Oct 2017 12:39:43 -0700
Subject: Fedora: FHS vs PYTHONPATH
In-Reply-To: <20171009060149.E571040605C@ip-64-139-1-69.sjc.megapath.net>
References: <20171009060149.E571040605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20171009123943.49d68edf@spidey.rellim.com>

Yo Hal!

On Sun, 08 Oct 2017 23:01:49 -0700
Hal Murray <hmurray at megapathdsl.net> wrote:

> >> My reading is that Fedora doesn't support /usr/local/,
> >> at least not the way we want/expect it to.  
> > Lost me.  What is expected/wanted?  Don't you expect Fedora to
> > respect the 40+ years of UNIX tradition and standards to led to the
> > current FHS?   
> 
> Their default python setup doesn't look for libraries in /usr/local/

Correct.  Always been that way, always should be that way, until the
user overrides it.  The user knows he has to add /usr/local/bin to
his PATH, not a big stretch to tell him to also add /usr/local/lib/pythonXX
to his PYTHONPATH.  An installation that installs partly in /usr/local/
and partly in /usr/ is schizophrenic.

> >> So we have 2 options:
> >>   Install in /usr/ (no local) and break FHS
> >>   Install in /usr/local/ and require PYTHONPATH to use it.  
> 
> > Yup.  But when the distros that have been waiting for NTPsec 1.0
> > start shipping NTPsec, it will be more than just ther FHS that gets
> > broken.  
> 
> Is that a suggestion to go with PYTHONPATH?

Suggestion?  No, I have a lot of better suggestions, but my track 
record getting Python people to take my suggestions is pretty bad.

I simply see PYTHONPATH as the only current way to respect a widely
used and frequently updated standard that embodies many decades
of best practice in UNIX administration.
	
> Should we add a trap to waf configure and/or install to check that 
> /usr/local/whatever is on the python search path?

There are a lot of things we could, and should do, if NTPssec goes
back to obeying UNIX best practice.  But I'm wasting my breath, we'll
have to wait until this inevitably causes problems before we fix it.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can?t measure it, you can?t improve it." - Lord Kelvin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171009/6bb247da/attachment.bin>

From mark.atwood at ntpsec.org  Tue Oct 10 15:02:10 2017
From: mark.atwood at ntpsec.org (Mark Atwood)
Date: Tue, 10 Oct 2017 08:02:10 -0700
Subject: we released 1.0.0, thank you, and take a break
Message-ID: <1507647730.114608.1134007944.463026A4@webmail.messagingengine.com>

Last night, we tagged and released 1.0.0 of NTPsec

blog post is https://blog.ntpsec.org/2017/10/10/version-1.0.0.html
updated historical narrative is https://www.ntpsec.org/history.html
press release is https://www.ntpsec.org/pressrelease-20171010.html
gitlab commit
https://gitlab.com/NTPsec/ntpsec/commit/3f72203164e9eebae0865f1b8dcd41b250bc232f

I would like to thank everyone who has worked so hard to get us here,
especially Hal Murray, Gary Miller, Daniel Franke, Eric Raymond, Cathy
Raymond, Ian Bruene , John Bell, Matt Selsky, Fred Wright, Achim Gratz,
and Susan Sons.  I'm sure I've missed people in that list, and I
apologize now for doing so.

I would like to thank our past and present institutional sponsors and
supporters, including Indiana University, the United States National
Science Foundation, OARcorp, Hewlett Packard Cloud Computing Service, 
the Linux Foundation Core Infrastructure Initiative, Software in the
Public Interest, Penguicon, Two Sigma Investments, Google Compute
Service, Oregon State University Open Source Lab, the Internet Civil
Engineering Institute, the Mozilla Foundation, and the Thiel Foundation.

Everyone, take a break from NTPsec.  I don't want to see any commits or
any planning emails for the rest of the week.

Thank you all again.

..m

--
Mark Atwood <mark.atwood at ntpsec.org>
Project Manager of the NTPsec Project
+1-206-604-2198 PSTN/GSM/SMS/Signal

From esr at thyrsus.com  Tue Oct 10 15:44:45 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Tue, 10 Oct 2017 11:44:45 -0400
Subject: we released 1.0.0, thank you, and take a break
In-Reply-To: <1507647730.114608.1134007944.463026A4@webmail.messagingengine.com>
References: <1507647730.114608.1134007944.463026A4@webmail.messagingengine.com>
Message-ID: <20171010154445.GA14315@thyrsus.com>

Mark Atwood via devel <devel at ntpsec.org>:
> Everyone, take a break from NTPsec.  I don't want to see any commits or
> any planning emails for the rest of the week.

Good plan.

I shall spend the week catching up on neglected projects.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.



From paul at anastrophe.com  Wed Oct 11 23:29:39 2017
From: paul at anastrophe.com (Paul Theodoropoulos)
Date: Wed, 11 Oct 2017 16:29:39 -0700
Subject: upgrading?
Message-ID: <d01c6edd-25fa-b787-b8d4-5ff7b2ec62f3@anastrophe.com>

Hey there everyone,
I've been running ntpsec-0.9.7 for a while now on Raspian 9.1.? I 
downloaded ntpsec-1.0.0 this afternoon, followed the build and install 
instructions, no errors, but when I start up timeservice - it simply 
does not see the GPS at all. Are there different requirements when 
upgrading?? I've rolled back to 0.9.7 for now (did a ./waf uninstall on 
1.0.0 then ./waf install in the 0.9.7 build dir), running fine.

Here's what I get upon running 0.9.7:
 ???? remote?????????????????????????????????? refid????? st t when poll 
reach?? delay?? offset?? jitter
=======================================================================================================
*SHM(1)????????????????????????????????? .PPS.??????????? 0 u 77? 128??? 
3?? 0.0000?? 0.8784?? 0.6017
xSHM(0)????????????????????????????????? .GPS.??????????? 0 u 108? 
128??? 3?? 0.0000 -56.5068? 29.3563
+clock.fmt.he.net??????????????????????? .CDMA.?????????? 1 u 32? 128??? 
7? 38.5736?? 6.2729?? 7.4006
+clepsydra.labs.hp.com?????????????????? .GPS.??????????? 1 u 98? 128??? 
3? 23.4886? -0.2906?? 1.5975
-stratum-1.sjc02.svwh.net??????????????? .CDMA.?????????? 1 u 32? 128??? 
7? 38.3806?? 6.7499?? 6.9528

and here's what I get upon running 1.0.0:
 ???? remote?????????????????????????????????? refid????? st t when poll 
reach?? delay?? offset?? jitter
=======================================================================================================
*clock.fmt.he.net??????????????????????? .CDMA.?????????? 1 u?? 14?? 
64??? 1? 23.7298?? 0.1724?? 4.0090
+clepsydra.hpl.hp.com??????????????????? .GPS.??????????? 1 u?? 14 
1024??? 1? 24.0723? -0.3156?? 4.4973
 ?stratum-1.sjc02.svwh.net??????????????? .CDMA.?????????? 1 u?? 13?? 
64??? 1? 23.6860?? 0.3439?? 0.5088

-- 
Paul Theodoropoulos
www.anastrophe.com


From hmurray at megapathdsl.net  Wed Oct 11 23:34:40 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Wed, 11 Oct 2017 16:34:40 -0700
Subject: upgrading?
In-Reply-To: Message from Paul Theodoropoulos via devel <devel@ntpsec.org>
 of "Wed, 11 Oct 2017 16:29:39 PDT."
 <d01c6edd-25fa-b787-b8d4-5ff7b2ec62f3@anastrophe.com>
Message-ID: <20171011233440.A5E8240605C@ip-64-139-1-69.sjc.megapath.net>


Did you feed the same parameters to waf configure?  Did you forget to include 
the SHM driver?

Check your log files to see what it says about SHM.


-- 
These are my opinions.  I hate spam.




From paul at anastrophe.com  Wed Oct 11 23:51:38 2017
From: paul at anastrophe.com (Paul Theodoropoulos)
Date: Wed, 11 Oct 2017 16:51:38 -0700
Subject: upgrading?
In-Reply-To: <20171011233440.A5E8240605C@ip-64-139-1-69.sjc.megapath.net>
References: <20171011233440.A5E8240605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <bd49364c-021c-8541-e979-41753e725c5b@anastrophe.com>

Shit. Yes, that's a big old 'duh' here on my part. Sorry for the list noise.

On 10/11/2017 16:34 PM, Hal Murray wrote:
> Did you feed the same parameters to waf configure?  Did you forget to include
> the SHM driver?
>
> Check your log files to see what it says about SHM.
>
>

-- 
Paul Theodoropoulos
www.anastrophe.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171011/f5b932b5/attachment.html>

From hmurray at megapathdsl.net  Fri Oct 13 02:57:34 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Thu, 12 Oct 2017 19:57:34 -0700
Subject: we released 1.0.0, thank you, and take a break
In-Reply-To: Message from Mark Atwood via devel <devel@ntpsec.org> of "Tue,
 10 Oct 2017 08:02:10 PDT."
 <1507647730.114608.1134007944.463026A4@webmail.messagingengine.com>
Message-ID: <20171013025734.723C540605C@ip-64-139-1-69.sjc.megapath.net>

> Last night, we tagged and released 1.0.0 of NTPsec

Congrats.

Should the version number on git be bumped to 1.0.1?


-- 
These are my opinions.  I hate spam.




From esr at thyrsus.com  Tue Oct 17 10:37:22 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Tue, 17 Oct 2017 06:37:22 -0400 (EDT)
Subject: Let's get back to work
Message-ID: <20171017103722.B318A13A6500@snark.thyrsus.com>

Our week of downtime has elapsed.  Matt got in the first commit this
morning, removing some code (yay for removing code).

Obviously the first thing we oufght to clean up is the FHS/library
path issue. Also Mark and I need to test the new release script

After that...I'd like to plan for a 1.1 in a roughly 90-day
timeframe. That's relatively short in order to get the build fix out.
Also I want people to know that we're not going to go off the radar
for another 2.5 years.

Let's try to figure out what we're going to focus on. If you have
a feature you'd like us to do next, reply with a proposal.  We'll
put them in a priority opder and start working.

Some possibilities I see:

>From Daniel, the data-hiding patch

>From Ian, the SNMP daemon.

>From Gary, rethinking initial jitter computation

>From me: Local monitoring mode to measure and graph the offet
between a noselected GPS and NTP time.

Other possibilities welcomed.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

.. a government and its agents are under no general duty to 
provide public services, such as police protection, to any 
particular individual citizen...
        -- Warren v. District of Columbia, 444 A.2d 1 (D.C. App.181)

From hmurray at megapathdsl.net  Tue Oct 17 10:53:40 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Tue, 17 Oct 2017 03:53:40 -0700
Subject: Let's get back to work
In-Reply-To: Message from "Eric S. Raymond via devel" <devel@ntpsec.org>
 of "Tue,
 17 Oct 2017 06:37:22 EDT." <20171017103722.B318A13A6500@snark.thyrsus.com>
Message-ID: <20171017105340.5287540605C@ip-64-139-1-69.sjc.megapath.net>


I was expecting VERSION to get bumped to 1.0.1 after the tarball and such.


> From me: Local monitoring mode to measure and
> graph the offet between a noselected GPS and NTP time.

Just add "noselect" and graph peerstats.

------

A possibly interesting feature in that area.

I have code that writes refstats for each refclock.  At poll time, it drops 
outliers from the fifo.  refstats includes the indexes into the fifo and the 
corresponding offset values.  You can get some idea of the scatter of your 
refclock samples.

We could add a flag to various drivers to write timestamps and data for each 
received line.  That's a lot of data for a typical NMEA device but that's 
what you need if you want to look at what's really going on.



-- 
These are my opinions.  I hate spam.




From ianbruene at gmail.com  Tue Oct 17 16:34:40 2017
From: ianbruene at gmail.com (Ian Bruene)
Date: Tue, 17 Oct 2017 11:34:40 -0500
Subject: Let's get back to work
In-Reply-To: <20171017103722.B318A13A6500@snark.thyrsus.com>
References: <20171017103722.B318A13A6500@snark.thyrsus.com>
Message-ID: <44a27b5b-a739-256e-ac22-cbfc4c147512@gmail.com>



On 10/17/2017 05:37 AM, Eric S. Raymond via devel wrote:
> Other possibilities welcomed.

Someone who understands waf: make the build run the python tests.

Note to whoever does it: you need to run them directly, python discover 
requires pip.

-- 
In the end; what separates a Man, from a Slave? Money? Power?
No. A Man Chooses, a Slave Obeys. -- Andrew Ryan


From esr at thyrsus.com  Thu Oct 19 13:48:21 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Thu, 19 Oct 2017 09:48:21 -0400 (EDT)
Subject: Library-path fix
Message-ID: <20171019134821.369E013A6891@snark.thyrsus.com>

Fred, can we get an updaterd MR and rationale for your library-path
fix proposal?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

That the said Constitution shall never be construed to authorize
Congress to infringe the just liberty of the press or the rights of
conscience; or to prevent the people of the United states who are
peaceable citizens from keeping their own arms...
        -- Samuel Adams, in "Phila. Independent Gazetteer", August 20, 1789

From fw at fwright.net  Fri Oct 20 19:24:19 2017
From: fw at fwright.net (Fred Wright)
Date: Fri, 20 Oct 2017 12:24:19 -0700 (PDT)
Subject: Library-path fix
In-Reply-To: <20171019134821.369E013A6891@snark.thyrsus.com>
References: <20171019134821.369E013A6891@snark.thyrsus.com>
Message-ID: <Pine.LNX.4.58L1X.1710201220580.23674@bolt.sonic.net>


On Thu, 19 Oct 2017, Eric S. Raymond via devel wrote:

> Fred, can we get an updaterd MR and rationale for your library-path
> fix proposal?

Yes, but I'm about to go away for the weekend.  I'll do that next week.

BTW, I do have an outstanding MR (561) for a minor doc fix that should be
non-controversial.

Fred Wright

From hmurray at megapathdsl.net  Sat Oct 21 06:37:23 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Fri, 20 Oct 2017 23:37:23 -0700
Subject: [time-nuts] inexpensive, black box,
 GPS or NTP based TTL time capture?
Message-ID: <20171021063723.30DBA40605C@ip-64-139-1-69.sjc.megapath.net>

>>> For under a $100 you could get a Raspberry Pi, a GPS HAT, and
>>> connect your input to a GPIO pin.  Configure ntpd to log the real
>>> PPS and the input as another 'PPS'.

>> Is there an option to log all individual PPS events?

>    # ppswatch /dev/pps0

That's a way to log stuff, but I don't think it comes under "Configure ntpd".

I remember some option to log lots of stuff, but I don't remember the 
details.  It could have been in a driver.  I couldn't find it in the man page 
for the PPS driver.

[amazon]
> Looks like $96 to me.  You can save some if you buy in bulk,

You can save $7 if you get the starter package that has only Pi, SD card, 
power, and case.  (Many starter packages include stuff you probably don't 
need and that raises the price.  But maybe one has a HDMI adapter.  I didn't 
look.)

Beware of using normal USB cables and/or normal USB power supplies.  The Pi 
is not happy with low voltage.  The drop in a USB cable can be significant.  
The setups intended for use with Pis normally have 5.1 or 5.25 volts and 
heavier gage wire in the cable.


[display, kbd, mouse...]
> Yeah, just for setup.  Shall we include the price of the desk it sits and
> the building it is in? 

I'm willing to assume somebody has a table and a roof.

The display and such are not a problem if you have a PC you can borrow them 
from.  (You probably need a HDMI adapter.)  But that doesn't work if all you 
have is a laptop or smart phone.

I think most of my friends have PCs but I wouldn't be surprised if some of 
them had a laptop and no PC.

If your PC is old enough, the keyboard and mouse may be PS2 rather than USB.


-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Sat Oct 21 06:45:39 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Fri, 20 Oct 2017 23:45:39 -0700
Subject: [time-nuts] inexpensive, black box, GPS or NTP based TTL time 
 capture?
In-Reply-To: Message from Hal Murray <hmurray@megapathdsl.net> of "Fri,
 20 Oct 2017 23:37:23 PDT."
 <20171021063723.30DBA40605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <20171021064540.08D6940605C@ip-64-139-1-69.sjc.megapath.net>

Arg, blush.  Wrong list.  Sorry for the clutter.

-- 
These are my opinions.  I hate spam.




From gem at rellim.com  Fri Oct 27 03:52:30 2017
From: gem at rellim.com (Gary E. Miller)
Date: Thu, 26 Oct 2017 20:52:30 -0700
Subject: =?UTF-8?B?4pyYUFRQ?= with NTPsec
Message-ID: <20171026205214.5d7a2dec@spidey.rellim.com>

Yo All!

I previsously had some NTPsec servers set up to also use PTP by
running the ptp4l daemon.  My setup is described in the gpsd time
service howto.

http://www.catb.org/gpsd/gpsd-time-service-howto.html#_providing_local_ntp_service_using_ptp

The configuration is supposed to put PTP time stamps in SHM NTP2, so ntpd
can see them.  My pshmmon no longer shows ptp4l creating or using NTP2.
My ntpmon does show my gpsd SHMs nicely.

Anyone else tried this lately?

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can?t measure it, you can?t improve it." - Lord Kelvin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171026/ff306801/attachment.bin>

From esr at thyrsus.com  Fri Oct 27 04:20:21 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Fri, 27 Oct 2017 00:20:21 -0400
Subject: =?utf-8?B?4pyYUFRQ?= with NTPsec
In-Reply-To: <20171026205214.5d7a2dec@spidey.rellim.com>
References: <20171026205214.5d7a2dec@spidey.rellim.com>
Message-ID: <20171027042021.GA7653@thyrsus.com>

Gary E. Miller via devel <devel at ntpsec.org>:
> Yo All!
> 
> I previsously had some NTPsec servers set up to also use PTP by
> running the ptp4l daemon.  My setup is described in the gpsd time
> service howto.
> 
> http://www.catb.org/gpsd/gpsd-time-service-howto.html#_providing_local_ntp_service_using_ptp
> 
> The configuration is supposed to put PTP time stamps in SHM NTP2, so ntpd
> can see them.  My pshmmon no longer shows ptp4l creating or using NTP2.
> My ntpmon does show my gpsd SHMs nicely.
> 
> Anyone else tried this lately?

Sorry, I have no PTP experience.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 811 bytes
Desc: not available
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171027/f9b646db/attachment.bin>

From fw at fwright.net  Sat Oct 28 23:07:22 2017
From: fw at fwright.net (Fred Wright)
Date: Sat, 28 Oct 2017 16:07:22 -0700 (PDT)
Subject: Library-path fix
In-Reply-To: <20171019134821.369E013A6891@snark.thyrsus.com>
References: <20171019134821.369E013A6891@snark.thyrsus.com>
Message-ID: <Pine.LNX.4.58L1X.1710281515340.1977@bolt.sonic.net>


On Thu, 19 Oct 2017, Eric S. Raymond via devel wrote:

> Fred, can we get an updaterd MR and rationale for your library-path
> fix proposal?

OK, here goes:

First of all, here's the case table for the "usability check" again:

Exists?         In sys.path?    Usable?
-------         ------------    -------
No              No              Unknown
No              Yes             Yes, but this can't happen (*)
Yes             No              No
Yes             Yes             Yes

* site.py's sys.path setup filters nonexistent directories

Gary took issue with the "No" in the third entry, but this "No" is based
on avoiding dependence on PYTHONPATH, which is consistent with current
policy and current code.  The current code also treats "Unknown" as "No".

The point of the table is to show that the check for directory existence
in the current code is superfluous.  This is not to say that the existence
*itself* is irrelevant (since it does affect the sys.path check); it just
means that *checking* the existence is superfluous.


The second "cosmetic" aspect concerns the way that the replacement path is
constructed, which is overly complicated due to the way it evolved.  The
history is as follows:

1) Originally, waf's result was used as is.  It constructs the path by
calling get_python_lib() with the PREFIX value supplied as the 'prefix'
argument.  This results in a path which is always relative to PREFIX, but
which may not actually work.  The PREFIX-relative aspect makes it
FHS-compliant provided that PREFIX is set appropriately.

2) The original FixConfig code unconditionally (provided that the path
wasn't overriden via environment or option) replaced the waf result with
the result of calling get_python_lib() without the 'prefix' argument.
This results in a path which always works, but which may not be
FHS-compliant, including in some cases where an FHS-compliant path is
usable.

3) The massage() function was added to FixConfig in order to improve
(albeit not guarantee) FHS compliance.  This constructs a path by
modifying the path from #2 to be based on PREFIX, but only applying this
modification if the result is in the default sys.path.

The "overly complicated" aspect comes about because, in the cases where
the modification applied by massage() is usable (i.e., passes the sys.path
test), the modified path is actually identical to the original waf result.
Hence, one can accomplish the same thing merely by *not overriding* the
waf result in the cases where the latter passes the sys.path test.


Based on the two considerations above, the path adjustment logic can be
simplified to:

-----------------------------------------------------------------------
If the path wasn't specified by either the environment variable or the
command-line option, and the waf result is not in sys.path, then replace
the waf result with the unprefixed get_python_lib() result.
-----------------------------------------------------------------------

This doesn't change the end result at all.


Separately from the above, there's a bug regarding the way that sys.path
is obtained for the "sys.path check".  Currently, it's obtained directly
from the Python running waf, but that's not guaranteed to be the same
Python that the code is actually targeting.  Thus, the code may not have
the intended result when targeting a non-default Python (or when using a
non-default Python to run waf, though that's much less likely).
Everything else that obtains information about Python for build/install
purposes does so by running the target Python as an external program to
execute the query (usually via the get_python_variables() function).  For
consistency, the sys.path for the aforementioned check needs to be
obtained in the same way.

Applying this fix to the updated logic from above, one gets:

-----------------------------------------------------------------------
If the path wasn't specified by either the environment variable or the
command-line option, and the waf result is not in the target Python's
sys.path, then replace the waf result with the unprefixed get_python_lib()
result.
-----------------------------------------------------------------------

This is precisely what the currently outstanding commit implements.
Although it could be split into separate cosmetic and functional commits,
the combined version is what I already tested (on a wide range of
platforms).  There's also some interdependence in the actual code, in that
the cosmetic change would need to precede the functional change in order
to avoid needing extra temporary code in the interim.

Fred Wright

From gem at rellim.com  Sun Oct 29 01:21:03 2017
From: gem at rellim.com (Gary E. Miller)
Date: Sat, 28 Oct 2017 18:21:03 -0700
Subject: Library-path fix
In-Reply-To: <Pine.LNX.4.58L1X.1710281515340.1977@bolt.sonic.net>
References: <20171019134821.369E013A6891@snark.thyrsus.com>
 <Pine.LNX.4.58L1X.1710281515340.1977@bolt.sonic.net>
Message-ID: <20171028182103.130e227d@spidey.rellim.com>

Yo Fred!

On Sat, 28 Oct 2017 16:07:22 -0700 (PDT)
Fred Wright via devel <devel at ntpsec.org> wrote:

> This is precisely what the currently outstanding commit implements.

Yeah, but it ain't FHS consistent, and violates decades of existing
practice.  It totally breaks dual installations.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can?t measure it, you can?t improve it." - Lord Kelvin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171028/dd77835c/attachment.bin>

From hmurray at megapathdsl.net  Sun Oct 29 06:36:20 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sat, 28 Oct 2017 23:36:20 -0700
Subject: NMEA driver - off by 1024 weeks but says 4096
Message-ID: <20171029063620.C631040605C@ip-64-139-1-69.sjc.megapath.net>

I'm working with an old NMEA device.  It sends things like:

  $GPRMC,062409,A,3726.0822,N,12212.2630,W,000.3,190.6,150398,015.5,E*6A
I've got a fudge time2 to fix that.  It seems to be working.

I'm seeing stuff like this in the log file:
  28 Oct 22:48:51 ntpd[2504]: NMEA(0) Changed GPS epoch warp to -4096 weeks

Anybody know that chunk of code?  Is that code actually doing anything?  If 
it is doing something, why is my fudge working as I expect?




-- 
These are my opinions.  I hate spam.




From hmurray at megapathdsl.net  Sun Oct 29 09:37:52 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Sun, 29 Oct 2017 02:37:52 -0700
Subject: version string
Message-ID: <20171029093752.1331040605C@ip-64-139-1-69.sjc.megapath.net>

VERSION currently says
  1.0.0

I was expecting that to get bumped to 1.0.1 after the release was out so that 
we could easily determine that we are running a development version rather 
than a release.

But maybe that should be 1.1.0?  Or 1.1.1?


devel/hacking.txt says:

We use a variant of three part Semantic Versioning, of the form X.Y.Z.
X, Y, and Z are non-negative decimal integers.

X is the "major" version number.
Y is the "minor" version number.
Z is the "revision" number.

Each release will result in an incremented version number, and the
version number string will be tagged into the git repository.

When the minor number is even, it refers to a "stable" release, and
when the minor number is odd, it refers to a "development" release.

------

When does the revision get bumped?


-- 
These are my opinions.  I hate spam.




From ianbruene at gmail.com  Sun Oct 29 16:27:07 2017
From: ianbruene at gmail.com (Ian Bruene)
Date: Sun, 29 Oct 2017 11:27:07 -0500
Subject: ntpsnmpd
Message-ID: <afea98b3-fdf8-24a6-5aad-e237e3f8fdca@gmail.com>


I have pushed the prototype SNMP AgentX daemon to the repo. In it's 
current state it implements the basic data retrieval packets (get, 
getnext, getbulk), returning dummy values for most items. At this point 
others banging on it is probably more useful than preventing commit clutter.

I make no guarantees that the viewer will not suffer from ocular 
bleeding when viewing the code. It is still very much a prototype.

Documentation will follow shortly.....

-- 
/"In the end; what separates a Man, from a Slave? Money? Power? No. A 
Man Chooses, a Slave Obeys."/ -- Andrew Ryan

/"Utopia cannot precede the Utopian. It will exist the moment we are fit 
to occupy it."/ -- Sophia Lamb

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171029/f5639487/attachment.html>

From esr at thyrsus.com  Sun Oct 29 19:08:08 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sun, 29 Oct 2017 15:08:08 -0400
Subject: ntpsnmpd
In-Reply-To: <afea98b3-fdf8-24a6-5aad-e237e3f8fdca@gmail.com>
References: <afea98b3-fdf8-24a6-5aad-e237e3f8fdca@gmail.com>
Message-ID: <20171029190808.GA12843@thyrsus.com>

Ian Bruene via devel <devel at ntpsec.org>:
> 
> I have pushed the prototype SNMP AgentX daemon to the repo. In it's current
> state it implements the basic data retrieval packets (get, getnext,
> getbulk), returning dummy values for most items. At this point others
> banging on it is probably more useful than preventing commit clutter.
> 
> I make no guarantees that the viewer will not suffer from ocular bleeding
> when viewing the code. It is still very much a prototype.
> 
> Documentation will follow shortly.....

So what's missing?  The Mode 6 transactions to get the read data?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.



From esr at thyrsus.com  Sun Oct 29 19:13:48 2017
From: esr at thyrsus.com (Eric S. Raymond)
Date: Sun, 29 Oct 2017 15:13:48 -0400
Subject: ntpsnmpd
In-Reply-To: <20171029190808.GA12843@thyrsus.com>
References: <afea98b3-fdf8-24a6-5aad-e237e3f8fdca@gmail.com>
 <20171029190808.GA12843@thyrsus.com>
Message-ID: <20171029191348.GC12843@thyrsus.com>

Eric S. Raymond via devel <devel at ntpsec.org>:
> Ian Bruene via devel <devel at ntpsec.org>:
> > 
> > I have pushed the prototype SNMP AgentX daemon to the repo. In it's current
> > state it implements the basic data retrieval packets (get, getnext,
> > getbulk), returning dummy values for most items. At this point others
> > banging on it is probably more useful than preventing commit clutter.
> > 
> > I make no guarantees that the viewer will not suffer from ocular bleeding
> > when viewing the code. It is still very much a prototype.
> > 
> > Documentation will follow shortly.....
> 
> So what's missing?  The Mode 6 transactions to get the read data?

Er, I meant *real* data.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.



From ianbruene at gmail.com  Sun Oct 29 20:22:49 2017
From: ianbruene at gmail.com (Ian Bruene)
Date: Sun, 29 Oct 2017 15:22:49 -0500
Subject: ntpsnmpd
In-Reply-To: <20171029191348.GC12843@thyrsus.com>
References: <afea98b3-fdf8-24a6-5aad-e237e3f8fdca@gmail.com>
 <20171029190808.GA12843@thyrsus.com> <20171029191348.GC12843@thyrsus.com>
Message-ID: <6460401d-b6b0-06b4-d053-91fc9b9963ee@gmail.com>



On 10/29/2017 02:13 PM, Eric S. Raymond wrote:
> Eric S. Raymond via devel <devel at ntpsec.org>:
>> So what's missing?  The Mode 6 transactions to get the read data?
> Er, I meant *real* data.

Currently all of mode 6, much of the AgentX protocol.

Reason for merging now is that ntpsnmpd has hit the level of basic 
functionality and Gary has wanted to poke at it.

-- 
/"In the end; what separates a Man, from a Slave? Money? Power? No. A 
Man Chooses, a Slave Obeys."/ -- Andrew Ryan

/"Utopia cannot precede the Utopian. It will exist the moment we are fit 
to occupy it."/ -- Sophia Lamb

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171029/63d2b203/attachment.html>

From fallenpegasus at gmail.com  Tue Oct 31 01:56:54 2017
From: fallenpegasus at gmail.com (Mark Atwood)
Date: Tue, 31 Oct 2017 01:56:54 +0000
Subject: NMEA driver - off by 1024 weeks but says 4096
In-Reply-To: <20171029063620.C631040605C@ip-64-139-1-69.sjc.megapath.net>
References: <20171029063620.C631040605C@ip-64-139-1-69.sjc.megapath.net>
Message-ID: <CANW5CYVk_-e7PWgD=wOSjf4Ci709W2h2Pj9cgBNwSgF61qPt0Q@mail.gmail.com>

I suggest running it with gpsd for a while instead of NTPsec, and see if
gpsd's logging identifies the issue.

Or if the ntpd log contains the NMEA strings, it may be possible to
reconstruct a gpsd playback file, and play it into gpsd, and see what it
says.  ESR and GEM might could help with that.

..m

On Sat, Oct 28, 2017 at 11:36 PM Hal Murray via devel <devel at ntpsec.org>
wrote:

> I'm working with an old NMEA device.  It sends things like:
>
>   $GPRMC,062409,A,3726.0822,N,12212.2630,W,000.3,190.6,150398,015.5,E*6A
> I've got a fudge time2 to fix that.  It seems to be working.
>
> I'm seeing stuff like this in the log file:
>   28 Oct 22:48:51 ntpd[2504]: NMEA(0) Changed GPS epoch warp to -4096 weeks
>
> Anybody know that chunk of code?  Is that code actually doing anything?  If
> it is doing something, why is my fudge working as I expect?
>
>
>
>
> --
> These are my opinions.  I hate spam.
>
>
>
> _______________________________________________
> devel mailing list
> devel at ntpsec.org
> http://lists.ntpsec.org/mailman/listinfo/devel
>
-- 

Mark Atwood
http://about.me/markatwood
+1-206-604-2198 Mobile & Signal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171031/935b5131/attachment.html>

From fallenpegasus at gmail.com  Tue Oct 31 02:25:16 2017
From: fallenpegasus at gmail.com (Mark Atwood)
Date: Tue, 31 Oct 2017 02:25:16 +0000
Subject: Let's get back to work
In-Reply-To: <20171017103722.B318A13A6500@snark.thyrsus.com>
References: <20171017103722.B318A13A6500@snark.thyrsus.com>
Message-ID: <CANW5CYV11qMKzvsG+HZD4JOkkTK-rY6gkxP512Cr46_6tHBthw@mail.gmail.com>

Some additional suggested work items:
* ntpsec.pool.ntp.org registered (I will take that one)
* do we KNOW that the DEB and RPM distro packaging works?
* get into the distros:
** setup an Ubuntu PPA (I will take that)
** setup a YUM repo (JDB, can you do that, working with GEM?)
** Debian as an alt
** Ubuntu as an alt
** Fedora as an alt
** Yocto
** ipkg  https://en.wikipedia.org/wiki/Ipkg
** opkg  https://en.wikipedia.org/wiki/Opkg
** Automotive Grade Linux
** Amazon Linux (I plan to shamelessly use my dayjob position here)
** Scientific Linux
* Once the snmp subagent is returning data:
** a TUI display app for it
** a GUI display app for it
** something cool: write another closely related snmp subagent for the GPSD
service

I agree with ESR about suggested work assignments:
* GEM: startup jittery and time-to-stable
* DFF: data hiding patch
* ESR: local monitoring mode
* Ian: make snmp return data
* Hal: keep doing your amazing work
* Everyone else: what would you like to do?

..m

-- 

Mark Atwood
http://about.me/markatwood
+1-206-604-2198 Mobile & Signal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20171031/80dcdb86/attachment.html>

From hmurray at megapathdsl.net  Tue Oct 31 06:29:49 2017
From: hmurray at megapathdsl.net (Hal Murray)
Date: Mon, 30 Oct 2017 23:29:49 -0700
Subject: NMEA driver - off by 1024 weeks but says 4096
In-Reply-To: Message from Mark Atwood <fallenpegasus@gmail.com> of "Tue,
 31 Oct 2017 01:56:54 -0000."
 <CANW5CYVk_-e7PWgD=wOSjf4Ci709W2h2Pj9cgBNwSgF61qPt0Q@mail.gmail.com>
Message-ID: <20171031062949.4852F40605C@ip-64-139-1-69.sjc.megapath.net>


fallenpegasus at gmail.com said:
> I suggest running it with gpsd for a while instead of NTPsec, and see if
> gpsd's logging identifies the issue.

> Or if the ntpd log contains the NMEA strings, it may be possible to
> reconstruct a gpsd playback file, and play it into gpsd, and see what it
> says.  ESR and GEM might could help with that. 

I know what the device is sending.  ntpd records it in clockstats.

It's off by 1024 weeks, and I can fudge it back to working well.

What I don't know is why the nmea driver is printing out something about 4096 
weeks and or what that code that doesn't appear to do anything other than 
print a misleading message is supposed to do.  If it doesn't do anything, we 
can just nuke it rather than debug it.

I was hoping that somebody would remember adding that code.  No off list 
responses yet.  I'll add it to my list of quirks to investigate.


-- 
These are my opinions.  I hate spam.