Resolution of the library-path mess

[Fred Wright]

On Sun, 1 Oct 2017, Eric S. Raymond wrote:
> Fred Wright via devel <devel at ntpsec.org>:
> > On Sun, 1 Oct 2017, Eric S. Raymond via devel wrote:
> > > Gary E. Miller via devel <devel at ntpsec.org>:
> > > > How do you plan that a local NTPsec install from source does not
> > > > overwite an NTPsec install from the native OS repositories?
> > >
> > > That now will never happen if the /usr/local/lb/python-X.Y directory exists;
> > > the install logic will notice that.
> >
> > That of course assumes that if the directory exists, it's in sys.path.
> > Perhaps that's a reasonable assumption, though it's hard to be sure.
>
> Duh.  I should have added that check.  Doing it now.

Sorry for the lateness, but I realized that the current code still has a
bug (as well as a couple of deficiencies of a more-or-less cosmetic
nature).  It's currently checking sys.path in the *running* Python, but it
needs to be checking it in the *target* Python.  Specifying a different
Python otherwise works, so this should respect the difference.

One ofthe "cosmetic" issues is that the directory existence check is
superfluous.  The case table looks like this:

Exists?		In sys.path?	Usable?
-------		------------	-------
No     		No          	Unknown
No     		Yes		Yes, but this can't happen (*)
Yes    		No		No
Yes    		Yes		Yes

* site.py's sys.path setup filters nonexistent directories

Since the install procedure would create the directory if needed, testing
for its preexistence is at best useless.

> Hm, I just realized that what I should be checking there isn't /usr
> but sys.prefix - which will normally be /usr under Linux, but might be
> /usr/local under *BSD.  That should make the transformation into a no-op
> in the case you just pointed out.

The other "cosmetic" issue is that there's no need to "manually" construct
'localized' at all, since that's what get_python_lib() with the prefix
supplied already does.  The only reason to avoid the value that waf comes
up with is that it might not be in sys.path, but *conditionally* using
that value is fine, and simpler.

Thus, the corrected and simplified logic becomes:

-------------------------------------------------------------------------
If the path isn't supplied explicitly and the value obtained by waf isn't
in the target Python's sys.path, then replace it with the target Python's
prefixless get_python_lib() result.
-------------------------------------------------------------------------

I'll make (and test) a version that does that.

On Sun, 1 Oct 2017, Hal Murray via devel wrote:

> > It would be easy enough to throw a warning if the desired /usr/local
> > directory doesn't exist.
>
> Warnings are easily lost in the noise.  So either create the directory or
> treat it as an error and bail.

There are two issues with just "creating the directory":

1) There's no guarantee that Python will actually use it.

2) Creating the directory requires root, and it would be undesirable to
require that 'configure' run as root.

Some improvement in this area is needed, but probably post-1.0.

On Mon, 2 Oct 2017, Daniele Nicolodi via devel wrote:

> On 01/10/17 22:28, Eric S. Raymond wrote:
> > Daniele Nicolodi via devel <devel at ntpsec.org>:
> >> Hello,
> >>
> >> I may be suggesting the obvious, but has anyone contacted the folks on
> >> the Python Distutils mailing list
> >> https://www.python.org/community/sigs/current/distutils-sig/ for their
> >> advise?  I'm surprised NTPSec it the first project facing those problems.
> >>
> >> Cheers,
> >> Daniele
> >
> > One of the things still on the todo list *is* filing an upstream bug
> > about this.
>
> I'm not sure a bug report is the way to go.  Despite that, contacting
> the Python folks may help in deciding how to handle this.  I think
> obeying the principle of less surprise and do what other projects do
> would be the best option here.
>
> Thinking a bit more about it, I think that autotools projects usually
> have a --with-python=<path to python interpreter> configure option that
> tells the install machinery which python interpreter the admin wishes to
> use to run the script or for which interpreter the modules should be
> available. Given that, the python install path is fixed.

That's not the issue.  It's already possibly to specify which Python to
target, and that works (modulo the bug noted above).  But there's no
reliable way to map that to a library directory which is both recognized
by Python and properly prefixed.

The whole concept of including Python-based programs in a base install may
be too new for anyone to have considered all the ramifications.  That's
not an excuse for stuff that's just broken, but it may explain why nobody
has been screaming about it.

On Sun, 1 Oct 2017, Hal Murray via devel wrote:

> > I would bet that /usr/local//lib/python-X.Y is NOT in the sys.path no matter
> > what.

You'd lose that bet on Debian and Ubuntu (at least the versions I have
here), provided that the directory exists.

> So are we back to PYTHONPATH?

No, since it now only uses that directory when it *is* in sys.path and
therefor doesn't have to be added via PYTHONPATH.

On Sun, 1 Oct 2017, Hal Murray via devel wrote:

> Does install have a single where-to-put-it parameter?  Or do we need several,
> one for ntpd (sbin), one for python code (bin) and one for python libraries?

That's what PREFIX is for (as the nominal parent of all those places);
it's just that Python doesn't always allow one to honor it and still have
something that actually works.

Fred Wright