Fix for Python library path problem

[Fred Wright]

On Tue, 26 Sep 2017, Gary E. Miller via devel wrote (different thread):
> On Tue, 26 Sep 2017 22:04:53 -0400 (EDT)
> "Eric S. Raymond via devel" <devel at ntpsec.org> wrote:
>
> > 2. Keep Fred's patch. Ship 1.0 with FHS non-conformance as a known
> >    and documented bug.
>
> Gack.  Opening a tech support nightmare.
>
> We constanlty have issues with conflicting system installed and user
> installed ntpd.  it will be a lot of fun when the distro updates
> ntpd and breaks the user installed ntpd.  That was not a problem
> before this patch.

Interestingly enough, this last paragraph is both completely irrelevant to
the issue at hand, and simultaneously gets to the crux of the matter. :-)

The whole issue being discussed here is the install location for *Python
libraries*.  Nothing else.  Since the classic NTP package doesn't use
Python at all, a conflict in this area is impossible.  There might, of
course, be conflicts in *programs*, *config files*, or whatever, but that
has absolutely nothing to do with the Python library path.

*However*, once distros start including ntpsec instead of classic ntpd,
this sort of conflict will become possible, and FHS compliance will
matter.  But if we can assume that no distro is including a pre-1.0
ntpsec, then the conflict doesn't exist for 1.0 by definition.  If we can
punt on FHS compliance for 1.0, then that removes the time pressure for
figuring out the right way to get Linux Python to play nicely with FHS.

> > 3. Hold 1.0 until we can write a replacement get_python_lib() that
> >    works right (e.g. produces an FHS-conformant path set by default.)
>
> Before someone rewrites get_python_lib() we better agree on what it
> should do.  I suspect the solution will require much more than just
> changes to get_python_lib().

Agreed.

On Thu, 28 Sep 2017, Gary E. Miller via devel wrote:
> On Wed, 27 Sep 2017 19:02:23 -0700 (PDT)
> Fred Wright via devel <devel at ntpsec.org> wrote:
>
> > One of the ways to do #1 is to use the path returned by
> > get_python_lib() without the prefix option.  This is what GPSD has
> > done for years.
>
> And caused subtl problems for years.

Can you name any actual problems, aside from offending the FHS gods?  If
it caused so many problems, why were you apparently unaware of it until I
pointed it out? :-)

> > 2.2) Make use of some sort of hook to augment sys.path, which is what
> > pip appears to do.  So far, this approach doesn't look promising,
> > partly because the only such mechanisms seem to require that the hook
> > itself be in a "non-local" location in order to be seen by Python
> > before being applied.  A chicken-and-egg problem.
>
> Huh?  Looks simple to me.  Looks like the preferred solution.
>
> Since pip, and many other programs do it, that is a lot of good precendent.

Precedent, yes.  Full understanding of the mechanism and constraints, no.

> > 2.3) Add code to the programs to augment sys.path prior to the import.
>
> Gack.  Breaks in a large number of ways.  It is very common for
> packaers to build to one path, package the results, then install
> elsewhere.

It would naturally be based on the *final* install location, not the
intermediate.  I believe packaging systems normally know the final
location at package build time, but I could be wrong about that.  If the
final install location isn't known at the time the package is built, then
*any* kind of sys.path adjustment would have to be handled by the
packaging system, since the ntpsec build scripts would have no way to know
what to do.  This applies whether one is patching code, generating special
hook files, or whatever.

I'm not saying tat I'm especially fond of this solution, but it is
something that would work.

Another possibility that occurred to me is placing the library directory
in the *program* directory, e.g., /usr/local/bin/ntp/, since module
directories parallel to a Python program are always recognized
automatically by Python.  This would work, but it's rather nonstandard.
If ntpsec assumes that it "owns" all paths of the form <prefix>/bin/ntp*,
then this wouldn't create any new conflicts, and AFAIK the NTP suite has
never had a program just called "ntp", anyway.

Again, a not very flavorful, but functional solution.

I attempted to see if there's any precedent for this, but the systems I
have here don't seem to be terribly useful examples.  My Ubuntu install
has exactly one program in /usr/local/bin/.  My CentOS and Fedora installs
don't even have /usr/local/bin/.  My Debian installs have lots of things
in /usr/local/bin/, but the only Python code is either from ntpsec or
GPSD, so it's not really an "outside opinion".


One of the this I've discovered in looking at this stuff is that having
code that relies on looking at sys.path would be fragile, for at least two
reasons:

1) Some directories get added by installs of other packages, but relying
on any such directories would break if the relevant packages were removed.

2) Directories that don't exist don't get added.  But the install process
creates directories as needed.  So if one came up with a directory that
Python knows about but doesn't currently exist, it would appear from
looking at sys.path at configure time that Python doesn't know about it,
even though it might work perfectly well after the install.

Note that a workaround for #1 is to launch Python with the -S option to
suppress processing site.py, but that also may exclude some directories
that *aren't* package-specific.


BTW, it's worth pointing out that this entire argument is over the
*default* value of PYTHONDIR (and technically also PYTHONARCHDIR, but
that's currently unused as noted in the comment in pylib/wscript).  Any
user or packaging system is free to supply a different value, either via
environment or via option.  At that point, it becomes the responsibility
of the user or packaging system to ensure that the specified directory is
either valid by default, or will be made valid when it's needed.  In fact,
I suspect packaging systems would tend to specify a lot of such things
explicitly in their own scripts, anyway, so this is mostly about getting
reasonable default behavior for a "bare" configure/build/install.

Fred Wright