Python libs on Debian/Raspbian

[Fred Wright]

On Wed, 20 Dec 2017, Richard Laager via devel wrote:

> Rather than debate hypotheticals further, I have submitted a merge
> request to implement the "option H" idea being discussed:
> https://gitlab.com/NTPsec/ntpsec/merge_requests/615
>
> I'm looking for feedback on that particular merge request. Only the last
> commit is particularly interesting, which addresses #414 by eliminating
> wafhelpers/fix_python_config.py and making the utilities search
> PYTHONDIR and PYTHONARCHDIR explicitly.

Actually, "option H" is one of the possibilities I raised for completeness
when this was being discussed back in September.  But I wasn't terribly
wild about this approach, and neither was anyone else.

One thing that makes "option H" unflavorful is that it adds kludgery to
every Python program just to program around a Python bug that only exists
in a subset of Linux distros (and not in any non-Linux cases).  With the
.pth file approach, the Python programs are written to assume that Python
works correctly, and if that turns out not to be the case, a .pth file is
added to fix that.

Another problem is that "option H" bypasses the normal versioning of
Python library locations.  As I mentioned in my earlier post in connection
with the similar downside of PYTHONPATH, "version bypass" could be a plus
where only pure Python libraries are concerned, but isn't when compiled
extensions like 'ntpc' are involved.  Extensions almost never work with a
different Python that the one they were compiled and linked for, with a
variety of possible failure modes.  I've seen at least one case where
importing ntpc into the wrong Python results in a segfault (which of
course can't be caught with a try/except block).  And the fact that it can
fail with something like a segfault means that one can't rule out the
possibility of a case where the import "succeeds" but has broken code.
The safest thing to do is to stick to the arrangement where libraries are
only findable by the one Python chosen at build/install time, and all
other cases get straightforward import errors.

Although the chance of running a program with the wrong Python version can
be substantially reduced via proper shebanging, it would still not be
impossible, so the "version bypass" remains undesirable.  Further
discussion of the shebanging issue (which is a related but separate
problem) really belongs in another thread, though, rather than increasing
the noise-to-signal ratio here.


AFAIK, the only concerns about the .pth file approach relate to its
violation of PREFIX, and whether it should be installed automatically
(when needed), or via instruction to the user.

On the latter point, first of all, I neglected to mention in my earlier
post that it would make sense to limit the automatic .pth file to the
default PYTHONDIR/PYTHONARCHDIR case.  The idea is that, if the user or
packaging system chooses to override the default location(s), then the
user or packaging system should assume responsibility for ensuring that
the chosen location(s) actually work(s).

That being said, in the normal default case, foisting the .pth fix off on
the user would be inconvenient and error-prone.  And pretending that a
PREFIX violation really isn't just because the user was invoked as a
script processor is a copout. :-) Thus, the automatic fix is more
appropriate in the default case.

On the PREFIX "violation", it's useful to step back and consider what the
actual motivation for the variable PREFIX is, rather than treating
honoring PREFIX as some sort of religious mandate.  The real purpose of
the variable PREFIX is to allow the same software to be installed from
multiple "origins" without colliding.  In the case of the out-of-PREFIX
.pth file, however, there would be no collision issue, for at least two
reasons:

1) The name of the .pth file can be derived from PREFIX, such that .pth
files for different prefixes are distinct even when installed in the same
directory.

2) In the primary case of interest, namely /usr/... vs. /usr/local/...,
there would be at most one .pth file needed anyway, since it would never
be needed in the /usr case.

It's also worth noting that, since the purpose of the .pth file is to fix
something which can be legitimately considered to be a bug in the Python
installation (a POV which is already accepted by at least some distros),
and since the relevant Python installation is itself outside PREFIX, it
actually makes sense that the patch to fix it is also outside PREFIX (in a
location "belonging to" the faulty Python).


BTW, regardless of the approach taken, completely removing FixConfig is
undesirable, for at least three reasons, two of which I mentioned
in my earlier post:

1) There are some cases where the unprefixed get_python_lib() returns a
location more in keeping with platform practice than the prefixed version,
while still being within PREFIX.  In such cases, it's still desirable to
override waf's choice.

2) There's another bug that I haven't fixed yet, related to waf's use of
python-config, and which will also need a fix within the FixConfig
framework.

3) The concept of cleaning up leftovers from earlier versions of the code
that chose different locations still remains, and that's also one of the
functions of the FixConfig code.  This is something which can eventually
go away, but needs to hang around for a while, probably for at least a
couple of releases.

Once the stale library cleanup is no longer needed, *and* waf comes up
with the most appropriate values for PYTHON[ARCH]DIR on its own in all
cases, *and* it either abandons the fundamentally flawed python-config or
at least locates it more correctly, *then* FixConfig can go away.


Fred Wright