Bite of the Buildbugs!

[Fred Wright]

(from a different thread)

On Fri, 8 Dec 2017, Richard Laager via devel wrote:

> I agree that your system does not have /usr/local in its sys.path by
> default.
[...]
> 1) Ignore prefix and install to /usr. This is Fred Wright's solution and
> is what is seemingly the point of fix_python_config.py. I think this is
> wrong; prefix should be respected unless explicitly overridden by
> --pythondir.

Ideally, yes, but encumbered by the absence of the /usr/local/... path in
the default sys.path on some Linux platforms.

> 2) Set PYTHONPATH. Fred Wright said there was a consensus that this is
> undesirable.

There are a number of reasons why PYTHONPATH is an undesirable solution:

1) Environment variables are not managed in a structured manner,
particularly in cases such as this where they need to be set up for *all*
users on the system.  Leaving the whole issue as an "exercise for the
reader" is obnoxious, and the formerly existing instructions for using
PYTHONPATH weren't even correct.  Setting this up automatically is hacky,
platform-specific, and sometimes site-specific.  Updating or removing (in
the event of an uninstall) environment variable setups reliably is even
messier.

2) Setting up environment variables via shell startup scripts doesn't work
in contexts where shell startup scripts aren't processed, such as ssh
remote commands.

3) There are some contexts where environment variables are suppressed for
security reasons.  That's admittedly not too likely in this particular
case, but nevertheless not impossible, especially when one allows for the
possibility of third-party programs relying on the ntp libraries.

4) Most Python library locations are versioned, but PYTHONPATH is not.
That can be either a plus or a minus, depending on the specifics, but when
compiled extensions (which are only valid for the Python version they're
built for) are involved, bypassing versioning is usually at best useless
and sometimes downright nasty.

5) Any use of a persistent PYTHONPATH applies to *all* Python programs on
the system, not just the ones that need it.  The combination of this with
#4 means that version-related screwups may be inflicted on *other* Python
programs via PYTHONPATH.

In short, relying on a persistent PYTHONPATH is like relying on a
persistent LD_LIBRARY_PATH, only worse.

> 3) Create a .pth file in the appropriate place, which is somewhere under
> /usr. There are probably existing ones:

I did investigate .pth files when this issue first arose.  At the time,
based on a strict interpretation of the requirements, I concluded that
.pth files don't help.  In the cases of interest, there are no locations
within /usr/local where Python looks for .pth files, so if the goal is to
avoid writing *anything* outside /usr/local while placing libraries under
/usr/local (and not in any default sys.path location), then there's an
insurmountable chicken-and-egg problem with .pth files.

That being said, it seems upon reflection that busting PREFIX for a .pth
file is a lot less onerous than busting it for the libraries themselves,
particularly given that one can use a naming convention that avoids
collisions (e.g., ntp-usr-local.pth in the /usr/local case).  So, assuming
that this degree of PREFIX violation is acceptable (as suggested above),
then installing a .pth file looks like a viable solution.

There are still a few details to be worked out:

1) Where to place the .pth file itself.  The result of the unprefixed
get_python_lib() looks like a plausible candidate, but that needs to be
verified.

2) There are certain cases (notably OSX) where the normal placement of
Python libraries is quite "unusual", and this isn't reflected in the
behavior of the prefixed get_python_lib() (essentially due to dumbness of
the latter).  Meanwhile, the result of the unprefixed get_python_lib() may
better conform to normal practice on the platform, and may also conform to
PREFIX, albeit semi-coincidentally.  In such cases, using the unprefixed
get_python_lib() is preferable for consistency with the platform's normal
practice (while still honoring PREFIX), but the exact condition for doing
this needs to be worked out.

3) It would be desirable to limit the installation of .pth files to cases
where they're actually needed (hopefully eventually becoming "nowhere" as
more Python setups get their acts together).  In principle, this means
checking whether the chosen location is in the default sys.path, but that
can't be determined reliably at configure time since nonexistent
directories are omitted from sys.path, and Python doesn't publish a
"potential" sys.path.  The current workaround for that issue is a note in
the documentation that the relevant directory may need to be created in
advance, but that's rather kludgy and unreliable.  Fortunately, for this
particular usage, the decision can be deferred until install time, and if
it's done as a post-install hook, then the directory will already have
been created if needed, and checking sys.path will have the desired effect
without any pre-creation needed.  That does of course mean that the .pth
file won't be a normal install product, and thus won't be automatically
tracked by waf, and will need to be handled explicitly for uninstall as
well.

4) It might be possible to fix PYTHONARCHDIR in the process, so that it
can actually be used as intended (as the install location for compiled
extensions).

If there are no objections to this approach, I'll go ahead and implement
and test it.

> Given that this is fixed in the distro, I think the right trade-off is
> to recommend #3, either in documentation and/or in a warning from waf if
> the configured pythondir is not in sys.path. I would like to see the
> fix_python_config.py code removed, as it is creating new breakage.

Removing FixConfig altogether isn't the right thing in any case, since
it's designed to be a general-purpose container for fixes to things that
waf gets wrong about Python.  E.g., there's another, subtler problem
(related to the fundamental brokenness of python-config) which I don't yet
have a completed solution for, but which will need more FixConfig code to
fix.

Fred Wright