Library-path fix

[Fred Wright]

On Thu, 19 Oct 2017, Eric S. Raymond via devel wrote:

> Fred, can we get an updaterd MR and rationale for your library-path
> fix proposal?

OK, here goes:

First of all, here's the case table for the "usability check" again:

Exists?         In sys.path?    Usable?
-------         ------------    -------
No              No              Unknown
No              Yes             Yes, but this can't happen (*)
Yes             No              No
Yes             Yes             Yes

* site.py's sys.path setup filters nonexistent directories

Gary took issue with the "No" in the third entry, but this "No" is based
on avoiding dependence on PYTHONPATH, which is consistent with current
policy and current code.  The current code also treats "Unknown" as "No".

The point of the table is to show that the check for directory existence
in the current code is superfluous.  This is not to say that the existence
*itself* is irrelevant (since it does affect the sys.path check); it just
means that *checking* the existence is superfluous.


The second "cosmetic" aspect concerns the way that the replacement path is
constructed, which is overly complicated due to the way it evolved.  The
history is as follows:

1) Originally, waf's result was used as is.  It constructs the path by
calling get_python_lib() with the PREFIX value supplied as the 'prefix'
argument.  This results in a path which is always relative to PREFIX, but
which may not actually work.  The PREFIX-relative aspect makes it
FHS-compliant provided that PREFIX is set appropriately.

2) The original FixConfig code unconditionally (provided that the path
wasn't overriden via environment or option) replaced the waf result with
the result of calling get_python_lib() without the 'prefix' argument.
This results in a path which always works, but which may not be
FHS-compliant, including in some cases where an FHS-compliant path is
usable.

3) The massage() function was added to FixConfig in order to improve
(albeit not guarantee) FHS compliance.  This constructs a path by
modifying the path from #2 to be based on PREFIX, but only applying this
modification if the result is in the default sys.path.

The "overly complicated" aspect comes about because, in the cases where
the modification applied by massage() is usable (i.e., passes the sys.path
test), the modified path is actually identical to the original waf result.
Hence, one can accomplish the same thing merely by *not overriding* the
waf result in the cases where the latter passes the sys.path test.


Based on the two considerations above, the path adjustment logic can be
simplified to:

-----------------------------------------------------------------------
If the path wasn't specified by either the environment variable or the
command-line option, and the waf result is not in sys.path, then replace
the waf result with the unprefixed get_python_lib() result.
-----------------------------------------------------------------------

This doesn't change the end result at all.


Separately from the above, there's a bug regarding the way that sys.path
is obtained for the "sys.path check".  Currently, it's obtained directly
from the Python running waf, but that's not guaranteed to be the same
Python that the code is actually targeting.  Thus, the code may not have
the intended result when targeting a non-default Python (or when using a
non-default Python to run waf, though that's much less likely).
Everything else that obtains information about Python for build/install
purposes does so by running the target Python as an external program to
execute the query (usually via the get_python_variables() function).  For
consistency, the sys.path for the aforementioned check needs to be
obtained in the same way.

Applying this fix to the updated logic from above, one gets:

-----------------------------------------------------------------------
If the path wasn't specified by either the environment variable or the
command-line option, and the waf result is not in the target Python's
sys.path, then replace the waf result with the unprefixed get_python_lib()
result.
-----------------------------------------------------------------------

This is precisely what the currently outstanding commit implements.
Although it could be split into separate cosmetic and functional commits,
the combined version is what I already tested (on a wide range of
platforms).  There's also some interdependence in the actual code, in that
the cosmetic change would need to precede the functional change in order
to avoid needing extra temporary code in the interim.

Fred Wright