Library-path glitch, again

[Fred Wright]

On Fri, 6 Oct 2017, Eric S. Raymond via devel wrote:
> Fred Wright writes:
> >Sorry for the lateness, but I realized that the current code still has a
> >bug (as well as a couple of deficiencies of a more-or-less cosmetic
> >nature).  It's currently checking sys.path in the *running* Python, but it
> >needs to be checking it in the *target* Python.  Specifying a different
> >Python otherwise works, so this should respect the difference.
> >
> >One ofthe "cosmetic" issues is that the directory existence check is
> >superfluous.  The case table looks like this:
> >
> >Exists?         In sys.path?    Usable?
> >-------         ------------    -------
> >No              No              Unknown
> >No              Yes             Yes, but this can't happen (*)
> >Yes             No              No
> >Yes             Yes             Yes
> >
> >* site.py's sys.path setup filters nonexistent directories
> >
> >Since the install procedure would create the directory if needed, testing
> >for its preexistence is at best useless.
> >
> >The other "cosmetic" issue is that there's no need to "manually" construct
> >'localized' at all, since that's what get_python_lib() with the prefix
> >supplied already does.  The only reason to avoid the value that waf comes
> >up with is that it might not be in sys.path, but *conditionally* using
> >that value is fine, and simpler.
>
> But Gary replies:
>
> >Uh, I disagree on the 3rd case (Yes, No).  I install, then add my
> >dir to PYTHONPATH.
> >
> >I guess we could document that: for people that want to be HFS
> >conformant, add PYTHONPATH before doing the install.

> Fred, your timing is awful and you shouldn't have submitted multiple
> changes as a single MR.  Three days ago I might have taken a single,
> simple change that fixed the one bug.  Now I can't justify doing that
> - especially when Gary has raised a question about your analysis.

The table above simply shows the known behavior of site.py in a clearer
way.  Gary's "question" is based on wanting to use PYTHONPATH to allow
directories not in the default sys.path, even though the undesirability of
that was supposedly a settled question.  PYTHONPATH seems to be some sort
of zombie that refuses to stay dead. :-)  My proposed change doesn't make
any policy change at all; it simply implements the current policy in a
simpler and more correct manner.

That being said, I don't think it's a big problem to wait until after 1.0,
especially since it's not the only broken thing about installing for a
non-default Python.

I combined the fixes since they interact at the coding level.  I posted
the updated logic and reasoning behind it here; the change simply
implements that logic.  The hard part was figuring out that getting
sys.path through waf's get_python_variables() is hopeless, and going
around it.  At least *my* code doesn't apply eval() to externally obtained
data, which is more than I can say for waf. :-)

> The one patch I would like to see is to devel/TODO explaining the
> source/target bug.

As things are now, it should simply be documented that installing for a
non-default Python version doesn't work.  The bug being addressed here
only has the effect of making it less likely to come up with an
FHS-compliant location in that case, while the lack of appropriate
shebanging would make that case not work at all.

Fred Wright