Resolution of the library-path mess

[Eric S. Raymond]

I've spent the last several days researching all the various proposals
for how we handle locating the Python libraries, and trying to fully implement
several of them.

My conclusion is that Mark called this one right on the 28th:

>My inclination is to keep [Fred's] patch, document the lack of FHS compliance,
>and roadmap a fix to get_python_lib, possibly by nudging the WAF or python
>communities to write it.

More specifically, the only information we have about where we
can put Python modules is the return from the distutils funcrion
get_python_lib().

My attempt to deduce a parallel 'local' location for that fails because
it is not in fact a rule on all our target platforms that if /usr/lib/X/Y
exists then /usr/local/lib/X/Y does as well (however, see below for
what we can do if this is true).

Messing with the default sys.path in any way (notably through .pth files)
is not a good idea either, because, as Fed Wright notes:

>One of the this I've discovered in looking at this stuff is that having
>code that relies on looking at sys.path would be fragile, for at least two
>reasons:
>
>1) Some directories get added by installs of other packages, but relying
>on any such directories would break if the relevant packages were removed.
>
>2) Directories that don't exist don't get added.  But the install process
>creates directories as needed.  So if one came up with a directory that
>Python knows about but doesn't currently exist, it would appear from
>looking at sys.path at configure time that Python doesn't know about it,
>even though it might work perfectly well after the install.

I'm also opposed to it because changing the default load path
could end up causing misbehavior in *other packages*.  Not good
to have sharp elbows like that.

However there is good news for Gary and anybody else who is really worried
about FHS compilance.  While we cannot *guarantee* being able to do that, it
was trivial to change the installation code so that if the FHS-compliant
/usr/$PREFIX directory corresponding to the return value of python_lib()
*already exists* (and can therefore be presumed to be scanned by the
Python library loader) it is used.

On most systems this will do the right thing.

I have implemented and tested this solution, along with full
documentation and packaging guidance.

We are now at T + 3 days and need to get a move on, so I'm limiting
the comment and objections period on this to end at close of business
Monday (2 oct).  If we don't have consensus on a clearly better
solution by then, we go to freeze and testing for a week, aiming to
ship on the 9th.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

If gun laws in fact worked, the sponsors of this type of legislation
should have no difficulty drawing upon long lists of examples of
criminal acts reduced by such legislation. That they cannot do so
after a century and a half of trying -- that they must sweep under the
rug the southern attempts at gun control in the 1870-1910 period, the
northeastern attempts in the 1920-1939 period, the attempts at both
Federal and State levels in 1965-1976 -- establishes the repeated,
complete and inevitable failure of gun laws to control serious crime.
        -- Senator Orrin Hatch, in a 1982 Senate Report