All hands - we need to test Fred's build changes pronto

[Fred Wright]

On Tue, 26 Sep 2017, Eric S. Raymond via devel wrote:

> Hal Murray <hmurray at megapathdsl.net>:
> > The permissions on the stuff in ntpclients had the execute bit removed so
> > local testing doesn't work any more.  I assume install fixes that since you
> > reported that ntpq worked.
> >
> > In particular, tests/options-tester.sh says things like:
> >
> > VERSION: ntpd ntpsec-0.9.7+1444 2017-09-26T09:54:52Z
> > VERSION: ./tests/option-tester.sh: line 40: ./ntpq: Permission denied
> > VERSION: ./tests/option-tester.sh: line 42: ./ntpdig: Permission denied
> > VERSION: ./tests/option-tester.sh: line 44: ./ntpmon: Permission denied
>
> Following Fred's change, client tests now need to be run from
> build/main/ntpcllients rather than the source directory. I have pushed
> a change to tests/options-tester.sh that does this; please verify.

Sorry, I should have caught that.  I was using my own test script to run
everything.  My original post did mention the possibility of personal
scripts possibly needed updates, but I failed to notice that there was one
in the project files.

> There's a small loss of convennience here that I regret, but updates to the
> Python service libraries already required a waf build.  It's more consistent
> that changes to the front ends do too, and eliminates an error where you
> don't get updated libraries because that step was forgotten.

Running from the build directory would be a bit more completion-friendly
if "buildprep" were renamed to something that doesn't start with "build".
:-)

On Tue, 26 Sep 2017, Ian Bruene via devel wrote:

> python3 build: fails, can't find ntp module, after installing p3 version
> it runs but crashes with a type conversion error (I'll get on this right
> away)

This is a bit short on detail, but you might just have been running afoul
of version dependencies.

For in-tree testing with pure Python libraries, one can usually switch
Python versions with no issues (within whatever versions are supported).
But compiled extensions are compiled and linked against a specific version
of Python, and in general shouldn't be expected to work with other
versions.  I.e., compiled extensions can only be polyglot at the source
level, not the binary level.  Recent versions of Python have started
putting the Python version in the extension filenames, so that they're
simply not found when using the wrong Python version.  But in other cases,
it may load an extension that just doesn't work properly.  I know of at
least one case where it results in an app crash.

For installed code, it's even worse, since the library location is almost
always version-specific.  So even pure Python libraries don't get found
when using a different Python version than was used for the build/install.
The only Python programs that don't suffer from this are ones that only
use the standard Python libraries.

On Tue, 26 Sep 2017, Hal Murray via devel wrote:
> hmurray at megapathdsl.net said:
> > Testing Python code is broken.  It's using the system libraries rather than
> > the new/local libraries.
>
> Author: Eric S. Raymond <esr at thyrsus.com>
> Date:   Tue Sep 26 13:59:10 2017 -0400
>
>     Restore accidentally removed creation of a magigic ntp/ link.
>
> Fixed.  Thanks.

This illustrates the subtler problem with in-tree testing.  The problem is
often framed as programs not working at all when the code hasn't been
installed to the system.  But when an older version is installed, code
relying on the system libraries may be using the older version, instead of
what one is trying to test.

This is the sort of scenario where using PYTHONPATH *could be* legitimate,
but since it's fairly easy to set things up to work without it, that's
usually a better choice.  When running a Python program, Python always
adds the program's own directory to sys.path, so as long as all local
imports are findable relative to that directory, they'll shadow any
installed versions that happen to be present.

Fred Wright