Removing the worst cruft

[Eric S. Raymond]

Heads up, Mark!  Policy issue, you're needed.

As part of working on the new visualization code, I've been looking
hard at the logfile report generators we inherited from Classic.
These consist of undocumented or very poorly documented scripts
underneath the util/ directory, written in a mixture of Perl, awk,
and a statistics-processing language called S.

Reading this stuff will give you a headache - the kind of headache
that means "Technical debt!  Remove or rewrite."  I particularly
dislike leaving such things in a distribution, because even when they
are small they are attractive nuclei for the formation of larger
hairballs.  You can't maintain what you don't understand, and it's
a bad habit to have a corner you throw junk into thinking "I'll
fix and/or document it it later". That's what util/ is.

For his reason, my hope is to eventually abolish or export the entire
contents of the util/ directory.  Some of the report generation
functions should be subsumed into ntpviz or another Python tool based
on the ntpstats.py class. Some of the standalone tools and Perl
support classes should be translated into Python and properly
documented, getting their own subdirectories under the distribution
root.

I've already removed a fair amount of crud that will never be useful
again.  ntploopstat was a representative example - a Perl script that
relied on using mode 7 queries to pull loopstats from remote machines
so they can be analyzed locally. We removed mode 7; in the very
unlikely event that this sort of thing is every required again the
right way to do it will be scp -r from the remote stats directory.

But there is a last category that is the actual reason for this note.
The volume of nasty undocumented code in util/ is dominated by report
generation and test tools specific to three clock drivers; AUSTRON,
IRIG, and CHU.  In effect, continuing to support those drivers commits
us to a much larger maintainance burden than is apparent from the
driver code itself. Not just in utils/, either; there's a chunk of C
code specific to the IRIG driver in (inexplicably) libntp.

The other drivers that I believe are obsolete don't have this kind of
second-order cost.  Yes, they bulk up the distribution, but the
code-complexity overhead of keeping them around is limited to one C
file each. I find this annoying, but tolerable to keep the few grognards
who might still be attached to them happy.

But AUSTRON/IRIG/CHU...I think there's a good (though not absolutely
dispositive) case for simply dropping them all.  To do that, we'd
have to change a policy decision made before you were PM, and we'd need
to have thought through the tradeoffs

Before we first attacked this question, very early in the project's
history, I did extensive research into every kind of hardware
supported in Classic. You can read the results here:

https://www.ntpsec.org/drivers.html

At the time the bulk of that document was written, our plan was to
drop support for hardware that had been EOLed for 7 years.  Before
actually executing on that, we moved to a more conservative, policy:
remove drivers only for specific technical reasons like "WWVB stopped
broadcasting that modulation in 2012 and never will again" or
"requires a proprietary kernel driver". That's how I've been rolling
ever since.

The reason for the change was that we wanted in general to position
NTPsec as a conservative, safe-pair-of-hands option for users looking
for a no-surprises exit from NTP Classic.  This in turn seemed to require
that we be extra-careful careful on behalf of the sort of time
grognard (/me points half-humorously at Hal Murray) who might actually
have long-EOLed device like a HP 58503A still running.

This has resulted in leaving in some drivers that I think are plain
silly. A major case in point is DUMBCLOCK.

    This driver supports a dumb ASCII clock that only emits localtime at a
    reliable interval. This has no provisions for leap seconds, quality
    codes, etc. It assumes output in the local time zone, and that the C
    library mktime()/localtime() routines will correctly convert back and
    forth between local and UTC.

In 2016 this merits a startled "WTF?". I have, nevertheless, not
succumbed to temptation and removed it.  I'm being *that careful*
about not removing things unless we have a compelling
why-we-dropped-this story to tell the grognards.

The DUMBCLOCK driver is a useless piece of historical baggage, but it
doesn't cost us anything other than carrying 321 lines of C and a
stupid manual page.  AUSTRON/IRIG/CHU are a different matter, and
that's why I'm now suggesting that a policy change may be in order.

The policy question is "How much of a maintainance pain in the ass
does a legacy driver have to imply before we write off the handful of
users it might still have in order to have a distribution we actually
understand and can maintain going forward?"

With that in mind, I'll now brief you on the three drivers that are
really, *conspicuously* bad this way.

AUSTRON: An obsolete line of 1PPS GPS receivers.  Austron was acquired
some time before 1996 and the product line apparently discontinued
then.  A few units were still in hobbyist use in 2006, but there don't
seem to be more recent references to the hardware itself even on the
time-nuts list. (Someone was trying to get the manuals scanned for
archival purposes in 2014.)  I can find none available for sale on the
web.

The real trouble with this driver isn't the driver itself, but that
Dave Mills liked the device enough to write a bunch of special-purpose
logging hacks into utils/.  They've been bitrotting in there,
undocumented, for about the last 20 years.

IRIG: This driver decodes *audio input*! It requires custom wiring to
connect an analog feed from a radio to a microphone or line-in port.
All for accuracy two orders of magnitude worse (500us) than a cheap
1PPS GPS, and no holdover capability.  Its performance *sucks* by
modern standards - it's a desperate kluge left over from the days
before GPSes and cheap DSP.

If we dropped this, we could also drop two significant-sized pieces
of C code in utils/ (never documented, of course) that exist to
generate test tones and probe the capability of the local audio
system.  Naturally there's some dependent undocumented Perl cruft
as well.

CHU: Similar bad idea to IRIG, decodes analog radio from the Canadian
national time authority, even worse accuracy (1ms). Carries with it
some code in libntp for controlling ICOM radios - not clear the models
it covers exist any more.  Also carries with it C code in utils/ for
calculating propagation delays from the station.

Wearing my software-architect that, I seriously want to nuke this
stuff from orbit - anybody still using these drivers in 2016 has taken
extracting extra service life from old gear well beyond the stupidity
point-of-no-return. But there's a political/PR context to the question
as well.  How do we trade off the compromise to our "safe pair of
hands" positioning against what we gain by lowering complexity and
expected future defect rates?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

The end move in politics is always to pick up a gun.
	-- R. Buckminster Fuller