Logfile visualization tools - request for comment

[Eric S. Raymond]

I've spent the last week reading code and preparing for a serious
effort to write logfile visualization tools for NTPsec.

There are at least two good reasons to do this, one retrospective and
one prospective.  The retrospective one is that the stats and
data-reduction tools now in the distribution are a huge mess. They're
archaic, often embodying assumptions that have long since passed their
sell-by date (one pair of tools relies, for example, on mode 7, which
we've eliminated).  They're poorly documented or not documented at
all. They're written in Perl, which is a serious maintainability
problem. The whole area cries to be cleaned up - or better yet, nuked
and replaced with better code.

The prospective reason is that I need a way to make sense out of my test
farm data.  I want to be able to answer a bunch of questions, beginning with
"How important are check servers to a machine with an GPS?"

The path forward that I'm considering is a Python translation of the
NTP branch of David Drown's chrony-graph software. It makes beautiful and
interesting visualizations, embodying a lot of domain knowledge about
which statistics and relationships are interesting.  And of course, that last
part is where my own knowledge is weakest. Co-opting his work will let me
concentrate on the software-engineering aspect of the problem.

I'm thinking Python translation for two reasons.  One is our general
Python-and-sh policy for scripting, to reduce maintainance complexity
down the road.

Another is that, as Gary Miller has pointed out, ddrown's collection of
shellscripts and Perl has terrible locality.  Gary says he can see in
his graphs artifacts from chrony-graph's disk overhead, and I have no
reason to disbelieve that. Gary suggests that a symbiont daemon, keeping
intermediate data in memory until the final graphs need to be produced,
would produce less noise.

So, translate chrony-graph to Python.  But this would leave us with
a coordination problem. It means either ddrown has to be prepared to
let the Python version be his new mainline, or we have to cross-port
all his improvements after the fork.

David, do you have any suggestions for making this less painful?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>