Yup, it's the peer structures

[Eric S. Raymond]

I have verified that I'm not getting replicability on replay because
I'm not capturing the state of the peer list.

This is bad news, because the peer list is a dumping ground for a lot
of poorly managed state.  It has over a hundred members.

This leaves me with difficult choices.  I could brute-force the thing by
dumping the entire peer list state on each packet capture, and picking
it up on replay.  Or I could try to narrow the interface of receive()
so that it only needs to see the peer structure for the packet's
origin.  Or I could try to clean up the peer structure so it doesn't
have so damn much junk in it.  I might have to do all three...

Or, I'm increasingly worried that the present replay architecture may
just be wrong, a bad fit for where the complexity lives. I thought I
could bypass the network I/O layer on replay, but it may be that the
protocol processing is too entangled with it for that to practical.

If that's so, I will have to spoof packet recipt in the UDP read calls
rather than spoofing the protocol machine's receive() function (this
in order to get the peer structures updated properly).  That might
have problems of its own near the polling logic, unless we're willing
for replays to run as slowly as NTP.

It's a good thing we don't seem to have broken anything
functional, because any way I jump it's going to be damned hard
work and require my total concentration for well into the
foreseeable future.

Accordingly, I apologize for seeming kind of absent lately but
that's going to be normal for a while.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

In the absence of any evidence tending to show that possession 
or use of a 'shotgun having a barrel of less than eighteen inches 
in length' at this time has some reasonable relationship to the 
preservation or efficiency of a well regulated militia, we cannot 
say that the Second Amendment guarantees the right to keep and bear 
such an instrument. [...] The Militia comprised all males 
physically capable of acting in concert for the common defense.  
        -- Majority Supreme Court opinion in "U.S. vs. Miller" (1939)