[Git][NTPsec/ntpsec][master] Deleted 66 commits: Implement BSD-style endian-codecs

Eric S. Raymond gitlab at mg.gitlab.com
Sun Oct 2 13:39:11 UTC 2016 

Eric S. Raymond pushed to branch master at NTPsec / ntpsec


WARNING: The push did not contain any new commits, but force pushed to delete the commits and changes below.


Deleted commits:
5fd21cc6 by Daniel Fox Franke at 2016-06-28T15:30:21-04:00
Implement BSD-style endian-codecs

- - - - -
6ab62db0 by Daniel Fox Franke at 2016-06-28T15:38:46-04:00
Whitespace cleanup

- - - - -
7ee0ffa5 by Daniel Fox Franke at 2016-06-28T16:12:14-04:00
Implement an RFC7822-compliant parser

- - - - -
3824e87f by Daniel Fox Franke at 2016-06-29T12:57:42-04:00
Make sendpkt() take void* for data

- - - - -
c96c0827 by Daniel Fox Franke at 2016-06-29T17:03:33-04:00
Add l_fp<->uint64_t conversion functions

- - - - -
25c8b0a3 by Daniel Fox Franke at 2016-07-05T16:00:16-04:00
Fix byte order in l_fp conversion functions

The lfp_to_uint64() and uint64_to_lfp() functions previously expected
the l_fp fields to be in network byte order. However, this is only
true of the l_fp fields in struct pkt; everywhere else they are in
host byte order. These functions now expect host byte order. To use
them with network-byte-order l_fp's, use the existing conversion
macros from ntp_fp.h.

- - - - -
d5df8f8d by Daniel Fox Franke at 2016-07-05T17:14:33-04:00
Refactor receive() and process_packet()

This is the first and biggest step of a complete rewrite of NTP's
receive-side logic. receive() now uses a struct parsed_pkt instead of
struct pkt, and the function is much better organized. Authentication
is now performed as early as possible rather than as late as possible;
this isn't the 1980s any more and checking a MAC is no longer
expensive. Other security-relevant checks such as orgin timestamp
validation are also moved to the fore.

Notable bug fixes introduced by this commit:

* Server packets are now immediately rejected as bogus if there is no
  request outstanding (peer->outcount == 0).

* The old code was sticking the packet's transmit timestamp into the
  peer structure's receive timestamp, a bug that appears to have been
  introduced in 2008 when interleaved mode was implemented. This is
  likely to have been harming timekeeping precision all this years,
  and the fix may be an especially big win for clients which talk to
  a stratum 1 server via a LAN.

* KoDs are handled more sensibly and will never bump the polling
  interval up to anything ridiculous.

Known issues as of this commit:

* Only basic client and server modes are currently working. Support
  for the myriad other association modes will be reintroduced in
  later commits.

* MS-SNTP support is not yet re-implemented.

* The call to record_raw_stats() is missing.

* Some system counters such as ntp_oldversion and ntp_newversion
  are not incremented at appropriate times.

* There are no logging calls in the new code.

Current testing status: I've verified that the code basically works in
the common cases both as a client and as a server. I haven't tested
authentication yet, and haven't verified that every sort of bogon is
correctly rejected.

- - - - -
c027cf3a by Daniel Fox Franke at 2016-07-06T08:11:11-04:00
Origin timestamp checking for symmetric modes

As a first step to getting symmetric mode working again, implement the
different, more relaxed rules for origin timestamp validation that
symmetric mode requires. This still only covers basic symmetric mode,
not interlaved.

- - - - -
5a150d47 by Daniel Fox Franke at 2016-07-06T09:06:25-04:00
Typo fix (add missing return)

- - - - -
10e4d987 by Daniel Fox Franke at 2016-07-06T09:41:08-04:00
Remove interleaved mode

Interleaved mode was an invention intended to improve timekeeping
precision in symmetric and broadcast mode. The problem it intended to
solve is that transmit timestamps have to be written before the packet
is sent, but right *after* the packet is sent, better information
becomes available because you know exactly when the packet made it
through the kernel and out onto the wire. So, the basic idea of
interleaved mode was to dump that better value into the *next* packet,
and have the peer follow along with that, always one packet behind.

This is a problem that PTP is clearly better suited to solving, but
interleaved mode still seems at least reasonable in theory. However,
there are two big problems.

First, interleaved mode adds a great deal of complexity to NTP's state
machine. This led to at least one terrible vulnerability (CVE-2016-1548)
which took two tries to fix (CVE-2016-4956), and probably indirectly
led to a few others.

Second, the implementation was flawed. "Drivestamps" were collected
simply by calling get_systime() immediately after sendpkt() returned.
However, on modern kernels, send() returns immediately unless the
network buffer is full. So the timestamp that NTP was collecting had
nothing to do with the time the packet actually went out, and was not
any more accurate than the transmit timestamp obtained in basic mode.

If interleaved mode ever provided a timekeeping improvement, there are
two possible explanations for why. One possibility is that the Solaris
boxen that Dave Mills tested it on had a simpler kernel networking
stack, so the timestamp he was collecting was something closer to a
true drivestamp. Another possibility is the presence of a simple bug:
before the recent refactor of receive(), in every mode except
interleaved mode, NTP was storing a transmit timestamp where a receive
timestamp belonged. Interleaved mode may have been improving
performance just by dodging this buggy code.

- - - - -
8d7afd29 by Daniel Fox Franke at 2016-07-07T10:32:53-04:00
Add extra security check for long-delayed responses

- - - - -
2d4764fa by Daniel Fox Franke at 2016-07-07T15:30:24-04:00
Handle manycast and pool mode

- - - - -
16a9997e by Daniel Fox Franke at 2016-07-08T12:28:23-04:00
Merge branch 'master' into proto-refactor

Resolved merge conflicts surrounding removal of xleave from the
parser, a change to the signature newpeer(), and removal of the
is_network_packet() function.

- - - - -
65271c18 by Eric S. Raymond at 2016-07-08T13:26:33-04:00
Remove unused 'group' structure member and freenode flag.

- - - - -
ca1ec22d by Eric S. Raymond at 2016-07-08T13:26:33-04:00
Simplify driver model - remoce buginfo slot.

None of the drivers has one any more.  And the function that used it had
no callsites.  Looks like this was a fossil from ancient times.

- - - - -
b843486c by Eric S. Raymond at 2016-07-08T13:26:33-04:00
Repair compilation without refclocks.

- - - - -
851ada79 by Daniel Fox Franke at 2016-07-08T16:43:23-04:00
Call record_raw_stats()

- - - - -
d9c6983b by Daniel Fox Franke at 2016-07-08T20:06:24-04:00
Fix extension length calculation

This was a nasty vulnerability, fortunately found quickly and before
merging to master.

- - - - -
50b59fdb by Eric S. Raymond at 2016-09-29T12:23:10-04:00
TESTFRAME: Withdraw the TESTFRAME code.

There's an incompatible split between KERNEL_PLL and non-KERNEL_PLL
capture logs - neither can be interpreted by the replay logic that
would work for the other.

Because we can't get rid of KERNEL_PLL without seriously hurting
convergence time, this means the original dream of a single set of
regression tests that can be run everywhere by waf check is dead.
Possibly to be revived if we solve the slow-convergence problem
and drop KERNEL_PLL, but that's far in the future.

Various nasty kludges could be attempted to partly save the concept
by, for example, having two different sets of capture logs.  But, as
the architect of TESTFRAME, I have concluded that this would be
borrowing trouble we don't need - there are strong reasons to suspect
the additional complexity would be a defect attractor.

One proble independent of the KERNEL_PLL/non-KERNEL_PLL split is that
once capture mode was (mostly) working, it became apparent that the
log format is very brittle in the sense that captures would easily be
rendered invalid for replay by even minor logic changes.

Best to fill in this rathole and move on.

- - - - -
21622bce by Eric S. Raymond at 2016-09-29T13:23:37-04:00
Attempted fix for build breakage on a non-KERNEL_PLL sytem.

Goes with the TESTFRAME excision.

- - - - -
783e28e6 by Eric S. Raymond at 2016-09-29T13:40:25-04:00
stoa -> socktoa.  Make the merge less complicated.

- - - - -
e7cc5d35 by Eric S. Raymond at 2016-09-29T14:03:32-04:00
Another try at fixing build breakage reported from Jenkins.

- - - - -
3c07a4d9 by Eric S. Raymond at 2016-09-29T14:53:02-04:00
Back out two build fix attempts, they seem to have made things worse.

- - - - -
fd7df66a by Eric S. Raymond at 2016-09-29T15:43:22-04:00
Once more into the build breakage, dear friends!

- - - - -
52f93c47 by Eric S. Raymond at 2016-09-29T15:56:57-04:00
To avoid modify/delete conflicts, nuke three drivers removed on master.

This will probably breeak building on this branch.

- - - - -
b5963bc1 by Eric S. Raymond at 2016-09-29T16:23:22-04:00
Merge branch 'proto-refactor'

Passes smoke test.  Broadcast and symmetric modes are probably broken and
will need to be fixed or perhaps removed before this will be releasable.

- - - - -
1f41e1b4 by Eric S. Raymond at 2016-09-29T17:04:19-04:00
Update docs and TODO for the protocol refactoring.

- - - - -
e8bcc2a5 by Gary E. Miller at 2016-09-29T15:30:59-07:00
Enable profiling of more options.

Profiling is enabled by -D9.  It only worked when all the graphs, and
html were generated.  By making the profile dump an atexit() the
prilfe now works on all ways you can run ntpviz.

- - - - -
00b10315 by Gary E. Miller at 2016-09-29T15:33:55-07:00
More standard def format for print_profile().

- - - - -
18d14108 by Eric S. Raymond at 2016-09-29T22:19:02-04:00
Remove duplicate code

- - - - -
486da5d7 by Eric S. Raymond at 2016-09-29T22:19:02-04:00
>From microsecond to nanosecond timescale.

- - - - -
747a038f by Eric S. Raymond at 2016-09-29T22:19:02-04:00
Typo fix.

- - - - -
1367fd4b by Eric S. Raymond at 2016-09-29T22:19:02-04:00
Update TODO list.

- - - - -
870c4e8b by Eric S. Raymond at 2016-09-29T22:19:02-04:00
Weaken header inclusions for better contaunment.

ntp_unixtime.h is included by timevalops.h.

- - - - -
0cd7f7ca by Gary E. Miller at 2016-09-29T19:49:24-07:00
Seriously refactor ntpviz.

Moved evertyhing related to a plot into the plot.  Each plot function
now returns it title, percentages, and html descrtipion in addition to
the plot.

This allowed moving the percentage display out of the plot and into
the html.  Easier to cut and paste.  Does not obscure the plot.

- - - - -
7a6bb71d by Gary E. Miller at 2016-09-29T20:03:22-07:00
Breakout deviation in histogram html.

- - - - -
bf9a6a64 by Gary E. Miller at 2016-09-29T20:15:20-07:00
Add some dynamic units to the html.

- - - - -
67902e8b by Eric S. Raymond at 2016-09-29T23:24:59-04:00
Move computation from timevals to timespecs.

- - - - -
54340bfb by Matt Selsky at 2016-09-30T00:18:32-04:00
Typos/grammar

- - - - -
ea61a0a3 by Matt Selsky at 2016-09-30T00:18:32-04:00
Partially revert "Weaken header inclusions for better contaunment."

Fixes:

../../ntpd/ntp_io.c:3241:11: warning: implicit declaration of function 'tval_stamp_to_lfp' is invalid in C99 [-Wimplicit-function-declaration]
                                nts = tval_stamp_to_lfp(*tvp);
                                      ^
../../ntpd/ntp_io.c:3241:9: error: assigning to 'l_fp' from incompatible type 'int'
                                nts = tval_stamp_to_lfp(*tvp);
                                    ^ ~~~~~~~~~~~~~~~~~~~~~~~
1 warning and 1 error generated.

Seen on OSX and NetBSD

- - - - -
3d67f3cc by Eric S. Raymond at 2016-09-30T05:51:05-04:00
Remove obsolete comments about replay.

- - - - -
b2ffb019 by Eric S. Raymond at 2016-09-30T06:49:11-04:00
Remove dead code for unused struct timeval operations.

- - - - -
d2d5278a by Eric S. Raymond at 2016-09-30T07:39:46-04:00
TESTFRAME: Remove some auxiliary code only used by TESTFRAME

- - - - -
d48a0cc6 by Eric S. Raymond at 2016-09-30T08:14:52-04:00
Dead-code removal and information hiding.

- - - - -
2a9f57d9 by Eric S. Raymond at 2016-09-30T08:51:14-04:00
Dead-code removal and information hiding.

- - - - -
222e7558 by Eric S. Raymond at 2016-09-30T10:15:52-04:00
Do more arithmetic at nanosecond scale.

- - - - -
cf409f9d by Gary E. Miller at 2016-09-30T11:06:06-07:00
Try to get Python to consistently output floats.

Sadly Python does not follow the longstanding printf() rules...

- - - - -
c9e0b1ac by Eric S. Raymond at 2016-09-30T14:20:55-04:00
change ntpdig to do its arithmetic using timespecs rather than timevals.

- - - - -
830b8132 by Gary E. Miller at 2016-09-30T16:22:13-07:00
Tweak the ntpviz explanations.

There the data comes from is now always the last item.

- - - - -
d03c6203 by Gary E. Miller at 2016-09-30T17:04:59-07:00
Add mean and sigma values to html where percentiles already are.

I'm not sure the value yet, but some stat freak will be happy.

- - - - -
ba5b9eb4 by Gary E. Miller at 2016-09-30T17:11:49-07:00
Add commas to glassary numbers.

- - - - -
e5ce3193 by Gary E. Miller at 2016-09-30T19:19:20-07:00
Special case percentage==100

- - - - -
601a5bf3 by Gary E. Miller at 2016-09-30T19:58:15-07:00
Move the spread out spread out stats work into class VizStats

Now just one place to tweak autoranging, html presentations.

- - - - -
d1cc3bf2 by Eric S. Raymond at 2016-09-30T23:12:58-04:00
Address GitLab issue #121: compiler warning on osx.

- - - - -
f9bea8a0 by Gary E. Miller at 2016-09-30T20:24:53-07:00
remove unused variables

- - - - -
ff783e17 by Gary E. Miller at 2016-09-30T20:24:53-07:00
remove some more unused vars.

- - - - -
873c81c2 by Eric S. Raymond at 2016-09-30T23:43:46-04:00
Yet another ntpdig warning fix.

- - - - -
2ec6429b by Eric S. Raymond at 2016-09-30T23:51:20-04:00
Remove unused variable.

- - - - -
7a4ab0fb by Eric S. Raymond at 2016-10-01T07:43:55-04:00
Generic driver no longer in-kernel; remove local copies of strlen/strcmp.

- - - - -
74804c62 by Eric S. Raymond at 2016-10-01T08:48:24-04:00
Change rawdcf clock to use timespec rather than timeval internally.

- - - - -
ea1ed04c by Eric S. Raymond at 2016-10-01T09:16:41-04:00
Removal of timeval operations.

Alas, some of the simpler macros had to be copied into ntptime and the
generic driver to accomplish this. Cleanup will involve fully resolving
these into timespec operations.

- - - - -
85da7698 by Eric S. Raymond at 2016-10-01T10:05:39-04:00
More struct timeval elimination.

- - - - -
4d8d6c87 by Eric S. Raymond at 2016-10-01T10:43:35-04:00
struct timeval elimination.

- - - - -
758b3b53 by Eric S. Raymond at 2016-10-01T10:48:11-04:00
struct timeval elimnination (from GPSD refclock driver).

- - - - -
42e2211d by Eric S. Raymond at 2016-10-01T11:09:41-04:00
A note on avoiding struct timeval.

- - - - -
f96d85a3 by Eric S. Raymond at 2016-10-01T17:28:48-04:00
Much-needed clarification of the system clock API.

- - - - -


30 changed files:

- devel/TODO
- devel/hacking.txt
- devel/tour.txt
- docs/decode.txt
- docs/includes/assoc-commands.txt
- docs/includes/assoc-options.txt
- docs/includes/ntpq-body.txt
- docs/includes/ntpviz-body.txt
- docs/includes/special.txt
- docs/index.txt
- docs/warp.txt
- − docs/xleave.txt
- include/ntp.h
- include/ntp_calendar.h
- + include/ntp_endian.h
- include/ntp_fp.h
- include/ntp_machine.h
- include/ntp_syscall.h
- − include/ntp_unixtime.h
- include/parse.h
- include/timespecops.h
- include/timetoa.h
- − include/timevalops.h
- libntp/clocktime.c
- libntp/humandate.c
- libntp/machines.c
- libntp/ntp_calendar.c
- + libntp/ntp_endian.c
- libntp/ntp_intres.c
- libntp/ntp_worker.c


The diff was not included because it is too large.


View it on GitLab: https://gitlab.com/NTPsec/ntpsec/compare/e3400e466e49bcea31fae24ca4358f1617e5da65...f96d85a3e781877ee26fdc0e8c067f6443b0c40b
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ntpsec.org/pipermail/vc/attachments/20161002/923ed749/attachment.html>

More information about the vc mailing list