Updated to-do list?
halmurray at sonic.net
Mon Apr 10 00:32:15 UTC 2023
[Context is a multi-threaded Go echo server]
> I can manage to do that.
I'll send you the C code off list.
Here is an outline of the big picture:
Linux and FreeBSD have a SO_REUSEPORT option. The idea is that you can open
several sockets on the same port number. The kernel will hash on source IP
address and source port. (so packets for a connection go to the same thread
-- the thread can save state for a connection)
The main server thread sets up worker threads
Each worker thread opens a socket
The worker threads bump counters.
The main thread will read/collect and print the info.
The worker threads have an option to spin for N microseconds between recvfrom
and sendto. The idea is to simulate crypto work.
The limiting factor on packet throughput is the kernel thread
That thread puts arriving packets on the right socket queue
and wakes up a waiting thread if there is one
So larger packets reduce the load on the kernel thread for a given traffic
load in bits/second (rather than packets/second).
The CPU load for a NTP server is roughly constant in terms of bits/second.
So you want to assign the kernel thread to one core (leaving the other
hyperthread idle) and disable interrupt coalescing.
Then fill up the rest of the CPUs on the chip with worker threads.
It's roughly a microsecond per packet for the kernel thread,
a microsecond each for recvfrom and sendto
and (handwave) a microsecond for basic NTP server work.
Up a level...
There is a companion client side that runs several worker threads sending to
You can run it on multiple old/slow PCs to get enough traffic to saturate the
There is an option to send N extra packets, like ping -l. I call them "in
Things are setup assuming that no packets are lost. So you have to be sane on
picking combinations of multiple clients and extra packets in flight.
Up another level...
I have some hackish scripts that
fire up a server
fire up several clients
read/reset the server stats
wait a bit
read server stats
loop for various CPU load/delays or packet lengths
These are my opinions. I hate spam.
More information about the devel