Using Heaptrack to Find Memory Leaks

[Jason Azze]

I mentioned on the IRC channel that I was playing with a tool called Heaptrack, which I read about on Hacker News, to search for why the memory footprint of ntpd is (is it?) larger than we expect. I found a leak that ESR said looked "pretty nasty." Because I have no idea what I'm doing with debugging symbols, profilers, etc. -- I'll write down the steps to reproduce the report I'm looking at.

Here's the Heaptrack release announcement https://www.kdab.com/heaptrack-v1-1-0-release/
Here's the source code https://github.com/KDE/heaptrack

I grabbed the Appimage from one of these mirrors https://download.kde.org/stable/heaptrack/1.1.0/heaptrack-v1.1.0-x86_64.AppImage.mirrorlist and saved it to my Ubuntu 16.04 box, which is named tin. 


# I got a fresh clone of NTPsec.
~/code $ git clone https://gitlab.com/NTPsec/ntpsec.git
~/code $ cd ntpsec
# Configure for a build with debugging symbols
[master]:~/code/ntpsec $ ./waf configure --enable-debug-gdb
[master]:~/code/ntpsec $ ./waf build
[master]:~/code/ntpsec $ sudo ./waf install
# Or, if you are GEM, do this as root.

Now become root in your favorite way. I didn't have luck with Heaptrack using sudo. (See, GEM is right.)
#Make sure ntpd isn't running.
# for systemd (my case)
root at tin:~# systemctl stop ntp
root at tin:~# ps -ef |grep [n]tpd
# Looks good
# Now start ntpd wit the Heapcheck Appimage. The -n (no fork) is important, otherwise I think you only catch ntpd dropping root.
root at tin:~# /home/jazze/installers/heaptrack-v1.1.0-x86_64.AppImage /usr/sbin/ntpd -p /var/run/ntpd.pid -g -n -c /etc/ntp.conf -u 121:130 
# Note, id 121:130 is ntp:ntp on my machine. ymmv

Let ntpd cook for as long as needed. I don't know how long that is. 

In another shell, find the ntpd PID and kill it.

root at tin:~# ps -ef |grep [n]tpd
root      3227 28886  0 14:02 pts/21   00:00:00 /bin/bash /tmp/.mount_heaptrUKPaLq/AppRun /usr/sbin/ntpd -p /var/run/ntpd.pid -g -n -c /etc/ntp.conf -u 121:130
root      3229  4024  0 14:02 ?        00:00:00 /home/jazze/installers/heaptrack-v1.1.0-x86_64.AppImage /usr/sbin/ntpd -p /var/run/ntpd.pid -g -n -c /etc/ntp.conf -u 121:130
root      3234  3227  0 14:02 pts/21   00:00:00 /bin/sh /tmp/.mount_heaptrUKPaLq//opt/bin/heaptrack /usr/sbin/ntpd -p /var/run/ntpd.pid -g -n -c /etc/ntp.conf -u 121:130
ntp       3249  3234  2 14:02 pts/21   00:00:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -n -c /etc/ntp.conf -u 121:130
root at tin:~# kill 3249

Now in the shell where you launched Heaptrack, you should see something like:
heaptrack stats:
	allocations:          	798
	leaked allocations:   	170
	temporary allocations:	127
Heaptrack finished! Now run the following to investigate the data:

  heaptrack --analyze "/root/heaptrack.ntpd.3234.zst"

The advice for launching the analyzer isn't perfect, because we're using the Appimage. Instead do:

root at tin:~# /home/jazze/installers/heaptrack-v1.1.0-x86_64.AppImage --analyze "/root/heaptrack.ntpd.3234.zst" 

You should get a GUI report. I believe Heaptrack has some CLI-only reports, which you may find more tasteful, but I can't figure out how to get heaptrack_print. Maybe if you build from source.

Best,
Jason