Tracking TCP lifespans using eBPF (2016)

Summary

"i really wish i had a command line tool that would give me stats on TCP connection lengths on a given port" # ./tcplife -D 80 PID COMM LADDR LPORT RADDR RPORT TX_KB RX_KB MS 27448 curl 100.66.11.247 54146 54.154.224.174 80 0 1 263.85 27450 curl 100.66.11.247 20618 54.154.164.22 80 0 1 243.62 27452 curl 100.66.11.247 11480 54.154.43.103 80 0 1 231.16 27454 curl 100.66.11.247 31382 54.154.15.7 80 0 1 249.95 27456 curl 100.66.11.247 33416 52.210.59.223 80 0 1 545.72 27458 curl 100.66.11.247 16406 52.30.140.35 80 0 1 222.29 27460 curl 100.66.11.247 11634 52.30.133.135 80 0 1 217.52 27462 curl 100.66.11.247 25660 52.30.126.182 80 0 1 250.81 [...] That's tracing destination port 80, you can also trace local port (-L), or trace all ports (default behavior). The quote and good idea is from Julia Evans on twitter, and I added it as a tool to the Linux BPF-based bcc open source collection. The output of tcplife, short for TCP lifespan, shows not just the duration (MS == milliseconds) but also throughput statistics: TX_KB for Kbytes transmitted, and RX_KB for Kbytes received. It should be useful for performance and security analysis, and network debugging. I am NOT tracing packets The overheads can cost too much to examine every packet, especially on servers that process millions of packets per second. To do this I'm not using tcpdump, libpcap, tethereal, or any network sniffer. So how'd I do it? There were four challenges: 1. Measuring lifespans The current version of tcplife uses kernel dynamic tracing (kprobes) of tcp_set_state(), and looks for the duration from an early state (eg, TCP_ESTABLISHED) to TCP_CLOSE. State changes have a much lower frequency than packets, so this approach greatly reduces overhead. But it ends up tricker than it sounds, since it's tracing the Linux implementation of TCP, which isn't guaranteed to use tcp_set_state() for every state transition. But it works well enough for now, and we can add a stable tracepoint for TCP state transitions to futur...