Nat? Nat. Nat!

#581
Tuna Please!

Note: Please read the updates as well, as the first part of this article has bugs.

So I had a moment a few weeks ago at work where we were trying to decide if a service was actually running on a machine. We had gotten to the point where we had actually ssh'd into the production machine, and were poking around trying to figure out what was up.

My coworker Fisher was driving, and I suggested he run nmap on the box to see what ports were open. He ran it, but then as soon as the results came back, he started running another command. As he typed, he said, quite loudly, "Tuna Please!", and then ran the command sudo netstat tunapl.

It outputs something like the following (truncated for brevity):

$ sudo netstat tunapl
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 192.168.1.18:ssh        125.88.177.98:30335     ESTABLISHED
tcp        0    160 192.168.1.18:ssh        cpe-72-231-30-23.:54860 ESTABLISHED
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ]         DGRAM                    4002     /run/systemd/notify
unix  2      [ ]         DGRAM                    9127     /run/user/109/systemd/notify
unix  2      [ ]         DGRAM                    4020     /run/systemd/shutdownd
unix  11     [ ]         DGRAM                    4022     /run/systemd/journal/dev-log
unix  2      [ ]         DGRAM                    1890496  /run/user/1001/systemd/notify
unix  7      [ ]         DGRAM                    4032     /run/systemd/journal/socket
unix  2      [ ]         DGRAM                    4571     /run/systemd/journal/syslog
...

I had never seen this combo before, but it was an outage, so I completely forgot about it.

Then last week, I was like "oh yeah, Fisher used tuna please to figure this out". So I ran the command sudo netstat tunaplz. It gave me the data I wanted, so I decided to go read the man page for the flags to see what it was doing. The following is what I gleaned from that page:

--tcp, -t TCP sockets

--udp, -u UDP sockets

Both make sense, I want both types of connections.

--numeric, -n Show numerical addresses instead of trying to determine symbolic host, port or user names.

I guess that's helpful.

-a, --all Show both listening and non-listening sockets.

Cool, I want all the things.

--program, -p Show the PID and name of the program to which each socket belongs.

Yeah, this is what I wanted to know in the first place!

--listening, -l Only show listening sockets.

Wait, why this and -a? Oh well.

The great part about this, and the amazing discovery, is that -z, AFAICT, is not a valid flag, but netstat just ignores it, instead of complaining!

So, if you're on a Linux box, and you wanna know about some sockets, just ask your computer for some Tuna Please!

sudo netstat tunaplz !!!

Update: As @saltyhorse pointed out, this is all dependent on versions and operating systems and what not. This has been tested in a few places, but specifically was written about a Debian install on a Raspberry Pi B+.

$ netstat --version
net-tools 1.60
netstat 1.42 (2001-04-15)
Fred Baumgarten, Alan Cox, Bernd Eckenfels, Phil Blundell, Tuan Hoang and others
+NEW_ADDRT +RTF_IRTT +RTF_REJECT +FW_MASQUERADE +I18N
AF: (inet) +UNIX +INET +INET6 +IPX +AX25 +NETROM +X25 +ATALK +ECONET +ROSE
HW:  +ETHER +ARC +SLIP +PPP +TUNNEL -TR +AX25 +NETROM +X25 +FR +ROSE +ASH +SIT +FDDI +HIPPI +HDLC/LAPB +EUI64

$ uname -a
Linux newyork 4.1.13+ #826 PREEMPT Fri Nov 13 20:13:22 GMT 2015 armv6l GNU/Linux

Update Number 2: So now I feel like a fool. Something seemed off when I was testing this late at night, and of course this article has been shared by an insane number of people, but @saltyhorse has shown me why -z was being ignored!

The thing that I was confused about is why programs were only showing up for domain sockets (they weren't even programs, they were paths), but it was late at night, so #shrugs. Now that I'm more awake, and understanding what's going on, you need to put a - in front of your args, and then -z isn't ignored. Which makes much more sense. So much more sense.

So here is the correct output and call.

$ sudo netstat -tunapl
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      508/nginx -g daemon
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      414/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      765/exim4
tcp        0     72 192.168.1.18:22         72.231.30.23:64218      ESTABLISHED 14189/sshd: nat [pr
tcp        0      0 192.168.1.18:22         183.3.202.111:25211     ESTABLISHED 15189/sshd: root [p
tcp6       0      0 :::80                   :::*                    LISTEN      508/nginx -g daemon
tcp6       0      0 :::22                   :::*                    LISTEN      414/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      765/exim4
udp        0      0 0.0.0.0:42428           0.0.0.0:*                           331/avahi-daemon: r
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           331/avahi-daemon: r
udp        0      0 0.0.0.0:68              0.0.0.0:*                           339/dhcpcd
udp        0      0 192.168.1.18:123        0.0.0.0:*                           451/ntpd
udp        0      0 127.0.0.1:123           0.0.0.0:*                           451/ntpd
udp        0      0 0.0.0.0:123             0.0.0.0:*                           451/ntpd
udp6       0      0 :::5353                 :::*                                331/avahi-daemon: r
udp6       0      0 :::42257                :::*                                331/avahi-daemon: r
udp6       0      0 fe80::8b6:728b:e11::123 :::*                                451/ntpd
udp6       0      0 ::1:123                 :::*                                451/ntpd
udp6       0      0 :::123                  :::*                                451/ntpd

Thanks to @saltyhorse for showing me the error of my ways. I'm saddened that the mnemonic is now less funny (no -z), but overall still works.

Update Three: This is why journalists fact check their articles. Fisher has informed me that this mnemonic actually came from @timball. And he, "like Zoidberg, want the credit."

Cheers!

/Nat