Hey there! In this instance of The Admins Toolbox, I’ll be discussing the tcpdump utility and some common uses. Sit back, crack open a frosty beverage of your choosing, and let’s learn about packet inspection for fun and profit. ๐
Tcpdump is by no means a new program. Written by Van Jacobson, Craig Leres, and Steven McCanne, while working at Berkley in 1987, tcpdumpโs popularity quickly rose due to its high caliber of features. Soon after itโs release, it was ported to just about every *nix flavor, taking off like wildfire.
Tcpdump Basics.
At it’s core, tcpdump is a wrapper for libpcap, which allows for full packet analysis of all traffic through the network stack of the interface you’re inspecting. The tcpdump app uses Berkeley filtering syntax for realtime packet filtering with extremely fine grained control, making it a “Swiss Army Knife” for server and network admins alike. Tcpdump allows for a proper view of network traffic that would be nearly impossible otherwise. A few examples of its use could be for detecting a (D)DoS attack or verifying an end-user really is entering the POP3 password they say they are.
Here’s a quick and dirty scenario you might see in your daily routine, and how tcpdump could save the day. ย Please keep in mind that encrypted protocols will not allow sniffing of this sorts, sorry!
Your ever faithful customer, Ron, is having email problems again. The cPanel admin for Ron’s company, Widgets.com, updated the passwords for every employee’s email address. Well, today’s your lucky day. Ron’s admin is out of town and gave everyone at Widgets.com your number “just in case”. You get this phone call…
You: “Thank you for calling Super Awesome Hosting Company, how may I help you today?”
Ron: “Hey there, it’s Ron again! I’ve got a tough one for you this time!”
You: “What are you having trouble with Ron?”
Ron: “So here’s the story, I’m putting my new password into my Outlook and your server is telling me it’s not right. You should be able to change this, right?”
You: “Well, sure thing, let me go ahead and generate a new password, then you should be good to go; one second please.”
<furious typing>
<3 seconds later>
You: “Alright Ron, your new password is going to be abc123, lowercase letters. You should be set now, please let me know if you run into any more problems!”
Ron: “Thanks again, you’re awesome!”
3 minutes pass…. RING.
You: “Thank you for calling Super Awesome Hosting Company, how may I help you today?”
Ron: “It’s me again!”
You: “Hey Ron, still having problems?”
Ron: “I put that password in JUST like you told me to and it’s not working either… Have you tried rebooting the server yet?”
You: “Well before we do that, do you mind if I check something else quick? When I tell you, would you mind clicking “Send and Receive” in Outlook?”
Ron: “Right on!”
Now, this is where the magic of tcpdump comes into play. You login to Ron’s server. Armed with the knowledge that Ron is using POP3 to retrieve his mail, you start checking out payloads on TCP port 110 and give Ron the go-ahead.
root@cpanelvps [~]# tcpdump -nn -i eth0 port 110
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
19:49:10.488790 IP 10.11.12.13.15263 > 172.16.134.69.110: S 2260515384:2260515384(0) win 64240 <mss 1460,nop,nop,sackOK>
19:49:10.488838 IP 172.16.134.69.110 > 10.11.12.13.15263: S 2161496985:2161496985(0) ack 2260515385 win 5840 <mss 1460,nop,nop,sackOK>
19:49:10.522786 IP 10.11.12.13.15263 > 172.16.134.69.110: . ack 1 win 64240
19:49:10.523022 IP 172.16.134.69.110 > 10.11.12.13.15263: P 1:21(20) ack 1 win 5840
19:49:10.556756 IP 10.11.12.13.15263 > 172.16.134.69.110: P 1:26(25) ack 21 win 64220
19:49:10.556770 IP 172.16.134.69.110 > 10.11.12.13.15263: . ack 26 win 5840
19:49:10.556862 IP 172.16.134.69.110 > 10.11.12.13.15263: P 21:26(5) ack 26 win 5840
19:49:10.590734 IP 10.11.12.13.15263 > 172.16.134.69.110: P 26:39(13) ack 26 win 64215
19:49:10.631034 IP 172.16.134.69.110 > 10.11.12.13.15263: . ack 39 win 5840
19:49:12.186189 IP 172.16.134.69.110 > 10.11.12.13.15263: P 26:55(29) ack 39 win 5840
19:49:12.220749 IP 10.11.12.13.15263 > 172.16.134.69.110: F 39:39(0) ack 55 win 64186
19:49:12.259941 IP 172.16.134.69.110 > 10.11.12.13.15263: . ack 40 win 5840
Hrm, that wasn’t very helpful there… What did we just see? Let’s start with the tcpdump commandline first and see what we get. We’re calling tcpdump with the ‘-nn’ flag, which tells tcpdump not to do reverse DNS lookups on all the IPs it picks up; also not to resolve the port names to service names. Without the ‘-nn’ flag, you would see something like this…
19:50:19.140525 IP office01.widgets.com.31773 > server18.superawesomehosting.com.pop3: . ack 1 win 64240
19:50:19.140648 IP server18.superawesomehosting.com.pop3 > office01.widgets.com.31773: P 1:21(20) ack 1 win 5840
19:50:19.174495 IP office01.widgets.com.31773 > server18.superawesomehosting.com.pop3: P 1:26(25) ack 21 win 64220
It could be an expensive time waster to have this additional information given over a high packet per second interface. This could leave the customer waiting for what seems to them like an eternity, in turn putting more pressure on you to quickly solve their issue.
Next up is the ‘-i’ flag, where we tell tcpdump which interface to attach to, in this case, eth0. Next we specify which port we want to focus on, in our example, port 110 for the POP3 protocol. If we don’t specify which port we’re using, your screen is going to be flooded with your SSH transactions alone. This is also our first glimpse of the BPF mechanism in play.
After looking over our first run with tcpdump, we decide we didn’t get the information needed, as tcpdump by default gives a very slim overview of the traffic. We need to dig a bit deeper here, using the ‘-A’ flag.
The ‘-A’ flag is fun for folks like us. In it’s default invocation, it’ll print the first 96 bytes of a packet in ASCII format. Yay, human readability! In this instance, using the ‘-A’ flag will be the key we need to see what’s going on with Ron, who’s patiently waiting.
root@cpanelvps [~]# tcpdump -nn -i eth0 port 110 -A
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
19:54:12.405746 IP 10.11.12.13.12660 > 172.16.134.69.110: S 3147514733:3147514733(0) win 64240 <mss 1460,nop,nop,sackOK>
E..0 e@.s..P.JyfE.u.1t.n..Cm....p...R1..........
19:54:12.405787 IP 172.16.134.69.110 > 10.11.12.13.12660: S 2480457399:2480457399(0) ack 3147514734 win 5840 <mss 1460,nop,nop,sackOK>
E..0..@.@.5.E.u..Jyf.n1t......Cnp...............
19:54:12.439721 IP 10.11.12.13.12660 > 172.16.134.69.110: . ack 1 win 64240
E..( f@.s..W.JyfE.u.1t.n..Cn....P...$T..
19:54:12.439859 IP 172.16.134.69.110 > 10.11.12.13.12660: P 1:21(20) ack 1 win 5840
E..<..@.@.<.E.u..Jyf.n1t......CnP.......+OK Dovecot ready.
19:54:12.474717 IP 10.11.12.13.12660 > 172.16.134.69.110: P 1:26(25) ack 21 win 64220
E..A g@.s..=.JyfE.u.1t.n..Cn....P....#..USER ron.c@widgets.com
19:54:12.474725 IP 172.16.134.69.110 > 10.11.12.13.12660: . ack 26 win 5840
@.@.<.E.u..Jyf.n1t......C.P....H..
19:54:12.474790 IP 172.16.134.69.110 > 10.11.12.13.12660: P 21:26(5) ack 26 win 5840
E..-..@.@.<.E.u..Jyf.n1t......C.P.......+OK
19:54:12.509666 IP 10.11.12.13.12660 > 172.16.134.69.110: P 26:39(13) ack 26 win 64215
E..5 h@.s..H.JyfE.u.1t.n..C.....P.......PASS ABC123
19:54:12.549339 IP 172.16.134.69.110 > 10.11.12.13.12660: . ack 39 win 5840
E..(..@.@.<.E.u..Jyf.n1t......C.P....6..
19:54:14.107478 IP 172.16.134.69.110 > 10.11.12.13.12660: P 26:55(29) ack 39 win 5840
E..E..@.@.<.E.u..Jyf.n1t......C.P....C..-ERR Authentication failed.
One thing you’ll be able to pick up is how to parse the packets more effectively. As you may have read above, the output has the answer we were looking for to help Ron!
Let me clean the output up a bit for you.
19:57:12.668150 IP 172.16.134.69.110 > 10.11.12.13.20989: P 1:21(20) ack 1 win 5840
E..<.y@.@.O0E.u..Jyf.nQ..:....J.P.......+OK Dovecot ready.
This is one packet, but not necessarily in it’s entirety (tcp packets can be upwards of 65k apiece; we’re only peeking at 96 bytes by default). The first line represents the two hosts having the conversation, the ports they’re talking on, and who sent the packet; essentially the header.
19:54:12.549339 IP 172.16.134.69.110 > 10.11.12.13.12660: . ack 39 win 5840
^---1 ^-2 ^-3 ^-4 ^-5 ^-6 ^-7 ^-8 ^-9 ^-10
- This is the timestamp when tcpdump received the packet, according to the system clock.
- The protocol, in this case, IP
- The source IP (SRC)
- The source port
- Which direction the traffic went (from host > to host)
- Our destination IP (DST)
- Our destination port
- Piggybacked ACK for the prior SYN
- Packet Byte count
- Window size
E..<.y@.@.O0E.u..Jyf.nQ..:....J.P.......+OK Dovecot ready.
In this part of the packet there’s a bit of padding on the front, then voila, our POP3 server response telling the customer, “I’m here, go for it!”
So, looking at the initial capture, cleaned up a bit with the aforementioned in mind, here’s what we have…
Server: +OK Dovecot ready
Client: USER ron.c@widgets.com
Server: +OK
Client: PASS ABC123
Server: -ERR Authentication failed.
You: “Hey Ron, are you there?”
Ron: “Sure am, are you rebooting the server now?”
You: “No Ron, but I did find the reason we’re having trouble. Can you do me a favor and turn your capslock off and enter that password again?”
Ron: “Is that necessary? I wanted my password to be REALLY secret…”
You: “Yes Ron, Yes… Do that and try it again”
Ron: “Oh hey, it works! Are you sure you didn’t reboot the server?”
So there you go; an easy to use basic usage of tcpdump you might see in your daily routine. We could’ve figured out Ron had a bad password from /var/log/maillog, however, it wouldn’t have shown us how it was bad. Given Ron’s frequency of calls into the helpdesk, tcpdump saved time by finding the specifics right away, and you never know, Ron may have learned a little something about passwords in the process.
Next up, we’ll be talking about more advanced usages of tcpdump, digging a little deeper into The Admins Toolbox.
