lsof is a great tool for debugging applications running on linux.
lsof is short for “list open files”. As the name suggests, the tool lists and searches for open files
on a linux host. Because everything is a file,
lsof can not only be used see who is touching what file, but can also be used to get a quick view of
what the network is doing.
Here are some useful snippets I find myself using frequently are some useful snippets I find myself using frequently.
This is not a comprehensive guide to
What applications are listening on a port?
You can find out what applications are listening on a port with
-i. Running this on my laptop, I can find out that
nginx is listening on port 80.
$ sudo lsof -i :80 -s TCP:LISTEN nginx 1305 nginx 6u IPv4 1613873 0t0 TCP *:http (LISTEN) nginx 1305 nginx 7u IPv6 1613874 0t0 TCP *:http (LISTEN) nginx 1306 nginx 6u IPv4 1613873 0t0 TCP *:http (LISTEN) nginx 1306 nginx 7u IPv6 1613874 0t0 TCP *:http (LISTEN) ...
what ports are being listened to by an application?
You can also go the other way. Say you know a host is running some application, but you’ve forgotten which port it’s listening on. First get the pid of the application, then use
$ sudo lsof -p 1305 | grep LISTEN nginx 1305 nginx 6u IPv4 1613873 0t0 TCP *:http (LISTEN) nginx 1305 nginx 7u IPv6 1613874 0t0 TCP *:http (LISTEN)
What logs are my applications writing to?
Sometimes you need to debug an application, but you don’t know where the logs are.
You can list all files open by a process. Often grepping for
log will reveal where a process is logging to:
nginx 1305 nginx 2w REG 253,0 606 2490627 /var/log/nginx/error.log nginx 1305 nginx 4w REG 253,0 606 2490627 /var/log/nginx/error.log nginx 1305 nginx 5w REG 253,0 833 2491029 /var/log/nginx/access.log
Note that this only works when the application is writing directly to log files itself. It’s more common for applications to be configured to log to stdout/stderr and delegate the log file management elsewhere. In those cases, these commands won’t work.
What network connections are my applications making?
This is helpful to verify that an application is making the right connections to other hosts. This is often helpful with services that talking to eachother, but I decided to see what my local Google Chrome was connecting to. This is only a subset of what I found, but you can see connections over IPv4 to Github and AWS.
$ sudo lsof -p 3129 | grep TCP chrome 3129 leeavital 58u IPv4 2168005 0t0 TCP localhost.localdomain:42752->lb-192-30-253-124-iad.github.com:https (ESTABLISHED) chrome 3129 leeavital 59u IPv4 2165690 0t0 TCP localhost.localdomain:42020->ec2-52-55-142-125.compute-1.amazonaws.com:https (ESTABLISHED) ...
Note here that
lsof is doing some rewriting of hostnames and ports to
make the output easier to read. For example,
lb-192-30-253-124-iad.github.com instead of the IP address, and
instead of port
443. Sometimes this helpful rewriting is slow or
wrong. It can be disabled with
$ sudo lsof -p 3129 -nP | grep TCP chrome 3129 leeavital 58u IPv4 2168005 0t0 TCP 192.168.0.165:42752->126.96.36.199:443 (ESTABLISHED) chrome 3129 leeavital 59u IPv4 2165690 0t0 TCP 192.168.0.165:42020->188.8.131.52:443 (ESTABLISHED)