Ever tried to run a server and find it won't start because some other program is already using the same port number to listen on?

In my case, I was installing a new package onto a Debian machine. For some reason, the new service didn't seem to be working. Checking the log of the newly installed server revealed a clue:

Cannot create listening socket on port 32101 - socket address already in use

Hmmm. To track down the problem, we need to find out what process is already listening on port 32101.

netstat can be used to give us the PID and program name associated with all listening sockets:

$ netstat -lnp -A inet

gives something like:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:32101           0.0.0.0:*               LISTEN     2759/TestService
...

Odd, TestService is the program I'm trying to run and I thought the previous version would have been uninstalled when installing the new package. Let's check the full path to the program using /proc with the PID given in the netstat output above.

$ ls -l /proc/2759/exe

lrwxrwxrwx  1 daemon daemon 0 2007-05-02 10:13 /proc/2759/exe -> /usr/local/TestService.v1/TestService (deleted)

Aha. So the problem isn't with the new version of TestService. Rather, the old package has a problem where uninstall does not kill the service. Bug raised.