How To Identify The Bad Processes On A Hacked Linux Box

### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
How To Identify The Bad Processes On A Hacked Linux Box
Mattias Geniar, Friday, August 8, 2008 - last modified: Saturday, August 8, 2015
### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###

Let's say you have a hacked Linux server. Let's say you want to find out who did it, as well as what the server is doing now. It's not always easy, but this guide will show you how to use tools like netstat and lsof to find which processes are running on which ports.

First of all, if you still have SSH access to the server, log in through a remote session. Otherwise, you'll need physical access to the server.

Most hacked servers have only 2 primary functions: try to hack other servers, in order to create a bigger botnet, or to wreck havoc on an unsuspecting server. They do have one thing in common: several outbound connections are made to other servers (portscan, irc-bot, ftp server, ...). We'll identify the ports used for those outbound connections, and the user/process that is executing them. Then we'll locate the files for those processes, and disable them (after analyzing them).

Note; this article turned out a bit larger than I had thought, so prepare for a wall of text, with the occasional comic -- to keep you entertained ;-)

Holy jokers Batman, how do we do that?!
So at first, a little warning. If you're trying to fix a hacked server, it's very possible that the binaries you're executing (such as ps, ls, lsof, netstat, ...) have been replaced by hacked versions, meaning they won't show you the full information.

Be cautious with trusting the information they show you, and try using the BackTrack live CD (if you have physical access to do so) to perform your analysis.

First we need to find out what the server is doing. Seeing as a hacked server is pretty useless if it isn't making contact with other servers, we can do this by checking all our incoming/outgoing connections. This can help us find out which ports are being used, and what the server's doing.

### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
To get a list of all current active connections, you can use the netstat command.

$ netstat -an
With the -a parameter, we tell the command to show all connections (including the LISTENING ones), instead of only those who are connected.

The -n parameter shows the different port numbers used. By default, it would display items with a textual description (such as :pop3, :smtp, :http). It's more useful when we can see the actual ports, shown as :110, :25, :80.

This will generate a list, much like this one (if it's a standard webserver, it could differ for your server(s) ). I've shortened it a bit, it's usually a lot longer.

$ netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address     Foreign Address     State
tcp        0      0 0.0.0.0:110       0.0.0.0:*           LISTEN
tcp        0      0 0.0.0.0:143       0.0.0.0:*           LISTEN
tcp        0      0 0.0.0.0:21        0.0.0.0:*           LISTEN
tcp        0      0 0.0.0.0:25        0.0.0.0:*           LISTEN
tcp        0      0 :::80             :::*                LISTEN
tcp        0      0 :::22             :::*                LISTEN
tcp        0      0 :::443            :::*                LISTEN

Active UNIX domain sockets (servers and established)
Proto Flags     Type      State        I-Node Path
unix  [ ACC ]   STREAM    LISTENING    548928452 /var/lib/mysql/mysql.sock
unix  [ ]       DGRAM                  548927555 @/org/kernel/udev/udevd
unix  [ ACC ]   STREAM    LISTENING    548928520 /tmp/spamd_full.sock
unix  [ ]       DGRAM                  548927910 /dev/log
unix  [ ACC ]   STREAM    LISTENING    548928786 /var/run/saslauthd/mux
unix  [ ]       DGRAM                  3285180947
unix  [ ]       DGRAM                  550618919
unix  [ ]       DGRAM                  550618862
unix  [ ]       DGRAM                  548928785
unix  [ ]       DGRAM                  548928764
unix  [ ]       STREAM    CONNECTED    548928535
unix  [ ]       STREAM    CONNECTED    548928534
You'll probably notice that these connections all look harmless. There are processes listening on port 80 (HTTP), 110 (POP3), 21 (FTP), 25 (SMTP), 22 (SSH) and 443 (HTTPS). For this example, we'll assume we want to find out which process is listening on port 80.

Note; if your server is hacked and being used as an IRC Bot you'll see many connections in the range of 6660-7000. You could follow the same steps, by just replacing port 80 by port 6660. It's just easier to show it this way.

### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
XKCD - Hacked, SQL Injection

Now, to find out which user and which process is making connections on this port, we'll use the lsof (List Open Files) command.

$ lsof -i tcp:80 -P -R
This command shows us all running processes that are using port 80 for any kind of communication.

The -i parameter specifies we want to list the processes, by identifying them with IPv4 or IPv6.
The tcp:80 part means we only want to show TCP connections (and ignore UDP for the time being) using port 80.
Using -P we specify we want to see the port numbers (80, 21, ...) instead of the names (HTTP, FTP) which are shown by default. Much like the -n parameter with netstat.
With -R we also show the Parent Process ID, to see who initiated this process.
It will show output like this.

$ lsof -i tcp:80 -P -R
COMMAND   PID  PPID   USER   FD   TYPE    DEVICE SIZE NODE NAME
httpd   13538 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   14279 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   16324     1   root    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   28042 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   29772 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   32424 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
Wow! Lots of numbers!
Take a deep breath. It looks more complicated than it is, trust me. What can we learn from this? We see 6 instances of httpd, the HTTP daemon that is acting as the webserver. Five of these processes are run by the apache user, and one by root.

All apache httpd processes were started by a main process (PID 16324) which was executed by the root user (PPID column).

The column "command" displays the name of the process. Under "PID" we can see the Process ID, this can be used to easily stop, or kill the process. With "PPID" we can see the process ID that started this specific process.

Beneath "USER" we see the user that is executing the process. We can now see that all httpd processes that are being run as the apache user, were started by Parent Process ID 16324, the httpd process executed as root.

Note; you should be alerted when you see a postgres user running all sorts of programs on all sorts of exotic ports. If that's the case, the server's probably hacked, and used for means it wasn't supposed to.

Now you have a list of all the processes that are using a specific port to communicate. Let's find out which files these processes use, where they are located, and then stop the process -- to prevent further harm. We won't delete the files that were used just yet, because we might learn something afterwards when we examine them. We will, however, move them outside the user's directory.

$ lsof -p 16324
With -p we can specify a Process ID (PID) to filter our results to. This'll only list open files for the PID 16324.

$ lsof -p 16324
COMMAND   PID USER   FD   TYPE    DEVICE    SIZE       NODE NAME
httpd   16324 root  cwd    DIR      0,61    4096   14624004 /
httpd   16324 root  rtd    DIR      0,61    4096   14624004 /
httpd   16324 root  txt    REG      0,61  312372   19768930 /usr/sbin/httpd
httpd   16324 root  mem    REG      0,61   11704   20455174 /usr/lib/libgpg-error.so.0.3.0
httpd   16324 root  mem    REG      0,61  346148   20455190 /usr/lib/libgcrypt.so.11.2.2
httpd   16324 root  mem    REG      0,61  213116   20455786 /usr/lib/libxslt.so.1.1.17
httpd   16324 root  mem    REG      0,61   67932   20455784 /usr/lib/libexslt.so.0.8.13
httpd   16324 root  mem    REG      0,61   23172   20513414 /usr/lib/php/modules/xsl.so
httpd   16324 root  mem    REG      0,61   32216   20513412 /usr/lib/php/modules/xmlwriter.so
....
This list usually goes on for a while, as any given process could have hundreds of files open. However, this list will tell you where the culprit is located. In our case, the program started in /usr/sbin/httpd (see line 3).

Let's end the process using kill, remove the execute permissions, and move the files to another location.

$ kill 16324
The kill command ends a process with a given PID, in this case 16324.

### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
$ chmod -x /usr/sbin/httpd
The chmod command lets you edit the permissions to this file. We'll use it to remove any execution rights it might have "-x".

$ mkdir /home/tmp-sandbox/
 mv /usr/sbin/httpd /home/tmp-sandbox/
We'll create the directory "/home/tmp-sandbox/" and copy the files there, so we can view them later without risk.

Ok Batman, I stopped the program. Can I go out and play now?
Oh no, not yet.

You may have ended the program, but you still have to prevent if from being started again. First of all, check to see if the program was filed under /etc/init.d . Every command located in this directory will be executed upon boot -- and you don't want the process to restart if you have to reboot the server.

### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
Next, make sure it's not created a crontab entry to be executed every X-seconds.

$ crontab -l
This will list all your cronjobs, for the current user -- probably root. If you see any command in there that is started from the location where you just found your culprit, remove it from the crontab.

$ crontab -e
This lets you edit the cronjobs, just like a normal vi. Remove the line, and save the file (:wq). Also make sure to log in to the user that executed your bad process (su <user>) and check those cronjobs.

Next up, update all your software. Either via yum or apt-get, or manually -- just make sure it's all up to date. Also, avoid using tools such as Webmin to manage a server -- it's highly vulnerable to these kinds of attacks.

You might want to check your /etc/, /tmp/ and /home/ folders just in case, for any files that were left there by your hacker. Remember: it's not because you stopped the process, that you're safe.

Assume there are other files, still hidden, on your server. Find them, and disable them. Then study their contents, and learn from it.

If this article was in any way useful to you, please leave a message and state how it helped you. If you see errors, of have other tips -- don't hesitate to share!

Hi! My name is Mattias Geniar. I'm a Support Manager at Nucleus Hosting in Belgium, a general web geek, public speaker and podcaster. If you're interested in keeping up with me, have a look at my podcast and weekly newsletter below. For more updates, follow me on Twitter as @mattiasgeniar.

Enter your e-mail here
SUBSCRIBE TO CRON.WEEKLY
I respect your privacy and you won't get spam. Ever.
Just a weekly newsletter about Linux and open source.
SysCast podcast
In the SysCast podcast I talk about Linux & open source projects, interview sysadmins or developers and discuss web-related technologies. A show by and for geeks!
cron.weekly newsletter
A weekly newsletter - delivered every Sunday - for Linux sysadmins and open source users. It helps keeps you informed about open source projects, Linux guides & tutorials and the latest news.
Share this post
Did you like this post? Will you help me share it on social media? Thanks!


### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
Comments
Rafal Los	Saturday, August 9, 2008 at 06:57
You appear to assume that the box’s processes and files haven’t been replaced with trojaned versions to hide the original malware…

Reply ↓
Mattias Geniar	Saturday, August 9, 2008 at 10:48
Well, this is indeed the simplified version. Most hacked servers are done using simple techniques, to do simple things, which can then simply be discovered.

Perhaps I’ll expand this to identify trojans and other malware too, for now it’ll just be this basic version.

Reply ↓
Pete	Monday, September 22, 2008 at 15:03
This was a very helpful little tutorial on the practical use of lsof. Thanks for taking the time to write this!

–Pete

Reply ↓
Tim	Tuesday, April 7, 2009 at 16:36
Great article! Very helpful. Much appreciated!

Reply ↓
MikeT	Thursday, May 6, 2010 at 19:03
This really opened my eyes. Thanks for using your time to write this…

MikeT

Reply ↓
Emre Sevinç	Friday, May 21, 2010 at 12:57
You said: “Be cautious with trusting the information they show you, and try using the BackTrack live CD (if you have physical access to do so) to perform your analysis.”

Booting the hacked server with a clean BackTrack Live CD and then analyzing the connections? How can that happen? I mean if you boot it with a clean, live CD then it is a clean, live server and no suspicious connections exist because you just booted it with a live CD. Your hacked server is not booted anymore. I’m a little bit confused.

Reply ↓
Matti	Sunday, May 23, 2010 at 10:06
@Emre; it only goes back to the “If you’re trying to fix a hacked server, it’s very possible that the binaries you’re executing (such as ‘ps’, ‘ls’, lsof’, ‘netstat’, …) have been replaced by hacked versions”-line. Those binaries shouldn’t be trusted, and can be “replaced” with the live CD ones.

The open connections are indeed useless at that point.

Reply ↓
zobin	Tuesday, October 26, 2010 at 18:48
Thank you for your article,and i found the spam process,but i don’t how to clear it in one time.
The below is the list which i ran lsof -p 2628(the pid of the main spam process)

COMMAND   PID USER   FD   TYPE     DEVICE    SIZE    NODE NAME
sendmail 2628 root  cwd    DIR        8,2 1699840 3899771 /var/spool/mqueue
sendmail 2628 root  rtd    DIR        8,2    4096       2 /
sendmail 2628 root  txt    REG        8,2  806460 6567022 /usr/sbin/sendmail.sendmail
sendmail 2628 root  mem    REG        8,2         7733281 /lib/libnsl-2.5.so (path inode=7733286)
sendmail 2628 root  mem    REG        8,2         7733338 /lib/libwrap.so.0.7.6 (path inode=7733452)
sendmail 2628 root  mem    REG        8,2         6563225 /usr/lib/libhesiod.so.0.0.0 (path inode=6563806)
sendmail 2628 root  mem    REG        8,2         7733446 /lib/libcom_err.so.2.1 (path inode=7734509)
sendmail 2628 root  mem    REG        8,2   14848 6654557 /usr/lib/sasl2/libplain.so.2.0.22
sendmail 2628 root  mem    REG        8,2         7733275 /lib/libcrypt-2.5.so (path inode=7734506)
sendmail 2628 root  DEL    REG        8,2         6565112 /usr/lib/liblber-2.3.so.0.2.31.#prelink#.oxts6J
sendmail 2628 root  mem    REG        8,2         6565055 /usr/lib/libkrb5.so.3.3 (path inode=6573533)
sendmail 2628 root  DEL    REG        8,2         6565057 /usr/lib/libkrb5support.so.0.1.#prelink#.QePuRn
sendmail 2628 root  mem    REG        8,2         6565045 /usr/lib/libk5crypto.so.3.1 (path inode=6573532)
sendmail 2628 root  mem    REG        8,2         6561754 /usr/lib/libz.so.1.2.3 (path inode=6567559)
sendmail 2628 root  mem    REG        8,2         7733295 /lib/libpthread-2.5.so (path inode=7733278)
sendmail 2628 root  mem    REG        8,2         7733264 /lib/ld-2.5.so (path inode=7733265)
sendmail 2628 root  mem    REG        8,2         7733339 /lib/libkeyutils-1.2.so (path inode=7733472)
sendmail 2628 root  mem    REG        8,2         7733332 /lib/libsepol.so.1 (path inode=7733300)
sendmail 2628 root  DEL    REG        8,2         7733467 /lib/libcrypto.so.0.9.8e.#prelink#.SBMg0S
sendmail 2628 root  mem    REG        8,2         7733277 /lib/libdl-2.5.so (path inode=7733296)
sendmail 2628 root  DEL    REG        8,2         6565114 /usr/lib/libldap-2.3.so.0.2.31.#prelink#.093P7r
sendmail 2628 root  mem    REG        8,2         7733297 /lib/libresolv-2.5.so (path inode=7734508)
sendmail 2628 root  mem    REG        8,2         7733469 /lib/libssl.so.0.9.8e (path inode=7734511)
sendmail 2628 root  mem    REG        8,2  905200 6654584 /usr/lib/sasl2/libsasldb.so.2.0.22
sendmail 2628 root  mem    REG        8,2         7733462 /lib/libdb-4.3.so (path inode=7733448)
sendmail 2628 root  mem    REG        8,2         7733271 /lib/libc-2.5.so (path inode=7733272)
sendmail 2628 root  mem    REG        8,2   50848 7733287 /lib/libnss_files-2.5.so
sendmail 2628 root  DEL    REG        8,2         6565041 /usr/lib/libgssapi_krb5.so.2.2.#prelink#.GnGQSz
sendmail 2628 root  mem    REG        8,2         7733442 /lib/libselinux.so.1 (path inode=7733304)
sendmail 2628 root  mem    REG        8,2   14752 6654553 /usr/lib/sasl2/liblogin.so.2.0.22
sendmail 2628 root  mem    REG        8,2         6562037 /usr/lib/libsasl2.so.2.0.22 (path inode=6563124)
sendmail 2628 root  mem    REG        8,2   14372 6652653 /usr/lib/sasl2/libanonymous.so.2.0.22
sendmail 2628 root    0r   CHR        1,3            1290 /dev/null
sendmail 2628 root    1w   CHR        1,3            1290 /dev/null
sendmail 2628 root    2w   CHR        1,3            1290 /dev/null
sendmail 2628 root    3u  unix 0xc3616580            7786 socket
sendmail 2628 root    4u  IPv4       7787             TCP localhost.localdomain:smtp (LISTEN)
sendmail 2628 root    5wW  REG        8,2      33 3900362 /var/run/sendmail.pid
Reply ↓
Mattias Geniar	Tuesday, October 26, 2010 at 23:25
For starters, stop sendmail so it no longer sends out messages. Then run some kind of rootkit hunter, and parse your log messages.
Sorry I can’t be more precise, but this differs from each server, and the way the server is being abused to send out spam. If possible, contact your hosting provider/sysadmin for more guidance.

Reply ↓
julian	Wednesday, February 9, 2011 at 23:43
Hi Matti,

you said:
@Emre; it only goes back to the “If you’re trying to fix a hacked server, it’s very possible that the binaries you’re executing (such as ‘ps’, ‘ls’, lsof’, ‘netstat’, …) have been replaced by hacked versions”-line. Those binaries shouldn’t be trusted, and can be “replaced” with the live CD ones.

I am ok with this idea if the replaced binaries are static but if they aren’t how could you trust the librairies used by those binaries ? I think you can’t turst a hacked server until you re-install it… you never now what they did with your server and how much this hacked machine could be corrupted.

Reply ↓
Caiya	Monday, November 14, 2011 at 16:16
Wait, I cannot fathom it being so srtaigthforward.
### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
//SOURCE: https://ma.ttias.be/how-to-identify-the-bad-processes-on-a-hacked-linux-box/