MoVP 1.4 Average Coder Rootkit, Bash History, and Elevated Processes

Month of Volatility Plugins

In this post I will begin showcasing some of Volatility’s new Linux features by analyzing a popular Linux kernel rootkit named “Average Coder”.  These new features will include recovering .bash_history from memory, finding userland processes elevated to root by kernel mode components of the rootkit, and discovering overwritten file operation structure pointers. 

If you would like to follow along or recreate the steps taken, please see the LinuxForensicsWiki  for instructions on how to do so.

Obtaining the Samples

To have samples to test against, I used two sources. The first was the sample stored on the SecondLook “Linux Memory Images” page  (big thanks to SecondLook for creating the samples and hosting them!), and the second sample was from my own virtual machine that I infected to showcase parts of the rootkit not used by SecondLook.

Average Coder Rootkit

The Average Coder Rootkit is a Linux kernel rootkit that operates as a loadable kernel module and provides the ability to hide processes, logged in users, and kernel modules. It also provides the ability for a userland process (e.g. bash) to elevate itself to root privileges by writing to a file inside of /proc.

Average Coder gains control over a computer by loading as a kernel module and then overwriting a number of file_operation structures within the kernel.  file_operations is a C structure used by the Linux kernel to provide generic handling of files by a number of subsystems within the kernel and contains function pointers such as read, write, readdir, open, close, etc.  Each active file in the operating system has its own file_operations structure that is referenced every time activity is performed on the file (e.g when a file is read, the read function pointer of the structure is eventually called to handle reading the file contents from disk). 

We will now go through each piece of functionality the rootkit offers, how it accomplishes it, and how we can detect it with Volatility.  

The Loadable Module

The first thing we notice, by using the linux_lsmod plugin, is that the rootkit does not unlink itself from the module list (it is named rootkit):

# python vol.py --profile=LinuxCentOS63x64 -f avgcoder.mem linux_lsmod | grep -i root

Volatile Systems Volatility Framework 2.2_alpha

rootkit 13438

Instead, as we will see, this rootkit hides from userland by hooking /proc/modules, and while this does effectively hide from userland, tools analyzing kernel memory will have no issues finding the module.

By reading the rootkit source, we also see that the rootkit takes optional parameters for configuration files and directories. We can recover these options with the –P/--params parameter to linux_lsmod:

# python vol.py --profile=LinuxCentOS63x64 -f avgcoder.mem linux_lsmod –P | grep –A 6 rootkit

Volatile Systems Volatility Framework 2.2_alpha

rootkit 13438

        mod_dir=(null)

        mod_name=(null)

        rc_dir=(null)

        rc_name=(null)

       

In this case we see that the SecondLook engineers did not pass any options when they loaded the rootkit, so all the values are null.

Since rootkits that are deployed in the wild will hide themselves from lsmod, to perform the rest of the analysis, I modified Volatility’s module finding code to skip the rootkit module in order to mimic a real compromise. We will see in a later blog post how Volatility detects modules that unlink themselves from the module list.

Hiding Users

Effect on Forensics

To hide users, Average Coder hooks the read member of /var/run/utmp in the kernel in order to hide logged in users from applications such as w and who.  This effectively hides logged in users from network administrators and incident response members working from userland on a live computer.

How Average Code Accomplishes it

The rootkit hides users in its ­init_users_hide_hook function (here). It does this by calling the kernel’s path_lookup function on /var/run/utmp in order to find the inode structure of the file. The path_lookup function works by enumerating the filesystem’s directory structure and then locating the file of interest.  Average Coder wants the inode structure because its i_fop member is a pointer to the particular file’s file_operations structure.  Once the i_fop member is found, it can simply be overwritten with the rootkit’s function that filters users from utmp on demand.

How Volatility Detects This

To detect this part of the rootkit, we need to verify that the file operations function pointers for /var/run/utmp are valid. Valid in this case means that the function pointers point to a function inside the base kernel or within a known kernel module. If the function pointers are invalid, we report them so they can be investigated. 

To do this for /var/run/utmp, we first find its inode using the linux_find_file command with utmp as the parameter:

# python vol.py -f avgcoder.mem --profile=LinuxCentOS63x64 linux_find_file -F "/var/run/utmp"

Volatile Systems Volatility Framework 2.2_rc1

Inode Number                  Inode

---------------- ------------------

          130564     0x88007a85acc0

This gives us the inode number and address of the inode structure. We can use the number to investigate the file on disk with tools, such as the Sleuthkit, and we can use the address to investigate the file in memory with Volatility.  To verify that the function pointers are correct we pass the address to linux_check_fop:

# python vol.py -f avgcoder.mem --profile=LinuxCentOS63x64 linux_check_fop -i 0x88007a85acc0

Volatile Systems Volatility Framework 2.2_rc1

Symbol Name                   Member                 Address

----------------------------- ---------------------- ------------------

inode at 88007a85acc0         read                   0xffffa05ce4d0

This plugin, when given the –i/--inode option, reads the inode at the given address and verifies each member of its i_fop pointer. As we can see, the plugin tells us that the read member is hooked and the address of the hooked function. This verifies what we see in the Average Coder source code and if this was an unknown rootkit, we could then perform binary analysis of the function if we desired.  

Since we now know for certain that w and who were being lied to on a live system, we want to figure out what the correct values were and who was actually logged in. To do that, we are again going to use the linux_find_file plugin but this time we will have it recover the /var/run/utmp file from memory for us:

# python vol.py -f avgcoder.mem --profile=LinuxCentOS63x64 linux_find_file -i 0x88007a85acc0 -O utmp

This writes the contents of /var/run/utmp from memory into the local “utmp” file. We can then use the who binary on our analysis system to determine who was logged in:

# who utmp

centoslive tty1         2013-08-09 16:26 (:0)

centoslive pts/0        2013-08-09 16:28 (:0.0)

Ignoring the fact that the SecondLook system had a skewed clock, we know now which user was logged in, how they were logged in (tty1, pts/0), and the time they logged in. Again, any IR script that ran on the live system would have missed these logins (we will soon see that centoslive was a user hidden by the rootkit), and we recovered this information by pulling /var/run/utmp directly out of memory and did not rely on the disk.

Hiding Processes, Kernel Modules, and Communicating with Userland

Effect on Forensics

Average Coder also allows users to hide processes and kernel modules from userland tools.  It does this by setting up a communications channel on which the kernel module accepts commands from userland. This has a very negative effect on forensics as investigators working on a live machine cannot trust the output of a majority of the tools and operating system interfaces that they normally rely on for gathering evidence.

How Average Code Accomplishes it

Commands are taken from userland by hooking the write member of /proc/buddyinfo, which is a file that normally does not support writes, and data written to this file (commands) can be used to hide process, users, and to elevate privileges.

Processes are hidden by hooking the readdir member of the root of the /proc filesystem. Every active process in Linux has a corresponding directory under /proc whose name is the PID of the process (e.g init has a directory of /proc/1/). To hide processes, the hijacked readdir function simply filters out the directories that correspond to hidden processes. This effectively hides the process from a number of userland tools.

Kernel modules are hidden from lsmod by hooking the read member of /proc/modules. The modules file is the only source used by lsmod to list loaded modules, so this effectively hides it from the application. Again, we will see in a future MoVP post how to find hidden modules on both a live system and from a memory image by leveraging sysfs.

How Volatility Detects This

To detect these overwritten file­_operations structures, we will use linux­_check­_fop in its default mode. This mode checks the file_operations structure for a number of common filesystems and files, including all of /proc:

# python vol.py -f avgcoder.mem --profile=LinuxCentOS63x64 linux_check_fop

Volatile Systems Volatility Framework 2.2_rc1

Symbol Name              Member           Address

------------------------ ---------------- ------------------

proc_mnt: root           readdir          0xffffa05ce0e0

buddyinfo                write            0xffffa05cf0f0

modules                  read             0xffffa05ce8a0

As we can see from the output, Volatility was able to report the three hooks placed by Average Coder (readdir from root of proc, write of buddyinfo, and read of modules), by enumerating all the files and directories under /proc and verifying their members. From here, the investigator knows the machine is compromised and can begin to investigate the rootkit.

Recovering .bash history from Memory

Effect on Forensics

.bash_history is a file on disk that stores all the commands run by a user directly on the bash command line. This file is a forensics goldmine when populated as the investigator can recreate everything done by a logged in user. Due to its importance to forensics, any reasonable attacker is going to make every effort to avoid having commands logged to this file. Common methods to accomplish this include:

• Logging in with ssh –T, which does not allocate a pseudo terminal and therefore does not spawn bash
• Setting HISTFILE to /dev/null or unsetting it from the process environment, which effectively stops logging
• Setting HISTSIZE to 0 which prevents any logging

The exclusion of these commands to disk makes traditional disk forensics much more difficult and means we have to rely on in-memory information.

How Volatility Handles This

Fortunately for investigators, no matter which of the previous anti-forensics tricks are used (or any others), bash still stores the history of commands in memory for the current session along with the time executed – regardless of if timestamps are enabled for disk logging.

To recover this information, the linux_bash plugin can be used. This plugin finds running bash instances, walks their history structures in memory, and pulls out the commands executed and their time ran (epoch).  Running linux_bash on the Average Coder memory sample produces the following output:

# python vol.py -f avgcoder.mem --profile=LinuxCentOS63x64 linux_bash -H 0x6e0950 -P

Volatile Systems Volatility Framework 2.2_rc1

Command Time         Command

-------------------- -------

#1376083693          dmesg | head -50

#1376085088          df

#1376085110          dmesg | tail -50

#1376085118          sudo mount /dev/sda1 /mnt

#1376085122          cd /mnt

#1376085122          ls

#1376085128          sudo insmod rootkit.ko

#1376085176          echo "hide" > /proc/buddyinfo

#1376085180          lsmod | grep root

#1376085194          w

#1376085218          echo "huser centoslive" > /proc/buddyinfo

#1376085220          w

#1376085229          sleep 900 &

#1376085241          echo "hpid 2872" > /proc/buddyinfo

#1376085253          ps auwx | grep sleep

<unallocated entries>

#1376085241          echo "hpid 2872" > /proc/buddyinfo

<unallocated entries>

#1376085128          sudo insmod rootkit.ko

As we study this output, we see a number of interesting results.  First we see that the rootkit was inserted into the kernel sudo insmod rootkit.ko. We then see “hide" being sent to /proc/buddyinfo which hides the kernel module from lsmod. Next, we see the person checking to be sure that rootkit does not appear in lsmod output. The user then hides the centoslive user from utmp (we covered this earlier). The user checks the success of this by running the w command. Finally, we see the user spawn and background the sleep command, which will print the PID of it to the command line, and then this PID is sent to buddyinfo to be hidden. The user then checks this to make sure sleep does not appear in ps auxw. 

Recovering these commands confirms much of what we have seen with the previous Volatility plugins and also would give us a starting point in a real investigation.  Due to the way this rootkit hides data, by modifying function pointers instead of in-kernel structures, we do not have to do anything special with Volatility to recover the hidden information, it is just simply there. We then verify the presence of malicious hooks using methods shown throughout the post.

Note: The -P/--printunalloc option to linux_bash tells the plugin to print unallocated entries as well as the currently allocated ones. This has the effect of printing overwritten entries to the terminal, so it is recommended to redirect this output to a file. In the above output I have taken out the unreadable unallocated entries and replaced them with a label.

Elevating Process Privileges

Effect on Forensics

Average Coder provides the ability for userland processes to send a command to /proc/buddyinfo in order to have their privileges elevated to root (user ID 0).  This is a very common method used by attackers in order to maintain persistence on compromised resources. 

The sample provided by SecondLook did not utilize this portion of the rootkit, so I made my own sample to test it out and to verify that the created plugin, linux_check_creds, detects it. The following shows my interaction with the rootkit to elevate my bash shell:

x@debian:~$ id

uid=1000(x) gid=1000(x) groups=1000(x),24(cdrom),25(floppy),29(audio),30(dip),44(video),46(plugdev)

x@debian:~$ echo $$

9673

x@debian:~$ echo "root $$" > /proc/buddyinfo

-bash: echo: write error: Operation not permitted

x@debian:~$ id

uid=0(root) gid=0(root) groups=0(root)

As can be seen, I made my user ‘x’ go from privileges of uid 1000 to uid 0 (root) by writing the PID of my bash shell to /proc/buddyinfo with the root command. I then took a memory capture of the infected VM using a version of LiME already compiled for my VM:

# ls

disk.c  lime-2.6.32-5-686.ko  lime.h  main.c  Makefile  Makefile.sample  out  tcp.c

# insmod ./lime-2.6.32-5-686.ko "format=lime path=avg.hidden-proc.lime"

This wrote the memory sample to avg.hidden-proc.lime

How Average Code Accomplishes it

On older 2.6 kernels, the user ID and group ID of a process were kept as simple integers in memory. For a rootkit to elevate the privileges of a process, it simply set these two values to zero.  This simplicity also made it very difficult to use only the information in the process structure itself to detect which processes had been elevated and which were simply spawned by root.

This changed in later versions of 2.6 as the kernel adopted a cred structure to hold all information related to the privileges of a process.  This structure is fairly complicated and forced rootkits to adapt their process elevation methods. Although the kernel provides the prepare_creds and commit_creds functions to allocate and store new credentials, a number of rootkits choose not to use this functionality. Instead, they simply find another process that has the privileges of root and that never exits, usually PID 1, and set the cred pointer of the target process to that of PID 1’s. This effectively gives the attacker’s process full control and the rootkit does not have to attempt the non-trivial task of allocating its own cred structure.

How Volatility Detects This

The borrowing of cred structures leads to an inconsistency that Volatility can leverage to find elevated processes. In the normal workings of the kernel, every process gets a unique cred structure and they are never shared or borrowed. The linux_check_creds plugin utilizes this by building a mapping of processes and their cred structures and then reports any processes that share them.   The following output shows the cred structure running on my infected VM and showing that PID 1 has the same cred structure as my elevated bash shell (PID 9673):

# python vol.py -f avg.hidden-proc.lime --profile=Linuxthisx86 linux_check_creds

Volatile Systems Volatility Framework 2.2_rc1

PIDs

--------

1, 9673

In real investigations, we could now focus our efforts on PID 9673 - using the bash plugin would be a good start – as we know that the attacker used that shell in conjunction with the rootkit!

Conclusion

We have thoroughly investigated the Average Coder rootkit, including its internals, artifacts left on a system, and interactions with the attackers who place it on a system.  This includes finding the module, locating its hooks throughout the kernel, finding interactions with the rootkit by userland processes, and recovering files that it filtered on the running machine.

In tomorrow’s post we will analyze another kernel rootkit that requires other plugins not covered in this post to detect. If you have any questions or comments please use the comment section of the blog or you can find me on Twitter (@attrc).