Friday, September 28, 2012

MoVP 3.5: Analyzing the 2008 DFRWS Challenge with Volatility

In this blog post I will go through analyzing the memory sample that was part of the 2008 DFRWS challenge.  This challenge was focused on a Linux computer that had sensitive files transferred from it. Due to its complexity and thoroughness, the challenge is well known throughout the forensics community and has been referenced by a number of research projects since being released.   

Obtaining the Sample
The sample for this blog post can be obtained from the DFRWS challenge file’s page here. The file is named ‘challenge.mem’ inside the compressed archive.

Obtaining the Volatility Profile
Since version 2.2 of Volatility has been placed into trunk, we have received numerous requests for built-in support of the DFRWS sample. This led us to creating and hosting a profile for it, which can be found on the Linux Profiles Wiki page. 

Analyzing the Image
We will walk through analyzing the image is two phases. First, we will compare the current capabilities of Volatility with those that were used in the winning submission of the competition. The complete set of analysis files uploaded by the winning team can be found here. Second, we will discuss relevant features of Volatility that perform deep analysis not available at the time of the challenge.

Parallel Capabilities
We will now show how to recreate much of the winning team’s output using the current state of Volatility. This serves as a useful explanation of Volatility’s current plugins and also guidance for people who have previously studied the winning challenge submission.  After the official 2.2 release next week, you can use the Volatility Command Reference (accessible from the Release22 page) for all the Linux plugins discussed.

These files contain the opened files for each process. The linux_lsof plugin recovers the same information.

This file contains the active network connections for each process. The linux_netstat plugin recovers the same information.

These files contain the kernel debug buffer. The linux_dmesg plugin recovers the same information.

This file contains high level information about the kmem cache. The linux_slabinfo Volatility plugin recovers complete information about each active kmem cache.

These files contain the loaded kernel modules active on the system. The linux_lsmod plugin performs the same functionality, and can also recover the parameters and sections for each module

This file contains high level information about the mounted filesystems. The linux_mount Volatility plugin recovers the same information.

The file contains the opened files per-filesystem. There is currently no Volatility plugin that operates the same as this crash module, but the linux_dentry_cache plugin can be used to recover a superset of this information. The dentry cache plugin recovers the filesystem in memory for each active mount point and can also recover filenames of previously deleted files as well.

These files contain information about running processes. The crash_ps_args.txt file also contains the command line arguments and environment variables for each process. The linux_pslist plugin can be used to list processes and the linux_psaux plugin can recover command line options. The next version of linux_psaux will also support environment variables.

These files were listed as having been recovered from memory, likely through strings, grep, or other primitive forensics techniques. The linux_find_file plugin can be used to directly recover these files from memory:

# python --profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_find_file -F /etc/passwd
Volatile Systems Volatility Framework 2.2_rc2
Inode Number          Inode
---------------- ----------
          975039 0xd100b4ac

# python --profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_find_file -F /etc/group
Volatile Systems Volatility Framework 2.2_rc2
Inode Number          Inode
---------------- ----------
          975309 0xd0cc8c5c

# python --profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_find_file -i 0xd100b4ac -O passwd

# python --profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_find_file -i 0xd0cc8c5c -O group

10.    linvm.txt
This file lists the memory maps for each process. The linux_proc_maps Volatility plugin recovers the same information.

11.    linpsscan.txt
This file contains a list of processes that were recovered through carving. While Volatility does not currently have a Linux process carver, it does have the linux_pslist_cache plugin. When run with the -u/--unallocated option, this plugin will recover processes that have previously terminated.

This file contains the ARP entries on a computer. The linux_arp plugin recovers the same information.

This plugin lists data about each network interface on a computer. This linux_ifconfig plugin recovers very similar data.

This plugin contains the routing cache at the time of the memory dump. The linux_route_cache plugin performs the same functionality.

New Capabilities

We will now discuss two key capabilities present in Volatility that were not available or developed by the winning team.

1.   Recovering Files from Memory
As we have seen throughout the Linux focused MoVP posts, the linux_find_file command is very powerful in a number of situations. This plugin allows the investigator to find the inode structure for a file and then recover file contents from memory.

In the writeup for the winning submission, there are references to a folder named “Admin_share”, which is where sensitive documents were stored. To find information about this folder, we can use the linux_dentry_cache plugin to list all files referenced in this folder from memory:

# python --profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_dentry_cache > dentry_cache_out

# python --profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_dentry_cache –u >> dentry_cache_out

This will recover all files and metadata information from their inode structure and write the results in bodyfile format. Once we have all the files listed, we can then grep for our folder of interest:

# grep Admin_share dentry_cache_out
0|Admin_share/negotiation notes.txt|262638|0|500|500|354|3469838020|3469838028|0|3469838036

We now have found references to a number of interesting files and can use linux_find_files to extract them:

# python --profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_find_file -F /mnt/hgfs/Admin_share/
Volatile Systems Volatility Framework 2.2_rc2
Inode Number          Inode
---------------- ----------
          262633 0xced18200

# python --profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_find_file -i 0xced18200 -O
Volatile Systems Volatility Framework 2.2_rc2
# file a /usr/bin/perl script text executable

2.    Orderly Recovery of Bash History
In previous MoVP posts we have used the linux_bash plugin to recover bash history from memory.   In order for this plugin to work though, it requires the address of the array of commands entered. To determine this value normally requires a copy of /bin/bash from the analyzed computer, but unfortunately we only have a memory image and linux_find­_file was unable to recover the bash binary from memory. We needed a different and new approach.

Our first thought was to simply explore the heap of any bash process and brute force all addresses within the heap to see if the plugin worked.  This hit a roadblock though as the linux_proc_maps plugin was not reporting any memory mapping as the heap. Confused, we added debugging prints to the plugin in order to determine where the heap should be based off the brk and brk_start members of mm_struct.  The debug prints showed that the brk and brk_start members actually pointed into separate and non-adjacent memory mappings!

This means that either the process had a fragmented heap, which we have never heard for Linux, or the tracking values were not updated correctly. To compensate for this, we had the plugin bruteforce across both memory ranges and report any addresses that looked like bash history structures:

# python --profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_bash -p 2585 -H 1
Volatile Systems Volatility Framework 2.2_rc1
Command Time         Command
-------------------- -------
good: 80f96b8 | l.
good: 80fa0e4 | ?H?X!?
good: 80faaa8 | ?  p??H?X!?
good: 80fab1c | uname -a
good: 80fb52c | xO??e
good: 80fb5e8 | shell-expand-line
good: 80fb638 | Mon

Of these, the command at 0x80fab1c looks most reasonable as it returns a real command (uname –a). I then reverted the changes to the plugin, used linux_pslist to determine instances of bash running, and specified our likely addresses as the one to recover from:

# python --profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_bash -p 2585 -H 0x80fab1c
Volatile Systems Volatility Framework 2.2_rc2
Command Time         Command
-------------------- -------
#1197861861          uname -a
#1197861861          who
#1197861861          ll -h
#1197861861          mkdir temp
#1197861861          ll -h
#1197861861          chmod o-xrw temp/
#1197861861          ll -h
#1197861861          cd temp/
#1197861861          cp /mnt/hgfs/Admin_share/*.xls .
#1197861861          cp /mnt/hgfs/Admin_share/*.pcap .
#1197861861          exit
#1197861861          uname -a
#1197861861          id
#1197861861          exit
#1197861861          X -v
#1197861861          X -V
#1197861861          X -version
#1197861861          cd temp
#1197861861          wget
#1197861861          tar -zpxvf xmodulepath.tgz
#1197861861          cd xmodulepath
#1197861861          ll
#1197861861          unset HISTORY
#1197861861          ./
#1197861861          exit
#1197861861          pwd
#1197861861          cd ..
#1197861861          cp /mnt/hgfs/Admin_share/intranet.vsd .
#1197861861          ll
#1197861861          ls -lh
#1197861861          exit
#1197861880          ls /mnt/hgfs/Admin_share/
#1197861980          zip /mnt/hgfs/Admin_share/acct_prem.xls /mnt/hgfs/Admin_share/domain.xls /mnt/hgfs/Admin_share/ftp.pcap
#1197862015          unset HISTFILE
#1197862019          unset HISTSIZE
#1197862066          zipcloak
#1197862105          ll -h
#1197862134          cp /mnt/hgfs/software/ .
#1197862137          ll -h
#1197862196          export http_proxy=""
#1197862200          env | less
#1197862316          ./
#1197865970          unset http_proxy
#1197865973          rm
#1197866006          dir
#1197866009          rm

# python --profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_bash -p 2921 -H 0x80fab1c
Volatile Systems Volatility Framework 2.2_rc2
Command Time         Command
-------------------- -------
#1197862634          uname -a
#1197862634          who
#1197862634          ll -h
#1197862634          mkdir temp
#1197862634          ll -h
#1197862634          chmod o-xrw temp/
#1197862634          ll -h
#1197862634          cd temp/
#1197862634          cp /mnt/hgfs/Admin_share/*.xls .
#1197862634          cp /mnt/hgfs/Admin_share/*.pcap .
#1197862634          exit
#1197862634          uname -a
#1197862634          id
#1197862634          exit
#1197862634          X -v
#1197862634          X -V
#1197862634          X -version
#1197862634          cd temp
#1197862634          wget
#1197862634          tar -zpxvf xmodulepath.tgz
#1197862634          cd xmodulepath
#1197862634          ll
#1197862634          unset HISTORY
#1197862634          ./
#1197862634          exit
#1197862634          pwd
#1197862634          cd ..
#1197862634          cp /mnt/hgfs/Admin_share/intranet.vsd .
#1197862634          ll
#1197862634          ls -lh
#1197862634          exit
#1197862646          netstat -tupan
#1197862716          netstat -tupan
#1197862955          netstat -tupan
#1197863124          netstat -tupan
#1197863195          netstat -tupan
#1197863230          netstat -tupan
#1197863281          netstat -tupan
#1197863543          netstat -tupan
#1197863623          netstat -tupan
#1197863726          netstat -tupan
#1197863945          netstat -tupan
#1197863991          netstat -tupan
#1197864058          netstat -tupan
#1197864123          ping
#1197864137          ping
#1197864248          ping
#1197864251          netstat -tupan
#1197864680          netstat -tupan
#1197864820          netstat -tupan
#1197864904          netstat -tupan
#1197865254          netstat -tupan
#1197865458          netstat –tupan

As we can see from the output, the machine was compromised and files were transferred from it. We also see that Matthew Geiger really enjoys running the netstat command (almost as much as Jesse K likes looking for his tools) ;-)

If there were no relevant instances of bash running when the memory capture was taken, we could also use linux_find_file  to recover .bash_history files from memory. While these records may not be as complete (i.e. they probably will not have timestamps), they are certainly better than being reduced to strings and grep.

We have demonstrated how to approach the 2008 DFRWS challenge image, as well as Linux incident response in general, using Volatility. We hope this serves as a good learning lesson for those wishing to perform further research and/or investigations in the field.

If you have any questions or comments please use the comment section of the blog or you can find me on Twitter (@attrc).

No comments:

Post a Comment