MotivationAlthough it would make our jobs quite interesting if every investigation involved analyzing new malware samples and families, the reality is that many malware investigations only require analyzing memory samples in order to verify (or hunt for) an infection by malware previously discovered in the wild. In these situations, the ability to quickly leverage existing tools and malware intelligence can lead to rapid identification of infected systems.
In this blog post, we document a methodology that we have successfully used in many investigations to confirm the presence, or absence, of known malware on a system. By following this workflow, you should be able to quickly sort through memory samples in your current investigations and to perform one aspect of proactive threat hunting in a highly efficient manner.
SetupFor the purpose of this blog post, we are going to analyze a memory sample that was previously available online. In particular, we are going to look at the stuxnet.vmem file that was produced to accompany The Malware Analyst's Cookbook. As you can likely guess, this memory sample came from a virtual machine infected with Stuxnet.
The real world scenarios where this applies are when you wish to verify if a system is infected, such as after receiving an AV or IDS alert, or when you wish to perform proactive threat hunting in order to stay ahead of potential threats in your environment. In these situations, you will first need to acquire a memory sample of the system you wish to analyze. A full discussion of this process is outside the scope of this blog post, but Chapter 4 of the Art of Memory Forensics provides extensive detail on the topic.
The remainder of this blog post assumes that you have a memory sample that was acquired using a stable tool and/or method.
The goal of this section is to showcase how, with only three Volatility plugins, an analyst can quickly extract a majority of the executables loaded into memory at the time of acquisition.
dlldumpTo start, the dlldump plugin will be used to extract the main application executable and all* loaded DLLs inside of each** process:
In the above invocation of dlldump, we set two options. The first, -D, specifies the directory in which to extract the executables. We will have each extraction plugin write to the same directory in order make running ClamScan easier. The second option, --memory, is likely only familiar to power Volatility users. The purpose of this flag is to instruct Volatility to extract all regions of memory around PE files instead of just relying on the file metadata in the PE header that specifies the original size of each section. This helps substantially when extracting packed executables from memory as you can get data both from the original file as well as the unpacked executable.$ python vol.py -f stuxnet.vmem --profile=WinXPSP2x86 dlldump —memory -D stuxout/ Volatility Foundation Volatility Framework 2.5 Process(V) Name Module Base Module Name Result
---------- -------------------- ------------- ----------------------- 0x820df020 smss.exe 0x048580000 smss.exe OK: module.376.22df020.48580000.dll 0x820df020 smss.exe 0x07c900000 ntdll.dll OK: module.376.22df020.7c900000.dll 0x821a2da0 csrss.exe 0x04a680000 csrss.exe OK: module.600.23a2da0.4a680000.dll 0x821a2da0 csrss.exe 0x07c900000 Error: DllBase is paged 0x821a2da0 csrss.exe 0x075b40000 CSRSRV.dll OK: module.600.23a2da0.75b40000.dll 0x821a2da0 csrss.exe 0x077f10000 GDI32.dll Error: DllBase is paged 0x821a2da0 csrss.exe 0x07e720000 sxs.dll Error: DllBase is paged 0x821a2da0 csrss.exe 0x077e70000 RPCRT4.dll Error: DllBase is paged 0x821a2da0 csrss.exe 0x077dd0000 ADVAPI32.dll Error: DllBase is paged 0x821a2da0 csrss.exe 0x077fe0000 Secur32.dll Error: DllBase is paged 0x821a2da0 csrss.exe 0x075b50000 basesrv.dll Error: DllBase is paged 0x821a2da0 csrss.exe 0x07c800000 KERNEL32.dll Error: DllBase is paged 0x821a2da0 csrss.exe 0x07e410000 USER32.dll OK: module.600.23a2da0.7e410000.dll 0x821a2da0 csrss.exe 0x075b60000 winsrv.dll OK: module.600.23a2da0.75b60000.dll 0x81da5650 winlogon.exe 0x001000000 winlogon.exe OK: module.624.1fa5650.1000000.dll <snip>
* There can be hidden DLLs. The next plugin, malfind, will take care of that.
** If a process is hidden from the active process list, then dlldump will not, by default, find it. This means the hidden executable and any DLLs it loaded will not be extracted using the above invocation. In this situation, you would want to use psxview to find the hidden process - but realize that the fact that a process is hidden already mean its malicious, and you do not need ClamScan to re-verify this for you.
malfindThe next plugin that we will use is malfind, which is a plugin that searches for malicious executables (usually DLLs) and shellcode inside of each process. Explaining the precise details of how malfind works is outside the scope of this post and not relevant in a triage situation - but again consult The Art of Memory Forensics if you want all the details. The following shows how to run malfind against the stuxnet sample:
$ python vol.py -f stuxnet.vmem --profile=WinXPSP2x86 malfind -D stuxout/ Volatility Foundation Volatility Framework 2.5
<snip> Process: services.exe Pid: 668 Address: 0x13f0000 Vad Tag: Vad Protection: PAGE_EXECUTE_READWRITE Flags: Protection: 6 0x013f0000 4d 5a 90 00 03 00 00 00 04 00 00 00 ff ff 00 00 MZ.............. 0x013f0010 b8 00 00 00 00 00 00 00 40 00 00 00 00 00 00 00 ........@....... 0x013f0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0x013f0030 00 00 00 00 00 00 00 00 00 00 00 00 08 01 00 00 ................ 0x013f0000 4d DEC EBP 0x013f0001 5a POP EDX 0x013f0002 90 NOP 0x013f0003 0003 ADD [EBX], AL 0x013f0005 0000 ADD [EAX], AL 0x013f0007 000400 ADD [EAX+EAX], AL 0x013f000a 0000 ADD [EAX], AL 0x013f000c ff DB 0xff <snip>
By default, for each suspicious memory region that malfind encounters, it will print attributes about the region such as which process it is mapped in, the starting and ending address of the region, and a hexdump and disassembly of the bytes at the beginning of the suspicious region. Since we added the -D flag to malfind, it will also extract each suspicious region into a separate file under our specified output directory.
By running the file command on just the files that malfind extracted, we have a good idea that we have already found the malware:
$ file stuxout/* stuxout/process.0x81c47c00.0x1000000.dmp: PE32 executable (GUI) Intel 80386, for MS Windows stuxout/process.0x81c47c00.0x680000.dmp: data stuxout/process.0x81c47c00.0x6f0000.dmp: data stuxout/process.0x81c47c00.0x80000.dmp: PE32 executable (DLL) (GUI) Intel 80386, for MS Windows, UPX compressed stuxout/process.0x81c47c00.0x870000.dmp: PE32 executable (DLL) (GUI) Intel 80386, for MS Windows, UPX compressed stuxout/process.0x81c498c8.0x1000000.dmp: PE32 executable (GUI) Intel 80386, for MS Windows stuxout/process.0x81c498c8.0x80000.dmp: PE32 executable (DLL) (GUI) Intel 80386, for MS Windows, UPX compressed stuxout/process.0x81e61da0.0xb70000.dmp: data stuxout/process.0x81e61da0.0xbf0000.dmp: data stuxout/process.0x81e61da0.0xd00000.dmp: PE32 executable (DLL) (GUI) Intel 80386, for MS Windows, UPX compressed stuxout/process.0x82073020.0x13f0000.dmp: PE32 executable (DLL) (GUI) Intel 80386, for MS Windows, UPX compressed stuxout/process.0x82073020.0x940000.dmp: data stuxout/process.0x820ec7e8.0x2550000.dmp: data stuxout/process.0x821a2da0.0x7f6f0000.dmp: data
By using dlldump and malfind, we have extracted every executable that Volatility will give us from userland (process memory) without having to manually dig ourselves. For the last plugin invocation, we will turn to moddump in order to extract all* the executables (drivers) from kernel memory as well.
$ python vol.py -f stuxnet.vmem --profile=WinXPSP2x86 moddump —memory -D stuxout/ Volatility Foundation Volatility Framework 2.5 Module Base Module Name Result ----------- -------------------- ------ 0x0804d7000 ntoskrnl.exe OK: driver.804d7000.sys 0x0806d0000 hal.dll OK: driver.806d0000.sys 0x0f7470000 update.sys OK: driver.f7470000.sys 0x0f89ba000 usbehci.sys OK: driver.f89ba000.sys 0x0f8a1a000 HIDPARSE.SYS OK: driver.f8a1a000.sys 0x0f8b5a000 CmBatt.sys OK: driver.f8b5a000.sys 0x0f855a000 pci.sys OK: driver.f855a000.sys 0x0f89aa000 usbuhci.sys OK: driver.f89aa000.sys 0x0bf800000 win32k.sys OK: driver.bf800000.sys 0x0f87fa000 raspptp.sys OK: driver.f87fa000.sys 0x0f89c2000 TDI.SYS OK: driver.f89c2000.sys 0x0b2c23000 ipnat.sys OK: driver.b2c23000.sys 0x0b23ce000 wdmaud.sys OK: driver.b23ce000.sys 0x0f84e5000 SCSIPORT.SYS OK: driver.f84e5000.sys
In this invocation, we have instructed Volatility to locate and extract all* kernel drivers from kernel memory into our output folder.
* moddump relies on the linked list of kernel modules in order to locate each extracted module. This list is susceptible to manipulation by malware and kernel-level malware can certainly run without being present in this list - in fact, it can even run without having any driver file associated with the malware at all (these are known as 'orphan threads' in Volatility terminology). We will avoid that rabbit hole in this blog post though.
Detection with ClamScan
So far we have leveraged dlldump, malfind, and moddump to automatically locate and extract a wide variety of executables from memory. At this point, we want to check to see if any known malware is on the system, and to do so we will use ClamAV. For those unaware, ClamAV is an open source anti-virus engine with millions of signatures for known malware. It is command-line based and very easy to automate.
The following invocation of clamscan shows running it against all the files in the stuxout directory, which is the directory that contains every executable extracted by the previous three plugins:
$ clamscan stuxout/ | grep -v ": OK$" stuxout/process.0x81c47c00.0x1000000.dmp: Win.Trojan.5873027-1 FOUND stuxout/process.0x81c47c00.0x6f0000.dmp: Win.Worm.Stuxnet-49 FOUND stuxout/process.0x81c498c8.0x1000000.dmp: Win.Trojan.5873027-1 FOUND stuxout/driver.f895a000.sys: Win.Trojan.Rootkit-8720 FOUND stuxout/process.0x81e61da0.0xb70000.dmp: Win.Worm.Stuxnet-49 FOUND stuxout/process.0x81c47c00.0x80000.dmp: Win.Worm.Stuxnet-49 FOUND stuxout/process.0x81c498c8.0x80000.dmp: Win.Worm.Stuxnet-49 FOUND ----------- SCAN SUMMARY ----------- Known viruses: 4514519 Engine version: 0.98.7 Scanned directories: 1 Scanned files: 1262 Infected files: 7 Data scanned: 614.46 MB Data read: 631.82 MB (ratio 0.97:1) Time: 196.320 sec (3 m 16 s)
In this output it is immediately obvious that we have a problem as 6 process (userland) files matched AV signatures. Furthermore, one of the drivers extracted from kernel memory matched a signature as well. While ClamScan will occasionally throw a false positive, the fact that multiple signatures matched multiple extracted files very likely means that we have an actual infection. In this example, we can see that ClamAV correctly identifies several of the malicious regions as being associated with Stuxnet.
Advantages to this approach
SpeedThe fact that we only need to run three Volatility plugins, all with static command line options, makes this a very fast process. Since none of the plugins require scanning the entire memory sample, each individual plugin should finish in less than one minute.
ClamAV can take anywhere from 1 minute to 10 minutes to scan all files extracted from memory samples that we regularly encounter.
ScalabilitySince the entire triage process described in the writeup can be scripted and will almost always be finished in less than 15 minutes, this approach to detecting known malware is highly scalable.
One of the main advantages to extracting malware through memory forensics is that the malware will be in its unpacked form (ignoring VM-based and other advanced packers). This means that the anti-virus engine will be able to scan the actual contents of the malicious files and not just its obfuscated, outer shell. This really helps when scanning with Yara rules as well.
Memory-Only Malware & Injected CodeThrough the use of malfind, we are able to find code injected into memory that may never exist on disk and that may have been injected in ways that traditional endpoint security tools would have missed.
No Need to Upload Samples
By running your testing local, you bypass the need to upload potential targeted malware to services such as VirusTotal or to your anti-virus vendor.
No OS Internals Needed
Performing this type of triage requires no advanced memory forensics or operating systems internals knowledge - all you have to do is run the plugins followed by clamscan.
Disadvantages to this approachWhile highly useful, this approach is not 100% effective. The following sections describe some of the drawbacks.
Known OnlyAs made clear in the post title, this approach will only detect known malware. If the organizations originally targeted by Stuxnet had applied this approach to their systems, they would not have detected it as there would be no signatures for it.
This is an example of where deep memory forensics comes into play as it can detect malware in a generic fashion - without the need for any signatures or malware-specific knowledge. MHL's blog post does a great job of showing how effective Volatility's generic anomaly detection plugins are against Stuxnet.
Not everything will be mapped
A second downfall to this approach is that the malicious code might not be resident in memory at the time of the acquisition - it will either in a file on disk or scattered throughout the page file. This situation requires deeper analysis beyond simple triage in order to verify the malware's presence.
ClamAV isn't PerfectNo security tool is perfect, and ClamAV is no exception to this rule. Just because ClamAV does not flag any executables does NOT mean that they are all actually legitimate. In general, we use ClamAV for quick, positive identification of known malware - not to gain confidence that no malware is on the system.
Warnings / Caution PointsBefore testing this approach in your own environment, there are two precautions to be aware of:
ClamAV bugsUnfortunately, ClamAV is not written in a type-safe programming language, and, as a result, has had many vulnerabilities. These vulnerabilities allow for malicious executables to exploit the ClamAV scanning engine in order to take over systems on which the malware is being analyzed. For this reason, we only perform this triage process on systems without critical data and almost always non-networked virtual machines. In general, you should take the same approach with all major anti-virus engines as they all have had numerous exploitable bugs reported in them.
Analysis on Windows systems / Shared VM foldersIf you perform this triage process on native Windows systems with AV installed or write your output files to a virtual machine shared folder where the host has AV installed, then it is very likely that the host AV will delete/quarantine any malicious files that you extract. This can cause you to miss infections as, by the time you get to run clamscan, the host AV has already deleted the files, which will prevent clamscan from alerting them to you.
What about dumpfiles?An experienced Volatility user may be wondering why we did not include dumpfiles into this triage process as dumpfiles can reconstruct cached files, including executables, directly from memory. The answer is that you can certainly include dumpfiles-based extraction into your triage process ---- but be aware of a few points first:
- This can make your output folder quite large
- One of the caches that dumpfiles examines contains a file contents as they appear on disk. This means that you will likely be scanning over packed files instead of the unpacked versions from memory
- dumpfiles can be considerably slower than the three plugins listed
This doesn't mean that you should always avoid dumpfiles, but instead just be cautious of its drawbacks in this particular situation.