In this blog post, we will cover how to automate the detection of previously identified malware through the use of three Volatility plugins along with ClamAV. Although this walk-through primarily focuses on Windows memory samples, at the end we explain how to port the approach to Linux and OS X samples.
Motivation
Although it would make our jobs quite interesting if every investigation involved analyzing new malware samples and families, the reality is that many malware investigations only require analyzing memory samples in order to verify (or hunt for) an infection by malware previously discovered in the wild. In these situations, the ability to quickly leverage existing tools and malware intelligence can lead to rapid identification of infected systems.
In this blog post, we document a methodology that we have successfully used in many investigations to confirm the presence, or absence, of known malware on a system. By following this workflow, you should be able to quickly sort through memory samples in your current investigations and to perform one aspect of proactive threat hunting in a highly efficient manner.
Setup
For the purpose of this blog post, we are going to analyze a memory sample that was previously available online. In particular, we are going to look at the
stuxnet.vmem file that was produced to accompany
The Malware Analyst's Cookbook. As you can likely guess, this memory sample came from a virtual machine infected with Stuxnet.
The real world scenarios where this applies are when you wish to verify if a system is infected, such as after receiving an AV or IDS alert, or when you wish to perform
proactive threat hunting in order to stay ahead of potential threats in your environment. In these situations, you will first need to acquire a memory sample of the system you wish to analyze. A full discussion of this process is outside the scope of this blog post, but Chapter 4 of the
Art of Memory Forensics provides extensive detail on the topic.
The remainder of this blog post assumes that you have a memory sample that was acquired using a stable tool and/or method.
Plugin Usage
The goal of this section is to showcase how, with only three Volatility plugins, an analyst can quickly extract a majority of the executables loaded into memory at the time of acquisition.
dlldump
To start, the
dlldump plugin will be used to extract the main application executable and all* loaded DLLs inside of each** process:
$ python vol.py -f stuxnet.vmem --profile=WinXPSP2x86 dlldump —memory -D stuxout/
Volatility Foundation Volatility Framework 2.5
Process(V) Name Module Base Module Name Result
---------- -------------------- ------------- -----------------------
0x820df020 smss.exe 0x048580000 smss.exe OK: module.376.22df020.48580000.dll
0x820df020 smss.exe 0x07c900000 ntdll.dll OK: module.376.22df020.7c900000.dll
0x821a2da0 csrss.exe 0x04a680000 csrss.exe OK: module.600.23a2da0.4a680000.dll
0x821a2da0 csrss.exe 0x07c900000 Error: DllBase is paged
0x821a2da0 csrss.exe 0x075b40000 CSRSRV.dll OK: module.600.23a2da0.75b40000.dll
0x821a2da0 csrss.exe 0x077f10000 GDI32.dll Error: DllBase is paged
0x821a2da0 csrss.exe 0x07e720000 sxs.dll Error: DllBase is paged
0x821a2da0 csrss.exe 0x077e70000 RPCRT4.dll Error: DllBase is paged
0x821a2da0 csrss.exe 0x077dd0000 ADVAPI32.dll Error: DllBase is paged
0x821a2da0 csrss.exe 0x077fe0000 Secur32.dll Error: DllBase is paged
0x821a2da0 csrss.exe 0x075b50000 basesrv.dll Error: DllBase is paged
0x821a2da0 csrss.exe 0x07c800000 KERNEL32.dll Error: DllBase is paged
0x821a2da0 csrss.exe 0x07e410000 USER32.dll OK: module.600.23a2da0.7e410000.dll
0x821a2da0 csrss.exe 0x075b60000 winsrv.dll OK: module.600.23a2da0.75b60000.dll
0x81da5650 winlogon.exe 0x001000000 winlogon.exe OK: module.624.1fa5650.1000000.dll
<snip>
In the above invocation of
dlldump, we set two options. The first,
-D, specifies the directory in which to extract the executables. We will have each extraction plugin write to the same directory in order make running ClamScan easier. The second option,
--memory, is likely only familiar to power Volatility users. The purpose of this flag is to instruct Volatility to extract all regions of memory around PE files instead of just relying on the file metadata in the PE header that specifies the original size of each section. This helps substantially when extracting packed executables from memory as you can get data both from the original file as well as the unpacked executable.
* There can be hidden DLLs. The next plugin,
malfind, will take care of that.
** If a process is hidden from the active process list, then
dlldump will not, by default, find it. This means the hidden executable and any DLLs it loaded will not be extracted using the above invocation. In this situation, you would want to use
psxview to find the hidden process - but realize that the fact that a process is hidden already mean its malicious, and you do not need ClamScan to re-verify this for you.
malfind
The next plugin that we will use is malfind, which is a plugin that searches for malicious executables (usually DLLs) and shellcode inside of each process. Explaining the precise details of how malfind works is outside the scope of this post and not relevant in a triage situation - but again consult
The Art of Memory Forensics if you want all the details. The following shows how to run malfind against the stuxnet sample:
$ python vol.py -f stuxnet.vmem --profile=WinXPSP2x86 malfind -D stuxout/
Volatility Foundation Volatility Framework 2.5
<snip>
Process: services.exe Pid: 668 Address: 0x13f0000
Vad Tag: Vad Protection: PAGE_EXECUTE_READWRITE
Flags: Protection: 6
0x013f0000 4d 5a 90 00 03 00 00 00 04 00 00 00 ff ff 00 00 MZ..............
0x013f0010 b8 00 00 00 00 00 00 00 40 00 00 00 00 00 00 00 ........@.......
0x013f0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x013f0030 00 00 00 00 00 00 00 00 00 00 00 00 08 01 00 00 ................
0x013f0000 4d DEC EBP
0x013f0001 5a POP EDX
0x013f0002 90 NOP
0x013f0003 0003 ADD [EBX], AL
0x013f0005 0000 ADD [EAX], AL
0x013f0007 000400 ADD [EAX+EAX], AL
0x013f000a 0000 ADD [EAX], AL
0x013f000c ff DB 0xff
<snip>
By default, for each suspicious memory region that malfind encounters, it will print attributes about the region such as which process it is mapped in, the starting and ending address of the region, and a hexdump and disassembly of the bytes at the beginning of the suspicious region. Since we added the -D flag to malfind, it will also extract each suspicious region into a separate file under our specified output directory.
By running the file command on just the files that malfind extracted, we have a good idea that we have already found the malware:
$ file stuxout/*
stuxout/process.0x81c47c00.0x1000000.dmp: PE32 executable (GUI) Intel 80386, for MS Windows
stuxout/process.0x81c47c00.0x680000.dmp: data
stuxout/process.0x81c47c00.0x6f0000.dmp: data
stuxout/process.0x81c47c00.0x80000.dmp: PE32 executable (DLL) (GUI) Intel 80386, for MS Windows, UPX compressed
stuxout/process.0x81c47c00.0x870000.dmp: PE32 executable (DLL) (GUI) Intel 80386, for MS Windows, UPX compressed
stuxout/process.0x81c498c8.0x1000000.dmp: PE32 executable (GUI) Intel 80386, for MS Windows
stuxout/process.0x81c498c8.0x80000.dmp: PE32 executable (DLL) (GUI) Intel 80386, for MS Windows, UPX compressed
stuxout/process.0x81e61da0.0xb70000.dmp: data
stuxout/process.0x81e61da0.0xbf0000.dmp: data
stuxout/process.0x81e61da0.0xd00000.dmp: PE32 executable (DLL) (GUI) Intel 80386, for MS Windows, UPX compressed
stuxout/process.0x82073020.0x13f0000.dmp: PE32 executable (DLL) (GUI) Intel 80386, for MS Windows, UPX compressed
stuxout/process.0x82073020.0x940000.dmp: data
stuxout/process.0x820ec7e8.0x2550000.dmp: data
stuxout/process.0x821a2da0.0x7f6f0000.dmp: data
In particular, we see multiple files whose file type is listed as a Windows executable. We also see several extracted regions whose file type was unknown (data), which means the region was either a false positive flagged by malfind or injected shellcode. Fortunately, since we are just using these plugins for automated triage, we can safely move on without examining each region individually.
moddump
By using
dlldump and
malfind, we have extracted every executable that Volatility will give us from userland (process memory) without having to manually dig ourselves. For the last plugin invocation, we will turn to
moddump in order to extract all* the executables (drivers) from kernel memory as well.
$ python vol.py -f stuxnet.vmem --profile=WinXPSP2x86 moddump —memory -D stuxout/
Volatility Foundation Volatility Framework 2.5
Module Base Module Name Result
----------- -------------------- ------
0x0804d7000 ntoskrnl.exe OK: driver.804d7000.sys
0x0806d0000 hal.dll OK: driver.806d0000.sys
0x0f7470000 update.sys OK: driver.f7470000.sys
0x0f89ba000 usbehci.sys OK: driver.f89ba000.sys
0x0f8a1a000 HIDPARSE.SYS OK: driver.f8a1a000.sys
0x0f8b5a000 CmBatt.sys OK: driver.f8b5a000.sys
0x0f855a000 pci.sys OK: driver.f855a000.sys
0x0f89aa000 usbuhci.sys OK: driver.f89aa000.sys
0x0bf800000 win32k.sys OK: driver.bf800000.sys
0x0f87fa000 raspptp.sys OK: driver.f87fa000.sys
0x0f89c2000 TDI.SYS OK: driver.f89c2000.sys
0x0b2c23000 ipnat.sys OK: driver.b2c23000.sys
0x0b23ce000 wdmaud.sys OK: driver.b23ce000.sys
0x0f84e5000 SCSIPORT.SYS OK: driver.f84e5000.sys
In this invocation, we have instructed Volatility to locate and extract all* kernel drivers from kernel memory into our output folder.
* moddump relies on the linked list of kernel modules in order to locate each extracted module. This list is susceptible to manipulation by malware and kernel-level malware can certainly run without being present in this list - in fact, it can even run without having any driver file associated with the malware at all (these are known as 'orphan threads' in Volatility terminology). We will avoid that rabbit hole in this blog post though.
Detection with ClamScan
So far we have leveraged
dlldump,
malfind, and
moddump to automatically locate and extract a wide variety of executables from memory. At this point, we want to check to see if any known malware is on the system, and to do so we will use ClamAV. For those unaware, ClamAV is an open source anti-virus engine with millions of signatures for known malware. It is command-line based and very easy to automate.
The following invocation of clamscan shows running it against all the files in the
stuxout directory, which is the directory that contains every executable extracted by the previous three plugins:
$ clamscan stuxout/ | grep -v ": OK$"
stuxout/process.0x81c47c00.0x1000000.dmp: Win.Trojan.5873027-1 FOUND
stuxout/process.0x81c47c00.0x6f0000.dmp: Win.Worm.Stuxnet-49 FOUND
stuxout/process.0x81c498c8.0x1000000.dmp: Win.Trojan.5873027-1 FOUND
stuxout/driver.f895a000.sys: Win.Trojan.Rootkit-8720 FOUND
stuxout/process.0x81e61da0.0xb70000.dmp: Win.Worm.Stuxnet-49 FOUND
stuxout/process.0x81c47c00.0x80000.dmp: Win.Worm.Stuxnet-49 FOUND
stuxout/process.0x81c498c8.0x80000.dmp: Win.Worm.Stuxnet-49 FOUND
----------- SCAN SUMMARY -----------
Known viruses: 4514519
Engine version: 0.98.7
Scanned directories: 1
Scanned files: 1262
Infected files: 7
Data scanned: 614.46 MB
Data read: 631.82 MB (ratio 0.97:1)
Time: 196.320 sec (3 m 16 s)
In this output it is immediately obvious that we have a problem as 6 process (userland) files matched AV signatures. Furthermore, one of the drivers extracted from kernel memory matched a signature as well. While ClamScan will occasionally throw a false positive, the fact that multiple signatures matched multiple extracted files very likely means that we have an actual infection. In this example, we can see that ClamAV correctly identifies several of the malicious regions as being associated with Stuxnet.
Advantages to this approach
Speed
The fact that we only need to run three Volatility plugins, all with static command line options, makes this a very fast process. Since none of the plugins require scanning the entire memory sample, each individual plugin should finish in less than one minute.
ClamAV can take anywhere from 1 minute to 10 minutes to scan all files extracted from memory samples that we regularly encounter.
Scalability
Since the entire triage process described in the writeup can be scripted and will almost always be finished in less than 15 minutes, this approach to detecting known malware is highly scalable.
Defeating Packers
One of the main advantages to extracting malware through memory forensics is that the malware will be in its unpacked form (ignoring VM-based and other advanced packers). This means that the anti-virus engine will be able to scan the actual contents of the malicious files and not just its obfuscated, outer shell. This really helps when scanning with Yara rules as well.
Memory-Only Malware & Injected Code
Through the use of malfind, we are able to find code injected into memory that may never exist on disk and that may have been injected in ways that traditional endpoint security tools would have missed.
No Need to Upload Samples
By running your testing local, you bypass the need to upload potential targeted malware to services such as VirusTotal or to your anti-virus vendor.
No OS Internals Needed
Performing this type of triage requires no advanced memory forensics or operating systems internals knowledge - all you have to do is run the plugins followed by clamscan.
Disadvantages to this approach
While highly useful, this approach is not 100% effective. The following sections describe some of the drawbacks.
Known Only
As made clear in the post title, this approach will only detect known malware. If the organizations originally targeted by Stuxnet had applied this approach to their systems, they would not have detected it as there would be no signatures for it.
This is an example of where deep memory forensics comes into play as it can detect malware in a generic fashion - without the need for any signatures or malware-specific knowledge.
MHL's blog post does a great job of showing how effective Volatility's generic anomaly detection plugins are against Stuxnet.
Not everything will be mapped
A second downfall to this approach is that the malicious code might not be resident in memory at the time of the acquisition - it will either in a file on disk or scattered throughout the page file. This situation requires deeper analysis beyond simple triage in order to verify the malware's presence.
ClamAV isn't Perfect
No security tool is perfect, and ClamAV is no exception to this rule. Just because ClamAV does not flag any executables does NOT mean that they are all actually legitimate. In general, we use ClamAV for quick, positive identification of known malware - not to gain confidence that no malware is on the system.
Warnings / Caution Points
Before testing this approach in your own environment, there are two precautions to be aware of:
ClamAV bugs
Unfortunately, ClamAV is not written in a type-safe programming language, and, as a result, has had many vulnerabilities. These vulnerabilities allow for malicious executables to exploit the ClamAV scanning engine in order to take over systems on which the malware is being analyzed. For this reason, we only perform this triage process on systems without critical data and almost always non-networked virtual machines. In general, you should take the same approach with all major anti-virus engines as they all have had numerous exploitable bugs reported in them.
Analysis on Windows systems / Shared VM folders
If you perform this triage process on native Windows systems with AV installed or write your output files to a virtual machine shared folder where the host has AV installed, then it is very likely that the host AV will delete/quarantine any malicious files that you extract. This can cause you to miss infections as, by the time you get to run clamscan, the host AV has already deleted the files, which will prevent clamscan from alerting them to you.
What about dumpfiles?
An experienced Volatility user may be wondering why we did not include
dumpfiles into this triage process as dumpfiles can reconstruct cached files, including executables, directly from memory. The answer is that you can certainly include
dumpfiles-based extraction into your triage process ---- but be aware of a few points first:
- This can make your output folder quite large
- One of the caches that dumpfiles examines contains a file contents as they appear on disk. This means that you will likely be scanning over packed files instead of the unpacked versions from memory
- dumpfiles can be considerably slower than the three plugins listed
This doesn't mean that you should always avoid dumpfiles, but instead just be cautious of its drawbacks in this particular situation.
Linux & OS X
This same process can be accomplished with the Linux and OS X counterpart plugins to the plugins used against the Windows stuxnet sample. Please see the official
Volatility Cheat Sheet (or the Art of Memory Forensics) for information on these plugins.
Conclusion
Hopefully this blog post has shown you a method that you can bring to your organization and immediately start triaging memory samples. If you have any questions on this process or the post itself, then please feel free to reach out - andrew @@@ AT @@@@ dfir.org -
PGP or through Twitter -
@attrc -
@volatility.
