Wednesday, June 3, 2015

Volshell Quickie: The Case of the Missing Unicode Characters

The other day someone reached out to me because they had a case that involved files with Arabic names.  Unfortunately the filenames were only question marks when using filescan or handles, so I set out to figure out why.

In order to figure out why, I created a few files with Hebrew names (which I can read and write, so I can verify if it is correct) and Arabic names (which was just me tapping on the keyboard, so they don't say anything). After creating them, I interacted with them to make sure they'd show up in filescan. Below you can see the filescan results:
[snip] $ python -f Win7x86.vmem --profile=Win7SP1x86 filescan 0x000000003d7008d0 16 0 RW-rw- \Device\HarddiskVolume2\Users\user\Desktop\????.txt 0x000000003ddfef20 18 1 RW-r-- \Device\HarddiskVolume2\Windows\Tasks\SCHEDLGU.TXT 0x000000003def9340 16 0 RW-r-- \Device\HarddiskVolume2\Users\user\Desktop\????????????????????.txt [snip]
In order to understand how the filename is output, you can look at the code in filescan:
for file in data: header = file.get_object_header() self.table_row(outfd, file.obj_offset, header.PointerCount, header.HandleCount, file.access_string(), str(file.file_name_with_device() or ''))
The function file_name_with_device() is defined in volatility/plugins/overlays/windows/ and the relevant part is highlighted in red:
class _FILE_OBJECT(obj.CType, ExecutiveObjectMixin): """Class for file objects""" def file_name_with_device(self): """Return the name of the file, prefixed with the name of the device object to which the file belongs""" name = "" if self.DeviceObject: object_hdr = obj.Object("_OBJECT_HEADER", self.DeviceObject - self.obj_vm.profile.get_obj_offset("_OBJECT_HEADER", "Body"), self.obj_native_vm) if object_hdr: name = "\\Device\\{0}".format(str(object_hdr.NameInfo.Name or '')) if self.FileName: name += str(self.FileName) return name
So we can take a look at this in volshell:
1 $ python -f Win7x86.vmem --profile=Win7SP1x86 volshell 2 [snip] 3 >>> file = obj.Object("_FILE_OBJECT", offset = 0x000000003d7008d0, vm = addrspace().base, native_vm = addrspace()) 4 >>> print file.FileName 5 \Users\user\Desktop\????.txt 6 7 >>> file2 = obj.Object("_FILE_OBJECT", offset = 0x000000003def9340, vm = addrspace().base, native_vm = addrspace()) 8 >>> print file2.FileName 9 \Users\user\Desktop\????????????????????.txt
On line 3 we create a _FILE_OBJECT object. We know the offset where this object resides (0x000000003d7008d0) from filescan. Since this object was obtained from the physical address space, we specify this by setting vm to addrspace().base, or the physical layer, since this is a raw memory sample. Since the _FILE_OBJECT's native address space is virtual, we specify this as well: native_vm = addrspace(). At this point we have instantiated a _FILE_OBJECT in a variable called "file". We then print out the FileName member on line 5 and see its output on line 6. We follow the same process for the second file, except we save the object as "file2". As you can see, the output is not very helpful. So now we need to know what type of member, FileName is. In order to accomplish this, we need to look at the vtypes in the volatility/plugins/overlays/windows/ file:
'_FILE_OBJECT' : [ 0x80, { 'Type' : [ 0x0, ['short']], [snip] 'FileName' : [ 0x30, ['_UNICODE_STRING']], [snip]
We have some functionality added to this type in volatility/plugins/overlays/windows/
class _UNICODE_STRING(obj.CType): [snip] def v(self): """ If the claimed length of the string is acceptable, return a unicode string. Otherwise, return a NoneObject. """ data = self.dereference() if data: return unicode(data) return data def dereference(self): length = self.Length.v() if length > 0 and length <= 1024: data = self.Buffer.dereference_as('String', encoding = 'utf16', length = length) return data else: return obj.NoneObject("Buffer length {0} for _UNICODE_STRING not within bounds".format(length)) [snip] def __format__(self, formatspec): return format(self.v(), formatspec) def __str__(self): return str(self.dereference()) [snip]
We know that the file_name_with_device() function uses str() in order to transform the _UNICODE_STRING into something readable and if we look at the overridden __str__() operator in the above code, we see that it uses the dereference() function. The dereference() function never casts the data as unicode, however, so the data is printed incorrectly. If we look at the above v() function, we see that there is a call to dereference() and that the resulting data is case as unicode, so let's see if we get valid data back by calling that function instead:
>>> print file.FileName.v() \Users\user\Desktop\שלום.txt >>> print file2.FileName.v() \Users\user\Desktop\تهحححتهحححتهحححتهححح.txt
Success! So let's modify the __str__() operator to use v() instead and see if that fixes filescan:
[snip] def __str__(self): return str(self.v()) [snip]
Now let's examine the filescan data:
0x000000003d7008d0 16 0 RW-rw- \Device\HarddiskVolume2\Users\user\Desktop\שלום.txt 0x000000003ddfef20 18 1 RW-r-- \Device\HarddiskVolume2\Windows\Tasks\SCHEDLGU.TXT 0x000000003def9340 16 0 RW-r-- \Device\HarddiskVolume2\Users\user\Desktop\تهحححتهحححتهحححتهححح.txt
Success! Just to make dually sure, I then created a user with a Hebrew name: גלידה, and created some files with Hebrew characters as well. If we look back to the file_name_with_device() function, you'll see that the complete file path is populated by using the NameInfo optional header (name = "\\Device\\{0}".format(str(object_hdr.NameInfo.Name or ''))). If you look in the volatility/plugins/overlays/windows/ file, you'll see the following definition:
class _OBJECT_HEADER(windows._OBJECT_HEADER): [snip] optional_header_mask = (('CreatorInfo', '_OBJECT_HEADER_CREATOR_INFO', 0x01), ('NameInfo', '_OBJECT_HEADER_NAME_INFO', 0x02), ('HandleInfo', '_OBJECT_HEADER_HANDLE_INFO', 0x04), ('QuotaInfo', '_OBJECT_HEADER_QUOTA_INFO', 0x08), ('ProcessInfo', '_OBJECT_HEADER_PROCESS_INFO', 0x10))
Now we know the object to find in the types file in order to figure out what type Name is. Look in volatility/plugins/overlays/windows/
'_OBJECT_HEADER_NAME_INFO' : [ 0x10, { 'Directory' : [ 0x0, ['pointer', ['_OBJECT_DIRECTORY']]], 'Name' : [ 0x4, ['_UNICODE_STRING']], 'ReferenceCount' : [ 0xc, ['long']], } ],
Since Name is of the same type (_UNICODE_STRING), we should be covered. In the words of Al Bundy, "Let's rock":
$ python -f Win7x86.vmem --profile=Win7SP1x86 filescan [snip] 0x000000003e8fb1c0 2 0 RW-rw- \Device\HarddiskVolume1\Users\גלידה\Desktop\עצם.txt 0x000000003f83c038 2 0 RW-rw- \Device\HarddiskVolume1\Users\גלידה\AppData\Roaming\Microsoft\Windows\Recent\עצם.lnk [snip]
Success! A lot of other objects use _UNICODE_STRINGs, including mutants, registry paths and symbolic links. So this was an important fix.

Changes have already been reflected in the master branch of Volatility. We hope that you have enjoyed this not so short, quickie ;-)

Friday, May 15, 2015

Using mprotect(.., .., PROT_NONE) on Linux

After deciding to revisit some old code of mine (ok, very old), I realized that there was something different about how Linux was allocating pages of data I wanted to hide.   At first, I was glad that I couldn't see the data using yarascan, but then I realized that I was unable to access the memory regions at all in linux_volshell to verify that they were, in fact, obfuscated. So I decided to take a look at using a smaller, stripped down program. Below is one such example, comments are included to explain what is happening:

int main( int argc, char *argv[]){ // pid: the process ID of this process // so we can print it out int pid; pid = getpid(); //size: an integer to hold the current page size int size; size = getpagesize(); //[1] create two pointers in order to allocate //memory regions char *buffer; char *buffer2; //unprotected buffer: //allocate memory using mmap() buffer2 = (caddr_t) mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 0,0); //[2] put some characters in the allocated memory //we're setting these characters one at a time in order //to avoid our strings being detected from the binary itself buffer2[0] = 'n'; buffer2[1] = 'o'; buffer2[2] = 't'; buffer2[3] = ' '; buffer2[4] = 'h'; buffer2[5] = 'e'; buffer2[6] = 'r'; buffer2[7] = 'e'; //protected buffer: //allocate memory with mmap() like before buffer = (caddr_t) mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 0,0); //[2] put some characters in the allocated memory //we're setting these characters one at a time in order //to avoid our strings being detected from the binary itself buffer[0] = 'f'; buffer[1] = 'i'; buffer[2] = 'n'; buffer[3] = 'd'; buffer[4] = ' '; buffer[5] = 'm'; buffer[6] = 'e'; //[3] protect the page with PROT_NONE: mprotect(buffer, size, PROT_NONE); //[4] print PID and buffer addresses: printf("PID %d\n", pid); printf("buffer at %p\n", buffer); printf("buffer2 at %p\n", buffer2); //spin until killed so that we know it's in memory: while(1); return 0; }
Now we'll just compile the above program and run it. In short, the above program [1] creates two buffers, [2] places characters in these buffers, [3] then calls mprotect() on one of them with PROT_NONE; [4] the program then prints out its process ID and the virtual addresses of the aforementioned buffers:

$ ./victim PID 29620 buffer at 0x7f2bec9a1000 buffer2 at 0x7f2bec9a2000
Let's set up a config file so that we don't have to type as much:

$ cat linux.config [DEFAULT] LOCATION=file:///Path/to/Virtual%20Machine/Linux%20Mint%20Cinnamon/Mint%2064-bit-d583834d.vmem PROFILE=LinuxLinuxMintCinnamonx64x64 PID="29620"
If we try to search for the strings we placed in the buffers we get:

$ python --conf-file=linux.config linux_yarascan -Y "not here" Volatility Foundation Volatility Framework 2.4 Task: victim pid 29620 rule r1 addr 0x7f2bec9a2000 0x7f2bec9a2000 6e 6f 74 20 68 65 72 65 00 00 00 00 00 00 00 00 0x7f2bec9a2010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0x7f2bec9a2020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [snip] $ python --conf-file=linux.config linux_yarascan -Y "find me" Volatility Foundation Volatility Framework 2.4

Well that was disappointing, we couldn't find the "find me" string. What you would expect, is to be able to access the memory contents using Volatility. Let's use the linux_volshell plugin to explore the victim process' memory and see if we can access the memory addresses (given to us from the program at run time) directly. First we'll examine the contents at 0x7f2bec9a2000:

$ python --conf-file=linux.config linux_volshell Volatility Foundation Volatility Framework 2.4 >>> Current context: process victim, pid=29620 DTB=0x3cfb1000 >>> db(0x7f2bec9a2000) 0x7f2bec9a2000 6e 6f 74 20 68 65 72 65 00 00 00 00 00 00 00 00 0x7f2bec9a2010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0x7f2bec9a2020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0x7f2bec9a2030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0x7f2bec9a2040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0x7f2bec9a2050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0x7f2bec9a2060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0x7f2bec9a2070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
As we see, we have the contents that we'd expect. Now to try to access the other memory location:

>>> db(0x7f2bec9a1000) Memory unreadable at 7f2bec9a1000
We see that we've been rejected. After a bit of investigation, we see that the point of failure is in the entry_present() function in the address space. In order to find the value of the entry that failed, we'll print it out. First we'll add a print statement to entry_present():

def entry_present(self, entry): if entry: if (entry & 1): return True arch = self.profile.metadata.get('os', 'Unknown').lower() # The page is in transition and not a prototype. # Thus, we will treat it as present. if arch == "windows" and ((entry & (1 << 11)) and not (entry & (1 << 10))): return True # we want a valid entry that hasn't been found valid: print hex(entry) return False

Now let's see what the entry is:
$ python --conf-file=linux.config linux_yarascan -Y "find me" Volatility Foundation Volatility Framework 2.4 0x2681d160
So let's look a little closer at the page table entry:

>>> print "{0:b}".format(0x2681d160) 100110100000011101000101100000
The way we read this is from right to left.  The entry_present() function checks the 0th bit to see if the entry is present. In this case the bit is not set (0); therefore this function returns False.

At this point, I decided to take a look at the Linux source code to see how mprotect() is actually implemented. As it turns out, the _PAGE_PRESENT bit is cleared when mprotect(...PROT_NONE) is called on a page and the _PAGE_PROTNONE bit is set [3]. Looking at how _PAGE_PROTNONE is defined [4][5][6] we'll see that it's actually equivalent to the global bit (8th bit) [1][2]. So let's look at our page table entry again, we'll notice that the 8th bit is indeed 1:

>>> print "{0:b}".format(0x2681d160) 100110100000011101000101100000

So let's patch the entry_present() function with our findings:

def entry_present(self, entry): if entry: if (entry & 1): return True arch = self.profile.metadata.get('os', 'Unknown').lower() # The page is in transition and not a prototype. # Thus, we will treat it as present. if arch == "windows" and ((entry & (1 << 11)) and not (entry & (1 << 10))): return True # Linux pages that have had mprotect(...PROT_NONE) called on them # have the present bit cleared and global bit set if arch == "linux" and ((entry & (1 << 8))): return True return False

Let's see if we get anything back:
$ python --conf-file=linux.config linux_yarascan -Y "find me" Volatility Foundation Volatility Framework 2.4 Task: victim pid 29620 rule r1 addr 0x7f2bec9a1000 0x7f2bec9a1000 66 69 6e 64 20 6d 65 00 00 00 00 00 00 00 00 00 0x7f2bec9a1010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0x7f2bec9a1020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [snip]

Success! So now we're able to see the "forbidden" string that we couldn't access before.  This goes to show that sometimes you have to dig a little into "why" something is not working as expected. In this case, we lucked out and had source code to examine, but sometimes things are not as easy as all that. Both and have been patched in order to accommodate memory sections that have had mprotect() called on them with PROT_NONE.


[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming

Monday, March 16, 2015

Windows Malware and Memory Forensics Training in the UK

Windows Malware and Memory Forensics Training by The Volatility Project is the only memory forensics course officially designed, sponsored, and taught by the Volatility developers. One of the main reasons we made Volatility open-source is to encourage and facilitate a deeper understanding of how memory analysis works, where the evidence originates, and how to interpret the data collected by the framework's extensive set of plugins. Now you can learn about these benefits first hand from the developers of the most powerful, flexible, and innovative memory forensics tool.

Our last public class in the UK sold out, so we are back again! This training is organized and hosted by CCL Group, the UK's largest provider of digital forensics.


Location: Stratford-Upon-Avon, United Kingdom
Cost: £ 2595 + VAT (lunch provided)
​Dates: Monday June 1st - Friday June 5th 2015
Times: 9 AM - 5 PM daily
Instructors: Michael Ligh, Jamie Levy, Andrew Case 
View this course's full page


For bookings, please contact or call 01789 261200 (UK).

Other Courses

The following courses are also open for registration. 

Monday, February 2, 2015

Advice from Det. Michael Chaves on Memory Forensics, KnTDD, and POS Malware

The following story was shared by Detective Michael Chaves. It describes how he's used Volatility, KnTDD, and memory forensics over the past year to investigate POS breaches at local businesses. Kudos to Michael for applying his skills in an effective and meaningful way, then taking the time to share experiences with others. Without a doubt, detectives in every police department have or will encounter situations like Michael describes.
It's been about year since I've taken the Volatility Windows Malware and Memory Forensics Training in NYC.  I wanted to take this time to share some of my experiences to hopefully help examiners/investigators early on in their exposure to Volatility and to help identify unknown malware.  Over the past 10 months I have responded to about a dozen POS breaches at local businesses; mainly liquor stores and restaurants. These breaches are identified rather quickly from local banks that call me with the details and I usually respond to a location within 2 days. It should come as no surprise that I have yet to respond to a location that was anywhere near being PCI compliant.

The large majority of POS terminals were running Windows XP some with SP2, most with SP3.  Two machines were even running Windows 2K and all had direct connections to the Internet.  Antivirus IF present was either out of date or turned off.  I still have yet to see a firewall present or any security policy in place.  My RAM capture tool of choice is Kntdd and I’ll use FTK Imager Lite to obtain all registry files, App Data directory, $log, $MFT and prefetch directory.  I carry with me several portable drives to make the acquisition from each POS location in the shortest amount of time possible as the store still needs to process customer purchases. 

For most of these breaches I have been able to identify the malware pretty easily.  I usually begin by running, pslist, psscan, psxview and connections (if supported).  In the majority of the breaches, the processes were not hidden and had an active process listed, usually called by ‘explorer’.  If I was not able to easily identify the malware process, I’d run dlllist to locate any programs running from odd locations followed by malfind  and yarascan.  Once I have identified a suspect process, I’d dump that process usually by procdump or dlldump.

During the early part of the investigation, I am not too concerned how the malware works or what the Initial Infection Vector was.   I want to know where the credit cards are going and how are they getting out.  I’d run strings on the exported out suspected malware file and I would generally find, the URL used to send out the cards via POST, an e-mail address associated with the malware and/or IP addresses.  The majority of my cases the POST command was used to send out the cards, in others it was via SMTP.  In 5 of my 12 breaches, the malware family was JACKPOS or Alina variants.  I will search the Internet on file names, URL’s and artifacts that usually result with great write ups that show what I may be investigating, as well researching with Virustotal.   It should also be noted that there have been a few times that I sought assistance from the Volatility community and other students from the NYC class.  They have been extremely receptive to my questions and information provided to was invaluable!

I realize there are many readers out there that have a far greater understanding of malware and memory forensics.  I’m slowly getting there, but I hope this helps out the people just beginning, like I am/was by describing my workflow and perhaps give confidence to some that may otherwise doubt their ability.

Tuesday, January 27, 2015

Incorporating Disk Forensics with Memory Forensics - Bulk Extractor

In this post we will take our first look at a tool that is primarily used for disk forensics and show how it can be useful during memory forensics analysis as well. In the coming weeks we will have several follow on posts highlighting other tools and techniques.


As you are likely aware, much of the information that is recoverable from disk and the network is also recoverable from memory - assuming you get a valid sample within a proper time frame.  If these conditions are met, then many forensics tools not usually associated with memory forensics can be incorporated into the memory analysis process. The artifacts recovered by these tools are often ones that memory tools do not focus on and are ill-suited to handle. Through incorporation of such tools and techniques, investigators can fully leverage all information available in volatile memory.

In many real-world investigation scenarios, you may only ever get a memory sample - see Jared's OMFW 2014 presentation for an excellent example of this [6] - and even less likely will you get full network flow (PCAP) during the time frame of an attack. This is unfortunate as the network often has information vital to an investigation, such as which servers were used to attack the network, which systems attackers laterally moved to, which domain(s) malware was downloaded from, and what commands were sent by a C&C server to local nodes. Even without a PCAP, all hope is not lost though, as network data must traverse main memory in order to be sent and received by applications.** This leaves the opportunity for historical network information to be left in memory long after it was active.

This idea was explored in great detail by Simpson Garfinkel and his co-authors in their 2011 DFRWS paper 'Forensic carving of network packets and associated data structures' [5]. In the paper they discuss not only historical network data in volatile memory, but also how that information can be stored on disk through system swapping and hibernation. This approach has the advantage of potentially getting network data from even previous reboots of the system.

To demonstrate the validity of their approach and evaluate the research, new modules were added to the open source bulk_extractor [1, 2, 3] tool in order to automate extraction and processing.

Since the publication of this paper and the popularization of bulk_extractor, extracting network data from memory captures has become a technique used by many analysts and it has even appeared in leading college digital forensics courses [4].

** With the exception of hardware rootkits within NIC firmware. If you believe this type of malware is active on a system that you need to investigate then you should check out the KntDD acquisition tool [14]. It can safely acquire memory from select NICs as well as acquire memory from other hardware devices.

Bulk Extractor

bulk_extractor is a highly-optimized open source tool that can scan inputs (disk, memory captures, etc.) and automatically find a wide range of information useful to investigators, such as email addresses, URLs, domains, credit cards numbers, and more. The project's wiki documents all of its scanning features [7]. Due to its multi-threaded, C++ design, bulk_extractor can often process inputs and extract all features at the speed at which the input can be read, making it very useful for efficient analysis.

When bulk_extractor is finished processing a target it then produces a number of output files. The exact files produced depends on which scanners were activated. For most scanners, in particular the network-related ones, not only is there a file of the raw data recovered, but there is also a histogram produced that orders entries based on the number of times found. This can be interesting in a number of ways, such as looking for the most common email addresses to determine who a user being investigated communicated with frequently.

Carving Network Packets and Streams from Memory

One of bulk_extractor's most useful features to memory forensics, and the one this blog will focus on, is the ability to profile network data in a sample as well as produce a PCAP of all packets found.

The profiled data includes recovered IP addresses, which can be mapped to known-bad lists or researched through threat intelligence tools, and Ethernet frames, which can be used to determine which local systems were contacted. Both of these profiled data sets include a histogram file.

To get this set of information you can run bulk_extractor as:

bulk_extractor -E net -o <output directory> <memory image path>

This will only run the net scanner and quickly produce the results described above in the directory specified with -o. Depending on your time constraints and desired data recovered, you can also run with other options, such as:

bulk_extractor -e net -e email -o <output directory> <memory image path>

This will run both the net and email scanners, which adds several more output files of interesting forensics data, including information about all domains found as well as email addresses and URLs. This can obviously give deep insight into what network activity occurred on the system being analyzed - potential phishing email accounts, malware URLs and domains, attacker C&C infrastructure, and so on.

Analyzing the PCAP File

In many of our own investigations we have found the PCAP file produced by bulk_extractor to be invaluable. In the connection traces we have uncovered evidence of data exfiltration, including (partial) file contents, commands sent by attackers to malware on client systems, input and output of tools on remotely controlled cmd.exe sessions, re-directed HTTP traffic from web servers, and much more.

The number of packets in a created PCAP file is limited to what is available in memory and not overwritten. For systems with large amounts of RAM, this can be many megabytes of data. For systems with less RAM, the in-memory caches will obviously be smaller, which often leads to less remnant information.  As we will show with the following forensics challenges though, even systems with small amounts of RAM can still retain crucial, historical network data long after it was processed.

Forensics Challenges

To show the usefulness of extracting network data from memory samples, we ran bulk_extractor against several memory captures included with popular forensics challenges. We choose these challenges for several reasons. First, they include both Linux and Windows samples. This shows the usefulness of the technique across several operating systems. Second, these challenges include small memory captures while still containing very useful data. Finally, by choosing public memory samples, anyone can perform their own research and follow the steps shown in the blog.

DFRWS 2005 Challenge

The first challenge we will analyze is the DFRWS 2005 challenge [8]. This is one the most well-known challenges since it directly inspired what we now consider modern memory forensics.

In this challenge, investigators were given a full network capture as well as a memory capture both before and after the suspect's laptop crashed. After running bulk_extractor, with -E net chosen, against the first sample we obtain a PCAP file (packets.pcap in BE's output folder) that has 130 packets. This is in comparison to 6304 packets in the full network capture, meaning roughly 2% of the related packets were recovered from a 126MB memory capture (no, the size is not a typo).

While this number may seem incredibly small, analysis shows that the recovered network data has highly useful information. As seen in the following screenshot from the PCAP loaded into Wireshark, proof of the "Back Orifice" (BO2k) malware being used on the system is present. This also includes partial file names and commands used on the system:

As discussed in the challenge's solutions, these backdoor interactions were a key part of solving what the attacker did to the system. Assuming a PCAP wasn't given along with the challenge, which we will see is not always the case, then volatile memory would be all investigators had to rely on for network clues.

DFRWS 2008 Challenge

The second challenge we will discuss is the one from DFRWS 2008 [9]. This included a Linux memory memory sample along with an accompanying PCAP file. As with the previous challenge, we will compare the network data obtained from memory to the provided PCAP along with analyzing the data from memory. This memory capture was 284MB and 122 packets were recovered. This is in comparison to 10243 contained in the full network capture (~1%).

Of these 122 packets, there was quite a bit of interesting data. As documented in the winning solution to the challenge [10], an encrypted ZIP file was exfiltrated from the network. Portions of this exfiltration appear in the capture. Also, the challenge details a Perl script used to extfiltrate the data using HTTP cookies on particular domains. Both fragments of the Perl script as well as several cookie instances are contained within the memory capture. The following shows one of the malicious requests encoding data within a cookie.

Note that the real is not contacted and a server under attacker control in Malaysia ( is contacted instead. For more information on this Perl's scripts chunking and encoding of data, please read the winning submission.

Honeynet 2010 Challenge

Honeynet's 2010 Challenge [11] focused on a end-user infected with a banking trojan. In the challenge you were given only a memory sample of the victim system. The given memory sample was 512MB and bulk_extractor recovered 256 packets. From our analysis, we believe this is a fairly significant amount of packets that were sent or received during creation of the challenge, even though it is a small number.

As shown in the following Wireshark screenshot, one of the packets recovered includes the request that downloaded the ZBot malware:

This is the focal point of the investigation, and if you read the challenge's official solution [12], you will see that once the malicious domain and connection are determined, the rest of the investigation falls into place.  Through the use of bulk_extractor we can immediately find the request in-tact, whereas the solution required an intricate mixing of strings and Volatility to map the pieces together. It is also interesting to note that when trying to definitively piece together the network traffic the solution states:

"It’s not possible to state it for sure since no network dump is available."

As we have just learned, we do not need to be given a network capture in order to get verifiable and useful network data from memory.

Honeynet 2011 Challenge

The 2011 Honeynet Challenge was an investigation against a memory sample and disk image of a compromised server. No network capture was provided. The memory capture is 256MB and 347 packets were recovered. As discussed in the solutions' for this challenge, the initial foothold was gained on the server through a heap-based vulnerability in the Exim mail server. The following from the PCAP of recovered packets shows this exploit in action:

Besides the large buffer of AAAAA's used to fill the heap, you can also see the IP address ( of the attacking system as well as its destination port (25) against the local server. This is an immediate warning sign of an attack against the mail server.

Mapping Connections to Processes

As useful information is recovered from memory, investigators often want to determine which process or kernel component was responsible for sending or receiving the particular packets. Volatility provides two ways to do this. The first is through the use of the strings plugin. The strings plugin can map a physical offset in a memory capture (the offset in the capture file) to its owning process or kernel module. Unfortunately, the strings plugins is not very straightforward with a PCAP file as the offset in physical memory of the packet is not encoded within the PCAP's data**.

To get around this limitation, you can use the yarascan plugin to search for unique data within the packet(s) that you find interesting. To do this, pass the data as the -Y option to yarascan. Note that you can pass arbitrary strings, bytes, and hex values to yarascan and you are not just limited to searching readable text. As yarascan searches for your signature it will list any processes or kernel components that contain the data. This can then immediately point you to the area of code responsible for generating or receiving the packet, which can greatly focus your analysis.

For those who closely follow Volatility, you know that there are also the ethscan [15] and pktscan [16] plugins that can perform the mapping as well. Unfortunately, pktscan is no longer supported and ethscan can take substantially longer to produce results versus bulk_extractor. For these reasons and others we choose to incorporate bulk_extractor in our analysis instead of using only Volatility components. 

** Someone please let me know if I am wrong about this. I have not seen this documented anywhere for bulk_extractor and I am not sure its possible using the old PCAP format that BE writes the capture in. The newer PCAP-ng format allows for arbitrary comments though and a nice addition to BE would be writing to the new format with a comment of the physical offset of where the particular packet was found.


This post was written to highlight the power of carving network data from memory. When full packet capture is not available, or when no network data is available, the ability to extract network data from memory may be your only chance of recovering it. As shown, bulk_extractor is a highly capable tool for this task. Not only can it recover packets to a PCAP, but it can also be used to recover a wide range of other information as well as arbitrary strings or regular expressions that you configure it for.

We strongly encourage our readers who are currently not utilizing this capability to start to incorporate in into your memory forensics workflows. We believe you will see immediate usefulness of the data recovered and bulk_extractor makes recovery trivial.



Wednesday, December 31, 2014

Acquiring Memor(ies) from 2014

2014 is extremely volatile. Any minute now, it will be gone. Thus, we wanted to take a minute and preserve some of the more exciting memories. Specifically, we wanted to summarize how the memory forensics field and Volatility community has progressed this year.
  • Volatility 2.4 was released - our most stable and fully featured code base, supporting all of the major versions of Windows, Linux, and Mac. The release includes Windows 8 and Server 2012 support, despite the encrypted debugging structures. Windows 10 beta support is also available.
  • We also moved to Github, which makes it easier for other developers to submit patches and pull requests. A separate repository stores Linux and Mac profiles. The 2.5 release is also under way. 
  • Extraction of TrueCrypt cached passwords and master keys become even easier than it was before, with new plugins that execute a structured approach at locating and recovering the data. The capability was also ported to Linux, and a member of the community completed a similar plugin for dm-crypt.
  • Our 5-day hands-on Malware and Memory Forensics training course proliferated to over 10 events (public and private) across the United States, Europe, and Australia. Testimonials from attendees are available here and registration is open for classes in 2015.
  • The Volatility Foundation was established as a 501(c)(3) non-profit organization. A new website was created to aggregate the foundation's resources. Among other things, the site describes the various ways to get involved with the community. The Volatility(R) name also became a registered trademark of the Volatility Foundation.
  • Volexity, LLC became a corporate sponsor of the Volatility Foundation and named core developer Michael Hale Ligh as its CTO. Volexity is a security firm based out of the Washington, D.C. area that specializes in assisting organizations with threat intelligence, incident response, forensics, and trusted security advisory.
  • The Volatility Plugin Contest received an extremely generous donation from Facebook and generated an enormous amount of new capabilities from various talented developers and researchers. 
  • The Volatility Team partnered with GMG Systems, Inc. to offer KnTTools (incl. KnTDD) at a discounted rate to students in our training course. KnTTools is the most reliable, robust, and fully featured memory acquisition suite for Windows. 
  • Volatility and memory forensics were represented at almost all security and technology related conferences, including (but not limited to) Blackhat USA, Defcon, OMFW, OSDFC, API Cybersecurity, SecTor, Archc0n, Alabama Cyber Security Summit, National Cyber Crime Conference, RSA USA, BSides, and Recon.
  • Volatility was used by Det. Michael Chaves to track down high profile ATM skimmers. It was used by European law enforcement to produce the primary source of evidence that put away a child sex offender for over 10 years. It was used by the United States Government on various occasions to investigate cases involving espionage, cyber terrorism, and major botnet rings.
  • Volatility developer Andrew Case and Golden G. Richard III won best paper at DFRWS 2014 for In lieu of swap: Analyzing compressed RAM in Mac OS X and Linux.
  • We enjoyed a record breaking number of attendees at Open Memory Forensics Workshop (OMFW) 2014 and a serious group of awesome talks. Later that week, Next Generation Memory Forensics was presented at OSDFC
  • The Art of Memory Forensics was published in August, a 900-page book that covers Windows, Linux, and Mac topics in depth. To date, its the most thorough and illustrative written source of memory forensics knowledge. Please remember to check the errata page and also the sample memory images that are accompanied with lab questions and answers (free online download). 
  • A Reddit Ask Me Anything (AMA) was conducted on Art of Memory Forensics. We appreciate all the great questions!  
  • An entire chapter of Black Hat Python was devoted to using Volatility for offensive purposes during pentests (extracting password hashes, injecting code, etc). 
  • We presented some Volatility capabilities at Blackhat Arsenal and later created a YouTube channel with recordings of the demos. There was also a book signing in the Blackhat bookstore, which was an exciting way for us to meet new Volatility users in person.
 Thanks to all who played a part! We look forward to an even more productive 2015!

Wednesday, October 29, 2014

Announcing the 2014 Volatility Plugin Contest Results!

The competition this year was fierce! We received a total of nearly 30 plugins to the contest. Ranking the submissions was one of the hardest things we’ve had to do. Each plugin is unique in its own way and introduces a capability to open source memory forensics that didn’t previously exist. Although a few people will receive prizes for their work, the real winners in this contest are the practitioners and investigators in the community that perform memory forensics. We’re talking about the federal government who used Volatility on some of the nation’s most prominent cases and the law enforcement groups that used it as the primary tool to force a child pornographer into a guilty plea (see you in about 10 years, wish it were more!). We’re talking about Det. Michael Chaves who used memory forensics to help crack a case involving POS breaches that lead to losses of over $100K. We’re talking about all the analysts who rely on open source forensics to identify and track malicious code and threat actors in their networks and those of their clients. 

Needless to say, we're very proud of everyone who submitted to the contest. Also a huge thanks goes out to Facebook for doubling the contest's cash prizes and supporting the research and development  of open source memory forensics. 

Here are this year’s rankings:
  1. Dave Lasalle wins first place and his choice of $2500 or free training. Dave submitted 14 plugins for recovering Firefox and Chrome activity (history, search terms, cookies, downloads) from memory, carving Java IDX files, and using fuzzy hashing to whitelist injected code and API hooks. 
  2. Curtis Carmony wins second place and $1250 for his plugin to extract dm-crypt disk encryption keys from Linux (and potentially Android) memory dumps.
  3. Adam Bridge wins third place and $750 with editbox – a plugin to recover the text within edit controls of GUI applications on Windows (including but not limited to notepad contents, username and password fields, browser URL and search forms, etc). 
  4. Thomas Chopitea wins fourth place for his autoruns plugin that enumerates automatically starting applications on Windows systems – a common first step in many different types of investigations.
  5. Takahiro Haruyama wins fifth place with openioc_scan, a plugin that combines the flexibility of the IOC language with the power of Volatility to give analyst’s quick and easy malware triage capabilities.

Here is a detailed summary of the submissions. We've included a link to the respective submissions on the Volatility Foundation website for archive purposes, however we recommend getting the code from the author's own GitHub repositories if that option exists. If you have feedback for the authors, we're sure they'd love to hear your thoughts.

(1st) Dave Lasalle: Forensic Suite

Dave’s 14 plugins are immediately useful for various different scenarios, from tracking user activity to parsing special file formats and whitelisting injected code and API hooks.

Previously, if you needed to inspect a suspect or victim’s browsing activity from memory in a structured manner (i.e. not brute forcing with regular expressions), you were limited to the iehistory (Internet Explorer) plugin. Now you can do the same, and more, for Firefox and Chrome. The two browsers use sqlite3 databases, but due to several reasons (including paging), you’re not likely to succeed in carving complete sqlite3 files from memory. Dave’s plugins leverage his sqlite3 memory API, which handles missing chunks of database files gracefully.

Dave’s Twitter: @superponible
Dave’s GitHub:
Dave's Blog:
Dave’s Submission:

Most wanted follow up(s): A plugin to extract the most recent Internet Explorer history records.  Porting Firefox and Chrome plugins to Linux and Mac memory dumps. 

(2nd) Curtis Carmony: Dmcrypt

The dm_dump plugin brings an exciting new capability to open source memory forensics. In his own words, “given a memory dump from a Linux system using full disk encryption and access to the disk, the output of this plugin gives you the arguments to pass to the dmsetup command to remount the original unencrypted file system on a different machine.” In addition, Curtis provided support for Linux kernels 3.0 to 3.14 and instructions on how to extend Volatility’s profile generation mechanism for future systems.

A unique aspect of this plugin is that the data it recovers can only be found in RAM. As such, it accomplishes something that no form of disk or network forensics can do and it really showcases the power of memory forensics. Similar to the existing truecrypt plugins, the dm_dump plugin works by traversing the internal data structures used by device-mapper to keep track of its devices. Thus it pinpoints the data in memory without scanning for constants or patterns in key schedules.

Curtis’ GitHub:
Curtis’ Submission:

Most wanted follow up(s): Testing the methodology on Android disk encryption. 

(3rd) Adam Bridge: Editbox

Adam’s submission provides powerful new capabilities for tracking suspect user activity. It recovers text from EditBox controls in the GUI subsystem, with experimental support of ComboBox and ListBox. As a result, it can extract the following data types:
  • Notepad window.
  • Run dialog.
  • Username and server name fields of Remote Desktop Connection.
  • Address bar and search bar of Internet Explorer.
  • Search bar of Windows Media Player.
  • Username field of Create New Account wizard.
  • Password of Change Password dialog.
In general, it is effective on any applications that leverage Microsoft's Common Control APIs. This plugin is particularly interesting, because the data it recovers is not available anywhere besides RAM. On a multi-user system, there would be no way to collect the data this plugin enumerates without logging into each account and taking a screen shot.

Adam’s Twitter: @bridgeythegeek
Adam’s Submission:

Most wanted follow up(s): Integration of edit box labels into the screenshot plugin.

(4th) Thomas Chopitea: Autoruns

In Thomas' own words, "Finding persistence points (also called "Auto-Start Extensibility Points", or ASEPs) is a recurring task of any investigation potentially involving malware." The plugin currently covers several of the most common registry locations, including services, appinit DLLs, winlogin notification packages, and scheduled tasks. After finding ASEPs, the plugin matches them with running processes in memory.

Thomas’ Twitter: @tomchop
Thomas’ GitHub:
Thomas’ Blog:
Thomas’ Submission:

Most wanted follow up(s): Adding Linux and Mac support.

(5th) Takahiro Haruyama: OpenIOC Scan

This plugin combines the flexibility of the IOC language with the power of Volatility to give analyst’s quick and easy malware triage capabilities. Takahiro solved several problems that he (and most certainly other analysts) faced when using the existing tools, such as ability to automate the tasks outside of a GUI and scan for terms with regular expressions and case sensitivity. Takahiro’s blog (below) shows several practical examples of quickly finding malicious code in memory. We’re really excited for investigators to start taking advantage of Takahiro’s work.

Takahiro’s Twitter: @cci_forensics
Takahiro’s Blog:
Takahiro’s Submission:

Most wanted follow up(s): A repository of memory related indicators. Also for performance reasons, using the Registry API to scan for keys, values, etc.

The following submissions appear in the order they were received. As previously mentioned, everyone succeeded in solving a specific problem that they (and undoubtedly others) faced. For this, they deserve huge props. We look forward to seeing future work by these authors!

Monnappa KA: Gh0stRat Decryption  

Monnappa’s plugin focuses on detecting and analyzing Gh0stRat in memory. In his own words, “Gh0stRat is a RAT (Remote Access Trojan) used in many APT/targeted attacks. This plugin detects the encrypted Gh0stRat communication, decrypts it and also automatically identifies the malicious Gh0stRat process, its associated network connections and the loaded DLL's. This can help the digital forensic investigators and incident responders to quickly narrow down on the Gh0stRat artifacts without having to spend time on the manual investigation.”

Although a chopshop module exists for decrypting Gh0stRat communications in packet captures, Monnappa’s Volatility plugin aims to solve several specific problems that analysts may regularly face, including the absence of a full packet capture from the victim machine and needing to trace connections in the pcap back to the suspect process or DLL.

Monnappa’s Twitter: @monnappa22
Monnappa’s Submission:

Most wanted follow up(s): Continued research into other malware families.

Jamaal Speights: MsDecompress

The msdecompress plugin by Jamaal Speights has high potential. It allows investigators to find and extract data compressed with the LZNT1 algorithm (Xpress and XpressH coming soon) from memory dumps and it reports the process in which the data was found. The RtlDecompressBuffer API is heavily used by malware authors to pack their code and minimize the size of command and control traffic before sending it across the network. Many kernel components and popular applications also use this compression algorithm, and we look forward to hearing about all the types of forensic evidence that can be uncovered using this plugin.

Jamaal’s Twitter: @jamaalspeights
Jamaal’s Blog:
Jamaal’s Code: 
Jamaal’s Submission:

Most wanted follow up(s): An analysis of the different types of compressed data frequently found in memory.

Cem Gurkok: Mac Rootkit and Bitcoin

Cem submitted a total of four plugins: two for detection of rootkit hooks in Mac OSX memory, one for in-depth investigation of Mac OSX threads, and one for finding bitcoin private keys and addresses.
  • mac_bitcoin allows for recovery of bitcoin keys and addresses. This can greatly help investigators that need to determine which transactions and activity a particular user was involved with. Due to the nature of bitcoin, this activity can be very well hidden within the network and only examination of a user’s system can put the pieces back together
  • The mac_check_call_reference plugin is used to check for modified call instructions in the kernel. This can catch a wide array of rootkits that directly modify control flow in order to manipulate the system.
  • The mac_threads plugin is able to enumerate threads of each running Mac task. The examination of thread state can lead to determination of which portions of code a thread was using and which operations it performed. This capability had been missing from Volatility’s Mac support while being supported by the Windows and Linux side in the last two releases.
  • mac_check_shadow_trustedbsd enables the detection of rootkits that modify a reference to the TrustedBSD policy list. Such a modification can allow a rootkit to add, modify, and delete system activity returned to other kernel components and user land tools that rely on TrustedBSD for a set of system state.
Cem’s Twitter: @CGurkok
Cem’s GitHub:
Cem’s Blog:
Cem’s Submission: and

Most wanted follow up(s): Support for bitcoin addresses found in any process (not just Multibit) on any memory dump (Windows, Linux, Mac) and also in free/deallocated memory.

Csaba Barta: Malware Analysis 

The plugins in this submission are focused on helping analysts perform malware investigations and malware research.  The first set of plugins highlight the differences between an infected memory sample and its baseline image.  This can help an analyst quickly determine the types of changes the malware has made to the system.   The current plugins focus on four important components of the operating system: processes,  DLLs, services, and drivers.  The final plugin, malprocfind, attempts to codify the rules an investigator may use to look for suspicious artifacts on a system.   The plugins help automate common analysis techniques used by analysts during malware investigations.

Csaba’s GitHub:
Csaba’s Submission:

Most wanted follow up(s): Further extension of the baseline artifacts and malprocfind rules.

Philip Huppert: OpenVPN 

Philip’s submission is the result of his University paper “Extracting private information of virtual machines using VM introspection.” In the paper, Philip described how to recover openvpn 2.x.x usernames and passwords entered by the user in addition to the password required for unlocking the private key. The submission also includes a plugin to extract base64/PEM encoded RSA private keys from memory.

The openvpn plugin is effective against any memory dump format (not just live VM memory using libvmi). Philip also did a really nice job of narrowing the search space for finding the usernames and passwords. He isolates the .data and .bss segments of openvpn.exe and looks for signs of a specific data structure (named “user_pass”).

Philip’s GitHub:
Philip’s Submission:

Most wanted follow up(s): A summary of the steps for decrypting an openvpn session from a packet capture, given the private key.  Also the ability to scan for the data structures in physical space (for example if the openvpn.exe process is no longer running).

Wyatt Roersma: Hyper-V Tools

Wyatt’s plugins will extract Hyper-V artifacts from a host system’s memory.  The first plugin hpv_vmconnect is used to extract information about which users were accessing virtual machines using the virtual connect console. The second plugin, hpv_vmwp, is used to map each virtual machine to its associated process on the host and extract temporal information about when the machine was started last. The final plugin hpv_clipboard is used to extract memory resident Hyper-V clipboard and hotkey artifacts. The plugins provide some insights into the types of artifacts that can be extracted from Hyper-V host memory and set the stage for future research.

Wyatt’s Twitter: @WyattRoersma
Wyatt’s Blog:
Wyatt’s GitHub:
Wyatt’s Submission:

Most wanted follow up(s): Further research into other Hyper-V artifacts and conversion tools.