Friday, May 7, 2021

Highlighting Research from the Next Generation of Memory Forensics Practitioners

Nearly 2 years ago, we published a blog post about our collaboration with Dr. Golden G. Richard III at the Louisiana State University (LSU) Center for Computation and Technology (CCT). We are very happy to report that this collaboration is still going strong, has been a huge success, and has helped the Applied Cybersecurity Lab at LSU flourish. The students from LSU who have finished their studies are now making a real impact in our industry, and those currently pursuing their degrees are continuing to push the state of the art in memory forensics and malware detection. 

Given that another semester has just wrapped up, and that the LSU Office of Research recently published a detailed article about our collaboration, we decided that it was time for us to write our own acknowledgement of these students and their efforts. As you might imagine, many of the students have focused their research on memory forensics and its use in real-world DFIR workflows. These research efforts have been strongly focused on "gaps" in current analysis techniques; significant improvements of existing techniques; and efforts to classify and test the accuracy and reliability of existing tools. For the remainder of this post, we will highlight several of these efforts, as well as introduce the students involved.

Reliability of DFIR Tools

The efforts aimed at the reliability of DFIR tools began with the creation of systems for targeted fuzzing of DFIR frameworks. The first publication from this research was the Gaslight fuzzing architecture aimed at memory forensic frameworks. Gaslight was first presented at DFRWS 2017, and it showcased the ability to efficiently fuzz only the portions of a memory sample analyzed during common investigations. It successfully found numerous error conditions in Volatility 2 that have since been patched, making the framework more reliable in the face of smearing.

A second version of Gaslight was then developed that greatly improved performance and scalability. This was targeted at the Sleuthkit Framework and found several inputs that would cause the library and associated programs to crash or exhaust available resources. This was published in Computers and Security (COSE) in 2020 and served as the basis of Shravya Paruchuri's Master's thesis.

The final iteration of Gaslight involved modifying its architecture to support distributed and high performance processing. This effort was also aimed at Volatility 2 and found many plugin code paths that did not properly account for smear, and that were not successfully triggered during testing with the first architecture version. This project was performed in coordination with the High Performance Computing center at LSU, and their allowance of abundant HPC resources allowed for millions of plugin fuzzing runs. This work was performed by Arian Shahmirza and led to successful completion of her Master's Degree titled "High Performance Fuzz Testing of Memory Forensics Frameworks". 

Beyond fuzzing, there was also a major effort to automate testing the accuracy of memory analysis frameworks. Given that the data structures in memory samples are a constantly moving target due to new operating system and application versions, it is imperative that analysis tools are able to keep up with the rapid changes. This is currently an extremely time-consuming and manual process, so research was performed to automate the analysis and comparison of different module versions to provide for an efficient and tested workflow. Ryan Maggio led this effort, and it contributed to his successful Ph.D. defense. The paper describing this research was accepted at DFRWS 2021 and will be presented this summer. We are also very excited for Ryan's future in the DFIR industry, as he recently accepted an offer to join the research team at MIT's Lincoln Laboratory after graduation.

Bringing Emulation to Memory Forensics

The second portion of Ryan's Ph.D. work involved another major research effort performed by our team and the LSU students: bringing emulation to the forefront of memory analysis research. Existing methods for analyzing malicious code in memory, such as shellcode and API hook stubs, require an expert investigator to manually reverse engineer each code block. Given that an average modern memory sample has thousands of such hooks, most of which are benign, this is no longer a feasible approach. 

To alleviate this issue, research was performed that explored the use of emulation of code inside memory samples to generate automated decisions for commonly seen code blocks, stubs, and patterns. This was a group research effort and led to the creation of HookTracer, a system we built on top of Unicorn that integrates directly with Volatility. While Unicorn provides a bare emulator, HookTracer adds Windows-specific functionality to the emulation environment. This includes support for FS/GS access; per-process address spaces; recording of API calls and parameters for functions of interest; and the ability to record all basic blocks and VADs traversed by particular code paths. All of these features are directly accessible to Volatility plugins through the new HookTracer API.

This research and development of HookTracer led to several peer-reviewed publications, as well as contributed to several successful M.S. and Ph.D. defenses. The first of these was presented at DFRWS 2019 and focused on the automated analysis of API hooks. As shown in this paper, default Windows installations with no 3rd-party products and no malware present still have thousands of active API hooks to support backwards compatibility and other related features. Through the development of a new Volatility plugin that leveraged HookTracer's APIs, all of these benign hooks were successfully filtered, while malicious ones planted by malware were successfully reported.

The second effort focused on message hooks, which are often abused by malware to perform keylogging, copy/paste buffer snooping, and inject code into remote processes. To remove the burden of manual analysis of such hooks, a new HookTracer plugin was developed that could automatically determine if a message hook was legitimate or benign. Furthermore, for all Windows APIs called by such hooks, the plugin is able to report the parameters passed to functions of interest. The following picture shows this plugin in action against a malware sample from the infamous Turla group:

In the output, you can see that HookTracer has successfully emulated the malware's message hook. This includes revealing that the malware recorded the current timestamp, application window title, current key pressed, and current working directory. The malware then writes these to a file on disk (msimm.dat). In many ways, this aspect of HookTracer can be thought of as a sandbox-like execution environment for code in memory. The full details of this plugin and its analysis capabilities can be found in the research team's paper published in COSE in 2020.

There was also a research effort by Austin Sellers to automate the analysis of networking APIs called by memory resident code. Network-based IOCs, such as IP addresses and hostnames, are used throughout all phases of the DFIR workflow. Existing memory analysis techniques enhance this information by automatically determining which process(es) have communicated with the hosts/IPs of interest. These techniques then leave the investigator to manually find the code region(s) that actually perform the communication, which is a crucial task when the ability to decode encrypted packet data is needed or when the C2 protocol needs to be reverse engineered. Austin's work sought to automate the location of these code regions by finding the places where networking APIs are called and extracting the parameters to the APIs. This work was the foundation of his Master's thesis. Since graduation, Austin has worked with several of our team members at Volexity as a software engineer helping to build commercial memory analysis capabilities.

We also have a fourth effort related to HookTracer that is currently being finished and will soon be sent for publication. Given the double-blind nature of most academic security conferences and journals, we cannot say too much yet except that it will provide another great leap forward for the field. We expect a follow-on blog post once it is finished.

Finally, HookTracer was developed for Volatility 2 given that this project was started several years ago. As Volatility 3 stabilizes and approaches feature parity with Volatility 2, we plan to port HookTracer to the new version so that the entire community can build on it and benefit from its use.

Memory Analysis Gaps

Although the memory forensics field has seen substantial research efforts over the last decade, the sheer number of relevant operating systems, applications, and runtime environments means that many research efforts are still needed. The combined research group aimed to address several of the most interesting and relevant of these research "gaps".

This began with Nathan Lewis publishing a paper at DFRWS 2018 on his efforts to build memory analysis capabilities for Linux systems powered by version 1 of the Windows Subsystem for Linux (WSL).  Before this research, all of the WSL-controlled Linux processes, file descriptors, network activity, and other artifacts were not accessible in a structured manner. This led to Volatility plugins being 'blind' to WSL activity. Given the technical effort and usefulness of this research, Nathan's paper was chosen for Best Student Paper award at DFRWS.

There were also efforts to modernize userland analysis of macOS systems. The last major research of the userland runtimes for Swift and Objective-C was published in 2016 and has fallen behind modern versions. To update the support as well as add new analysis features, Ph.D student Modhuparna Manna worked on two major efforts. The first was the analysis of the macOS page queues, which "hide" many present pages in memory by marking them as invalid, even though they are actually in a given memory sample. Without the analysis of the queues, many pages in process memory are unavailable to analysis plugins and results are often very limited. Recovering these pages and making them accessible to Volatility plugins was the subject of a paper that the group published at DFRWS 2020. Modhu also updated the existing Volatility plugins for analysis of the macOS userland runtimes to support modern versions as well as recover several new artifact types. These are covered in a paper that has been accepted by the Digital Investigation journal and is awaiting publication. We will update this blog post with the link to the paper after publication.

Analysis of the Android userland runtimes was the focus of Sneha Sudhakaran's research, in collaboration with her co-advisor Aisha Ali-Gombe, a faculty member at Townson University. Given the popularity of Android devices, along with the amount of malware targeting the platform, this research is highly relevant to modern investigations. The results of this research were novel methods for analysis and recovery of Android application activity. There were several publications as a result of this effort, the first being AmpleDroid for analysis of Large Object Files. The second, DroidScraper, was published at the top-tier academic security conference RAID and focused on structured recovery of application data and artifacts. Sneha is currently working on a third related publication as she wraps up her Ph.D. studies.

Scholarships for Service

Before closing, we wanted to highlight that due to the efforts of Dr. Richard and his colleagues Dr. Ram, Dr. Sun, Dr. Mahmoud and Dr. Peng, LSU is now a part of the National Science Foundation's Scholarship for Service (SFS) program. The full details are on the corresponding LSU webpage, but briefly, this scholarship provides up to 3 years of full support to students. This support includes tuition; a very nice stipend ($25,000/year for undergrads and $34,000/year for graduate students); funding for professional training and certification; and internship opportunities. 

In exchange, students agree to work for a US government institution for the number of the years that the scholarship was used. Past examples of institutions students work for/with include 3-letter agencies, energy-related departments, and other high-profile organizations that operate major components of the country. These are jobs that would be of interest to many Computer Science graduates, and the SFS program provides a direct path to them.

If you know someone who may be interested in the SFS program, please see the link above. We are hoping that it will continue to attract motivated students to the LSU research group, and that we can continue to help mold the future practitioners and researchers of the DFIR industry. 

We also would like to note that no one on the Volatility Team has any involvement with selecting or screening students through this process. If you have any questions, the best place is the previously linked page.

Final Thoughts

In closing, we would again like to acknowledge the great amount of effort put forth by the students in the LSU research group. Please reach out to them if you have questions on their research, or if your organization may be hiring in the future. As a final note, we would also like to congratulate a current Master's student in the group, Raphaela Mettig, for excelling during the intern interview process at Tesla and being awarded a security team internship for this summer.

-- The Volatility Team

Tuesday, January 26, 2021

Malware and Memory Forensics Training Goes Virtual!

We are very excited to announce that our popular Malware and Memory Forensics with Volatility training is now available in a self-paced, online format!

Brought to you by members of the Volatility Team, this course gives you the opportunity to learn directly from the people behind the research and development of Volatility, and it offers you a chance to support our ongoing efforts.

The Course

Our course provides a deep examination of Windows internals, malware operations, attacker toolkits, DFIR workflows, and how memory forensics can be leveraged throughout all of your investigations. The end result of the lectures on these topics is a complete understanding of how memory analysis tools operate, along with the traces left behind by malicious actors and applications. You will gain knowledge that applies to all memory analysis investigations and frameworks, both now and in the future.

The Labs

You will get significant hands-on experience using Volatility against Windows, Linux, and MacOS samples during the training labs. These labs are mirrored directly to match our real-world investigations (1, 2, 3, 4, 5), and they are constantly updated as new threats emerge. After each lab, there is a pre-recorded walkthrough that shows you the precise steps taken to solve the lab as well as explains the rationale behind the workflow used. A written version of the walkthrough is also provided in the Lab Guide.

The Content

Signing up for the course provides you with 4 months of access to all material, including the pre-recorded lectures, demos, and lab walkthroughs. You will also be given a trial version of Surge Collect Pro, which provides reliable and secure memory acquisition across Windows, Linux, and Mac systems. For support, we have a dedicated channel on our Slack server and periodic office hours via Zoom; students can directly ask us questions about any portion of the course, with optional screen sharing. After the course ends, you may retain access to the written materials (Art of Memory Forensics, course slides, lab guide) and the course virtual machine. You are also welcome to stay in the student-only Slack channel and join our alumni-only mailing list.

For those wondering about Volatility 2 vs Volatility 3, our course currently uses Volatility 2 for demos and labs as it is the stable and fully featured version of The Volatility Framework. As mentioned previously, the skills learned in this course are transferable to any memory analysis framework – even those not based on Volatility. We plan to add training modules specific to Volatility 3 slowly over time as it stabilizes and further approaches feature- and plugin-parity with Volatility 2. In general, if you know how to use Volatility 2, then using Volatility 3 will be very simple in the future. As students of this course, you will be the first to gain access to Volatility 3 training material as it is released.

Sign Up!

You can request access to the course here. We provide discounts for military, law enforcement, and groups of students from the same organization. We also provide significant discounts to course alumni who wish to take the course again, as well as to full-time college students studying in a related field. Please inquire about these discounts if you meet the requirements. We can also accommodate private sessions for large group trainings.

We would like to thank the memory forensics community for their years of continued support. We are excited that our course is now available 24/7/365 for students around the world!

- The Volatility Team

Wednesday, November 18, 2020

The 2020 Volatility Plugin Contest results are in!

We would like to begin by thanking the participants for their hard work and contributions to Volatility. It’s always exciting to see continued innovation in the field of memory forensics from research teams across the globe! Despite the challenges of this unprecedented year, we had 8 submissions, including a number of new plugins, an output renderer, and a new graphical user interface. This year’s participants also had the additional challenge of learning all the new paradigms and interfaces of Volatility 3.

Independent open source projects and communities only remain viable because of contributors who are willing to sacrifice their time and resources. Please show your appreciation for the contestants’ contributions by following them on Twitter/GitHub/LinkedIn, providing feedback on their ideas, and helping to improve their code with testing, documentation, or contributing patches. 

We would like to thank Volexity for being a sustaining sponsor of the Volatility Foundation and, in particular, for contributing to this year’s contest. We would also like to thank Paessler AG for their donation toward the contest prizes!

Placements and Prizes for the 2020 Volatility Plugin Contest:

1st place and $2000 USD cash or One Free Seat at Malware and Memory Forensics Training by the Volatility Team goes to:

Gustavo Moreira for Netfilter  

2nd place and $1000 USD cash goes to:

Jelle Vergeer for SSHKeys 

3rd place and $750 USD cash goes to:

Aviel Zohar for Volatility Explorer, StructAnalyzer, WinObjGUI, FileScanGUI, P2V, PFNInfo, RAMMap, Winobj

Below is a detailed summary of all submissions, ordered alphabetically by first name. If you have feedback for the participants, we're sure they'd love to hear your thoughts! As previously mentioned, these developers deserve huge props. We look forward to seeing future work by these authors! 

Aviel Zohar: Volatility Explorer, StructAnalyzer, WinObjGUI, FileScanGUI, P2V, PFNInfo, RAMMap, Winobj

This contest submission includes a number of components intended to make Volatility more approachable and user-friendly to investigators. Volatility Explorer is a graphical user interface that provides a user experience similar to Sysinternal’s Process Explorer but only leveraging the information extracted from volatile memory. The Struct Analyzer plugin allows an analyst to parse memory resident data structures in a graphical hierarchy. WinObjGUI provides a capability to view kernel objects with a similar user experience provided by Sysinternal’s WinObj tool. FileScanGui allows an analyst to view files by providing a Window’s Explorer user interface experience. The submission also includes tools for mapping physical to virtual addresses, extracting the provenance of memory regions, and exploring forensics artifacts in physical memory: P2V and PFNInfo. The submission leverages these capabilities to create a tool that provides a similar user experience to Sysinternal’s RAMMap. Finally, the submission also includes an updated version of WinObj.

Related References:

Bjorn Stelte:  EvtxLogs Plugin

This submission provides the capability to extract evtx entries from physical memory of Windows systems. The EvtxLogs plugin leverages the work done by Willi Ballenthin's python-evtx/evtxtract libraries. By focusing on memory samples, it allows a forensics analyst to get insights into events that were happening at the time of acquisition, and events that happened in the past on the suspected system. In particular, this can augment the data used for temporal reconstruction of system events. 

Related References: 

Bjorn Stelte:  Syslog Renderer

This submission involved a Volatility 3 renderer for sending Volatility output to a syslog server. The syslog output renderer is intended for situations where a forensics analyst is attempting to analyze a large number of memory samples. An analyst could integrate Volatility into an automated pipeline and send the results to a unified syslog server. Then, the analyst could access the server to triage, using strings and other analysis methods which systems need deeper forensic analysis.

Related References:

Gustavo Moreira: MountInfo Plugin 

The use of containers has seen significant growth in production Linux environments. Previous Volatility file system analysis capabilities did not fully enumerate information related to containers, which left much work on part of the analyst. This plugin closes that gap by replicating the per-process mount information as exported in the /proc/<pid>/mountinfo file on live systems.

Related References:

Gustavo Moreira: Netfilter Plugin

This plugin provides full support for the enumeration of Netfilter hooks for all modern versions of the Linux kernel. The Netfilter subsystem is routinely abused by kernel rootkits to control the network stack, such as the recently disclosed Drovorub and Cloud Snooper malware samples. Abusing Netfilter hooks provides a rootkit with the ability to monitor for 'magic packets' containing command-and-control data; sniff outgoing and incoming network connections; and even silently modify packet data. Properly implemented rootkits will also abuse Netfilter to hide malicious traffic from userland network monitors, such as tcpdump and Wireshark, running on the local system.

Related References: 

Jelle Vergeer: SSHKeys Plugin

This submission implemented a Volatility 3 plugin to extract SSH session keys, which are used to encrypt and decrypt network traffic between an SSH client and server. The plugin can extract keys from Linux OpenSSH ssh (client) and sshd (server) processes by locating and parsing a structure that holds the symmetric key cipher name, the keys used for encryption/decryption, and the initialization vectors (IVs) used for encryption/decryption. For each SSH session, two key/IV pairs are recovered, client to server and server to client.  Given the output of the author's Volatility plugin (a JSON containing key and IV) and a PCAP of the network traffic between SSH client and server, users can decrypt and parse the network traffic using another tool the author developed, The result is complete transparency into the entire SSH network session.

Related References:

Or Chechik and Inon Weber: check_parent_spoof Plugin

This submission provides a practical method to detect stealth techniques used by malware in the wild and post-exploitation toolkits, such as Cobalt Strike. Although there are ways to identify parent process spoofing on live machines using ETW and via dead disk forensics by parsing event logs, the check_parent_spoof plugin for Volatility 3 adds this capability to memory analysis. In many cases, after creating a child process with a spoofed parent, code is injected into the child, which warrants analyzing memory anyway. The ability to triage all of these aspects with a single source of evidence provides a powerful capability.

Related References:

Or Chechik and Inon Weber: check_peb_spoof Plugin

In addition to parent process spoofing, another common obfuscation technique is PEB-Spoofing or PEB-Masquerading, which provides a mechanism for bypassing UAC. This plugin provides the ability to detect spoofed process names and detect malware using this technique. This is accomplished by comparing the name in the PEB with the name from the EPROCESS kernel structure. 

Related References:
Here are a few additional resources for previous contests and community-driven plugins:

Volatility Foundation Contest Home Page:

Volatility 2019 Plugin Contest Results:
Volatility 2018 Plugin Contest Results:
Volatility 2017 Plugin Contest Results:
Volatility 2016 Plugin Contest Results: 
Volatility 2015 Plugin Contest Results:
Volatility 2014 Plugin Contest Results:
Volatility 2013 Plugin Contest Results:

Volatility Community GitHub Repository: 


Wednesday, May 27, 2020

When Anti-Virus Engines Look Like Kernel Rootkits

While analyzing real-world systems, memory analysts will often encounter anti-virus (AV) engines, EDRs, and similar products that, at first glance, look suspiciously like malware. This occurs because these security products leverage the same techniques commonly employed by malware—such as API hooking, system call hooking, and registering callbacks—in order to gain the insight they need to detect and analyze threats.

A recent journal publication by members of our team examined this issue for userland API hooks. In this blog post we will show an example of what AV kernel hooks often look like. The sample investigated in this post was sent by a friend who wanted clarification of the behavior, as it was part of a real investigation. In particular, Volatility's ssdt plugin was showing many system call table entries as being hooked, and our friend wanted to know if the hooks were malicious or benign.

System Calls Background

System calls are the common mechanism by which unprivileged code in userland (process) memory requests services through the kernel to access privileged resources, such as the hard drive to create/read/write/delete files, the network stack to send/receive packets, the monitor to display information to the user, and numerous other resources. System calls are also used to enumerate the active system state, such as the list of running processes, active network connections, loaded kernel modules, and nearly everything else that live forensic tools and endpoint agents gather for analysis. This obviously makes the system call table an attractive target for malware to tamper with, as well as a high-value location for security agents to monitor.

To check for signs of rootkits, Volatility's ssdt plugin locates each system call table present in a memory sample and then enumerates each system call table entry. For each entry, it prints either the containing module, if found, or UNKNOWN if the entry points to a memory location not associated with the kernel itself or a module contained within the kernel module list. Several common kernel rootkit methods of hiding code will trigger the classification as UNKNOWN, such as unlinking the malicious module from the module list, or allocating a RWX memory region and then using it to host code after the malicious module is unloaded (see this blog post on analyzing a detached kernel thread).

Unfortunately, the use of system call hooking by AV and EDR engines frequently triggers UNKNOWN entries in ssdt output, as security agents' attempts to "hide" on a system often leverage the same tactics used by malware.

Initial Analysis

To start the analysis, the ssdt plugin was run against the sample:

$ python -f sample.raw --profile=Win7SP1x86  ssdt > ssdt-output.txt 
Volatility Foundation Volatility Framework 2.6
$ grep -c UNKNOWN ssdt-output.txt 
$ grep UNKNOWN ssdt-output.txt | head -20
  Entry 0x000d: 0x886ea580 (NtAlertResumeThread) owned by UNKNOWN
  Entry 0x000e: 0x881f70b0 (NtAlertThread) owned by UNKNOWN
  Entry 0x0013: 0x886e4d88 (NtAllocateVirtualMemory) owned by UNKNOWN
  Entry 0x0016: 0x88217ca0 (NtAlpcConnectPort) owned by UNKNOWN
  Entry 0x002b: 0x886e9e80 (NtAssignProcessToJobObject) owned by UNKNOWN
  Entry 0x004a: 0x886ea390 (NtCreateMutant) owned by UNKNOWN
  Entry 0x0056: 0x886e9c48 (NtCreateSymbolicLinkObject) owned by UNKNOWN
  Entry 0x0057: 0x88438858 (NtCreateThread) owned by UNKNOWN
  Entry 0x0058: 0x886e9d00 (NtCreateThreadEx) owned by UNKNOWN
  Entry 0x0060: 0x886e9f30 (NtDebugActiveProcess) owned by UNKNOWN
  Entry 0x006f: 0x886e4910 (NtDuplicateObject) owned by UNKNOWN
  Entry 0x0083: 0x886eaba8 (NtFreeVirtualMemory) owned by UNKNOWN
  Entry 0x0091: 0x88436198 (NtImpersonateAnonymousToken) owned by UNKNOWN
  Entry 0x0093: 0x886ea4d0 (NtImpersonateThread) owned by UNKNOWN
  Entry 0x009b: 0x880d3478 (NtLoadDriver) owned by UNKNOWN
  Entry 0x00a8: 0x886eaae0 (NtMapViewOfSection) owned by UNKNOWN
  Entry 0x00b1: 0x886ea2e0 (NtOpenEvent) owned by UNKNOWN
  Entry 0x00be: 0x886e4de0 (NtOpenProcess) owned by UNKNOWN
  Entry 0x00bf: 0x886e4db8 (NtOpenProcessToken) owned by UNKNOWN
  Entry 0x00c2: 0x886ea1a8 (NtOpenSection) owned by UNKNOWN

Looking at the output, it appears that 45 entries are hooked by UNKNOWN module(s). Furthermore, the hooked functions are commonly targeted by both malware and security agents.

Classifying One Hook

To begin the process of determining if the hooks are benign or malicious, volshell was used to examine the code of a few hooks. volshell is a Volatility plugin that allows you to interactively explore a memory sample, including viewing, searching, and disassembling arbitrary addresses in any context (the kernel or a particular process).

To start, volshell was loaded and the hook for Entry 0x000d: 0x886ea580 (NtAlertResumeThread) was disassembled (numbers, dashes, and arrows added for ease of explanation):

$ python -f sample.raw --profile=Win7SP1x86  volshell
> dis(0x886ea580)
0x886ea580 55                               PUSH EBP                               <----- [1]
0x886ea581 8bec                           MOV EBP, ESP
0x886ea583 ff750c                         PUSH DWORD [EBP+0xc]
0x886ea586 b9fca56e88                MOV ECX, 0x886ea5fc            <-----  [2]
0x886ea58b ff7508                         PUSH DWORD [EBP+0x8]
0x886ea58e 51                               PUSH ECX
0x886ea58f 8b01                            MOV EAX, [ECX]                      <-----  [3]
0x886ea591 ff5004                         CALL DWORD [EAX+0x4]        <-----  [4]
0x886ea594 83c40c                       ADD ESP, 0xc
0x886ea597 5d                               POP EBP
0x886ea598 c20800                       RET 0x8                                    <-----  [5]
0x886ea59b cc                               INT 3
0x886ea59c 0000                           ADD [EAX], AL
0x886ea59e 0000                           ADD [EAX], AL

After reading the disassembly, several things stand out. First, there is a real function prologue with the instructions of "PUSH EBP; MOV EBP, ESP;" [1] and, shortly after, a proper function epilogue [5]. Examining the code in between, a hardcoded address is stored in ECX [2]. That address is dereferenced and stored in EAX [3]. A dereference of EAX + 4 is then used as the target address of a CALL instruction [4].

This code flow can be re-implemented in volshell through the use of its dd (display double word) and dis functions.

> dd(0x886ea5fc, 4)                   
886ea5fc  964c72b4

> dd(0x964c72b4 + 4, 4)          
964c72b8  964ba922

> dis(0x964ba922)
0x964ba922 55                               PUSH EBP
0x964ba923 8bec                           MOV EBP, ESP
0x964ba925 51                               PUSH ECX
0x964ba926 51                               PUSH ECX
0x964ba927 53                               PUSH EBX
0x964ba928 56                               PUSH ESI
0x964ba929 57                               PUSH EDI
0x964ba92a bf010000c0                MOV EDI, 0xc0000001

The first invocation of dd is at the hardcoded address from [2] earlier. As can be seen, the address stored here is 0x964c72b4. The second dd invocation dereferences this address + 4 just as the disassembly in [4] from earlier shows. This gives a CALL target address of 0x964ba922, which the address used in the above dis call.

Looking at the instructions here, there is the beginning of a long function which appears to be the real hook implementation. To determine which, if any, module is hosting the real payload, the drivermodule plugin can be invoked with the address of the function:

$ python -f sample.raw --profile=Win7SP1x86 drivermodule -a 0x964ba922
Volatility Foundation Volatility Framework 2.6
Module                   Driver          Alt. Name    Service Key
--------------------- ------------- ------------- -----------
SYMEVENT.SYS     SymEvent   SymEvent   \Driver\SymEvent

This output shows that the function is inside of the SYMEVENT.SYS driver, which is part of the Symantec AV/Endpoint protection suite installed on the system.

Classifying All Hooks

So far, it has only been shown that 1 of the 45 hooks belongs to Symantec. To ensure that all of them belong to Symantec, the remaining 44 must be checked. Repeating the volshell steps for two more of the system calls showed the same pattern as before: a hardcoded address (different address for each hook) being dereferenced, followed by the value of the first dereference having 4 added then being dereferenced itself. The consistent result between the hooks of this sample was that the final destination was the same: 0x964ba922. With this in mind, it became clear that the process of checking each hook could be automated to see if they all ended at 0x964ba922. 

To accomplish this, awk was first used to extract just the UNKNOWN addresses from the ssdt output saved earlier:
grep UNKNOWN ssdt-output.txt | awk '{ print $3 }' > unknown-addresses
This placed all of the UNKNOWN addresses in the unknown-addresses file. Next, code was written inside of volshell that repeated the read hardcoded address-> dereference -> dereference+4 pattern. Note: volshell runs inside of the Python shell, so whatever you can do in Python, you can do in volshell as well.  The following shows this code in its entirety, with line numbers added to the beginning of each line.
1.    > for line in open("unknown-addresses", "r").readlines():
2.   ...:    address = int(line.strip(), 16)
3.   ...:    insts = addrspace().zread(address + 6, 5)
4.   ...:    if ord(insts[0]) != 0xb9:
5.   ...:        print "invalid instruction at %x | %x" % (address, ord(insts[0]))
6.   ...:        break
7.   ...:     
8.   ...:    first_addr = struct.unpack("<I", insts[1:])[0]
9.   ...:     
10. ...:   second_addr_str = addrspace().zread(first_addr, 4)
11. ...:   second_addr = struct.unpack("<I", second_addr_str)[0]
12. ...:     
13. ...:   function_addr_str = addrspace().zread(second_addr + 4, 4)
14. ...:   function_addr = struct.unpack("<I", function_addr_str)[0]
15. ...:     
16. ...:   if function_addr != 0x964ba922:
17. ...:        print "incorrect function_address: %x" % function_addr
18. ...:     

This code starts by looping for each line in the file, which contains one line for each UNKNOWN hook address. Next, the address is converted to an integer. On line 3, the hook address plus 6 is read for 5 bytes inside of the kernel address space. This corresponds to reading what should be the MOV ECX, <hardcoded address> instruction at [2] in the first disassembly listing. The first byte of the instruction is verified to be 0xb9, which is the MOV instruction opcode. On line 8, the remaining 4 bytes are then converted to an integer, as they are the hardcoded address in little endian. On lines 10  and 11, the hardcoded address is dereferenced and then converted to an integer. On lines 13 and 14, the dereferenced valued plus 4 is dereferenced and converted to an integer. It is then verified to be the target CALL address of 0x964ba922.

Running this loop over the sample showed that all system call hooks transferred control flow to 0x964ba922, which means that all hooks belonged to the same Symantec driver.

Closing Thoughts

Through a mix of manual reverse engineering and automated comparisons based on knowledge learned, the system call table was automatically verified as clean, and our friend was given a repeatable methodology to apply to other samples in his investigation.

If you like this type of analysis and want to challenge yourself to write plugins that automate memory forensic techniques, then consider a submission to our 2020 Volatility Plugin Contest. This year's contest is based on Volatility 3, and you can learn about the new, exciting features of Volatility 3 in our recently recorded presentation.

If you would like to stay in touch with the Volatility Team and community then consider following us on Twitter, joining our Slack Server, and subscribing to our email list.

Friday, May 15, 2020

The 8th Annual Volatility Plugin Contest!

We are excited to announce that the 8th annual 2020 Volatility Plugin Contest is now accepting submissions until October 1, 2020!

Winners will receive over 3750 USD in cash prizes!

Volatility 3

The 2020 Volatility Plugin Contest encourages research and development in the field of memory analysis. With the announcement of the Volatility 3 Public Beta (blog post and recent presentation), and the current development focus on the Volatility 3 Official Release later this summer, we decided this is an exciting time to shift the Contest to Volatility 3 plugins. As in previous years, the main goal is to encourage innovation in the field of memory analysis.

Submissions provide an opportunity to get industry-wide visibility for your work, put groundbreaking capabilities immediately into the hands of investigators, and contribute back to the open source forensics community. By building on top of Volatility 3, your contributions will help lay the foundation for the next generation of memory forensics! If you are looking for inspiration for the Volatility Plugin Contest, check out the previous results.


We would like to thank Volexity and our other sustaining donors for their continued support.

Friday, November 15, 2019

Results from the 2019 Volatility Contests are in!

We would like to begin by thanking the participants in this year’s contests! This was one of the hardest years for our panel of judges since it had so many outstanding submissions. In the Plugin Contest, there were 11 submissions, which included over 30 new plugins across 3 operating systems. It was also great to see repeat contestants in both the Plugin Contest and the Analysis Contest, and submissions from community members around the globe. Despite the contest going into its 8th year, it’s inspiring to see all the exciting innovation still happening in the field of memory forensics. 

Independent open source projects and communities only remain viable because of contributors who are willing to sacrifice their time and resources. Please show your appreciation for the contestants’ contributions by following them on Twitter/GitHub/LinkedIn, providing feedback on their ideas, and helping to improve their code with testing, documentation, or contributing patches. Looking forward, we are excited to announce that next year’s contest will focus on Volatility 3!

We would also like to thank Volexity for being a sustaining sponsor of the Volatility Foundation and, in particular, for sponsoring this year’s contests.

Placements and Prizes for the 2019 Volatility Plugin Contest:
1st place and $1500 USD cash or One Free Seat at Malware and Memory Forensics Training by the Volatility Team goes to:

Antoine Brodin for FreeBSD Support

2nd place and $750 USD cash goes to:

Blaine Stancill for Windows 10 Memory Compression

3rd place and $250 USD cash goes to:

Shusei Tomonaga for MalConfScan

4th place and Volatility swag goes to:

Shachaf Atun for Winobj and Tokenimp

5th place and Volatility swag goes to:

Fabio Pagani for Linux kallsyms

Placements and Prizes for the 2019 Volatility Analysis Contest:
1st place and $1500 USD cash or One Free Seat at Malware and Memory Forensics Training by the Volatility Team goes to:

TEAM. HSLFL (South Korea) for “Memory Analysis in Data Leakage Cases”

Here is a detailed summary of the submissions. If you have feedback for the participants, we're sure they'd love to hear your thoughts!

Analysis Contest

1st: Memory Analysis in Data Leakage Cases by TEAM. HSLFL (South Korea)

This analysis report, written by TEAM. HSLFL (Hope Seyoung Lee Find Love) from South Korea, focuses on exploring how memory analysis with Volatility can be used during data breach investigations. In particular, they present three scenarios involving a remote administration tool (QuasarRAT), FTP, and cloud storage (Google Drive). The simulated scenarios involved two systems, an attacker machine, and a victim machine. The report authors leveraged memory analysis to find suspicious commands, processes, and network activity. They also used memory analysis techniques to extract memory-resident strings and executables. Finally, they demonstrated how numerous types of memory-resident files, including prefetch, hosts, FTP logs, and browser history, could be extracted and post-processed to reveal valuable insights about the attackers’ activities. The submission includes the analysis report and memory samples from each of the three targeted Windows 7 systems.

Related References:

Plugin Contest

1st: Antoine Brodin: FreeBSD Support

This submission provides a very complete toolkit for performing memory analysis on modern FreeBSD systems. In addition to adding support to Volatility 2.6.1 for a new operating system and writing 13 new plugins, the author has also written a memory acquisition tool for FreeBSD and integrated full profile support. One of the factors that judges consider is the “analysis curve” or the difference in analytical capabilities with and without the author’s work. Prior to this submission, investigations would either not be able to capture RAM or they’d be limited to strings and grep. They now can leverage the full power of Volatility! Amazing and useful!

Related References:

2nd: Blaine Stancill (FireEye): Windows 10 Memory Compression

This submission enabled the automatic incorporation of the compressed stores created by Windows 10 to implement in-memory paging. By incorporating analysis of these stores in a new Volatility address space, compressed pages are transparently decompressed as plugins access them. The clever use of inheritance and caching by the authors minimizes the performance impact of the decompression routines and makes the benefits far outweigh the impact. Aleksander Østerud pioneered some of these capabilities in last year’s plugin contest, and it is exciting to see Blaine and the FireEye team dive deeper into what we consider one of the most drastic changes to memory analysis that any OS vendor has made in the past couple years.

Related References:

3rd: Shusei Tomonaga (JPCERT): MalConfScan

This suite of plugins dumps configuration data, decoded strings or DGA domains from over 25 common malware families and red team tool kits. The plugins can be launched individually or in batch mode by the main entry point, malconfscan. Most configurations are identified with yara signatures (which, too, can be leveraged outside of the framework, if desired) and then parsed out of process memory, using freely available decompression libraries (i.e., aPLib), custom decoding functions, or encryption algorithms – using keys hard-coded within the malware or dynamically referenced from elsewhere in the memory sample. The authors also provide a Cuckoo Sandbox integration, which can save analysts a significant amount of time. We'd love to see the broader community submit merge requests to this project and help upkeep the (already extensive) library of decoders.

The authors also thought one step ahead and developed a generic malware string extractor (malstrscan plugin) that works independently of specific malware families. By default, the string extractor searches for injected code regions (similar to how malfind works), disassembles the code, and follows pointers/references to strings. In "full" mode, it extracts strings from bordering memory regions that were possibly allocated contiguously, but that might not appears as injected on their own (for example, if they're RW and not RWX).

Related References:

4th: Shachaf Atun (KSL Group): Winobj and Tokenimp

The winobj and tokenimp plugins introduce some powerful new capabilities. As always, plugins built to satisfy a particular deficiency (amongst existing plugins) during investigations are pretty much guaranteed to be valuable. In this case, the authors wanted to determine which processes elevated via UAC or UAC bypasses, and to hunt instances of token impersonation. Tools such as Mimikatz, Tokenvator, and Invoke-TokenManipulation were used for testing, and pointers were provided in the documentation to threat groups and malware variants that leverage the same techniques. We were impressed by this plugin because it also required analyzing a fair amount of undocumented data structures and flags. The winobj plugin allows extracting artifacts from the object directories. One of the documented use cases is detecting attacks against KnownDlls, which has been exploited heavily in the past. It also provides a valuable alternate method for finding drivers, devices, file systems, mutexes, symbolic links, and various other artifacts relevant to forensics. Allowing interactive browsing of object name spaces was also a nice touch.

Related References:

5th: Fabio Pagani: Linux Kallsyms

This submission provided several new capabilities to Volatility. First, new algorithms for enumerating processes in kernel memory were discussed and implemented as Volatility plugins. What is particularly interesting about these new methods is that they were derived from an automated system that can determine where and how data structures are accessed during runtime and then translate these accesses to memory forensic capabilities. The second part of the submission involved automatically finding and reconstructing the full set of kernel symbols from a memory sample. This also had an interesting twist in that it used the Unicorn emulation engine to perform the decompression of the symbol sets instead of code implemented in Volatility. The use of emulation allowed for the exact algorithm from the kernel to be used.

Related References:

The following submissions appear in the order they were received. As previously mentioned, these developers deserve huge props. We look forward to seeing future work by these authors!

Cesare Pizzi (Sorint.Lab): Powershell

Powershell has become increasingly common as a malicious payload in real malware samples. The goal of this plugin is to assist analysts in finding Powershell processes and narrowing down the amount of related data to focus on for subsequent analysis. The submitted plugin accomplishes this by first using a yara rule, rather than relying on a filename, for finding Powershell processes in memory. Additionally, the plugin provides an advanced scanning option that filters the VADs of found Powershell processes, looking for ones that may contain Powershell-related commands or scripts.

Related References:

Or Chechik and Inon Weber: Ropfind

The ropfind and linux_ropfind plugins find evidence of return oriented programming, or ROP, and stack pivoting in a memory sample. The plugins scan thread stacks for evidence of ROP gadgets and small memory allocations for evidence of stack pivoting. There are a number of checks performed to reduce the number of false positives reported. The plugins additionally provide a means for whitelisting gadgets based on opcodes and make them more specific by whitelisting based on specific symbols.

Related References:

Rolf Govers and Max de Bruijn (Fox IT): Windows Toast Notifications

The ToastPlugin is the first Volatility plugin to extract Windows notification artifacts. In particular, the plugin focuses on a particular type of notification called toast notifications. Toast notifications allow an application to relate relevant information and timely events to users that may require actions. A toast notification can contain text, images, and custom actions. For example, this could include information about connected devices, calendar events, friend requests, or email/messaging notifications. As a result, they can provide investigators with valuable insights into historical user activity and context about how the system was being used. The ToastPlugin searches though memory samples for the toast XML structures. It also lays the groundwork for a lot of future work related to Windows notifications.

Related References:

Elmar Nabigaev: VMware Tools

Virtualization platforms have become exceedingly common and typically provide capabilities to perform various actions on a guest operating system from the host. VMware's vmtools is one example and this plugin can help analysts by finding recently executed commands run on a guest through vmtools. These include operations such as copying files to and from the guest, listing files, running processes, etc. This plugin can also capture credentials an attacker may have used on the guest OS when executing a command.

Related References:

Angelo Mirabella: Linux Coredump

linux_coredump is a new Volatility 2.6.1 plugin that extracts detailed information about individual processes and generates a core dump file for the process. The resulting core file can be loaded into GDB or other debuggers that support the core file format. This plugin is particularly useful for situations where an analyst wants to dig into the details of a particular process found in a Linux memory dump and needs to leverage capabilities provided by debuggers but not memory analysis tools.

Related References:

Ryan D. Maggio, Raphaela Mettig, Sweta Ghimire (LSU): Shemu

shemu (shell + emulator) allows for the emulation of in-memory code directly inside volshell. With current Volatility capabilities, analysts who wish to perform reverse engineering inside of volshell must instead use the dis() function, which simply provides static disassembly listing. Automated emulation of code provides for much deeper, accurate, and efficient analysis of memory resident code and will be a great feature for users of the framework.

Related References:

Here are a few additional resources regarding previous contests and community-driven plugins.

Volatility Foundation Contest Home Page:

Volatility 2018 Plugin Contest Results:
Volatility 2017 Plugin Contest Results:
Volatility 2016 Plugin Contest Results:
Volatility 2015 Plugin Contest Results:
Volatility 2014 Plugin Contest Results:
Volatility 2013 Plugin Contest Results:

Volatility Community GitHub Repository: