Announcing the Volatility 3 Public Beta!

The Volatility Team is very excited to announce the first public beta release of Volatility 3!

We presented this beta for the first time to OSDFCon attendees and received a very warm reception both during and after our presentation. As always, we are very grateful to our community for the years of support given to our trainings, book, invited speaking engagements, plugin contests, and other activities.

With this blog post, we now want to document the beta release for the entire community.

This beta release is meant to give an early view of the future direction of Volatility along with the ability to experience the new framework well in advance of its first official release next summer.

Motivation
Since being initially developed in the mid-2000s, Volatility 2 has become the de-facto framework for memory analysis research, development, and real-world analysis. Parallel to this rise in popularity, significant changes have occurred in the memory forensic landscape, including the:

• Large increases in the size of RAM samples to be analyzed
• Inclusion of multiple RAM samples in common investigative scenarios
• Number of analysis tasks (plugins) that are available and utilized in investigations
• Introduction of rapid kernel development cycles across Windows, Linux, and OS X

These changes, along with others, drove the decision to redesign and reimplement Volatility 3 as a completely new framework - meaning that every line of code has been written from scratch. This decision gave us the full ability to design a framework that not only meets the needs of current analysis, but also of analysis for years to come.

New Features

During the design phase of Volatility 3, it became clear that the choice to design the framework from scratch provided the flexibility to fulfill many of the requests made over the years by members of our community. We are happy to say that a number of these requests have now been satisfied with Volatility 3 and will be fully realized in the first official release.

For users, the highlights of the new features include:

• Major performance boosts
• The removal of reliance on --profile in order for the framework to determine which symbol table (profile) is needed to match the operating system version in the memory sample 
• Proper evaluation of 32bit code on 64bit systems, such as Window's wow64 
• Automatic evaluation of in-memory code to avoid as much manual reverse engineering on part of the analyst as possible

For developers:

• Much simpler integration into user 3rd-party interfaces and libraries
• Extensive API documentation
• The ability for plugins to directly call other plugins
• Plugin versioning
• Direct integration of custom symbol tables and data structures

Code and Documentation

The official repository for Volatility 3 is on Github within the same organization as Volatility 2.

The official documentation can be found on our Read the Docs page.

Roadmap

In order to fully meet the needs of our community, including keeping Volatility 2 stable and up-to-date with the latest operating system versions, along with the rapid development of Volatility 3, we have decided on a dual framework development cycle.

For Volatility 3, our goal is to have the first full release in August 2020. This release is slated to have complete feature parity (plugins, address spaces, etc.) with Volatility 2 as well as a number of completely new analysis features. We then expect official periodic releases to continue for years after.

For Volatility 2, the core development team will keep it fully up-to-date with features and plugins until August 2021.

In August 2021, the core development team will stop work on Volatility 2 and all efforts will be given to Volatility 3. 

This means that, for the roughly next two years, Volatility users can expect a fully featured framework in Volatility 2 while we work to bring Volatility 3 to full realization.

Getting Involved!

By releasing a beta version of Volatility 3 in the middle of the development cycle, we hoped to inspire members of the community to help with our efforts related to development, documentation, testing, and everything else involved with making Volatility 3 become the new de-facto framework of the field. While the core of Volatility 3 is implemented and stable, there is much work to be done in porting plugins and adding the envisioned brand new features.

Whether you are a college student looking to gain real-world development experience or a seasoned professional looking to have a major impact on your field, there are many tasks where help would be appreciated and recognized.

If you would like to discuss these possibilities with us, then please see our community resources, described next.

Community Resources

For the members of our community who wish to engage directly with the core development team as well as other members, we now have two resources available:

• Our new Slack server
• Our existing mailing list - vol-users

Both of these resources can be used to ask questions, get help with analysis or development tasks, keep up with our latest announcements, and for anything else related to Volatility and memory forensics.

Final Thoughts

We would again like to thank our community for the continued support!

The huge, warm reception at OSDFCon made for a very nice experience, and we always enjoy being able to engage directly with Volatility users.

Our 2020 calendars are already packed with trainings, conferences, and other community events. If you see an announcement of us participating at an event that you will also be attending, then please stop by and say hello.



-- The Volatility Team

The (5th Annual) 2017 Volatility Plugin Contest is Live!

Its that time again, folks! The 2017 Volatility Plugin contest is now live and accepting submissions until October 1st, 2017. Winners of this year's contest will be receiving over $2,250 in cash prizes as well as plenty of Volatility swag (t-shirts, stickers, mugs, sync stops, etc).

The purpose of the contest is to encourage open memory forensics research and development. It is a great opportunity for students to become familiar with memory forensics, develop a master's thesis or PhD project, as well as gain experience that will be very desirable by future employers. For those already in the field, submitting to the contest is a great way to gain experience and visibility in the memory forensics community. After the contest is over we promote the work in our conference presentations, blogs, and social media.

If you are looking for inspiration or to see the past winners, please check out the pages from 2013, 2014, 2015, and 2016. You will find projects that allow for inspection of virtual machines guests from the view of the host, recovery of in-memory browser artifacts, methods to detect stealthy rootkits, and much more.

If you have any questions please feel free to reach out to us. 

We are looking forward to another year of innovative open source research!

Malware and Memory Forensics 2017 Schedule (Now with Linux, Mac, and Surge Collect Pro)

Our most popular training course just got even better! We're happy to announce the curriculum for Malware and Memory Forensics by The Volatility Project now includes the following:

• Linux and Mac OS X memory analysis
• Windows memory acquisition with Volexity Surge Collect Pro
• Several new and challenging hands-on labs based on high profile, real world incidents

We currently have 5 MMF courses open for registration, including our first wave of events in 2017. Registrants of all classes will receive a free copy of The Art of Memory Forensics in hard/paper or electronic (eBook, Mobi, or PDF) format.

2016

August 29th - September 2nd, 2016 in Amsterdam (**nearly sold out)
October 3rd - 7th, 2016 in Herndon, VA (**nearly sold out)

2017

April 3rd - 7th, 2017 in Herndon, VA
September 18th - 22nd, 2017 in London, UK (**our first UK class in 3 years)
October 16th - 20th, 2017 in Herndon, VA

Starting with the 2017 courses, we're offering discounts on Volexity Surge Collect Pro (for Windows) at the time of registration. For more information on this package deal, see our Memory Forensics Training FAQ.

If you're interested in attending, contact us. We look forward to meeting and training with everyone!

Results from the 2015 Volatility Plugin Contest are in!

The competition this year was fierce! We received 12 plugins to the contest. Similar to last year, ranking the submissions was one of the hardest things we’ve had to do. Each plugin is unique in its own way and introduces a capability to open source memory forensics that didn’t previously exist. Although a few people will receive prizes for their work, the real winners in this contest are the practitioners and investigators in the community that perform memory forensics. 

Needless to say, we're very proud of everyone who submitted to the contest. 

Here are this year’s rankings:

1. Fred House, Andrew Davis, and Claudiu Teodorescu for the shimcachemem plugin. 
2. James Habben for the Evolve web interface to Volatility. 
3. Philip Huppert for the VM migration address space. 
4. Ying Li for the linux python strings and SSH agent plugins. 
5. Adam Bridge for the NDIS packet scanning plugin. 

Here is a detailed summary of the submissions. If you have feedback for the authors, we're sure they'd love to hear your thoughts.

1st: Shimcache Memory Scan 

This plugin by FireEye and Mandiant researchers Fred House, Andrew Davis, and Claudiu Teodorescu parses the Windows Application Compatibility Database (aka, ShimCache) from the module or process memory that contain the database. In the authors' own words:

Shim cache is a highly valuable forensic artifact used to identify evidence of file execution. In addition to recording potential file executions, the cache is ordered, meaning that an analyst can identify other files that may have executed before or after a file of interest.

Most forensic tools that parse the shim cache rely on the cache stored in the Windows registry. The cache in the registry is only updated when a system is shutdown so this approach has the disadvantage of only parsing cache entries since the last shutdown. On systems that are not rebooted regularly (e.g., production servers) an analyst must either use out-of-date shim cache data or request a system reboot.

This plugin parses the shim cache directly from the module or process containing the cache, thereby providing analysts access to the most up-to-date cache. The plugin supports Windows XP SP2 through Windows 2012 R2 on both 32 and 64 bit architectures.

Fred's Twitter: @0xF2EDCA5A

Claudiu's Twitter: @cteo13

FireEye's blog post: https://www.fireeye.com/blog/threat-research/2015/10/shim_shady_live_inv.html

Shimcachemem Submission: https://github.com/volatilityfoundation/community/tree/master/ShimcacheMemory

2nd: James Habben: Evolve Web Interface

This submission provides a web interface to Volatility built with AJAX, jQuery, and JSON. It allows the user to run several plugins at once and leverage the power of multi-processing. It shows the plugins' status and places the completed plugins near the top for easy access. The output is easily searchable given its sqlite3 backend and there is a "morph" option as well for highlighting interesting artifacts. In fact, users can create their own morphs using a plugin-based template system. Existing morphs include associating country codes with IP addresses and marking filenames that do not match the NSRL list. 

James' Submission: https://github.com/JamesHabben/evolve

James' Twitter: @JamesHabben
James' GitHub: https://github.com/jameshabben
Evolve Videos: https://youtu.be/55G2oGPQHF8 and https://youtu.be/mqMuQQowqM

3rd: Philip Huppert: VM Live Migration 

This submission includes an address space plugin for accessing memory samples found in network data captured during a VMotion live migration. In particular, it currently supports analysis of samples collected during VMotion migrations between ESXi hosts. This allows an analyst to access the entire runtime state of a virtual machine as it is being transferred between two physical hosts over the network. This has a number of valuable applications toward virtual machine introspection and forensics investigations of cloud environments. 

Philip's plugin represents interesting implications toward VM's that migrate between different cloud providers or countries. Wireshark has the capability to analyze the traffic protocols but not the payload of the traffic, so this is a new and exciting capability for both forensic researchers and offensive security analysts. 

Philip's Submission: https://github.com/volatilityfoundation/community/tree/master/PhilipHuppert

Philip's Twitter: @oilheap

Philip's GitHub: https://github.com/Phaeilo

4th: Ying Li: Python Strings and SSH Keys 

This submission includes a collection of plugins for analyzing memory samples acquired from 64-bit Linux systems. The plugins were initially presented at PyCon 2015. The first plugin in the submission, linux_python_strings, extracts memory resident Python strings from the heap of Python processes. This is accomplished by scanning the heap for Python string objects. The advantage of this approach is that we are able to leverage context that the interpreter has about the particular string and how it is being used. The author also includes a plugin for extracting strings stored in Python dictionaries. The context provided by this plugin could allow an analyst to determine which strings were associated as key-value pairs. The final plugin, linux_ssh_keys, will extract RSA keys from the heap of ssh-agent processes. These keys could be useful when investigating lateral movement or performing an investigation of a suspected person's machine.  

Ying's plugins continue the trend of moving further up the analysis stack into the application to extract more memory resident context. As for RSA keys, extracting crypto artifacts is always interesting. To date, we haven't seen anyone extract the RSA keys from ssh-agents with Volatility. Additionally, Ying created an automated test harness for generating memory samples, which we thought was particularly useful. 

Ying's Submission: https://github.com/volatilityfoundation/community/tree/master/YingLi

Ying's Twitter: @cyli
Ying's GitHub: https://github.com/cyli
Ying's PyCon 2015 Talk: https://www.youtube.com/watch?v=tMKXcc2-xO8

5th: Adam Bridge: NDIS Packet Scan 

This submission carves packets and ethernet frames from NDIS shared memory sections (regions of RAM shared between the OS and DMA NIC). Although tools exist to scan an arbitrary binary file for packets, the extra context that Adam's plugin provides is extremely valuable. For example, the methodology is far less likely to produce false positives or report fake/decoy packets. In addition to outputting text and pcap formatted results, the plugin also decodes NetBIOS names found in DNS traffic and extracts slack space between packet payloads that other tools may miss. This research represents the beginning of a very exciting realm of future plugins that focus on NDIS private data structures including data waiting in sent/received buffers. 

Adam's Submission: https://github.com/volatilityfoundation/community/tree/master/AdamBridge

Adam's Twitter: @bridgeythegeek
Adam's GitHub: https://github.com/bridgeythegeek

The following submissions appear in the order they were received. As previously mentioned, everyone succeeded in solving a specific problem that they (and undoubtedly others) faced. For this, they deserve huge props. We look forward to seeing future work by these authors!

Joe Greenwood: Hacking Team RCS Attribution  

This plugin searches a memory dump for evidence of the Hacking Team Galileo Remote Control System (RCS), and attempts to attribute the infection to particular Hacking Team client. One of our favorite aspects of this plugin is that it detects RCS based on its predictable named shared memory sections, a creative alternative to scanning for the typical byte signatures and mutexes. Its also a really nice touch to provide the extra attribution context via the watermark lookups. 

Joe's Submission: https://github.com/volatilityfoundation/community/tree/master/JoeGreenwood

Joe's Twitter: @SeawolfRN
Joe's (4ARMED) GitHub: https://github.com/4ARMED
Joe's Blog: https://www.4armed.com/blog/memory-forensics-detecting-galileo-rcs-windows/

Alexander Tarasenko: Pykd/Windbg Address Space

This submission lets you integrate Volatility into Windbg through Pykd. You're then able to query for addresses of critical data structures, functions, and other variables using the debugging APIs. Additionally, you can connect Volatility to a running Windows system in debugging mode and run Volatility plugins against the live system, which particularly comes in handy for malware/rootkit analysis (not necessarily IR/forensics). 

The author of this address space also mentioned a new project named Karmadbg (see the link below) which is a GUI intended for development of new debugging and memory analysis scripts. Check it out! Unfortunately, documentation in English is not yet available. 

Alexander's Submission: https://github.com/volatilityfoundation/community/tree/master/AlexanderTarasenko

Pykd's Twitter: @pykd and @pykd_dev
Pykd's Website: https://pykd.codeplex.com
Karmadbg's Website: https://karmadbg.codeplex.com

Loïc Jaquemet: Haystack 

These plugins are an interface between the Volatility framework and the haystack framework. While Volatility establishes a forensic framework to analyse a system's RAM, the haystack framework is intended to analyse a process's RAM, allowing an analyst to search for defined structures in a process's memory. You can build structure definitions from known/public C header files or with Python's ctypes library (based on undocumented or reverse engineered data structures) and then plug them into this framework to scan across process heaps, all process memory allocations, etc. The author provided examples of constraints that can find openssl cipher contexts, session keys, and passphrases, but its surely not limited to those types of data. Give it a shot and see what you can find! 

Loïc's Submission: https://github.com/volatilityfoundation/community/tree/master/Lo%C3%AFcJaquemet

Loïc's Twitter: @trolldbois
Loïc's GitHub: https://github.com/trolldbois

The python-haystack module: https://github.com/trolldbois/python-haystack

May Medhat (et. al.): GVol Tool

This tool by EG-CERT researchers May Medhat and Mohamad Shawkey provides a (thick) GUI front end for Volatility written in Java. It lets you run preconfigured batch scripts against a memory dump with just a couple mouse clicks or you can customize your own batch scripts. You can choose from categories such as Rootkits, Kernel Artifacts, Networking, etc. and they link up with Volatility plugins in the backend. We imagine this tool will be very handy for analysts who are not comfortable on the command line or with Volatility usage in general. Additionally, for those who are seasoned Volatility users, this tool can reduce the amount of time you spend typing commands before you actually dive into the details of your case. 

May's GitHub: https://github.com/May-Medhat
Mohomad's GitHub: https://github.com/m-shawkey

GVol User Guide: https://github.com/eg-cert/GVol/blob/master/docs/GVol-User%20Guide.pdf

EG-CERT's GitHub: https://github.com/eg-cert/GVol

Monnappa Ka: Linux Memory Diff 

This plugin uses the Volatility advanced memory forensics framework to run various plugins against a clean and infected Linux memory image and reports the changes. Many times while doing memory analysis (or malware analysis) an analyst is presented with an abundance of data and the analyst has to manually find the malicious artifacts from that data which takes time and effort. This tool helps in solving that problem by comparing the results between the clean and infected memory images. This tool helps speed up analysis, reduce manual effort and allows you to focus on the relevant data.

Monnappa's Submission: https://github.com/volatilityfoundation/community/tree/master/MonnappaKa

Monnappa's Twitter: @monnappa22
Monnappa's GitHub: https://github.com/monnappa22
Monnappa's Blog: http://malware-unplugged.blogspot.com

Bart Inglot: Scheduled Task and Job Scanners 

This plugin scans for job files and prints out their information. It's useful from a DFIR perspective, since job files are often used by attackers in order to run programs with SYSTEM privileges, run at scheduled moments, or to move laterally within a network. Aside from the job plugin for Volatility, adapted from Jamie Levy's jobparser.py (http://gleeda.blogspot.com/2012/09/job-file-parser.html) Bart also has written standalone carvers for jobs and scheduled tasks that work against memory samples, disk images, and other binary files. 

Bart's Submission: https://github.com/volatilityfoundation/community/tree/master/BartoszInglot

Bart's Twitter: @BartInglot
Bart's GitHub: https://github.com/binglot
Bart's Blog: http://passionateaboutis.blogspot.com

Volatility Updates Summer 2015

Summer 2015 has been quite a busy time for the memory forensics community. We wanted to write a quick update to talk about some recent events and research as well as upcoming news.

Conferences

Black Hat Vegas 2015

We wanted to again thank everyone who came out and supported us during Black Hat. Between our Arsenal demo, book signing, and party, we met hundreds of Volatility users and fans. Your support and enthusiasm is greatly appreciated. Come back next year for twice the champagne, twice the suite size, and twice the fun!

HTCIA International Conference (Orlando) 

We're putting on a lab session at HTCIA's International Conference in Orlando next week. You can also stop by the Volexity booth for a chance to win a free seat at any upcoming Windows Malware and Memory Forensics Training course.

Open Source Digital Forensics Conference (OSDFC) 2015

The Volatility team will once again be presenting the latest in memory forensic research at OSDFC 2015. This year we will be focusing on the anti-forensic capabilities of PlugX as well as new Volatility capabilities that can auto-detect them. We will also be discussing the results of the 2015 Volatility Plugin contest. 

OSDFC has a great lineup this year so you should try and attend. We hope to see many Volatility users while we are out there.

Research

A New Paper on OS X Memory Forensics

At DFRWS 2015, Dr. Golden Richard and I published a paper on OS X memory forensics entitled: Advancing OS X Rootkit Detection. The purpose of this paper was to document gaps in existing OS X rootkit detection techniques and then develop new methods, in the form of Volatility plugins, to close these gaps. The plugins introduced in the paper will be committed to GitHub in the coming weeks.

DFRWS had a number of good submissions this year, and we recommend browsing the program for other interesting forensics research.

Volatility vs Hacking Team

The malware used by Hacking Team to control victim's computers is known as Galileo RCS, and in a well done blog post, Joe Greenwood shows how to use Volatility to detect RCS in a number of ways.

RCS now joins a long list (Stuxnet, Careto, Flame, and more) of 'advanced', 'stealthy' malware that immediately falls to inspection by memory forensics.

Volatility at PyCon

At PyCon 2015, Ying Li showed how to use newly developed Volatility capabilities in order to find artifacts of Python scripts that were executing on the system. This was very interesting work, and we suggest watching the video of her talk.

Projects Building on Volatility

VolDiff

VolDiff is a project that compares the results of a number of Volatility plugins against two memory samples and automatically the reports the differences. Compared to manually running and comparing the plugins, this can save a substantial amount of time.

The purpose of VolDiff is to compare in-memory artifacts both before (clean state) and after (post-install state) an application, such as a malware sample, has executed. The new artifacts that appear post-installation can be immediately isolated for further analysis and/or for the creation of highly-effective IOCs.

If you wanted to get started with VolDiff then you should read the author's post on analyzing DarkComet with VolDiff.

Evolve

Evolve is an open source web interface to Volatility. It is under very active development and is constantly having new features added. Consult the README on GitHub for the latest features, and be sure to follow the tool's author on Twitter.

To see Evolve in action, check out a video showing the basic features here and advanced features here.

As much as we love the command line, it is sometimes nice to have a GUI visualize and shuffle data for you!

Plugin Contest

The 2015 Volatility Plugin Contest is underway and accepting submissions until October. The contest is a great way to win cash and other prizes, gain recognition in the community and become more familiar with Volatility and Python. We feature the research submitted to the contest on this blog, during conferences, presentations, and all throughout social media and our mailing lists.

New Volatility Capabilities

Windows 10 Support

While not merged in the official Volatility branch (yet), the Windows 10 branch is currently under active development. Nearly all of the plugins are working at the current time, and we would be very appreciative of any bug reports you may have when testing the branch. Please post any bugs to the GitHub issue tracker. Currently supported functionality includes process and kernel module listing, all pool scanning plugins (files, mutexes, processes, drivers, etc.), handles, DLLs, PE file extraction, process memory (VAD) parsing, service enumeration, cached file extraction, and memory signature scanning with Yara.

OS X 10.10.x Support and New Plugins

Volatility now has official support for OS X 10.10.4 and 10.10.5, which are the latest two versions. We have also tested Volatility on a preview release of 10.11, and it appears that all of the plugins work as expected. We will release an official profile for 10.11 once Apple releases debug kits (we are currently using special custom built profiles).

Volatility also has a new plugin named mac_get_profile. This plugin allows Volatility to auto-detect which profile (OS version) matches the given memory sample. To use this plugin you do NOT need any OS X profiles installed. Instead, you can run a fresh checkout of Volatility, determine the profile by using mac_get_profile, and then download the correct profile from our profiles repository.

To use mac_get_profile, simply pass the path to your memory sample as the -f option and then put mac_get_profile after it:

$ python vol.py -f <path to memory sample> mac_get_profile

Linux

Volatility Linux support has now been tested through kernel version 3.19. As many of the data structures do not change between versions, we expect that most or all of the plugins will work with bleeding edge developments kernels as well. Please file an issue if you encounter any bugs.

Memory Forensic Trainings

Our memory forensics training class in Amsterdam is now SOLD OUT.  We appreciate the support and word of mouth praise from past attendees as well as fans of the project.

Our current course is 5 days of memory forensics and malware analysis training against Windows systems. Full information on the course, as well as upcoming dates and locations, can be found here. If our current set of public offerings does not work for your company then please contact us about conducting a private training at one of your facilities.

Using mprotect(.., .., PROT_NONE) on Linux

After deciding to revisit some old code of mine (ok, very old), I realized that there was something different about how Linux was allocating pages of data I wanted to hide.   At first, I was glad that I couldn't see the data using yarascan, but then I realized that I was unable to access the memory regions at all in linux_volshell to verify that they were, in fact, obfuscated. So I decided to take a look at using a smaller, stripped down program. Below is one such example, comments are included to explain what is happening:

int main( int argc, char *argv[]){
    // pid: the process ID of this process 
    // so we can print it out
    int pid;
    pid = getpid();

    //size: an integer to hold the current page size
    int size;
    size = getpagesize();

    //[1] create two pointers in order to allocate 
    //memory regions
    char *buffer;
    char *buffer2;

    //unprotected buffer:
    //allocate memory using mmap()
    buffer2 = (caddr_t) mmap(NULL,
                      size,
                      PROT_READ|PROT_WRITE,
                      MAP_PRIVATE|MAP_ANONYMOUS,
                      0,0);

    //[2] put some characters in the allocated memory
    //we're setting these characters one at a time in order
    //to avoid our strings being detected from the binary itself
    buffer2[0] = 'n';
    buffer2[1] = 'o';
    buffer2[2] = 't';
    buffer2[3] = ' ';
    buffer2[4] = 'h';
    buffer2[5] = 'e';
    buffer2[6] = 'r';
    buffer2[7] = 'e';
    
    //protected buffer:
    //allocate memory with mmap() like before
    buffer = (caddr_t) mmap(NULL,
                      size,
                      PROT_READ|PROT_WRITE,
                      MAP_PRIVATE|MAP_ANONYMOUS,
                      0,0);

    //[2] put some characters in the allocated memory
    //we're setting these characters one at a time in order
    //to avoid our strings being detected from the binary itself
    buffer[0] = 'f';
    buffer[1] = 'i';
    buffer[2] = 'n';
    buffer[3] = 'd';
    buffer[4] = ' ';
    buffer[5] = 'm';
    buffer[6] = 'e';

    //[3] protect the page with PROT_NONE:
    mprotect(buffer, size, PROT_NONE);

    //[4] print PID and buffer addresses:
    printf("PID %d\n", pid);
    printf("buffer at %p\n", buffer);
    printf("buffer2 at %p\n", buffer2);

    //spin until killed so that we know it's in memory:
    while(1);
    return 0;
}

Now we'll just compile the above program and run it. In short, the above program [1] creates two buffers, [2] places characters in these buffers, [3] then calls mprotect() on one of them with PROT_NONE; [4] the program then prints out its process ID and the virtual addresses of the aforementioned buffers:

$ ./victim
PID 29620
buffer at 0x7f2bec9a1000
buffer2 at 0x7f2bec9a2000

Let's set up a config file so that we don't have to type as much:

$ cat linux.config
[DEFAULT]
LOCATION=file:///Path/to/Virtual%20Machine/Linux%20Mint%20Cinnamon/Mint%2064-bit-d583834d.vmem
PROFILE=LinuxLinuxMintCinnamonx64x64
PID="29620"

If we try to search for the strings we placed in the buffers we get:

$ python vol.py --conf-file=linux.config linux_yarascan -Y "not here"
Volatility Foundation Volatility Framework 2.4
Task: victim pid 29620 rule r1 addr 0x7f2bec9a2000
0x7f2bec9a2000  6e 6f 74 20 68 65 72 65 00 00 00 00 00 00 00 00   not.here........
0x7f2bec9a2010  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0x7f2bec9a2020  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
[snip]
$ python vol.py --conf-file=linux.config linux_yarascan -Y "find me"
Volatility Foundation Volatility Framework 2.4

Well that was disappointing, we couldn't find the "find me" string. What you would expect, is to be able to access the memory contents using Volatility. Let's use the linux_volshell plugin to explore the victim process' memory and see if we can access the memory addresses (given to us from the program at run time) directly. First we'll examine the contents at 0x7f2bec9a2000:

$ python vol.py --conf-file=linux.config linux_volshell
Volatility Foundation Volatility Framework 2.4
>>> Current context: process victim, pid=29620 DTB=0x3cfb1000
>>> db(0x7f2bec9a2000)
0x7f2bec9a2000  6e 6f 74 20 68 65 72 65 00 00 00 00 00 00 00 00   not.here........
0x7f2bec9a2010  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0x7f2bec9a2020  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0x7f2bec9a2030  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0x7f2bec9a2040  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0x7f2bec9a2050  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0x7f2bec9a2060  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0x7f2bec9a2070  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................

As we see, we have the contents that we'd expect. Now to try to access the other memory location:

>>> db(0x7f2bec9a1000)
Memory unreadable at 7f2bec9a1000

We see that we've been rejected. After a bit of investigation, we see that the point of failure is in the entry_present() function in the amd64.py address space. In order to find the value of the entry that failed, we'll print it out. First we'll add a print statement to entry_present():

    def entry_present(self, entry):
        if entry:
            if (entry & 1): 
                return True

            arch = self.profile.metadata.get('os', 'Unknown').lower()

            # The page is in transition and not a prototype.
            # Thus, we will treat it as present.
            if arch == "windows" and ((entry & (1 << 11)) and not (entry & (1 << 10))):
                return True

            # we want a valid entry that hasn't been found valid:
            print hex(entry)
        return False

Now let's see what the entry is:

$ python vol.py --conf-file=linux.config linux_yarascan -Y "find me"
Volatility Foundation Volatility Framework 2.4
0x2681d160

So let's look a little closer at the page table entry:

>>> print "{0:b}".format(0x2681d160)
100110100000011101000101100000

The way we read this is from right to left.  The entry_present() function checks the 0th bit to see if the entry is present. In this case the bit is not set (0); therefore this function returns False.

At this point, I decided to take a look at the Linux source code to see how mprotect() is actually implemented. As it turns out, the _PAGE_PRESENT bit is cleared when mprotect(...PROT_NONE) is called on a page and the _PAGE_PROTNONE bit is set [3]. Looking at how _PAGE_PROTNONE is defined [4][5][6] we'll see that it's actually equivalent to the global bit (8th bit) [1][2]. So let's look at our page table entry again, we'll notice that the 8th bit is indeed 1:

>>> print "{0:b}".format(0x2681d160)
100110100000011101000101100000

So let's patch the entry_present() function with our findings:

    def entry_present(self, entry):
        if entry:
            if (entry & 1): 
                return True

            arch = self.profile.metadata.get('os', 'Unknown').lower()

            # The page is in transition and not a prototype.
            # Thus, we will treat it as present.
            if arch == "windows" and ((entry & (1 << 11)) and not (entry & (1 << 10))):
                return True

            # Linux pages that have had mprotect(...PROT_NONE) called on them
            # have the present bit cleared and global bit set
            if arch == "linux" and ((entry & (1 << 8))):
                return True

        return False

Let's see if we get anything back:

$ python vol.py --conf-file=linux.config linux_yarascan -Y "find me"
Volatility Foundation Volatility Framework 2.4
Task: victim pid 29620 rule r1 addr 0x7f2bec9a1000
0x7f2bec9a1000  66 69 6e 64 20 6d 65 00 00 00 00 00 00 00 00 00   find.me.........
0x7f2bec9a1010  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0x7f2bec9a1020  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
[snip]

Success! So now we're able to see the "forbidden" string that we couldn't access before.  This goes to show that sometimes you have to dig a little into "why" something is not working as expected. In this case, we lucked out and had source code to examine, but sometimes things are not as easy as all that. Both intel.py and amd64.py have been patched in order to accommodate memory sections that have had mprotect() called on them with PROT_NONE.

References

[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming http://developer.amd.com/wordpress/media/2012/10/24593_APM_v21.pdf

[2] Intel(R) 64 and IA-32 Architectures Software Developer's Manual Volume 3A: System Programming Guide. Section 4.3 http://www.intel.com/products/processor/manuals/index.htm

[3] https://www.kernel.org/doc/gorman/html/understand/understand006.html


[4] http://www.cs.fsu.edu/~baker/devices/lxr/http/source/linux/arch/x86/include/asm/pgtable_types.h#L63

[5] http://www.cs.fsu.edu/~baker/devices/lxr/http/source/linux/arch/x86/include/asm/pgtable_types.h#L29


[6] http://www.cs.fsu.edu/~baker/devices/lxr/http/source/linux/arch/x86/include/asm/pgtable_types.h#L18

New Volatility 2.4 Cheet Sheet with Linux, Mac, and RTFM

Our Windows Malware and Memory Forensics Training class is intense and rigorous, because its designed to reflect real world investigations. When you have a limited amount of time and you're being pressured for reliable answers - every minute counts. Sometimes you just gotta cheat...and when you do, you might as well use an Official Volatility Memory Analysis Cheat Sheet!

The 2.4 Edition features an updated Windows page, all new Linux and Mac OS X pages, and an extremely handy RTFM-style insert for Windows memory forensics.

For in-depth examples and walk-throughs of using the commands in this cheat sheet, make sure to get your copy of The Art of Memory Forensics!

New Paper: In Lieu of Swap: Analyzing Compressed RAM in Mac OS X and Linux

A research paper (slides here) that I worked on with Golden G. Richard was recently published at DFRWS 2014 and received the Best Paper award! The paper, In Lieu of Swap: Analyzing Compressed RAM in Mac OS X and Linux, analyzed the in-memory, compressed swapped stores on recent Mac and Linux versions.

As you are likely aware, operating systems will traditionally swap unneeded pages out to disk in order to free the pages for currently running applications to use. This swapping process has historically presented issues for forensics analysis as attempting to acquire both physical memory and the swap file(s) in a consistent state can be a daunting task outside of virtual machine environments. Furthermore, Mac has provided encrypted swap (Secure virtual memory) since Mountain Lion and Linux users could optionally utilize encrypted swap for years. Similarly, Windows has the EncryptPagingFile option that can be turned on by administrative users. Recovering the keying materials for these encrypted stores can require modification to existing acquisition tools as well as specialized research.

A new advancement in operating system design, the creation of memory-only, compressed swap stores, presents both challenges and opportunities for memory forensics practitioners. When enabled, these stores attempt to avoid writing swapped pages to disk for performance reasons and instead compresses and stores them into a reserved pool of memory. When a page is later recalled from the pool, a simple decompression operation is all that is needed to be performed. This is much quicker than reading from even the fastest PCI-E storage provided by Apple devices.

At first, this store can be challenging to investigators as simple methods of investigation, such as strings analysis or file carving, will fail since the data is in its compressed state. This store can be helpful though as normal physical memory capture techniques will completely collect the store. This is opposed to traditional swap collection that required timing memory acquisition with disk acquisition to avoid smearing.

The paper presents the algorithms used by both Mac and Linux to implement the compressed stores. We also describe our Volatility plugins that can fully locate and decompress all pages kept within the stores. Finally, we present data sets from sample machines that we tested the plugins and analysis on.

If you have any questions on the paper feel free to reach out to either myself or Golden.

Presenting Volatility Foundation Volatility Framework 2.4

The release of this new Volatility version coincides with the publication of The Art of Memory Forensics. It adds support for Windows 8, 8.1, 2012, and 2012 R2 memory dumps, Mac OS X Mavericks (up to 10.9.4), and Linux kernels up to 3.16. New plugins include the ability to extract cached Truecrypt passphrases and master keys from Windows and Linux memory dumps, investigate Mac user activity (such as pulling their contact database, calendar items, PGP encrypted mails, OTR Adium chat messages, etc), and analyze advanced Linux rootkits. See below for a detailed change log.

Binary releases, including pre-built executables for Windows and Mac OS X can be found on the Volatility Foundation website: http://www.volatilityfoundation.org. We've also now moved our source code repository to Github: https://github.com/volatilityfoundation. Note that there's a separate repository containing over 160 Linux profiles for 32- and 64-bit OpenSuSE, Redhat, Debain, Ubuntu, Fedora, and CentOS (thanks Kevin!); and all Mac OS X profiles from 10.5 to 10.9.4. 

The detailed change log is below:

Windows Memory Forensics 

• Truecrypt plugins (summary, cached passphrases, master keys)
• Apihooks support for 64-bit memory images 
• Apihooks plugin detects JMP FAR hook instructions 
• Hashdump, Cachedump, and Lsadump plugins updated for x64 and Win8/2012
• Callbacks and timers plugins work on 64-bit memory images 
• Mftparser identifies NTFS alternate data streams 
• Mftparser -D option extracts MFT-resident files to disk
• Ability to scan for multiple executive object types concurrently with a single pass through the memory dump 
• Procmemdump and procexedump condensed into "procdump" (and --memory option available)
• Envars plugin has a --silent flag to ignore common/default environment variables 
• Vadtree plugin in graphviz output mode (--output=dot) color codes nodes per heap, stack, mapped file, DLL, etc.
• Getsids plugin automatically resolves user and service SIDs 
• Timeliner plugin supports --machine to identify the source in multi-source timelines 
• Verinfo (PE version info) plugin updated and moved into core framework 
• Strings translator prints "FREE MEMORY" for data found in deallocated regions (used to skip them)
• Vadinfo plugin allows --addr to specify one region rather than printing them all 
• Yarascan plugin allows you to control --size (bytes in preview) and --reverse (show data *before* a hit)
• Volshell plugin has new APIs proc(), addrspace(), getprocs(), and getmods() for easy access
• All process based plugins accept --name (process name regular expression filter)
• Added the auditpol plugin to check audit policies 
• Added the cmdline plugin to show process command line arguments 
• Volshell plugin can recursively print structure members (similar to windbg's dt /r)
• New pooltracker plugin allows analysis of kernel pool tag statistics 
• New bigpools plugin allows finding big page pool allocations 
• Svcscan plugin prints service start type (manual, automatic, disabled, etc)
• Added a plugin to find and print text on the Notepad application's heap
• PE dumping plugins (procdump, dlldump, moddump) support --fix to fix the image base value 
• Joblinks plugin for getting information for job objects

Address Spaces / File Formats

• Support for QEMU virtual machine memory images 
• Support for "split" VMware files (memory in .vmem and metadata in .vmss/.vmsn)
• Support for Windows BitMap crash dumps (created by Windows 8 / 2012 on BSOD)

Mac Memory Forensics 

• Support for Mavericks through 10.9.4
• Mac string translation added 
• Recover sent and received Adium messages, including those protected by OTR 
• Enumerate contacts from the Contact application's database
• Extract the HTML content of notes from the Notes application 
• Ability to reveal clear-text PGP emails sent or received with the Mail application 
• Locate Apple Keychain encryption keys in memory (for cracking with Chainbreaker)
• Find API hooks in both the kernel and process memory
• List IP and socket filters
• Extract loaded kernel extension to disk
• Find suspicious process mappings (i.e. injected code) 
• Find hidden kernel extensions
• Recovered files cached in memory

Linux Memory Forensics 

• Support for Linux kernels through 3.16
• Linux string translation added
• Detect API hooks in both userland processes and the kernel
• Detect GOT/PLT overwrites
• Find hollowed executables
• Find suspicious process mappings
• Library listing using the loader’s data structures
• Extract process ELF executables and libraries to disk
• List network interfaces in promiscuous mode
• List processes that are using raw sockets
• Find hidden kernel modules
• List Netfilter hooks
• Extract cached Truecrypt passphrases 

The Art of Memory Forensics

By now, some of you may have realized that The Art of Memory Forensics is available for pre-order on Amazon.

The Art of Memory Forensics: Detecting Malware and Threats in Windows, Linux, and Mac Memory

This book is written by 4 of the core Volatility developers - Michael Ligh (@iMHLv2), Andrew Case (@attrc), Jamie Levy (@gleeda) and AAron Walters (@4tphi). We've been collaborating for well over 6 years to design the most advanced memory analysis framework and we're excited to be collaborating on a book.

The content for the book is based on our Windows Malware and Memory Forensics Training class, which has been executed in front of hundreds of students across the US and Europe. As an added bonus, the book will also cover Linux and Mac memory forensics.

Unlike some other books, which will run you $50 for 250 pages (we call those expensive brochures), The Art of Memory Forensics will be over 700 pages. In fact, its still possible that we may grossly exceed that and have to split it into two editions, which will royally piss off our publisher.

An electronic book will be available for Kindle initially and at some point, also on Safari Bookstore.

Results are in for the 1st Annual Volatility Framework Plugin Contest!

We are excited to announce the results of the 1st Annual Volatility Plugin Contest. We were pleasantly surprised with 8 submissions to the contest. Each submission provides an exciting new capability to the memory analysis community or demonstrates the power of Volatility to solve a variety of real world problems. The submissions included everything from new plugins to new address spaces and operating system plugins to application plugins.  We also had submissions for Linux, Windows, and OS X. It's great to see so many people giving back to the memory analysis community and taking this opportunity to do pioneering research in such an exciting field.

Given the number of deserving submissions we received, the judging took a little longer than we originally anticipated. We wanted to make sure that we were able to thoroughly test and verify each submission. We would like to thank all the participants for their submissions. A number of these submissions will be highlighted in upcoming blog posts and at OMFW 2013!

The winners of the 1st Annual Volatility Framework Plugin Contest are:

1. Mariano Graziano from EURECOM with Actaeon, Intel VT-x introspection.
2. Cem Gurkok with OS X rootkit detection and Window's security auditing plugins.
3. Jeff Bryner with the Facebook and Twitter artifact extraction.
4. Carl Pulley with a plugin to find the nearest function/method within a symbol table & Edwin Smulders with his Linux process information, stack analysis, and syscall register plugins [Note: Carl and Edwin tied for 4th place, this is not a joint submission]
5. Jamaal Speights with extracting networking packets from memory samples.

Honorable Mention: Jeremy Jones from Delphix with a plugin to convert VMware suspended state to Illumos debug format

1st Place 

Mariano Graziano from EURECOM with Actaeon, Intel VT-x introspection.
Description:
This submission enables memory forensics of guest operating systems in virtualization environments using Intel VT-x technology. This includes the ability to locate memory resident hypervisors and nested virtualization.  It's current implementation enables virtual machine instrospection of 32-bit Windows guests. It was tested with KVM, Xen, VMware Workstation, VirtualBox and HyperDbg.
Download Link:
downloads.volatilityfoundation.org/contest/2013/MarianoGraziano_Actaeon.zip
Related Links:
http://www.s3.eurecom.fr/tools/actaeon/
https://github.com/eurecom-s3/actaeon
http://www.s3.eurecom.fr/docs/raid13_graziano.pdf
Author's Twitter: @emd3l

2nd Place 

Cem Gurkok with the Window's security permission plugin
Description:
This plugin displays the security permission information for files, processes, services, tokens, threads, devices, and registry keys. The information includes DACLs, SACLs (Discretionary/System access control lists), object integrity level, and object ownership. Security permission information is obtained from the object’s security descriptor. This plugin can help administrators proactively assess the security of their systems and can also help determine possible "holes" that led to successful privilege escalation attacks. This plugin dumps verbose information that you can categorize and filter for your needs, and it also supports all major 32-bit and 64-bit Windows operating systems.
Download Link:
downloads.volatilityfoundation.org/contest/2013/CemGurkok_WindowsSecurity.zip
Author's Twitter: @CGurkok

Cem Gurkok with the OS X rootkit detection plugins
Description:
This submission provides detection capabilities for a number of rootkit hooking techniques within 64 bit OS X:

• Direct syscall table modification
• Syscall function inlining (ie DTrace hooks)
• Patching the syscall handler (ie, shadow sycall table)
• Hooked functions in kernel/kext symbol tables
• Modified IDT descriptors
• Modified IDT handlers

There is published research and example rootkits that are leveraging the exact hooking techniques detected by this plugin, and the addition of the plugin into the core framework will greatly enhance Volatility's Mac rootkit detection capabilities.
Download Link:
downloads.volatilityfoundation.org/contest/2013/CemGurkok_OSXDetect.zip
Related Links:
http://siliconblade.blogspot.com/2013/07/idt-hooks-and-detecting-them-in-osx.html
http://siliconblade.blogspot.com/2013/07/back-to-defense-finding-hooks-in-os-x.html
http://siliconblade.blogspot.com/2013/07/offensive-volatility-messing-with-os-x.html
http://siliconblade.blogspot.com/2013/05/checkdtrace-volatility-plugin-arises.html

Author's Twitter: @CGurkok

3rd Place 

Jeff Bryner with the Facebook and Twitter artifact extraction
Description:
This submission provides plugins for carving Twitter and Facebook artifacts from a process' address space.  This is accomplished by scanning the address space for the json/html structures that are used by the social media applications.  Examples of information extracted include: Twitter direct messages, identifying user information, Facebook direct messages, etc.
Download Link:
downloads.volatilityfoundation.org/contest/2013/JeffBryner_FacebookTwitter.zip
Related Links:
https://github.com/jeffbryner/volatilityPlugins
http://www.youtube.com/watch?v=K_gBpdK936o
Author's Twitter: @0x7eff

4th Place (tie)

Carl Pulley with a plugin to find the nearest function/method within a symbol table.
Description:
This submission demonstrates the usefulness of being able to dynamically extract Window's symbol information.  It includes a plugin that will automatically extract symbol information from PDB files associated with memory resident modules.   The submission also includes a "profile modification" that creates a new member of the _EPROCESS object, which facilitates "nearest symbol" lookups of addresses. This can be very useful when investigating unknown pointers or the control flow history of a corrupted execution stack.
Download Link:
downloads.volatilityfoundation.org/contest/2013/CarlPulley_Symbols.zip
Related Links:
https://github.com/carlpulley/volatility/blob/master/symbols.py
https://code.google.com/p/pdbparse/issues/detail?id=13
Author's Github: https://github.com/carlpulley

Edwin Smulders with his Linux process information, stack analysis, and syscall register plugins
Description:
This submission provides plugins for extracting information from x86_64 Linux memory samples. They provide extensive insight into the state of the system at the time of the memory samples. Examples of the extracted information include:

• Detailed process information
• Networking data structures
• Detailed analysis and annotation of the execution stacks
• System call context

Download Link:
downloads.volatilityfoundation.org/contest/2013/EdwinSmulders_Symbols.zip
Related Links:
https://github.com/Dutchy-/volatility-plugins

Author's Twitter: @0x445554434859

5th Place 

Jamaal Speights with a plugin that extracts networking packets from memory samples.
Description:
The ethscan plugin provides the ability to recover Ethernet frames from memory samples. It provides extraction support for both IPV4 and IPV6. It also provides the option to extract the frame data to either binary files or to a pcap file. It should work against any binary file (not just memory dumps).
Download Link:
downloads.volatilityfoundation.org/contest/2013/JamaalSpeights_Network.zip
Related Links:
https://code.google.com/p/jamaal-re-tools/source/browse/volplugins/ethscan.py
Author's Twitter: @jamaalspeights

Honorable Mention

Jeremy Jones from Delphix with a plugin to convert VMware suspended state to Illumos debug format
Description:
This submission includes a plugin that converts a VMware suspended state file (.vmss) into a format supported by Illumos debugging tools (mdb, adb, etc.). It was tested on an OpenIndiana VM, which it converted successfully and whose files worked well when anaylzed with mdb. This plugin was created to solve a real world system administration challenge of collecting a crash dump from a system that was hanging during the boot process. It demonstrates the power of memory analysis beyond just forensics and security.
Download Link:
downloads.volatilityfoundation.org/contest/2013/JeremyJones_Illumos.zip


The Volatility Foundation would like to thank everyone who participated in this year's contest. Memory analysis is one of the most exciting and important fields in information security and digital forensics. It requires a deep technical skill set and an investigative mindset that are often rare to find. The participants in the plugin contest were not only competing for prizes but they were also contributing pioneering research that will help shape the future of the field. While this page highlighted research that was officially submitted to the contest, there were also a few other efforts worth mentioning: a new Xen address space written by Nehal Bandi from Citrix Systems and patches to mftparser and timeliner that facilitate including memory artifacts into Plaso and log2timeline by David Nides and Kristinn Gudjonsson.

If you have any questions about a particular submission, please contact the participant directly.

MoVP II - 3.5 - Utilizing the kmem_cache for Android Memory Forensics

This post will discuss utilizing plugins that leverage the kmem_cache in order to perform deep memory forensics of Android devices. In previous Linux posts, we have briefly mentioned these plugins, but a full walk through has not been done. Also, as we will see, these plugins are highly effective on Android, while not as effective on recent Intel-based systems due to differences in memory allocation and de-allocation strategies.

kmem_cache, SLAB & SLUB

The Linux kmem_cache is part of the kernel's infrastructure for handling memory allocations and de-allocations that cannot experience delays (e.g. process creation, network packet handling). To accomplish this, memory pools for a particular structure type and size are pre-allocated upon system startup. Then, when a structure allocation is requested, the request can be immediately serviced without needing to allocate more regions. Also, the allocator keeps track of all free pages and can reuse chunks immediately after they are freed.

From a memory forensics perspective, we can leverage these features in a few ways. First, we can use active (allocated) structures in order to find instances of a particular structure type. This is best seen in the kmem_cache column of linux_psxview. This column is populated by enumerating all of the task structures active in the task_cachep kmem_cache at the time of the memory capture.

The second way we can leverage kmem_cache is to walk the free lists and recover as much information as possible from de-allocated entries. Structure members that are stored within the structure, such as simple integers or character arrays, will be left in-tact until the host structure is re-used and overwritten. Depending on how much time lapsed between de-allocation and the memory capture being taken, we may even be able to recover reference values as the pointer can still point to valid data.

The internals of these memory allocations and de-allocations depend entirely on the allocator chosen. There are currently two main allocators in-use, SLAB and SLUB, and they have a large effect on which objects are recoverable. SLAB was the original allocator and left many freed entries active and kept direct references to all active objects. SLUB on the other hand makes life much more difficult in both instances.

Luckily for us, all Android ROMs we have tested chose to use the SLAB allocator. This means that we cannot not only find active structures at will, but also recover a wealth of previous information "forgotten" by the operating system.

Recovering Data from kmem_cache

linux_vma_cache

Each memory mapped file in Linux is represented by a vm_area_struct structure. The vma kmem_cache is responsible for allocating and de-allocating them. 

$ python vol.py --profile=LinuxEvo4GARM -f Evo4GRodeo.lime linux_vma_cache
Process          PID    Start      End        Path
---------------- ------ ---------- ---------- ----
                        0x48a4b000 0x48a4c000
                        0x46dfa000 0x46e12000
                        0x45496000 0x454a0000 app/htccalendarwidgets.apk
                        0x45136000 0x45137000 app/htccalendarwidgets.apk
                        0x443da000 0x443db000 app/htccalendarwidgets.apk
                        0x4513e000 0x451d6000 app/HtcDialer.odex

In this we can see information on each mapping, including the path. Note that in this particular sample the Process and PID columns are empty due to the kernel not implementing the 'owner' member of mm_struct (the parent struct of a process' memory mappings). For kernels with this option enabled the columns would be populated.

If we pass the -u flag to any of the kmem_cache plugins, the unallocated entries will be recovered. For memory mappings this can be extremely useful as it shows files and paths that were mapped into a processes' address space, even if the process has since exited or the file deleted from disk.

linux_pslist_cache

This plugin recovers processes and their associated information. Data found includes the process ID, user ID and group ID, and the start time of the process. By recovering unallocated entries, not only can we determine processes that previously executed, but we can include their start time within a timeline of in-memory data.

$ python vol.py --profile=LinuxEvo4GARM -f Evo4GRodeo.lime linux_pslist_cache
0xcafb6000 Binder Thread #      1856            10034           10034  0x23d80000 2012-08-05 02:32:21 UTC+0000
0xcafb6400 oid.voicedialer      1841            10087           10087  0x27d78000 2012-08-05 02:32:21 UTC+0000
0xcafb6c00 HeapWorker           1842            10087           10087  0x27d78000 2012-08-05 02:32:21 UTC+0000
0xc3d3a400 Signal Catcher       884             10092           10092  0x28d7c000 2012-08-05 02:21:44 UTC+0000
0xc3d3a800 RefQueueWorker@      1647            10009           10009  0x2cc04000 2012-08-05 02:30:49 UTC+0000
0xc3d3ac00 com.htc.bg           1157            10009           10009  0x2cc04000 2012-08-05 02:22:16 UTC+0000
0xc8ecc000 Binder Thread #      888             10092           10092  0x28d7c000 2012-08-05 02:21:44 UTC+0000

linux_dentry_cache

Each opened and/or mapped file is represented by a file structure that contains a pointer to a dentry structure that holds the name of the file as well as other metadata. Enumeration of this cache gathers all opened files across the system. This can  be very useful to locate malware-specific files or processes that are interacting with files they should not. Unallocated entries can also be used to find files that were previously opened and now deleted. Use of this plugin reports all of the files across in the cache in body file format, so you can immediately add the MAC times of recovered files to your timeline.

$ python vol.py --profile=LinuxEvo4GARM -f Evo4GRodeo.lime linux_dentry_cache
0|data/com.htc.socialnetwork.flickr/databases/flickr.db|1140|0|10009|10009|7168|3550353848|3550353856|0|3550353864
0|data/com.htc.socialnetwork.provider|724|0|10009|10009|2048|3550256496|3550256504|0|3550256512
0|data/com.htc.socialnetwork.provider/databases|777|0|10009|10009|2048|3550941704|3550941712|0|3550941720
0|data/com.htc.socialnetwork.provider/databases/SocialNetwork.db|797|0|10009|10009|30720|3550846648|3550846656|0|3550846664
0|data/com.htc.sync.provider.weather|607|0|10009|10009|2048|3550345296|3550345304|0|3550345312
0|data/com.htc.sync.provider.weather/databases|1173|0|10009|10009|2048|3550522768|3550522776|0|3550522784
0|data/com.htc.sync.provider.weather/databases/weathersync.db|928|0|10009|10009|5120|3550520144|3550520152|0|3550520160
0|data/com.htc.wdm|627|0|1000|1000|2048|3550397888|3550397896|0|3550397904
0|data/com.htc.wdm/databases|871|0|1000|1000|2048|3550958056|3550958064|0|3550958072
0|data/com.htc.wdm/databases/wdm.db|881|0|1000|1000|7168|3550960024|3550960032|0|3550960040
0|data/com.l33t.seccncviewer|1270|0|10093|10093|2048|3550269112|3550269120|0|3550269128
0|data/com.l33t.seccncviewer/lib|1037|0|1000|1000|2048|3549299864|3549299872|0|3549299880
0|data/com.l33t.seccncviewer/lib/libl33tcrypto.so|1054|0|1000|1000|37840|3549301176|3549301184|0|3549301192
0|data/com.rosedata.android.rss|614|0|10071|10071|2048|3550213600|3550213608|0|3550213616
0|data/com.rosedata.android.rss/databases|926|0|10071|10071|2048|3548935288|3548935296|0|3548935304
0|data/com.rosedata.android.rss/databases/Data|1191|0|10071|10071|31744|3548934304|3548934312|0|3548934320

linux_mount_cache

Each mount point in the system is tracked through the kmem_cache. By recovering allocated entries, we can see all mountpoints on the system:

$ python vol.py --profile=LinuxEvo4GARM -f Evo4GRodeo.lime linux_mount_cache
none                      /acct                               cgroup       rw,relatime
/sys/kernel/debug         /sys/kernel/debug                   debugfs      rw,relatime
sysfs                     /sys                                sysfs        rw,relatime
proc                      /proc                               proc         rw,relatime
devpts                    /dev/pts                            devpts       rw,relatime
tmpfs                     /dev                                tmpfs        rw,relatime
/dev/block/vold/179:1     /mnt/sdcard                         vfat         rw,relatime,nosuid,nodev,noexec
/dev/block/vold/179:1     /mnt/secure/asec/.android_secure    vfat         rw,relatime,nosuid,nodev,noexec
[snip]

If we pass the -u option, we can find mount points that previously existed. This can be useful when external media was present on the system, but later removed.

Building New kmem_cache Plugins

Volatility currently only supports a few caches that we have found useful during investigations. If you find another cache that seems interesting (note: linux_slabinfo can be used to list them all), then you can leverage our simple API to gather all the structures of the same type.

If you read the dentry_cache plugin, you see that one simple line of code, that takes the name of the cache and type of each cache member as parameters, is all that is needed to enumerate from kmem_cache. The API determines if the system is using SLAB or SLUB, enumerates the proper entries, and instantiates them as the chosen type.

Conclusion

This post has highlighted a number of plugins that can help discover the current and historical context of a number of user and application (malware) related activities on an Android system. Incorporating kmem_cache analysis into this process is a huge advantage for investigators and can recover details not accessible by any other memory forensics tool.

MoVP II - 3.3 - Automated Linux/Android Bash History Scanning

Recovering bash command history from Linux and Android memory dumps just got a lot easier.

In previous releases of Volatility, extracting commands and the associated timestamps was possible, but with one caveat - you needed to know the offset into the /bin/bash binary where a pointer to the start of the command history list existed. As described in the linux_bash documentation, you could get the offset in a few ways - by using gdb on the live system or analyzing /bin/bash with IDA. Unfortunately, one of those ways requires live access to the victim machine (which you may not have) and the other requires reverse engineering experience.

Starting with Volatility 2.3, the linux_bash command doesn't need any help. It scans through the heap of bash processes and automatically finds command histories. This is a major enhancement to the plugin and brings it a lot closer to the practical world of memory forensics. You get a whole lot more by doing less.

Glancing Back 

Here's a look at the linux_bash output prior to Volatility 2.3. As you can see, the -H/--history-list argument is required.

$ python vol.py -f avgcoder.mem --profile=LinuxCentOS63x64 linux_bash --history-list= 0x6ed4a0 
Volatile Systems Volatility Framework 2.2_rc1
Command Time         Command
-------------------- -------
#1376083693          dmesg | head -50
#1376085088          df
#1376085110          dmesg | tail -50
#1376085118          sudo mount /dev/sda1 /mnt
#1376085122          cd /mnt

If the history list offset varies per system, and you need to know it in advance before analyzing a memory dump, what options do you have? As mentioned earlier, one way is to load /bin/bash in gdb and disassemble the history_list function.

mhl@ubuntu:~$ gdb /bin/bash 
GNU gdb (Ubuntu/Linaro 7.4-2012.02-0ubuntu2) 7.4-2012.02
[snip]
Reading symbols from /bin/bash...(no debugging symbols found)...done.

(gdb) disassemble history_list
Dump of assembler code for function history_list:
   0x00000000004a5030 <+0>:  mov    0x248469(%rip),%rax  # 0x6ed4a0
   0x00000000004a5037 <+7>:  retq   
End of assembler dump.

The number you see in the comment (0x6ed4a0) is the offset you pass to the plugin as the --history-list value. As previously stated, however, running gdb on a live system or reversing the binary in IDA is not always the optimal solution, especially for forensic investigators who don't have debugging and reverse engineering skills.

Moving Forward

Yesterday, we discussed the new Linux volshell command, which you can use when actively developing a new plugin or an extension/improvement to an existing plugin. For example, before we can scan for history commands we have to know what a history structure looks like. You can find out like this:

$ python vol.py -f avgcoder.mem --profile=LinuxCentOS63x64 linux_volshell
Volatile Systems Volatility Framework 2.3_alpha
Current context: process init, pid=1 DTB=0x366ec000
Welcome to volshell! Current memory image is:
To get help, type 'hh()'
>>> dt("_hist_entry")
'_hist_entry' (24 bytes)
0x0   : line              ['pointer', ['String', {'length': 1024}]]
0x8   : timestamp         ['pointer', ['String', {'length': 1024}]]
0x10  : data              ['pointer', ['void']]

What you see is a 24-byte structure with a member at offset 0 named line. This is a pointer to a string which is the actual command entered. Also there is a timestamp at offset 8, which is a rather unique - its a string of digits (representing epoch seconds) preceded by a pound/hash (#) character. If we knew, for example, that all commands entered started with a common value such as "sudo", we could easily scan memory for all instances of "sudo" and then print them. But alas, that's theoretical at best. We can, however, rely on the fact that all timestamps start with a # character and are followed by at least 10 digits (unless you're dealing with a memory dump from before 2001, which is highly unlikely).

So the plan of action becomes:

1. Scan the heap of all running /bin/bash instances, or all processes period if --scan-all is supplied. The ---scan-all allows you to ignore the process name, in case an attacker copied a /bin/bash shell to /tmp/a and then entered commands. Furthermore, since we're only scanning the heap of the process, its much quicker than a whole process address space scan.
2. Look for # characters in heap segments. With the address in process memory for each # character, do a second scan for pointers to that address elsewhere on the heap. The goal is to find the timestamp member of the _hist_entry structure. We're essentially linking up data with pointers to the data.
3. With each potential timestamp, we subtract 8 bytes (since it exists at offset 8 of the structure). That should give us the base address of the _hist_entry. Now we can associate any other members of _hist_entry (in particular the line member) with the timestamp.
4. Once the scan is finished, collect all _hist_entry structures and place them in chronological order by timestamp. Then report the results.

If you want to explore the steps from a code perspective, check out the source file here.

Getting More by Doing Less

Here's an example of the described algorithm in action. Notice we don't have to supply any parameters to the linux_bash plugin anymore:

$ python vol.py -f avgcoder.mem --profile=LinuxCentOS63x64 linux_bash
Volatile Systems Volatility Framework 2.3_alpha
Pid      Name         Command Time                   Command
-------- ------------ ------------------------------ -------
    2738 bash         2013-08-09 21:28:13 UTC+0000   dmesg | head -50
    2738 bash         2013-08-09 21:51:28 UTC+0000   df
    2738 bash         2013-08-09 21:51:50 UTC+0000   dmesg | tail -50
    2738 bash         2013-08-09 21:51:58 UTC+0000   sudo mount /dev/sda1 /mnt
[snip]

Important Reminders 

There are a few things you should note about the linux_bash plugins which aren't immediately obvious:

• The plugin works flawlessly on Android memory dumps also
• If you supply the -P/--printunalloc parameter, the plugin will print potentially unallocated entries (can be verbose)
• Although the default mode is to use the brute force scanning approach, if you know the --history-list value, you can still supply it. The advantage with using the list head pointer is you see commands in exactly the same order they were entered by an attacker. The brute force scanning approach is a little different. Assuming multiple commands were entered within the same second, there's no way to determine which of those few commands were first.

Conclusion

Recovering commands and timestamps from memory dumps is one of the most powerful investigation techniques. What more could you ask for than the ability to figure out exactly what an attacker did on the machine? In the new Volatility 2.3 release, this task is even easier - it doesn't require any debugging or reversing and it now converts the epoch seconds to a human-readable timestamps (you can also set the timezone with the --tz option if you're generating timelines from artifacts in memory)