Memory Forensics R&D Illustrated: Detecting Hidden Windows Services

As mentioned in a recent blog post, our team is once again offering in-person training, and we have substantially updated our course for this occasion. Over the next several weeks, we will be publishing a series of blog posts, offering a sneak peek at the types of analysis incorporated into the updated Malware & Memory Forensics training course.

Introduction

To begin the series, this post discusses a new detection technique for hidden services on Windows 7 through 11. Since not all readers will be familiar with hidden services and the danger they pose on live systems, we will start with some brief background. We will then walk through how services.exe stores service information, and how we can recover it in an orderly manner. This will lead to how we developed two new Volatility 3 plugins to help automate detection of hidden services.

The power of these plugins will be showcased against the powerful GhostEmperor APT rootkit that was discovered in the wild by researchers at Kaspersky. GhostEmperor employs a kernel mode rootkit and a userland DLL to maintain persistence and control the victim system. This DLL operates as a service that is hidden from live analysis and DFIR triage tools, and it interacts directly with the rootkit driver in kernel memory. As will be demonstrated, by automatically detecting the hidden service of GhostEmperor through memory analysis, we can quickly find the rest of its components, including those hidden on the live system.

Services Background

Services are a powerful feature of Windows that allow malware to run in one of three possible forms. The first allows malware to register a DLL that will be loaded into a shared svchost.exe process, hiding it amongst other DLLs loaded inside the same process, as well as the many svchost.exe instances that run on a normal system. The second form allows malware to run as its own process. The third, and most dangerous, form is when malware creates a service to load a kernel driver (rootkit).

When services are created and started using standard methods, a few artifacts are left behind for investigators to find. The first is a set of registry keys and values under CurrentControlSet\Services\<service name>. The second is the service’s entry within a linked list maintained by services.exe. This list is enumerated when system APIs, such as EnumServiceStatus{A,W,Ex}, and tools, such as sc.exe query, are used to enumerate services on the running system.

Given the power of services, malware often abuses the ability to create or hijack services for its own purposes. This leads to the inspection of services on a running system by endpoint detection and response solutions (EDRs) and threat hunting teams to look for any suspicious signs. To avoid detection while keeping a service active, malware has historically targeted both sources of artifacts—the registry keys and the services.exe list—with registry keys being targeted in two ways: deleting or hiding them.

In the first approach, malware will delete its registry keys while running, and then rewrite them before system shutdown or reboot. This has a major disadvantage though, as sudden system crashes or service stops prevent the malware from re-registering its persistence.

This deficiency led to the current approach malware takes, including by GhostEmperor, which is to simply hide its keys from the running system. The following screenshot shows Kaspersky’s report on the malware's approach:

As discussed in Kapersky's report, the CmRegisterCallback usage effectively allows the malware to hide its service’s keys from tools on the live system. Detecting this malicious callback is possible with Volatility’s callbacks plugin though, and there are also EDRs capable of enumerating callbacks from within kernel memory. To avoid these EDRs, some rootkits found during recent APT campaigns have implemented a completely new method of registry key hiding, known as GetCellRoutine hijacking, that we will cover in an upcoming post along with another new Volatility 3 plugin.

Beyond the registry, malware also wants to hide its malicious service from tools on the live system that query services.exe to enumerate running services. To accomplish this, malware will inject code into the services.exe process, and then unlink the malicious service of interest. This will effectively hide the service from live DFIR triage tools and built-in Windows commands. It's the detection of these unlinked services using new memory forensics capabilities that we cover further in this blog post.

Note: Chapter 12 in The Art of Memory Forensics is devoted to discussion of Windows services, ways malware abuses them, and several historical methods of detection. If you would like a complete treatment of the subject after reading this blog post, then we suggest reading this chapter.

Detecting Unlinked Services

As mentioned, a wide variety of malware samples will unlink their malicious services for anti-forensics purposes. The following screenshot from the Kaspersky report on GhostEmperor describes this for the malware sample:

In our analyzed memory sample, the name of the hidden service is “msdecode”, which is one of the possibilities listed in the report.

The Art of Memory Forensics details one method for detecting unlinked services with Volatility. This method relies on scanning physical memory for services records, and then drawing a dot graph of how each service is linked to other services. This linkage is based on the previous and next pointers of the doubly linked list. During normal operations, each service record should have one service’s forward pointer referencing it, and one service’s backwards pointer referencing it. In the case of an unlinked service, the hidden service will have no services that reference it. The following image from The Art of Memory Forensics shows how this detection logic is applied to an unlinked wscsvc service:

As can be seen, all services other than wscsvc have previous and next pointers (green and red arrows) pointing to them from other services. This is a direct visual indication that the wscsvc service is unlinked.

Unfortunately, this detection method is no longer viable for two main reasons. First, the Connected Devices Platform subsystem creates a wide variety of temporary services during system operations. This means that smear (changes to memory during acquisition) will often cause these services to appear as unlinked when using the method that detected wscsvc. The second reason is that scanning physical memory will find copies of service records relating to services that have since changed state (restarting, start<->stop). The ability to recover these historical records is a powerful aspect of memory forensics, but unfortunately clutters the results for this particular use case, as the historical records are no longer tracked by services.exe.

To illustrate these issues, we created a Volatility 3 plugin, svclinks, that reports a text-based version of the visual graph. We ran svclinks against our memory sample infected with GhostEmperor. This plugin reports only services that it thinks are unlinked, and the results are shown below:

 

As can be seen, while our target service, Msdecode, is in the reported list of unlinked services, so are several other services, all of which are false positives. Given the inability to rely on our old method, we needed to develop a new one.

Replicating services.exe's Enumeration of Services

Knowing that the live system uses the list inside of services.exe to report services, along with the fact that malware takes great effort to hide from this list, we chose to use it as a source for detecting unlinked services. This detection relies on cross-comparing the services found through scanning, which Volatility 3 already supports, versus the list walking performed in our new plugin. This is similar to using pslist and psscan (or psxview) to detect unlinked processes within the kernel.

Over the years, Microsoft has made substantial changes to the methods services.exe uses to track services, but, luckily for us, we only have to be concerned with changes in the data structure layout and the name of global variables. The following screenshots show how the services.exe database is declared across Windows versions:

Windows 10+:

Windows 7:

For data structure layouts, Volatility 3 already contained definitions for most of the types needed. All we had to add was the CServiceDatabase type and the offset to the first service record. Luckily, this was at a constant offset for all versions tested.

Enumerating the Service List in Volatility 3

To automate detection of unlinked services, two Volatility 3 plugins were developed. The first, svclist, locates and then enumerates the list of services maintained by services.exe. The second, svcdiff, compares the services obtained from scanning with the services obtained from walking the list. We will now discuss how these plugins are implemented with several screenshots of code. If you would like to read a nearly line-by-line breakdown of implementing a Volatility 3 plugin that performs similar actions, please see our post on detecting the skeleton key attack of Mimikatz.

Finding the Service Database

Obtaining the address of the service database inside of a particular memory sample is easy, since Volatility 3 supports automatic symbol resolution through PDB files. This tells us our plugin precisely where to find the database within the memory sample. 

To start this recovery, Volatility’s process enumeration API is used to find the _EPROCESS object for services.exe. Next, the following code is used to automatically download and parse the PDB file for the executable, and then search the variations of the service database's symbol name: 

The end result of this code is that the svclist plugin will automatically know where to find the services database, which then tells the plugin how to find the beginning of the list.

Enumerating Services from the List

Once the list is found, Volatility’s traverse API for services can be used to walk the list; svclist then has little work left to do, as the existing svcscan plugin already contains a get_record_tuple API that gathers the information about a service (name, path, PID, etc.) to report to the analyst:

Using this, the output from our plugin then looks the same as when svcscan runs.

Detecting Unlinked Services in Volatility 3

Our detection of unlinked services in the new svcdiff plugin is based on comparing the set of services generated by the svcscan plugin and our new svclist plugin. In particular, each of these plugins is programmatically run, and then the names of any service found through scanning—but not through list walking—is reported. 

By keying in on the name, we work around the issues found when linked-list pointers are used. This fix works because even if a service is stopped and restarted (which creates multiple data structures in memory), the name will be the same between runs. The name-based approach also removes the chance of false positives from the temporary services generated on Windows 10+. 

The following screenshot shows the core of this plugin and how easy it is to leverage existing APIs in Volatility 3 to produce powerful new capabilities:

In this code, the services_scan API is first used to gather the names of services based on scanning. As shown in the get_tuple_record screenshot, the name of the service is the sixth entry. Next, service_list from our new plugin is used to gather services like services.exe does on the live system. Finally, a simple set difference is used to determine names found from scanning that were not found in the list. These are then reported to the output rendering API.

Detecting Ghost Emperor

With our new plugin available, automatically detecting GhostEmperor’s unlinked service is as simple as one Volatility invocation:

In this invocation, svcdiff reports only one service, Msdecode, which we know is the one hidden by GhostEmperor. 

Exploring the Hidden Service

With this information in hand, we can investigate further by determining other components and actions of this service. To start, we can look at the list of DLLs inside of the process (reported as PID 4756 by svcdiff):

In this abbreviated output, we see DLLs inside of system32, as well as the msdecode.dll of the malware. By applying the --dump option to dlllist, the plugin will extract all of a processes DLLs to disk. Looking at the strings output of this extracted file shows the name of several APIs used for gathering sensitive system information and anti-forensics purposes:

The extracted DLL file can then be loaded into your reverse-engineeering (RE) tool of choice, scanned with YARA signatures, and other static analysis techniques.

Kernel Mode Components

After examining loaded DLLs, we can then examine the handles of the process to determine which system resources it is accessing. Since we know a kernel rootkit is involved, we search for any references to Device files within the handles output. Device files are created by drivers to allow userland processes to "speak" directly with the driver. This is the most commonly used interface by rootkits to allow the controlling process to specify filenames, registry keys, and processes to hide, as well as actions like enabling privilege escalation. 

Looking at the Device files being accessed by the Msdecode service process shows us an interesting entry to a device named dump_audio_codec0;  the other entries are present on all Windows systems:

Attempting to investigate dump_audio_code0 further instantly shows that it is malicious in nature. The following screenshot shows the output of the modules and driverscan plugins of Volatility 3 while searching for the driver:

As seen, the driver does not appear in modules output, which only happens when anti-forensics techniques are used. This is verified by the output of driverscan that shows the module's base address and size have both been set to "0". This is a common anti-forensics technique to hide a module on a live system and prevent its direct extraction from memory.

This technique is, in fact, so common that Volatility has a special-purpose plugin called drivermodule to detect discrepancies between module and driver data structures:

In this output, two modules are reported. The first, RAW, will trigger in all nearly all memory samples but, as seen in the first column, it is reported as a known exception. However, dump_audio_code0 is not, and as we verified multiple times already, this driver is definitely worthy of deep investigation. 

Between the usage of our new svclist plugin, along with the drivermodule plugin, we have directly detected both the userland and kernel components of the rootkit, and we have done so without any existing IOCs specific to GhostEmperor. As demonstrated, memory forensics continues to be a necessary component to accurately detect modern rootkits and malware.

Conclusion

In this blog post we have demonstrated a new memory-forensics technique to detect hidden services in a smear-resistant manner. Given the number of malware samples that hide services from the live system, as well as the danger posed by these services, it is essential that malware can be detected in a reliable manner. 

If you have any questions about this blog post, please let us know! You can email us, or find us on Mastodon and Twitter. We also have our own Slack Server.  If you enjoyed this content, then be sure to check out the announcement of our updated training class. During the course, students are taught how to detect modern malware, such as the sample discussed in this blog post, as well as gain significant hands-on experience through many real-world labs. 

Finally, we will be presenting new research on triaging modern Windows rootkits at BSidesCharm in Baltimore in a few weeks, so please come say hello if you will be there!

The 2022 Volatility Plugin Contest results are in!

Results from the 10th Annual Volatility Plugin Contest are in! There were 8 submissions this year, including submissions from 2 contestants from previous years who have continued to build on their previous work. Submissions included updates to graphical interfaces, plugins to detect Linux rootkits, plugins to extract threat actor activity despite anti-forensics techniques, and a new analytical capability for leveraging handle information to augment investigations. As usual, we would like to thank the participants for their hard work on their submissions and contributions to Volatility community!

Independent open source projects and communities only remain viable because of contributors who are willing to sacrifice their time and resources. Please show your appreciation for the contestants’ contributions by following them on Twitter/GitHub/LinkedIn, providing feedback on their ideas, and helping to improve their code with testing, documentation, or contributing patches. 

The Return of In-Person Volatility Malware and Memory Forensics Training!

We are excited to announce that we are resuming our in-person Malware and Memory Forensics with Volatility training course! From Fall 2012 until Spring 2020, this course ran multiple times a year and taught hundreds of students how to apply memory forensics to their incident response and malware analysis workflows. Since Spring 2020, the course has been delivered in a virtual, self-paced format. With the return of our in-person training, students now have the option of attending in-person delivery or the virtual version. 

The first in-person course of 2023 will take place May 8–12, 2023, in Reston, VA. We are also exploring potential venues for a Fall 2023 course in Europe.  Detailed course information, including registration procedure, format, and deliverables can be found on the course page. 

This course is taught by members of the Volatility Team and teaches students how to detect and respond to modern, advanced threats through comprehensive analysis of volatile memory and key file system artifacts. All material for this course is based on the instructors’ experience detecting and responding to some of the most sophisticated threat groups in the world (1,2,3,4,5). The knowledge and insight gained during these investigations has been transitioned into training content and labs.

Course Updates for 2023

The rapid advancement of malware and attacker toolkits, along with major changes by operating system vendors, means that incident response handlers must constantly update their skill sets and knowledge. The 2023 version of our course will include many of these changes in the form of updated lectures and new labs. These updates will be delivered in person, as well as incorporated into the virtual course.

 These updated topics include the following:

• Significant artifact changes in later versions of Windows 10 and Windows 11
• New Windows rootkit techniques that bypass driver signing enforcement and PatchGuard monitoring
• Modern credential dumping attacks
• Modern code injection techniques meant to bypass EDR and AV monitoring
• EBPF-based Linux rootkits (see our research from Black Hat 2021)
• A deep dive into in-the-wild keylogging techniques (see our research from Black Hat 2022)
• Memory analysis of Apple Silicon devices 

During the course, we will also be showing off many new features and plugins of Volatility 3 so students can see the latest updates to the framework.

If you would like to receive updates on the course and general Volatility developments, please join our Slack server; follow us on Twitter and Mastodon; and join our mailing list.

Our team is really looking forward delivering in-person trainings again, and we hope to see many of you in Reston in May!

-- The Volatility Team

The 10th Annual Volatility Plugin Contest!

This year not only marks 15 years since the first public release of Volatility, we are also excited to announce the 10th annual Volatility Plugin Contest is now open! Submissions will be accepted until December 31, 2022.

Volatility Plugin Contest

The 2022 Volatility Plugin Contest is your chance to get industry-wide visibility for your work, put groundbreaking capabilities into the hands of investigators, and contribute back to the open source forensics community. Since its inception, the contest has encouraged research and development in the field of memory analysis. Over the last 10 years, participant contributions from all around the world have helped to lay the foundation for the next generation of memory forensics.  

Winners this year will receive over 6000 USD in cash prizes! 

Get visibility for your work, contribute to an important open-source project, and get a chance to win a cash prize! For more information, see the full details here: 2022 Volatility Plugin Contest  

If you are looking for inspiration, check out the 2021 Volatility Plugin Contest Results.

Acknowledgements

We would like to thank Volexity and our other sustaining donors for their continued support.

The 2021 Volatility Plugin Contest results are in!

Results from the 9th Annual Volatility Plugin Contest are in! And this year, there were 7 submissions from 7 different countries! Submissions included a new web interface, a new address layer, 6 updates to existing plugins, and 15 new Volatility 3 plugins. Once again, we would like to thank the participants for their hard work on their submissions and contributions to Volatility. As in previous years, it was great to see contestants who had submitted in prior contests and submissions from across the global Volatility community.  

It's now 15 years since the first public release of Volatility! It has been exciting to see researchers in the memory forensics field continue to innovate. Later this year, we are planning something special to commemorate all the contributors who have joined us on this journey. 

Independent open source projects and communities only remain viable because of contributors who are willing to sacrifice their time and resources. Please show your appreciation for the contestants’ contributions by following them on Twitter/GitHub/LinkedIn, providing feedback on their ideas, and helping to improve their code with testing, documentation, or contributing patches. 

---

We would like to thank Volexity for being a sustaining sponsor of the Volatility Foundation and, in particular, for contributing to this year’s contest. We would also like to thank the core Volatility developers and the previous winners of the contest who helped review and deliberate the submissions.

---

Placements and Prizes for the 2021 Volatility Plugin Contest:

1st place and $3000 USD cash or One Free Seat at Malware and Memory Forensics Training by the Volatility Team goes to:

 Amir Sheffer & Ofek Shaked: Linux Namespaces Support and Docker Plugin

2nd place and $2000 USD cash goes to:

Kevin Breen: Symbol Generator & Public ISF Server, Cobalt Strike Plugin, Rich Header Plugin, and LastPass Credential Recovery Plugin

3rd place and $1000 USD cash goes to:

Frank Block: PTE Analysis Plugins 

---

Below is a detailed summary of all submissions, ordered alphabetically by first name. If you have feedback for the participants, we're sure they'd love to hear your thoughts! As previously mentioned, these developers deserve praise for their amazing work. We look forward to seeing future work by these authors!

---

Amir Sheffer & Ofek Shaked: Linux Namespaces Support and Docker Plugin

Container technology is widely used in production Linux settings, and the highly focused analysis of per-container information can help to greatly focus investigations and identify key related artifacts. This submission provides a suite of Volatility 3 plugins for memory forensics of Docker containers.  This included expanding core capabilities in Volatility 3 by making them aware of Linux namespaces and augmenting the number of supported kernel versions.  For example, an analyst can quickly detect the presence of a container, collect information about the container and its capabilities, display information about its mount points, and provide detailed network configuration data.

Related References:

https://github.com/amir9339/volatility-docker
https://github.com/oshaked1
https://github.com/amir9339

Felix Guyard: VolWeb

This submission provides an exciting new web interface to Volatility 3 built using the Django framework. The objectives for the project were to improve investigator efficiency, centralize collaborative analysis, and make memory analysis more "human" friendly. VolWeb also allows investigators to manage memory analysis investigations and search for string-based indicators of compromise. It provides a promising new platform for future work and integrations.

Related References:

https://twitter.com/k1nd0ne
https://k1nd0ne.github.io/index.html
https://github.com/k1nd0ne/VolWeb

Frank Block: PTE Analysis Plugins

The author contributes several Windows plugins for Volatility 3 that extend the code injection detection capabilities of malfind, while also adding low-level PTE enumeration functionality similar to !pte in Windbg. Building on the author's novel research, he has identified potential false negatives in malfind that can occur when the Windows VAD data does not match the underlying page protections, encoded in the PTEs. The author has written a comprehensive library for enumerating and inspecting Windows PTEs and a set of example capabilities on top. All-in-all, it's a great contribution to the Volatility 3 ecosystem! It is also extremely well documented with research publications, blog posts, and a great talk on the subject.

Related References:

https://insinuator.net/2021/12/release-of-pte-analysis-plugins-for-volatility-3
https://github.com/f-block/volatility-plugins

Gerhart: Hyper-V Volatility Introspection Layer

Virtual memory introspection is a technique for monitoring the runtime state of a virtual machine. This submission adds the ability to analyze live Windows Hyper-V virtual machines without acquiring a full memory dump. The new Volatility 3 layer for Hyper-V adds an interface reminiscent of LiveCloudKd or Sysinternals LiveKd, but with the power of Volatility 3's extensive plugins.

Related References:

https://twitter.com/gerhart_x
https://hvinternals.blogspot.com
https://github.com/gerhart01

Kevin Breen:  Symbol Generator & Public ISF Server, Cobalt Strike Plugin, Rich Header Plugin, and LastPass Credential Recovery Plugin

This submission includes a number of components that can help analysts with modern investigations. The submission includes the following 3 plugins that bring new or updated functionality to Volatility 3:

Password Managers: LastPass is a widely used password manager and thus provides a highly valuable forensics target. This submission ports a popular Volatility 2 plugin for extracting LastPass credentials that were stored in memory at the time of acquisition.

 

Rich Header Plugin: A common technique during investigations is to try and identify masquerading processes running on suspected systems. This plugin extracts the Rich header from PE files compiled with Visual Studio which can help identify masquerading processes or aid in wider threat hunting or incident response investigations.

 

Cobalt Strike Plugin: Cobalt Strike is one of the most popular frameworks used by modern attackers and is frequently encountered during investigations. This plugin scans processes for signs of a Cobalt Strike configuration block and provides the ability to extract relevant configuration information.  

In addition to the aforementioned plugins, the submission also provides tools to reduce the hurdles some people experience when analyzing Linux memory samples: 

A Linux symbol server with currently over 1000 Volatility 3 ISF symbol files: The server can be provided to Volatility 3 as a remote symbol server and, if a sample has a matching banner, it can automatically use the associated symbols for analysis. Individual symbol files can also be searched for either by banner or kernel name.

If a symbol file does not exist on the server, a separate Symbol Maker tool can be used to create a symbol file.  By specifying a supported distribution and an optional kernel, the tool will download the necessary files and use dwarf2json to create a symbol file that can be used with Volatility 3. The tool currently supports Ubuntu (Main, AWS, Azure and GCP Variants) and Debian (Main, AWS).

Related References:

https://twitter.com/kevthehermit
https://github.com/kevthehermit/volatility_plugins/blob/main/vol3/passwordmanagers/passwordmanagers.py
https://github.com/Immersive-Labs-Sec/volatility_plugins/tree/main/richheader
https://github.com/Immersive-Labs-Sec/volatility_plugins/tree/main/cobaltstrike
https://isf-server.techanarchy.net
https://github.com/kevthehermit/volatility_symbols

Leonardo Dias da Silva: MultiYara

Many investigators often use YARA to help detect suspicious activity in memory samples.  This submission was intended to help investigators optimize and automate their investigation workflows by making it easier to pull down updated rules from remote locations and leverage multiple YARA rules. 

Related References:

https://www.linkedin.com/in/leonardo-dias-silva

MoonGyu Lee, JeongToon Kang, HyeonDeok Jeongm JunSung Park, Mintaek Lim (BoB Tracer of Coin): CryptoScan

Cryptocurrency is becoming increasingly important during digital investigations ,and there aren’t many forensics tools focused on extracting cryptocurrency artifacts.  In Korea, malicious actors are leveraging hardware wallets to bypass government-required authentication and gain anonymity. This submission is a plugin to detect and extract cryptocurrency transaction records and artifacts related to hardware wallet usage.  In particular, their research explores the Ledger Nano and Trezor One hardware wallets.  By interfacing with several cryptocurrency websites, the plugin can also be used to support investigations related to tracking cryptocurrency transactions.

Related References:

https://github.com/BoB10th-BTC/CryptoScan/blob/master/cryptoscan.py

---

Here are some additional resources for previous contests and community-driven plugins:

---

Volatility Foundation Contest Home Page:  http://www.volatilityfoundation.org/contest

Volatility 2020 Plugin Contest Results: https://www.volatilityfoundation.org/2020
Volatility 2019 Plugin Contest Results: https://www.volatilityfoundation.org/2019
Volatility 2018 Plugin Contest Results: https://www.volatilityfoundation.org/2018
Volatility 2017 Plugin Contest Results: http://www.volatilityfoundation.org/2017
Volatility 2016 Plugin Contest Results: http://www.volatilityfoundation.org/2016 
Volatility 2015 Plugin Contest Results: http://www.volatilityfoundation.org/2015
Volatility 2014 Plugin Contest Results: http://www.volatilityfoundation.org/2014-cjpn
Volatility 2013 Plugin Contest Results: http://www.volatilityfoundation.org/2013-c19yz

Volatility Community GitHub Repository: https://github.com/volatilityfoundation/community3 

Malware and Memory Forensics Training in 2022!

Over the last few months, we have received many questions about when our Malware and Memory Forensics training would return to in-person learning. Given that a new year is nearly here, and the rate of inquiries has continued to increase, we wanted to document our plans going forward in a publicly available blog post, as opposed to only fielding questions individually.

Virtual Course Remains Available

We would like to start by saying that our course is currently available in virtual format to students across the globe. We announced this availability earlier this year, and since then have had many students successfully complete the course. 

Our online course is self-paced and includes the full material (pre-recorded lectures, copies of the slides, labs, lab guide, etc.) given and presented in the normal 5-day course. Students also have direct access to the instructors through a private Slack channel on the Volatility Foundation’s Slack server. The self-paced format of the course has received very positive feedback, particularly as students have benefited from being able to message and screen share with instructors for help and have the ability to re-listen to lectures to reinforce learning:

"The class is one of the most technical courses I've taken. The Labs provided real world examples and make you think of various aspects of incident response (Network, Disk, File, Memory Forensics). I would recommend this class for incident responders, defenders, and those looking to get a better understanding of memory forensics. I've taken other Level 600/Advanced classes from other vendors and this is right up there with the content and quality." ~ Carlos M.

"I don't say it lightly: this is the best course I've ever taken. The instructors are incredibly fast (and helpful) to answer any questions about the subject, be it directly related to a module or on a real life scenario. It's also not about running plugins blindly. The course has taught me priceless information on Windows Internals, why certain suspicious activities in memory are suspicious in the first place and, best of all, showed me a structured analysis framework I can apply to all my future investigations." ~ Alexandre S.

"I have waited five years to finally attend, participate in and complete this excellent course. I wasn't disappointed. Very relevant, very in depth and has left with me with the skills and enthusiasm to seek out and explore more." ~ Matthew K.

Plans for Public, In-Person Training

As for in-person, public trainings, we are currently evaluating offering these in the Summer or Fall of 2022. We decided against hosting the training in Spring 2022 given the ongoing uncertainty surrounding the pandemic, including restrictions on travel from governments and companies, as well as constantly changing local regulations—particularly in cities we have historically held training events. Our aim is to return to in-person training as soon as the uncertainty clears and as the pandemic allows. 

Private, In-Person Training Beginning in 2022

We will be available for private trainings starting in 2022 within the United States. At this time, this seems the most reasonable option given that a single organization would control the pandemic-related parameters of the training. If your company is interested in a private, in-person training, please contact us. Private trainings can be customized, including modifying the number of days or focusing the course on the areas most critical to the particular organization’s success and goals. 

Keep in Touch!

We hope this update addresses any questions you may have, but if not then please let us know. We also hope to be back presenting at conferences in-person next year along with being able to host our training at public events. 

-- The Volatility Team

Memory Forensics R&D Illustrated: Detecting Mimikatz's Skeleton Key Attack

In this blog post, we are going to walk you through the research and development process that leads to new and powerful memory analysis capabilities. We are often asked about what this workflow looks like, and how the abuse of an API by malware or a new code injection technique can be successfully uncovered by a Volatility plugin. To showcase this process, we are going to analyze the Skeleton Key feature of Mimikatz, and then develop a brand-new Volatility 3 plugin that can successfully detect this backdoor technique across memory samples. While Volexity Volcano customers have had this capability, we wanted to contribute this back to the Volatility community, since there was no publicly available plugin. This post will also reveal a number of entirely new features.

To reach this goal, we will first study the relevant Mimikatz source code; then we will reverse engineer the API that Mimikatz uses to locate its victim data structure; and then we will write a plugin that can replicate this search and look for signs of tampering. As you will see shortly, the new Skeleton Key detection plugin is fully documented and shows how to perform a wide range of tasks using the APIs of Volatility 3. 

Our hope with this blog post is to inspire more members of the community to challenge themselves to develop their own new capabilities, and to experience what real-world malware and operating systems investigations entail. If you find this work interesting and decide to develop your own plugin(s), please consider submitting them to our 2021 Volatility Plugin Contest and take a chance at winning several prizes, including cash or a free spot in our popular Malware and Memory Forensics training. 

Skeleton Key Background

The Skeleton Key technique was first detected in the wild by the DFIR team at SecureWorks. Their blog post walks through the steps taken by the malware sample they uncovered. They also worked with Microsoft on a follow-up paper about the attack type and its variations. The implementation in Mimikatz is very similar to the one described in their research. 

The idea behind the Skeleton Key technique is to backdoor the authentication subsystem of Windows Active Directory domain controllers. This is accomplished by injecting code into the running lsass.exe process, and then hooking the routines used when verifying a domain account’s credentials. With the hooks in place, attackers are able to authenticate as any valid user in an AD domain by using a hard-coded password (termed the Skeleton Key by SecureWorks). 

The ability of attackers to log in as any user makes several traditional incident remediation procedures largely ineffective. As an example, it is very common during incidents to temporarily disable accounts that attackers are/were using or to at least force a password reset of these accounts. When a Skeleton Key is active, these procedures are not helpful, since any account can be used. This problem is also compounded by the fact that all user accounts in a domain could potentially be abused by attackers, and a significant amount of log review (assuming logs are available) is necessary to trace the abuse of accounts and any associated lateral movement. This ability to authenticate as any user to any system is incredibly powerful and significantly expands the scope of DFIR engagements. 

Analyzing the Skeleton Key Capability of Mimikatz 

Activating the Skeleton Key attack of Mimikatz requires using its misc::skeleton command after running the usual privilege::debug command. There are many great blog posts that document this process by showing the related Mimikatz output and other related information, such as here, here, and here. Cycraft also documented malware from the Chimera APT group that used a significant amount of code from misc::skeleton to implement its own Skeleton Key attack. The end result of this command is a Skeleton Key attack being active on the system; the attacker is able to authenticate with the malware-controlled credentials. 

Running the misc::skeleton command will lead to the kuhl_m_misc_skeleton function being called inside the active Mimikatz instance. This function is responsible for patching the needed code and data inside the lsass.exe process to make the Skeleton Key active. The first steps in this process are shown in the following image:

In this code, Mimikatz first gets the process ID of lsass.exe and stores it in the processId variable. Next, it calls OpenProcess to obtain a handle to the lsass.exe process. This handle gives the ability to read and write the memory of the lsass.exe process from the calling process. Mimikatz then calls kull_m_memory_open, which is an internal Mimikatz function that stores the handle for later use. 

After Mimikatz is able to read and write memory of the lsass.exe process, it then searches for the Kerberos-Newer-Keys string in memory so that it can find the data structure related to AES-based authentication. It then manipulates this structure so that authentication is downgraded to the weaker RC4 without the use of a salt. Note that this is the same approach described in the previously linked Cycraft report. Older reports, such as the one from Microsoft, describe how malware can also achieve the same result by hooking the SamIRetrieveMultiplePrimaryCredentials function and forcing it return an error when the Kerberos-Newer-Keys package is used. The end result of both methods is the same: all authentication attempts to an infected domain controller will use the weaker RC4 algorithm.

After downgrading the domain controller to RC4, Mimikatz will then attempt to locate and patch the data structure that handles RC4-based authentication. The following image shows the beginning of this code:

First, the module information is gathered for the “cryptdll.dll” module loaded within the lsass.exe process. This module is responsible for implementing the different encryption packages, which are also known as systems. Next, the address of where cryptdll.dll is loaded within the Mimikatz process is gathered by calling GetModuleHandle. 

The undocumented CDLocateCSystem function is then called with an argument of KERB_ETYPE_RC4_HMAC_NT and the address to store the resulting lookup (&pCrypt). CDLocateCSystem determines the address of the data structure that handles the given (KERB_ETYPE_RC4_HMAC_NT) authentication system and copies its contents into the passed-in pCrypt address. In this instance, it will be for the system that implements RC4-based authentication, and it will contain the information shown below:

The actual data structure definition for this type is not documented by Microsoft, so the above image is directly from the Mimikatz source code. The arrows point to the members relevant to our plugin, which include the encryption type; the function pointers for the initialization, encryption, decryption, and finish operation handlers; and the pointer to the string name of the system.

After finding the _KERB_ECRYPT instance for RC4 through the use of CDLocateCSystem, Mimikatz then hooks the legitimate initialization (Initialize) and decryption (Decrypt) members of the structure. These hooks point the handlers to Mimikatz’s malicious handlers (kuhl_misc_skeleton_rc4_init and kuhl_misc_skeleton_rc4_init_decrypt) instead of the legitimate ones. The malicious handlers are injected into the address space of lsass.exe through the use of the WriteProcessMemory function. Combined, these malicious handlers are what implement the Skeleton Key attack, as they give Mimikatz control over all future authentication attempts to the infected domain controller.

Devising a Detection Strategy

Now that we understand how Mimikatz implements its attack—forcing a downgrade to RC4 followed by hooking the RC4 initialization and decryption routines—we can devise a strategy to detect the attack in memory.

We could start by attempting to detect the RC4 downgrade, but this has a few limitations. First, the string needed to find this data structure (Kerberos-Newer-Keys) is zeroed out as part of the attack, removing the possibility of a scanning-based approach to finding it. Second, attempting to detect that this string has been zeroed out would lead to many false positives due to paging of data out to disk, as well as the possibility of the page holding the string being smeared. Third, there are other methods to force a downgrade to RC4 without directly altering this string (as discussed in earlier references), meaning several approaches would be needed to completely detect it. Finally, finding proof of the downgrade only gives a clue that a Skeleton Key attack might have been performed, but it does not offer direct evidence. 

On the other hand, by examining the RC4 data structure directly, we can inspect the handlers for the initialization and decryption routines and determine if they were altered at runtime. This not only definitively tells us if a Skeleton Key attack occurred, but it also tells us exactly where the malicious handlers are inside of the infected lsass.exe process. Given that this approach gives direct evidence of the attack, as well as directly points out the malicious code, inspecting these handlers was chosen as the detection method for our plugin.

Reverse Engineering CDLocateCSystem

Before we can locate the handlers to then verify them, we need to be able to find the RC4 data structure in a repeatable and consistent manner. As shown previously, Mimikatz locates the address of the RC4 data structure by calling CDLocateCSystem. This tells us that if we can replicate the algorithm of CDLocateCSystem—or at least build an algorithm that is equal—we can reliably locate the RC4 structure to then verify its handlers. 

Since cryptdll.dll contains the CDLocateCSystem function implementation and is closed source, we will need to reverse engineer the function to determine its algorithm. As you will see next, this function is pretty simple, so do not panic if you have never reverse engineered before; the concept will be straightforward.

The following images show the IDA Pro decompiler and graph view of CDLocateCSystem:

As seen above, the function is pretty small and simple. It begins (the first instruction of CDLocateCSystem in the disassembly view) by copying the current value of cCSystems global variable into the r8d register, which is an alias for the lower 32 bits of 64-bit r8 register. It then tests (CDLocateCSystem+22) if the value is zero and bails with an error (+27) if it is. If the value is anything but zero, then it moves to basic block, starting at offset +9. This basic block begins by decrementing the r8d register (+9). This code pattern of storing a variable, checking if it is zero, and then decrementing the value tells us that this is likely a counter for looping (iterating) through a data structure. Looking ahead, the red line leading from +20 back to +22 in the graph confirms this, as the code at +22 will be evaluated every time the basic block starting at +9 fails to exit. This is exactly how loops look in IDA Pro and other basic block graphing tools. 

Further studying the basic block starting at +9, we see the address of the CSystems global variable copied into the r9 register. Next (+13), the value in r8d is copied into eax, and then rax is shifted left by 7

(+16), which is the same thing as being multiplied by 128 (2 to the 7^th power). This computed value is then stored into r9, and the data r9 points to is compared with the value in ecx (+1D). If this comparison matches, then the function returns. Otherwise, the flow starting at +9 repeats. 

Breaking this down, the code is using the current value of r8d multiplied by 128 (shifted left by 7) as an index into CSystems. This is exactly what iterating through an array looks like. As each array element is stored contiguously in memory, by knowing the size and count, you can successfully locate each element. This understanding of the code now tells us two things: 

1. cCSystems holds the number of elements in CSystems. 
2. The size of each CSystems element is 128 bytes.

For the basic block at +9, the only remaining parts to understand are which values are being compared at +1D and the purpose of that comparison. Since the loop breaks dependent on that comparison, it is likely critical to the function’s overall purpose. Looking at the two values, [r9] and ecx, we know a few things. First, the comparison will be comparing two 32-bit values, as that is the size of ecx, which is the lower 32 bits of rcx. Second, the brackets around r9 mean to treat the value of r9 as an address in memory and then retrieve the value at that address, which is known as dereferencing an address (pointer). From our previous discussion, we know that r9 holds the address of the current CSystems element being inspected. Dereferencing it as [r9] is equivalent to dereferencing [r9+0], which tells us that the first 4 bytes (32 bits) of the referenced structure are being accessed. 

As for ecx, the instruction at +1D is the first time ecx (or any of rcx) is referenced. This means the value must have been set before the function was called. Consulting the Microsoft documentation on function-calling conventions, we see that the rcx register is used on 64-bit systems to store the first parameter sent to a function. Earlier, when we examined how Mimikatz called CDLocateCSystem, we noted that the first argument was the KERB_ETYPE_RC4_HMAC_NT constant, which is defined in NTSecAPI.h of the Windows SDK as 0x17 hex (23 decimal). This means that CDLocateCSystem will be searching for an element of CSystems that has 0x17 (23) as the first integer. 

Looking at the end of the function (+2E -> +33), we see that r9 is stored into the address pointed to by rdx; the previous Microsoft documentation tells us rdx stores the second parameter to a function. We know for CDLocateCSystem that this is the address of where the calling code (Mimikatz) wants Windows to store the address of the requested authentication system (RC4).

Seeing that the address of the CSystems element found in the loop is directly returned to the caller tells us that the data structure returned is also of type _KERB_ECRYPT, since we know that is the type of the second parameter to CDLocateCSystem. This then tells us that the integer at offset 0, that is compared in the loop, is actually the EncryptionType member of structure. This makes sense, since it holds the integer value for the particular authentication system type. It also means that the elements in CSystems are the ones actually used by Windows during the authentication process, since these are the ones directly targeted by Skeleton Key attacks.

In summary, reverse engineering has showed us that the active RC4 authentication system structure can be located by enumerating CSystems and then looking for the element that has an EncryptionType of 0x17 (23). This precisely matches how CDLocateCSystem uses its first parameter to determine which element of the CSystems array to return to the caller. It also tells us that the type of each element is KERB_ECRYPT, which is very handy since we already have the definition for this type.

Reverse Engineering the RC4 Structure Origin

After learning how CDLocateCSystem operated, the next analysis step taken was to determine if the RC4 structure inside the CSystems array could be found directly. While enumerating the array is not difficult nor time consuming, in memory forensics research we aim to find the most direct path to data to avoid analysis issues that can be caused by smear. 

To begin this analysis, we wanted to determine how elements of CSystems were registered, with particular interest in the RC4 system. Examining cross-references (meaning, finding code that references), CSystems showed only a few locations inside of cryptdll.dll. Of these, the CDRegisterCSystem function sounded the most promising, as it would hopefully lead us to RC4 being registered. 

The following image shows the decompiled view of this function:

As can be seen, this is a pretty simple function that first (line 6) checks against the maximum number of registered systems (0x18), and then bails if already at the maximum. Next, the function determines the offset into CSystems (line 9) by using cCSystems shifted by 7. This matches our understanding of cCSystems and the shifting by 7 from earlier. The function then simply copies in the values from the passed in data structure (a1) into the correct offsets of CSystems. In summary, whatever values are in the system being registered are copied separately inside of CSystems, duplicating them in memory.

Following cross-references to CDRegisterCSystem leads us to many references inside of LibAttach; a decompiled view is shown below:

This function is exactly what we were looking for, as we can see all the different systems being registered. We also see our system of interest, csRC4_HMAC, being registered on line 5. If we examine the data at this address, we can verify this with seeing 0x17 (23) as the first integer. We learned earlier that this is the EncryptionType targeted by Mimikatz.

As seen above, not only is the 0x17 (23) present at the first offset, but a little further down we also see the string defined for the system (RSADSI RC4-HMAC), as well as the handlers for events the system must support. Looking at the list of functions, we find the legitimate handlers for the initialization (rc4HmacInitialize) and decryption (rc4HmacDecrypt) routines that Mimikatz targets. This gives us the specific symbol names that should correspond to the handlers we find inside of analyzed memory samples.

In summary, this reverse-engineering effort to find the origin structure led us to two import conclusions. First, even though we know the symbol name of the static RC4 structure (csRC4_HMAC), we cannot analyze this directly, as a copy of its values will be placed inside of CSystems. This means we will still need to enumerate CSystems to get the “active” values, but it also means that we can potentially choose to leverage the duplicate, original data in our plugin. Second, by knowing the symbol names of the legitimate initialization and decryption handlers, we can make the sanity checks performed by our plugins as specific as possible.

With these two reversing efforts complete, we can now start to develop our plugin!

Designing the windows.skeleton_key_check Plugin

Our previous analysis gave us all the information we need to design and implement our plugin; we saw exactly how the operating system retrieves our desired data structure. As a direct approach, this would include the following steps:

1. Find the address of CSystems
2. Walk each element to find the active RC4 system
3. Compare its initialization and decryption handlers to the known-good symbols

After the handlers are processed, the plugin would then report whether the handler’s value is legitimate or if a Skeleton Key attack has been performed. 

Creating a New Plugin

To start, we must create a base Volatility 3 plugin that is capable of processing Windows samples. A major goal of Volatility 3 was to have significant and always-up-to-date documentation for both users and developers. This documentation is stored on the Volatility 3 page of readthedocs. There is also a section specifically on writing a basic plugin here. 

At a high level, all plugins must define their requirements, a run method, and a generator method. The run method executes first and calls the generator method to create the data sets that will be displayed on the terminal (or output in whatever format other interfaces support). For more information, please see the documentation above.

For our Skeleton Key plugin, we use the basic starting form to then implement the steps listed previously. Note that the plugin being described in this blog post is already available in Volatility 3 here. Since line numbers change after each new commit, we instead will be referencing portions of the plugin by the function name. Also, we will be showing screenshots of code portions being discussed with the line numbers starting at 1. This will guide the discussion in a consistent manner. 

Implementation - Writing the run Function

The run function is called first when a plugin’s execution begins. The expected return value is a TreeGrid that the calling user interface will then display for the analyst. The following image shows the run function, along with the process filter from our Skeleton Key plugin:

On line 12, the return statement begins with the construction of the required TreeGrid instance. The first parameter to the TreeGrid constructor is the list of columns that the plugin will display. Each column is specified with its name and type. For this plugin, we have chosen to display the process ID and name of analyzed lsass.exe instances; whether or not a Skeleton Key attack was found; and the addresses of the initialization and decryption handlers. Note that the handlers are listed by their address in memory, which Volatility 3 will automatically print in hexadecimal due to the format_hints.Hex specifier. This is similar to the [addrpad] specifier of Volatility 2.

Next, the generator function is called. For plugins that operate on data not made available by another plugin, the generator function will be called with no arguments. For Skeleton Key, since we only want to analyze lsass.exe processes, we can leverage list_processes to perform the filtering for us. This filtering occurs through the use of the filter_func argument, which specifies a callback that evaluates if a process object should be yielded to the caller. Our filtering function, lsassproc_filter, is very simple; it only needs to evaluate if the process name is lsass.exe. 

Implementation – Leveraging PDBs

 

Our reverse-engineering effort showed us that four symbols—cSystems, cCSystems, rc4HmacInitialize, rc4HmacDecrypt—hold the key data we need to write a complete plugin. Luckily, one of the new features of Volatility 3 is the ability to automatically download and incorporate PDB (symbol) files into the analysis flow of plugins. This is accomplished by locating the PE file (.exe, .dll, .sys) of interest and parsing it with the PDB utility API. Since cryptdll.dll holds the symbols our plugins need, the first step is to find the DLL within the address of lsass.exe:

The above image shows that _find_cryptdll—a function that receives the process object for lsass.exe—iterates through its memory regions (line 12), retrieves the filename for the current region (line 13), and checks for the file of interest (13-15). Once cryptdll.dll is found, its base address and size are returned (16-17). 

Once cryptdll.dll has been located, its information can then be passed to the PDB utility APIs:

As shown, calling into the PDB API is straightforward, but this actually triggers quite a bit of activity inside the core of Volatility 3. First, the memory range specified for the PE file is scanned to find its GUID, which is unique identifier for the file. Next, the local Volatility cache is checked to see if the PDB for this GUID has already been downloaded and processed during previous plugin runs. If so, then the cached file is parsed and returned to the caller.

If the GUID is not in the cache, then Volatility will attempt to download the PDB file from the Microsoft symbol server.  If successful, then the PDB will be parsed, converted to Volatility’s symbol table format, and stored within the cache. 

Assuming the PDB is successfully downloaded and parsed, then our plugin has direct access to the offsets of the needed symbols within the particular version of cryptdll.dll. This allows us to trivially find their values within a particular memory sample:

The code shown gathers the runtime address for each of the four desired symbols. For the handlers, we only need their address in memory to compare to the ones in the active RC4 system. For cCSystems, we treat it separately, as we do not want processing to fail simply because the page holding the count is unavailable. 

We also treat CSystems separately, as we need to construct an array type to cleanly enumerate its elements. Constructing this object requires not only the address of where CSystems is in memory, but also the structure definition for the array elements. Unlike the PDB file for the kernel, which includes both symbol offsets and type information, the PDB file for cryptdll.dll only includes the symbol offsets. This means we need to manually inform Volatility of the data structure layout. This is performed in Volatility 3 by creating a JSON file that describes the data structure(s) a plugin requires. You can view this file for the _KERB_ECRYPT structure here, which was based on the definition from Mimikatz discussed earlier.

Once the array is constructed, it can then be enumerated as shown below:

Volatility has built-in support for enumerating arrays, so the for loop will walk each element, creating the csystem variable as the _KERB_ECRYPT type. Before processing an element, it is checked for being valid (mapped) into the process address space (lines 2-3). Next, the EncryptionType value is compared with our type of interest (lines 6-7). To determine if a Skeleton Key is present, we compare the Initialize and Decrypt members of the system found in memory to the expected values from the PDB file. If either of these have been modified, then a Skeleton Key attack has occurred, or at a minimum, a modification has occurred that an analyst would want to know about. 

With all of the values computed, displaying the results to the analyst requires just a simple yield of the data. This can be seen in lines 13-17 and will result in the process name and PID, presence of a Skeleton Key, and handler addresses being displayed. This immediately informs the analyst if a Skeleton Key was found, and if so, where the malicious handler values are in memory. 

The following image shows a run of our new plugin against an infected memory sample:

Adding Resiliency to windows.skeleton_key_check

So far, our plugin is able to successfully detect Skeleton Key attacks by leveraging the cryptdll.dll PDB file to determine where our four symbols of interest are located in memory. Unfortunately, real-world memory forensics is not always this straightforward, and the data we would like may not be memory resident or it may be smeared. Thus, it is also advantageous to consider other approaches.

In the case of leveraging a PDB file for analysis, there are a few situations that could prevent us from determining which PDB file is needed for analysis, as well as obtaining that PDB file.

1. The page containing cryptdll.dll’s GUID could be paged out or smeared. 
2. The analysis system may be offline and unable to download the PDB file from Microsoft’s symbol server.
3. Although rare, Microsoft has published corrupt/broken PDB files for modules shipped with stable versions of Windows.

In these situations, we would still like to be able to detect Skeleton Key attacks, but we need a different approach to gather the required data. 

Finding CSystems Without a PDB File

Using knowledge gained from previous work on the plugin, we know that the CDLocateCSystem function directly references two of the four symbols we need: CSystems and cCSystems.  This means that by performing static binary analysis of CDLocateCSystem, we should be able to determine the address of these symbols, since the function’s instructions will reference the addresses themselves. This is a common tactic in memory analysis and reverse engineering tasks to find symbols that are not exported or where a symbol file cannot be obtained. 

To attempt to find CDLocateCSystem without the use of a PDB file, we parse the export directory of cryptdll.dll, since it exports CDLocateCSystem by name. The following image shows how this is performed in Volatility 3:

First, a reference is obtained to the type information for PE files (lines 1-7). Next, a Volatility 3 PE file object is constructed starting at the base address of cryptdll.dll (lines 9-11). This object contains a number of convenience methods for accessing common data, such as the data directories. This is leveraged on line 15 to parse the export directory, and then loop through its exported symbols starting on line 22. The body of this loop then looks for CDLocateCSystem, and when found, attempts to read the bytes (opcodes of the instructions) from its location in memory. 

If these bytes can be read, then the _analyze_cdlocatecsystem function is called, which leverages capstone to perform the static disassembly necessary to locate both symbols. After locating them, it will construct the array object using the same method as described when the PDB file symbols were used. 

Assuming the export table and opcodes for CDLocationCSystem are present, this method will successfully find CSystems and allow us to locate the RC4 structure as we did previously. 

Finding rc4HmacInitialize and rc4HmacDecrypt

So far, we have been able to locate the RC4 structure without the PDB file. Unfortunately, there are no direct references to the legitimate initialize and decrypt handlers that we can leverage. This leaves us with two options. The first option is to verify the memory region holding the handlers, which will be discussed in this section. The second option is to attempt to scan for the values, which is discussed in the next section. Each has advantages and drawbacks, as we will discuss.

Each memory region within a process’s address space is tracked by a virtual address descriptor (VAD). Information in the VAD includes the starting and ending address of the region; the initial protection of the region; and the number of committed pages. For executables, such as lsass.exe and cryptdll.dll, one VAD will track all regions of the executable, including its code and data. Knowing this, we can check if the values of the initialization and decryption handlers are within the region for cryptdll.dll. This is shown in the following image:

This simple check ensures that the value of the handler is within the starting and ending range of the VAD for cryptdll.dll.  Although this check is not as precise as having the exact, legitimate values from the PDB file, this method still detects all forms of Skeleton Key attacks found in the wild, as they all allocate new VADs to hold the shellcode of the malicious handlers.

Note: Theoretically, an in-memory code cave could be used to place redirection stubs within cryptdll.dll and this check would be bypassed, but no malware—in the wild or proof-of-concept—has leveraged this approach. Furthermore, the PDB-based method and the one described in the next section would still detect these, rendering them not particularly stealthy. These types of attacks are also much less portable to differing operating system versions, which is one of the reasons they are uncommon in the real world.

Adding Scanning as a Last Resort to windows.skeleton_key_check

We currently have two methods to gather the data needed for Skeleton Key attacks: PDB files and export table analysis. As discussed previously, the PDB file method can be unavailable for a number of reasons, and unfortunately, the export table method can be as well. The most common reason for this is the PE header metadata being paged out or the page(s) holding the export table information are paged out. The end result is that we cannot use the export table to tell us directly where to look for our needed information.

In these situations, there is a long history of memory forensic tools scanning for the data they need. Since we have access to all pages that are present within a process’s address space, we can simply scan them in hopes of finding what we need. In the case of our Skeleton Key plugin, we were able to develop a highly effective and efficient scanner to meet our needs.

To begin, we used our knowledge that the data we need is contained within cryptdll.dll. This means we only have to scan a very small space (the size of the DLL). Second, as shown before, the layout of the active structure starts with the integer for the encryption type, which we know is 0x17 for RC4. Other research showed that the second member, BlockSize, had a value of 1 in all of our test samples. Using this knowledge, we developed a scanner based on Volatility 3’s scanning API:

The scanner is configured to look for an 8-byte pattern of 0x17 followed by 1 in little-endian integer format. It attempts to instantiate a _KERB_ECRYPT type at each address where this pattern is found. To strengthen the check, we also verify that the Encrypt and Finish members our potential structure reference addresses are inside of cryptdll.dll. Neither of these are targeted by Skeleton Key attacks and validating their values provides a strong check against false positives.

The following shows the output of our plugin when the scanning method is used:

Note that there are two lines of output. This occurs beause the scanner finds both the active version of the RC4 structure and the version that is statically compiled into the application. Having both outputs provides some advantages: there is direct visual confirmation that the active structure is hooked, and the statically compiled version reveals the addresses for the legitimate handlers, even without PDB usage.

Wrap Up

In this blog post, we have walked through the entire process for memory forensics research and development. We analyzed a target (Mimikatz’s Skeleton Key attack), analyzed the subsystem it abuses (the authentication systems managed by cryptdll.dll), and developed a new Volatility plugin that can automatically analyze this subsystem for abuse. This is a common workflow used to develop Volatility plugins.

If you find this type of research interesting, please consider developing a new plugin and submitting it to our Volatility Plugin Contest. Note that your submission does not have to be anywhere near as thorough as the plugin presented here; even submitting a new capability with just one of the discovery methods (PDB files, export analysis, scanning) used would be sufficient for an entry. We showed the full range here to display many of Volatility 3's new capabilities, but we certainly do not expect all plugins to meet this level of complexity. 

We hope you have enjoyed this post. If you have any questions or comments, please let us know. You can find us on Twitter (@volatility) and our Slack server.

-- The Volatility Team