Thursday, May 16, 2013

MoVP II - 1.3 - VMware Snapshot and Saved State Analysis

VMware is arguably the most popular virtualization software used in production and research. However, there are various versions of VMware (Workstation, Fusion, ESX Server, etc) and not all of them write raw memory dumps with .vmem extensions for guest VMs. Specifically, some products create saved state (.vmss) and snapshot (.vmsn) files instead. These files also contain physical memory of guest VMs, but in an entirely different format than the raw/dd-style dumps that you're familiar with. The ability to parse and analyze memory dumps in the .vmss/.vmsn formats was researched and developed by Nir Izraeli. Nir wrote an initial address space for Volatility and a standalone Python utility called vmsnparser to deal with these special file types.

With the release of Volatility 2.3, we've adapted and ported Nir's code into the core framework, giving you even more flexibility with the types of memory dumps you can analyze. Unlike other memory dump file formats that simply store basic metadata, the .vmss and .vmsn files contain a fairly complex structure layout which the physical memory runs, the VM configuration data, CPU registers, and even PNG thumbnails of the VM's screen.

File Format 

At offset 0 of the .vmss/.vmsn file, there's a _VMWARE_HEADER structure, which looks like this:

>>> dt("_VMWARE_HEADER")
'_VMWARE_HEADER' (12 bytes)
0x0   : Magic                     ['unsigned int']
0x8   : GroupCount                ['unsigned int']
0xc   : Groups                    ['array', <function <lambda> at 0x1046b7320>, ['_VMWARE_GROUP']]

As you can see there's a magic value, which must be 0xbed2bed0, 0xbad1bad1, 0xbed2bed2, or 0xbed3bed3 for the file to be considered valid. Following the magic value, there is an array of _VMWARE_GROUP structures (of count GroupCount), each of which look like this:

>>> dt("_VMWARE_GROUP")
'_VMWARE_GROUP' (80 bytes)
0x0   : Name                      ['String', {'length': 64, 'encoding': 'utf8'}]
0x40  : TagsOffset                ['unsigned long long']

The Name field is a string that identifies the type of data contained in the group. Here are a few of the groups just to give you an idea of what's inside:
  • Checkpoint: VMware product that produced the memory dump, along with version and build numbers, platform, memory size, and FSB size.
  • cpu: values for all of the cpu's 32-bit registers (EIP, EAX, EBX, etc), 64-bit registers if applicable (RAX, RCX, RDX, etc), EFLAGS, IDT, control registers, debug registers, MSRs (sysEnterCS, sysEnterESP, sysenterEIP), and floating point registers.
  • memory: information on the physical memory runs, including page number, size of each region, and contents of the region. 
  • Snapshot: contents of the .vmx configuration file at the time of the snapshot, including the nvram file and extended configuration file. 
  • scsiX, ideX: details on the connected SCSI and IDE devices (X is the 0-indexed device number)
  • PCIBridgeX: details on the connected PCI devices (X is the 0-indexed device number)
  • VGA, SVGA: colormaps describing the current state of the video adaptor 
  • EthernetX: ethernet adaptor state, including eeprom, flash, IP, and other data 
That's just the tip of the iceberg. There are also groups named DMA, CMOS, FlashRam, Keyboard, serial, MemoryHotplug, vmDebugControl - all of which can be very interesting sources of information in forensic investigations. 

Now, how exactly is data organized within these groups? The TagsOffset member is an offset in the .vmss/.vmsn file where an array of _VMWARE_TAG structures can be found. An example of the structure is shown below:

>>> dt("_VMWARE_TAG")
'_VMWARE_TAG' (None bytes)
0x0   : Flags              ['unsigned char']
0x1   : NameLength         ['unsigned char']
0x2   : Name               ['String', {'length': <function <lambda> at 0x1046b7398>, 'encoding': 'utf8'}]

The tag structures are a bit more complex than others. They have a Name field and a Flags field which devotes some bits to the length of the data associated with the tag, and a set of indices (not shown) which allow you to distinguish between multiple tags with the same name within a group.

The data for a tag follows the _VMWARE_TAG structure, however there are a few intricacies that Nir researched and are better understood by reading the source code of the address space. If the system has less than 4 GB of RAM, there will be a single physical memory run stored in a group named "memory" and a tag named "Memory" using indices [0][0]. For systems with greater than 4 GB of RAM, there will be multiple runs, also in a group named "memory" but including tags named "Memory", "regionPPN", "regionPageNum", and "regionSize."


You can dump meta-data from .vmss/.vmsn files using the vmwareinfo plugin. If the data in a tag is 1, 2, 4, or 8 bytes, it's formatted as a number. If its more than 8 bytes and you supply --verbose to the plugin, you'll get a hexdump.

The example below shows a VMware saved state file from ESX 4.1.0. The group and tag names are combined in the Name column, but separated with a forward slash. You can see the values of the volatile CPU registers and the beginning of the .vmx file contents for the snapshot.

$ python -f Win7SP1x64-d8737a34.vmss vmwareinfo --verbose 

Magic: 0xbad1bad1 (Version 1)
Group count: 0x5c

File Offset PhysMem Offset Size      
----------- -------------- ----------
0x000010000 0x000000000000 0xc0000000
0x0c0010000 0x000100000000 0xc0000000

DataOffset   DataSize Name                                               Value
---------- ---------- ------------------------------------------------- -----
0x00001cd9        0x4 Checkpoint/fileversion                             0xa
0x00001cfc      0x100 Checkpoint/ProductName                             
0x00001cfc  56 4d 77 61 72 65 20 45 53 58 00 00 00 00 00 00   VMware.ESX......
0x00001d0c  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0x00001e1d      0x100 Checkpoint/VersionNumber                           
0x00001e1d  34 2e 31 2e 30 00 00 00 00 00 00 00 00 00 00 00   4.1.0...........
0x00001e2d  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0x00002046        0x4 Checkpoint/Platform                      0x1
0x00002055        0x4 Checkpoint/usageMode                     0x1
0x00002062        0x4 Checkpoint/memSize                       0x1800
0x00002071        0x4 Checkpoint/maxFBSize                     0x800000
0x00002085        0x4 cpu/cpu:numVCPUs                         0x1
0x00002095        0x4 cpu/eflags[0]                            0x86
0x000020a2        0x8 cpu/rip[0]                               0xfffff80002c1c0ba
0x000020b3        0x4 cpu/eip[0]                               0x2c1c0ba
0x000020c3        0x1 cpu/halted[0]                            0
0x00005eea        0x4 cpu/CR[0][0]                             0x80050031
0x00005efa        0x4 cpu/CR[0][1]                             0x0
0x00005f0a        0x4 cpu/CR[0][2]                             0x865eb08
0x00005f1a        0x4 cpu/CR[0][3]                             0x187000
0x00005f2a        0x4 cpu/CR[0][4]                             0x6f8
0x00005f3c        0x8 cpu/DR64[0][0]                           0x0
0x00006020        0x8 cpu/DR64[0][6]                           0xffff0ff0
0x00006034        0x4 cpu/DR[0][6]                             0xffff0ff0
0x00006046        0x8 cpu/DR64[0][7]                           0x400
0x0000605a        0x4 cpu/DR[0][7]                             0x400
0x0000606c        0x2 cpu/GDTR[0][0]                           127
0x0000607c        0x4 cpu/GDTR[0][1]                           0x3cd5000
0x0000608e        0x4 cpu/GDTR[0][2]                           0xfffff800
0x000060a0        0x2 cpu/IDTR[0][0]                           4095
0x000060b0        0x4 cpu/IDTR[0][1]                           0x3cd5080
0x000060c2        0x4 cpu/IDTR[0][2]                           0xfffff800
0x180011963    0x2c29 Snapshot/cfgFile                                   
0x180011953  2e 65 6e 63 6f 64 69 6e 67 20 3d 20 22 55 54 46   .encoding.=."UTF
0x180011963  2d 38 22 0a 63 6f 6e 66 69 67 2e 76 65 72 73 69   -8".config.versi
0x180011973  6f 6e 20 3d 20 22 38 22 0a 76 69 72 74 75 61 6c   on.=."8".virtual
0x180011983  48 57 2e 76 65 72 73 69 6f 6e 20 3d 20 22 37 22   HW.version.=."7"
0x180011993  0a 70 63 69 42 72 69 64 67 65 30 2e 70 72 65 73   .pciBridge0.pres
0x1800119a3  65 6e 74 20 3d 20 22 74 72 75 65 22 0a 70 63 69   ent.=."true".pci
0x1800119b3  42 72 69 64 67 65 34 2e 70 72 65 73 65 6e 74 20   Bridge4.present.
0x1800119c3  3d 20 22 74 72 75 65 22 0a 70 63 69 42 72 69 64   =."true".pciBrid
0x1800119d3  67 65 34 2e 76 69 72 74 75 61 6c 44 65 76 20 3d   ge4.virtualDev.=
0x1800119e3  20 22 70 63 69 65 52 6f 6f 74 50 6f 72 74 22 0a   ."pcieRootPort".

Even More Metadata

We previously introduced you to the ability of taking screenshots from memory dumps using data structures found in GUI memory. These were primitive, wire-frame diagrams, with labels for window titles. If you have a VMware .vmss/.vmsn file, there is another way to extract a full color screenshot of the guest VM at the time the snapshot was taken or the VM's state was saved. In a group named MKSVMX and a tag named imageData, there is an embedded PNG file. All you have to do is pass the --dump-dir parameter to Volatility and the screenshot will be extracted to your desired output directory.

For example:

$ mkdir screenshot
$ python -f Win7SP1x64-d8737a34.vmss vmwareinfo --verbose --dump-dir=screenshot/

Now check what you've got in your screenshot directory:


We must extend a huge thanks to Nir Izraeli for creating the vmsnparser project and providing the foundation for Volatility's VMware address space. VMware snapshot and saved state files are full of valuable information even beyond the contents of the VM's physical memory. The only downside is that some VMware products have been reported to create both a .vmss and .vmem file, where the .vmss only contains metadata describing the contents of the .vmem file. In these cases, especially for larger memory dumps (from VMs with more than 4GB RAM), there can be issues parsing the .vmem files. Until further notice, Volatility does not fully support (as in YMMV) these types of memory dumps that are split across multiple files, however in our experience the conditions are seemingly rare.

No comments:

Post a Comment