MoVP 2.5: Investigating In-Memory Network Data with Volatility

Month of Volatility Plugins

In this post I will discuss Volatility’s new Linux features related to recovering network information.  This will include enumerating sockets, network connections, and packet contents.  The post will discuss each plugin along with its implementation, how to use it, output on a sample memory capture, and which forensics scenarios it applies to.

If you would like to follow along or recreate the steps taken, please see the LinuxForensicsWiki  for instructions on how to do so.

linux_arp

The linux_arp plugin recovers the ARP table and prints out the IP address, MAC address, and network interface for each entry. It simulates running arp –a on Linux systems.

Running the Plugin

# python vol.py --profile=LinuxDebianx86 -f network.lime linux_arp

Volatile Systems Volatility Framework 2.2_rc1

[::                ] at 00:00:00:00:00:00    on lo

[192.168.181.129   ] at 00:0c:29:1f:f7:24    on eth0

[192.168.181.255   ] at ff:ff:ff:ff:ff:ff    on eth0

[192.168.181.1     ] at 00:50:56:c0:00:08    on eth0

Forensics Use

ARP entries are commonly used in two situations. The first is to prove or disprove the existence of ARP spoofing attacks on a network. If any ARP entries associate an IP address with an incorrect MAC address, the existence of this attack can be verified. The second use is to determine which paths were taken by attackers during lateral movement on a network. Any type of network reconnaissance or full connections to other computers on the local network will generate ARP entries for the contacted resources. This can be used in conjunction with other Volatility plugins and with network forensics data to retrace attacker’s or malware’s steps.

Implementation

The plugin works by walking the neigh_tables hash table and recovering each neighbor structure. Each neighbor structure contains the information for one entry in the ARP table.

linux_ifconfig

This plugin simulates the linux_ifconfig command and for each network interface prints the name, IP address, MAC address, and if the interface is in promiscuous mode.

Running the plugin

# python vol.py --profile=LinuxDebianx86 -f network.lime linux_ifconfig

Volatile Systems Volatility Framework 2.2_rc1

Interface   IP Address      MAC Address        Promiscuous Mode

---------   ---------       -----------        ---------

lo         127.0.0.1        00:00:00:00:00:00  False

eth0       192.168.181.128  00:0c:29:ea:35:c9  True

Forensics Uses

The recovery of each interface immediately allows us to correlate its IP and MAC address information during network forensics. The promiscuous mode column tells us if the device was sniffing the network at the time of the memory sample.  In this sample, tcpdump was running on eth0 at the time of capture. Many userland rootkits run a network sniffer on the local machine and then hide the process from other userland tools.  These tools can also be used to sniff all packets on a local switched network if an ARP spoofing attack is used in conjunction with it.  You can also find proof of devices entering and leaving promiscuous mode searching system logs on disk or using the linux_dmesg plugin:

# python vol.py --profile=LinuxDebianx86 -f network.lime linux_dmesg | grep eth0

<snip>

<6>[23716.356788] device eth0 entered promiscuous mode

<6>[23734.599028] device eth0 left promiscuous mode

<snip>

Implementation

This plugin recovers the interfaces using one of two methods depending on which kernel version was used in the sample. For older kernels, the dev_base pointer is located which points to the net_device structure for the first interface in the system. The internal list of this structure, based off the next member, is then walked. Each net_device structure contains an in_device structure at its ip_ptr member. Together, these two structures contain all the information needed to reproduce ifconfig output.

On newer kernels, the list of devices is gathered by walking the net_namespace_list list and then recovering each net_device structure from each element’s dev_list member.

linux_netstat

This plugin simulates the netstat command and for each network connection prints the source and destination IP address and port, state of the socket if applicable, and the process that owns the socket.

Running the plugin

# python vol.py --profile=LinuxDebianx86 -f network.lime linux_netstat

Volatile Systems Volatility Framework 2.2_rc1

UNIX /dev/log

TCP   :::445      :::0       LISTEN      smbd/1138

TCP   0.0.0.0:22  0.0.0.0:0  LISTEN      sshd/1291

TCP   192.168.181.128:22  192.168.181.1:64462 ESTABLISHED sshd/1871

UDP   192.168.181.128:137 0.0.0.0:0 nmbd/1307

In the sample output, I have filtered the output to show unique sets of information. The first line shows a UNIX socket opened on /dev/log. The second and third lines show listening TCP sockets, where :::  indicates that the smbd process is listening on all IPv6 interfaces while 0.0.0.0 indicates that sshd is listening on all IPv4 interfaces. The next line shows an established (active) connection between port 22 on 192.168.181.128 and port 64462 on 192.168.181.1. The last line shows that the nmbd process is listening on port 137 on 192.168.181.128 for the UDP protocol.

Note: The plugin can be given the –U option to have it skip printing UNIX sockets. Depending on the distribution and its configuration, some samples may have hundreds of UNIX sockets of no real forensics value.  

Forensics Use

Being able to determine all active connections on a machine is often essential during incident response and network forensics. Backdoor connections from attackers, active data exfiltration, and the presence of rogue services and servers can all be detected from linux_netstat.

Implementation

This plugin works by leveraging the linux­_lsof plugin to enumerate the opened files of all processes. It then filters these files to those that are sockets by checking if the file structure’s file_operations member points to socket_file_ops or if the dentry_operations points to sockfs_dentry_operations. 

For each socket, it finds the socket structure, which embeds the file’s inode structure, and then obtains its inet_sock structure, which is the sk member of socket.  The inet_sock structure then contains all of the members needed to produce the output of netstat.

linux_route_cache

The route cache plugin recovers the kernel’s routing cache. This cache keeps a pairing of each IP address connected to by a computer along with the route gateway used to connect to it. The kernel keeps this cache so that repeated connections to a server outside the network do not require repeated lookups in the route table.

Running the plugin

# python vol.py --profile=LinuxDebianx86 -f network.lime linux_route_cache

Volatile Systems Volatility Framework 2.2_rc1

Interface        Destination          Gateway

---------------- --------------------      -------

eth0             173.194.43.41        192.168.16.2

eth0             173.194.43.38        192.168.16.2

eth0             173.194.43.39        192.168.16.2

eth0             173.194.73.106       192.168.16.2

Forensics Use

The route cache can be very helpful during incident response and traditional forensics investigations as it contains a record of foreign IP addresses contacted by the computer. This information can serve as a starting point when examining records from network security devices or when going through packet captures of the network.

Implementation

This plugin works by locating and enumerating the rt_hash_table hash table. This table contains elements of type rt_hash_bucket and the plugin walks the chain member of each bucket. Each member in the chain list contains the information reported by the plugin.

linux_sk_buff_cache

As will be discussed in a future Month of Volatility Plugins post, the kmem_cache can be leveraged to find both allocated and de-allocated structures of a particular type in an orderly manner. The linux_sk_buff_cache plugin leverages this to find network packets that are in kernel memory and writes them to disk.

Running the plugin

# python vol.py --profile=LinuxDebianx86 -f network.lime linux_sk_buff_cache -D recovered_packets

Volatile Systems Volatility Framework 2.2_rc1

Wrote 20 bytes to de2c60c0

Wrote 1430 bytes to de2da900

Wrote 60 bytes to de21c680

Wrote 42 bytes to de2cc600

Wrote 1430 bytes to de284f00

Wrote 68 bytes to def720c0

Wrote 68 bytes to def72540
# strings recovered_packets/*
<snip>

GET /safebrowsing/rd/<removed>
HTTP/1.1
Host: safebrowsing-cache.google.com
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
Cookie: PREF=ID=<removed>:LM=1346093776:S=_zT51pWTC5-mvK0t
Pragma: no-cache
Cache-Control: no-cache
<snip>

The plugin enumerates all of the packets from memory and writes them to a file named as the virtual address of where the owning structure was found.  As can be see in the strings output, this plugin is very effective at recovering packets still referenced by the kernel.

Note: This plugin can be run with the –u/--unallocated option to recover packet structures that were previously de-allocated and are no longer in use.

Forensics Use

Being able to recover packets from memory, including historical ones, has obvious forensics value.  This can be used to prove that data seen in network captures or logs was transferred to or from a machine. It can also be used to determine actions taken by an attacker over plaintext protocols. 

Implementation

This plugin works by enumerating the skbuff_head_cache and skbuff_fclone_cache kmem_caches.

These caches both store elements of type sk_buff, which represents a single packet.  To recover the packet contents, the plugin reads starting at the location pointed to by the data member of the structure and reads for len in length bytes.

This effectively recovers the data portion of the packet. Future versions of the plugin may recover the protocol headers as well.

linux_pkt_queues

When a socket is attempting to send packets out onto the network at rates that the network cannot handle, or when the kernel has processed received packets that the corresponding userland service has not yet picked up, these packets are placed on per-socket send and receive queues.

The linux_pkt_queues plugin enumerates these queues for each active socket in the kernel and writes the recovered packets to disk.

Running the Plugin

# python vol.py --profile=LinuxDebianx86 -f network.lime linux_pkt_queues -D recovered_packets

Volatile Systems Volatility Framework 2.2_rc1

Wrote 32 bytes to receive.1466.3

Wrote 128 bytes to receive.2565.3

Wrote 32 bytes to receive.2839.3

As the plugin finds queued packets, it writes them out with a filename of <receive or send>.<PID>.<file descriptor number>.  The owning process can then be referenced by its PID in linux_pslist / linux_psaux  and the file descriptor can be matched with output from linux_lsof on a per-process basis.

Forensics Use

Recovery of the send queue of sockets can help prove that specific data and files were transferred by a particular process on a computer. For example, if an attacker is exfiltrating files over FTP or HTTP at the time of the memory capture, by recovering the queues, the investigator can find the portions that have not yet been transferred. These portions of the file can be matched with those already transferred that are contained in a network packet capture.

Recovery of the receive queue can prove that files were being downloaded during the memory capture. For example, if a user of the computer is downloading a file with his web browser, portions of the file will be stored in the receive queue until the browser has processed them. The receive queues are also able to prove that files were being uploaded to any network services running on the computer (FTP, SFTP, HTTP, etc).

Implementation

This plugin works by gathering inet_sock structures from the netstat plugin and then walking the sk_receive_queue and sk_write_queue of the inet_sock.sk member. Each element of the queue is a sk_buff and is processed the same as those found by sk_buff_cache.

Conclusion

In this post we have demonstrated a number of Volatility plugins that allow for deep inspection of the network activity that occurred on a computer. Used in conjunction with traditional network forensics, these plugins can definitively tie packets and data streams found in packet captures and network device logs to specific processes and user activity.

If you have any questions or comments please use the comment section of the blog or you can find me on Twitter (@attrc).