/dev/crash Driver

As you may or may not know, some distributions (RHEL, Fedora, Ubuntu) block some reads and writes to /dev/mem and have for a while. I first came across this when writing my thesis at John Jay College. Since I was trying to test the memory encryption library I had written and original tests comprised of scanning all of memory, the /dev/mem barrier was a bit cumbersome. I had gotten around it by using a Python script called Zeppoo-dump.py (project no longer maintained) to overwrite the offending instructions.

The following code only allows access within the first 256 pages of memory. (/usr/src/linux/drivers/char/mem.c):

#ifdef CONFIG_STRICT_DEVMEM
static inline int range_is_allowed(unsigned long pfn, unsigned long size)
{
u64 from = ((u64)pfn) << PAGE_SHIFT;
u64 to = from + size;
u64 cursor = from;

while (cursor < to) {
if (!devmem_is_allowed(pfn)) {
printk(KERN_INFO
"Program %s tried to access /dev/mem between %Lx->%Lx.\n",
current->comm, from, to);
return 0;
}
cursor += PAGE_SIZE;
pfn++;
}
return 1;
}
#else
static inline int range_is_allowed(unsigned long pfn, unsigned long size)
{
return 1;
}

and from (/usr/src/linux/arch/x86/mm/init_32.c):

int devmem_is_allowed(unsigned long pagenr)
{
if (pagenr <= 256)
return 1;
if (!page_is_ram(pagenr))
return 1;
return 0;
}

You can find a nice writeup by Anthony Lineberry from BH Europe 2009.

So suppose you want to collect the memory image from /dev/mem? What will happen if you try to do so on a machine that has CONFIG_STRICT_DEVMEM enabled? If you try to collect memory using dd you will see the following:

# dd if=/dev/mem of=mem.dd
dd: reading `/dev/mem': Operation not permitted
2056+0 records in
2056+0 records out
1052672 bytes (1.1 MB) copied, 0.159965 s, 6.6 MB/s

You should also see the following in /var/log/messages though some values will obviously vary:

Aug 23 14:37:15 [comp name] kernel: [17415.953941] Program dd tried to access /dev/mem between 101000->101200.

So there has be a way around this, right? Checking the Redhat Crash Utility listserv yielded some good advice. There are three courses of action proposed:

(1) Rebuild your kernel without the CONFIG_STRICT_DEVMEM restriction.
(2) Port the Fedora /dev/crash driver (./drivers/char/crash.c) to your kernel.
(3) Write a kretprobe module that tinkers with the return value of the
kernel's devmem_is_allowed() function such that it always returns 1.

I don't want to recompile the kernel since I'll loose whatever is currently in memory, so I'll focus on (2). Since I am currently using Ubuntu instead of Fedora, I knew I would have to port the code over. So I found a copy of crash.c and crash.h and set to work. You can find the ported crash driver here as well as a Makefile.

Now, I take NO responsibility for what may happen to your machine if something goes wrong during installation. This is for a 32bit system, and I have only tested this on Ubuntu Ibex kernel 2.6.27-14-generic. I still need to do some testing and will probably have more to say about that later... That being said, we'll continue.

Grab the tar file from above and extract:

#tar -xvzf crash_driver_ubuntu.tgz
crash_driver/
crash_driver/crash.h
crash_driver/Makefile
crash_driver/crash.c

Go inside the newly created folder and compile the kernel module:

# cd crash_driver/
# ls
crash.c crash.h Makefile

# make
make -C /lib/modules/2.6.27-14-generic/build M=/home/levy/crash/crash_driver modules
make[1]: Entering directory `/usr/src/linux-headers-2.6.27-14-generic'
CC [M] /home/levy/crash/crash_driver/crash.o
Building modules, stage 2.
MODPOST 1 modules
CC /home/levy/crash/crash_driver/crash.mod.o
LD [M] /home/levy/crash/crash_driver/crash.ko
make[1]: Leaving directory `/usr/src/linux-headers-2.6.27-14-generic'

At this point you should have the following files:

# ls
crash.c crash.h crash.ko crash.mod.c crash.mod.o crash.o Makefile Module.markers modules.order Module.symvers

The file of interest is the crash.ko kernel module. We will load this into the kernel and check that it is installed correctly:

# insmod crash.ko

# lsmod |grep crash
crash 10368 0

# ls -l /dev/crash
crw-rw---- 1 root root 10, 59 2009-08-23 15:04 /dev/crash

# tail -n 1 /var/log/messages
Aug 23 15:04:10 [comp name] kernel: [19030.855920] crash memory driver: version 1.0

So now we have a new device we can use to access memory: /dev/crash

# dd if=/dev/crash of=crash.dd
dd: reading `/dev/crash': Bad address
6812680+0 records in
6812680+0 records out
3488092160 bytes (3.5 GB) copied, 157.964 s, 22.1 MB/s

I'm not yet sure what the "Bad address" error means, but I suspect it is because dd tried to read beyond the 3.3 GB of memory that I have available.

You can remove the crash.ko module like so when you are finished:

# rmmod crash

Now let's test the newly obtained memory dump to see if it works. I'm going to use the RH Crash Utility with the volatile patch which you can find here:

# ./crash -f /boot/System.map-2.6.27-14-generic /usr/src/linux-source-2.6.27/vmlinux crash.dd --volatile

crash 4.0-8.9
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...

SYSTEM MAP: /boot/System.map-2.6.27-14-generic
DEBUG KERNEL: /usr/src/linux-source-2.6.27/vmlinux (2.6.27.18)
DUMPFILE: crash.dd
CPUS: 2
DATE: Mon Aug 23 12:31:54 2009
UPTIME: 02:44:55
LOAD AVERAGE: 0.10, 0.17, 0.17
TASKS: 252
NODENAME: --
RELEASE: 2.6.27-14-generic
VERSION: #1 SMP Tue Aug 18 16:25:45 UTC 2009
MACHINE: i686 (1994 Mhz)
MEMORY: 3.2 GB
PID: 0
COMMAND: "swapper"
TASK: c0471340 (1 of 2) [THREAD_INFO: c04aa000]
CPU: 0
STATE: TASK_RUNNING

crash>

So far so good :-)

crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
0 0 0 c0471340 RU 0.0 0 0 [swapper]
> 0 0 1 f744e480 RU 0.0 0 0 [swapper]
1 0 0 f7448000 IN 0.1 3056 1900 init
2 0 1 f7448c90 IN 0.0 0 0 [kthreadd]
3 2 0 f7449920 IN 0.0 0 0 [migration/0]
4 2 0 f744a5b0 IN 0.0 0 0 [ksoftirqd/0]
5 2 0 f744b240 IN 0.0 0 0 [watchdog/0]
6 2 1 f744bed0 IN 0.0 0 0 [migration/1]
7 2 1 f744cb60 IN 0.0 0 0 [ksoftirqd/1]
8 2 1 f744d7f0 IN 0.0 0 0 [watchdog/1]
9 2 0 f744f110 IN 0.0 0 0 [events/0]
10 2 1 f7460000 IN 0.0 0 0 [events/1]
11 2 0 f7460c90 IN 0.0 0 0 [khelper]

[snip]

crash> foreach files
PID: 0 TASK: c0471340 CPU: 0 COMMAND: "swapper"
ROOT: / CWD: /
No open files

PID: 0 TASK: f744e480 CPU: 1 COMMAND: "swapper"
ROOT: / CWD: /
No open files

PID: 1 TASK: f7448000 CPU: 0 COMMAND: "init"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
0 f69c8300 f700c550 f695a3e0 CHR /dev/console
1 f69c8300 f700c550 f695a3e0 CHR /dev/console
2 f69c8300 f700c550 f695a3e0 CHR /dev/console
3 f69c8f00 f720d770 f7243c38 FIFO
4 f69c8780 f720d770 f7243c38 FIFO
5 f69c86c0 f7218990 f7045228 SOCK
6 f69c8b40 f70216e8 f70b6000 DIR inotify

PID: 2 TASK: f7448c90 CPU: 1 COMMAND: "kthreadd"
ROOT: / CWD: /
No open files

PID: 3 TASK: f7449920 CPU: 0 COMMAND: "migration/0"

[snip]

crash> foreach net
foreach: WARNING: net command requires -s or -S option

PID: 0 TASK: c0471340 CPU: 0 COMMAND: "swapper"
No open sockets.

PID: 0 TASK: f744e480 CPU: 1 COMMAND: "swapper"
No open sockets.

PID: 1 TASK: f7448000 CPU: 0 COMMAND: "init"
FD SOCKET SOCK FAMILY:TYPE SOURCE-PORT DESTINATION-PORT
5 f7045200 f6bfe380 UNIX:DGRAM

[snip]

In order to compile the kernel so that you can use the RedHat Crash Utility, you can follow the Ubuntu tutorial. After installing the appropriate packages you may have to run the following command:

sudo apt-get build-dep linux

You should end up with a linux-source*.tar.bz2 file under /usr/src . You should also have at least one folder with the kernel headers for your current kernel. You can extract the .tar.bz2 file as so:

# tar -xjf linux-source*.tar.bz2

Go into the resulting folder and set following flag in the Makefile:

CFLAGS_KERNEL = -g

Copy the .config file from your /usr/src/linux-headers-$(uname -r) folder into the /usr/src/linux-source-$(uname -r) folder.

Now type make. After the kernel is finished compiling, you should end up with a vmlinux file. This is the uncompressed kernel image with the debugging information that you need in order to run the RH crash utility.

For more information on the RH Crash Utility, check out:
Official RH Crash Utility Website
Linux Memory Forensics by A. Walters, M. Cohen and D. Collett.
slides from CEIC

Briefly: VDP Project

I have volunteered to help with the Volatility Documentation Project (VDP) for Volatility. If you have something you would like to contribute, please feel free to email me at

jamie -{dot}- levy -{at}- gmail -{dot}- com

Contributed documents will appear on the Volatility Google Code website.

We have two new contributions by SAL:

VolReport(win) with an accompaning batch script as well as a manual covering the visual capabilities of Volatility.

Keep them coming :-)

Installing Volatility Plugins

So you've already installed Volatility using SVN and you want to try out some of the community plugins that people are raving about. Publicly known plugins are listed on the forensics wiki. The wiki contains links to plugins as well as links to blogposts/articles for further information on installation, dependencies and how they work.

Most plugin installation is straightforward where one may simply place the plugin in the memory_plugins directory within the Volatility directory. Some are only slightly more complicated by needing a helper library installed in addition to the plugin itself. Others are even more complicated and require some installation of Python libraries which may or may not need the help of other compiled libraries. Therefore we have three cases for plugin installation (please visit the forensics wiki for more information):

1. Simple Case - only in memory_plugins
   • volshell
   • IDT
   • cryptoscan
   • orphan_threads
   • keyboardbuffer
   • getsids
   • moddump
   • objtypescan
   • symlinkobjscan
   • driverscan
   • fileobjscan
   • pstree
2. More Complex Case - also supporting file(s)
   • driverirp (needs driverscan)
   • threadqueues (needs lists.py)
   • ssdt (needs lists.py)
3. Most Complex Case - installation of supporting libraries
   • malfind (needs pydasm and pefile)
   • kernel_hooks (needs pefile)
   • usermode_hooks (needs pefile)
   • volreg (needs pycrypto)
   • VolRip (needs volreg and Inline::Python)

Simple installation of volshell

For an example of a simple installation, we will install the volshell plugin. Simply download the volshell.py file and place it into your memory_plugins directory. You can test to make sure that is installed correctly by running Volatility without any arguments and volshell should appear under "Supported Plugin Commands" highlighted below in Figure 1. All other "simple case" plugins should install the same way.



Figure 1: Installation of volshell

More Complex Cases

The ssdt and threadqueues plugins require that the lists.py library file be placed in the forensics/win32 directory in addition to placing the ssdt.py and threadqueues.py into the memory_plugins folder as before. The driverirp plugin requires the driverscan plugin in order to work. Both of these plugins are placed in the memory_plugins directory.

After placing the files in the appropriate places, check to make sure that the plugins are properly installed by running volatility without any arguments as before and checking under "Supported Plugin Commands" (Figure 1).

Most Complex Cases

For the "most complex cases" other libraries must be installed for the plugin to work properly. First we will look at installing the malfind plugin. First of all, download the malfind.py plugin file and place it in the memory_plugins directory. Now you must install the pydasm and pefile libraries.

In order to install the pydasm library, you will have to do some initial setup including by installing a gcc compiler and make. For this tutorial, we will use MinGW.



Figure 2: Sourceforge download site for MinGW

Download the windows installer for MinGW from the sourceforge website (Figure 2). Double click to install (Figure 3-9).



Figure 3: Choose "Download and Install"



Figure 4: Click "Agree"



Figure 5: Choose "Current"



Figure 6: Choose compilers and MinGW make

You do not necessarily have to install all compilers however, for simplicity, do a full install.



Figure 7: Choose location for installation. The default is fine.



Figure 8: Installing



Figure 9: installation complete

Once the installation is complete and you have clicked finish, you will have to make a few adjustments to the installation in order to get things working properly. First of all, we need to have an executable called "make.exe". The make executable for MinGW is appropriately named mingwmake.exe. Simply copy this executable and paste it into the same directory (C:\MinGW\bin) which should result in an identical copy named "Copy of mingwmake.exe". Rename this executable to "make.exe" as shown in Figures 10-11.



Figure 10: "Copy of mingwmake.exe"



Figure 11: Rename to "make.exe"

Now we have to modify our path to include the executables for MinGW. If you have a regular start menu, click on start and then right click on “My Computer” and choose properties. If you have the classic start menu, just right click on “My Computer” and choose properties. Click on the “Advanced” tab and then click on “Environmental Variables”. Click on the Path system variable towards the bottom of the window and click the “Edit” button. We will append the path of our Python installation to the end of the existing Path variable. Where it says “Variable Value” go to
the end of the line and add the following (if you installed MinGW in a different location, modify appropriately):

;C:\MinGW\bin



Figure 12: Adding C:\MinGW\bin to the path variable

Now for installing pydasm. Download the source code for libdasm. The easiest way to extract the contents from this tar ball is using 7zip. Once you have 7zip installed, you can associate all zip files by starting the 7zip Filemanager (Start->Programs->7-zip->7-zip File Manager) and clicking on "Tools->Options" and clicking "Select all" in the system tab and "OK" (Figure 13).



Figure 13: Associating zip file types

At this point you are ready to extract the libdasm/pydasm source code. Double click the downloaded pydasm tar ball. You should see the following:



Figure 14: Opening libdasm tar ball with 7-zip

Double click on the libdasm*.tar file inside from within the 7-zip application until you see a folder icon with the name libdasm-1.5 (or other version number):



Figure 15: libdasm folder

Highlight the folder and then click on the extract button and say OK. The folder will extract with all source code inside to the path you choose, or by default the current directory:



Figure 16: Extracting the libdasm source code

Now open a command prompt and change directories until you are in the newly extracted libdasm folder. Type the following commands:

make
cd pydasm
python setup.py build -c mingw32
python setup.py install

That's it! You've installed pydasm.

Now you are ready to install the pefile library. Grab the zip file or tar ball of the source code and extract is as you did before. Go into that resulting folder and type the following:

python setup.py build
python setup.py install

Now you've installed pefile. Now you should see the malfind plugin listed under supported plugins for Volatility. All the other plugins that were depend on pefile should work as well if they are installed in the memory_plugins directory.

Installing the volreg plugin requires pycrypto. Simply go the gitweb interface for this project and download the latest git snapshot by clicking on "snapshot". This will download a tar ball file of the source code. Simply extract it as you did before, then open the command prompt and change into that directory. Then type the following:

python setup.py build
python setup.py install

You've now installed the pycrypto library. Download the volreg tarfile and extract the contents into your Volatility folder by double clicking as before, selecting all three folders and changing the extraction path to your Volatility folder. All files should be placed into the correct location:



Figure 17: Extraction of volreg into Volatility directory.

Next time we will cover the volrip plugin after I figure out how to get Inline::Python working under windows...

Volatility SVN

Since the last post on Volatility some of you may be wondering how you may download the newest source of Volatility from the Google SVN repository. Well for Linux it's very easy. After you install subversion using yum or apt-get, you simply follow the instructions on the website:

$ svn checkout http://volatility.googlecode.com/svn/trunk/ [folder name]

where [folder name] is replaced by the name of the folder you want to contain the downloaded code.

For Windows users, it's only slightly more complicated. First you must install a subversion client. For this post we will use Tortoise SVN.


Figure 1: Tortoise SVN website

Go to the downloads section and choose the appropriate installer. For this post we are choosing the 32 bit version.


Figure 2: Tortoise SVN website - downloads section

Once you have downloaded the installer, run it. You may see the following security warning, just click Run. All defaults should be fine, so keep clicking next until the installation finishes.



Figure 3 and 4: Running the Tortoise SVN installer



After the installation is complete, you will have to restart your computer. After restart you should see the following menu added when you right click:


Figure 5: Tortoise SVN right click menu options

Create a folder for the repository (for this run through we will create a folder called Volatility on the root of the drive (C:\). Right click and choose "SVN Checkout" noted in the picture above. After choosing this menu, you should see the following:


Figure 6: Filling in options to download Volatility

Paste the url of the repository: http://volatility.googlecode.com/svn/trunk/ in the first text box and the location of the newly created folder in the second text box (as shown above). Leave the other options the same as shown above. Press OK. You should see the following as it begins downloading and then finishes:



Figure 7 and 8: Downloading Volatility from SVN



The newly created folder should now contain the SVN repository. This includes another folder named "Volatility" that contains the actual source code. If you go inside the inner Volatility folder you should see the python source code files as shown in Figure 10.




Figure 9 and 10: Newly created folder containing Volatility SVN repository.



To test the newly downloaded code, open a command shell, go inside the Volatility folder (which is inside your newly created folder) and type "python volatility" without the quotes. (This is assuming you have already installed Python, which is covered in the installation manual.) See below:


Figure 11: Running Volatility

Now you're set. You have the latest source code for Volatility. Next will be how to install plugins...

Briefly: recordmydesktop

Occasionally I have needed to make screencasts for my students so that they would have something to look at in their own time. There are two tools that make this easy on Linux (both should available in the yum or apt repositories):

• recordmydesktop
• ffmpeg.org

Recordmydesktop does as it sounds: it records the desktop. It has options to set the size of the area to record as well as the window you would like to record. I like to choose the window option myself. I also like to record without sound, but you can figure out how to modify the script to remove that option if you so choose.

FFmpeg is a nice tool that allows you to convert, record and stream audio and video. I use it to convert the resulting video from recormydesktop to flv format in order to upload to photobucket or elsewhere.

To make my life easier, I have created the following script that takes in 1-2 arguments. The first argument should be the desired name of the resulting video file. The second argument is an (optional) amount of time to wait before recording. The default wait time is 3 seconds.

When you run the script it waits for you to click on the window that you wish to record by using xwininfo to get the window id number. You will notice that the mouse changes to a + sign as it is waiting for you to click. Once you click the window, it will begin recording that window area after the appropriate wait time has transpired. The video is converted to [chosen filename].flv after you have stopped recording (CTRL+C in terminal from which you started the script).

Feel free to do with as you please. The script can be found below:

#!/bin/bash
#
# Warning: this does not have robust error checking!

bad=67
if [ $# -lt 1 ]
then
echo "Usage: $0 [filename] [[optional time]]"
exit $bad
fi

if [ $# -eq 1 ] #check for arguments
then
time=3 #if one (only filename) exists, sleep for 3 seconds
filename=$1 #set filename
else
time=$2 #else, we'll sleep for $2 seconds
filename=$1 #set filename
fi

recordmydesktop -windowid `xwininfo |grep "Window id:"|sed -e "s/xwininfo\:\ Window id:\ //;s/\ .*//"` -o $filename.ogv -delay $time --no-sound

ffmpeg -i $filename.ogv -b 384000 -s 640x480 -pass 1 -passlogfile log-file $filename.flv