/dev/crash Driver

As you may or may not know, some distributions (RHEL, Fedora, Ubuntu) block some reads and writes to /dev/mem and have for a while. I first came across this when writing my thesis at John Jay College. Since I was trying to test the memory encryption library I had written and original tests comprised of scanning all of memory, the /dev/mem barrier was a bit cumbersome. I had gotten around it by using a Python script called Zeppoo-dump.py (project no longer maintained) to overwrite the offending instructions.

The following code only allows access within the first 256 pages of memory. (/usr/src/linux/drivers/char/mem.c):

#ifdef CONFIG_STRICT_DEVMEM
static inline int range_is_allowed(unsigned long pfn, unsigned long size)
{
u64 from = ((u64)pfn) << PAGE_SHIFT;
u64 to = from + size;
u64 cursor = from;

while (cursor < to) {
if (!devmem_is_allowed(pfn)) {
printk(KERN_INFO
"Program %s tried to access /dev/mem between %Lx->%Lx.\n",
current->comm, from, to);
return 0;
}
cursor += PAGE_SIZE;
pfn++;
}
return 1;
}
#else
static inline int range_is_allowed(unsigned long pfn, unsigned long size)
{
return 1;
}

and from (/usr/src/linux/arch/x86/mm/init_32.c):

int devmem_is_allowed(unsigned long pagenr)
{
if (pagenr <= 256)
return 1;
if (!page_is_ram(pagenr))
return 1;
return 0;
}

You can find a nice writeup by Anthony Lineberry from BH Europe 2009.

So suppose you want to collect the memory image from /dev/mem? What will happen if you try to do so on a machine that has CONFIG_STRICT_DEVMEM enabled? If you try to collect memory using dd you will see the following:

# dd if=/dev/mem of=mem.dd
dd: reading `/dev/mem': Operation not permitted
2056+0 records in
2056+0 records out
1052672 bytes (1.1 MB) copied, 0.159965 s, 6.6 MB/s

You should also see the following in /var/log/messages though some values will obviously vary:

Aug 23 14:37:15 [comp name] kernel: [17415.953941] Program dd tried to access /dev/mem between 101000->101200.

So there has be a way around this, right? Checking the Redhat Crash Utility listserv yielded some good advice. There are three courses of action proposed:

(1) Rebuild your kernel without the CONFIG_STRICT_DEVMEM restriction.
(2) Port the Fedora /dev/crash driver (./drivers/char/crash.c) to your kernel.
(3) Write a kretprobe module that tinkers with the return value of the
kernel's devmem_is_allowed() function such that it always returns 1.

I don't want to recompile the kernel since I'll loose whatever is currently in memory, so I'll focus on (2). Since I am currently using Ubuntu instead of Fedora, I knew I would have to port the code over. So I found a copy of crash.c and crash.h and set to work. You can find the ported crash driver here as well as a Makefile.

Now, I take NO responsibility for what may happen to your machine if something goes wrong during installation. This is for a 32bit system, and I have only tested this on Ubuntu Ibex kernel 2.6.27-14-generic. I still need to do some testing and will probably have more to say about that later... That being said, we'll continue.

Grab the tar file from above and extract:

#tar -xvzf crash_driver_ubuntu.tgz
crash_driver/
crash_driver/crash.h
crash_driver/Makefile
crash_driver/crash.c

Go inside the newly created folder and compile the kernel module:

# cd crash_driver/
# ls
crash.c crash.h Makefile

# make
make -C /lib/modules/2.6.27-14-generic/build M=/home/levy/crash/crash_driver modules
make[1]: Entering directory `/usr/src/linux-headers-2.6.27-14-generic'
CC [M] /home/levy/crash/crash_driver/crash.o
Building modules, stage 2.
MODPOST 1 modules
CC /home/levy/crash/crash_driver/crash.mod.o
LD [M] /home/levy/crash/crash_driver/crash.ko
make[1]: Leaving directory `/usr/src/linux-headers-2.6.27-14-generic'

At this point you should have the following files:

# ls
crash.c crash.h crash.ko crash.mod.c crash.mod.o crash.o Makefile Module.markers modules.order Module.symvers

The file of interest is the crash.ko kernel module. We will load this into the kernel and check that it is installed correctly:

# insmod crash.ko

# lsmod |grep crash
crash 10368 0

# ls -l /dev/crash
crw-rw---- 1 root root 10, 59 2009-08-23 15:04 /dev/crash

# tail -n 1 /var/log/messages
Aug 23 15:04:10 [comp name] kernel: [19030.855920] crash memory driver: version 1.0

So now we have a new device we can use to access memory: /dev/crash

# dd if=/dev/crash of=crash.dd
dd: reading `/dev/crash': Bad address
6812680+0 records in
6812680+0 records out
3488092160 bytes (3.5 GB) copied, 157.964 s, 22.1 MB/s

I'm not yet sure what the "Bad address" error means, but I suspect it is because dd tried to read beyond the 3.3 GB of memory that I have available.

You can remove the crash.ko module like so when you are finished:

# rmmod crash

Now let's test the newly obtained memory dump to see if it works. I'm going to use the RH Crash Utility with the volatile patch which you can find here:

# ./crash -f /boot/System.map-2.6.27-14-generic /usr/src/linux-source-2.6.27/vmlinux crash.dd --volatile

crash 4.0-8.9
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...

SYSTEM MAP: /boot/System.map-2.6.27-14-generic
DEBUG KERNEL: /usr/src/linux-source-2.6.27/vmlinux (2.6.27.18)
DUMPFILE: crash.dd
CPUS: 2
DATE: Mon Aug 23 12:31:54 2009
UPTIME: 02:44:55
LOAD AVERAGE: 0.10, 0.17, 0.17
TASKS: 252
NODENAME: --
RELEASE: 2.6.27-14-generic
VERSION: #1 SMP Tue Aug 18 16:25:45 UTC 2009
MACHINE: i686 (1994 Mhz)
MEMORY: 3.2 GB
PID: 0
COMMAND: "swapper"
TASK: c0471340 (1 of 2) [THREAD_INFO: c04aa000]
CPU: 0
STATE: TASK_RUNNING

crash>

So far so good :-)

crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
0 0 0 c0471340 RU 0.0 0 0 [swapper]
> 0 0 1 f744e480 RU 0.0 0 0 [swapper]
1 0 0 f7448000 IN 0.1 3056 1900 init
2 0 1 f7448c90 IN 0.0 0 0 [kthreadd]
3 2 0 f7449920 IN 0.0 0 0 [migration/0]
4 2 0 f744a5b0 IN 0.0 0 0 [ksoftirqd/0]
5 2 0 f744b240 IN 0.0 0 0 [watchdog/0]
6 2 1 f744bed0 IN 0.0 0 0 [migration/1]
7 2 1 f744cb60 IN 0.0 0 0 [ksoftirqd/1]
8 2 1 f744d7f0 IN 0.0 0 0 [watchdog/1]
9 2 0 f744f110 IN 0.0 0 0 [events/0]
10 2 1 f7460000 IN 0.0 0 0 [events/1]
11 2 0 f7460c90 IN 0.0 0 0 [khelper]

[snip]

crash> foreach files
PID: 0 TASK: c0471340 CPU: 0 COMMAND: "swapper"
ROOT: / CWD: /
No open files

PID: 0 TASK: f744e480 CPU: 1 COMMAND: "swapper"
ROOT: / CWD: /
No open files

PID: 1 TASK: f7448000 CPU: 0 COMMAND: "init"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
0 f69c8300 f700c550 f695a3e0 CHR /dev/console
1 f69c8300 f700c550 f695a3e0 CHR /dev/console
2 f69c8300 f700c550 f695a3e0 CHR /dev/console
3 f69c8f00 f720d770 f7243c38 FIFO
4 f69c8780 f720d770 f7243c38 FIFO
5 f69c86c0 f7218990 f7045228 SOCK
6 f69c8b40 f70216e8 f70b6000 DIR inotify

PID: 2 TASK: f7448c90 CPU: 1 COMMAND: "kthreadd"
ROOT: / CWD: /
No open files

PID: 3 TASK: f7449920 CPU: 0 COMMAND: "migration/0"

[snip]

crash> foreach net
foreach: WARNING: net command requires -s or -S option

PID: 0 TASK: c0471340 CPU: 0 COMMAND: "swapper"
No open sockets.

PID: 0 TASK: f744e480 CPU: 1 COMMAND: "swapper"
No open sockets.

PID: 1 TASK: f7448000 CPU: 0 COMMAND: "init"
FD SOCKET SOCK FAMILY:TYPE SOURCE-PORT DESTINATION-PORT
5 f7045200 f6bfe380 UNIX:DGRAM

[snip]

In order to compile the kernel so that you can use the RedHat Crash Utility, you can follow the Ubuntu tutorial. After installing the appropriate packages you may have to run the following command:

sudo apt-get build-dep linux

You should end up with a linux-source*.tar.bz2 file under /usr/src . You should also have at least one folder with the kernel headers for your current kernel. You can extract the .tar.bz2 file as so:

# tar -xjf linux-source*.tar.bz2

Go into the resulting folder and set following flag in the Makefile:

CFLAGS_KERNEL = -g

Copy the .config file from your /usr/src/linux-headers-$(uname -r) folder into the /usr/src/linux-source-$(uname -r) folder.

Now type make. After the kernel is finished compiling, you should end up with a vmlinux file. This is the uncompressed kernel image with the debugging information that you need in order to run the RH crash utility.

For more information on the RH Crash Utility, check out:
Official RH Crash Utility Website
Linux Memory Forensics by A. Walters, M. Cohen and D. Collett.
slides from CEIC