Author Archives: Hal Pomeranz

Independent IT/InfoSec Consultant, SANS Institute Faculty Fellow

Mounting Images Using Alternate Superblocks (Follow-Up)

Comments Off
Posted by Hal Pomeranz on October 5, 2009 – 2:16 pm
Filed under Computer Forensics, Evidence Acquisition, Evidence Analysis

Hal Pomeranz, Deer Run Associates

Several months ago, I blogged about using alternate superblocks to fake out the ext3 drivers so you could mount file system images read-only, even if they were needing journal recovery.  However, due to recent changes in the ext file system driver the method I describe in my posting is no longer sufficient.  Happily, there’s a quick work-around.

Let’s try the solution from the end of my previous posting under a more recent Linux kernel:

# mount -o loop,ro,sb=131072 dev_sda2.dd /mnt
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
 missing codepage or helper program, or other error
 In some cases useful info is found in syslog - try
 dmesg | tail  or so

This looks like the original error output we got when using the primary superblock.  Looking at the relevant dmesg output we see something different, however:

[163135.527484] JBD: Ignoring recovery information on journal
[163135.795917] Buffer I/O error on device loop0, logical block 0
[163135.795931] lost page write due to I/O error on loop0
[163135.795944] Buffer I/O error on device loop0, logical block 1
[163135.795949] lost page write due to I/O error on loop0
[163135.795958] Buffer I/O error on device loop0, logical block 2
[163135.795963] lost page write due to I/O error on loop0
[163135.795973] Buffer I/O error on device loop0, logical block 3
[163135.795977] lost page write due to I/O error on loop0
[163135.795986] Buffer I/O error on device loop0, logical block 18
[163135.795991] lost page write due to I/O error on loop0
[163135.795999] Buffer I/O error on device loop0, logical block 32
[163135.796034] lost page write due to I/O error on loop0
[163135.796232] Buffer I/O error on device loop0, logical block 73
[163135.796238] lost page write due to I/O error on loop0
[163135.796248] Buffer I/O error on device loop0, logical block 74
[163135.796253] lost page write due to I/O error on loop0
[163135.796261] Buffer I/O error on device loop0, logical block 94
[163135.796267] lost page write due to I/O error on loop0
[163135.796275] Buffer I/O error on device loop0, logical block 96
[163135.796280] lost page write due to I/O error on loop0
[163135.796516] JBD: recovery failed
[163135.796520] EXT3-fs: error loading journal.

It would appear that even though we’re using an alternate superblock that’s marked as not requiring journal recovery, the ext file system driver is still finding the uncompleted journal entries and trying to apply them.  This is arguably “more correct” behavior than the old driver used, but it doesn’t help us very much.

The simple work-around is to tell the ext file system driver to ignore the journal by forcing the file system to be mounted as ext2:

# mount -t ext2 -o loop,ro,sb=131072 dev_sda2.dd /mnt
# ls /mnt
bin   dev  home    lib         mnt  proc  sbin  usr
boot  etc  initrd  lost+found  opt  root  tmp   var

Excellent!  With this small modification our trick is working again.  Hurrah!

You might well wonder what happens if you just try to mount our image as ext2 without using the alternate superblock.  Unfortunately, simply mounting as ext2 is not sufficient because the primary superblock is still marked as needing journal recovery.  Though I wonder why this flag should be relevant to an ext2 file system, it’s enough to prevent the mount from happening.  So the result is that you need to both mount using an alternate superblock and (at least on modern Linux kernels) mount the file system as ext2 to stop the file system driver from looking at the journal.

Hal Pomeranz is an independent IT/Computer Security consultant and a SANS Faculty Fellow.  He actually discovered this problem when attempting to give a live demo in the middle of a class.  Unfortunately, the solution only occurred to him after class was concluded.  This is one of the reasons why being a SANS Instructor can be so… invigorating.

Tricking the “script” Command

Comments Off
Posted by Hal Pomeranz on July 29, 2009 – 12:29 pm
Filed under Computer Forensics, Evidence Acquisition, Incident Response, Linux IR

Keeping a record of all of the commands you type as well as their output is obviously useful during a forensic investigation.  On Unix and Linux systems, we have the “script” command which does precisely this.  You run “script <filename>” and the script command spawns a new shell: everything you type and all output you receive in return is automatically captured to the specified file.

From a forensic perspective, however, the classic problem is that script insists on writing its output to a file in the local file system.  This is particularly a problem during the initial stages of incident response when you’re operating on a live system trying to verify whether or not it has been compromised.  If you capture your session with the script command, you may be trampling important data as your output file grows.  Of course you could attach a portable storage device and write your output there, but that could be problematic on many levels.

This topic came up recently when Ed Skoudis and I were working on an article for our Command Line Kung Fu blog (a useful resource for Incident Response professionals, since we provide helpful command line short-cuts using both the Windows and Unix command shells).  During our conversation, I realized that there might be a work-around that allows the script command to send its output over the network.  Sure enough, after a little bit of hacking I came up with a devious little method to accomplish exactly this.

Suppose we had another machine on the same network as the host we are investigating.  Let’s suppose this other machine has IP address 192.168.100.1 and there’s a netcat process listening on port 9999 and redirecting its output to some file (”nc -l 9999 >myoutput”).  Now, on the machine you’re investigating, run the following commands:

$ mkfifo /tmp/fifo
$ cat /tmp/fifo >/dev/tcp/192.168.100.1/9999 &
[1] 1523
$ script -f /tmp/fifo
Script started, file is /tmp/fifo
$

Some explanation is clearly in order here:

  • The mkfifo command creates a special type of object in the file system called a FIFO (short for “first in, first out”).  The FIFO allows one command to write data into this “file” and another command to read data out of it.  This is perfect for our purposes.
  • The next command uses the cat to read data out of the FIFO and redirect that output into /dev/tcp/192.168.100.1/9999.  This “file name” is a specially recognized syntax in the bash shell that means “write the data over the network to host 192.168.100.1 on port 9999/tcp” (for more information see Ed’s excellent “Netcat Without Netcat” presentation).  We use “&” to put this command in the background where it will sit and wait for us to start putting data into the FIFO.
  • Finally we fire up the script command and tell it to write its data into the FIFO.  We also add the “-f” (”flush”) argument so that the script command doesn’t buffer its output– the remote side will see the commands being typed and get the output instantly.  We’re now in the subshell spawned by the script command and anything that happens from here on out should start appearing in the output file on the remote system.

The “/dev/tcp/…” syntax is a useful little bit of bash shell trickery.  But even if you didn’t have this, you could do something similar if your incident response kit included netcat.  Just change the second command in the output above to “cat /tmp/fifo | nc 192.168.100.1 9999″.

The real trick here is obviously knowing how to create and manipulate FIFOs, and for whatever reason this doesn’t seem to be something that’s commonly covered in most Unix classes.  But you have to admit it’s an impressively useful thing to know for situations such as this when you have a command like script that insists only on writing to a local “file”.

Obviously, on principles of strict forensic soundness it must be admitted that creating the FIFO does alter the file system slightly.  The FIFO itself requires an inode, though no data blocks will be consumed.  Also the contents of the directory the FIFO is placed in will be modified.  However, I will point out that many Unix systems are configured by default to use memory-based file systems for directories such as /tmp or /var/run.  If this is the case on the system you’re investigating, I would recommend creating your FIFO in one of these memory-based file systems to minimize any impact to the file system on disk.

Hope you find this little trick useful in future investigations!

Hal Pomeranz, Deer Run Associates, is an independent IT/Computer Security consultant and a SANS Faculty Fellow.  He spends far too much time hanging out in the sketchy parts of the Unix operating system.

Perl Fu: Email Discovery

1
Posted by Hal Pomeranz on July 6, 2009 – 3:21 pm
Filed under Email Investigations, eDiscovery

Hal Pomeranz, Deer Run Associates

I hope Mike Worman doesn’t hate on me for stealing his “Perl Fu” idea, but I recently have been dealing with a task that is perfect for Perl.  One of my customers is having to do a laborious discovery process through a huge email archive that is in “Unix mailbox format”– meaning large text files with the email messages all concatentated togther.  They need to find any one of a list of relevant keywords in messages stored in these hundreds of gigabytes of large text files and output the entire text of the matching email messages.

Unix mailbox format is a file format that I’ve dealt with a lot, and I’ve written many scripts to parse these kinds of files.  So it probably took me less time to write the script to do this than it’s going to take me to write this blog post.  But I figured this is a task that other readers of the blog might encounter from time to time, so here’s the code:

#!/usr/bin/perl
# mgrep -- match patterns and output messages from Unix mailbox files
# Usage: mgrep [-i] [-f file] [pattern] file1 ...

use strict;
use Getopt::Std;

my %opts = ();
getopts('if:', \%opts);

my $pattern = undef;
if (length($opts{'f'})) {
    open(FILE, "< $opts{'f'}") ||
	die "Can't open pattern file $opts{'f'}: $!\n";
    my @lines = <FILE>;
    close(FILE);
    chomp(@lines);
    $pattern = '(' . join('|', @lines) . ')';
}
else {
    $pattern = shift(@ARGV);
}
$pattern = "(?i)$pattern" if ($opts{'i'});

my $message = undef;
while (<>) {
    if (/^From\s/) {
	print $message if ($message =~ /$pattern/s);
	$message = undef;
    }
    $message .= $_;
}
print $message if ($message =~ /$pattern/s);

The actual meat of the program is the “while (<>) …” loop down in the bottom third of the code.  We spend more code processing arguments and setting up the pattern match than on actually processing the input files.  But here are some notes to help you make sense of what’s happening in the program:

  1. First we “use strict” to have Perl help us enforce good programming practice in our script, like pre-declaring variables with “my” to help prevent typos and other errors.
  2. Then we incorporate the standard Perl command line argument processing library (”use Getopt::Std”) and call getops() to process the command line arguments.  Here we’re specifying that our program accepts both “-i” (case insensitive matching) and “-f” to specify a file name containing a list of patterns to match against.  The “:” after the “f” in the getops() string means that”-f” expects an argument, namely the file name.  Any options that getopts() finds will be stored in the “%opts” array.
  3. Next our “if” block checks to see if the “-f” option was set.  If so, then we attempt to open the specified file name and read in its contents (”die” causes the program to abort if the file can’t be opened).  We use chomp() to remove the newlines from the lines we read in and then we concatenate all of the patterns together to form a pattern string like “(pattern1|pattern2|…)” (”pattern1 or pattern2 or …”).  Note that if “-f” was not set, then we just read the pattern in from the command line like the normal Unix grep program (that’s the “else { … }” block).
  4. Next we check to see if the “-i” (case-insensitive match) option is set.  If so, then we add “(?i)” at the front of our pattern.  In a Perl pattern match, this is one way to express case-insensitive matching.
  5. Now we’re finally ready to start processing our input files.  The “while (<>) { … }” construct is a useful bit of Perl shorthand that emulates the standard Unix command-line processing.  Specifically it means that if there are any remaining command-line arguments, they should be treated as file names and opened sequentially and all lines processed one at a time from each file.  If there are no unused arguments on the command line after our argument processing, then the program should look for its input from the standard input.
  6. Within the body of the loop, we’re processing our input one line at a time.  At the end of the loop we’re simply concatenating the lines we read into the “$message” variable that holds our message text.  “$_” is the magic Perl variable that represents the text of the line we’re currently processing, and “$message .= $_” means “append $_ to the text already in $message”.
  7. Now for the uninitiated, Unix mailbox format is nothing but a large text file with messages concatenated one after the other.  You can recognize the start of each new mail message when you find a line that begins “From<whitespace>“.  Our “if { … }” block at the top of the loop matches this pattern as an indication that we’ve reached the end of one message and are starting in on another.  If the message we’ve collected so far matches the pattern specified by the user then we print the entire contents of the mail message.  Then we empty our “$message” variable and so we can start collecting the next mail message.
  8. After we’ve processed all of our input files, we still need to determine whether or not we should output the last message from the last file we processed.  That’s why there’s one more print statement after the end of the loop.

Whew!  That’s a lot of words for a simple script, but I hope it helps you wrap your head around some of the more obscure bits of Perl syntax and gives you some ideas for writing your own scripts.  By the way, because I chose to use Perl for this task, one of the happy accidents is that we can actually use the Perl regular expression syntax for the patterns we give as input to the program (whether we put them in a file or specify them on the command line).  This is good news because Perl’s pattern matching syntax is much more flexible and expressive than the one used by the regular Unix grep command.

Happy email hacking!

Hal Pomeranz is an independent IT/Computer Security Consultant and a SANS Faculty Fellow.  He is available as a strolling Perl programmer for weddings and bar mitzvahs.

Directory Link Counts and Hidden Directories

1
Posted by Hal Pomeranz on June 19, 2009 – 1:47 pm
Filed under Computer Forensics, Evidence Analysis, Incident Response, Linux IR

by Hal Pomeranz, Deer Run Associates

One of the things I love about teaching at SANS is that the students are smart people and come up with great ideas.  Sometimes these ideas even lead to useful tools, as was the case a few years ago when we were talking about hidden directories in the Digital Forensics section of Sec506.

First, a little background information.  Unix file systems keep track of a “link count” to all objects in the file system.  This “link count” value is the number of different directory entries that all point to the inode associated with the object.  In the case of a regular file, the link count is the number of hard links to that file.

However, Unix file systems don’t let you create hard links to directories, yet the link count on a directory is always at least two, and even increases by one for each sub-directory in that directory.  Why is this so?

  • Any object in the file system must have a directory entry that connects it into the file system.  For example, if you have a directory like “/tmp”, there’s a pointer in the root directory (”/”) that points to the “tmp” directory entry.  So that gives you one link.
  • Every directory contains the “.” link that points back to itself.  So that gives us the minimum value of 2 links per directory.
  • Every subdirectory has a “..” link that points back to its parent, incrementing the link count on the parent directory by one for each subdirectory created.

For one thing, the above behavior is why it’s important to monitor the link count on critical directories in your file system using a file integrity assessment tool like Tripwire, Samhain, or AIDE.  You can detect people adding or deleting directories when the link counts change.

But consider the following output from a compromised system:

# ls -a /foo
.  ..
# ls -dl /foo
drwxr-xr-x    3 root     root         4096 Jun 12 18:46 /foo

Our “/foo” directory is empty except for the normal “.” and “..” links, meaning we’d expect the link count to be 2.  Yet we see from the “ls -l” output that the link count on this directory is listed as 3 (look for the link count in the second column after the permissions flags and before the file ownership).  What’s going on here?

What’s happening is that I’ve used a kernel-level rootkit to hide a subdirectory of “/foo”.  However the rootkit was not able to decrement the link count of the parent directory without causing a file system discrepancy that would show up the next time you fsck-ed the file system.  In fact, I’ve never encountered a kernel-level rootkit that has attempted to mask the parent directory link count in any fashion when it hides a directory.

As my students pointed out, this suggests an obvious heuristic for detecting the presence of hidden directories on a system.  Simply write a tool that traverses the entire file system searching for directories where the number of subdirectories in a given directory is not equal to the “link count minus two”.  While this technique will only tell you that a hidden directory exists and not necessarily give you the name of the hidden directory, it will pinpoint exactly where to start looking once you get a chance to analyze the file system image on a system that doesn’t have the kernel-level rootkit loaded.

In any event, the tool was extremely simple to write.  It’s called “chkdirs” and it’s now part of the chkrootkit distribution.  And it’s all thanks to some smart and interested SANS students.

Hal Pomeranz is an independent IT/Computer Security Consultant and a SANS Faculty Fellow.  He believes that when the student is ready the master will appear… even if that master is one of your students!

When “Redundant” Yields Different Results

Comments Off
Posted by Hal Pomeranz on June 17, 2009 – 12:30 pm
Filed under Evidence Acquisition, Incident Response

by Hal Pomeranz, Deer Run Associates

One question that often comes up with I’m talking about Digital Forensics in SANS Sec506 is, “There are so many ways to get at the same data on a Linux/Unix system, which method should we choose?”  My response is, “All of them.”  And then I show them this little example to explain why.

Let’s take the case of active network connections on the system.  There are all sorts of ways to get at this data, including “lsof” and “netstat”:

# lsof -i :22
# netstat -anp | grep :22
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 172.1.1.136:22          172.1.1.1:39967         ESTABLISHED -

This is definitely a Whiskey-Tango-Foxtrot kind of situation.  First we’re asking “lsof” to show us all current network connections associated with port 22, and it says there’s nothing going on.  However when we run “netstat” to dump the same information and “grep” for the port 22 traffic, we see both a local server process listening on that port as well as an active network connection with another system.  If you look closer at the “netstat” output you’ll see that where the “-p” option should be providing process ID information about these processes in the last column of output, we’re only getting a “-”, indicating that “netstat” is unable to associate a process with these network connections.

What’s happening on this system is that I’ve used a kernel-level rootkit to hide the SSH daemon process in the kernel process table.  Since “lsof” comes at the problem by scanning through the process table, it misses our hidden processes and their open sockets.  On the other hand, “netstat” starts from the list of open sockets and (if requested with “-p”) tries to find related processes in the process table.  Even if it can’t find an associated process, however, it still can output the fact that the network sockets are in use.

This is why it’s so critical to capture data using multiple tools when you’re trying to verify an incident or capture volatile data about the current state of a compromised system.  If we had only relied on “lsof”, we would have missed important information.  In this case, the discrepancy in the output of these two tools tells us a lot about what’s happened on the system.  So make sure you use all the tools in your toolkit.

Hal Pomeranz is an independent Computer Security/IT consultant and SANS Institute Faculty Fellow.  He once developed the perfect kernel-level rootkit, but it was so good at hiding itself that it disappeared.

Missed It By That Much!

Comments Off
Posted by Hal Pomeranz on March 5, 2009 – 8:31 am
Filed under Computer Forensics, Evidence Analysis

Hal Pomeranz, Deer Run Associates

One primitive forensic technique I show my students in my SANS Sec506 class is the tried and true method of using grep to display byte offsets of “strings of interest” found in a disk image.  For example, I have my students go looking for “love” in the file system of the VMware image we use in class:

# grep -abi ' love ' /dev/sda6
452925733:# This is a comment. I love comments.
...

Once you have the byte offsets from grep, all you have to do is divide by the block size of the file system (hint: use fsstat) to get the number of the block that the string resides in.  In the example, /dev/sda6 is a small file system that only uses 1024 byte blocks, so the number of the block where love is hiding is: 452925733 / 1024 = 442310.

However, when we go to dump that block, something interesting occurs:

# dcat /dev/sda6 442310
###
### Begin Red Hat Mailcap
###

audio/mod; /usr/bin/mikmod %s
# play is apparently a security hole
#audio/*; /usr/bin/play %s

image/*; gthumb %s

application/msword; ooffice %s
application/pdf; evince %s
application/postscript ; evince %s

text/html; /usr/bin/htmlview %s ; copiousoutput

Apparently there’s something wrong with our technique, because the string we matched with grep doesn’t appear in the block whose number we calculated from the byte offset!  Let’s try the next block:

# dcat /dev/sda6 442311
# This is a comment. I love comments.

# This file controls what Internet media types are sent to the client for
...

OK, there’s our string– right at the beginning of the next block.  What the heck is going on here?

We’ve just encountered on of the curious side-effects of using a line-oriented tool like grep on binary file system image data.  It’s easier to understand what’s happening if you look at the hex dump output of the original block whose number we calculated from the byte offset:

# dcat -h /dev/sda6 442310
0       23232320 0a232323 20426567 696e2052     ###  .###  Beg in R
16      65642048 6174204d 61696c63 61700a23     ed H at M ailc ap.#
...
272     77202573 203b2063 6f70696f 75736f75     w %s  ; c opio usou
288     74707574 0a000000 00000000 00000000     tput .... .... ....
304     00000000 00000000 00000000 00000000     .... .... .... ....
320     00000000 00000000 00000000 00000000     .... .... .... ....
...

What we’ve got here is a very small file that ends at byte 292 with a final trailing newline (hex 0×0a).  The rest of the slack space in the block is filled with nulls.

However, grep is looking to display the byte offset of matching lines, so from it’s perspective the next “line” starts at byte 293 in our block, right after that trailing newline.  Since there are no newlines in the rest of the block, that “line” continues into the next block and includes our “string of interest”.  Remember that grep is just treating its input as a stream of data, it knows nothing about block boundaries because the lower-level system call interfaces hide those details from grep.

If what I’m telling you is the truth then the original byte offset that was displayed by grep (452925733) should equal 442310 blocks of 1024 bytes each, plus 293 bytes.  You can do the calculation for yourself, but here’s my result (presented in true Unix nerd fashion):

# expr 442310 \* 1024 + 293
452925733

Theory confirmed.

Now don’t get me wrong: the “grep for strings of interest” technique is a useful arrow in your forensics quiver.  You just need to be aware that little bumps in the road like this will occur whenever you’re using tools like grep in ways that they were not originally intended to be used.

Change Controls: Ur Doin It Rong

1
Posted by Hal Pomeranz on February 2, 2009 – 2:30 pm
Filed under Incident Response

by Hal Pomeranz, Deer Run Associates

More details are emerging in the case of Rajendrasinh Makwana, a former consultant at Fannie Mae, who allegedly planted malicious code on Fannie Mae’s servers after he had been terminated.  If the code had not been detected, it apparently would have destroyed data on a large number of Fannie Mae’s servers on January 31st.

There’s been a great deal of hand-wringing over the fact that Makwana continued to have sufficient access after he was terminated to allow him to plant the malicious code.  Well, let’s review the facts as presented by FBI Agent Jessica Nye’s affidavit:

“On October 24, 2008 between 1:00 and 1:30pm, MAKWANA was terminated as an employee of [Fannie Mae]… At approximately 4:45pm, MAKWANA dropped of his badge and laptop to a [Fannie Mae] employee… Access to [Fannie Mae's] computers…[was not terminated] until late in the evening on October 24, 2008.”

Certainly it’s regrettable that Makwana had upwards of 12 hours of access to Fannie Mae’s network after he was notified of his termination.  But, while I’m not condoning this practice, it’s a fact that managing to terminate Makwana’s access the same day he left the building puts Fannie Mae in the upper 90th percentile of most IT organizations I’ve encountered.

The information in the FBI Agent’s affidavit that really made me sit up and take notice was the following:

“On October 29, 2008, SK, [a Fannie Mae] senior Unix engineer, discovered malicious script embedded within a pre-existing, legitimate script… It was only by chance that SK scrolled down to the bottom of the legitimate script to discover the malicious script.”

In other words, Fannie Mae got very, very lucky here.  What I want to know is why Fannie Mae had to trust to luck to detect an attack that (again according to the FBI affidavit), “would have caused millions of dollars of damage and reduced if not shut down operations at [Fannie Mae] for at least a week.”

Where the hell are the controls on Fannie Mae’s change management process?  How is it possible that Makwana was able to modify code that ran on Fannie Mae’s production systems without that modification being detected?  Having a front-end change control process is nearly useless if you don’t have back-end controls to verify that the process is being followed correctly.

Fellow SANS Instructor Randy Marchany is fond of saying, “The time it takes your organization to deal with an incident is equal to the time it takes you to notice the problem plus the time it takes you to react.”  Well in this case, it was only through sheer luck apparently that Fannie Mae’s “time to notice the problem” wasn’t infinite.  That’s unacceptable in a high-performing IT organization.  In fact, there’s research that shows that effective change management and lack of tolerance for unplanned changes may be the key benchmark for the highest-performing IT organizations.  And Fannie Mae clearly isn’t there.

And there’s such an easy fix for this problem: file integrity assessment tools like Tripwire, AIDE, and Samhain.  Yes, there is overhead to using these tools in any IT organization, but it’s easy to justify these tools if the alternative is potentially losing millions of dollars and having your operations offline or degraded for a week.  That’s a catastrophic event that would put many companies out of business.

Many of you reading this blog deal more with the aftermath of failures of IT processes than with establishing those processes themselves.  But I think we’d all agree that the best incident response scenario is one where we don’t have to respond to an incident at all because the incident wasn’t allowed to happen.  As the current economic climate leads to more layoffs and a greater potential for malice by disgruntled former employees, it’s important that IT organizations look to shoring up the controls that will help them defend their infrastructures against possible threats.

Hal Pomeranz is an independent IT/Computer Security Consultant and a SANS Faculty Fellow.  He’s extremely happy to report that he has no financial relationships with Fannie Mae.
–

Recovering Open But Unlinked File Data

Comments Off
Posted by Hal Pomeranz on January 27, 2009 – 10:05 am
Filed under Computer Forensics, Evidence Acquisition, Incident Response, Linux IR

By Hal Pomeranz, Deer Run Associates

If you’ve ever been a Unix system administrator, you may have encountered “open but unlinked” files in the course of your normal duties.  The typical scenario is a user who’s launched a process that creates an unexpectedly large output file which consumes all of the free space in the partition.  In a panic, the user deletes the output file but leaves the process running.  Unfortunately, the operating system is not allowed to reclaim the space until the last process that has the output file open actually exits.  So until the user kills their process, the space is still in use and the file system is full.  But when you as the system administrator logs in to free some space in the partition, you’re unable to find the massive file that’s consuming all of the space with your normal file system tools because the file has been unlinked (deleted) from the file system.  Finding the process that’s holding the file open and killing it would free the space, but that requires some specialized knowledge and trickery which we’ll see a little later.

In an incident, attackers have been known to use open but unlinked files to hide their data.  For example, suppose the attacker were running a packet sniffer that was capturing usernames and passwords off your network and storing it in a file.  Perhaps they have another process that’s reading the data as it’s placed in the file and using some covert channel to move it off system.  At this point the attacker could delete the data file: the packet sniffer would continue writing data to the file and their reader process could continue reading the data because they opened the file before it was removed from the file system, but the system administrator would have trouble locating the file because it’s now unlinked from the file system.  In fact, the attacker can even delete the executables for the packet sniffer and the reader process from the file system and the current processes will continue to run.

This kind of open but unlinked file data can be difficult to recover from a “cold” system image, because the minute the system is shut down and the attacker’s processes are terminated, the data in these files just becomes part of the free block collection and must be recovered like any other deleted file data.  However, if you have the luxury of analyzing the running system, it is extremely easy to spot and recover this kind of file data.

Creating Our Test Case

As a stand-in for our attacker’s hypothetical packet sniffer, I’m going to start a tcpdump process and have it dump its packet captures into a file.  Since I plan on removing the tcpdump binary as part of my demonstration, I’m first going to make a copy of the binary in /tmp:

# cp /usr/sbin/tcpdump /tmp/tcpdump
# /tmp/tcpdump -w /tmp/capture &
[1] 12437
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
# ls -l /tmp/tcpdump /tmp/capture
-rw-r--r-- 1 root root   4096 2009-01-21 18:08 /tmp/capture
-rwxr-xr-x 1 root root 639416 2009-01-21 18:07 /tmp/tcpdump

So far, so good.  Our tcpdump process is running and has captured a little bit of data.  Now I’m going to remove the binary and the output file and verify that the process is still running:

# rm /tmp/tcpdump /tmp/capture
# ls -l /tmp/tcpdump /tmp/capture
ls: cannot access /tmp/tcpdump: No such file or directory
ls: cannot access /tmp/capture: No such file or directory
# ps -ef | grep tcpdump
root     12437 12289  0 18:08 pts/1    00:00:00 /tmp/tcpdump -w /tmp/capture

Great!  The process is still active, but neither the binary nor its output file are visible to standard Unix tools like ls.  Now let’s have some fun.

The Power of lsof

It turns out that the highly indispensible lsof utility has an option for detecting exactly these open but unlinked files:

# lsof +L1
COMMAND     PID USER   FD   TYPE DEVICE     SIZE NLINK   NODE NAME
init          1 root    0u   CHR    5,1              0    623 /dev/console (deleted)
init          1 root    1u   CHR    5,1              0    623 /dev/console (deleted)
init          1 root    2u   CHR    5,1              0    623 /dev/console (deleted)
wineserve 10510  hal   31u   REG  254,1 16777216     0  42233 /tmp/.wine-1000/server-fe05-4ee00e/anonmap.WKvI4J (deleted)
firefox   10826  hal   60u   REG  254,4     8200     0 139466 /var/tmp/etilqs_wkK3U9h1sAcQpD3 (deleted)
tcpdump   12437 root  txt    REG  254,1   639416     0  25463 /tmp/tcpdump (deleted)
tcpdump   12437 root    4w   REG  254,1    65536     0  25500 /tmp/capture (deleted)

The “+L1″ option is translated as “show me all files with link count less than one”– in other words, “show me all files with link count zero” which is just another way of saying “files which have been unlinked from the file system”.  It’s almost anti-climactic how easy it is to spot these files with lsof.

However, the above output also demonstrates another important aspect of this discussion: not all open but unlinked files are necessarily cause for concern.  There are a few processes that are common to the Unix/Linux platform that sometimes make use of open but unlinked files as part of their normal operations.  This is clearly a situation where one must “know thy systems”– be familiar with how the operating systems you’ll be investigating appear during normal operations– so that you can spot discrepancies.  That being said, programs that I’ve seen regularly using open but unlinked files include the Linux init process, Firefox, the Wine Windows emulator (all of which you can see in the output above), and VMware Server.

OK, at this point we’ve spotted the open but unlinked files, but how can we recover the data that’s in them?  The lsof output above gives us the inode numbers for the files if you look in the “NODE” column (25463 for our deleted tcpdump binary and 25500 for the output file).  This means we could use a tool like icat from the Sleuthkit to dump the contents of these files.  But it turns out that there’s another approach that doesn’t require any tools that aren’t already present in most Unix-like operating systems.

/proc to the Rescue!

The lsof output also gives us the process ID of the processes with open but unlinked files.  In this case, our tcpdump process is PID 12437.  Let’s head over to the /proc/12437 directory and see what we can see:

# cd /proc/12437
# ls
attr		 cpuset   io	    mountinfo	oom_score  smaps
auxv		 cwd	  latency   mounts	pagemap    stat
cgroup		 environ  limits    mountstats	root	   statm
clear_refs	 exe	  loginuid  net		sched	   status
cmdline		 fd	  maps	    numa_maps	schedstat  task
coredump_filter  fdinfo   mem	    oom_adj	sessionid  wchan

There’s a lot of interesting data in /proc, but for our purposes the “exe” object is the first interesting thing:  “exe” is always a copy of the binary image of the running process.  So all we have to do to recover the deleted executable is make a copy of this “file” under /proc.  Obviously in a real incident you wouldn’t want to copy the file back into the file system of the compromised machine because you’d potentially be overwriting important evidence.  You’d either want to copy it to a portable drive you’ve connected to the system, or move the executable off the machine via the network with a command line like:

# cat /proc/12437/exe | nc 192.168.1.1 9999

This assumes you’ve got a machine acting as a “collector” at 192.168.1.1 with another netcat process listening on port 9999/tcp and writing the incoming data into a file.

Just to prove to you that this works, however, let me just copy the binary into my local file system and demonstrate that the result is a working executable:

# cp /proc/12437/exe /tmp/testing
# /tmp/testing --version
testing version 3.9.8
libpcap version 0.9.8
Usage: testing [-aAdDeflLnNOpqRStuUvxX] [-c count] [ -C file_size ]
        [ -E algo:secret ] [ -F file ] [ -i interface ] [ -M secret ]
        [ -r file ] [ -s snaplen ] [ -T type ] [ -w file ]
        [ -W filecount ] [ -y datalinktype ] [ -Z user ]
        [ expression ]

Again, in a real incident you wouldn’t just blindly execute a suspicious program you’d recovered in this manner.  You’d only want to experiment with a copy of the executable in an isolated “sandbox” type system.

OK, so what about recovering the data file that the tcpdump process is writing?  If you look at the directory listing of /proc/12437, you’ll notice that there’s an object called /proc/12437/fd.  “fd” here stands for “file descriptor” and /proc/12437/fd is actually a directory that contains links to all files this process currently has open.  Let’s take a look at this directory in more detail:

# ls -l /proc/12437/fd
total 0
lrwx------ 1 root root 64 2009-01-21 18:14 0 -> /dev/pts/1
lrwx------ 1 root root 64 2009-01-21 18:14 1 -> /dev/pts/1
lrwx------ 1 root root 64 2009-01-21 18:12 2 -> /dev/pts/1
lrwx------ 1 root root 64 2009-01-21 18:14 3 -> socket:[4197672]
l-wx------ 1 root root 64 2009-01-21 18:14 4 -> /tmp/capture (deleted)

Notice that the link names are named for the internal file descriptor number used by the process, but the names of the files associated with these file descriptors can be clearly seen, along with the fact that the /tmp/capture file has been deleted.

The cool thing is that you can use these links as arguments to normal Unix file system commands.  For example, if you wanted to copy the data in the deleted file over the network to your capture workstation:

# cat < /proc/12437/fd/4 | nc 192.168.1.1 9999

There’s a little bit of subtlety to the command above.  Notice that I’m not just doing “cat /proc/12437/fd/4″, because that would only give me a snapshot of the contents of the file at a particular instant in time.  Instead I’m using input redirection (”<”) to dump the current state of the file and then continue reading data and dumping it into netcat as it’s being written into the file.  Our cat process will only terminate when the tcpdump file closes its output file.  This is one of the things that makes this approach superior to just using a tool like icat to dump the current state of the file.

Some Final Thoughts

It’s fairly uncommon to find attackers making use of open but unlinked files in this manner.  But if you’re accessing the compromised system anyway– for example to capture a memory dump before you shut the machine down– it doesn’t hurt to run a quick lsof command to check for any instances of these files.  In the cases where they do exist, the techniques described above can allow you to quickly recover some information that is typically critical to analyzing what the attacker is doing to your system.

Also, don’t forget about the /proc/<pid>/cwd link, which is a link to the current working directory of the process.  If the attacker was foolish enough to start the suspicious process from their rootkit installation directory, you may be able to zero-in on the compromise in record time!

Hal Pomeranz is an independent Computer Security/IT consultant and SANS Institute Faculty Fellow.  He wants to name his first child “Elesso’ef Slashproc”, which probably means it’s a good thing he’s not planning on having kids.

A Quick Idiom for Pretty-Printing /proc Data

Comments Off
Posted by Hal Pomeranz on January 22, 2009 – 8:15 am
Filed under Incident Response, Linux IR

This is just a short note about a useful little idiom that a lot of people I run into seem to have never seen before.  You’re all aware that the /proc file system contains a great deal of information about processes that’s useful in an incident response situation.  However, when you start looking at this data it can sometimes be difficult to read:

$ cd /proc/self
$ cat environ
GNOME_KEYRING_SOCKET=/tmp/keyring-r8yNJT/socketLOGNAME=halGDMSESSION=default...

Yuck!  All of the environment variables are jammed together in an unreadable mess.

The reason the output appears this way is that the various strings in the /proc structures use nulls (ASCII zero) instead of newlines as terminal characters (just like strings in C).  You don’t usually see the nulls because they’re non-printable characters.

But with a little help from the “tr” command you can convert the nulls to newlines and make everything much more readable:

$ cat environ | tr \\000 \\n
GNOME_KEYRING_SOCKET=/tmp/keyring-r8yNJT/socket
LOGNAME=hal
GDMSESSION=default
[...]

Notice the use of double backslashes in the command above — the extra backwhack makes sure that the arguments to “tr” end up being \000 and \n after being interpolated by the shell (or you could use single quotes).

I hope you find this little trick useful.  I find myself using it constantly.

Hal Pomeranz is an independent IT/Computer Security consultant and a SANS Faculty Fellow.  He spends far too much of his life herding Unix/Linux systems.

Understanding Indirect Blocks in Unix File Systems

Comments Off
Posted by Hal Pomeranz on December 24, 2008 – 10:37 am
Filed under Computer Forensics, Evidence Analysis

When I’m covering Linux Digital Forensics on the last day of Sec506 (that’s my SANS Linux/Unix Security track for those sluggards out there that haven’t memorized the SANS course numbering scheme), questions always come up about the function of indirect blocks in standard Unix file systems (meaning ext2/ext3 on Linux, UFS on Solaris, FFS on BSD, and others derived, directly or indirectly, from Kirk McKusick’s original Fast File System from 4.2BSD).  Generally these questions always arise in one of two contexts:

  • What’s that extra information at the end of my istat output?
  • Why do I always need the -d option when using foremost on Unix file system images?

It turns out that even people who’ve been using Unix for a long time are a little fuzzy on the subject of indirect blocks, what they are, how they work, and so on.  So let’s talk about indirect blocks, and why you care about them from the perspective of file system forensics.

A Quick Review of Unix File Systems

In a standard Unix file system, files are made up of two different types of objects.  Every file has an index node (inode for short) associated with it that contains the metadata about that file: permissions, ownerships, timestamps, etc.  The contents of the file are stored in a collection of data blocks.  At this point in the discussion, a lot of people just wave their hands and say something like, “And there are pointers in the inode that link to the data blocks.”

As it turns out, there are only fifteen block pointers in the inode.  Assuming standard 4K data blocks, that means that the largest possible file that could be addressed directly would be 60K– obviously not nearly large enough.  In fact, only the first 12 block pointers in the inode are reserved for direct block pointers.  This means you can address files of up to 48K just using the direct pointers in the inode.

Beyond that, you start getting into indirect blocks:

  • The thirteenth pointer is the indirect block pointer.  Once the file grows beyond 48K, the file system grabs a data block and starts using it to store additional block pointers, setting the thirteenth block pointer in the inode to the address of this block.  Block pointers are 4-byte quantities, so the indirect block can store 1024 of them.  That means that the total file size that can be addressed via the indirect block is 4MB (plus the 48K of storage addressed by the direct blocks in the inode).
  • Once the file size grows beyond 4MB + 48KB, the file system starts using doubly indirect blocks.  The fourteenth block pointer points to a data block that contains the addresses of other indirect blocks, which in turn contain the addresses of the actual data blocks that make up the file’s contents.  That means we have up to 1024 indirect blocks that in turn point to up to 1024 data blocks– in other words up to 1M total 4K blocks, or up to 4GB of storage.
  • At this point, you’ve probably figured out that the fifteenth inode pointer is the trebly indirect block pointer.  With three levels of indirect blocks, you can address up to 4TB (+4GB from the doubly indirect pointer, +4M from the indirect block pointer, +48K from the direct block pointers) for a single file.

Here’s a picture to help you visualize what I’m talking about here:

By the way, while 4TB+ seemed like an impossibly large file back in the 1970s, these days people actually want to create files that are significantly larger than 4TB.  Modern file systems deal with this by recognizing that addressing every single block in the file is incredibly wasteful.  Instead the file system allocates collections of consecutive blocks, usually referred to as extents, and then stores pointers to the first block in the extents.  Even with a relatively small extent size, you can index huge files– 64K extents mean you can create a 64TB file!

Why All This Matters for Forensics

While all of this is fascinating in a deeply nerdy sort of way, why am I discussing this on a digital forensics blog?  First, this answers the question of why you always must use the -d option with foremost– without -d foremost won’t follow the indirect block pointers and you’ll never recover more than the first 48K of the files that foremost finds.  I’ve often wondered why the -d behavior isn’t simply the default when foremost detects a Unix file system, and I speculate it’s because the foremost developers are primarily Windows-oriented and are interested in analyzing Windows file systems, which are obviously far more numerous in the general population.

You can also observe indirect blocks using tools from the Sleuthkit (TSK).  Take a look at this snippet of istat output:

# istat /dev/mapper/elk-home 802839
inode: 802839
Allocated
Group: 98
Generation Id: 2610194750
uid / gid: 0 / 0
mode: -rwxrwx---
size: 2145910784
num of links: 1

[...]

Direct Blocks:
8552962 8552963 8552964 8552965 8552966 8552967 8583175 16566928
16566929 16566930 16566931 16566932 16566934 16566935 16566936 16566937
[...]

Indirect Blocks:
16566933 16578347 16578348 16579373 16580398 16581943 16582968 16583993
[...]

As you can see, this is a huge file– 2GB in size, which means we should be using doubly indirect pointers to index all of the data blocks.  Indeed, at the end of the istat output you can see the list of indirect blocks.

What’s perhaps more interesting is that if you look at the list of direct blocks, you’ll notice that there’s a gap in the numbering between the 12th (block number 16566932) and 13th blocks (16566934).  You’ll find block number 16566933 in the list of indirect blocks.  So the file system assigned the first 12 direct blocks to the direct block pointers in the inode, then realized it was going to need an indirect block pointer and just grabbed the next contiguous block to use as the indirect block and then started using the subsequent blocks as data blocks (whose addresses will appear in the indirect block).  This illustrates a couple of important aspects of the Unix file system:

  1. Consecutive data blocks are allocated whenever possible
  2. The file system will just greedily grab the next available block when it needs an indirect block pointer

I’m going to be exploiting these tendencies in just a moment to get some interesting information out of the file system.

But first let’s actually use dcat to dump the contents of one of the indirect blocks:

# dcat -h /dev/mapper/elk-home 16566933
0       96cafc00 97cafc00 98cafc00 99cafc00     .... .... .... ....
16      9acafc00 9bcafc00 9ccafc00 9dcafc00     .... .... .... ....
[...]
4064    23f7fc00 24f7fc00 25f7fc00 26f7fc00     #... $... %... &...
4080    27f7fc00 28f7fc00 29f7fc00 2af7fc00     '... (... )... *...

This is actual output from my Linux AMD_64 laptop, and consequently the block pointers in the indirect block are in little-endian byte order.  That means that the actual address of the first block is 0×00fcca96, or 16566934 in decimal (i.e. the thirteenth block listed in istat’s summary of direct blocks in the file).  You can see the block addresses increasing monatomically from that point on.  Again there’s a sort of geeky coolness factor here, but is there a practical application for this information?  Well, there’s no question in my mind about that.

Consider that when a file is deleted on an ext3 file system, the contents of the inode associated with that file are zeroed before the inode is returned to the free list.  That means you lose all of the block pointers in the inode, and reconstructing the file is extremely difficult in cases where tools like foremost don’t recognize the type of file you’re searching for.  You can use string searching to find particular strings of interest in the middle of a file, and because the file system tends to allocate blocks contiguously you can often find larger chunks of the file by dumping the blocks before and after the block containing your string of interest.  But this process is time consuming and error-prone.

Leveraging Indirect Blocks

However, suppose you could locate one of the indirect blocks for the file– suddenly you have the exact addresses of up to 4MB of data from the file!  But how do we go about tracking down an indirect block and recognizing it once we find it?  Consider that whatever string of interest we locate by searching our file system image is probably in the middle of a collection of at most 1024 blocks addressed by an indirect block pointer.  If the file was created in a single write operation (the most common situation for pretty much every file in the file system, except possibly log files and other incrementally growing files), the indirect block pointer is sitting right at the beginning of that collection of 1024 blocks.  That means you should run into the indirect block pointer before too long if you start searching backwards through the file system from your string of interest.  How will you know when you get there?  Just look for a block where the first four bytes of the block translate to the address of the next block– and where the rest of the block is probably full of monotomically increasing 4-byte hex values, which are the addresses of the next 1023 blocks.  Can’t miss it.

But it gets better:

  • Chances are that the block after the last data block address in your indirect block pointer is yet another indirect block pointing at another 4MB of the deleted file’s original contents (and very often that block is exactly 1025 blocks away from the indirect block you’ve found).
  • You may be able to find what appears to be two indirect blocks back-to-back.  This means you’ve found the beginning of the double indirect block collection and can recover up to 4GB of data from the file just by following the block pointers appropriately.  Extra bonus points for finding the three blocks that start the trebly indirect block collection.

I was interested in playing around with these notions, so I threw together a quick little Perl script called idcat for dumping the contents of an indirect block.  The syntax is similar to TSK’s dcat utility, but it will dump the contents of all of the data blocks stored in the indirect block you specify on the command line (because I was lazy, the program uses dcat to dump the block contents, so you’ll need to have TSK installed).  If you use the -a option, idcat will dump the block addresses, rather than dumping the block contents.  This is just a proof-of-concept– the interested reader is invited to create the a more useful tool that actually heuristically analyzes file systems looking for collections of indirect blocks necessary to reconstruct the entire file.

Hal Pomeranz is an independent IT/Security consultant and SANS Institute Faculty Fellow.  While not a professional forensic analyst, he has been known to grovel through Unix file system internals for fun.  They don’t let Hal out very much.

–