Forensics from the sausage factory

SQLite overflow pages and other loose ends...

2011-07-02T23:10:00.003+01:00

This is the fourth post dealing with the elements making up SQLite databases and complements the previous three:

We will remember from these previous posts that:

The entire database file is divided into equally sized pages - SQLite database files always consist of an exact number of pages
The page size is always a power of two between 512 (2⁹) and 65536 (2¹⁶) bytes
All multibyte integer values are read big endian
The page size for a database file is determined by the 2 byte integer located at an offset of 16 bytes from the beginning of the database file
Pages are numbered beginning from 1, not 0
Therefore to navigate to a particular page when you have a page number you have to calculate the offset from the start of the database using the formula:
offset = (page number-1) x page size

and that the database may have the following possible page types:

An index B-Tree internal node
An index B-Tree leaf node
A table B-Tree internal node
A table B-Tree leaf node
An overflow page
A freelist page
A pointer map page
The locking page

In this post I am going to take a closer look at Overflow pages.

Overflow pages are required when a record within a database requires more space than that available within a cell in one database page. One SQLite database of forensic interest is the Cache.db file maintained by the Apple Safari web browser. One of the tables within this database is entitled cfurl_cache_blob_data which uses the receiver_data field to store the cached item itself (e.g. cached jpgs, gifs, pngs, html et al ) as a BLOB. A BLOB is a Binary Large OBject. These cached objects often require overflow pages and we can demonstrate the mechanics of them by walking through a record within Cache.db.

If you run a file carver across a Cache.db file searching for pictures you are likely to carve out a number corrupt pictures as shown in the example within Figure 1.

Double click to enlarge

Figure 1

It can be seen that this picture starts at File offset 4583317 within Cache.db. By examining the two bytes at offset 16 within this SQLite database we have established that the database page size is 1024 bytes. The record that contains this picture has six fields as shown in Figure 2.

Double click to enlarge

Figure 2

As discussed in my earlier post An analysis of the record structure within SQLite databases the data making up this record is store in a serialised way (the data representing field 1 is immediately followed by the data representing field 2 and so on with no delimiters). It can be seen therefore that a cell storing a record within the cfurl_cache_blob_data table is almost bound to overflow the 1024 byte database page.

In our example our corrupt picture starts at FO 4583317. To calculate the database page it is stored in we divide the offset by the page size 4583317/1024=4475.8955078125 and round up to establish the page number. Our corrupt picture header is in page 4476.

The SQLite.org file format states that overflow pages are chained together using a singly linked list. The first 4 bytes of each overflow page is a big-endian unsigned integer value containing the page number of the next page in the list. The remaining usable database page space is available for record data.

We know that our picture is likely to be stored in a number of overflow pages and we can establish the next page by looking at the first 4 bytes of the page that is in. Using the formula offset = (page number-1) x page size I can calculate that the offset of these 4 bytes at the start of the page is 4475 x 1024=4582400. This offset can be seen in Figure 3.

Double click to enlarge

Figure 3

The 4 bytes in hex are 00 00 11 7D which decoded big endian is 4477. The next page therefore in the linked list is page 4477. The first 4 bytes of this page found at offset 4583424 in hex are 00 00 11 7E which decoded big endian is 4478. The first 4 bytes of this page found at offset 4584448 in hex are 00 00 11 7F which decoded big endian is 4479. The first four bytes of this page found at offset 4585472 are in hex 00 00 00 00. This value signifies the last page in the linked list.

I can be seen in this example that our corrupt picture starts in page 4476 and overflows into pages 4477, 4478 and 4479. Obviously the overflow pages are contiguous in this case, so in theory at least, if I copy the data from the jpg header of the corrupt picture to the jpg footer and edit out the 'corruption' I should end up with a complete picture. The corruption was likely caused by the overflow page values at the start of each page so using a hex editor I can remove these and hey presto:

Essentially what we have done here is start in the middle of the record and work forwards to the end. Because overflow pages are chained together using a linked list this is relatively straightforward.

But what do we do if we want to locate the earlier pages in the record? This is a little more complicated because each overflow page does not contain the page number of the previous page. The Safari cache.db SQLite 3 database is an auto-vacuum database so we could utilise Pointer Map pages to locate the parent page of the page (4476) where our corrupt picture header is stored. You will recall from my previous post that Pointer Map pages store a 5 byte record relating to every page that follows the Pointer Map page. Pointer Map pages found in Safari Cache.db files will have a lot of entries that relate to overflow pages. The 5 byte records are structured with 1 byte indicating a Page Type and then 4 bytes, decoded big endian, referencing the parent page number as follows:

0x01 0x00 0x00 0x00 0x00
This record relates to a B-tree root page which obviously does not have a parent page, hence the page number being indicated as zero.
0x02 0x00 0x00 0x00 0x00
This record relates to a free page, which also does not have a parent page.
0x03 0xVV 0xVV 0xVV 0xVV (where VV indicates a variable)
This record relates to the first page in an overflow chain. The parent page number is the number of the B-Tree page containing the B-Tree cell to which the overflow chain belongs.
0x04 0xVV 0xVV 0xVV 0xVV (where VV indicates a variable)
This record relates to a page that is part of an overflow chain, but not the first page in that chain. The parent page number is the number of the previous page in the overflow chain linked-list.
0x05 0xVV 0xVV 0xVV 0xVV (where VV indicates a variable)
This record relates to a page that is part of a table or index B-Tree structure, and is not an overflow page or root page. The parent page number is the number of the page containing the parent tree node in the B-Tree structure.

We can expect to see a lot of Page Types 0x03 and 0x04. So how do we find the pointer map page? We know that the Pointer Map page may contain up to (database page size/5) records (rounded down if necessary) - in this case 1024/5=204.8 so there are 204 records in each Pointer Map page. The first Pointer Map page is Page 2. This is followed by 204 pages and then another Pointer Map page, page 207, followed by 204 pages and then another Pointer Map page, page 412 and so on. In other words there is a Pointer Map page every 205th page, starting page 2. In our example we know that our corrupt picture header is in database page 4476 and the applicable Pointer Map page is prior to it. To calculate the applicable Pointer Map page number we divide 4476 by 205 = 21.834146341463415, round down to 21 and multiply by 205 and then add 2 which equals 4307. The applicable Pointer Map page for page 4476 of the database is page 4307. Using the formula offset = (page number-1) x page size I can calculate that the offset to this page is 4409344. This page can be seen in Figure 4. Each Page Type flag where it references an Overflow page is bookmarked in green, other Page Types are in blue. The first 5 byte record relates to database page 4308, the second 5 byte record page 4309 and so on. The record for page 4476 is the 169th record on the Pointer Map page (4476-4307).

Double click to enlarge

Figure 4

It can be seen that the 169th record is in hex 04 00 00 11 7B. This record has a flag 0x04 which indicates that this record relates to a page that is part of an overflow chain, but not the first page in that chain. The parent page number is 00 00 11 7B decoded big endian to page 4475. The record for page 4475 is the 168th record on the page 04 00 00 11 7A. This record also indicates that the page is part of an overflow chain but not the first page. The parent page number is 00 00 11 7A decoded big endian to page 4474. The record for page 4474 is the 167th record on the page 03 00 00 11 6E. This record has a flag 0x03 which indicates that this record relates to the first page in an overflow chain. The parent page number is the number of the B-Tree page containing the B-Tree cell to which the overflow chain belongs. The parent page number is 00 00 11 6E decoded big endian to page 4462.

Using the formula offset = (page number-1) x page size I can calculate that the offset to the beginning of this page is 4568064. We can now decode the page header (detailed more fully in the post An analysis of the record structure within SQLite databases ) shown in Figure 5.

Double click to enlarge

Figure 5

Figure 5 shows the page header of the B-tree leaf node page. The first byte 0D is a flag indicating the page is a table B-tree leaf node. The next two bytes 00 00 indicate that there are no free blocks within the page. The next two bytes 00 03 read big endian indicate that there are three cells stored on the page. The next two bytes at offset 5 within the page header 00 EB decoded big endian give a value of 235 which is the byte offset of the first byte of the cell content area relative to the start of the page. The last byte of the eight byte page header 00 is used to indicate the number of fragmented free bytes on the page, in this case there are none. The remaining highlighted three pairs of bytes 02 60, 01 F1 and 00 EB are the cell pointer array for this page. These three values are offsets to the start of each cell when decoded big endian are 608, 497 and 235 respectively. We will focus on the cell at offset 235. At offset 235 we find two Varints representing the Length of Payload and Row ID (see Figure 6).

Double click to enlarge

Figure 6

The varints are B1 66 and 82 11. The calculation needed to decode them follows in Figure 7:

Double click to enlarge

Figure 7

Following the Length of Payload and Row ID are varints representing the Length of the Payload Header and the serial type codes of the entry_ID, response_object, request_object, receiver_data, proto_props and user_info fields respectively as shown in Figure 8:

Double click to enlarge

Figure 8

Figure 8 shows highlighted in blue and green the first three elements of the Cell make up - the Payload Length, the Row ID and the Payload Header. We have already decoded the Payload Length B1 66 and the Row ID 82 11. The next byte h0A denotes the length of the Payload Header which is in this case 10 bytes (including the Payload Header Length byte). It can be seen therefore that the remaining 9 bytes contain the varints 00, 9B 54, 96 3E, B1 4A, 00 and 00. To determine what each varint indicates we have to consult the Serial Type Code chart detailed in the post An analysis of the record structure within SQLite databases . Each Serial Type Code details the type and length of the data in the payload that follows the payload header.

00 This serial type code indicates that the first field is NULL and the content length is 0 bytes. We know that the first field in our record relates to Row ID however the SQLite.org file format states If a database table column is declared as an INTEGER PRIMARY KEY, then it is an alias for the rowid field, which is stored as the table B-Tree key value. Instead of duplicating the integer value in the associated record, the record field associated with the INTEGER PRIMARY KEY column is always set to an SQL NULL.
9B 54 This serial type code has a value of 3540 which is greater than 12 and an even number. The chart indicates therefore that this field is a BLOB (3540-12)/2 bytes in length [1764 bytes]
96 3E This serial type code has a value of 2878 which is greater than 12 and an even number. The chart indicates therefore that this field is a BLOB (2878-12)/2 bytes in length [1433 bytes]
B1 4A This serial type code has a value of 6346 which is greater than 12 and an even number. The chart indicates therefore that this field is a BLOB (6346-12)/2 bytes in length [3167 bytes]
00 This serial type code indicates that the field is NULL
00 This serial type code indicates that the field is NULL

The serial type code for the response_object indicates that this field is a BLOB 1764 bytes in length. The entire database page would not be big enough to store this BLOB and the cell is even less capable. Figure 9 shows each cell highlighted alternately blue and green:

Double click to enlarge

Figure 9

The first blue shaded area is the cell the elements of which are duscussed above. The last four bytes of this cell highlighted in darker blue is the hex value 00 00 11 7A which when decoded big endian gives the value 4474. This is the page number of the first Overflow page for this cell and is consistent with the information found in the Pointer Map page discussed above.

Next Post

Following on from my earlier SQLite blog posts James Crabtree has been kind enough to code a Varint decoder and Alex Caithness of CCL has supplied me with his fully featured SQLite record recovery tool EPILOG. I'll review this software next time. Thanks to James and CCL.

References
http://www.sqlite.org/fileformat.html
http://www.sqlite.org/fileformat2.html

An analysis of the record structure within SQLite databases

2011-05-10T18:11:00.001+01:00

My two previous posts Carving SQLite databases from unallocated clusters and SQLite Pointer Maps pages looked at the structure of SQLite databases as a whole. Information contained in those posts may hopefully facilitate the carving of complete SQLite databases. This post is aimed at examining the potential of carving individual records within an SQLite database but should be read in conjunction with the Carving SQLite databases from unallocated clusters post.

Background
Carving individual SQLite database records in certain circumstances may be more fruitful than carving whole databases. There are in fact a number of applications that do exactly this for some types of SQLite database. For example Firefox 3 History Recovery (FF3HR) is an application written to recover Firefox records. A paper entitled Forensic analysis of the Firefox 3 Internet history and recovery of deleted SQLite records written by Murilo Tito Pereira also deals with the recovery of Firefox records.

SQLite databases can be considered as a mini file system in their own right. Within this file system are areas that are marked as free that may contain deleted data. Record based recovery may help identify records that have been deleted but are still contained within the parent SQLite database. More obviously record based recovery is indicated where only deleted and partially overwritten databases are available. However for record based recovery to be useful the data you wish to recover must be stored within one table within the SQLite database concerned. If it is necessary to query two or more tables to extract useful data record based recovery is probably not going to be appropriate.

Table Record

Figure 1

Figure 1 shows, as viewed with the SQLite Database Browser software, a record within the Google Chrome History file URL table at row ID 608. This record is stored within the Google Chrome History SQLite database within a B-tree table leaf node in an area known as a cell. It can be seen from the column headers that this record consists of an id, a url, a title, a visit_count, a typed_count, a last_visit_time, a hidden flag and lastly a favicon _id. To aid viewing I will repeat the record data below:

ID
608
URL
http://www.sqlite.org/fileformat2.html
title
File Format For SQLite Databases
visit_count
1
typed_count
0
last_visit_time
12949409092779476
hidden
0
favicon_id
46

The urls table is stored within one table B-tree which will consist of a root page and possibly a number of internal and leaf node pages. I have established that the data representing the record detailed above is stored in a cell that exists within a B-tree table leaf node database page.

Cells

Lets recap some of the information dealt with in the earlier post Carving SQLite databases from unallocated clusters. SQLite databases are divided into equal sized pages, the size of which is detailed in two bytes, decoded as a 16 bit integer big endian, at offset 16 of the database file within the database header. Most of an SQLite database consists of B-tree structures consisting of one or more B-tree pages. Each B-tree page has either an 8 or 12 byte page header (depending on whether it is a leaf or internal node).

Figure 2

As can be seen in Figure 2 the cells tend to be at the end of each database page in an area referred to as the cell content area. These cells are used to store the database records, one record per cell. The first cell to be written in a database page is stored at the end of the page and additional cells work back towards the start of the page.

The number of cells and their location within a database page is stored within the B-Tree page header at the following offsets.

Offset 1 2 bytes 16 bit integer read big endian
Byte offset of first block of free space on this page (0 if no free blocks)

Offset 3 2 bytes 16 bit integer read big endian
Number of entries (cells) on the page

Offset 5 2 bytes 16 bit integer read big endian
Byte offset of the first byte of the cell content area relative to the start of the page. If this value is zero, then it should be interpreted as 65536

Figure 3 Page Header and Cell Pointer array

Figure 3 shows the page header of the B-tree leaf node page that contains the record detailed in Figure 1 above. The first byte 0D is a flag indicating the page is a table B-tree leaf node. The next two bytes 00 00 indicate that there are no free blocks within the page. The next two bytes 00 03 read big endian indicate that there are three cells stored on the page. The next two bytes at offset 5 within the page header 0B 8E decoded big endian give a value of 2958 which is the byte offset of the first byte of the cell content area relative to the start of the page. The last byte of the eight byte page header 00 is used to indicate the number of fragmented free bytes on the page, in this case there are none.

The remaining highlighted three pairs of bytes 0C 44, 0B EC and 0B 8E are the cell pointer array for this page. The SQLite.org file format notes helpfully state that the cell pointer array of a b-tree page immediately follows the b-tree page header. Let K be the number of cells on the b-tree. The cell pointer array consists of K 2-byte integer offsets to the cell contents. The cell pointers are arranged in key order with left-most cell (the cell with the smallest key) first and the right-most cell (the cell with the largest key) last. The key value referred to is the row ID. In this case we have three cells and therefore three offsets which when decoded big endian are 3140, 3052 and 2958. These offsets allow us to find the start of each cell, it is worth pointing out that there may be free blocks or fragments between each cell so we can not use the offsets to determine the length of each cell.

The record detailed in Figure 1 is contained within the cell at offset 2958 within the page. We will decode the contents of this cell but first we better look at the make up of a cell.

Figure 4 Cell make up

Figure 4 indicates four areas of interest. The payload is the data forming the record as detailed in this example in Figure 1 and as suggested in the diagram it is stored in a serialized way with all the relevant data concatenated together. The payload header details how we can identify each field within the concatenated data (see the Payload Header section below for details of how this works). The Row ID number and the Payload length are stored using variable length integers known as varints. To successfully decode the Cell and Payload headers we have to be able to decode a varint.

Varint
http://www.sqlite.org/fileformat2.html and http://www.sqlite.org/fileformat.html#varint_format provides some detail in respect to how varints are structured. I will try here to simplify things and provide a few example decodings when we decode the cell relating to the record detailed at figure 1.

Varints are variable length integers between 1 and 9 bytes in length depending on the value stored
They are a static Huffman encoding of 64-bit twos-complement integers that uses less space for small positive values
Where the most significant bit of byte 1 is set this indicates that byte 2 is required, where the most significant bit of byte 2 is set this indicates that byte 3 is required, and so on
Varints are big-endian: bits taken from the earlier byte of the varint are the more significant than bits taken from the later bytes
Seven bits are used from each of the first eight bytes present, and, if present, all eight from the final ninth byte

Figure 5

Figure 5 shows the beginning of the cell at offset 2958 within the page. As shown in figure 4 the first value is the payload length represented by a varint. The first byte is 5B. We have to establish the value of the most significant bit and this can be done by converting the hex 5B to binary:

It can be seen that in this case the most significant bit is zero and therefore not set. This varint is only one byte long and indicates that the payload length is 91 bytes. The payload length is the length in bytes of both the payload header and the payload.

The next byte is 84. Converting this to binary:

The most significant bit here has the value of one and therefore is set. This indicates that this varint includes, at least, the next byte 60 which converted to binary:

It can be seen that the most significant bit is zero and therefore not set. This byte therefore is not followed by another and is the last byte of this varint. To establish the value of this varint we now have to take the least significant 7 bits of each of the two contributing bytes and concatenate them together:

00001001100000

We discard the leading zeros and convert the binary 1001100000 to decimal, giving a value of 608. This varint represents the row ID and we can see in figure 1 that the row ID is confirmed as 608.

The calculation we have carried out can be represented by a formula. If we say that the value of the varint is N and the unsigned integer value of the first byte is x and the unsigned integer value of the second byte is y we can use the formula:

N = (x-128) x 128 + y

If we substitute the value of our unsigned integers 132 and 96 into the formula:

(132-128) x 128 + 96 = 608

This formula works for two byte varints that can represent a maximum value of 16383. I suspect we are not likely to encounter larger varints in the SQLite databases we have an interest in with the possible exception of SQLite databases used to store browser cache. It is also worth noting that the most significant bit if included and allowed to contribute to the unsigned integer value would have a value of 128 (hence the [x-128]). Therefore if the first byte of a varint is less than 128 you can exclude the possibility of there being a second byte in the varint.

Payload Header and Payload

We have already looked at two of the four areas of interest within a cell, the payload length and row ID. Next up is the Payload Header and Payload.

Figure 6 Payload Header make up

Figure 6 shows the make up of the payload header of a record within the URLs table of the Google Chrome History SQLite database. The payload is the data forming the record stored in a serialized way with all the relevant data concatenated together. The payload header details how we can identify each field within the concatenated data and will vary from table to table, the contents of which is dictated by the fields required in each record. All payload headers will have however a Payload Header Length followed by one or more Serial Type Codes. The Serial Type Code is used to denote the type of data found in a field within the payload and it's length. All possible Serial Type Codes are varints and are detailed in a chart provided by SQLite.org at Figure 7:

Figure 7

Lets have a look at the Payload Header of our example record detailed in Figure 1.

Figure 8

Figure 8 shows highlighted in blue and green the first three elements of the Cell make up shown in Figure 4 - the Payload Length, the Row ID and the Payload Header. We have already decoded the Payload Length 5B and the Row ID 84 60. The next byte h09 denotes the length of the Payload Header which is in this case 9 bytes (including the Payload Header Length byte). It can be seen therefore that the remaining 8 bytes shown in hex are 00, 59, 4D, 01, 01, 06, 01 and 01. These bytes represent varints so we have to consider that a value may be represented by more than one byte, however in this case the unsigned integer value of each byte is less than 128. We can conclude therefore that each varint is only a single byte in length. To determine what each varint indicates we have to consult the Serial Type Code chart shown at figure 7. Each Serial Type Code details the type and length of the data in the payload that follows the payload header. The multi byte integers are decoded big endian.

00 This serial type code indicates that the first field is NULL and the content length is 0 bytes. We know that the first field in our record relates to Row ID (see figure 1) however the SQLite.org file format states If a database table column is declared as an INTEGER PRIMARY KEY, then it is an alias for the rowid field, which is stored as the table B-Tree key value. Instead of duplicating the integer value in the associated record, the record field associated with the INTEGER PRIMARY KEY column is always set to an SQL NULL.
59 This serial type code has a value of 89 which is greater than 13 and an odd number. The chart indicates therefore that this field is a text string (89-13)/2 bytes in length [38 bytes]
4D This serial type code has a value of 77 which is greater than 13 and an odd number. The chart indicates therefore that this field is a text string (77-13)/2 bytes in length [32 bytes]
01 This serial type code has a value of 1 indicating the next field is an 8 bit integer using 1 byte
01 This serial type code has a value of 1 indicating the next field is an 8 bit integer using 1 byte
06 This serial type code has a value of 6 indicating the next field is an 64 bit integer using 8 bytes
01 This serial type code has a value of 1 indicating the next field is an 8 bit integer using 1 byte
01 This serial type code has a value of 1 indicating the next field is an 8 bit integer using 1 byte

It can be seen that our payload is 82 bytes in length (38+32+1+1+8+1+1). The payload header was 9 bytes and therefore the overall payload length is 91 (82+9) bytes, as previously calculated, and represented by the byte 5B.

Figure 9

Figure 9 shows each element of the payload highlighted alternately in green and blue. The first element is http://www.sqlite.org/fileformat2.html 38 bytes in length, the next element is File Format For SQLite Databases 32 bytes in length. The next element is represented by the byte 01 which denotes the visit_count of 1. This is followed by the byte 00 denoting the typed_count of 0. Next are the eight bytes 00 2E 01 6B 41 06 BD D4 decoded as a 64 bit integer big endian giving a value of 12949409092779476, the last_visit_time (stored in the Google format). The next byte is 00, the hidden flag, followed lastly by 2E decoded as 46, the favicon_ID. The next record in this case immediately follows at offset 3052 within the database page.

Notes
I have glossed over some possible combinations of data found in stored records in order to try and simplify things a little. It is possible for a record to require more space than the space available in a cell within one database page. In this eventuality pages known as overflow pages come into play. I will leave any commentary on this to another day :-)

Carving Considerations
It can be seen that each record of the Google Chrome History URL table may vary in content and length. This precludes simple carving of records using known headers. It may be possible to define a scheme to assist with carving however by focussing on the parameters of each element of the record. It is clear that for the Google Chrome History URL table the scheme would be fairly complicated, allowing for very large URLs and Page titles which may well induce many false positives. For databases using a simpler record structure things are a bit easier. A presentation presented by Alex Caithness of CCL details an approach that can be adopted for carving iPhone calls.db databases.

Deleted Data within Live Databases
This area will require another blog post on another day! I am aware of two programs possibly written to recover this deleted data. SQL Undeleter from Chirashi Security and Epilog from CCL. If the developers will let me test these programs out I will report the results to you.

References
http://www.sqlite.org/fileformat.html
http://www.sqlite.org/fileformat2.html
http://www.ccl-forensics.com/images/f3%20presentation3.pdf
http://mobileforensics.wordpress.com/2011/04/30/sqlite-records/

SQLite Pointer Maps pages

2011-05-04T12:09:00.000+01:00

This blog post complements and should be read in conjunction with the previous post Carving SQLite databases from unallocated clusters. In that post I looked at the information available within an SQLite database that may assist in carving one from unallocated clusters.

You will remember from the earlier blog post that SQLite databases are divided into equally sized pages and SQLite database files always consist of an exact number of pages. The page size for a database file is determined by the 2 byte integer located at an offset of 16 bytes from the beginning of the database file. The first page in an SQLite database is numbered page 1.

Auto-vacuum capable
Auto-vacuum capable SQLite databases make use of Pointer Map pages along with the other page types detailed in the earlier blog post. It is probably helpful to provide some information about what an auto-vacuum capable database is.

In a non-auto-vacuum-capable SQLite database when information is deleted the pages where it was stored are added to a list of free pages and these pages can be reused the next time data is inserted. Therefore, should data be deleted the file size of the database does not decrease. If a lot of data is deleted and it becomes necessary to shrink the database size the SQL VACUUM command can be run. This has the effect of reorganising the database from scratch and removing any free pages completely, thus making the database smaller.

When Auto-vacuum is enabled all free pages are moved to the end of the database file and the database file is truncated to remove the free pages at every transaction commit, thus removing free pages automatically.

Pointer Map Pages
The purpose of the Pointer Map is to facilitate moving pages from one position in the database file to another as part of auto vacuum. When a page is moved, the pointer in its parent must be updated to point to the new location. Pointer Maps are used to provide a lookup table to quickly determine what a pages parent page is. They only exist within auto-vacuum capable databases, which require the 32 bit integer value, read big endian, stored at byte offset 52 of the database header to be non-zero.

In auto-vacuum-capable SQLite databases page 2 of the database is always a Pointer Map page. Pointer Map pages store a 5 byte record relating to every page that follows the Pointer Map page. For example if we have an auto-vacuum-capable database that has 24 pages (each of 4096 bytes in size) in total, page 1 will contain the database header and the database schema and the next page, page 2, will be a Pointer Map page. This Pointer Map page will contain a 5 byte record for every one of the remaining 22 pages taking up 110 bytes of space within the page. The first 5 byte record begins at the very beginning of the Pointer Map page and therefore in a 4096 byte page a maximum of 819 (4096/5) records can be stored. If the database has more than 821 pages (when using a page size of 4096 bytes) page 822 would be an additional Pointer Map page that would contain records for the next 819 pages following this second Pointer Map page. Further additional Pointer Map pages can be added in the same way. Pointer Map pages do not store records relating to Pointer Map pages or page 1 of the database.

Pointer Map 5 byte records are structured with 1 byte indicating a Page Type and then 4 bytes, decoded big endian, referencing the parent page number as follows:

0x01 0x00 0x00 0x00 0x00
This record relates to a B-tree root page which obviously does not have a parent page, hence the page number being indicated as zero.
0x02 0x00 0x00 0x00 0x00
This record relates to a free page, which also does not have a parent page.
0x03 0xVV 0xVV 0xVV 0xVV (where VV indicates a variable)
This record relates to the first page in an overflow chain. The parent page number is the number of the B-Tree page containing the B-Tree cell to which the overflow chain belongs.
0x04 0xVV 0xVV 0xVV 0xVV (where VV indicates a variable)
This record relates to a page that is part of an overflow chain, but not the first page in that chain. The parent page number is the number of the previous page in the overflow chain linked-list.
0x05 0xVV 0xVV 0xVV 0xVV (where VV indicates a variable)
This record relates to a page that is part of a table or index B-Tree structure, and is not an overflow page or root page. The parent page number is the number of the page containing the parent tree node in the B-Tree structure.

Figure 1

The screenshot at Figure 1 shows the Pointer Map page of an iPhone SMS.db which uses a page size of 4096 bytes. The Page Type bytes are highlighted in light blue and occur at every fifth byte. The byte highlighted in green (0x00) is the first byte of the sequence that is not one of the page type bytes as described above and therefore indicates that there are no more records stored in this pointer map as can be seen within the highlighted darker blue area.

Extrapolating the database size from the Pointer Map page
If you count the Page Type bytes highlighted within Figure 1 in light blue you will find there are 22. This is because we have 22 pages following the Pointer Map page and therefore require 22 records. This allows us to conclude that there are 24 pages in total within this database (page 1, the Pointer Map page and then the 22 pages). By examining the 2 byte integer located at an offset of 16 bytes from the beginning of the database file we have determined that the page size within this database is 4096 bytes. 24 multiplied by 4096 equals 98304. The file size of this particular database is therefore 98,304 bytes which can also be seen within Figure 1.

Carving Considerations
To carve auto vacuum capable databases the following steps would be needed:

Identify first page of database by detecting the SQLite format 3 header
Establish page size by reading the 2 bytes at offset 16 as a 16 bit integer big endian
Check the 4 byte 32 bit integer at offset 52 for a non zero value indicating that the database is auto vacuum capable
Go to page 2 of the database at Offset page size
If value is 0x01 or 0x02 or 0x03 or 0x04 or 0x05 set a counter to 1
Move five bytes forward and if value is 0x01 or 0x02 or 0x03 or 0x04 or 0x05 increment the counter by 1
or if the value is not 0x01 or 0x02 or 0x03 or 0x04 or 0x05 begin the calculation of database file size using the formula
database size = (counter value + 2) x page size

The above discounts the possibility of their being more than one pointer map page. Some additional logic would be needed to cater for this eventuality. Pointer map pages may contain page size/5 records. If the counter increments to a point where it equals this value it would be necessary to locate the next pointer map page using the formula:

next pointer map page number = (page size/5) + 2 + number of existing pointer map pages.

To calculate the offset to this page use the formula:

offset = (page number-1) x page size.

References
http://www.sqlite.org/fileformat.html
http://www.sqlite.org/fileformat2.html
http://www.sqlite.org/src/artifact/cce1c3360c

Carving SQLite databases from unallocated clusters

2011-04-27T19:57:00.000+01:00

Have you missed me?

Background
Carving SQLite databases from unallocated clusters is problematic because although these types of database have a header, there is no footer and the length of the file is not normally stored within the file either. Given that SQLite databases are used by so many programs now (e.g.Firefox, Google Chrome and numerous Mac OSX and IoS applications) to store data of forensic interest it would be useful to recover them from unallocated clusters. I am aware that there has been some comment in this area within our industry blogs and forums. A paper entitled Forensic analysis of the Firefox 3 Internet history and recovery of deleted SQLite records written by Murilo Tito Pereira became available in 2009 (at a cost) but my interest was piqued more recently when Rasmus Riis (who I believe works for Law Enforcement in Denmark) posted an enscript he had written - Chrome SQlite db finder v1.4 to Guidance Software's enscript download center. Rasmus Riis's approach to carving SQLite files used by Google Chrome is to identify the header and then carry out additional checking throughout the file for known values within page headers. I had mixed results using this enscript and also wondered whether it could be adapted to search for other SQLite databases, in particular the SMS.db used by the iPhone. So as a result I took a closer look at the problem.

SQLite Database Structure and File Format
The SQLite file format is well documented at http://www.sqlite.org/fileformat.html and also at http://www.sqlite.org/fileformat2.html. What I will try to do here is pick out the salient points that may be relevant to carving SQLite files from unallocated clusters, and also provide some commentary where it may be useful.

Size and numbering

The entire database file is divided into equally sized pages - SQLite database files always consist of an exact number of pages
The page size is always a power of two between 512 (2⁹) and 65536 (2¹⁶) bytes
All multibyte integer values are read big endian
The page size for a database file is determined by the 2 byte integer located at an offset of 16 bytes from the beginning of the database file
Pages are numbered beginning from 1, not 0
Therefore to navigate to a particular page when you have a page number you have to calculate the offset from the start of the database using the formula:
offset = (page number-1) x page size

Possible Page Types
Each page is used exclusively to store one of the following:

An index B-Tree internal node
An index B-Tree leaf node
A table B-Tree internal node
A table B-Tree leaf node
An overflow page
A freelist page
A pointer map page
The locking page

However not every database will include all of these items.

The first page (page 1)
The first page of the database is a special page for two reasons; it contains within the first 100 bytes of the page the database header and it also contains the Database Schema (the structure of the database [tables, indexes etc.] described in formal language).

The database header begins with the following 16 byte sequence:

0x53 0x51 0x4c 0x69 0x74 0x65 0x20 0x66 0x6f 0x72 0x6d 0x61 0x74 0x20 0x33 0x00

which when read as UTF-8 encoded text reads SQLite format 3 followed by a nul-terminator byte.

Other significant values at offsets within the database header are as follows:

Offset 16 2 bytes 16 bit integer read big endian
Page Size
Offset 28 4 bytes 32 bit integer read big endian
The logical size of the database in pages which is only populated when the database was last written by SQLite version 3.7.0 or later. This field is only valid if it is nonzero and in all examples of SQLite databases I have examined this value was zero, so unfortunately not as exciting as it first appears!
Offset 32 4 bytes 32 bit integer read big endian
Page number of first freelist trunk page
Offset 36 4 bytes 32 bit integer read big endian
Total number of freelist pages including freelist trunk pages
Offset 52 4 bytes 32 bit integer read big endian
The highest numbered root page number if the database is auto-vacuum capable, for non-auto-vacuum databases, this value is always zero. The majority of the databases we are likely to be interested in are non-auto-vacuum databases.

B-Tree pages

The SQLite.org file format notes helpfully state that:

A large part of any SQLite database file is given over to one or more B-Tree structures. A single B-Tree structure is stored using one or more database pages. Each page contains a single B-Tree node. The pages used to store a single B-Tree structure need not form a contiguous block.

So from a carving perspective we note that most of what we wish to carve are B-Tree pages but they are not necessarily stored contiguously. We can however identify B-Tree pages because they have a page header 8 bytes in length if the page is a leaf node page and 12 bytes in length if the page is an internal node page. In all pages, with the exception of page 1, the header starts at the beginning of the page at offset 0. On page 1 the header starts at offset 100.

The first byte of all B-Tree page headers is a flag field, each flag is detailed as follows:

0x02
flag indicating index B-Tree internal node
0x0A
flag indicating index B-Tree leaf node
0x05
flag indicating table B-Tree internal node
0x0D
flag indicating table B-Tree leaf node

These flags therefore allow us to potentially identify B-Tree pages (of all types) by examining the first byte of each page to see if it is either 0x02, 0x0A, 0x05 or 0x0D.

The B-Tree page header also contains some other potentially useful values (offsets from start of page):

Offset 1 2 bytes 16 bit integer read big endian
Byte offset of first block of free space on this page (0 if no free blocks)
Offset 3 2 bytes 16 bit integer read big endian
Number of entries (cells) on this page
Offset 5 2 bytes 16 bit integer read big endian
Byte offset of the first byte of the cell content area relative to the start of the page. If this value is zero, then it should be interpreted as 65536

Freelist pages

Once again the SQLite.org file format notes help us out:

A database file might contain one or more pages that are not in active use. Unused pages can come about, for example, when information is deleted from the database. Unused pages are stored on the freelist and are reused when additional pages are required.

Each page in the freelist is classified as a freelist trunk page or a freelist leaf page. All trunk pages are linked together into a singly linked list. The first four bytes of each trunk page contain the page number of the next trunk page in the list, formatted as an unsigned big-endian integer. If the trunk page is the last page in the linked list, the first four bytes are set to zero.

The operation of the freelists might be better understood by a quick forensic examination of them:

The four bytes highlighted in blue are at offset 32 within the database header and decoded as a 32 bit integer big endian give a decimal value of 61 - this is the page number of the first freelist trunk page. The four bytes highlighted in green are at offset 36 within the database header and decoded as a 32 bit integer big endian give a decimal value of 70 - this is the total number of free pages including freelist trunk pages.

The page number of the first freelist trunk page is 61. To calculate the offset from the start of the database for this page we use the formula offset = (page number-1) x page size which in this case is (61-1) x 4096 = 245760.

The offset 24570 takes us to start of the first freelist trunk page. There we find an array of 4 byte big endian integers. The first four bytes (highlighted in green) decoded big endian denote the page number of the next freelist trunk page - in this case the value of the first four bytes is zero indicating that there is no more free freelist trunk pages. The second four bytes (highlighted in blue), in this case 0x00 0x00 0x00 0x45 when decoded as a 32 bit integer big endian give a value of decimal 69 - this is the number of leaf page pointers to follow. The remaining 69 blocks of 4 bytes (alternately highlighted in green and blue in the screen shot) each represent, when decoded as a 32 bit integer big endian, the page number of a free page. Examination of each free page revealed that the entire page had been zeroed out - although this may be a function of the application using the database, not a function of SQLite itself.

Pointer map pages
Pointer map pages will only exist if the database is auto-vacuum capable and the value within the 4 bytes of the database header at offset 52 is non zero. In a database with pointer map pages, the first pointer map page is page 2. The first byte of a pointer map page is one of five values 0x01, 0x02, 0x03, 0x04 or 0x05. Many of the databases we have a forensic interest in are not auto-vacuum capable and therefore do not have pointer map pages, however where they do (iPhone SMS.db for example) it is possible to calculate the length of the of the SQLite database by extrapolating information from the pointer map page entries. I will cover this in my next blog post.

Locking Page
The locking page is the database page that starts at byte offset 2³⁰ (1,073,741,824) and always remains unused (i.e all zeros). Most databases will not be big enough (> 1GB) to require a locking page.

Overflow pages
Once again the SQLite.org file format notes help us out:

Sometimes, a database record stored in either an index or table B-Trees is too large to fit entirely within a B-Tree cell. In this case part of the record is stored within the B-Tree cell and the remainder stored on one or more overflow pages. The overflow pages are chained together using a singly linked list. The first 4 bytes of each overflow page is a big-endian unsigned integer value containing the page number of the next page in the list. The remaining usable database page space is available for record data.

We can calculate that for the first byte to be any value other than 0x00 there must be at least 16,777,216 pages within the database (0x01 0x00 0x00 0x00 decoded big endian). At a page size of 4096 bytes this would equate to a database size of 64 Gigabytes. In most cases therefore we can discount the first byte of overflow pages being anything other than 0x00.

So how does this help with carving SQLite databases then?
We can use the database header and the first byte value of each page thereafter to determine whether the data within each page size block is valid. So from a carving perspective we can identify the first page with the database header, calculate the page size, read the next page and validate the first byte and so on until the first byte validation fails.

This essentially is what Rasmus Riis's Chrome SQlite db finder v1.4 enscript does for SQLite databases created by the Google Chrome web browser. His description of the enscripts functionality states that it checks the first byte of each page, other than the first page, for the values 13,10, 2, 5 or 0. Convert these values into hex and you get 0x0D, 0x0A, 0x02, 0x05 and 0x00. The first four of these values are the flags found at the first byte of the B-Tree page headers discussed above. Additionally he checks for 0x00 which may well be the first byte of either a free page or an overflow page. The script does not however allow 0x01, 0x03 or 0x04 to be the first page byte. This therefore does not allow for an auto-vacuum capable database. The databases used by Google Chrome are not auto-vacuum capable, however other databases such as the iPhone SMS.db database are.

Rasmus's enscript also carries out a test of the two bytes at offset 100 within the first database page. The enscript according to the description in the download center looks for the values 2243, 3853 or 3331. The coding however shows that it checks for 3343 or 3853 or 3331 decoded big endian. Converted to hex the first two bytes would be 0x0D 0x0F or 0x0F 0x0D or 0x0D 0x03. I am not sure what Rasmus had in mind for the second value 3853 (which is 0x0D 0x0F converted little endian) but focussing on the first and the third pairs of two bytes the script is taking the flag byte and the first byte of the two bytes used to store the Byte offset of first block of free space on the page. The first page of all SQLite databases beyond the 100 byte database header store the database schema and are therefore constant (in other words because each particular database has a unique set of tables and indexes the first page of a particular database does not change from instance to instance). Because the database schema does not change the combining of the first and second bytes seems to work in order to identify Chrome databases.

This lead me on to consider whether an analysis of the following values within the B-Tree page header of the first page containing the database schema would allow the identification of other types of SQLite databases. The following table appears to suggest that it would be possible to establish a fingerprint for each database type. I have not collected enough test data to be certain about this yet.

With regards to Rasmus's enscript because he was kind enough to share and also not Enpack it is possible to make some small changes to the code to allow it to parse unallocated for all types of SQLite.db. I am in touch with my programming friends to create a script that can carve and identify SQLite databases from the fingerprint discussed above and also include a greater amount of error checking.

Other stuff of note
When you examine data within SQLite databases have you noticed that most of the meaningful stuff is bunched up at the end of each page? The SQLite.org file format notes help us out here:

The total amount of free space on a b-tree page consists of the size of the unallocated region plus the total size of all freeblocks plus the number of fragmented free bytes. SQLite may from time to time reorganize a b-tree page so that there are no freeblocks or fragment bytes, all unused bytes are contained in the unallocated space region, and all cells are packed tightly at the end of the page. This is called "defragmenting" the b-tree page.

To Do
I have not as yet covered the part the journal files may play in the recovery of SQLite data from unallocated, a future post perhaps.

Thanks
to Rasmus Riis for sharing his enscript
and to Chris Vaughan for his help in firming up some of the theory.

References
http://www.sqlite.org/fileformat.html
http://www.sqlite.org/fileformat2.html
http://www.sqlite.org/requirements.html
http://www.sqlite.org/hlr30000.html
https://support.guidancesoftware.com/forum/downloads.php?do=file&id=1116

Reporting and Exporting Emails from Encase

2011-01-04T16:26:00.000Z

Regular readers will know that I champion Encase for most forensic tasks but I have to admit that my favourite forensic tool does not handle the investigation of email very well.

My friend Oliver Smith, over at Cy4or, had cause to run a keyword search across a number of emails. These emails were contained in a number of email containers including dbx and pst files and the Encase email search had been carried out. The emails were reviewable in the records tab. There was a need however for the client to review emails that contained certain keyword hits. Encase provides an export to .msg facility whereby emails can be exported in the .msg format allowing a review using Microsoft Outlook. It is a therefore a simple task to filter the records tab to display only email with search hits (that is those with a value in the Search Hits column). Then by selecting those records you can export them as .msg files.

The problem with this approach is that it is difficult to marry up the exported .msg files to a report detailing each msg files provenance. So in a case where many thousands of emails have been exported it is a real pain to provenance the relevant emails after the client's review. Depending on the email container concerned (e.g. pst, dbx etc.) Encase names the .msg file either by its subject or by some arbitrary description (Inbox.dbx-0.msg, Inbox.dbx-1.msg and so on). In situations where the client has copied notable emails out of the original export directory it can be very difficult to quickly trace the source email container.

To address this problem Oliver has written an enscript to export selected emails to .msg along with a report detailing their provenance.

The html report contains hyperlinks to each message along with the emails provenance and it's metadata. The file name of each exported email marries up to the reference number in the report.

As always if you would like a copy please email me.

Richard Drinkwater

Storage in Forensic Labs

2010-11-08T23:14:00.000Z

As you probably appreciate the Sausage Factory type of computer forensics lab has to store and retain vast quantities of data. In the early days, even in the Sausage Factory, we imaged individual hard drives to individual hard drives. But because of the volume of data and the economics of this methodology we realised that we had to use some form of centralised storage. That was in 2002 and since then we picked up a few tips along the way.

I know of a number of LE labs that have invested large sums (£100k plus) buying their storage area networks. Unfortunately further down the road they could not afford to increase capacity, had maintenance issues, or had other difficulties exacerbated by the shear complexity of their set up. At the other end of the scale I know of sizeable outfits who stick to imaging to hard drives because they believe that they would never acquire the budget to go down the centralised storage route.

I believe there is a middle ground. It is possible to buy 26TB of useable RAID6 storage (32TB raw), a Server and a backup solution for circa £15k. This solution is scalable with further units of 26TB useable storage costing circa £7k each. With a sensible set of operating procedures this type of solution will remain serviceable and fit for purpose for a number of years.

The observant amongst you will have counted nine raid enclosures in the picture. The youngest unit is a Jetstor 516F which when equipped with 16 2TB enterprise class SAS hard drives provides 26TB usable storage and costs less than £10k. The oldest Infortrend unit is over five years old (and does not store production line data any longer). None of these units have ever lost data. They routinely recover from the inevitable hard drive failures. Although these units are not in the same league as EMC et al they are manufactured for the enterprise and in my experience have longevity built in. It is possible to provide similar levels of storage even cheaper with consumer grade equipment but this would probably be a false economy.

All of these units are directly attached (via fibre) to a server. I have found that both Intel and HP manufacture (and support) servers that will probably last forever. Again I look after servers that have not missed a beat in five years.

Although I have found that this type of kit will last I think it is sensible to plan to cycle replacement of primary production line equipment over a three to four year period. Since 2002 I have learnt a lot about this type of kit but have also found that choosing a supplier that will hold your hand when necessary can be particularly useful. In the UK I have found that VSPL understand the needs of LE computer forensic labs and most importantly have always been available to support me when required.

This type of setup, in my experience, has worked well in supporting the production line nature of our forensics work. However a certain way of operating it is required. Which if I had to sum up in two points the first is that storage performance is best alongside processor performance - on the forensic workstation, and secondly if you want data resilience keep two copies of your data (in one form or another) at all times.

Obviously there is a little bit more to it than that. If you are interested in finding out more please let me know,

FTK Imager 3

2010-10-09T10:24:00.000+01:00

FTK Imager has always been the crème de la crème of free forensic tools and now with the introduction of FTK Imager 3 it is even better.

Access Data have added some amazing functionality to this programs already extensive list of capabilities - in fact to steal a phrase - its almost magical and it is certainly available at an unbelievable price. So what am I referring to?

The answer of course is the new image mounting feature which allows a user to mount an image as a drive or physical device. Encase evidence files, Smart image files, Advanced Forensic Format images and dd images are supported. Additionally Encase Logical Evidence Files and Access Data's AD1 custom content images can be mounted logically. Full details in the Release Notes.

This functionality is accessed via File/Image Mounting

In this screen shot I have chosen to mount a drive from a Mac which includes a Bootcamp partition

This resulted in the EFI partition, the HFS+ partition and the NTFS Bootcamp partition all being given a drive letter. The whole drive is allocated the Physical Drive Number 4 in this example.

All of these resources are now available natively upon the machine that FTK Imager 3 is running on. The Physical Disk however is not listed in Disk Management nor does this functionality appear to install any devices within Device Manager.

Logical mounting of non windows partitions (HFS+, EXT3 et al) will present an explorer view of these file systems as FTK imager itself sees them (à la Encase VFS).

This functionality provides many benefits and at first look at least, renders the costly alternatives of PFS/VFS and Mount Image Pro redundant. It also raises the bar in how we can construct virtual machines from images due to the ability to mount more than one drive at once, thus simplifying the creation of multi drive VMs. The functionality also facilitates non techies (lawyers, fraud investigators et al ) to easily peruse images.

FTK Imager 3 also introduces support for VXFS, ex FAT and EXT4 file systems. As we sometimes say in England it's the dogs...

Hiberfil Xpress

2010-09-07T06:14:00.000+01:00

Departing on platform 2 .... I seem to have lost my train of thought ..... ever since I started drafting this post I have had to cope with lyrics of Crosby, Stills and Nash's Marrakesh Express floating around in my brain. OK I know I've lost two thirds of my readership already - Crosby Stills and WHO?

This post, once I've overcome a touch of nostalgia, is about the use of compression by Microsoft in the Hiberfil.sys file. From a forensic point of view this fact can be quite important and I have seen reference to this compression in a few of the other forensics blogs as the result of the work of Matthieu Suiche. I also know that functionality exists in Xways to decompress Hiberfil.sys but until now this functionality was absent in Encase.

The reason Microsoft uses compression is to minimise the footprint of Hiberfil.sys. The compression seeks to reduce Hiberfil.sys to about 75% of physical memory size. The presence of this compression can be identified easily - it exists in chunks typically 16 x 4096 bytes in size, each chunk having a header \x81\x81xpress . Not all hiberfil.sys files utilise this compression.

The reason it matters to us can be demonstrated by looking at a fairly common task for us forensicators; finding traces of Windows Live Messenger conversations. In the worst case scenario, when logging is turned off and the user has not saved their conversation, traces of conversations may only be found in memory (or artefacts of memory created on disk). Hiberfil.sys is used to store the contents of memory when the computer concerned is hibernated and therefore potentially may contain Microsoft Notification Protocol messages relating to WLM conversation. A fairly typical grep keyword used to find these traces is \x20PF= . When run over a hiberfil.sys containing xpress compression results may appear similar to the following screenshot:

It can be seen that the message and the surrounding MSNP is a little garbled. This is because this message is within a xpress compressed block. Decompressing the block and viewing the same message results in:

It can be seen that the MSNP and the message is now in plain text. Until now achieving the decompression for Encase users required the use of another tool but I am pleased to report that after discussing this issue with Guidance Software's Simon Key he wrote an enscript for this purpose. The script can decompress all xpress blocks within hiberfil.sys and write them out to a logical evidence file. Alternatively it will decompress each block in turn and then perform a keyword search against it. Blocks containing search hits are written into a logical evidence file. The script is available at GSI's download center.

Finding traces of MSNP is only one use, you can find index.dat contents, Limewire search terms and many other interesting artefacts in Hiberfil.sys - happy searching!

References
http://www.forensicswiki.org/wiki/Hiberfil.sys
http://msdn.microsoft.com/en-us/library/ee915356(PROT.13).aspx

USN Change Journal

2010-08-06T14:08:00.000+01:00

This post includes

a new method to recover USN Change Journal artefacts from unallocated
some background information
some commentary on benefitting from the existing work of Lance Mueller and Seth Nazzaro

Background
The examination of USN Change Journals is nothing new and was commented on as long ago as September 2008 in Lance Mueller's blog. My interest was piqued more recently when Harlan Carvey discussed the python script written by Seth Nazzaro to parse these Journals.

The update sequence number (USN) change journal provides a persistent log of all changes made to files on the volume. As files, directories, and other NTFS objects are added, deleted, and modified, NTFS enters records into the USN change journal, one for each volume. Each record indicates the type of change and the object changed. New records are appended to the end of the stream. Programs can consult the USN change journal to determine all the modifications made to a set of files. The part of the USN change journal that matters to us is the $USNJRNL•$J file found in the $Extend folder at the root of applicable NTFS volumes. This file is a sparse file which means that only non zero data is allocated and written to disk - from a practical point of view the relevance of this will become obvious in the next section of this post. The capability to create and maintain USN Change Journals exists in all versions of NTFS from version 3.0 onwards. This means that they can exist in all Windows versions from Windows 2000 onwards, however the functionality was only turned on by default in Vista and subsequent Windows versions.

You might be thinking by now - why from an evidential perspective does the USN Change Journal matter? A good question and in many cases with data in a live file system USN Change Journal entries might not assist. However it may be relevant to establish the latest event to occur to a file. The event is recorded by creating a reason code in each record within the journal. These reason codes are detailed in Lance's post and by Microsoft here. Where I think the journal entries may be more useful is in establishing some information about a file that no longer exists in the live file system but is referenced elsewhere.

Lance Mueller's Enscript
Lance's script is designed to parse a live $USNJRNL•$J file and output the parsed records into a CSV file. Like Seth Nazzaro I found that when I tried to run the Enscript Encase hung. This turned out not to be a problem with the script but a problem with how Encase presents sparse files. My $USNJRNL•$J file was recorded as being over 6GB in size. Only the last 32MB (or thereabouts) contained any data, the preceding data was a lot of zeroes -00 00 00 00 ad infinitum. Because the file is a sparse file the zeroed out portion of the file is not actually stored on disk - it is just presented virtually. However it appears that the script needed to parse through the almost 6GB of zeroes before it got to the juicy bits which gave the appearance of the script hanging (or resulting in Encase running out of memory). The solution to this was simple - copy out the non zero data into a file entitled $USNJRNL•$J. Create a folder named $Extend and place your extracted file into it. Drag the folder into a new Encase case as Single Files. Rerun the script which will then process the entries almost instantaneously.

Seth Nazzaro's Python Script
Seth wrote his script because he had difficulty in running the Enscript -possibly for the reasons described above. I have described how to run the script in my earlier Python post. The script is useful in validating results ascertained by other means and particularly for the comprehensive way it parses the reason codes (many record entries contain more than one reason code and the way they amalgamate together can be a bit confusing). The script also outputs to a CSV file.

Recovering USN Change Journal Records from unallocated
Regular readers will know that I am particularly keen in recovering file system and other data from unallocated. I am pleased to see I am not alone. In many cases because of OS installations over the top of the OS where your evidence was created we have no choice but to recover evidence from unallocated.

It is possible to locate large numbers of deleted USN Change Journal Records in unallocated clusters. There is a clear structure to them.

To carve these from unallocated I use my file carver of choice Blade. I have created a Blade data recovery profile which recovered a very large number of records from my test data.

Profile Description: $UsnJrnl·$J records
ModifiedDate: 2010-07-14 08:32:57
Author: Richard Drinkwater
Version: 1.7.10195
Category: NTFS artefacts
Extension: bin
SectorBoundary: False
HeaderSignature: ..\x00\x00\x02\x00\x00\x00
HeaderIgnoreCase: False
HasLandmark: True
LandmarkSignature: \x00\x3c\x00
LandmarkIgnoreCase: False
LandmarkLocation: Static: Byte Offset
LandmarkOffset: 57
HasFooter: False
Reverse: False
FooterIgnoreCase: False
FooterSignature: \x00
BytesToEOF: 1
MaxByteLength: 1024
MinByteLength: 64
HasLengthMarker: True
UseNextFileSigAsEof: False
LengthMarkerRelativeOffset: 0
LengthMarkerSize: UInt32

You may wish to be more discriminatory and carve records relating to just avi and lnk files for example. A small change to the Landmark Signature achieves this.

LandmarkSignature: \x00\x3c\x00[^\x2E]+\x2E\x00[al]\x00[vn]\x00[ik]

The next step is to process the recovered records. Given we already have two separate scripts to do this all we have to do is to present the recovered records to the scripts in a form they recognise. This is achieved by concatenating the recovered records contained within the blade output folders

This can be achieved at the command prompt, folder by folder > copy *.bin $USNJRNL•$J. However if you have recovered a very large number of records and have a considerable number of Blade output folders this can be a bit tedious. To assist with this John Douglas over at QCC wrote me a neat program to automate the concatenation within the Blade output folders (email me if you would like a copy). John's program Concat creates a concatenated file within each output folder in one go. Once you have the concatenated $USNJRNL•$J files you can then run either script against them. Please note the folder structure the enscript requires as referred to above.

Carving individual records in this fashion will result (at least in my test cases) in the recovery of a lot (possibly hundreds of thousands) of records. There will be considerable duplication. Excel 2007 or later will assist with the de-duplication within the scripts output.

Given the potentially large number of records that are recoverable I found it sensible to

run a restricted Blade Recovery profile for just the file types you are interested in (e.g. avi and lnk)
Run John Douglas's concat.exe across Blades output
In Windows 7 use the search facility to locate each concatenated $USNJRNL•$J file
copy them all into one folder allowing Windows to rename the duplicates
at the command prompt use a for loop to process them along the lines of
>for /L %a in (1,1,40) do python UsnJrnl-24NOV09.py -f "UsnJrnl•$J (%a)" output%a -t
or
>for /L %a in (1,1,40) do python UsnJrnl-24NOV09.py -f "UsnJrnl•$J (%a)" -s >> output.tsv
or drag the concatenated files back into Encase as single files and process further with Lance's script.

References
http://www.opensource.apple.com/source/ntfs/ntfs-64/kext/ntfs_usnjrnl.h?txt
http://www.microsoft.com/msj/0999/journal/journal.aspx
http://technet.microsoft.com/en-us/library/cc976808.aspx
http://wapedia.mobi/en/USN_Journal
http://code.google.com/p/parser-usnjrnl/
http://www.forensickb.com/2008/09/enscript-to-parse-usnjrnl.html

Python

2010-07-23T13:41:00.000+01:00

As regular readers will know here in the Sausage Factory our primary forensics tool is Encase. From time to time however we need to try out other tools to validate our results. Recently I wanted to utilise two python scripts widely discussed elsewhere and as a result had to figure out the mechanics of getting these scripts to run on a forensic workstation running Windows 7. I thought I'd share the process with you. Now some of you are highly geeky programmer types who write and run scripts for breakfast - if thats you turn away now. This blog post is in no way definitive and is intended for python newbies wishing to run python scripts in their forensicating but who until now didn't know how.

First off we need to install and configure Python

Download Python - I downloaded Python 2.7 Window X86-64 installer for my Windows 7 64 bit box
Run the installer
Right click on the Computer icon, select properties, select Advanced system settings and click on the Environment Variables button.
In the System Variables pane you will have a variable entitled Path, select it and click on edit
Add to the entries already there ;C:\Python27 (assuming you installed Python 2.7 to the default location)

The two scripts I wanted to run were David Kovar's analyzeMFT and the $USNJRNL parser written by Seth Nazzaro. They are designed to parse MFTs and USN Change Journals respectively which can be copied out of an image or made available via VFS or PDE. More about analyzeMFT can be found at the author's blog. Detailing how I ran these scripts will give a clear indication of how to run these, and many other python scripts, and utilise their output.

analyzeMFT
Download script by visiting http://www.integriography.com/ and right clicking on the Downloaded Here link in the Downloads section (for the source code) and saving the download as a text file. Once downloaded change the file extension to .py.

Save it somewhere and then run IDLE (installed with Python) and open the analyzeMFT.py script. Locate the words noGUI = False and edit to read noGUI = True and save.

To run

open command prompt
at prompt type Python C:\Path_to_the_script\analyzeMFT.py -f U:\Path_to_your_extracted_or_mounted_MFT\$MFT -o $MFT_parsed
The above command runs the script against your extracted or mounted $MFT and outputs the results to a file $MFT_parsed
Open $MFT_parsed using the text import wizard in Excel selecting the text format for each column.

Thanks to David Kovar for making this script available.

$USNJRNL•$J Parser
This script can be downloaded at http://code.google.com/p/parser-usnjrnl/.

To run

open command prompt
at prompt type Python C:\Path_to_the_script\UsnJrnl.py -f U:\Path_to_your_extracted_or_mounted_USNJRNL•$J\USNJRNL•$ -o Output_file -c
The above command runs the script against your extracted or mounted $USNJRNL•$J and outputs the results to Output_file.csv

Notes
Typing at the command prompt Python path_to_script.py wil give some help about a scripts options. For example Python UsnJrnl.py results in the output

Usage: UsnJrnl.py [options]
Options:
-h, --help show this help message and exit
-f INFILENAME, --infile=INFILENAME
input file name
-o OUTFILENAME, --outfile=OUTFILENAME
output file name (no extension)
-c, --csv create Comma-Separated Values Output File
-t, --tsv create Tab-Separated Values Output File
-s, --std write to stdout

I have installed Python 2.7. There are other (and later) versions available including some that are not completely open source. It is also possible to install Python modules to provide a GUI. I have not installed these - takes the fun out of running scripts!

Gatherer Transaction Log Files - a Windows Search artefact

2010-07-19T22:32:00.000+01:00

A recurring theme in many examinations is the prevalence of evidence in unallocated clusters. Reinstallation of the OS is often to blame and a recent case where XP was installed on a drive where the previous OS was Vista further complicated matters. All relevant data had been created during Vista's reign and the challenge was to determine what files and folders existed under this OS. The Encase Recover Folders feature assisted to an extent as did Digital Detective's Hstex 3. Loading the output of Hstex 3 into NetAnalysis allowed me to identify the download of a number of suspect files and some local file access to files within the Downloads folder.

The next step was to carry out a keyword search utilising the suspect file names as keywords. This is always a good technique and results in the identification of useful evidence in a variety of artefacts (e.g. index.dats, link files, registry entries, NTFS file system artefacts et al) but because in this case every thing was unallocated identifying all the artefacts was a little tricky. A considerable number of the search hits were clearly within some structured data but the data was not an artefact I was familiar with.

I have highlighted Record Entry Headers to draw attention to the structured nature of the data. This screen shot is of test data where the file names/path are stored as unicode as opposed to ASCII in the case I was investigating.

A bit of googling led me to page 42 of Forensic Implications of Windows Vista - Barrie Stewart which identified the structured data I had located as being part of Gatherer Transaction Log files created by the search indexer process of Windows Search. These files have a filename in the format SystemIndex.NtfyX.gthr where the X is replaced by a decimal number and on a live Vista system can be found at the path

C:\ProgramData\Microsoft\Search\Data\Applications\Windows\Projects\SystemIndex\

These files have the words Microsoft Search Gatherer Transaction Log. Format Version 4.9 as a file header. The files are a transaction log of entries committed to the Windows search database indexing queues. The SearchIndexer process monitors the USN Change Journal which is part of the NTFS file system used to track changes to a volume. When a change is detected (by the creation of a new file for example) the SearchIndexer is notified and the file (providing it is in an indexable location - mainly User folders) is added to the queue to be indexed. The USN Change Journal is also something that may contain evidentially useful information and I will look at it in more depth in a later blog post.

Sometimes artefacts are only of academic interest but it was fairly apparent that this data could have some evidential value. Each file or folder has a record entry; parts of which had been deconstructed by Stewart. I was able to identify two additional pieces of information within each record - the length of the Filename block and a value that is possibly a sequence or index number or used to denote priority. I also observed some variations in some parts of the record that had been constant in Stewart's test data.

Record Header 0x4D444D44 [4 bytes]
Unknown variable data [12 bytes]
FILETIME Entry [8 bytes]
FILETIME Entry [8 bytes] or a value of 0x[0100]00000000000000
Unknown variable data [12 bytes]
Length of file path following plus 1 byte (or plus 2 bytes if file path stored as unicode) [4 bytes] stored as 32 bit integer
Name and fullpath of file/folder (ASCII or Unicode -version dependant) [variable length]
0x000000000000000000FFFFFFFF [13 bytes]
FILETIME Entry [8 bytes] or a value of 0x[0100]00000000000000
0xFFFFFFFF [4 bytes]
FILETIME Entry [8 bytes] or a value of 0x[0100]00000000000000
Unknown variable data [4 bytes]
Sequence or index number? [1 byte] stored as 8 bit integer
Unknown variable data [15 bytes]
FILETIME Entry [8 bytes] or a value of 0x[0100]00000000000000
FILETIME Entry [8 bytes] or a value of 0x[0100]00000000000000
Unknown variable data [20 bytes]

Microsoft do not seem to have publicly documented the record structure. To establish how useful this data can be I came to the conclusion that I needed to recover all of these records from unallocated. I needed an enscript and Oliver Smith over at Cy4or kindly wrote one for me. I wanted the enscript to parse out the file and path information, sequence or index number, the six time stamps and a hex representation of each unknown range of data into a spreadsheet. The script searches for and parses individual records (from the live systemindex file and unallocated) as opposed to entire files. I was astonished at just how much information the script parsed out - email me if you want a copy. Setting the spreadsheet to use a fixed width font (Courier New) lines up the extracted hex very well should anyone want to reverse engineer these records further. As it stands the file paths and timestamps can provide some useful evidential information, particularly when the recovered records have been recovered from unallocated clusters and relate to a file system older than the current one.

Timestamps
Obviously once you have run this enscript or manually examined the records the first question that arises is what are the timestamps. Establishing this has not been as easy as it could be and hopefully a little bit of crowd sourcing will sort this out for all of us. Post a comment if you can help in this regard. One approach is to use the hex 64 bit filetime value as a keyword and see where you get hits. Hits in another timestamp indicates that the timestamp is the same down to the nanosecond. Carrying out this process will result in hits in OS system files and fragments of them. I have found on the limited test data set I have used that Timestamp 3 matched the File Modified (File Altered) date within MFT for the file concerned and the timestamp for the same file in the USN Change Journal. The timestamp in the USN Change Journal record is the absolute system time that the change journal event was logged (1). It is worth reminding readers who are Encase users that Encase uses different terminology for the time stamps within the MFT - file modified is referred to as Last Written. I think it likely that timestamps 1 and 2 are linked to the indexing function (e.g. time submitted for indexing) given the journalling nature of the file but can not either prove this by testing or confirm this within Microsoft documentation. I can say that in testing sorting on Timestamp 1 gave a clear timeline of the file system activity I had provoked within User accessible folders.

Example CSV output of Enscript (click to enlarge)

References
Good citizenship when developing background services for Windows Vista - Microsoft
Forensic Implications of Windows Vista - Barrie Stewart
Forensic Artefacts Present in Microsoft Windows Desktop Search - John Douglas MSc Thesis
Indexing Process in Windows Search - Microsoft MSDN
(1) http://msdn.microsoft.com/en-us/library/cc232038%28PROT.10%29.aspx

Safari Internet History round up

2010-06-28T21:56:00.000+01:00

The last few posts all concern the recovery of internet history created by the Safari browser. I like to think of internet history in the wider sense and consider any artefact that demonstrates that a user visited a URL at a particular time.

Recovering Safari browser history from unallocated deals with history.
Safari browser cache -examination of Cache.db deals with the cache.
Never mind the cookies lets carve the crumbs - Safari Cookie stuff looks at Cookies.
Safari History - spotlight webhistory artefacts examines Spotlight snapshots of web pages accessed with Safari.

To round things up I will briefly list some other files or locations that may provide internet history created by the Safari browser (the ~ denotes the path is within a user profile)

~/Library/Safari/LastSession.plist
Used to store details of the last browser session allowing a user to select Reopen All Windows from Last Session from the safari history menu.

~/Library/Safari/WebpageIcons.db
~/Library/Safari/Icons.db
Used to store the associations between websites and their favicons.

~/Library/Safari/TopSites.plist
~/Library/PubSub/Feeds/............... .xml
~/Library/Caches/com.apple.Safari/Webpage Previews
TopSites is a gallery of recently visited web sites. The binary TopSites.plist details the websites featured in this gallery. The image representing each webpage is stored within the Webpage Previews folder. This folder also stores any Quicklook representation of a webpage, for example when managing Bookmarks or reviewing History. File names of files in the Webpage Previews folder are the MD5 of the associated URL. Safari monitors whether a page has altered since it was last viewed and appends a blue star to the TopSites view for those sites that have. The xml files in PubSub/Feeds are connected with the monitoring.

~/Library/Safari/Downloads.plist
An xml plist the contents of which are self explanatory.
~/Library/Caches/Metadata/Safari/History/.tracked filenames.plist
A binary plist that may be connected to Safari spotlight web history artefacts.

Never mind the cookies lets carve the crumbs - Safari Cookie stuff

2010-06-22T09:50:00.000+01:00

Safari versions 3, 4 and 5 amalgamates Cookie data into one large file Cookies.plist stored at the path ~/Library/Cookies. This plist is an XML plist. The Encase Internet History search will parse these files and when set to Comprehensive search will find fragments of them in unallocated. However perhaps due to its lack of granularity this search takes forever to run across a Mac and in my experience often fails to complete

As is becoming a recurring theme with my Safari examinations I have turned to Blade to carve out Safari Cookie data from unallocated. The Cookie.plist consists of an array of dictionary objects.

Using Apple's Property List Editor it can be seen that this Cookie.plist has an array of 7074 Dictionary objects. Each Dictionary object is a Cookie in its own right.

Looking at the underlying XML you can see how each dictionary object is structured.

In creating a recovery profile I considered whether I wanted to carve out deleted cookie plists in their entirety or whether I should carve each dictionary object separately. These dictionary objects are fragments of the cookie.plist - hence the crumb reference in the title -after all fragments of cookies are clearly crumbs. I decided that it would be a more thorough search if I carved for the dictionary objects themselves and the following Blade data recovery profile did the business (this data is extracted from Blade's audit log -another neat feature).

Profile Description: Safari Cookie records
ModifiedDate: 2010-06-17 06:33:30
Author: Richard Drinkwater
Version: 1.3.10168
Category: Safari artefacts
Extension: plist
SectorBoundary: False
HeaderSignature: \x3C\x64\x69\x63\x74\x3E\x0A\x09\x09\x3C\x6B\x65\x79\x3E\x43\x72\x65\x61\x74\x65\x64\x3C\x2F\x6B\x65\x79\x3E\x0A\x09\x09\x3C\x72\x65\x61\x6C\x3E
HeaderIgnoreCase: False
HasLandmark: True
LandmarkSignature: <key>Expires</key>
LandmarkIgnoreCase: False
LandmarkLocation: Floating
LandmarkOffset: 0
HasFooter: True
Reverse: False
FooterIgnoreCase: False
FooterSignature: \x3C\x2F\x73\x74\x72\x69\x6E\x67\x3E\x0A\x09\x3C\x2F\x64\x69\x63\x74\x3E\x0A
BytesToEOF: 19
MaxByteLength: 9728
MinByteLength: 200
HasLengthMarker: False
UseNextFileSigAsEof: True
LengthMarkerRelativeOffset: 0
LengthMarkerSize: UInt16

Processing the Carved Files

If your case is anything like mine you will carve out thousands and thousands of individual cookies (or at least the cookie data represented in XML). There are a number of options to process this data further.

Option 1
Drag output into Encase as single files.
Run Encase Comprehensive Internet History search.
View results on records tab.
There are two issues with this method. Firstly Encase does not parse the Cookie created date which is stored as an CFAbsolute timestamp. Secondly there is the issue of duplicates. You will have thousands and thousands of duplicates. These can be managed by hashing the carved files. I would also recommend running the data recovery profile over any live cookie.plists, loading the output into Encase as single files, hashing the output and then creating a hash set. This hash set will allow you to spot additional cookies over and above those in the live cookie plists in any cookies carved from unallocated.
Option 2
Concatenate the contents of each output folder by navigating to the folder at the command prompt and executing the command copy *.plist combined.plist.
With a text editor add the plist header and array tag at the beginning of combined.plist and the closing plist and array tags at the end.
Make sure the formatting of combined.plist looks OK with a text editor.
Process combined.plist with Jake Cunningham's safari cookie plist parser.
The utility is run from the command prompt using a command in the form
>[path to Safari_cookies.exe] [path to combined.plist] > cookies.txt
This parses the plist into the file cookies.txt
This text file may contain many thousands of Cookies. Ideally it would be nicer to port this data into a spreadsheet. To do this I (there is probably a far more elegant way to do this BTW) open cookies.txt in a hex editor (PSPad Hex) and delete all the carriage returns 0D0A. I then find the string Path [50617468] and replace it with 0D0A7C50617468 -in other words preface path with a carriage return and the pipe symbol |. I then find and replace the strings Domain, Name, Created, Expires and Value and replace each in turn with the same string prefaced with | (e.g. |Domain, |Name etc. etc.)
I then use Excel's text import wizard to import the edited cookies.txt setting the delimiter to the pipe symbol | only.
This results in each row relating to one cookie. You can then utilise Excel's very powerful duplicate removal tool.
Both the Mac and Windows versions work OK and the utility converts the CFAbsolute formatted cookie created timestamp.

Safari History - spotlight webhistory artefacts

2010-06-15T10:59:00.000+01:00

June is Safari month here in the Sausage Factory and this post is the third in the series. Just imagine having an observation point in the house across the road from your suspect. When the suspect surfs the internet the man in the OP (with the help of a good pair of binoculars) makes notes of what he reads on screen (OK.. he may use a long lens instead of binoculars and take photos but bear with me). Essentially this is exactly what Spotlight does when a user utilises the Safari web browser (versions 3,4 and 5) to view web pages - it writes the URL, Web Page Title and all the text content in the web page into a file.

These files filenames are in the format URL.webhistory
Their internal structure is that of a binary plist with three strings to each record Full Page Text, Name and URL
They are stored at the path ~/Library/Caches/Metadata/Safari/History
The file created date of these files represents the time that the URL was first visited (since History was last cleared)
The file modified date represents the time that the URL was last visited

It can be seen that it is possible to deduce information from these files that amounts to internet history and therefore it it may be appropriate to consider this data along with records extracted from history.plist and cache.db files.

Recovery from Unallocated
These files are deleted when a user clears Safari history. However it is possible to recover these files from unallocated. Using my file carver of choice - Digital Detective's Blade I wrote an appropriate Data Recovery Profile (which I will happily share with you upon request)

Click on image for larger version

Running this profile resulted in the recovery of over ten thousand files. I then added the recovered files into Encase as single files. I noticed that a small percentage of these files had the text content stored as ascii and not unicode text. I am at this stage not sure why.

Investigation of Live and Recovered Spotlight Webhistory Files using Encase
If you review these files using Encase you will see in the View (bottom) pane the relevant data -the URL is at the start of the file, followed by the text in unicode and then the webpage title near the end of the file. If the content is relevant reporting on it is a pain -potentially three sweeping bookmarks are required using two different text styles. The unicode text sweeping bookmark is also likely to be truncated due its length. Therefore reviewing any number of these files this way is not a good plan.

The eagle eyed amongst you will have observed that in my Blade Data Recovery Profile I gave the recovered files a plist file extension (as opposed to a webhistory file extension). This because these files have a binary plist structure and I use Simon Key's binary Plist Parser v3.5 enscript to parse them. This excellent enscript allows the option to create a logical evidence file which creates a file for each plist name/value pair. I run the enscript with this option, add the logical evidence file back into my case and the review the contents with just a unicode text style selected and bookmark as appropriate. This method is much quicker and removes the need to mess about with unicode formatting. It also makes keyword searching easier. For example to view all URLs green plate (set include) your logical evidence file, apply a sort to the name column in the table pane, scroll down to cause each URL to appear in turn in the view pane. Use a similar method for the Full Page Text and Name items.

Click on image for larger version

Miscellaneous Information in relation to the webhistory file format
Prior to considering the Plist Parser enscript to parse these files I briefly looked at its format with a view to tempting some programming friends to write me a parser. I established that

The file is a binary plist. I do not want to too far into the intricacies of how these plists are assembled. We are interested in objects within the object table. Binary plists use marker bytes to indicate object type and size. The objects we are interested in are strings, either ASCII or unicode. Looking at Apple's release of the binary plist format (scroll about a fifth of the way down the page) it can be seen that the Object Format Marker byte for ASCII strings found in this file is in binary 01011111, followed by an integer count byte. In hex these marker bytes as seen in this file are 5Fh 10h. The Object Format Marker byte for unicode strings found in this file is in binary 01101111, followed by an integer count byte. In hex these marker bytes as seen in this file are 6Fh 11h.
The byte immediately prior to the URL (generally starting http) and after the marker 5Fh 10h decoded as an 8 bit integer denotes the length of the URL. However if the URL is longer than 255 bytes the marker will be 5Fh 11h indicating the following two bytes are used to store the length decoded as 16 bit big endian
Following the URL there is a marker 6Fh 11h - the next two bytes decoded 16 bit big endian is the number of characters of text extracted from the web page - multiply by 2 to calculate the length of the unicode text element of the record
Following the unicode text element is a marker 5Fh 10h -the next byte immediately prior to the webpage title decoded as an 8 bit integer denotes the length of the webpage title
the last four bytes of the file formatted 32 bit big endian is the record size (detailing the number of bytes from the start of the URL to the end of the fifth byte from the end of the file)

Example file format

Click on image for larger version

References
http://opensource.apple.com/source/CF/CF-550/CFBinaryPList.c
http://developer.apple.com/mac/library/documentation/Cocoa/Conceptual/PropertyLists/AboutPropertyLists/AboutPropertyLists.html#//apple_ref/doc/uid/10000048i-CH3-SW2

Safari browser cache - examination of Cache.db

2010-06-08T19:06:00.000+01:00

Following on from my post about Safari browser history I want to touch upon Safari cache. My suspect is running Mac OSX 10.5.6 Leopard and Safari 3.2.1. This version stores browser cache in an sqlite3 database ~/Users/User_Name/Library/Caches/com.apple.Safari/Cache.db. Earlier versions of Version 3 and Version 1 and 2 store cache in a different format, and/or a different place. The Episode 3 Shownotes of the Inside the Core Podcast cover this succinctly so I will not repeat it here but FWIW I have cached Safari artefacts in all three forms on the box I have examined. Currently Netanalysis and Encase do not parse the Safari Cache.db file so another method is required.

Safari Cache.db basics
What follows I believe relates to versions 3, 4 and 5 of Safari running in Mac OSX.
The file contains lots of information including the cached data, requesting URL and timestamps. The file is a Sqlite3 database file which has become a popular format to store cached browser data. The cache.db database contains four tables. For the purposes of this post think of each table as a spreadsheet with column headers (field names) and rows beneath representing individual records.
Two tables are of particular interest:

cfurl_cache_blob_data
cfurl_cache_response

cfurl_cache_blob_data contains one very notable field and a number of slightly less useful ones. The notable field is receiver_data which is used to store the cached item itself (e.g. cached jpgs, gifs, pngs, html et al ) as a BLOB. A BLOB is a Binary Large OBject. Two other fields request_object and response_object contain information relating to the http request/response cycle also stored as a BLOB which when examined further are in fact xml plists. The entry_ID field is the primary key in this table which will allow us to relate the data in this table to data stored in other tables.

cfurl_cache_response contains two notable fields - request_key and time_stamp. The request_key field is used to contain the URL of the cached item. The time_stamp field is used to store the time (UTC) the item was cached. The entry_ID field is the primary key in this table which will allow us to relate the data in this table to data stored in cfurl_cache_blob_data.

In a nutshell cfurl_cache_blob_data contains the cached item and cfurl_cache_response contains metadata about the cached item.

Safari cache.db examination methods
I would like to share three different methods using SQL queries and a few different tools.

Safari cache.db examination methods - contents quick and dirty
Safari cache.db examination methods - metadata quick and dirty
Safari cache.db examination methods - contents and metadata

Safari cache.db examination methods - contents quick and dirty
Depending on what you wish to achieve there are a number of different methods you can adopt. As regular readers will know I work on many IPOC cases. If all you want to do is quickly review the contents of cache.db (as opposed to the associated meta data) I can not recommend any application more highly than File Juicer. This application runs on the Mac platform (which I know is a gotcha for some) and parses out all cached items into a neat folder structure.

I drag the File Juicer output folders into Encase as single files and examine the contents further there. File Juicer is not a forensic tool per se but the developer has at least considered the possibility that it may be used as such. If using a Mac is not an option a Windows app SQL Image Viewer may suffice (with the caveat that I have not actually tested this app).

Safari cache.db examination methods - metadata quick and dirty
Sometimes overlooked is the fact that most caches contain internet history in the form of urls relating to the cached item. The cfurl_cache_response table contains two fields - request_key and time_stamp containing useful metadata. We can use an SQL query to parse data out of these fields. I use (for variety more than anything else) two different tools (i.e. one or the other) to carry out a quick review of meta data.

Method A using Sqlite3 itself (http://www.sqlite.org/download.html scroll down to the Precompiled Binaries for Windows section)
extract your cache.db file into a folder
copy sqlite3.exe into the same folder [to cut down on typing paths etc.]
launch a command prompt and navigate to your chosen folder
Type sqlite3 cache.db
then at the sqlite prompt type .output Cache_metadata.txt [this directs any further output to the file Cache_metadata.txt]
at sqlite prompt type Select time_stamp, request_key from cfurl_cache_response; [don't forget the semi colon]
allow a moment or three for the query to complete the output of it's results
Launch Microsoft Excel and start the Text Import Wizard selecting (step by step) delimited data, set the delimiters to Other | [pipe symbol] and set the Column data format to Text
Click on Finish then OK and bobs your uncle!

Click image to view full size

Method B using SQLite Database Browser as a viewer in Encase
from your Encase case send the Cache.db to SQLite Database Browser
on the Execute SQL tab type in the SQL string field enter Select time_stamp, request_key from cfurl_cache_response
Review results in the Data returned pane
Or
from your Encase case send the Cache.db to SQLite Database Browser
File/Export/Table as CSV file
Select the cfurl_cache_response Table name
Open exported CSV in Excel and adjust time_stamp column formatting (a custom date format is required to display seconds)

Safari cache.db examination methods - contents and metadata
What we need to do here is extract the related data from both tables - in other words be able to view the time stamp, URL and the cached object at the same time. This can be done using SQLite2009 Pro Enterprise Manager. This program has a built in BLOB viewer that will allow you to view the BLOB data in hex and via a image (as in picture) viewer if appropriate.

Once you have launched the program open your extracted Cache.db file
In the Query box type (or copy and paste) all in one go
SELECT cfurl_cache_blob_data.entry_ID,cfurl_cache_blob_data.receiver_data, cfurl_cache_response.request_key,cfurl_cache_response.time_stamp
FROM cfurl_cache_blob_data, cfurl_cache_response
WHERE cfurl_cache_blob_data.entry_ID=cfurl_cache_response.entry_ID
Then key F5 to execute the query
This will populate the results tab with the results
To view the cached object BLOB data in the receiver_data field highlight the record of interest with your mouse (but don't click on BLOB in the receiver_data field). This will populate the hex viewer (bottom left) and the BLOB viewer (bottom right).
To view a full sized version of a cached image click with your mouse on BLOB in the receiver_data field which launches a separate viewing window

Click on image to view full size

References
SQLite Database File Format
Sindro.me weblog - Extracting data from Apple Safari's cache
http://www.devx.com/dbzone/Article/17403/1954
Inside the Core Episode 3 Show Notes
Define relationships between database tables -Techrepublic

Recovering Safari browser history from unallocated

2010-06-06T14:29:00.000+01:00

One of my cases involves the examination of an Apple Mac running Mac OSX 10.5.6 Leopard . The primary web browser in use is Safari version 3.2.1. Typically with Safari I run the Comprehensive Internet History search in Encase but in this case the search would not complete so I had to consider another method to recover and review internet history. Browsing history is stored in a binary plist ~ /Users/User_Name/Library/Safari/History.plist however the live one was empty. I recalled from a much earlier case that you can carve deleted plists from unallocated. I had documented a method for doing this over at www.forensicwiki.com but at the time of writing this resource is still offline.

One of the best file carvers around is Blade and I decided to use it to recover the deleted History.plists. Blade has a number of pre-configured built in Recovery Profiles but there wasn't one for Safari. However one of the neat things about Blade is that you can write your own profiles and share them with others. In conversation I had found out that Craig Wilson had written a Safari history.plist recovery profile which he kindly made available to me (after all why re-invent the wheel). I imported it into my copy of Blade and I was then good to go.

Click image for a full size version

Another really neat feature with Blade is that you can run it across the Encase evidence files without having to mount them. Having done this in my case Blade recovered over three thousand deleted History.plist files. I then loaded the recovered plist files into Netanalysis 1.51 resulting in over 300,000 internet history records to review. Cool.

Prefetch and User Assist

2010-05-27T11:14:00.000+01:00

It seems to me that more and more cases I see only have evidence within unallocated clusters. It is also a frustration that the CPS seem less and less interested in any artefact found there. They seem to have the view that any thing currently living in unallocated clusters somehow magically arrived there and has nothing whatever to do with the computer's user.

Obviously we try and address this misconception, by trying to investigate how the evidence in question came to be on the computer, and to a lesser extent how it came to be deleted. Which brings me on to another frustration - file wiping software. This is another thing I see more and more. Properly configured file wiping software eliminates the little fragments of evidence we use to piece our cases together.

Recently I was faced with this scenario - evidence could only be found in unallocated and there was file wiping software sat there in program files. Sentencing Advisory Panel guidelines allude to the presence of file wiping software being an aggravating factor to consider when sentencing. But in this case it occurred to me that it would be evidentially useful to know just how often my suspect used the file wiping software concerned. File time stamps may indicate when the program was last executed and installation dates can be discerned from a variety of locations (registry entries, folder creation dates and so on) but where do you establish how often the program was used? You never know -it may write to a log file or create event log entries but many don't. In my case the answer lay in two areas - Prefetch and User Assist.

Prefetch
My suspect was using Microsoft Windows XP. This OS (as the later Vista and Windows 7) performs application and boot prefetching. This process is designed to speed up the loading of applications (with regards to application prefetching) by storing data required by the program during the first ten seconds of use in a file - a prefetch file. These files are stored in the Windows/Prefetch folder and have a .pf file extension. The file names are a combination of the applications name and a hash of its file path. The hash may be useful in some cases because it could indicate that an application lives in more than one location (which is often suspicious). Some work on analysing the hash algorithm has been carried out by Yogesh Khatri at 42llc. The files themselves contain some useful information including last time of execution, the number of times the program was run and references to files and the file system utilised by the program in its first ten seconds of use. Unfortunately prefetch files are not differentiated by user. In my case the file wiping software had a prefetch file. There are a number of options open to us to analyse the prefetch file.

If all you need is the time of last execution and number of time the application was run for just one file you may as well do it manually. For Windows XP at file offset 120 an 8 byte Windows Filetime is stored which is the Last Execution Time. At file offset 144 the number of executions is stored as a four byte Dword. For Vista and Windows 7 the offsets are different - 128 and 152 respectively.

Bookmarking Last Execution Time and Date

Bookmarking number of times the application was run

If you have a number of prefetch files to analyse or you wish to corroborate your findings you could try the Mitec Windows File Analyzer program or run an enscript. Guidance Software's download center has two enscripts that fit the bill. PfDump.Enpack and Prefetch File Analysis. Pfdump outputs to the console and the Prefetch File Analysis enscript outputs to bookmarks.

UserAssist
UserAssist is a method used to populate a user's start menu with frequently used applications. This is achieved by maintaining a count of application use in each users NTUSER.DAT registry file. I use Access Data's Registry Viewer application to parse and decode this information. Simon Key has written a cool enscript which is bang up to date with Windows 7 support. Detailed information, including the changes introduced with Windows 7, and the script can be found within GSI's download center.

In my case I encountered a possible anomaly in that the Prefetch and UserAssist run counts were different. With multiple users you would expect this as the Prefetch run count is not user specific. I had only one user in my case and the UserAssist count was significantly greater albeit that both were four figure numbers. A possible explanation is that if the application's prefetch file is deleted when the application is next used the prefetch run count starts again from 1.

References
https://42llc.net/index.php?option=com_myblog&show=Prefetch-Files-Revisited.html&Itemid=39
http://en.wikipedia.org/wiki/Prefetcher
http://members.rushmore.com/~jsky/id14.html
http://members.rushmore.com/~jsky/id37.html
http://jessekornblum.com/presentations/dodcc08-2.pdf

C4P Import to Encase enscript and Lost Files

2010-05-11T06:37:00.000+01:00

Many C4P users experience problems when importing bookmarks back into Encase from C4P. A common problem is that files bookmarked in Unallocated Clusters don't match up to actual picture data. Almost always the cause of this problem is that the user has run the Recovered Folders process in Encase after running the C4P Graphics Extractor enscript thus altering the amount of unallocated clusters (as calculated by Encase). Trevor has a two page pdf on the C4P website addressing all the potential issues.

I have noticed another problem. A large number of my notable files are in Lost Files. Lost Files in Encase on an NTFS volume are files that have an MFT entry but their parent folder has been deleted. It is possible to have a number of files in the virtual Lost Files folder that have the same file name (and path). In my current case where I have duplicate file names in Lost Files the C4P Import enscript has not always bookmarked the correct file, bookmarking another file with the same name and path instead. This is sometimes further complicated by the incorrect file being deleted and overwritten.

The symptoms of this problem are easy to detect. Viewing your C4P import within the Encase bookmarks tab in gallery view results in a number of pictures not being displayed. When checking the bottom pane in text view you see that the bookmarked data for the non displaying pictures does not relate to a picture. Alternatively the picture you see does not relate to the C4P category it should be. To review this I am currently selecting (blue ticking) all non displaying pictures or wrongly bookmarked pictures and then tagging these selected files. Having done this in Entries view I am sorting by selection (blue tick) then highlighting a blue ticked file, then sorting by name. This brings all the other files with the same name together in Entries view. I am then checking the others to find the file that was meant to be bookmarked.

The underlying problem is a small bug in the C4P Import v2 enscript. Trevor has now kindly fixed it for me and will no doubt circulate the revised script. However in the meantime to fix the script

Find the following file in the import script folder: ..\include\ProcessReportClass.EnScript

In there, find the following function:

EntryClass FindByFullPath(ImportRecordClass irc, CaseClass c)

It’s a short function, only eight lines – highlight them, and replace with the following:

EntryClass FindByFullPath(ImportRecordClass irc, CaseClass c){

EntryClass e = c.EntryRoot();

e = e.Find(irc.DeviceName + "\\" + irc.Path);

if(e){

if(e.PhysicalLocation() == irc.PhysicalLocation)

return e;

else

return null;

}

else

return null;

}

Save and update.

HTH someone :)

Volume Shadow Copy Forensics - the Robocopy method Part 2

2010-04-13T15:56:00.000+01:00

Without further ado this post will build upon Volume Shadow Copy Forensics - the Robocopy method Part 1. In part one we looked at using Robocopy to extract data from a single shadow copy at a time. We will now look at a method to extract data from a range of shadow copies in one go. I will also cover some slightly more advanced options.

What are we going to need?
For what follows we will need a Windows 7 box (real or a VM), Encase with the PDE module and some storage space formatted NTFS. Robocopy is pre-installed within Windows 7.

Method
You will already have an Encase image of the drive you wish to investigate. When this is loaded up into an Encase case you need to gather some information in respect to the shadow copies you wish to investigate further. You will need to note the File Creation dates and if you wish to be more precise establish the Shadow Copy IDs stored at File Offset 144 for 16 bytes - bookmark as a GUID in Encase. Next you will have to mount the volume containing the shadow copies as an emulated disk using the Encase PDE module with caching enabled. On my box the mounted volume was allocated the drive letter J. I am using a Windows 7 box - if you are using a Windows 7 VM add the PDE mounted disk to the VM as an additional hard disk. Then on your box or in the VM:

Run a Command Prompt as Administrator and type the command (substituting J for the drive letter allocated to your mounted volume and G:\Shadows with the path of your export directory):

vssadmin list shadows /for=J: > G:\Shadows\list_of_shadow_copies.txt

This will create a text file containing a list of available shadow copies. From the list we can identify a range of shadow copies that we wish to investigate further. We now need to create symbolic links to them using the command:

for /l %i in (22,1,24) do mklink /d c:\Users\Richard\Desktop\Symbolic\SC%i \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy%i\

note: there is not a space after the ?

This command will create symbolic links for all shadow copy IDs starting at 22 up to 24. Obviously vary the (22,1,24) part to suit - 22 is the start, 1 increments by 1 and 24 is the end value. The symbolic links in this example are being created in a folder C:\Users\Richard\Desktop\Symbolic that I have allocated for this purpose. Many walk throughs, including ones I have prepared, often create the symbolic links at the root of C. Vista and Windows 7 do not like files being stored there so I think it is better practise to create the symbolic links in a user area.

OR
If you do not wish to process a range of shadow copies but need to process more than one or two you can instead use the command:

for %i in (18 20 22) do mklink /d c:\Users\Richard\Desktop\Symbolic\SC%i \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy%i\

note: there is not a space after the ?

In this example the command process only shadow copy IDs 18, 20 and 22.

Next we will run robocopy over the range of shadow copies we have selected:

for /l %i in (22,1,24) do robocopy c:\Users\Richard\Desktop\Symbolic\SC%i\Users G:\Shadows\SC%i *.jpg *.txt /S /COPY:DAT /XJ /w:0 /r:0 /LOG:G:\Shadows\Robocopy_log_SC%i.txt

OR
for %i in (18 20 22) do robocopy c:\Users\Richard\Desktop\Symbolic\SC%i\Users G:\Shadows\SC%i *.jpg *.txt /S /COPY:DAT /XJ /w:0 /r:0 /LOG:G:\Shadows\Robocopy_log_SC%i.txt

where you are interested in just specific shadow copies.

This command will create output folders named after each selected shadow copy along with a log of what has been copied. These items are being stored within an export folder prepared for the purpose. In this example I have drilled down to just the Users folder and copied out only jpg and txt files. Please see Part 1 for a detailed explanation of the options used in the command. The output folders can be dragged into Encase as single files. All paths and timestamps have been preserved.

Network Shares instead of Symbolic Links alternative
In part 1 I touched upon possible permission and copying errors. Troy Larson from Microsoft commented that creating shares instead of symbolic links may overcome some issues. So as an alternative the command:

for /l %i in (22,1,24) do net share SC%i=\\.\HarddiskVolumeShadowCopy%i\

will create network shares entitled SC22, SC23 and SC24 for the shadow copy IDs 22-24. We can now use robocopy to copy data out of these shares:

for /l %i in (22,1,24) do robocopy \\localhost\SC%i\Users G:\Shadows\SC%i *jpg *.txt /S /COPY:DAT /XJ /w:0 /r:0 /LOG:G:\Shadows\Robocopy_log_SC%i.txt

In this example I am accessing the shares on the same box hence localhost but of course you can run this across a network. The resulting data is as before.

Incorrect Function
You may run into what I think is a permission related error - clicking on the symbolic link results in

or you see

2010/04/12 15:25:28 ERROR 1 (0x00000001) Accessing Source Directory c:\Users\Richard\Desktop\Symbolic\SC22\Users\Incorrect function.

in your log file.

I have tried myriad ways to overcome this - trying to take ownership of the Shadow Copies using cacls, icacls and everything else but the kitchen sink. However I did eventually find a workaround. In Volume Shadow Copy Forensics.. cannot see the wood for the trees? I discussed imaging shadow copies using George Garner's Forensic Acquisition Utility. This utility appears not to have this issue so the command

for /l %i in (22,1,24) do dd if=\\.\HarddiskVolumeShadowCopy%i of=G:\Shadows\%i.img bs=512 count=1 --localwrt

will image just one sector of each shadow copy in our range. This takes just a few seconds. Then after imaging make your symbolic links or network shares. The Incorrect Function issue is overcome. Don't ask me why.

Cleaning Up
At the conclusion of your investigations you will want to remove the symbolic links or network shares you have created.

To remove the symbolic links

for /l %i in (22,1,24) do rd c:\Users\Richard\Desktop\Symbolic\SC%i

To remove the shares

for /l %i in (22,1,24) do net share SC%i /delete

Dealing with the storage issues
If you want to copy substantial amounts out of a large number of shadow copies you are faced with the problem of where you can store it. In Volume Shadow Copy Forensics.. cannot see the wood for the trees? I observed that there is considerable duplication of files in each shadow copy. I have found that a utility like Duplicate and Same Files Searcher can be useful. This utility can search across your export folders and identify duplicates. You can then opt to retain the first file and then create hard links for all the duplicate files. This utility can also move duplicate files, thus allowing you to focus on just the unique files.

References
Windows 7: Current Events in the World of Windows Forensics Harlen Carvey, Troy Larson
Reliably recovering evidential data from Volume Shadow Copies in Windows Vista and Windows 7 QCC

Volume Shadow Copy Forensics - the Robocopy method Part 1

2010-04-06T16:33:00.000+01:00

There is always more than one way to skin a cat and so I make no apologies for discussing another approach to processing volume shadow copies. This approach - I'll call it the Robocopy method - has been researched and developed by the chaps over at QCC, John Douglas, Gary Evans and James Crabtree and they have kindly let me crib from their notes. This post is Part 1 - I have simplified QCC's approach but have also removed some functionality. In Part 2 I will expand on the simplified approach and add back in some functionality.

Robocopy is a robust file copying utility developed by Microsoft. This method allows us to copy out folders and files of interest from any notable shadow copies. The process will preserve folder and file paths and timestamps. The key advantages are that it is efficient -both in storage and speed.

This blog post complements my previous posts Vista Volume Shadow Copy issues and Volume Shadow Copy Forensics.. cannot see the wood for the trees? and the method documented below is similar in the early stages.

Method
You will already have an Encase image of the drive you wish to investigate. When this is loaded up into an Encase case you need to gather some information in respect to the shadow copies you wish to investigate further. You will need to note the File Creation date and if you wish to be more precise establish the Shadow Copy ID stored at File Offset 144 for 16 bytes - bookmark as a GUID in Encase. Next you will have to mount the volume containing the shadow copies as an emulated disk using the Encase PDE module with caching enabled. On my box the mounted volume was allocated the drive letter I. I am using a Windows 7 box - if you are using a Windows 7 VM add the PDE mounted disk to the VM as an additional hard disk. Then on your box or in the VM:

Run a Command Prompt as Administrator and type the command (substituting I for the drive letter allocated to your mounted volume)

vssadmin list shadows /for=I:

This will result in a list of all available shadow copies on the selected volume

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Contents of shadow copy set ID: {2202d8a9-1326-4254-9818-252ece858b17}
Contained 1 shadow copies at creation time: 10/12/2009 14:41:25
Shadow Copy ID: {ad2e71d0-48d6-44b9-9715-f5ff6b5a5643}
Original Volume: (I:)\\?\Volume{34e5a98a-1a1d-11df-a259-00236cb6de69}\
Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy5
Originating Machine: Richard-MBP-Vis
Service Machine: Richard-MBP-Vis
Provider: 'Microsoft Software Shadow Copy provider 1.0'
Type: ClientAccessibleWriters
Attributes: Persistent, Client-accessible, No auto release, Differentia
l, Auto recovered

Contents of shadow copy set ID: {e13bb9d9-c522-422b-b92a-37f6d12363d9}
Contained 1 shadow copies at creation time: 15/12/2009 12:17:37
Shadow Copy ID: {d0e1c613-7892-47e1-9b7e-f638adac9d16}
Original Volume: (I:)\\?\Volume{34e5a98a-1a1d-11df-a259-00236cb6de69}\
Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy6
Originating Machine: Richard-MBP-Vis
Service Machine: Richard-MBP-Vis
Provider: 'Microsoft Software Shadow Copy provider 1.0'
Type: ClientAccessibleWriters
Attributes: Persistent, Client-accessible, No auto release, Differentia
l, Auto recovered

Marry up your bookmarked GUID to the Shadow Copy ID number to identify the Shadow Copy Volume you wish to process. The next step is to create a symbolic link to the selected shadow copy (ShadowCopy6 in this example) by typing the command

mklink /d C:\shadow_copy6 \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy6\

which results in the output

symbolic link created for C:\shadow_copy6 <<===>> \\?\GLOBALROOT\Device\Harddisk
VolumeShadowCopy6\

We are now going to use robocopy to copy out data from the mounted shadow copy - for this example I have created a folder called SC6 on my export volume. The command I used for this example is

robocopy C:\shadow_copy6\Users\Richard G:\SC6 /S /XJ /COPY:DAT /NFL /NDL /w:0 /r:0

This results in robocopy outputting a Job Header to the console

---------------------------------------------------------------------------
ROBOCOPY :: Robust File Copy for Windows
---------------------------------------------------------------------------
Started : Mon Apr 05 11:23:18 2010
Source : C:\shadow_copy6\Users\Richard\
Dest : G:\SC6\
Files : *.*
Options : *.* /NDL /NFL /S /COPY:DAT /XJ /R:0 /W:0
---------------------------------------------------------------------------

The header usefully sums up what I have asked robocopy to do.

I am copying only the Richard user profile (Users\Richard) to my export folder SC6
*.* indicates that I am copying all files
/NDL suppresses directory listings to the console
/NFL suppresses file listings to the console
/S copies the source folder and all sub folders and files
/COPY:DAT copies data, attributes and timestamps
/XJ exclude junction points
/R:0 number of retries on failed copies (in other words -do not re try)
/W:0 wait between retries

Nothing much happens at the command prompt now unless a failed file copy is encountered when you will receive output to the console similar to

2010/04/06 11:24:48 ERROR 5 (0x00000005) Copying File C:\shadow_copy6\Users\Rich
ard\AppData\Local\Temp\~DFA2ED.tmp
Access is denied.

When the copying is completed a summary is outputted to the console

It can be seen that 17 directories have been skipped. This means they have not been copied -probably because of permission issues. Also notable is the copying speed which is much quicker than imaging.

The output folder now contains a copy of the Richard users profile. Drag the contents of the export folder into Encase which processes the contents as single files. You may wish to create a logical evidence file of these single files.

Alternative Robocopy Configuration
As inferred from the Job Header above it is possible to take a fairly granular approach to what is copied out of your shadow copy. For example the command

robocopy C:\shadow_copy6\Users G:\SC6 *.jpg *.bmp *.png /S /XJ /COPY:DAT /NFL /NDL /w:0 /r:0

will copy out all jpg, bmp and png files from all User profiles. With reference to the two examples in this post and the robocopy manual it is possible to configure the copy operation in many different ways. For example you could just copy files that have timestamps in a particular range or files that are greater than a particular size.

Incorrect Function
If you play with VSCs often you will run into this rather helpful Microsoft error message. Tips to overcome it in Part 2.

References
Windows Vista/7 Recovering evidential data from Volume Shadow Copies John Douglas
http://technet.microsoft.com/en-gb/library/cc733145(WS.10).aspx
http://blogs.sans.org/computer-forensics/2009/01/08/robocopy-–-a-computer-forensics-tool/
http://42llc.net/index.php?option=com_myblog&task=tag&category=robocopy&Itemid=39

Internet History Examination Tools - you generally get what you pay for

2010-03-22T18:38:00.000Z

My digital Life of Grime case and another ongoing case have caused me to look more closely at the tools we use to analyse internet history. Life of Grime's suspect had a penchant for the Flock browser and the other ongoing case involves Firefox 3.0.1 running on an Apple iBook. The version of Flock in use is built on Firefox 3 also. It seems to me that I am seeing Firefox as the browser of choice in the majority of my cases. I know that most available statistics still put Internet Explorer in top spot but I still maintain Firefox has overtaken IE in my cases. Certainly Firefox is (worryingly?) by far the most popular browser used to read this blog.

Now as far as the data we think of as internet history is concerned, Firefox 3.x.x stores this data in a Sqlite database, places.sqlite and sometimes additionally places.sqlite-journal. However in the wider sense internet history can also be determined from the Firefox cache because the page elements stored there contain each elements URL and associated times and dates. Firefox stores it's cache in a folder entitled Cache within the xxxxxxxx.default folder (it is possible for a user to have more than one profile -so there may be other profile names not having the word default appended). Cached items may be stored in a file by itself (albeit renamed without a file extension) or more often within cache block files (e.g. _CACHE_001_). These cache block files can contain many cached items and therefore an index file is needed to record where an item is stored in the cache. This index is called a cache map and is the file _CACHE_MAP_.

Many of us have made use of a number of free utilities to examine Firefox 3 internet history. These tools have become popular due to our mainstream, paid for, tools falling a little behind the curve and not supporting Firefox 3. These free utilities include:

http://www.firefoxforensics.com/
http://forensic-software.co.uk/foxanalysis.aspx
http://www.machor-software.com/firefox_forensics
http://www.nirsoft.net/web_browser_tools.html
http://www.woany.co.uk/firefoxforensics/
http://manuel.santander.name/wbf.html

All bar the Nirsoft tool can not parse any data out of Firefox cache. Now for me that is a problem -not necessarily with the tools, but with the methodology I am going to use in a production line forensics environment. I know there are other approaches -substituting the suspects profile into a test installation of Firefox for example but ideally I would prefer a one stop shop approach.

Regular readers will know that I work in an Encase shop and the Encase 6.15 comprehensive internet search will parse out live Firefox records from cache. However in both the cases I am working on now, the cache map file is zeroed out. In this situation Encase does not parse records still recorded within the cache block files (and for that matter neither does the Nirsoft utility).

Which brings me to NetAnalysis. As our more experienced readers will know NetAnalysis is the industry standard, tried and tested tool for the analysis of browser history and cache, particularly Internet Explorer history and cache. NetAnalysis has always supported other browser artefacts but until now not Firefox version 3. NetAnalysis 1.50 and the associated program HstEx 3 now support Firefox 3 and then some.

In both my cases I have copied out the profile folders and quickly added them to a NetAnalysis workspace using the Open all history from folder option. Because the cache map file is empty the cache is effectively deleted which is where HstEx 3 comes in. HstEx 3 is a deleted history record extractor which can locate deleted records from physical disks, raw images and new to this version, directly from Encase evidence files. In both my cases HstEx 3 was able to parse records from the cache block files that none of my other tools, including Encase 6.15, could do. The advantage of being able to run HstEx 3 directly over Encase evidence files can not be over stated. By running multiple sessions of Hstex 3, productivity can be considerably enhanced and at the same time freeing up image mounting tools for other uses.

NetAnalysis 1.50 also introduces many other new features -probably the headline one being the significantly enhanced cached web page rebuilding feature. This is very cool. Simply load in your suspects browser cache, press F6 and the workspace filters cached webpages that can be rebuilt. Double clicking on them displays the page in a viewer and within an export folder the underlying html and associated page elements can be found. Each rebuilt page folder can be copied for use in an html report for example. I can't do this feature justice here but a Digital Detective knowledge base article covers it in more detail. Other enhancements include the provision of an extensive audit log. I think this facility is anticipating the tightening of regulation here in the UK with the advent of the Forensic Science Regulator. More immediately it allows the creation of cracking contemp notes. Another notable addition is the ability to provide more information about redirect and referral URLs.

As a final thought, we all love free tools, many of them written by Harlan Carvey, Mark Woan, Tim Coakley, John Douglas and others we know and respect. But we do need to be careful - one or two of the free tools I have referred to above have been written by people who, for whatever reason, have had to obfuscate their identity. I do not doubt any of these tools, but what if a new tool was released that extracted data that none of our current tools could. If we do not know who is behind it how do we know that the said tool was not on a certain date in the future going to scour our forensic network and modify all the Encase evidence files it came across? As I said at the start you generally get what you pay for. Till next time.

Flock shepherds in a Life of Grime

2010-03-21T13:11:00.000Z

After writing that title I am wondering if I wouldn't be better employed writing crossword clues. Back at the sausage factory I am bogged down investigating a digital Life of Grime. You probably have done a case like it - there are several hard drives in the suspect's box all with multiple partitions. Every nook and cranny is stuffed with IPoC and IVoC in a semi organised way. Almost as if all the material was neatly stored at one time until my suspect got lazy and started storing stuff on the floor, in the hall, under the table, that sort of thing. There are new folders after new folders -you don't see a New Folder (13) everyday. Not content with filling up his hard drives my suspect had also felt the need to back his stuff up to CD-R or DVD. Repeatedly.

The sheer quantity of files and folders in this case presents many problems and I thought I would share a few of them with you. Most of the problems are linked to pictures, that is the vast quantity of pictures, all of which have to be categorised and counted.

As many of my readers will know in a case like this C4P is an essential tool (going off at a tangent -googling C4P led to the discovery that c4p dot com is an online community for swingers - and now having said that we better watch out for some dodgy google ads appearing to the right ->> but hey they might improve my minuscule return from them ;-) ). C4P stores the results of categorisation, whilst the case is being worked on, in an Access database that has a 2GB file size limit. Now depending on how deeply your MDP folder is buried within your case folder structure this equates to about 1.9 million picture files. A long standing problem has been that if the c4p graphics extractor enscript carves out more than about 1.9 million picture files C4P itself would fail to create a case, due to the underlying Access database maxing out. In this case well over 2.1 million images were carved from the optical media alone and I was pleased to discover that C4P version 4.01 at least dealt with the maxing out issue fairly gracefully by creating a .c4p4 file and then a second .c4p4 file for the overspill. Case creation failed but I was subsequently able to create a new case by opening the double clicking on the first c4p4 file, which is associated with C4P. Doing this gave me a case with exactly 1,900,000 pictures which when viewed pragmatically is better than starting again. It is clear that the Access database problem will become more of a problem as main stream hard disks become bigger and bigger and I was pleased to discover that a beta version (4.03) of C4P is available that utilises an SQLite database (and whilst at it, runs happily on 64 bit boxes). This version does not limit the number of pictures you can have in one case.

In a case like this another C4P bugbear also rears its ugly head. The latest C4P graphics extractor enscript (4.03) carves out embedded thumbnails within jpgs, so for every picture file potentially you may have three carved images- the original, a thumbnail and a preview image. I am not sure why the latest script does this as a feature of earlier scripts was that it didn't carve embedded images to avoid duplication. It is that word -duplication- that is the problem. C4P will allow you to create a report directly from the program but if you need to compile statistics (how many Level 1s, 2s etc.) the report will be inaccurate. I do not use C4P for reporting and instead use the C4P Import enscript to bring back the C4P results into my Encase case. Once in Encase, sorting the C4P bookmarks by file offset provides a good indicator of how many embedded images there are. Luckily it is a fairly simple exercise to remove the duplicate bookmarking -simply selecting (blue checking) each category folder in turn and tagging selected files, and then from Entries view rebookmarking the files concerned will effectively sort out most duplication. Obviously different considerations will apply to pictures embedded into files for other reasons.

Moving away from C4P another issue in this case is establishing where the pictures came from. Obviously establishing which browsers are being used is a job that need doing and an Enscript that does just that has been written by Mark Woan over at Woanware. The WebBrowserInformationFinder enscript outputs to the console enabling you to copy and paste into your contemporaneous notes -very neat. Talking of contemp notes I use John Douglas's Case Notes - another very neat program. Looking at the problem a little wider it is worth considering some other angles to this problem, the registry contains a few pointers too, Harlan Carvey's post Browser Stuff covers this in more detail.

As it turned out my suspect used Flock as his default browser. This browser is built upon Firefox 3 and stores internet history and cache in a profile in the same way as Firefox. In the XP box I was looking at the profile folder was stored at the path C:\Documents and Settings\USER_NAME\Application Data\Flock\Browser\Profiles\xxxxxxxx.default. I have analysed this with NetAnalysis 1.50. More on NetAnalysis, Firefox 3 and the tools available to analyse this browser's artefacts in my next post.

Volume Shadow Copy Forensics.. cannot see the wood for the trees?

2010-02-19T23:07:00.006Z

There has been a great deal written about volume shadow copy forensics lately, much of it very technical. The purpose of this post is to provide some of this information in an abridged form and to document a methodology to investigate them that works for me. I work in an Encase shop so much of what follows is tailored for Encase users. It is a followup to my earlier post.

What is a Volume Shadow Copy?
It is a snapshot of a NTFS formatted volume. Typically they are created every day or two in Vista and on a weekly basis in Windows 7. Both operating systems will also create volume shadow copies prior to the installation of new software (including Windows updates). The Volume Shadow Copy Service (VSS) monitors all the changes made to a volume from the time that a shadow copy was created until the creation of the next one. For the purpose of monitoring a volume is divided into 16 kilobyte blocks. When a block is about to be overwritten it is backed up in the shadow copy. Only changed blocks are backed up. If there is a requirement to revert to a snapshot the original blocks are restored, replacing the changed ones, in a sense reconstituting the volume back to its state when the snapshot was taken. Certain versions of Vista and Windows 7 allow users to revert to previous versions of files and folders, not just at the volume level.

Where are Volume Shadow Copies stored?
In the System Volume Information folder stored at the root of each protected volume.

Why might I need to investigate the contents of a Volume Shadow Copy?
You might not, but I am seeing many cases now where the Shadow Copies contain relevant evidence. For instance, during your IPOC cases the file finder or the C4P graphics extractor enscript or Blade have identified pictures within the shadow copy file which do not exist on the live volume. At that stage you simply do not know for example, whether these pictures were part of a web page accidentally viewed and then immediately closed down, embedded within an unsolicited email message, or there as a result of the intended actions of your suspect. Or you may have experienced cases where Encase lists many files as deleted and overwritten but that have very very dodgyfile names.

The following screenshot is from an Encase case which contains a live volume and images of two shadow copies of the same volume. Four deleted and overwritten files are listed, the metadata of which has been recovered by Encase from the MFT on the live volume. It can be seen that these files are intact within the shadow copies which allows your file carver of choice to recover them.

There are of course many other scenarios and examples I could give but the long and the short of it is that if deleted data from the live volume could be relevant to your case, you may need to investigate the shadow copies.

Which shadow copies should I look at further?
It has been suggested that, because each shadow copy is effectively a volume in it's own right which mirrors the size of the source volume, that if you have twenty shadow copies you will have to investigate twenty times as much data. In Production Line forensics this simply is not going to happen. Therefore we need to make some educated guesses as to which shadow copies to look at. Have our file carvers flagged any as meriting further investigation? Do keyword searches point to any? Were any shadow copies created just before or just after a particular event we are looking at? Are there a rash of shadow copies created within a short time of each other by software installation or windows updates which we can exclude? Once we have made our choices we can proceed further but it is not the case that the amount of data to be investigated is a multiplied by the number of shadow copies you chose to investigate. This is because the majority of data within each shadow copy is exactly the same as the data on the live volume. We are really only interested in the differences and I will discuss a method to do this later in this blog post.

What are we going to need?
For what follows we will need a Windows 7 box, Encase with the PDE module and some storage space formatted NTFS and ideally with File and Folder compression enabled. We will also need George Garner's Forensic Acquisition Utility.

Method
You will already have an Encase image of the drive you wish to investigate. When this is loaded up into an Encase case you need to gather some information in respect to the shadow copies you wish to investigate further. You will need to note the File Creation date and if you wish to be more precise establish the Shadow Copy ID stored at File Offset 144 for 16 bytes - bookmark as a GUID in Encase.

Next you will have to mount the volume containing the shadow copies as an emulated disk using the Encase PDE module. On my box the mounted volume was allocated the drive letter I.

I am using Windows 7 - every thing that follows should work in Vista and has worked for many people. However for me, in testing using Vista it did not always work as expected -generating various permissions related errors -so far - touch wood- Windows 7 just works.

Run a Command Prompt as Administrator and type the command (substituting I for the drive letter allocated to your mounted volume)

vssadmin list shadows /for=I:

This will result in a list of all available shadow copies on the selected volume

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Contents of shadow copy set ID: {2202d8a9-1326-4254-9818-252ece858b17}
Contained 1 shadow copies at creation time: 10/12/2009 13:41:25
Shadow Copy ID: {ad2e71d0-48d6-44b9-9715-f5ff6b5a5643}
Original Volume: (?)\\?\Volume{34e5a98a-1a1d-11df-a259-00236cb6de69}\
Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy4
Originating Machine: Richard-MBP-Vis
Service Machine: Richard-MBP-Vis
Provider: 'Microsoft Software Shadow Copy provider 1.0'
Type: ClientAccessibleWriters
Attributes: Persistent, Client-accessible, No auto release, Differentia
l, Auto recovered

Contents of shadow copy set ID: {e13bb9d9-c522-422b-b92a-37f6d12363d9}
Contained 1 shadow copies at creation time: 15/12/2009 11:17:37
Shadow Copy ID: {d0e1c613-7892-47e1-9b7e-f638adac9d16}
Original Volume: (?)\\?\Volume{34e5a98a-1a1d-11df-a259-00236cb6de69}\
Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy5
Originating Machine: Richard-MBP-Vis
Service Machine: Richard-MBP-Vis
Provider: 'Microsoft Software Shadow Copy provider 1.0'
Type: ClientAccessibleWriters
Attributes: Persistent, Client-accessible, No auto release, Differentia
l, Auto recovered

Contents of shadow copy set ID: {4db5347c-214a-4e5c-b785-0fd993f1dc33}
Contained 1 shadow copies at creation time: 15/12/2009 11:18:25
Shadow Copy ID: {b7621ae2-5efb-4929-aa35-39af3d6e39ac}
Original Volume: (?)\\?\Volume{34e5a98a-1a1d-11df-a259-00236cb6de69}\
Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy6
Originating Machine: Richard-MBP-Vis
Service Machine: Richard-MBP-Vis
Provider: 'Microsoft Software Shadow Copy provider 1.0'
Type: ClientAccessibleWriters
Attributes: Persistent, Client-accessible, No auto release, Differentia
l, Auto recovered

Contents of shadow copy set ID: {2445d9b6-58fd-4ac6-b73e-7bd0ebbec6cc}
Contained 1 shadow copies at creation time: 15/12/2009 12:14:44
Shadow Copy ID: {709f74d5-ded2-4294-a292-c7cc4db0e67b}
Original Volume: (?)\\?\Volume{34e5a98a-1a1d-11df-a259-00236cb6de69}\
Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy7
Originating Machine: Richard-MBP-Vis
Service Machine: Richard-MBP-Vis
Provider: 'Microsoft Software Shadow Copy provider 1.0'
Type: ClientAccessibleWriters
Attributes: Persistent, Client-accessible, No auto release, Differentia
l, Auto recovered

Contents of shadow copy set ID: {c9e7850a-87db-4dbc-9d72-40749665d80d}
Contained 1 shadow copies at creation time: 09/02/2010 07:26:39
Shadow Copy ID: {2a871fe5-21f7-4fb6-b8e9-65e194a62901}
Original Volume: (?)\\?\Volume{34e5a98a-1a1d-11df-a259-00236cb6de69}\
Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy8
Originating Machine: Richard-MBP-Vis
Service Machine: Richard-MBP-Vis
Provider: 'Microsoft Software Shadow Copy provider 1.0'
Type: ClientAccessibleWriters
Attributes: Persistent, Client-accessible, No auto release, Differentia
l, Auto recovered

It can be seen that the Shadow Copy ID number I bookmarked as a GUID is identified by VSSAdmin as HarddiskVolumeShadowCopy5. The next step is to image this shadow copy using the Forensic Acquisition Utility (a version of dd that works in Windows). At the command prompt navigate to the path of the utility and type the command

C:\FAU.x64>dd if=\\.\HarddiskVolumeShadowCopy5 of=G:\shadow5.img --localwrt

It can be seen that the input file is referenced as \\.\HarddiskVolumeShadowCopy5 and the output file which I chose to name shadow5.img is located on G:. This volume is an NTFS formatted volume with file and folder compression enabled. --localwrt is a switch required to allow the utility to write to the G drive. The response at the command prompt will be similar to

The VistaFirewall Firewall is active with exceptions.
Copying \\.\HarddiskVolumeShadowCopy5 to G:\shadow5.img
Output: G:\shadow5.img
44048285696/57868808192 bytes (compressed/uncompressed)
55187+1 records in
55187+1 records out
57868808192 bytes written

Succeeded!

Enabling file and folder compression reduced the dd image from about 59GB to 44GB. Whilst imaging make sure that you have carried out an hash analysis in Encase of your source volume and create a hash set from it. Add only this hash set to your library.

Once imaging completes add the image to Encase (via the add raw image option) and carry out a hash analysis. Create a condition which filters out files that match a hash set.

The following screen shot shows that there are 237024 files contained within the image of HarddiskVolumeShadowCopy5 (broadly the same as the source volume)

Apply the condition which filters out files that have the same hash value as those in the original image and it can be seen that the amount of new or different files is considerably reduced to 7864 files (with the caveat that some of the original files may have been moved -which may or may not be relevant in your case).

I think it can be seen therefore that it is possible to reduce the number of files you need to consider by filtering out the duplication between the original volume and the shadow copy.

A Note of Caution about the Shadow Copy Images
When you image a Shadow Copy and add it into an Encase case as described above it is easy to think of the image as a point in time snapshot of the entire volume including the areas of the volume we refer to as unallocated clusters. Unfortunately it does not appear to be. However your forensic tool (Encase in my case) is likely to process it as if it was the entire volume. In my test case used to illustrate this blog post I imaged at 14:47hrs on 9th February 2010 two shadow copies -one created at 15/12/2009 11:17:37 and one created at 09/02/2010 07:26:39. At 15/12/2009 11:35 I copied a folder entitled Sony from an external device to the desktop of my user account on this Vista box. It can be seen in the screen shot below. It is a live folder containing files and has not been deleted.

It can also be seen within the shadow copy created on 9th February 2010 as expected.

And as expected it cannot be seen in the earlier shadow copy created prior to the folder being copied.

However unexpectedly when I ran the Encase Recover Folders feature across the HarddiskShadowcopy5 volume it found traces of the Sony folder and in fact many other files post dating the creation of the shadow copy.

The Encase Recover Folders feature parses unallocated clusters looking for folder metadata. It seems that it found data in unallocated clusters relating to the current volume. Therefore I believe that any deleted but recoverable data within the shadow copies needs to be treated with caution.

Other Methods
So far in this post I have discussed imaging the shadow copy pseudo-device. It is possible to mount the shadow copy as a symbolic link. There are various methods discussed using VMs etc. The most streamlined approach seems to me to be:

Mount shadow copy with Encase PDE
create a symbolic link (in this example at the root of C) using the command

mklink /d c:\shadow_copy5 \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy5\

drag the created symbolic link into a separate instance of Encase (for me using the same instance as the one running PDE blue screened my box) which causes Encase to treat the files and folders accessed via the symbolic link as Single files
Create a logical evidence file of these Single files (you may encounter some permissions issues along the way)

Methods involving accessing the shadow copies via Symbolic links will only provide access to the logical contents. It may be that the object you are seeking is deleted but recoverable (but note warning above).

References

http://blogs.sans.org/computer-forensics/2008/10/10/shadow-forensics/
http://www.digital-detective.co.uk/cgi-bin/digitalboard/YaBB.pl?num=1251461771
http://blog.szynalski.com/2009/11/23/volume-shadow-copy-system-restore/
http://windowsir.blogspot.com/2009/11/working-with-volume-shadow-copies.html
http://blogs.technet.com/filecab/archive/2006/09/01/452845.aspx
http://en.wikipedia.org/wiki/Shadow_Copy

TIM released - first test

2010-01-21T11:52:00.001Z

Tableau have released their Tableau high performance software imaging product today.

To install it you first must have installed the latest Tableau firmware updater. Once installed the interface is fairly simple to use. I found some of the terminology in the GUI a bit odd. For example the types of compression selectable for Encase e.01 files are No Compression, Fast Compression or Good Compression whilst Encase provides the options none, good or best. It seems that fast maps to good and good maps to best. There also does not seem to be the ability to actually name the created e.01 files- just their containing folders, again a little strange.

Now as far as performance is concerned in our environment it is slightly faster. We image to a mapped network drive on a san across a gigabit ethernet network. I imaged a Hitachi 80GB sata laptop hard disk via a Tableau T35i write blocker using FTK Imager 2.5.5 using the Encase e.01 format - imaging completed in 51 minutes. Using TIM to image the same drive took 44 minutes. In both cases maximum compression was selected. Obviously at this stage this testing is far from scientific but TIM seems to be between 10 to 15% faster. I will report how it goes on with larger 3.5 inch sata hard drives later.

C4P, MySQL and Windows 7

2010-01-07T11:36:00.000Z

I can just about remember the thrill of upgrading to a new OS. When I was the new kid on the block I couldn't wait to upgrade from Windows 98SE to XP Pro (somehow we missed out Windows ME). Anyhow the newer kids on the block have migrated to Windows 7 (it is prettier, more stable, blah blah). Now I am a dinosaur with XP Pro 64 bit!

Anyway I had a call today about getting various aspects of C4P to play nicely with MySQL on a Vista 64 bit box. In our office we don't actually use Vista on any of our forensic boxes so I thought I'd check out the issues on one of Windows 7 64 bit boxes. As you know (because if you are still reading this you probably use C4P along with a MySQL DB) C4P interacts with the MySQL database in two areas:

within Encase at Enscript level if the pre categorization option is selected
or via Data Migration/ Special/ Update Case Direct from C4P Hash Database within the Categorizer for Pictures program itself.

Both of these connections require an MySQL ODBC connector driver to communicate with the running MYSQL C4P hash database using a suitable database connection string. Essentially in this scenario we have a choice of four MySQL ODBC drivers:

All bar the 3.51 64 bit driver are installed via a Windows installer. The 3.51 64 bit driver is slightly trickier to install - you need to unpack the zip, run a command prompt as administrator, navigate the command prompt to your unpacked zip folder and then run the command

Install 0

Enscript level communication

At Enscript level it is possible to modify the database connection string which allows you to specify which ODBC connector driver to use. In testing on a Windows 7 64 bit box I have found that both the 3.51 and 5.1 64 bit drivers work (if the drivers fail you generally get a long unintelligible error message). The C4P 4.02 enscript allows the user to configure their own database connection string. The string that works for me is:

Provider=MSDASQL;DRIVER={MySQL ODBC 3.51 Driver};SERVER=Your_server_name_or_IP_address;DATABASE=c4p_hash;UID=c4p_user;PASSWORD=password;OPTION=3

Simply change 3.51 to 5.1 if you are using the later driver.

Categorizer for Pictures communication

The database connection string used by this program is hard coded and not user configurable. The program requires the 3.51 driver. However I could not get the Data Migration/ Special/ Update Case Direct from C4P Hash Database option to work on the Windows 7 64 bit box using the 3.51 64 bit driver. I suspect this is due to a permissions issue and tried to run C4P as administrator but I still failed to connect to the MySQL C4P hash db. However I was able to get the Data Migration/ Special/ Update Case Direct from C4P Hash Database option to work using the MySQL ODBC 3.51 32 bit driver.

Conclusion

Other combinations may work but on a Windows 7 64 bit box I recommend installing the MySQL ODBC 5.1 64 bit driver and the MySQL ODBC 3.51 32 bit driver to get C4P and the C4P graphics extractor enscript to play nicely with the MySQL C4P hash database.

Web Browser Session Restore Forensics

2010-01-05T12:07:00.001Z

As this is the first post of the year I would like to wish you a happy new year.

The posting title is the title of an excellent paper written by Harry Parsonage relating to Session Restore files created by the latest Mozilla (Firefox) and Internet Explorer 8 browsers. These files may contain enough information to allow the browser to display a users workspace exactly as it was prior to a forced restart. Obviously these files may contain significant evidence. I am not going to steal Harry's thunder so download his paper from http://computerforensics.parsonage.co.uk/other/other.htm

I know that Harry is not keen on blogs simply regurgitating information found elsewhere so I will try and add a little value.

Safari v4
Session Restore functionality is now a must have in modern browsers. Another browser to have similar functionality is Safari v4. The last session information is contained in a file entitled LastSession.plist

In Mac OSX 10.6 this file is stored at /Users/<user name>/Library/Safari

In XP this file is stored at C:\Documents and Settings\<User name>\Application Data\Apple Computer\Safari

I use the mac application - property list editor to review plists, there are windows applications to do this as well.

Firefox v3.5.6 running in Mac OSX 10.6
Harry's paper applies here in the main.

The sessionstore.js file is stored at /Users/<User Name>/Library/Application Support/Firefox/Profiles/XXXXXXX.default

Navman S30 Satnav device

2009-12-13T09:53:00.000Z

I have seen a number of different Navman models now. All of them have subtle variations on how to process them. Although the underlying OS is Microsoft WinCE.NET 5.0 Core Version I did not have to access this device via Mobile Device Centre. After applying some write blocking to my USB port this device was accessible as a mass storage device allowing me to image it with Encase. I ended up with 114,759 sectors being imaged.

The next step I took was to track down and consult a manual for the device in order to establish its capabilities. I expect all devices to have saved Home, Favourites and Recent destinations and this device was no exception.

In addition however this device can save trip logs, pre-planned itineraries, pictures and be paired with a mobile phone.

Almost all the relevant data is stored within xml formatted files. Microsoft Excel 2007 is an excellent tool for examining these files and subsequently reporting on them. I use the Get External Data / From Other Sources / From XML Data Import option via the data tab and allow Excel to sort out the formatting.

A good place to start is the file paths.xml stored at the root of the partition. This file details the location of some of the relevant files.

This is a more definitive list:

MyFavouriteLocations.xml - used to store the home location and favourites
MyRecentLocations.xml - used to store Recents and also Journey Starts
MyMultiStopLocations.xml - used to store saved multi stop journeys
MyRoute.xml - used to store the current journey which is in effect the last journey -on the device I examined this file was deleted but recoverable
UserSettings.xml – used to store device settings including where the unit was turned off

When a user enters a new address the menu shows previously entered towns or cities, road names and postcodes. This data is stored in the following files:

DWRecentPocode.xml – Previously entered postcodes, most recent first
DWRecentRoad.xml – Previously entered road names, most recent first
DWRecentPlace.xml – Previously entered towns or cities, most recent first

There are a number of presumably back up files also containing the same (as far as I could see) xml formatted data:

MyFavouriteLocations_bak.xml
MyRecentLocations_bak.xml
MyMultiStopLocations_bak.xml

There are also two files that appear to be temporary files which were deleted but recoverable, containing xml formatted data:

MyRecentLocations_New.xml
MyMultiStopLocations_New.xml

All of the above mentioned xml files are parsed very tidily using Microsoft Excel 2007. I use the same program to create an html version of the worksheet after a little tidying up. The longitude and latitude values need to be divided by 100,000. I populate a new column using the formula:

=HYPERLINK("http://maps.google.co.uk/maps?q="&(K3/100000)&"+"&(L3/100000)&"","Click here to view in Google Maps")
The cell K3 contains the Latitude and L3 the Longitude. The formula creates a clickable hyperlink to the Lat/Long in google maps.

There are one or two other files of interest:

destdata.dat - which contains the address used for the last navigated journey
gpslog.ini - detailing the location of trip log data
default_settings.xml -which in the FAVOURITES/ RECENTS/ MULTI-STOP section appears to detail the maximum number of favourites and recents
.pcd - on the unit I examined I could not locate a .pcd file however I understand from Andy Sayers that this file if it exists contains the phonebook from a paired mobile phone
Log001.log - again I did not see this file but if it exists it contains GPS track logs

Because we have physical access there is also a possibility of recovering relevant data from unallocated clusters. I located records in unallocated using the keyword <lat> .

References
Sat Nav Examination Guidance Notes (Andy Sayers)
Navman S-Series (S30, S50, S70 & S90i) User Manual

Binatone X350 UK&ROI 2nd edition GPS

2009-12-07T19:43:00.001Z

This device can be purchased very cheaply now from places like Asda and ebuyer. It runs Astrob Turbodog4 satellite navigation software within a Microsoft WinCE.NET 5.0 Core Version OS. Although I have not examined one I believe a number of Navigo devices run similar software. It has an SD card slot which was unpopulated in the one I looked at. The internal memory can be accessed like many similar devices via Mobile Device Centre in Vista which makes available a volume entitled ResidentFlash. I disable writing to USB devices by modifying the registry (there are many utilities about to do this). Simply paste the text below into a text file, give it a .reg file extension and then execute it and then reboot.

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\StorageDevicePolicies]
"WriteProtect"=dword:00000001

After copying and creating a logical evidence file of ResidentFlash I found three notable files within the MobileNavigator folder:

RecentDest.dat
FAV.DAT
SystemSet.dat

RecentDest.dat stores up to fifty of the recently navigated to locations. These locations are stored in records of 104 bytes in length. The first record starts at the first byte of RecentDest.dat, so by viewing the file in Encase with the view pane set to hex and dragging the view to show 104 bytes per line (assuming you have twin monitors), it is possible to see all the relevant data. Each location record stores Longitude and Latitude as 8 byte doubles which unfortunately Encase does not natively decode. The data interpreter in Winhex can do this. The hex editor 0xED on a mac can also do this but rounds up to fewer decimal places than Winhex. So given a fully populated RecentDest.dat file you have one hundred doubles to decode. I turned to my friend Oliver Smith over at Cy4or who wrote me an enscript which parses out the records to a csv file. Email me with a brief note about who you are for a copy

Recent Destinations also log the time entered, so very unusually some meaningful time and date information may be extracted. Providing these times have been recorded whilst the device can see satellites they are accurate and stored at the configured time zone. If a destination is entered when the unit can not see the sky and if the battery had been previously discharged it appears that the recorded time and date would be soon after 00:00 hours 1st January 2007.

I have decoded each 104 byte record as shown in figure 1

Figure 1 (click on image for larger version)

FAV.DAT contains user configured Favourites stored in 536 byte records. Once again the Longitude/Latitude are stored as eight byte doubles in the first sixteen bytes of each record.

SystemSet.dat appears to store the users Home location and again the Longitude/Latitude are stored as eight byte doubles.

Within the MobileNavigator folder there is a folder entitled Trace. This was empty in the one I looked out however the manual states:

The unit is capable of logging all positioning information received from the GPS satellites during navigation. It then uses this information to draw a track of the route on the map. This enables you to review the route information at a later time.

I imagine that should this feature be enabled a file of some sort will be stored in this folder.

References

http://binatonegps.com/gps/download/manual/X350II%20User%20Guide%20(Turbo%20Dog)%20-%2020080910.pdf
http://en.wikipedia.org/wiki/Double_precision_floating-point_format

Sony PSP internet history

2009-11-13T09:30:00.000Z

A recent case resulted from an entry in a compromised web server log. The GET request included the string "Mozilla/4.0 (PSP (PlaySation Portable); 2.00)". Our suspect had used a PSP to do dodgy stuff and the PSP eventually came my way. I looked around for some information but did not find a large amount of information, essentially the most useful items were an Encase Message Board post and Chapter 9 of a book entitled Advances in Digital Forensics V which I read via Google Books.

Sony PlayStation Portable hand held consoles have an inbuilt wi-fi adaptor and can therefore connect to the internet. The device utilises the Netfront browser. There are a number of different versions and firmware versions. The one I looked at had a label indicating that it was a PSP1001. This site details the many different types available. A PSP1001 is known as a PSP Fat (as opposed to a PSP Slim). The one I looked had version 4.05 firmware. These type of PSPs have a small amount of internal NAND flash memory and a Memory Stick ProDuo flash media card.

As far as I can ascertain it is not possible to examine the internal NAND memory of devices beyond 1.5 firmware because you would require hacked firmware and modified hardware to do it. The browser does store its cache in this area but I believe as a default only 512KB is used for this purpose. Some information can be derived from the internal memory via a manual exam. Essentially then, we are left with the Memory Stick ProDuo flash media card. Our Tableau USB write blocker would not recognise the card I had however I was able to image it using our Helix imaging box and Guymager. The card had a FAT16 file system and was examinable with Encase.

Files of interest
On the card I looked at only two files were of interest both in the folder \PSP\SYSTEM\BROWSER.

bookmarks.html contained what you would expect -user created bookmarks
historyv.dat contained internet history

Scott Conrad, Carlos Rodriguez, Chris Marberry and Philip Craiger's paper within Advances in Digital Forensics V refer to two further files of interest historyi.dat and historys.dat. I got my hands on a test PSP1001 with the same firmware as my suspects (4.05) and in testing I was not able to populate these files with any data. The files existed but I was not able to cause details of either Google Searches or user typed URLs to be stored in these files. My suspects card had an unpopulated historyi.dat file and no historys.dat file. As noted by Conrad et al I found in testing that I could only cause writes to historyv.dat by shutting the browser down gracefully. Simply turning off the PSP without shutting down the browser did not commit that sessions history to historyv.dat.

Structure of historyv.dat
The structure of historyv.dat is discussed by Conrad et al however they suggest that elements of the file were best decoded by introducing the data into a test PSP for decoding. For example the date of each history entry could be ascertained this way. I would prefer to carry out a completely static examination if possible, not least because on my suspects card I had recovered a number history records in slack space and a manual examination can be a little laborious. I have therefore decoded the records a little bit further as shown below at Figure 2 and Figure 3. Each historyv.dat file is headed with 66 bytes of data starting with the string Ver.01. Within this 66 bytes are two further bits of plain text - NFPKDDAT and BrowserVisit. Immediately following BrowserVisit is the first history record. The most recent record is listed first, the oldest last. Each record can be located using a GREP expression to search for the header - in Encase \x03\x00\x01\x00 -see Figure 1 below. Records can be found in slack space and unallocated clusters.

Click on image for larger version

Figure 1

Figure 2

Figure 3

A significant addition to the research of Conrad et al is the decoding of the date for each record. The date is recorded in the two bytes following the URL and is stored Little Endian. In Encase sweep these two bytes and right click, select Go To and check Little-endian. The value is the number of days since the Unix epoch (1st January 1970). This web site provides a good date calculator.
IMPORTANT NOTE re dates: the dates stored are in accordance with the PSPs internal clock. The clock resets when the battery is exhausted. With the firmware I looked at the reset date was 1st January 2008. This date is 13879 days from the Unix epoch. I speculate that the average user is unlikely to reset the date each time the battery exhausts, therefore I would expect to see a lot of dates in January 2008.

References and thanks
http://www.edepot.com/reviews_sony_psp.html
http://en.wikipedia.org/wiki/PlayStation_Portable#Web_browser
Forensic Analysis of the Sony Playstation Portable - Scott Conrad, Carlos Rodriguez, Chris Marberry and Philip Craiger
https://support.guidancesoftware.com/forum/showthread.php?t=33057&highlight=psp
http://www.computerforensicsworld.com/modules.php?name=Forums&file=viewtopic&t=654&highlight=psp

Thanks to Pete Lewis-Jones and Simon Maher for their help brain storming the date problem

Garmin Streetpilot C510

2009-11-01T08:44:00.000Z

I blogged earlier this year about the Garmin Nüvi 200 Sat Nav device and I have now had a crack at a Garmin Streetpilot C510.

The Streetpilot like the Nüvi 200 stores waypoints in a file Current.gpx found within the Garmin folder. This folder is accessible when the device is connected to a computer due to the fact that the device is designed to act as a mass storage device. It is probably worth expanding on what a waypoint is. Garmin's FAQs define them as

Waypoints may be defined and stored in the unit manually, by taking coordinates for the waypoint from a map or other reference. This can be done before ever leaving home. Or more usually, waypoints may be entered directly by taking a reading with the unit at the location itself, giving it a name, and then saving the point.

Essentially as far as both the Garmin devices discussed here are concerned the waypoints recovered from Current.gpx are the users favourites and home location. Apologies for teaching granny to suck eggs but it is probably worth stating that waypoints are not Track Logs. Most Streetpilots and Nüvi 200s do not store any tracking information (there is an unsupported hack which allows the modification of some units firmware to store tracking information).

As commented in my previous posting and within the SatNav forensics forum over at Digital Detective these Garmin devices do store other data not contained in Current.gpx. This data is the Recently Found locations which are effectively the last fifty locations a user chose to navigate to (or at least look at on the device). Evidentially this data may be useful. Up to now a manual exam using something like Fernico ZRT has been the answer. I have tried out a slightly different methodology.

Suggested Methodology for the examination of Garmin Streetpilot C510
(May work with other models)

Download Garmin USB drivers
Download Garmin xImage version 2.3
Download G7toWin
Install Ximage on your workstation
Turn on Garmin sat nav and press and hold your finger over the onscreen battery symbol for about 10 seconds
This should take you into a hidden diagnostics menu

On your Forensic Examination workstation run the Garmin USB drivers executable and work through to this screen

Connect Garmin sat nav to your Forensic Workstation and complete the USB driver installation (the sat nav must be displaying the hidden service mode - if it isn't it will act as a mass storage device)

Run G7toWin on your workstation (it does not need to be installed) and adjust the configuration to allow communication via USB

Within G7toWin via the menu bar select GPS/Download from GPS/ All
All available waypoints will display
Via File/Save As you can save the data to your filetype of choice (e.g. .gpx, .kml, .xml)
It is possible that one of the fields may contain an illegal character - in my testing the comment field did. I dealt with this in my exported kml and xml files with a decent text editor (PSPad) and the find and replace feature. Applications that support xml and Google Earth are not usually tolerant of any illegal characters/formatting.

Downloading of the waypoints is now taken care of. Next I want to deal with the

Recently Found

locations. I am going to suggest two approaches, which although relatively simple I have not seen documented elsewhere. The version of the device you are using may dictate which approach you try.

Approach 1
You should still be at the Diagnostics Menu - press the Exit icon
Via the main navigation menu select Where to?/ Recently Found
You should now see the first five Recently Found locations
On your Forensics Workstation launch xImage, your device should appear in the Device field then click Next
Select Get Images from the GPS then click Next
Set Image Type to Screen Shot
Clicking Next will allow you to save a screen shot of the currently displayed screen on the device
Using this method you can quickly screenshot all the screens you would have photographed in a manual exam, after each screenshot click back to prepare for the next one

Approach 2 is more invasive, however I think principal 2 of ACPO guidelines applies.

Approach 2
You don't initially have to have your device connected to your workstation for this to work
On the device select Settings/ Display
In the Display menu enable Screen Shot
This will cause a small camera icon to appear in the top right of the display
Pressing this icon will cause a screen shot to be saved into the Garmin/scrn folder upon the device
Screen shot all the screens you would have photographed in a manual exam
Connect device to workstation as mass storage device and cut and paste screenshots from it

UPDATE RE GARMIN Nüvi 310

Artemus has been looking at a Garmin Nüvi 310. He tried Approach 1 above and found that to enter the diagnostics mode he had to push and hold the top right of the display (as opposed to the battery symbol). HOWEVER he then encountered a message asking if he wished to delete all user data, so I guess for Nüvi 310 Approach 1 is a no go. So he tried Approach 2. He enabled the Screen Shot feature however on this device no camera icon appears. Screen shots are created by pressing the power button. Screen shots are saved into a folder entitled Screenshot on the media card.

TIM

2009-10-26T10:00:00.000Z

TIM is an acronym for Tableau Imager which unsurprisingly is new imaging software developed by Tableau. Tableau promise astounding imaging speeds. Apparently it will be available in beta form anytime soon. Given the quality of the Tableau write blockers I think this software is definitely worth watching out for. The latest info can be found here.

Video triage revisited

2009-10-11T07:26:00.001+01:00

Back in July 2009 I blogged about the potential of video triage. I was commenting on its effectiveness and had used a program written by John Douglas to explore what was possible. Mark Woan added a very interesting comment to that post, introducing a program he had written - Forensic Video Triage. I have now tried out a series of enscripts written by Oliver Höpli which aim to provide the same functionality as both John and Mark's programs.

Essentially all three approaches utilise a third party video playing and manipulation program to create and store thumbnails of frames at set intervals throughout a video clip. The investigator can then triage the video clip by reviewing the thumbnails as opposed to playing the video. The gallery feature in Encase for example makes reviewing the thumbnails a considerably quicker experience than playing the videos.

John's program utilises VLC, Mark's uses ffmpeg and Oliver's enscript calls upon mplayer for thumbnail creation. Each of these video utilities have inbuilt codecs and their capabilities may vary - in other words a video clip may play with one and not the others.

Oliver Höpli has integrated the process much more closely with Encase with his suite of enscripts, and for me this can only be a good thing. If you are an Encase shop the pre-processing is considerably reduced and the whole process is more seamless leading to greater productivity. The main enscript runs across selected (as in blue checked) movie files within your case and parses out thumbnails into a logical evidence file. Another enscript creates a folder structure within the Encase bookmarks tab based on the contents of the logical evidence file. Each video clip has a folder within bookmarks making it an easy process to review the thumbnails.

To get it all working the main enscript needs a little configuration which I found a little fiddly. Ahead of time you need to install a version of mplayer suitable for Windows, the installer I used was MPUI.2009-07-24.Full-Package.exe. This appears to have been superseded by MPUI.2009-10-12.Full-Package.exe which is available here (at least today - download locations seem to change quite often). Oliver directs you to the standard mplayer site which I found a bit difficult to navigate. Once mplayer is installed you need to configure the main enscript by editing it to include the location of mplayer.exe and the location of a suitably large temp directory. On my box the lines of the enscript are (note the double \\)

////////////////////////////////////////// Configuration ///////////////////////////

//Path to MPClassic.exe
mpclassic = "C:\\Program Files (x86)\\MPlayer for Windows\\mplayer";

//Tempfolder which will be used to extract the movies and create the thumbnails
expDir = "C:\\Temp";

//time interlace between to frames
//films under 1 minute
OneU = 5;

//films between 1 and 5 minutes
FiveU = 10;

//films between 5 and 30 minutes
BetweenFiveAndThirty = 20;

//films over 30 minutes
ThirtyU = 30;

///////////////////////////////////////////////////////////////////////////////////

Oliver's enscript can be found in the Guidance Software Download Center and comes with a Readme which needs reading.

Windows Photo Gallery

2009-09-20T11:42:00.000+01:00

Windows Photo Gallery is built in to all Vista editions and allows the management of photographs and other pictures together with the ability to carry out a number of basic photo editing tasks. Two forensic artefacts of this program are discussed in this post.

Original Images Folder

The program allows users to revert to the original picture with one click should they not like the results of their editing. This feature provides investigators with a very useful artefact. When a picture has been edited the original unmodified version is stored at

%LOCALAPPDATA%\Microsoft\Windows Photo Gallery\Original Images

The file name of this original unmodified version is renamed - the relevant Microsoft Knowledge Base Article details the file name construction

When the original unmodified version of the image is saved, the image file is renamed by using a combination of a unique ID and the original file name. The unique ID is determined by theSystem.Image.ImageID file property. If there is no System.Image.ImageID file property value, a GUID is created. The following is the new file name construction:
'{' + unique ID + '}' + '-' + file name
The following is an example of a renamed original file:
{198EB054-44E6-441e-87C8-9B29C5198DE6}-image1.jpg

To example this I have edited and renamed the Windows sample picture Toro-toucan.jpg (quite apt considering the forthcoming Arthur's day) using Windows Photo Gallery

The Original Images folder is created the first time a picture is edited with the application and is a hidden folder. From a forensic point of view we might need to identify the edited picture which may have been renamed. We can locate the edited picture by searching for the unique ID referred to above. Essentially take the original file name:
{1F7BA35C-33F2-499E-92A1-0FBE9477C8CA}-Toco Toucan in my example)

and strip it down to

1F7BA35C33F2499E92A10FBE9477C8CA

This value is embedded within metadata stored within the edited file known as an XMP Message block and also in one further location. Using FTK Imager we can see this value stored in the two locations within the edited file (click on screenshots to see a larger version)

In the second screenshot part of the XMP message block can be seen. The editing application is also detailed.

Pictures PD4

Windows Photo Gallery stores metadata about the pictures indexed by it in a database file Pictures.PD4 at the location

C:\Users\YourUser\AppData\Local\Microsoft\Windows Photo Gallery.

Tim Coakley's Simple Carver Suite contains a program Windows Photo Gallery Viewer to parse this file. I have found that substituting a test Pictures.PD4 file (in a Vista VM lets say) with your suspects Pictures.PD4 file can produce some meaningful results. I found that the best results can be achieved when the test Windows Photo Gallery is set to display tiles view. Although a blog discussing the transfer of Pictures.PD4 files from machine to machine suggests that the test machines Volume Serial Number needs to match that of the suspects. This can be done with the Windows Sysinternals utility Volume ID v2.0.

References
http://support.microsoft.com/default.aspx/kb/944370
http://blogs.msdn.com/pix/archive/2006/08/16/702780.aspx
http://www.adobe.com/devnet/xmp/pdfs/XMPSpecificationPart3.pdf
http://aaron-kelley.net/blog/2008/03/migrating-vistas-windows-photo-gallery-database/

Vista Volume Shadow Copy issues

2009-08-17T19:59:00.001+01:00

Volume shadow copies in Vista are often the elephant sat in the corner in many cases. We know they exist and we know they can contain lots of data, but we often choose to ignore them.

A recent case required some keyword searches and an examination of picture files. A the completion of the keyword search most of the hits were within files with names similar to

{bab9c293-d150-12dc-a44f-021d253da909}{3708876a-d176-4f38-b7bb-05036c6bb821}

The view pane within Encase 6.14 displayed the contents in a nice light blue colour which I now know is a new feature in 6.14 to indicate the contents of uninitialised files. The files were all located within the System Volume Information folder on the root of the volume and are the Vista Volume Shadow Copies. By default 15% of the capacity of the volume is allocated by Vista to store these copies. The C4P graphics extractor enscript carved most of the notable pictures out shadow copies also.

At this stage I have known examiners report their findings alluding to the fact that that the notable artefacts are within the file {bab9c293-d150-12dc-a44f-021d253da909}{3708876a-d176-4f38-b7bb-05036c6bb821}. In most cases I think you need to drill down further. In order to do this I mounted my Vista image with Encase PDE and used Liveview 0.7b to create a working VM using VMWare Workstation 6. Having logged into my suspects account I ran a command prompt as administrator and entered the command

vssadmin list shadows /for=c:\

This provided a nice list of available shadow copies. Having selected one I entered the command (updated 13th Jan 2010)

mklink /d c:\shadow_copy7 \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy7\

This created a symbolic link in the root of C which in Windows Explorer at any rate appears exactly like a shortcut to a folder. Clicking on it produced the error message shown below

I believed this error is probably generated by a permissions issue (SEE UPDATE BELOW), however I was not able to overcome it and Rob Lee over at Sans Computer Forensics suggests this methodology does not work. I think Jimmy Weg however has had some success with a program written by Dan Mares - VSS.exe. I therefore turned to ShadowExplorer version 0.4.382.0. This program allows the user to view the contents of Volume Shadow Copies that exist on any volumes within the installed system. The contents are displayed in an Explorer like view allowing the user to export out any file or folder to an export directory. I exported the User profile I was interested in to an export directory. Unfortunately it seems that only the Last Written date is preserved in this process and all other time stamps are tripped. I then tried to copy this export directory out of the VM to my workstation and encountered errors (probably due to files within the profile with illegal windows file names). To overcome this I zipped up the export directory and copied the zip out of the VM. Once unpacked I then added the exported folders into Encase as single files and created logical evidence files from them.

Having done this I was able to resolve most of my keyword search hits and pictures to actual files as opposed to being simply within a volume shadow copy.

UPDATE 13th January 2010

The issue I had with the mklink command was due to a missing \ but not the trailing slash referred to in some comments below. The correct command is

mklink /d c:\shadow_copy7 \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy7\

References
http://blogs.msdn.com/adioltean/archive/2008/02/28/a-simple-way-to-access-shadow-copies-in-vista.aspx
http://blogs.sans.org/computer-forensics/2008/10/10/shadow-forensics/

You wait all day for a bus then two come along at once...

2009-08-02T18:02:00.000+01:00

Probably not an entirely accurate title but I came across two enscripts the other day both of which are aimed at quickly triaging the results of a comprehensive Internet History search. Users of this functionality within Encase version 6 will know that often you can be faced with reviewing hundreds of thousands of entries on the records tab. Many times all you need is evidence of user inputted search terms. There are conditions available to start sorting the wheat from the chaff however it is difficult for these conditions to be totally focussed due to the variation in url formation. This is where both enscripts come in as they are both designed to parse the actual search term used from a variety of search engine urls.

Searchterms V 1.1 parses out the search term used and where possible the time and date it was carried out into note bookmarks. The enscript has been written to support a claimed 145 separate search engines.

Internet Search Term Finder parses out unique search terms to Log Record bookmarks and stores the term along with its associated url. The script is in fact an Enpack so it is difficult to determine exactly how it works, however it seems to base its search on elements from the query url. A neat feature is that it is configurable, allowing the addition of a new prefix (to the query string) to cater for a different or new search engine.

Within an XP Pro SP3 VM I carried out a series of searches utilising the Firefox v3.0.11, Internet Explorer v8, Opera v9.64 and Safari v4.0.2 browsers. I ran the Search for internet history Comprehensive search option within Encase 6.14 and established that all my searches had been parsed into the records tab, with the exception of those carried out with Safari v4.0.2. It turns out that Encase 6.14 does not support parsing internet history from this version of Safari.

I then ran both Enscripts and can report that both parsed out my test search terms from the records tab. The results can be viewed within bookmarks. For me the output of the Internet Search Term Finder is preferable and it usefully creates a Log Records bookmark which allows the easy export of results into a spreadsheet. Both successfully hit the spot in respect to users quickly reviewing search terms within internet history.

Update 15 Sept 2009
Dan Fenwick has kindly updated his Internet Search Term Finder (to v1.1.1). The script now can remove duplicates and separates the results by device. Even more useful - thanks Dan.

Link Files within System Restore Points

2009-07-22T06:29:00.000+01:00

A recent case involved the download of a contraband file and I was asked to establish what happened just before the download in order to try and establish who was responsible. This scenario is fairly commonplace and I usually start with a timeline analysis of the file system activity around the event in question. An invaluable enscript for this is Geoff Black's Timeline Report which can also be found within the Guidance Software Enscript Resource center. The html report produced by this script is particularly cool.

In my case my analysis showed there were a number of link files in a number of system restore points all created at a time and date just before the download. They were all named in the form A000XXXX.lnk (xxxx being a variable number) and I could see from a rough and ready examination of the data that they all pointed to one particular file saved on the users desktop. As these link files were stored within restore points the first hurdle to overcome was to establish the link files original name and path. This information is stored within the changelog files of each respective restore point. Manually searching through this file for the restore point file name (e.g. A000XXXX.lnk) will reveal the files original path. There used to be an enscript for parsing the changelog files but it was written for version 5, however I was able to track down a version that worked in version 6 at Paul Bobby's excellent blog/web site (this enscript can also be found within the Guidance Software Enscript Resource Center). The changelog files contain a lot of information and all I really needed was the original filename and path - the scripts output may be a little bit of an overkill*. Another utility out there is the Mandiant Restore Point Analyzer. I used this utility to determine the original paths and file names.

All of the link files related to one link file stored within a users Recent folder. In my case the file the link file linked to was created and stored upon the desktop, thus causing the initial link file to be created. Over a period the target file was opened and additional information written into it, thus causing the link file to be updated. Harry Parsonage's paper on link files illuminates this further.

I copied the link files out of my image and loaded them into Sanderson Forensics Linkalyzer. This program decodes and displays the contents of link files into a grid much like a spreadsheet ( I was going to post a screenshot but sanitising the contents became too much of a pain) and very quickly allowed me to see that the target file was being regularly accessed and modified. Because the target file size is also stored within the link file I could also see that the file size was growing over time. The program produces good reports and has many other abilities beyond the scope of this blog post, but in short I thoroughly recommend it.

Now as far as my case was concerned the target file was clearly linked to the suspect and it proved worthwhile delving into those restore point link files.

References
http://computerforensics.parsonage.co.uk/downloads/TheMeaningofLIFE.pdf
Forensic Analysis of System Restore Points in Microsoft Windows XP

*Now what would be really useful is an enscript that simply parsed out the original path and filename of only all user selected (blue checked) files sitting in restore points. I would envisage the output to be a three column csv file - Current Filename, Original Path and Filename, Restore Point Creation Date.

Video Triage

2009-07-10T14:45:00.000+01:00

Paul Sanderson's VidReport has been referred to here and there lately. C4M also is regularly brought up in conversations I have with people (such an interesting life I lead). Triage is certainly the flavour of the month right now. So I thought it worth writing a few lines about my recent experiences of triaging videos.

I have often voiced the opinion that reviewing a couple of hundred video files in a case is not that bigger a deal and on that basis I have not been too keen on using C4M. Anyhow I've just had a case with about 170 video clips to review and thought it would be a good case to try out the video triage approach. My normal approach is to use VLC as a file viewer in Encase to preview each video. This took about an hour and a half (some of the videos were quite good ;-) ).

I then used John Douglas's video triage program (which I think he supplies free to LE) to review the same video clips. To use this program you copy out the clips you wish to review and point the program at the folder containing them. It processes each clip by taking a screen capture at a configurable interval and putting each screen capture into a subfolder named after the videos file name. Once the program has processed all the clips you will have a sub folder for each one, containing the screen captures. I then simply dragged the folders into Encase as single files and previewed the contents of each folder in gallery view. I previewed all the clips in fifteen minutes. My scepticism of video triage was clearly unfounded.

Bing in-line video previews

2009-06-18T11:09:00.000+01:00

The new Microsoft search engine Bing has been in the news lately.

One of the facilities it provides is a video search which in itself is old hat, however the results page features in-line video previews. A user can turn safe search off and perform a search which results in a screen of thumbnails of the located videos. Hovering the mouse over a thumbnail results in a short preview of the video being played within the thumbnail.

The thumbnail videos are cached as FLV (flash video) files however the interesting feature is that the URL host of the flv files in my early tests was ts3.images.live.com. The ts3 part was variable. Microsoft are processing the video using Smart Motion Preview technology producing effectively a trailer of the most relevant parts. Microsoft on or about 12th June 2009 began to serve all explicit video smart motion previews from ts4.explicit.bing.net. The ts4 part is variable.

These in-line video previews allow the viewing of contraband material without leaving a significant footprint. At least for Bing the search query is saved within the browsers internet history and the smart motion preview is cached as a FLV file with the word explicit helpfully added into the cached items url.

The video search at ask.com also provides in-line video previews however these previews seem to be streamed - another kettle of fish altogether!

USB Prober

2009-05-29T11:03:00.000+01:00

From time to time the subject of linking USB flash drives to a particular PC crops up. A week or so ago I saw a post on the Guidance boards touching on this subject and chipped in with a link to a paper referencing Harlan Carvey's original research in this area. The nub of this issue is that many USB flash drives have a unique device serial number which is recorded into the registry of Windows boxes that have hosted said flash drive.

When investigating this issue establishing a USB flash drives device serial number may be achieved by utilising a utility such as UVCView. In our lab we use the Tableau T8 USB write blocker to do this. When checking out the subject again prior to posting to the thread on the Guidance boards referred to above I discovered that my Mac Book Pro also has a utility that can establish a USB flash drives device serial number. The utility is an application called USB Prober which is installed as part of the XCode developer tools (which can be found on the separate DVD along with the Mac OS disc for those that have a Mac).

To use USB Prober for this purpose the Mac needs to configured so that it does not mount the USB flash drive. To do this disk arbitration needs to be turned off. In Leopard in terminal the command is:

sudo launchctl unload /System/Library/LaunchDaemons/com.apple.diskarbitrationd.plist

Once disk arbitration simply launch USB Prober (via spotlight is the quickest way) and drill down to the device serial number.

References
http://www.macosxforensics.com/Technologies/DiskArbitration/DiskArbitration.html
http://scissec.scis.ecu.edu.au/conference_proceedings/2007/forensics/23_Luo_Tracing_USB_Device_artefacts_on_Windows_XP.pdf
http://developer.apple.com/documentation/MacOSX/Conceptual/OSX_Technology_Overview/Tools/Tools.html

Helix Imaging PC

2009-05-05T07:29:00.002+01:00

When we upgrade our Forensic Workstations we cascade the older machines onto administrative and imaging tasks. One particular ex Forensic Workstation had supported a tape drive for a year or two but now was about to become totally redundant. Instead of suffering this fate I decided to dedicate it to running Helix. The box itself is a Supermicro chassis sporting a Supermicro X6-DAL-TG motherboard, twin Xeon Nocona 3.4 ghz processors, 2GB ram and a hot swap drive bay.

I had read Andre Ross's blog post Installing Helix 2008R1 and Jess Garcia's How to install Helix to Disk webpage and decided that installing to hard disk was the way to go.

The process I followed to do this successfully (guided by Andre Ross's post in the main) was:

Equip box with an unformatted wiped hard disk - using a partitioned (with ext2 and linuxswap) disk caused the installation routine to hang.
Boot box to Helix 2008R1 CD and commence installation by going to System->Administration->Install
At the point the installer hangs (Who are you screen) click cancel and then quit
Commence installation routine again and create a user - I called mine Helix
Configure Network Adaptor to connect to the internet via System->Administration->Network
Launch Update Manager via System->Administration->Update Manager and update all packages.
Applications->Forensics & IR->Root Terminal
:~#apt-get install smbfs
:~#apt-get install winbind

Part 1 of the job is done. A little bit of configuration is needed to make the machine more usable in it's main role as an imaging machine. I am not a Linux guru so apologies for the Janet and John approach for those that are. Also my imaging machines are in a secure environment and not normally connected to the internet so I felt relaxing security a little may be OK.

Relaxing Security

System->Administration->Login Window
On the Security tab you may wish to enable Automatic Login for the Helix user
Applications->Forensics & IR->Root Terminal
:~# nano /etc/sudoers
Use arrow keys to scroll to end of file then type
Helix ALL=(ALL) NOPASSWD: ALL
(presuming helix was the name of the user account you created, if not substitute helix with the name of your account)
Type CTRL+o to save then press enter then type CTRL+x to exit nano text editor. The syntax is critical - if sudoers is messed up your OS may not boot. The reason this is done is that most of the applications we wish to use run at root. However user accounts do not have root privileges. This is overcome by using the sudo command which periodically requires you to enter a password which is a pain. Editing the sudoers file as shown above removes the requirement to enter a password when sudo is used.
By default there are three icons in the panel (like Windows Quick Lauch) on the taskbar at the top of the desktop (Firefox, help and terminal). Right click on Terminal and Remove from Panel.
Access Applications->Forensics & IR->Root Terminal in the menu and right click and select Add to Panel

Imaging Applications

I work in an Encase shop so I am going to concentrate on applications that image to EWF format (aka e.01 files). There are currently two applications installed that do this - Linen and EWFacquire.

Linen

Linen needs some configuration to run from the shortcut Applications->Forensics & IR->Linen. This shortcut (I think the proper linux terminology is launcher) runs a script called sl in /usr/bin. sl needs editing.

Applications->Forensics & IR->Root Terminal (or click on Root Terminal in the Panel)
:~# nano /usr/bin/sl
Use nano to delete the line
cp /cdrom/IR/bin/linen /usr/local/bin
Type CTRL+o to save then press enter then type CTRL+x to exit nano text editor.

At this stage Linen does not reside in /usr/local/bin - we need to put an up to date copy there.

On a Windows box where Encase version 6 is installed copy the Linen file from the root Encase folder within Program Files to a thumb drive.
On the Helix box copy Linen from the thumb drive to /usr/local/bin as follows:
Launch root terminal from panel on task bar and mount your thumb drive by clicking on it's icon on the task bar and selecting Mount
:~# cp /media/sdc1/linen /usr/local/bin (where sdc1 is your thumb drive)

Linen should now be launchable via the menu. But in true windows style I created a desktop shortcut by right clicking the Linen menu item and selecting add launcher to desktop.

EWFacquire

EWF Acquire is installed and will run from the root terminal. This program is part of the libewf project. The syntax is

ewfacquire /dev/sdb

where /dev/sdb is the drive to be imaged. Again I created a desktop shortcut by:

Right clicking on the desktop and selecting Create Launcher
Change the type to Application in Terminal
Set the name appropriately
In the command box type sudo /usr/bin/ewfacquire /dev/sdb
Click OK

It is probably worth noting that you would not want to launch EWFacquire from the desktop launcher unless you had established the path of each drive by typing fdisk -l into the root terminal.

Guymager

Guymager is another imaging tool that utilises Libewf. It is controlled from a GUI and is a desirable addition to our imaging tools. I intend to do a mini review of it along with steps I have carried out to validate it in a forthcoming blog post. It is not installed on the Helix CDRom but can be installed to our hard disk installation.

Launch a Root Terminal
:~# nano /etc/apt/sources.list
Use arrow keys to scroll to end of file then type deb http://apt.pinguin.lu/i386 ./
Type CTRL+o to save then press enter then type CTRL+x to exit nano text editor.
Whist still connected to internet type in root terminal
:~# apt-get update
:~# apt-get install guymager smartmontools hdparm libewf-tools

Once the process is completed guymager can be launched from a root terminal. Again I created a desktop shortcut by:

Right clicking on the desktop and selecting Create Launcher
Change the type to Application in Terminal
Set the name appropriately
In the command box type sudo /usr/bin/guymager
Click OK

Guymager utilises a configuration file - guymager.cfg. For my setup I wanted to make some changes. The program advises that changes should be made to local.cfg, however I did not have much success with this. I edited guymager.cfg with nano:

Launch a Root Terminal
:~# nano /etc/guymager/guymager.cfg
and modify entries to the following
Language='en'
EwfFormat=Encase5
EwfCompression=Best
EwfSegmentSize=1500
and in the Table LocalDevices area add a new line beneath the line of ------------
containing the serial number of the hard disk drive where Helix is installed
e.g. '1ATA_Maxtor_6B300S0_B605MV0H'
The best way to establish the serial no. is probably with Guymager itself.
Many other changes can be made as documented within guymager.cfg
Type CTRL+o to save then press enter then type CTRL+x to exit nano text editor.

Adepto

Although Adepto does not image to EWF files I know some people use it. Some changes need to be made to get it to work.

Launch a File Browser with root permissions by launching a root terminal and typing nautilus
Use the file browser to navigate to /home/helix (helix being the name of the user account I created during the installation routine - if you used another account name navigate to /home/theAccountNameYouUsed )
Right click or use the edit menu to create a folder then name it Adepto
Double click Adepto and create a subfolder within Adepto called Logs
Right click on Logs and Make Link
Right click on the resulting Link to Logs and Cut
Navigate to /usr/local/adepto and paste your link file
Right click on the existing Logs file and delete
Rename Link to logs to logs

Adepto should work now.

Some Networking Stuff

In our lab we image to a file server running Microsoft Windows Server 2003. When I have used the Helix CDs in the past it was always a pain to image to an attached hard drive then transfer the image to the file server later. I wanted the Helix Imager to image direct to our file server and be part of our Windows Workgroup.

To do this:

via System->Administration->Network configure to connect to your internal network
on the windows file server create a share (I called mine Helix) and create a user named Helixuser (having done this you can apply appropriate security to this user at the Windows end)
Create a mount point to the windows share by:
Launch a Root Terminal
:~#mkdir /media/helix
:~# nano /etc/nsswitch.conf
modify (add wins prior to dns) the following line to read

hosts: files mdns4_minimal [NOTFOUND=return] wins dns mdns4

Type CTRL+o to save then press enter then type CTRL+x to exit nano text editor
:~# nano /etc/fstab
Append the line below to the end of the fstab file

//server/Helix /media/helix cifs username=user,password=*,iocharset=utf8,file_mode=0777,dir_mode=0777 0 0

where server is your server name, Helix is the name of your Windows share, helix is the name of the linux mount point, user is the name of an account on your Windows server and * is substituted for whatever your password is.
Type CTRL+o to save then press enter then type CTRL+x to exit nano text editor
:~# mount -a
Configure the way the Helix Imager box is recognised within our Windows Workgroup
at the root terminal :~# nano /etc/samba/smb.conf
Within the global settings area modify entries to the following
workgroup = THENAMEOFYOURWORKGROUP
server string = %h

Now that a mount point has been created to your windows share specifying /media/helix as the path to image to in Linen, EWFacquire or Guymager will output the image to the Windows File Server.

Tableau T9 Firewire write-blocker

2009-04-25T23:22:00.001+01:00

Most forensic practitioners will prefer to use hardware write-blockers over software. However when the device you wished to image only had a firewire interface the choice was limited. Hardware write-blockers for firewire didn't exist. Now somewhat late in the day Tableau have introduced the Tableau T9. This write-blocker will allow you to image firewire external storage drives as well as Apple Macs booted into target disk mode. Given the increase of Macs submitted to our lab I can see the T9 becoming very useful. Data Duplication will sell the T9 in the UK for around £240.

Facebook revisited and other chat related stuff

2009-04-22T17:59:00.000+01:00

My blog post about facebook chat generated a lot more email than usual.

In particular Jad Saliba wrote about a program he has written to search for and report on facebook chat. Jad's program is called Internet Evidence Finder and essentially at this time it searches for Facebook chat, Facebook pages, Yahoo chat and MSN chat. Jad points out that the program may be useful in a non Encase shop and I agree. In fact it will be useful anywhere as it did a very good job.

I have had some fun testing it today and found that it parses all the messages that my two previously documented methods had found. I used the program by mounting the drive image I wished to search with Encase PDE and then running the program across the mounted drive. On my box the search ran at a speed of about 27 MB/sec. The resulting spreadsheet was nicely formatted and gave the Physical Sector of each hit. Jad's program is freeware and can be found at http://www.jadsoftware.com.

With respect to MSN chat and the other chat clients Jad's website deal with what can be achieved. In testing I am running right now with MSN a large number of false positives have been found however this is probably the nature of the beast.

Now before someone mentions tool validation my view is that I don't validate my tools - I validate my results. Generally I do this with dual tool verification as in the example above.

Till next time...

Adobe Flash Player Local Shared Objects

2009-04-08T13:29:00.000+01:00

The value of cookies and other internet history related artifacts is well known. Not as widely commentated on are Local Shared Objects created by Adobe Flash Player. They have a .sol file extension and on the vista box I am looking at at least they are stored at:

\Users\your user\AppData\Roaming\Macromedia\Flash Player\#SharedObjects\

These Local Shared Objects are data files that can be created on a computer by visited websites and in many respects are similar to cookies. It appears however that the conventional forensic software I use to analyse internet history ignores these files (I use Netanalysis and Encase v6 Comprehensive Internet History search).

To parse .sol files into a more readable form I use the SharedObject Reader plugin of FlashDevelop3.

References
http://www.adobe.com/products/flashplayer/articles/lso/
http://en.wikipedia.org/wiki/Local_Shared_Object

Garmin nüvi 200 Sat Nav device

2009-04-03T10:44:00.000+01:00

This device has two memory chips hard wired onto the internal pcb, therefore the only regular means of accessing this memory is via the USB port. These sat nav devices will act as mass storage devices when connected via USB. I imaged one whilst connected to a Tableau USB write blocker. Please note that the time the device is switched on is recorded within current.gpx referred to below.

There are few human readable files most notably current.gpx. This file contains the users home location and user selected favourites along with the location of a number of Garmin offices. If a user saves a favourite from a location on a map the favourite will be entitled 001, 002 and so on.

There are a number of ways to investigate the contents of current.gpx. Effectively it is an xml formatted file which I use Microsoft XML Notepad 2007 to review. You can also use a utility such as EasyGPS or open the file with Google Earth.

To report the contents of current.gpx I use Microsoft Excel 2007. In order to do this successfully change the file extension to xml and use the xml data import facility (Data/ From Other Sources/ From XML Data Import) allowing Excel to create the schema. You will end up with a nicely formatted table.

Recently Found locations unfortunately do not appear to be saved within the user accessible memory.

The hidden service menu of the device can be accessed by turning on and then holding a finger on battery symbol on screen for 10 seconds. It is possible once in this menu to interface with the device via USB without it behaving as a mass storage device. Garmin USB drivers are required to do this. I am not sure whether this will be useful forensically at any stage.

UPDATE
A later post relating to a StreetPilot C510 may be some help.

References
http://en.wikipedia.org/wiki/GPS_eXchange_Format
Download nuvi 200 manual
http://www.techonline.com/showArticle.jhtml?articleID=210601020

Seagate Barracuda Firmware problems

2009-03-24T14:45:00.000Z

As has been widely reported elsewhere many Seagate Barracuda hard disk drives have faulty firmware that causes them to effectively freeze.

In our lab we have 12 of the affected Seagate Barracuda 7200.11 1TB drives and 50 Seagate ES.2 1TB drives. Two of the 7200.11 have failed with symptoms that suggest that faulty firmware was the cause. I have flashed the firmware of all of the affected working drives and hope that I can look forward to a long fault free period.

One of the failed 7200.11 drives was one third of a Raid 0 array in one of our forensic workstations. This workstation needed an OS reinstallation (onto a separate single drive) and temporarily some data on the OS drive (normally backed up to the Raid) only existed on the Raid. During subsequent software installations a number of reboots were required triggering the firmware bug and a failed raid. Oh bother!

Normally data on our Raid 0 arrays is backed up but due to the aforementioned OS reinstall there was a small amount of data that lost. I therefore had to find a way to unfreeze the locked Seagate Barracuda. A considerable trawl of the internet led me to this post. Gradius2 details a fix involving a significant amount of down and dirty electronics and low level hard drive programming. Not having all the necessary adapters/serial cables etc led me to call Disk Labs. They quoted £1000 - that was a no go then. Luckily Hugh Morgan at Cy4or successfully repaired the drive broadly following Gradius2's fix for significantly less. I reintroduced the drive to the two other drives in the Raid 0 array and Bob's Your Uncle!

Monetizing Helix

2009-03-22T20:50:00.000Z

The forensics community has benefitted from the free Linux forensic distro Helix3 for some time. This distro was developed by Drew Fahey and distributed via e-fense.com (archived Helix 3 website). I suppose, like many free things, the issue of how to you support it and develop it when you are not making money from it became an issue for e-fense. I was under the impression that a revenue stream was available via Helix3 training courses (run by CSI Tech in the UK). I know that both Nick Furneaux and Jim Gordon were very busy with these courses, and having attended one myself, I thought they were a great success.

Anyhow it seems that training provision wasn't enough. Late 2008 e-fense invited e-fense helix forum members to make donations. Unsurprisingly take up wasn't that great. This resulted in a slightly hectoring email from e-fense announcing that Helix3 was now only available to those who subscribed for access to their forum. The subscription is around US$20 per month. So be it but as someone who has already paid circa US$1000 for a training course to use a product I cannot now download without subscription I am left feeling slightly disappointed.

Nothing stands still in this arena however. I have posted in the past about WinFE and some subsequent comments led me to a Grand Stream Dreams blog post written by Claus Valca. He referred to two free forensic Linux distros:

Perhaps one of these is the new helix?

It seems one or two others have commented on the same subject -it seems they are not planning to subscribe either.

I noticed a bit late in the day that there is an extensive thread over at Forensic Focus about this issue also.

Facebook Chat Forensics

2009-03-20T10:24:00.003Z

Background
Facebook has a built in instant messaging facility which has grown in popularity along with the Facebook social networking site itself. Many cases involve potential grooming offences in which the use of instant messaging needs to be investigated.

The instant messaging facility creates a number of artefacts which are easily found and I know have been commentated on elsewhere. The purpose of this blog post is to suggest a methodology to automate the discovery and reporting of Facebook messages.

For those who have not looked at this area in detail yet messages are cached in small html files with a file name P_xxxxxxxx.htm (or .txt). These messages can be found in browser cache, unallocated clusters, pagefiles, system restore points, the MFT as resident data and possibly other places. It is possible for the messages to be cached within the main Facebook profile page (although I have never seen them there - the main facebook page does not seem to be cached that often).

An example of a message is shown below:

for (;;);{"t":"msg","c":"p_1572402994","ms":[{"type":"msg","msg":{"text":"Another Message","time":1237390150796,"clientTime":1237390150114,"msgID":"3078127486"},"from":212300220,"to":1123402994,"from_name":"Mark PPPPPP","to_name":"Richard XXXX","from_first_name":"Mark","to_first_name":"Richard"}]}

The bulk of the message is in fact formatted as JavaScript Object Notation normally referred to as JSON. The format is a text based and human readable way for representing data structures. The timestamps are 13 digit unix timestamps that include milliseconds - they can be divided by 1000 to get a standard unix timestamp.

Although keyword searches will find these messages they are difficult to review particularly if you are only interested in communication between selected parties. Having found relevant hits you then have to create a sweeping bookmark for each one. For these reasons I follow the following methodology.

Suggested Methodology

Create a Custom File Type within the Encase Case Processor File Finder module entitled Facebook Messages using the Header "text":" and the footer }]} making sure GREP is not selected.

Run the file finder with the Facebook Messages option selected.
When the file finder completes you will have a number of text files in your export directory (providing there are messages to be found).
These text files are in the form of the example above. They do not have Carriage Return and Line Feed characters at the end of the text. We need to remedy this by utilising a DOS command at the command prompt.
At the command prompt navigate to the directory containing your exported messages (please note Encase creates additional sub directories beneath your originally specified directory).
Then run the following command:
FOR %c in (*.txt) DO (Echo.>>%~nc.txt)
This command adds a Carriage Return and Line Feed to the end of the extracted message.
Next we want to concatenate the message text files into one file using the command at the DOS prompt: copy *.txt combined.txt
Alternatively create (or email me for) a batch file that executes these two commands direct from windows.
An additional file combined.txt will be created in your export directory.
Launch Microsoft Excel and instigate the Text Import Wizard specifying Delimited with the Delimiter being a comma and the text qualifier " .
Put the data into your worksheet (or cell J3 of my pre-formatted worksheet).
All that's needed now is to tidy up the worksheet with some Excel formulas the full details of which can be found within my example pre-formatted worksheet. The formula to process the time values (which are Unix time stamps) is (RIGHT(K2,13))/1000/86400+25569 where K2 is the cell containing the source time data.
Perform a sanity check and remove obviously corrupt entries.
It can be seen below that after applying a data sort filter you can sort by time or user.

The spreadsheet also allows you to de-duplicate the found messages. In my recent case over half the recovered messages were duplicates. In Excel 2007 these duplicate (rows) are easily removed (Data/DataTools/Remove Duplicates). In Excel 2003 an add-in called The Duplicate Master will do this for you.

Further Thoughts
Non Encase users may be able to use an alternative file carver (e.g. Blade) to carve out the messages. I am sure that the header and footer could be refined a bit to reduce false positives, however for me the ratio of legitimate versus false positives is OK. UPDATE 22nd April 2009 - non encase users may wish to look at my more recent post.

I have the pre-formatted spreadsheet in template form. Please email me for a copy (with a brief explanation of who you are - thanks).

To further investigate the data you recover you may wish to check out http://www.facebook.com/profile.php?id=xxxxxxx. Just substitute the xxxxx with the User ID's you recovered.

Enscript Method
I have collaborated with Simon Key and now have an enscript to parse out JSON objects including messages. It outputs to a csv spreadsheet and in my tests parsed 160GB in about an hour. It might not be as tolerant of corrupt strings as the method detailed above. The script will only run in 6.13 or newer. I have a template that tidies up the formatting of the csv- email me if you want a copy.

References and thanks
http://coderrr.wordpress.com/2008/05/06/facebook-chat-api
http://video.google.com/videoplay?docid=6088074310102786759
http://json.org/
http://en.wikipedia.org/wiki/JSON
http://www.wilsonmar.com/datepgms.htm#UNIXStamp

Thanks to Glenn Siddall for sparking my interest and providing me with some notes of his research.
Thanks to Mark Payton for his assistance in researching this.

The need for speed

2009-03-19T07:18:00.001Z

We are lucky in our lab that our workstations are upgraded on a regular basis so once in use we don't often make many changes.

The most important bits of my current spec are as follows:

Supermicro X7-DWA-N fitted into a Supermicro CSE-733TQ-645 chassis
Two Intel Xeon X5482 processors
16GB DDR2 800 Ram
Western Digital 300GB VelociRaptor10,000 rpm hard drive for the OS
3 x 1TB Samsung HE103UJ Spinpoint F1 hard drives in a RAID 0 array
Microsoft XP 64 bit
256MB ATI FirePro V3700 GPU

At one time our primary forensic software Encase would max out the processor when carrying pretty much any process. With the advent of multi-core dual processors we aim to max out one core (which on my box is 13% cpu utilisation in task manager). As processors get faster and faster I have noticed that often the CPU core is not maxing out. Something else is slowing us down! We store our Encase evidence files on the Raid 0 array (and just before someone posts a comment about the lack of data resilience etc., the way our lab is set up all the data on my Raid 0 array is mirrored elsewhere). We do this for speed and capacity. When Encase (and most other forensic utilities for that matter) is processing it has a voracious appetite for data. Just look at the Read bytes value in Task Manager. The multi-core processors allow us to run other forensic programs (FTK, Netanalysis hstex etc etc) along with Encase, we can even run other instances of Encase, and because we can - we do. The net result of all these programs running is that they compete to read data from the Raid 0 array in my case (and from wherever you store yours in yours) - the net result once your data storage is maxed out is things slow down. It follows then that performance can be increased by having faster data storage.

One way to achieve this would be faster hard drives. We use sata hard drives for capacity reasons and to an extent cost. SAS hard drives are faster but don't provide the capacity. So as things stand three hard drives in a Raid 0 array was the best that could be done. I decided to see how I could make some improvement.

Currently the three hard drives (and the OS drive) connect to the Intel ESB2 raid controller integrated on the motherboard. Conventional wisdom would have it that by adding a fourth hard drive to the raid 0 array would speed things up.

HD Tach details an average sequential read speed of around 200 MB/s for a three drive array utilising the default stripe size (128kb) with NTFS formatted with the default cluster size.

Adding a fourth drive slowed the sequential read speed to around 180 MB/s.

I tested a variety of different stripe sizes and aligned the partitions but came to the conclusion that the Intel ESB2 controller just did not scale up to four drives very well. The arrays were created via the utility accessed via the controller bios during boot up. This utility is very basic and does not allow much configuration. Intel also provides a Windows utility called Intel Matrix Storage Console. When running this utility I found that by default Volume Write-back cache was disabled. Enabling it made a significant improvement.

Conventional wisdom has it that a hardware Raid controller would improve performance over the Intel ESB2 and in my testing this seems to be the case. I have used an Areca 1212 PCI-E raid card and achieved a sequential read speed of over 600 MB/s.

This array has four 1TB sata hard drives with a 64kb stripe, is partition aligned at 3072 sectors and has one NTFS volume with the default cluster size. Using Syncback to write to the array from our file server across a copper gigabit ethernet network produces some pretty impressive network utilization stats.

Yahoo mailbox

2009-03-08T13:45:00.000Z

An MLAT request brought CD-R to my door recently. The OIC informed me that the CD contained a Yahoo mailbox but wanted my help because he could not read them. I found that the CD contained a tar.gz file.

Once this archive was unpacked I saw it contained two very large text files. These files were generic Mbox files. The next problem was how to view the contents. I found that Apple Mail would happily import Mbox files (File/Import Mailboxes) however I live in a mainly windows world so needed a Windows method for the OIC to preview the emails.

Thunderbird came to mind, however although this program uses the mbox format for its mailboxes it does not offer an easy way to import them. I did track down an extension to Thunderbird that provided this functionality but it only worked on one of my two mbox files. I also found that Opera 9 would also import my mbox files.

The problem with both Thunderbird and Opera is that the boxes available to the OIC in this case, and our customers in general, mostly do not have these programs installed. Ideally a way of getting the email messages into Outlook Express would be the best bet. The solution to this is provided by using the Mid Michigan Computer Forensics Group's M2CFG Yahoo! Email/Text Parser. This program parses out the email messages into .eml files which can be dragged into Outlook Express (and a number of other Email clients).

As it turned out the two mboxes I had extracted for the OIC were so full of emails with attachments that it was too complicated for him to process efficiently. So they came back to me to investigate. I added the mbox text files into Encase v6.12.1 and searched for email with the mbox option selected which resulted in Encase parsing out the emails and attachments very well. Reporting them was another matter!

Corel Paint Shop Pro Photo X2

2009-02-02T17:28:00.035Z

Paint Shop Pro has a slightly longer title nowadays. From a forensic perspective the older version had a thumbnail cache stored within .jbf files. Tim Coakley wrote a tool for parsing the contents out. I stumbled across this new version when I saw some subfolders in my suspect's My Pictures folder entitled 2008.01.02 and a subfolder entitled Thumbs at the path C:\Documents and Settings\username\Local Settings\Application Data\Corel\Thumbs. This folder had a number of jpgs within it with filenames in the format yyyy.mm.dd.jpg.

The software can be installed as a trial. It has two components that leave good evidence - an Organizer that creates a thumbnail database of selected folders (by default the current users C:\Documents and Settings\Username\My Pictures, C:\Documents and Settings\All Users\My Pictures and My Corel Shows) and the Corel Photo Downloader.

Organizer
When the program is first run it will catalogue all supported picture types in the default folders. This allows the user to view a thumbnail gallery of the catalogued folders within the program. The thumnails are created and stored at the path:

C:\Documents and Settings\Username\Local Settings\Application Data\Corel\Thumbs

These catologued thumbnails retain the filename of the original picture. Metadata is stored in a database imagedb.db.

Corel Photo Downloader
This utility was responsible for creating some good evidence in my recent case. It is installed as part of Corel Paint Shop Pro Photo X2 and is intended to automate the download of photographs from cameras, flash media cards and CDs. The following dialogue box shows the relevant configuration settings:

Photos downloaded with this utility form the folders and files I had found.

Thumbnails of them are catalogued as part of the Organizer function - a function your suspect may be unaware of.

Thumbnail issues

When Coral Paint Shop Pro Photo X2 is running thumbnails are pruned if the underlying source folders/files are deleted. However if the program is not run after the underlying source folders/files are deleted the thumbnails remain. In my case it appears that the trial expired which allowed a considerable number of thumbnails to remain at the path C:\Documents and Settings\username\Local Settings\Application Data\Corel\Thumbs.

Imagedb.db

Both the organizer and the photo downloader utilise a SQLite3 database to store metadata in relation to the catalogued pictures. The database can be viewed using the SQLite database browser. It is possible to walk through the database to establish the original filenames of any thumbnails and the source folder. Other useful information is stored in the database including program configuration. In my case the date of download was useful.

Figure 1
Details pictures downloaded (in this case from a casio camera). The highlighted picture has been given an image ID number - 5210 and has a Download ID number 1.

Figure 2
Image ID 5210 has been allocated the filename 20090129_3.jpg and is stored in Folder 851.

Figure 3
Folder 851's entry detailing the folder name and path

Figure 4
Showing Download ID number 1's download date.

Encase v6 Comprehensive Internet History Search and Firefox Mork Databases

2009-01-14T16:27:00.032Z

Encase version 6 now includes considerable functionality in recovering Internet History records for a number of browsers.

In a recent case using v6.12.1 I ran a search for internet history with the Comprehensive search option selected. My results included relevant hits in unallocated clusters which Encase attributed to Mozilla History/Forms. The results are recorded within the Encase Records tab and when highlighting a record some data was highlighted in the view pane which made me scratch my head - and luckily loosen some cobwebs in a long ago abandoned area of my brain. What was Encase showing me - I wasn't sure but my brain was telling me it was something to do with Mork.

Click on screenshot 1 to see a larger image

My suspect was using Firefox version 1.8. Firefox can save (subject to user configuration) information entered into web forms and the search bar to make form filling and searching faster. This information is saved in a file known as formhistory.dat. Encase had found data within unallocated that was a fragment of a deleted formhistory.dat file. In this version of Firefox formhistory.dat contained a Mork database. Encase had highlighted in the view pane what I will loosely refer to as the address of the data that it had parsed out.

[8(^83^83)(^82^8A)]

To understand this better we need to look more closely at all the data in the database. Because the hit was in unallocated I needed to find the start of the deleted formhistory.dat file. The file signature of the file is highlighted in the screenshot below

Click on screenshot 2 to see a larger image

so I scrolled up in the view pane, found this header and swept down to the footer (which is the } after the last address) and exported out the data as a file

Click on screenshot 3 to see a larger image

Open the file with Notepad and find/replace all the $00 strings with an empty string. The file now looks a lot more readable

Click on screenshot 4 to see a larger image

In the example we are working through the address we are interested in is

[8(^83^83)(^82^8A)]

which can be seen shown in red at the bottom of Screenshot 4 above. This address is a row within the Mork database. The row is delimited with open and close square brackets [] and is made up of a Row Object ID followed by a series of cells delimited by open and close brackets (). The cells contain a column name and a value.

The column names are defined within a dict delimited with <> shown in green at the start of Screenshot 4 above. It can be seen that two column names are relevant here name and value, given object ids of 83 and 82 respectively.

The values are defined within the next dict down. The one relevant to our address are shown in red.

[8(^83^83)(^82^8A)]

decodes to

[8(^Name^searchbar-history)(^Value^vmware fusion)]

The Name column contains the type of record - searchbar-history means what it says on the tin, other values may relate to various fields found on web page forms. The query field on the Google Firefox start page is represented by q .

Encase does not report these records particularly well. I chose to export relevant records into a spreadsheet and manually add the physical sector and sector offset of each row.

References
https://support.guidancesoftware.com/forum/showpost.php?p=115379&postcount=2
http://www.mozilla.org/mailnews/arch/mork/primer.txt
https://developer.mozilla.org/en/Mork_Structure

Goodmans GNAV12 sat nav device

2008-11-10T12:23:00.008Z

This device runs destinator software within a Windows 4.2 CE OS and is similar to the Medion MDPNA150 I looked at earlier. It has an external SD card slot which was not populated on the one I looked at. It also has internal flash memory.

I accessed the device via Mobile Device Centre in Vista and copied off the contents of the ResidentFlash volume. At the path DestinatorApps\Destinator\UK_Ireland I found Previous.dat. This file contains Recent Locations which are locations that the user chose to navigate to.

I have deconstructed the records found within Previous.dat.

Key (click on table to view larger image)

Example Records (click on records to view larger image)

Each record contains two sets of longitude and latitude co-ordinates stored one after the other. I speculate that one set is actual and the other set is nearest road. In my sample they were either very similar or the same. Each longitude or latitude value is stored as a double which requires 8 bytes therefore 32 bytes are required to store both sets. The two sets are followed by a further 16 bytes of data -use unknown, which completes the record.

To locate these co-ordinates I found it easier to count back from the start of the following record. The other problem to overcome is how to convert the doubles to a decimal value. Encase does not have a easy way to do this. The data interpreter in Winhex can do this. The hex editor 0xED on a mac can also do this but rounds up to fewer decimal places than winhex.

I can supply on request an Enscript (written by my friend Oliver Smith over at Cy4or) that will parse out these records.

HP QuickPlay version 2.3

2008-11-10T09:28:00.012Z

A HP media centre laptop came through our lab recently. Its main OS was XP Media Center edition. However it appeared to have another XP OS installed on a separate 1GB partition. This was a case where the suspect was suspected of hiding stuff and regularly re-installing his OS to cover his tracks - so this second OS was of interest.

Encase listed three partitions

Mounting the drive PDE and looking at the disk in Disk Management showed

The Windows Initialise case module reported the following for the OS on the 1GB partition

The 1GB partition has a Partition Type of D7

A quick google began to throw some light on the matter. It seems that the laptop has HP Quickplay 2.3 installed. This technology allows users to access multimedia disks without booting into the main operating system. The version of XP on the partition with partition type D7 is XP embedded. This OS facilitates the quickplay function. Later versions of HP Quickplay do not use this method.

It seems that a number of other manufacturers use the D7 partition type for similar purposes.

References

http://en.wikipedia.org/wiki/QuickPlay
http://www.goodells.net/dellrestore/mediadirect.htm
http://www.asifism.com/installing-hp-quickplay-on-your-laptopnotebook-vista-xp/

Sony NV-U50 G Sat Nav device

2008-10-08T16:25:00.011+01:00

Some of you may be thinking that this blog has stopped being a computer forensics blog and is now only covering sat navs. It seems that way lately, but it is just that my computer cases are all run of the mill whilst sat navs are nearly always new and exiting!

The Sony NV-U50 G sat nav I looked at is running Sony Personal Navigation System version 1.06 software within Windows CE. It has 512mb of internal memory and no external flash media. I accessed it via Mobile Device Centre in Vista (probably Active Sync will suffice but I did not test with this) and discovered a My Flash Disk volume as normal. A folder named Sony will be accessible and within the NAV-U sub folder the following notable files can be found:

recent.txt
favourites.txt
prefs.ini

All three files are plain text.

prefs.ini is used to store user preferences but also contains three useful values:

hometarget
lasttarget
LastVisibleArea

hometarget contains the postcode and latitude and longtitude coordinates of the user set home location. lasttarget was not populated on the device I examined but I understand from colleagues that it can contain the last navigated to location. Both these values comprise of seventeen fields separated by the pipe symbol (|).

LastVisibleArea contains the lat/long coordinates of the bottom left and top right corners of the last map displayed on the device prior to being switched off. I had seen that the manual for the device contained the note:
The Sony Personal Navigation System always opens with the screen that was active at the time you switched off the device.

recent.txt and favourite.txt contained recently navigated to locations and user stored favourite destinations respectively. Each location record comprised of seventeen fields separated by the pipe symbol (|) in both files allowing them to be imported into an excel spreadsheet using the text data import wizard.

Example of a single favourite record.

GFRIEND|-|48|12366|-|-|SW1 1AA|-|-|-|-|-0.142076|51.50107|-|-|-|-|

Field 3 in most cases contained the value 48 which I understand to be the Country Code used by Sony/Navteq for the UK, Field 4 had in some cases a five digit number beginning with the digits 12. I speculate that these fields have something to do with the RDS/TMC facility available with this device. Fields 12 and 13 contained the Longtitude and Latitude coordinates of the location record (stored in decimal notation). Fields 16 and 17 if populated contained a second set of Longtitude and Latitude coordinates which I can only speculate may be journey origins.

The device can also store pre-planned itineraries and these are stored within files with a .rte file extension in the Sony/NAV-U/Routes folder. These files are plain text formatted the same way as recent.txt and favourite.txt .

By carrying out a live examination of the device most the data above can be ascertained, however where a user has allocated a name to a favourite or itinerary only the allocated name will be displayed - not the underlying address and lat/long coordinates.

UPDATED 17th December 2008

I have been asked whether the order of the entries in Recent.txt had any significance. I have carried out some further testing and established:

i) Recent.txt will contain a maximum of thirty entries.

ii) If the entry is generated by a user choosing to navigate to a Favourite via the Favourites menu button the display name will be stored along with other location information within Recent.txt.

iii) The most recently entered location is recorded last.

iv) Once there are thirty entries within Recent.txt when a new location is added the oldest record at the top of the list is deleted. An exception to this is if a new location duplicates an entry already stored in Recent.txt the older entry is deleted (wherever it was stored in the list) and a new location appended to the end of the list.

I also had another look at the secondary lat/long values and I am now of the view that they DO NOT contain journey origins. During testing all the values I was able to populate in these fields were fairly near to the location chosen to navigate to. I could not however discern their relevance.

Transition to XP 64

2008-10-07T19:39:00.015+01:00

We upgrade our forensic workstations every eighteen months or so have just taken delivery of four shiny new ones. This time I decided that the time had come for us to enter the brave new world of 64 bit operating systems. I have read many peoples experiences of this on Digital Detective, Forensic Focus and so on and was expecting a number of difficulties. However the driver for change is utilising 8gb ram so XP 64 bit was specified. To be on the safe side I had the machines built in a dual boot configuration with XP 64 bit on one partition and XP 32 bit on another. As an additional safeguard on the 64 bit side I installed VMWare Workstation and created an XP 32 bit VM.

So if it helps anyone here is a list of what works (and what doesn't).

XP 64 Software installations

64 bit programs

Encase 6.11.2 x64

Encase installed OK. Help About shows that the PDE, VFS, EFS and Fastbloc SE modules are installed and I have successfully tested PDE.

Tomtology

Tomtology x64 1.162, .NET 3.5 Redistributable and the keylock dongle drivers all installed and working.

http://forensicnavigation.com/#/downloads/4527959036

7Zip

http://downloads.sourceforge.net/sevenzip/7z460-x64.msi

Ultramon

http://www.realtimesoft.com/files/UltraMon_2.7.1_en_x64.msi

Firefox

http://wiki.mozilla-x86-64.com/Download

CodeMeter

CodeMeter Runtime 3.30a is needed for the FTK codemeter dongle.

Gimp 2.5.4

http://downloads.sourceforge.net/gimp-win/gimp-2.5.4-r26790-x64-setup.exe?modtime=1219882476&big_mirror=0

32 bit programs that work within the 64 bit OS

Encase 6.11.2 32 bit version

The 32 bit version installs and runs OK.

Access Data FTK

I had hoped to upgrade to FTK2 but a 64 bit version has not been released. I did however decide to transfer our licences from the green keylock dongle to the Codemeter dongle. There are detailed instructions on how to do this within the FTK2 box. I used an internet connected laptop with the latest version of Licence Manager installed to carry out the transfer. Once the licences were on the codemeter dongle I installed CodeMeter Runtime 3.30a (64 bit) onto the XP 64 bit box and then inserted the dongle. It was successfully recognized. FTK 1.81 and Registry Viewer 1.5.3 have been installed and they work OK.

http://accessdata.com/downloads.html

Microsoft Office 2007

The office programs are 32 bit but are designed to work with a 64 bit MS OS.

VFC

I have installed VFC 1.2.4.3, the green keylock dongle drivers and VMWare Diskmount Utility. All seem to work OK.

http://www.vmware.com/download/eula/diskmount_ws_v55.html

C4P

C4P v3.3.4 runs without issue. However some work is needed to get the C4P graphics extractor enscript running.

For Encase v6 64 bit running on XP 64 bit install the 64 bit MySQL 3.51 ODBC connector driver found at http://dev.mysql.com/downloads/connector/odbc/3.51.html#winx64
and also a MS hotfix found at http://www.microsoft.com/downloads/details.aspx?FamilyID=000364db-5e8b-44a8-b9be-ca44d18b059b&displaylang=en

The latest C4P v4 graphics extractor enscript contains a mySQL database connection string which needs modification (if you are using mysql). The string needs to be edited to read:

Conn.Open("PROVIDER=MSDASQL;DRIVER={MySQL ODBC 3.51 Driver};SERVER=" + Com.serverName + ";DATABASE=" + DbName + ";" + "UID=c4p_user;PASSWORD=password;OPTION=3")

NetAnalysis
NetAnalysis 1.38 beta 6 is running OK

Irfanview
Irfanview 4.20 is a running OK

Saminside
Saminside v2.5.7.1 is running OK

Isobuster 1.9 Pro
Isobuster 1.9 Pro works OK

Various other utilities such as PSPad, FileAlyzer, WRR and WFA are also working OK.

Things that did not work

Well not much really. The Microsoft powertoy picture resizer does not work but is adequately replaced with http://adionsoft.net/fastimageresize/