path: root/README

README for ext4magic V-0.2.1


You accidentally deleted files ?
=================================
 
Now, you can try it with ext4magic - probably you will find many - but not all
deleted files. ext4magic will not change the data on your partition.
It write copies of found deleted files to a directory on a different file system.
For that you need enough disk space on a ext4 or ext3 Linux file system. 

This tool requires a working file system. If the partition table ore the file system
damaged, ext4magic can not help. Then you should use a different recover tool.
In addition to the recovery functions a lot of other functions are included.
These functions allow a deep look into the file system and can also help to find 
data and files which are not automatically recover.



How does this work ?
=====================

A file in an ext3/4 filesystem consists of several parts. The name of the file
and a Inode nummer are in data blocks of the directory. This Inode nummer is 
a serial number for a data structure in a tabel of these structures. 
These structures are called Inode and are the most important part of the file.
In the Inode are included all properties of the file and the reference to 
there data blocks. In the data blocks store the data of the file. In example, 
all the bytes for a jpg image

During the deletion of a file, be completely destroyed all refer to the data
blocks in inode data. The content of data blocks are not destroyed, but the 
block now marked as free.
If you write new files, this free data blocks can reused for new files.
The old inode is also marked as free and is also ready for reuse.
Name and Inode number in the directory block are only marked deleted, 
they are skipped for now when searching for file names in this directory.

Deleted files can not re-assembled, the Inode data are unsuitable for 
this purpose.  Exactly what the developers say.

But there is the filesystem Journal. Journaling ensures the integrity of the 
filesystem by keeping a log of the ongoing disk changes.
After deleting a file, there you found a copy of the data block in which the 
deleted Inode is included. Well, this copy is not usable for a recover. 
The Inode is deleted, but perhaps there is also still an even older copy of 
the same data block.
If you find such an older block in the Journal, then you can find there the old
intact Inode copy of the deleted file. And with such an old Inode, you can now
undelete the file. You find in the Inode the properties and all refer to the 
data blocks. In the directory you find the old file name.  With a little luck,
the data blocks are not reused.

This is the principle of ext4magic to recover from inode copies. 

In the Journal there are not only inode copies.  You will also find tables with
the block and inode allocation. This data are used in the magic functions
for controlling the file carving. The functions of the file carving matched
exactly to the respective properties of the file system types and these functions
included into a multi-stage recover process. This feature is new in this version
and currently only usable for ext3.



----------------------------------------------------------------------------------


How you can use ext4magic ?
===========================

You need, of course, the file system from which you want try to recover deleted 
files. The safest way is to create an image of the partition. 
Important, for this, the filesystem must umounted or readony mounted.

For example: the filesystem is on /dev/sda1

 # dd  if=/dev/sda1  of=/path/to/image_name  bs=4096


With the shell, you change to a directory, where enough free space to write 
the data recovers.  You need also some options, but that later. 

You can use ext4magic:

 # ext4magic  /path/to/image_name  options




Not enough free space for a imagefile of the entire filesytem ?
-------------------------------------------------------------

If you can use ext4magic from a rescue system or from a LINUX Live-system,
or on a other LINUX system, do not mount the partition with the deleted files and use it directly 

  # ext4magic /dev/sda1 options




You can not restart the computer or umount the partition ?
---------------------------------------------------------

Attempts to mount the partition readonly. The best way try to "umount" and then 
"mount -o ro /dev/sda1" . If this ist noch posible?  try the following:

 # mount -o remount,ro,noload  /dev/sda1

if the partition is now mounted readonly, use also
 # ext4magic /dev/sda1 options




It is impossible to mount readonly ? 
------------------------------------

ext4magic still has a solution, but highly experimental. Please use only in 
exceptional cases. Never use ext4magic for a not readonly mounted partition. 
ext4magic read over the filesystem buffer from journal but the kernel write 
unbuffered to journal.

The first read of the journal is often ok, but all subsequent reads can read
wrong data blocks from journal. So long the journal file is buffered, you read 
wrong data blocks at the moment of the first read. The file system is operating
normally without errors, but ext4magic reads wrong blocks from the Journal.   


Workaround : ext4magic supports external journal. 
You can create a copy of the filesystem journal with the "debug2fs" command. 
Use this copy as external Journal for the mounted file system.
But, if mounted readwrite, here also only the first backup will work good,
after read the journal by debug2fs, the journal is also buffered and the next
read by debug2fs results also a bad journal copy.


  # debug2fs  -R  "dump <8> /path/to/journalbackup" /dev/sda1

you can use this copy of Journal  

  # ext4magic /dev/sda1 -j  /path/to/journalbackup options 

ext4magic then only read journal data from this journal backup. 
  
Warning: This procedure is tested, it works, but please be very careful 
with this feature. Remember, for ext4magic the file system is frozen at the time at
which the journal copy created. Any subsequent changes will not recognize by ext4magic.
This works only for a limited time if you continue to write into the file system.




--------------------------------------------------------------------------------------------------


A few words about the new magic functions (current version 0.2.0)
 ============================================================
 These functions are designed to make undo of recursive deletes. It is a  multi-level recover
 and also restore files if no old journal copies can be found for this file.


   1. recover files of the file system tree with the help of old inode copies.
   2. recover all other inode copies which were not found in first stage. 
   3. (currently only ext3) recover the remaining data blocks, using a file carving function (we say magic function)  
 


 After an accidental deletion:  prevent all writing into this file system and if possible also
 prevent reading of this file system. Also reading overwrites old journal data
 which are needed for the restore. 

 Umount the file system, and use ext4magic before you mount the file system again,
 or create a copy of the file system and use this for the recover.
 Perform no file system check on this file system before.


 The magic functions are very user friendly because very few command options are required.

 Extensive testing has confirmed that magic-scan-functions are now stable with libmagic of file-5.04.
 Good support exists for: all text file types, a lot of image formats, 
 often-used video and audio file types, Open Office documents,
 PDF, RAR, TAR, CPIO, BZ2, ZIP, GZIP, 7Z ...
 
 Many other file types are also found and restored with default function, but without examining
 the contents of the files. This works more or less. 

 Problems still exist with some multimedia formats and some documents. Not every file type
 can be restored only based on head and foot patterns. Some types of multimedia streams, splited or
 truncated files are hard to recover.
 The recovery of CD/DVD images and other file system containers is also problematic. This can only work 
 in file systems with 4KB block size.
 Sparse files, and very large files if not deleted in one step, can not be restored with this 
 function. (Bug:#017607)
 Of course, you can only find files when the "file" command recognize this file type. It is theoretically
 possible to enable the restore of unknown file types by an entry in the configuration file to "magic".
 

 Some files are one (or few) byte too short. These are final zero byte.
 Most of these files can be repaired by appending zeros.
 The following command illustrates how attach two zero byte to a file.

 #echo -en "\0\0" >> file


 Some files are one or more bytes to long. These are often zero byte at the end of the restored file.
 You can see this at the end of a file. "hexdump-C file | tail "
 These files can be opened usually normal, possibly with a warning. Only a few programs block the
 processing of such files.  Here is an example,  how this can be fixed (xz compressed file)

# ls -l test.xz
-rw-r--r-- 1 rob users 1005  4. Dez 12:54 test.xz
# xz -t test.xz
xz: test.xz: Compressed data is corrupt
# xz -d test.xz
xz: test.xz: Compressed data is corrupt
# dd if=test.xz of=test_.xz bs=1 count=1004
1004+0 Datensätze ein
1004+0 Datensätze aus
1004 Bytes (1,0 kB) kopiert, 0,0164605 s, 61,0 kB/s
# xz -t test_.xz
# xz -d test_.xz

 
 The magic functions do not work particularly fast, but very efficient and can find some files
 that other tools can not recover. It also find very long files when the data are fragmented in the
 file system. Others file carving tools find here often no complete files, or recover data trash.
 Because of the previously running recover stages, the hit rate of this function is often very good.
 But, at very high fragmentation the chances are low for a successful recovery for many files.
 
 In real file systems the magic function find also unfortunately some very old files.
 The idea, to prevent this by using the metadata from the journal, is definitely good, but, 
 in a real file system it works only limited. In test file systems it works very well, but in a real
 file system journal you find not always enough of these metadata to prevent the recover of very old files.

 

--------------------------------------------------------------------



Instructions to experimenting with new features, the magic functions.
---------------------------------------------------------------------

Use no file system specially created for this purpose.
Why?
 If you create a test file system, it is likely that all inode copies are included
 in the Journal. The first stage can restore all files, and you'll never see the
 magic functions in the third stage.


Better is the following:
 Use an existing ext3 filesystem. The last hours should no run a global "find" or a backup tool
 in this file system. That too would write to many inode copies and to be easy to recover.
 umount this file system, and create a 1-to-1 copy of the file system.
 Now mount the file system copy and delete all or many files. Then umount the file system copy.



Now you can test ext4magic with the deleted copy.
 You need free space for writing the recovered files.
 Assuming, the copy is "/dev/sdb1" and you have enough free
 space at "/home/test/"

 # ext4magic /dev/sdb1 -d /home/test/RECOVER -M
 if you have deleted all files.

 or
 # ext4magic /dev/sdb1 -d /home/test/RECOVER -m
 if not all files were deleted.


 It will automatically search for the time of the last deletion. 
 And with a little delay should start the recover. You can now only wait. Depending on the
 number of deleted files can take a long time. Then you can compare the files with 
 the original file system.


--------------------------------------------------------------------------------------------

The Options of ext4magic
=========================

ext4magic has a lot of options, here are just a small overview.
Detailed information take from the manpage.

One option must always be specified, the file system.


Information Options  -S -J -H -T 
---------------------------------
This can display information from the file system, the journal ore the transactions from journal.



Selections  -I -B  -f
-----------------------
select the specific inode, blocks or file names for the information- and action options.



Time Options -a -b -t
------------------------
These are important control options. This indicates the time window for searching for files.



File input and output options  -d -i -j
---------------------------------------
This can be specified, the output directory, a input file list and an external journal file



Action Options  -l -L -r -R -m -M
----------------------------------
For select of the various listing- and recover actions.



(NEW 2.0.1) Expert Options  -s -n -c
-------------------------
available only if enabled by configure
Allow access to damaged file systems, backup superblocks, ....



---------------------------------------------------------------------------------------------


Some common problems

Command not found 
------------------
ext4magic is installed to /usr/local/sbin/ 
This directory is only included in the PATH if you use root as a login shell.
For a full root environment  use "su -l" for the user change.



Manpage not found
-----------------
The manpage is installed under /usr/local/*/man/man8/
Check if the MANPATH variable include the following directories.

 /usr/local/man /usr/local/share/man



ext4magic nothing works
-----------------------
two possible causes: 
- either you are not root
- or the time options are not set correctly. Only the magical functions automatically search 
  for the best time window, all other options use default time values. (See manpage) 



----------------------------------------------------------------------------------------------

Known Bugs

Only on big endian environments, there are some incorrect outputs of time stamps, and missing of
deleted directory entries. (BUG #017304 #017305) 
These errors occur only if the journal is not read and so only called functions of libext2fs for
printout of inode and directory. All journal options and the file restoring are not affected. 
The error is not within ext4magic and can not be compensated in ext4magic. This would be patched
in libext2fs. The error is very rare and not significant. If anyone needs a patch for this,
no problem, within ext4magic the problem is solved. It is also possible to write an unofficial patch
for libext2fs. I just think that nobody will really need it. Otherwise, send a request to the ext4magic 
mailing list.