From 891877f331bc93afd5fe8b33ee89acdee67ac162 Mon Sep 17 00:00:00 2001 From: robi Date: Sun, 3 Apr 2011 18:29:12 +0000 Subject: convert to unix line terminators --- README | 728 ++++++++++++++++++++++++++++++++--------------------------------- 1 file changed, 364 insertions(+), 364 deletions(-) (limited to 'README') diff --git a/README b/README index 28db56d..13f3c74 100644 --- a/README +++ b/README @@ -1,364 +1,364 @@ -README for ext4magic V-0.2.0 - - -You accidentally deleted files ? -================================= - -Now, you can try it with ext4magic - probably you will find many - but not all -deleted files. ext4magic will not change the data on your partition. -It write copies of found deleted files to a directory on a different file system. -For that you need enough disk space on a ext4 or ext3 Linux file system. - -This tool requires a working file system. If the partition table ore the file system -damaged, ext4magic can not help. Then you should use a different recover tool. -In addition to the recovery functions a lot of other functions are included. -These functions allow a deep look into the file system and can also help to find -data and files which are not automatically recover. - - - -How does this work ? -===================== - -A file in an ext3/4 filesystem consists of several parts. The name of the file -and a Inode nummer are in data blocks of the directory. This Inode nummer is -a serial number for a data structure in a tabel of these structures. -These structures are called Inode and are the most important part of the file. -In the Inode are included all properties of the file and the reference to -there data blocks. In the data blocks store the data of the file. In example, -all the bytes for a jpg image - -During the deletion of a file, be completely destroyed all refer to the data -blocks in inode data. The content of data blocks are not destroyed, but the -block now marked as free. -If you write new files, this free data blocks can reused for new files. -The old inode is also marked as free and is also ready for reuse. -Name and Inode number in the directory block are only marked deleted, -they are skipped for now when searching for file names in this directory. - -Deleted files can not re-assembled, the Inode data are unsuitable for -this purpose. Exactly what the developers say. - -But there is the filesystem Journal. Journaling ensures the integrity of the -filesystem by keeping a log of the ongoing disk changes. -After deleting a file, there you found a copy of the data block in which the -deleted Inode is included. Well, this copy is not usable for a recover. -The Inode is deleted, but perhaps there is also still an even older copy of -the same data block. -If you find such an older block in the Journal, then you can find there the old -intact Inode copy of the deleted file. And with such an old Inode, you can now -undelete the file. You find in the Inode the properties and all refer to the -data blocks. In the directory you find the old file name. With a little luck, -the data blocks are not reused. - -This is the principle of ext4magic to recover from inode copies. - -In the Journal there are not only inode copies. You will also find tables with -the block and inode allocation. This data are used in the magic functions -for controlling the file carving. The functions of the file carving matched -exactly to the respective properties of the file system types and these functions -included into a multi-stage recover process. This feature is new in this version -and currently only usable for ext3. - - - ----------------------------------------------------------------------------------- - - -How you can use ext4magic ? -=========================== - -You need, of course, the file system from which you want try to recover deleted -files. The safest way is to create an image of the partition. -Important, for this, the filesystem must umounted or readony mounted. - -For example: the filesystem is on /dev/sda1 - - # dd if=/dev/sda1 of=/path/to/image_name bs=4096 - - -With the shell, you change to a directory, where enough free space to write -the data recovers. You need also some options, but that later. - -You can use ext4magic: - - # ext4magic /path/to/image_name options - - - - -Not enough free space for a imagefile of the entire filesytem ? -------------------------------------------------------------- - -If you can use ext4magic from a rescue system or from a LINUX Live-system, -or on a other LINUX system, do not mount the partition with the deleted files and use it directly - - # ext4magic /dev/sda1 options - - - - -You can not restart the computer or umount the partition ? ---------------------------------------------------------- - -Attempts to mount the partition readonly. The best way try to "umount" and then -"mount -o ro /dev/sda1" . If this ist noch posible? try the following: - - # mount -o remount,ro,noload /dev/sda1 - -if the partition is now mounted readonly, use also - # ext4magic /dev/sda1 options - - - - -It is impossible to mount readonly ? ------------------------------------- - -ext4magic still has a solution, but highly experimental. Please use only in -exceptional cases. Never use ext4magic for a not readonly mounted partition. -ext4magic read over the filesystem buffer from journal but the kernel write -unbuffered to journal. - -The first read of the journal is often ok, but all subsequent reads can read -wrong data blocks from journal. So long the journal file is buffered, you read -wrong data blocks at the moment of the first read. The file system is operating -normally without errors, but ext4magic reads wrong blocks from the Journal. - - -Workaround : ext4magic supports external journal. -You can create a copy of the filesystem journal with the "debug2fs" command. -Use this copy as external Journal for the mounted file system. -But, if mounted readwrite, here also only the first backup will work good, -after read the journal by debug2fs, the journal is also buffered and the next -read by debug2fs results also a bad journal copy. - - - # debug2fs -R "dump <8> /path/to/journalbackup" /dev/sda1 - -you can use this copy of Journal - - # ext4magic /dev/sda1 -j /path/to/journalbackup options - -ext4magic then only read journal data from this journal backup. - -Warning: This procedure is tested, it works, but please be very careful -with this feature. Remember, for ext4magic the file system is frozen at the time at -which the journal copy created. Any subsequent changes will not recognize by ext4magic. -This works only for a limited time if you continue to write into the file system. - - - - --------------------------------------------------------------------------------------------------- - - -A few words about the new magic functions (current version 0.2.0) - ============================================================ - These functions are designed to make undo of recursive deletes. It is a multi-level recover - and also restore files if no old journal copies can be found for this file. - - - 1. recover files of the file system tree with the help of old inode copies. - 2. recover all other inode copies which were not found in first stage. - 3. (currently only ext3) recover the remaining data blocks, using a file carving function (we say magic function) - - - - After an accidental deletion: prevent all writing into this file system and if possible also - prevent reading of this file system. Also reading overwrites old journal data - which are needed for the restore. - - Umount the file system, and use ext4magic before you mount the file system again, - or create a copy of the file system and use this for the recover. - Perform no file system check on this file system before. - - - The magic functions are very user friendly because very few command options are required. - - Extensive testing has confirmed that magic-scan-functions are now stable with libmagic of file-5.04. - Good support exists for: all text file types, a lot of image formats, - often-used video and audio file types, Open Office documents, - PDF, RAR, TAR, CPIO, BZ2, ZIP, GZIP, 7Z ... - - Many other file types are also found and restored with default function, but without examining - the contents of the files. This works more or less. - - Problems still exist with some multimedia formats and some documents. Not every file type - can be restored only based on head and foot patterns. Some types of multimedia streams, splited or - truncated files are hard to recover. - The recovery of CD/DVD images and other file system containers is also problematic. This can only work - in file systems with 4KB block size. - Sparse files, and very large files if not deleted in one step, can not be restored with this - function. (Bug:#017607) - Of course, you can only find files when the "file" command recognize this file type. It is theoretically - possible to enable the restore of unknown file types by an entry in the configuration file to "magic". - - - Some files are one (or few) byte too short. These are final zero byte. - Most of these files can be repaired by appending zeros. - The following command illustrates how attach two zero byte to a file. - - #echo -en "\0\0" >> file - - - Some files are one or more bytes to long. These are often zero byte at the end of the restored file. - You can see this at the end of a file. "hexdump-C file | tail " - These files can be opened usually normal, possibly with a warning. Only a few programs block the - processing of such files. Here is an example, how this can be fixed (xz compressed file) - -# ls -l test.xz --rw-r--r-- 1 rob users 1005 4. Dez 12:54 test.xz -# xz -t test.xz -xz: test.xz: Compressed data is corrupt -# xz -d test.xz -xz: test.xz: Compressed data is corrupt -# dd if=test.xz of=test_.xz bs=1 count=1004 -1004+0 Datensätze ein -1004+0 Datensätze aus -1004 Bytes (1,0 kB) kopiert, 0,0164605 s, 61,0 kB/s -# xz -t test_.xz -# xz -d test_.xz - - - The magic functions do not work particularly fast, but very efficient and can find some files - that other tools can not recover. It also find very long files when the data are fragmented in the - file system. Others file carving tools find here often no complete files, or recover data trash. - Because of the previously running recover stages, the hit rate of this function is often very good. - But, at very high fragmentation the chances are low for a successful recovery for many files. - - In real file systems the magic function find also unfortunately some very old files. - The idea, to prevent this by using the metadata from the journal, is definitely good, but, - in a real file system it works only limited. In test file systems it works very well, but in a real - file system journal you find not always enough of these metadata to prevent the recover of very old files. - - - --------------------------------------------------------------------- - - - -Instructions to experimenting with new features, the magic functions. ---------------------------------------------------------------------- - -Use no file system specially created for this purpose. -Why? - If you create a test file system, it is likely that all inode copies are included - in the Journal. The first stage can restore all files, and you'll never see the - magic functions in the third stage. - - -Better is the following: - Use an existing ext3 filesystem. The last hours should no run a global "find" or a backup tool - in this file system. That too would write to many inode copies and to be easy to recover. - umount this file system, and create a 1-to-1 copy of the file system. - Now mount the file system copy and delete all or many files. Then umount the file system copy. - - - -Now you can test ext4magic with the deleted copy. - You need free space for writing the recovered files. - Assuming, the copy is "/dev/sdb1" and you have enough free - space at "/home/test/" - - # ext4magic /dev/sdb1 -d /home/test/RECOVER -M - if you have deleted all files. - - or - # ext4magic /dev/sdb1 -d /home/test/RECOVER -m - if not all files were deleted. - - - It will automatically search for the time of the last deletion. - And with a little delay should start the recover. You can now only wait. Depending on the - number of deleted files can take a long time. Then you can compare the files with - the original file system. - - --------------------------------------------------------------------------------------------- - -The Options of ext4magic -========================= - -ext4magic has a lot of options, here are just a small overview. -Detailed information take from the manpage. - -One option must always be specified, the file system. - - -Information Options -S -J -H -T ---------------------------------- -This can display information from the file system, the journal ore the transactions from journal. - - - -Selections -I -B -f ------------------------ -select the specific inode, blocks or file names for the information- and action options. - - - -Time Options -a -b -t ------------------------- -These are important control options. This indicates the time window for searching for files. - - - -File input and output options -d -i -j ---------------------------------------- -This can be specified, the output directory, a input file list and an external journal file - - - -Action Options -l -L -r -R -m -M ----------------------------------- -For select of the various listing- and recover actions. - - - - ---------------------------------------------------------------------------------------------- - - -Some common problems - -Command not found ------------------- -ext4magic is installed to /usr/local/sbin/ -This directory is only included in the PATH if you use root as a login shell. -For a full root environment use "su -l" for the user change. - - - -Manpage not found ------------------ -The manpage is installed under /usr/local/*/man/man8/ -Check if the MANPATH variable include the following directories. - - /usr/local/man /usr/local/share/man - - - -ext4magic nothing works ------------------------ -two possible causes: -- either you are not root -- or the time options are not set correctly. Only the magical functions automatically search - for the best time window, all other options use default time values. (See manpage) - - - ----------------------------------------------------------------------------------------------- - -Known Bugs - -Only on big endian environments, there are some incorrect outputs of time stamps, and missing of -deleted directory entries. (BUG #017304 #017305) -These errors occur only if the journal is not read and so only called functions of libext2fs for -printout of inode and directorys. All journal options and the file restoring are not affected. -The error is not within ext4magic and can not be compensated in ext4magic. This would be patched -in libext2fs. The error is very rare and not significant. If anyone needs a patch for this, -no problem, within ext4magic the problem is solved. It is also possible to write an unofficial patch -for libext2fs. I just think that nobody will really need it. Otherwise, send a request to the ext4magic -mailing list. - +README for ext4magic V-0.2.0 + + +You accidentally deleted files ? +================================= + +Now, you can try it with ext4magic - probably you will find many - but not all +deleted files. ext4magic will not change the data on your partition. +It write copies of found deleted files to a directory on a different file system. +For that you need enough disk space on a ext4 or ext3 Linux file system. + +This tool requires a working file system. If the partition table ore the file system +damaged, ext4magic can not help. Then you should use a different recover tool. +In addition to the recovery functions a lot of other functions are included. +These functions allow a deep look into the file system and can also help to find +data and files which are not automatically recover. + + + +How does this work ? +===================== + +A file in an ext3/4 filesystem consists of several parts. The name of the file +and a Inode nummer are in data blocks of the directory. This Inode nummer is +a serial number for a data structure in a tabel of these structures. +These structures are called Inode and are the most important part of the file. +In the Inode are included all properties of the file and the reference to +there data blocks. In the data blocks store the data of the file. In example, +all the bytes for a jpg image + +During the deletion of a file, be completely destroyed all refer to the data +blocks in inode data. The content of data blocks are not destroyed, but the +block now marked as free. +If you write new files, this free data blocks can reused for new files. +The old inode is also marked as free and is also ready for reuse. +Name and Inode number in the directory block are only marked deleted, +they are skipped for now when searching for file names in this directory. + +Deleted files can not re-assembled, the Inode data are unsuitable for +this purpose. Exactly what the developers say. + +But there is the filesystem Journal. Journaling ensures the integrity of the +filesystem by keeping a log of the ongoing disk changes. +After deleting a file, there you found a copy of the data block in which the +deleted Inode is included. Well, this copy is not usable for a recover. +The Inode is deleted, but perhaps there is also still an even older copy of +the same data block. +If you find such an older block in the Journal, then you can find there the old +intact Inode copy of the deleted file. And with such an old Inode, you can now +undelete the file. You find in the Inode the properties and all refer to the +data blocks. In the directory you find the old file name. With a little luck, +the data blocks are not reused. + +This is the principle of ext4magic to recover from inode copies. + +In the Journal there are not only inode copies. You will also find tables with +the block and inode allocation. This data are used in the magic functions +for controlling the file carving. The functions of the file carving matched +exactly to the respective properties of the file system types and these functions +included into a multi-stage recover process. This feature is new in this version +and currently only usable for ext3. + + + +---------------------------------------------------------------------------------- + + +How you can use ext4magic ? +=========================== + +You need, of course, the file system from which you want try to recover deleted +files. The safest way is to create an image of the partition. +Important, for this, the filesystem must umounted or readony mounted. + +For example: the filesystem is on /dev/sda1 + + # dd if=/dev/sda1 of=/path/to/image_name bs=4096 + + +With the shell, you change to a directory, where enough free space to write +the data recovers. You need also some options, but that later. + +You can use ext4magic: + + # ext4magic /path/to/image_name options + + + + +Not enough free space for a imagefile of the entire filesytem ? +------------------------------------------------------------- + +If you can use ext4magic from a rescue system or from a LINUX Live-system, +or on a other LINUX system, do not mount the partition with the deleted files and use it directly + + # ext4magic /dev/sda1 options + + + + +You can not restart the computer or umount the partition ? +--------------------------------------------------------- + +Attempts to mount the partition readonly. The best way try to "umount" and then +"mount -o ro /dev/sda1" . If this ist noch posible? try the following: + + # mount -o remount,ro,noload /dev/sda1 + +if the partition is now mounted readonly, use also + # ext4magic /dev/sda1 options + + + + +It is impossible to mount readonly ? +------------------------------------ + +ext4magic still has a solution, but highly experimental. Please use only in +exceptional cases. Never use ext4magic for a not readonly mounted partition. +ext4magic read over the filesystem buffer from journal but the kernel write +unbuffered to journal. + +The first read of the journal is often ok, but all subsequent reads can read +wrong data blocks from journal. So long the journal file is buffered, you read +wrong data blocks at the moment of the first read. The file system is operating +normally without errors, but ext4magic reads wrong blocks from the Journal. + + +Workaround : ext4magic supports external journal. +You can create a copy of the filesystem journal with the "debug2fs" command. +Use this copy as external Journal for the mounted file system. +But, if mounted readwrite, here also only the first backup will work good, +after read the journal by debug2fs, the journal is also buffered and the next +read by debug2fs results also a bad journal copy. + + + # debug2fs -R "dump <8> /path/to/journalbackup" /dev/sda1 + +you can use this copy of Journal + + # ext4magic /dev/sda1 -j /path/to/journalbackup options + +ext4magic then only read journal data from this journal backup. + +Warning: This procedure is tested, it works, but please be very careful +with this feature. Remember, for ext4magic the file system is frozen at the time at +which the journal copy created. Any subsequent changes will not recognize by ext4magic. +This works only for a limited time if you continue to write into the file system. + + + + +-------------------------------------------------------------------------------------------------- + + +A few words about the new magic functions (current version 0.2.0) + ============================================================ + These functions are designed to make undo of recursive deletes. It is a multi-level recover + and also restore files if no old journal copies can be found for this file. + + + 1. recover files of the file system tree with the help of old inode copies. + 2. recover all other inode copies which were not found in first stage. + 3. (currently only ext3) recover the remaining data blocks, using a file carving function (we say magic function) + + + + After an accidental deletion: prevent all writing into this file system and if possible also + prevent reading of this file system. Also reading overwrites old journal data + which are needed for the restore. + + Umount the file system, and use ext4magic before you mount the file system again, + or create a copy of the file system and use this for the recover. + Perform no file system check on this file system before. + + + The magic functions are very user friendly because very few command options are required. + + Extensive testing has confirmed that magic-scan-functions are now stable with libmagic of file-5.04. + Good support exists for: all text file types, a lot of image formats, + often-used video and audio file types, Open Office documents, + PDF, RAR, TAR, CPIO, BZ2, ZIP, GZIP, 7Z ... + + Many other file types are also found and restored with default function, but without examining + the contents of the files. This works more or less. + + Problems still exist with some multimedia formats and some documents. Not every file type + can be restored only based on head and foot patterns. Some types of multimedia streams, splited or + truncated files are hard to recover. + The recovery of CD/DVD images and other file system containers is also problematic. This can only work + in file systems with 4KB block size. + Sparse files, and very large files if not deleted in one step, can not be restored with this + function. (Bug:#017607) + Of course, you can only find files when the "file" command recognize this file type. It is theoretically + possible to enable the restore of unknown file types by an entry in the configuration file to "magic". + + + Some files are one (or few) byte too short. These are final zero byte. + Most of these files can be repaired by appending zeros. + The following command illustrates how attach two zero byte to a file. + + #echo -en "\0\0" >> file + + + Some files are one or more bytes to long. These are often zero byte at the end of the restored file. + You can see this at the end of a file. "hexdump-C file | tail " + These files can be opened usually normal, possibly with a warning. Only a few programs block the + processing of such files. Here is an example, how this can be fixed (xz compressed file) + +# ls -l test.xz +-rw-r--r-- 1 rob users 1005 4. Dez 12:54 test.xz +# xz -t test.xz +xz: test.xz: Compressed data is corrupt +# xz -d test.xz +xz: test.xz: Compressed data is corrupt +# dd if=test.xz of=test_.xz bs=1 count=1004 +1004+0 Datensätze ein +1004+0 Datensätze aus +1004 Bytes (1,0 kB) kopiert, 0,0164605 s, 61,0 kB/s +# xz -t test_.xz +# xz -d test_.xz + + + The magic functions do not work particularly fast, but very efficient and can find some files + that other tools can not recover. It also find very long files when the data are fragmented in the + file system. Others file carving tools find here often no complete files, or recover data trash. + Because of the previously running recover stages, the hit rate of this function is often very good. + But, at very high fragmentation the chances are low for a successful recovery for many files. + + In real file systems the magic function find also unfortunately some very old files. + The idea, to prevent this by using the metadata from the journal, is definitely good, but, + in a real file system it works only limited. In test file systems it works very well, but in a real + file system journal you find not always enough of these metadata to prevent the recover of very old files. + + + +-------------------------------------------------------------------- + + + +Instructions to experimenting with new features, the magic functions. +--------------------------------------------------------------------- + +Use no file system specially created for this purpose. +Why? + If you create a test file system, it is likely that all inode copies are included + in the Journal. The first stage can restore all files, and you'll never see the + magic functions in the third stage. + + +Better is the following: + Use an existing ext3 filesystem. The last hours should no run a global "find" or a backup tool + in this file system. That too would write to many inode copies and to be easy to recover. + umount this file system, and create a 1-to-1 copy of the file system. + Now mount the file system copy and delete all or many files. Then umount the file system copy. + + + +Now you can test ext4magic with the deleted copy. + You need free space for writing the recovered files. + Assuming, the copy is "/dev/sdb1" and you have enough free + space at "/home/test/" + + # ext4magic /dev/sdb1 -d /home/test/RECOVER -M + if you have deleted all files. + + or + # ext4magic /dev/sdb1 -d /home/test/RECOVER -m + if not all files were deleted. + + + It will automatically search for the time of the last deletion. + And with a little delay should start the recover. You can now only wait. Depending on the + number of deleted files can take a long time. Then you can compare the files with + the original file system. + + +-------------------------------------------------------------------------------------------- + +The Options of ext4magic +========================= + +ext4magic has a lot of options, here are just a small overview. +Detailed information take from the manpage. + +One option must always be specified, the file system. + + +Information Options -S -J -H -T +--------------------------------- +This can display information from the file system, the journal ore the transactions from journal. + + + +Selections -I -B -f +----------------------- +select the specific inode, blocks or file names for the information- and action options. + + + +Time Options -a -b -t +------------------------ +These are important control options. This indicates the time window for searching for files. + + + +File input and output options -d -i -j +--------------------------------------- +This can be specified, the output directory, a input file list and an external journal file + + + +Action Options -l -L -r -R -m -M +---------------------------------- +For select of the various listing- and recover actions. + + + + +--------------------------------------------------------------------------------------------- + + +Some common problems + +Command not found +------------------ +ext4magic is installed to /usr/local/sbin/ +This directory is only included in the PATH if you use root as a login shell. +For a full root environment use "su -l" for the user change. + + + +Manpage not found +----------------- +The manpage is installed under /usr/local/*/man/man8/ +Check if the MANPATH variable include the following directories. + + /usr/local/man /usr/local/share/man + + + +ext4magic nothing works +----------------------- +two possible causes: +- either you are not root +- or the time options are not set correctly. Only the magical functions automatically search + for the best time window, all other options use default time values. (See manpage) + + + +---------------------------------------------------------------------------------------------- + +Known Bugs + +Only on big endian environments, there are some incorrect outputs of time stamps, and missing of +deleted directory entries. (BUG #017304 #017305) +These errors occur only if the journal is not read and so only called functions of libext2fs for +printout of inode and directorys. All journal options and the file restoring are not affected. +The error is not within ext4magic and can not be compensated in ext4magic. This would be patched +in libext2fs. The error is very rare and not significant. If anyone needs a patch for this, +no problem, within ext4magic the problem is solved. It is also possible to write an unofficial patch +for libext2fs. I just think that nobody will really need it. Otherwise, send a request to the ext4magic +mailing list. + -- cgit v1.2.3