How ext3grep can save you hours of work

Posted: July 18, 2008 in Linux

Linux files systems work great. Especially ext3 is a mature file system and very robust. With journaling turned the filesystems stays consistent also on an unexpected power loss or system crash (wich normally never happens on linux :-) ) and you don’t need long disk checks to repair the file system in such a case. There is almost no disk fragmentation and a lot of more cool features … But the best linux feature is: if you remove a file or directory it’s gone :-) It’s not moved to trash, its not marked as deleted, there are no annoying messageboxes, it just does what you say. I love that!
But from time to time also experienced users make mistakes and you delete something wrong. As Murphy’s law says: “if anything can go wrong it will” and so you will not have a copy of the deleted files and the backup will be outdated. So what to do now?
I want to explain the procedure by using the problem I had as example.

How the trouble came up for me

As usual I was working on my Gentoo Linux laptop. I just wanted to add a new folder with sources to my subversion repository. I wanted to add first the folder, then some files within the folder…

  1. Adding a folder to svn is simple, sometimes too simple…
    svn add uaconsoleclientsample

    Now I saw that this command added all files recursively, not just the folder. That means a lot of temporary files where added too.
    The good thing with svn is, everthing you change is just done on your local working copy and not on the server. So it’s not a problem until you commit your changes. So lets remove the wrong files again before committing the changes.

  2. Removing the files…
        svn remove uaconsoleclientsample
        svn: Use --force to override this restriction
        svn: 'uaconsoleclientsample' has local modifications

    Ok, remove didn’t work, use –force.

    svn --force remove uaconsoleclientsample

    Bad idea!!! The “svn remove” command not only removes files from scheduled add operations like I assumed. It removes it also from your local working copy.

    So now my uncommited files were lost. This was the time where ext3grep came into play.

Recover lost files with ext3grep

I knew every new file could overwrite the data of my deleted files. So quickly unmount the partition. (The great thing, linux users have multiple partitions. We can unmount /home and continue to work as root)

  1. Unmounting /home
    • Close my X Session
    • Alt-F1 -> log in as root
    • /etc/init.d/xdm stop
    • umount /home Cannot umount, files still in access
    • lsof | grep home -> artsd is still accessing home
    • killall artsd
    • umount /home Succeeded.
  2. OK, done. Nothing can delete any more data now. Now relax, breath through and go to another Desktop PC, start the browser and ask google for a solution. -> ext3grep Ok, lets try this out. Login into laptop from the desktop, that’s more comfortable, just copy paste URLs to the shell etc. ssh -l root laptop

  3. Installing ext3grep…
    wget http://ext3grep.googlecode.com/files/ext3grep-0.7.0.tar.gz
    tar -xvzf ext3grep-0.7.0.tar.gz
    cd ext3grep
    ./configure
    make
    cd src
    ./ext3grep --help

    Done, ext3grep is built and working

  4. Lets search for my deleted folder:My laptop has an LUKS encrypted hard disk, and I devided this encrypted partition using LVM. For that reason my home partition is not a physical one like /dev/hda, but /dev/mapper/vg-home. Replace that with your partition you are using.
  5. lt_gergap src # ./ext3grep /dev/mapper/vg-home --search uaconsoleclientsample
    Running ext3grep version 0.7.0
    Number of groups: 240
    Minimum / maximum journal block: 1547 / 35890
    Loading journal descriptors... sorting... done
    Journal transaction 1202732 wraps around, some data blocks might have been lost of this transaction.
    Number of descriptors in journal: 30993; min / max sequence numbers: 1202700 / 1204860
    Blocks containing "uaconsoleclientsample": 240017 242352 242355 242356 (allocated) 242357
    (allocated) 242358 (allocated) 242359 255079 (allocated) 336393 336518 336526 395434 395435
    395457 (allocated) 737282 (allocated) 984250 1346129 1868670 (allocated) 1869273 (allocated)
    1950436 3915933 3915935 4069411 4087953 4216611 4292193 4292196 4292275 4530219 4538370
    4538371 4538372 4538376 4538378 4538382 4538385 4543743 4543750 4543752 4544514 4544517
    4544528 4544539 4550683 4550707 4655509 4655533 4670417 4670423 4689385 4689746 4785120
    5046823 6525842 (allocated) 7370457 7805912

    Wow, there are a lot of blocks. Why are some allocated? Lets check them all.

    lt_gergap src # ./ext3grep /dev/mapper/vg-home --ls --block 240017
    Running ext3grep version 0.7.0
    Number of groups: 240
    Minimum / maximum journal block: 1547 / 35890
    Loading journal descriptors... sorting... done
    Journal transaction 1202732 wraps around, some data blocks might have been lost of this transaction.
    Number of descriptors in journal: 30993; min / max sequence numbers: 1202700 / 1204860
    Group: 7
    Block 240017 is Unallocated.

    Hmm, seems not to be a directory, at least it contains nothing. Try the next one …

    This operation takes some time and there are a lot of blocks. I’m lazy and before I do something twice I write a little script to automate this procedure.


    1 #!/bin/sh
    2
    3 blocks=255079 336393 336518 336526 395434 395435 395457 737282 984250 1346129 1868670 1869273 1950436 3915933 3915935 4069411 4087953 4216611 4292193 4292196 4292275 4530219 4538370 4538371 4538372 4538376 4538378 4538382 4538385 4543743 4543750 4543752 4544514 4544517 4544528 4544539 4550683 4550707 4655509 4655533 4670417 4670423 4689385 4689746 4785120 5046823 6525842 7370457 7805912
    4
    5 for block in $blocks; do
    6 ./ext3grep /dev/mapper/vg-home –ls –block $block | tee -a output.txt
    7 done

    This script does the same procedure for each block in the list, shows the output on console and writes it to the file output.txt using tee. This way you can recheck the result later also in vim.

    Analyzing the file output.txt I have seen only one interesting block:

    ...
    Running ext3grep version 0.7.0
    Number of groups: 240
    Minimum / maximum journal block: 1547 / 35890
    Loading journal descriptors... sorting... done
    Journal transaction 1202732 wraps around, some data blocks might have been lost of this transaction.
    Number of descriptors in journal: 30993; min / max sequence numbers: 1202700 / 1204860
    Group: 57
    
    Block 1869273 is a directory. The block is Allocated
    
              .-- File type in dir_entry (r=regular file, d=directory, l=symlink)
              |          .-- D: Deleted ; R: Reallocated
    Indx Next |  Inode   | Deletion time                        Mode        File name
    ==========+==========+----------------data-from-inode------+-----------+=========
       0    1 d  657336                                         drwxr-xr-x  .
       1    2 d  657111                                         drwxr-xr-x  ..
       2    3 d  657337                                         drwxr-xr-x  .svn
       3    4 d  657345                                         drwxr-xr-x  qtestclient
       4    5 d  657354                                         drwxr-xr-x  uaserverc
       5    6 d  657444                                         drwxr-xr-x  performanceclient
       6    7 d  657534                                         drwxr-xr-x  uademoserver
       7    8 d  657687                                         drwxr-xr-x  comdaproxy
       8    9 d  657696                                         drwxr-xr-x  examples_bin
       9   10 d  657903                                         drwxr-xr-x  uaconsoleclient
      10   11 d  672247                                         drwxr-xr-x  uaconsoleclientcpp
      11   12 d  672766                                         drwxr-xr-x  demoserver
      12   13 d  672784                                         drwxr-xr-x  uaserver
      13   14 d  657946                                         drwxr-xr-x  ascserver
      14   15 d  672981                                         drwxr-xr-x  server_ads
      15   16 d  657947                                         drwxr-xr-x  uaexpert
      16   18 d  657949                                         drwxr-xr-x  uasimpleclient
      17   18 d  675465  D 1215690762 Thu Jul 10 13:52:42 2008  drwxr-xr-x  bin
      18  end d  656495                                         drwxr-xr-x  uaconsoleclientsample
    ...

    Yes, there it is. Let’s check the contents of this folder.

    lt_gergap src # ./ext3grep /dev/mapper/vg-home --ls --inode 656495
    Running ext3grep version 0.7.0
    Number of groups: 240
    Minimum / maximum journal block: 1547 / 35890
    Loading journal descriptors... sorting... done
    Journal transaction 1202732 wraps around, some data blocks might have been lost of this transaction.
    Number of descriptors in journal: 30993; min / max sequence numbers: 1202700 / 1204860
    Inode is Allocated
    Loading vg-home.ext3grep.stage2.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
    WARNING: Rejecting a dir_entry (block 5714387) because it contains legal but unlikely characters.
    Use --ls --block 5714387 to examine this possible directory block.
    If it looks like a directory to you, and '24'
    looks like a filename that might belong in that directory, then add
    --accept='24' as commandline parameter AND remove both stage* files!
    ........................................................................................................................................................................................... done
    The first block of the directory is 1348508.
    Inode 656495 is directory "kde4/work/opcua/applications/uaconsoleclientsample".
    Directory block 1348508:
    .-- File type in dir_entry (r=regular file, d=directory, l=symlink)
    |          .-- D: Deleted ; R: Reallocated
    Indx Next |  Inode   | Deletion time                        Mode        File name
    ==========+==========+----------------data-from-inode------+-----------+=========
    0    1 d  656495                                         drwxr-xr-x  .
    1   11 d  657336                                         drwxr-xr-x  ..
    2    4 r 2328216  D 1215690762 Thu Jul 10 13:52:42 2008  rrw-r--r--  Makefile
    3    4 r 2263308  D 1215690762 Thu Jul 10 13:52:42 2008  rrw-r--r--  main.cpp
    4    5 r 2328213  D 1215690762 Thu Jul 10 13:52:42 2008  rrw-r--r--  qdevelop-settings.db
    5   11 r 2328221  D 1215690762 Thu Jul 10 13:52:42 2008  rrw-r--r--  uaconsolesampleclient.pro
    6    7 r 2328215  D 1215690762 Thu Jul 10 13:52:42 2008  rrw-r--r--  main.o
    8    9 d  675466  D 1215690762 Thu Jul 10 13:52:42 2008  drwxr-xr-x  moc
    9   11 r 2262157  D 1215690762 Thu Jul 10 13:52:42 2008  rrwxr-xr-x  uaconsolesampleclient
    10   11 d  675465  D 1215690762 Thu Jul 10 13:52:42 2008  drwxr-xr-x  PKI
    11  end d  656701                                         drwxr-xr-x  .svn
    12  end r 2263307  D 1215690762 Thu Jul 10 13:52:42 2008  rr--r--r--  .main.cpp.swx

    Yes, everything is there. Now just restore the files you want using the right inode.

    lt_gergap src # ./ext3grep /dev/mapper/vg-home --restore-inode 2263308
    Running ext3grep version 0.7.0
    Number of groups: 240
    Minimum / maximum journal block: 1547 / 35890
    Loading journal descriptors... sorting... done
    Journal transaction 1202732 wraps around, some data blocks might have been lost of this transaction.
    Number of descriptors in journal: 30993; min / max sequence numbers: 1202700 / 1204860
    Restoring inode.2263308
    mv RESTORED_FILES/inode.2263308 /home/kde4/work/opcua/applications/uaconsoleclientsample/main.cpp
    lt_gergap src # ./ext3grep /dev/mapper/vg-home --restore-inode 2328221
    Running ext3grep version 0.7.0
    Number of groups: 240
    Minimum / maximum journal block: 1547 / 35890
    Loading journal descriptors... sorting... done
    Journal transaction 1202732 wraps around, some data blocks might have been lost of this transaction.
    Number of descriptors in journal: 30993; min / max sequence numbers: 1202700 / 1204860
    Restoring inode.2328221
    mv RESTORED_FILES/inode.2328221 /home/kde4/work/opcua/applications/uaconsoleclientsample/uaconsolesampleclient.pro</pre

That’s all. ext3grep did a great job for me. Thanks to the developers of that tool.

Use svn the correct way
Adding just a folder to svn would have been easy. Just use the -N option to work non recursive way.

svn add -N uaconsoleclientsample
About these ads
Comments
  1. bob says:

    /opt/ext3grep/bin/ext3grep /dev/hda2 –restore-inode 12
    Running Revision: 115
    WARNING: EXT3_FEATURE_INCOMPAT_RECOVER is set. This either means that your partition is still mounted, and/or the file system is in an unclean state.
    Number of groups: 23
    Minimum / maximum journal block: 689 / 17091
    Loading journal descriptors… sorting… done
    The oldest inode block that is still in the journal, appears to be from 33685516 = Mon Jan 25 16:05:16 1971
    Number of descriptors in journal: 1312; min / max sequence numbers: 203 / 675
    WARNING: Not recovering “inode.12″, which is a *UNKNOWN*

  2. gergap says:

    Should this be a question?
    Looks like your partition is still mounted where you try to run ext3grep.
    That one block is from 1971 looks also strange.
    Keep in mind, ext3grep is to recover deleted files,
    it cannot repair a corrupted file system.

  3. dude says:

    very nice, quick tutorial ;)

  4. King-Cake says:

    Hello Nice Howto :)
    But Your Script is not running, that makes nothing.
    root@ubuntu:~# ./blocks.sh
    ./blocks.sh: 2: 28186752: not found

    Whats my Problem ? I have a very many blocks ;)

    THX

    • gergap says:

      You have to store this script as “blocks.sh” and make it executable using “chmod 700 blocks.sh”.
      And of course have to replace my block numbers with yours.
      Also replace /dev/mapper/vg-home with your device file.
      This is just a sample ;-)

      – start –
      #!/bin/sh

      blocks=“255079 336393 336518 336526 395434 395435 395457 737282 984250 1346129 1868670 1869273 1950436 3915933 3915935 4069411 4087953 4216611 4292193 4292196 4292275 4530219 4538370 4538371 4538372 4538376 4538378 4538382 4538385 4543743 4543750 4543752 4544514 4544517 4544528 4544539 4550683 4550707 4655509 4655533 4670417 4670423 4689385 4689746 4785120 5046823 6525842 7370457 7805912“

      for block in $blocks; do
      ./ext3grep /dev/mapper/vg-home –ls –block $block | tee -a output.txt
      done
      – end –

      The result is shown on the output window and is stored in the file output.txt.
      Good luck.

  5. Shaakunthala says:

    Thanks! I was bit confused with this article, which I think written by the developer. But your one is more clear. :)

  6. [...] How ext3grep can save you hours of work [...]

  7. Vangelis says:

    You saved me many hours of works with your post!! :D
    Thank you very very much!!!

  8. Meenakshi says:

    Great post man… Thanks… I am able to restore my data…. :)

  9. Trifaux666 says:

    I had an entire folder full of images I scavenged during the day on the internet that I deleted by mistake, I had a backup of yesterday but your guide allowed me to recover the images that I downloaded during this very day.

    It only took a while for the block analysis (I went to cinema in the while :P), but this is really a great tool for recovering data which is not stored anywhere else and can’t be found on the internet.

    Thank you!

  10. Hi, I think your website might be having browser compatibility issues.
    When I look at your website in Firefox, it looks
    fine but when opening in Internet Explorer, it has some overlapping.
    I just wanted to give you a quick heads up! Other then that, great blog!

  11. Bravo says:

    It will not work with the mounted file system ? I have to mount the disk on read only mode ? or mount in some other system ?
    Also I am looking for a file (not directory) which was deleted by mistake, the search procedure will remain same ?

    • gergap says:

      The filesystem must not be mounted. Read-only may work, I don’t know, I’ve not tested this. If it’s a system partition you should start from a boot CD. Don’t use your system anymore!
      My procedure also recovered files, so this should work.

  12. BlueCola says:

    I’m a linux newbie. I have my full system on just one partition (expect home, which is mounted), and I need to recover a file out of /var/lib. When I do a search I get the error that the filesystem is still mounted, (that’s correct, since I can hardly unmount the filesystem with the systemfiles on it). Is there any way I could recover these files using ext3grep?

    • gergap says:

      Yes, shutdown your system immediately and boot from a Linux Boot CD like SystemRescueCD (www.sysresccd.org).
      In such cases also magic SysRq key helps. You can unmount all file systems (Alt-Print-U) and shut off your system (Alt-Print-o).
      This is the fastest way to safely shutdown without any further file operation.
      See http://en.wikipedia.org/wiki/Magic_SysRq_key
      Every further operation of your running system or rebooting it could potentially overwrite your deleted file(s) your want to recover.

  13. [...] If you are using the ext3 filesystem, then ext3grep is your new best friend. A good tutorial (among many others) on using ext3grep can be found here. [...]

  14. roterfunken says:

    And how do I undo the unmounting (Alt-Pint-U)?

    • gergap says:

      I don’t get your question. Undo umount is simply mount, but why do you need that? In case of an emergency you want to umount to avoid loosing the data of your deleted file. If you used SysRq to umount you can reboot the machine too using SysRQ and boot from a CD like SystemRescue. Afterwards you start the ext3 rescue procedure.
      Note, the normal SysRq way to safely shutdown is to use the Alt-Print-E,I,S,U,B sequence, but if you have a deleted file on your system partition and you want to stop writing to it as quickly as possible just using U,B will be quicker, because you don’t wait for applications to quit. Of course unsaved data of those applications will be lost then. So it depends on you if you want that.
      In my blog above I didn’t use SysRq, my data was on a separate /home partition, so just unmounting /home the normal way was enough. You see, its always good to not have everything on a single partition.
      Good luck!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s