Cannot restore $DATA attribute(s)

[crotron]

When I recovered my drive with RecuperaBit, I noticed that some files were producing an error like this one:

ERROR:root:Cannot restore $DATA attribute(s) for File(#4360743, ^^5^^, PMSFX_SAMPLER_MARCH_2021_Part1.zip, offset = 8754294 sectors)

I wanted to know what was causing this, so I did a bit of looking around with a different recovery tool and through RecuperaBit itself. Surprisingly, I found that the recovery tool was able to find the $DATA attribute corresponding to the problematic files. All of the files that couldn't be recovered appear to have the same characteristics. They are all extremely large (> 800 MB). The tool says that there are 5 attributes for each of them: $STD_INFO, $ATTR_LIST, $FILE_NAME, $SECURITY, and $DATA. $STD_INFO, $FILE_NAME, and $SECURITY are tagged R (I assume that means resident) and the remaining two are tagged NR (I assume that means non-resident).

One interesting thing that I'm seeing from the tool is that the $STD_INFO, $ATTR_LIST, $SECURITY, and $DATA attributes are all allocated in one sector (8752866) and $FILE_NAME is allocated in a different sector (8754294).

When I go into RecuperaBit and visit the get_partitions() function in ntfs.py, I notice that both the MFT entries at 8752866 and 8754294 are present in self.found_file. However, when the MFT entry at 8752866 is processed, it seems like only the attributes at that sector come out. Since $FILE_NAME is in a different sector, it isn't present in the attribute dictionary, which causes this data to be ignored (if '$FILE_NAME' not in attrs then continue). On the other hand, when the MFT entry at 8754294 is parsed, it produces a $FILE_NAME and $DATA attribute, with the $DATA attribute having an empty runlist. Since $FILE_NAME is present, this does get added to the list of files, but I assume what happens later on in the get_content(partition) function in ntfs.py is that, when recovering this file, the program checks the $DATA field, sees that there's nothing there due to the empty runlist, and declares a failure.

Something that isn't really clear to me is how the recovery tool is able to connect the two sets of attributes together and consider them part of the same file even though they reside in different sectors. Maybe it has something to do with the $ATTR_LIST attribute. Do you have any insight into what could be going on here and if it is possible to use RecuperaBit to recover these files?

Here are some screenshots from the other recovery tool, if they help. I can provide more data if needed.

Also, here is the raw data as seen in a hex editor.

Sector 8752866:

Sector 8752867:

Sector 8754294:

PS: I didn't know a whole lot about the structure of NTFS going into this, so I've been learning as I go along. Let me know if there's anything confusing that I said due to my lack of understanding.

PPS: Thank you so much for developing this tool! While I wasn't able to recover the extremely large files, I was at least able to recover the vast majority of the smaller ones, which mattered most to me. I was genuinely surprised at how well it worked.

[crotron]

I asked around, and it seems like this behavior is a feature of the attribute list. The exact problem seems to be that when the attribute list in one MFT record points to another MFT record with its filename, the data for the first MFT record is dropped (because it can't find the filename inside it) whereas the second MFT record is kept, but only with its own attributes and not those of the first.

[Lazza]

Have you checked that the files are actually put in the same partition by RecuperaBit? Attributes across different records should work (provided that the attribute list is not damaged) but only if these records are put in the same partition.

[crotron]

Thanks for the reply. In my case, the whole disk was just a single partition (probably unwise in hindsight for such a large drive), so that should not be a problem. It seems to me that the issue is that inside get_partitions (if I haven't simply confused myself), parse_file_record reads in the contents of all of the attributes but doesn't dig into the attribute list to find additional attributes before a check is done to skip files that have no filename. So if the attribute list happens to have all of the filename data, then the entire file will be skipped even though there is logic to dig into the attribute list later on in the code that would uncover the filename.

I managed to partially hack together something that worked for me by calling a modified _integrate_attribute_list (with hardcoded values in place of variables like spc/offset as those have yet to be computed by that point) if $FILE_NAME wasn't found by parse_file_record. With this, I managed to completely recover some of the missing files and I am no longer seeing that error message (though I'm now seeing some files missing sections of data for some reason - still trying to see if this is just being caused by corruption on the drive or something else; it's probably not related to this specific issue).

[Lazza]

Sorry if my previous comment was unclear, I did not intend to ask you if you have a single partition on the drive.

Rather, I wanted to make sure that RecuperaBit did indeed classify both records in the same partition or if it put them into two different partitions.

Can you please check? Thank you.

[crotron]

I reran the code and it is reporting this before presenting me with the option selection menu:

INFO:root:MFT for partition at offset 0 is fragmented. Trying to merge 2 parts...

DEBUG:root:Merging partition with MFT offset 32808 into Partition (NTFS, 3.64 TB, 65241 files, Recoverable, Offset: 0, Offset (b): 0, Sec/Clus: 8, MFT offset: 32, MFT mirror offset: 3907018576) (fragmented MFT)

INFO:root:1 partitions found.

So it looks like RecuperaBit found only one partition in the end. As for whether both records were determined to be in the same partition or different partitions before the merge described in the console output, I'm not sure how to check that at the moment.

[Lazza]

Offset: 0, Offset (b): 0

This is a bit unusual (although will probably work)... are you running the software on a image of the single partition like /dev/sda1 rather than the whole disk like /dev/sda?

Trying to merge 2 parts...

By the way, yes, the merge feature does not always work flawlessly. It looks like you might have found a case when it doesn't.

[crotron]

I'm currently running the software on the whole disk (in my case it is /dev/sdf).

[Lazza]

Sorry but that doesn't seem to be right. RecuperaBit says the file system offset is 0.

How can the file system start at offset 0 if you have a whole drive?
Where is the partition table then?

Or did you just mkfs -t ntfs the whole thing?

[crotron]

When I initially formatted the drive, I used the GNOME Disks Utility in Ubuntu instead of the command line. I don't know the details of how it does this or if it's equivalent to the above, but I do know I didn't bother with splitting the drive into multiple partitions and just formatted the whole drive into one big 4 TB chunk of data.

Sorry I can't really offer a whole bunch of detailed info on this. I formatted the drive a while back and didn't think much of it at the time.

[Lazza]

Let me mention #92 here so we have them cross-linked.