What Unlost reads on a drive, and why

The one thing everything here rests on

Deleting a file does not erase it. The file system removes the entry that says where the file lives and marks its clusters available again; the bytes stay exactly where they were until something new is written on top of them. Everything below is about the different places a drive writes down where the bytes were, and how much of that survives a delete.

Which is also why the volume being recovered is opened for reading and nothing else. Every handle to a source drive — the NTFS reader's, the FAT reader's, the carver's — is created with read access and nothing else, and there is no write path anywhere behind any of them. That is not a promise about intentions — a handle opened without write access cannot write, and Windows is the one enforcing it. What Unlost does to your drive, and what it doesn't sets out the rest of that.

NTFS: what a delete actually changes

On NTFS every file has a record in the Master File Table. The record holds the name, the dates, the size, and — the part that matters — the run list: the addresses of the stretches of disk the file occupies. A fragmented file's run list is the only thing on the volume that knows the order of its pieces.

When you delete a file, three separate things happen, and they degrade at three different speeds.

• The MFT record is marked free. It is not wiped. It sits there, run list and all, until the next file created on that volume is handed the same slot. Until that happens the deleted file is fully addressable.
• The parent folder's index entry is unlinked from the folder's B-tree — but the bytes of the entry are left where they were. So a folder routinely still remembers the name, size and parent of a file whose own record was given away long ago.
• The clusters go back on the free list in $Bitmap, one bit per cluster. Nothing is written into them at that moment.

Why $Bitmap is the honest source

A record's run list says where a file was. Whether those clusters are still that file's own is a different question, and $Bitmap is the only thing on the volume that can answer it. A cluster that belonged to a deleted file but is marked in use today has been handed to something else and written over; it is not coming back. That comparison is what draws the bar you see next to each file before you recover anything: solid where the file's own clusters are still its own, broken where they have been handed to something else. It is deliberately a map and not a score — a percentage would say a photograph is 80 per cent there, and what the drive actually knows is which stretches went, not how much of the picture they were.

The mistake that files a photograph under a stranger's name

Every reference into the MFT carries a sequence number alongside the record number, because a slot is handed to the next file the moment the one in it is deleted. Following a stale reference without checking the sequence number is exactly how a recovered photograph ends up filed in somebody else's folder. Unlost checks it. Where the parent chain no longer reaches the root, the file goes into a single folder named [unknown folder] rather than a plausible guess — someone sorting through a thousand recovered files is helped by knowing the path is unknown and misled by being shown a wrong one.

Four things that are easy to skip and expensive to have skipped

• The attribute list is followed. A heavily fragmented file cannot fit its run list in one record and keeps the rest in extension records. A reader that stops at the base record reports a whole file as a shred of one.
• Compressed data is decompressed. NTFS-compressed files are read through their compression rather than handed back as the compressed bytes.
• Small files have no clusters at all. A file short enough to fit inside its own MFT record is stored there, and comes back from the record rather than from the disk.
• Backups are used when originals will not read. The NTFS boot sector has a copy in the volume's last sector and the MFT has a mirror; both are read when the primary is damaged. When there is no usable table at all — the case after a reformat — the volume is swept for record signatures and a table is rebuilt. That is the difference between giving files back with their names and giving them back as file0001.jpg.

FAT and exFAT: a smaller record, and one honest danger

FAT32 and exFAT are what cameras and USB sticks use, and they keep much less. A directory entry holds the name, the size and the first cluster; the rest of the file is followed through a separate allocation table that chains one cluster to the next.

Deleting does two things. The first character of the short name is overwritten with a delete marker — which is why recovered FAT names often come back missing their first letter. Unlost shows that letter as _ and does not guess it: a name invented here would be read as fact by somebody hunting for one photo among thousands. And the cluster chain is cleared, which is the real loss.

There is one more trick worth describing because of what it is not used for. Every step of the FAT short-name checksum is reversible, so exactly one first byte can produce the checksum stored in the entry. That byte is never printed as the missing letter. It is used as a test: an answer that could not legally have been in a filename proves that the long name sitting in front of this entry belonged to some other file whose entry has since been reused — so the long name is dropped instead of being attached to the wrong photo.

The journals: where files come back that the table has forgotten

This is the part most recovery tools skip, and it is the single biggest difference between a short list of recovered files and a long one. Everything else on a volume answers with the file system's current state. The journals are the only place that remembers what it used to say.

NTFS keeps two, and they answer different questions

• The change journal ($UsnJrnl) is a flat list of events. Each one carries the file's record number, its name, the record of the folder it was in, when it happened and why. So a file whose MFT record has since been handed to something else still has a name here — and a delete carries the wall-clock time the delete happened. It also carries the previous name after a rename, which matters when somebody is looking for invoice.pdf and the table only ever knew it as Untitled.pdf.
• The transaction log ($LogFile) holds what NTFS was about to write, and what it would have to put back to undo the write. The undo side of "free this record" is the whole MFT record, run list and all. That is the whole point: a file whose record has already been reused can still be recovered in full, not merely named.

On Linux volumes — ext2, ext3, ext4 — the equivalent is jbd2, and the mechanism is even more direct. Deleting a file on ext4 truncates the inode first, and truncating an extent-addressed inode empties the extent tree inside it: the name, the size and the second it went all survive, but where the data was does not. Except that the inode lives in a block of the inode table, the inode table is metadata, and jbd2 writes a copy of every metadata block into the journal before the change lands. The version of that inode with its extent tree still in it is sitting in the journal ring. A file that could only be named becomes a file that can be read back.

Three things the journals give that nothing above them can

1. A name for a file the walk could only number, and a folder for one whose parent chain no longer reaches the root.
2. Addresses for a file the folder index could only name. An index entry holds a name, a size and a parent — it never held a run list. Without one, the file is a row you can see and cannot save. The journal turns it into a file.
3. A file that neither the table nor the index has left at all, because the record was handed to somebody else and the entry naming it went with the folder. Here the copy in the log is the only thing on the volume that still says where this file's bytes are.

They are read every time, not only when things look bad

Reading the journals only when the table came back thin is the wrong saving. A record handed to a live file leaves the table walk looking perfectly healthy while the log still holds the whole of what was lost. Both journals are fixed-size files — tens of megabytes of sequential reading, against a deep pass measured in hours. They are skipped in exactly one case: a volume whose table had to be rebuilt by sweeping has been reformatted, so its journals belong to the file system that was written over the one being recovered.

A run list out of a journal is still measured against $Bitmap before anything is claimed for it. If every cluster has since gone to something else, the file drops back to being a name, and says so.

What is deliberately not decoded

The transaction log has around thirty operation codes. Six are decoded here: the ones that carry a record image or a directory index entry. The rest are counted and passed over, and the count is reported as a number rather than an impression. Half a parse of an operation whose layout is guessed at would put invented names in front of somebody looking for their own files. A journal is also read with damage assumed: a bad length resynchronises on the next boundary, a torn page is skipped, a record cut off by the end of a read window is carried into the next one. On ext, a transaction whose commit never arrived is counted as torn and not believed, because believing one would mean attributing one disk block's contents to another block's address.

Signature carving: what is left when there is no table

After a quick format, or on a card whose directory area will not read, there is nothing to walk. The names are gone for good; the bytes usually are not. So the whole volume is read in one sequential pass and files are recognised by the bytes they start with.

Three rules make this useful instead of misleading.

• A carved file has no name. The names produced are sequence numbers with the right extension, and they never pretend to be the original. The name, the folder and the dates lived in the file table, and the file table is what is gone.
• A guessed length is never presented as a measured one. Where the format states its own length, it is read. Where it does not, the file is capped and the name carries _partial.
• A candidate with nothing behind it is not offered at all — short magic bytes, no length, no end marker, no neighbour. A list of files that open into nothing costs you more than a shorter list does.

How the end of a file is found

Finding the start is the easy half. Eleven different rules decide where a file stops, because the formats do not agree on how to say it. Some state their length in the header — the cheapest answer and the only one that cannot be fooled by what the file happens to contain. Some are walked structure by structure: a JPEG through its segment lengths, an MP4 or HEIC through its box headers, a ZIP or Office document through its entry list, an old .doc through the document's own internal allocation table. Audio without a container is followed frame header to frame header and ends at the first place the chain stops holding together, which is what stops a carve from handing back an hour of silence. And one rule is the weakest in the set and is used only when nothing else applies: the file ends where the next recognised header starts.

The reason for all that machinery is that the obvious approach does not work. A JPEG's end-of-image marker turns up in erased space roughly once every sixty-five kilobytes, so searching for the last one returns rubbish and searching for the first one cuts the photo off at the end of its own embedded thumbnail. Short magic bytes have the same problem: a four-byte signature comes up by chance about once in four billion sectors, while a two-byte one comes up once in sixty-five thousand — so any signature shorter than four bytes has to be confirmed by parsing the structure behind it before it is believed, and the build fails if a new format is added without one.

The pass recognises 64 file types by their contents, weighted towards what people actually arrive here having lost: photographs and video get the exact length rules and the widest coverage, including eleven camera raw formats. And when a deep pass follows an ordinary scan, it reuses the reads the scan already made rather than reading the drive twice — a dying flash controller and a failing head both have some number of reads left in them, nobody knows what the number is, and a second full pass spends it for nothing.

Why checking a recovered file gives three answers, not two

A recovery can hand back a file of exactly the right length whose insides belong to something else. From the outside it is an ordinary JPEG of an ordinary size. You find out at home, opening photographs one at a time. So every recovered file is checked against its own format, and the check has three possible outcomes rather than two.

The third answer exists because the formats are genuinely not equal, and pretending otherwise is where the lie would be.

• PNG and ZIP-based files — which includes .docx, .xlsx and .pptx — store a checksum beside every chunk and every entry. For these the answer is arithmetic, not opinion. A single changed byte anywhere in the picture fails it. That is the strongest statement any format in this list can make.
• JPEG, GIF, PDF, the MP4 family and old Office files carry no checksum, but they carry a structure that has to close: the segments must chain, the boxes must land exactly on the end of the file, the PDF's cross-reference offsets must all point inside it. That answer is weaker, and it is described as weaker. A photograph whose middle was overwritten can still walk cleanly from marker to marker.
• Bare MP3 and the TIFF-based camera raw formats carry neither. The only honest thing to say about them is that nothing can be said — so the app says exactly that, and tells you to open the file.

A photographer's raw files are the most valuable thing this program ever touches. Reporting them intact because nothing disagreed would be the most expensive lie it could tell.

The format check does not get to overrule what the scan already knew

The two can only disagree in one direction that matters. A format check that fails is a fact and wins outright. A format check that passes is not a promise that the picture is the right picture. So when clusters were known to be missing before the file was ever opened, that knowledge stands: the file is reported broken with the reason the scan gave, rather than intact with the reason the format gave.

One clarification that matters more than it sounds: what counts is the number of clusters that went to something else, not the number of pieces the file is in. A file scattered across a drive in a dozen fragments can be perfectly whole. Reading the fragment count instead would mean telling people their photograph is ruined when it is not.

And a read error on the destination drive is never reported as damage to the file. That is a fact about the destination, not about the recovered bytes.

Drives that are physically failing

Damage is treated as expected rather than exceptional. Reads happen in large blocks for speed; when one fails, the same span is retried in 64 KB blocks so the good sectors around the bad ones are still recovered. The bad blocks are filled with zeros, which is the honest filling — no signature matches zeros, so nothing gets invented out of a hole, and no other file's data ends up in the gap. A media error is never fatal to a scan: the person reading this is holding the failing card, and a scan that stops at the first bad sector recovers nothing.

For a drive that is actually dying, the right move is to stop scanning it and copy it first. A scan reads for hours and seeks all over the disk; one sequential pass front to back is the gentlest thing that can be asked of a failing controller, and after that the copy can be searched a hundred times for free. Unlost images a drive into an ordinary file, writes down the address of every stretch that would not read in a log beside the image, and — this is the payoff — a file that lands on one of those stretches is reported as damaged when it is found, instead of discovered to be damaged when the photograph opens grey. An image that exists is a finished one: an interrupted copy leaves a partial file or nothing, never something that looks complete. The guide on failing drives covers when to do this.

Encryption: identified, never guessed at

Unlost does not decrypt anything, and says so before it does anything else.

A BitLocker volume is recognised before any other probe runs — including BitLocker To Go, which hides behind a working FAT32 boot record and which every other probe would happily hand over as a FAT32 volume. That check comes first for one reason: carving encrypted data is the one thing this program must never do. Encrypted bytes are uniform noise, every signature match in them is a false one, and a list of invented photographs handed to somebody whose real photographs are all still there, safe behind a key, is the worst outcome available. What it does instead is read the unencrypted metadata and tell you what you are looking at: the encryption method, which unlock methods the volume has, and above all the recovery key identifier — the one fact that turns "somewhere in my Microsoft account there are nine recovery keys" into "this one".

Windows EFS-encrypted files are found, named and listed, and marked as what they are: the data is right here in full and the key is in a Windows account. They are kept separate from files that are missing their addresses, because the two refuse for opposite reasons and the thing you would do about them is different.

Getting to the volume in the first place

Half of "the drive doesn't show up in Windows" is a container problem, not a file problem. Partition tables are read by three methods, in order of how much each has to assume: the MBR table and its chain of extended entries; the GPT header and entry table, with checksums over each, plus the second complete copy at the far end of the disk that makes a destroyed table survivable; and failing both, a sweep of the medium testing every sector for the start of a file system, where a hit is believed only once the structures behind it have been parsed and made to agree with each other. That last one is slow, and it is the one that finds a partition nothing has recorded.

Sector size is settled once against the medium rather than assumed, because on a 4K-native drive a table read in 512-byte units puts every partition at an eighth of its real address — not slow, wrong. Linux LVM volumes are read out of the LVM metadata, including the metadata ring's history, which is the only thing on a disk that still says where a logical volume that has since been removed used to be. A logical volume whose extents run onto a disk that is not present is reported as incomplete and named, rather than handed over as though the missing half were zeros.

The full list of what is read and what is only recognised is on what Unlost can read.

How this is tested

Raw access to a real drive is an administrator privilege, so the test suite does not touch one. It does the next most honest thing: it builds a real FAT32 volume, a real exFAT volume and a real NTFS volume inside ordinary files, puts files with known contents into them, deletes those files exactly the way Windows deletes one, and then runs the shipping readers, the shipping carver and the shipping recovery code over the result. The measurement at the end is a SHA-256 of the original against a SHA-256 of what came back — because a length is not a measurement, and a file that comes back the right size with the wrong bytes has to fail somewhere.

Three things are asserted. A contiguous deleted file comes back byte for byte. A file whose clusters something else has taken does not — and the engine says so before the recovery runs rather than after. And the carver, given a volume whose directory has been wiped, finds the file by its leading bytes and returns the same bytes again.

There are more than 450 automated tests in the engine, close to 300 of them in that suite. None of this is a substitute for running the program on your own card. It is a substitute for guessing.

Things it will not guess

Most of the design decisions above are the same decision in different places: where the drive does not say, the program does not invent. Collected in one list, because it is the fastest way to judge any recovery tool, including this one.

• A FAT name's missing first letter is shown as _, never reconstructed.
• A file with no recoverable parent goes to [unknown folder], never a plausible path.
• A carved file gets a numbered name, never a made-up one.
• A guessed length is marked _partial in the file name itself.
• Deletion dates are printed as days, never times of day — "it went at 14:07" is a number that looks more precise than the thing it describes.
• An unreadable allocation map means every file is drawn as untouched, and you are told that this is the absence of a check rather than a check that passed.
• A listing that hit its limit says so, so that anything you cannot find has not been looked for rather than been shown to be gone.
• Nothing is written to the destination until there is room for all of it, and a read that fails part way through takes its half-written file with it. Half a photograph under the right name is worse than none.