Langa Letter: File & Disk Tools

It was late. I was tired. And I screwed up, big time.
I'd been on the West Coast for a week, and was on the daylong west-to-east return trip. Thanks to frequent-flier miles, I had a good seat on the plane, and was able to work on my laptop almost all the way across North America. I landed with hundreds of E-mails queued on my hard drive, and dozens of new files created or altered. All I needed was to drive home, transfer the files to my main system, reconnect to the Net, and I'd be off and running.

But when I arrived home, I started doing too many things at once: While checking the house, turning on lights and the heat, I booted my home office server and primary desktop system, and then started unloading the car. In between trips, I hooked up the laptop to the LAN, and started synching files with the desktop PC.

After a couple minutes of ferrying more stuff in from the car, some foggy synapse finally fired: Uh-oh. I stopped short, realizing that I'd synched in the wrong direction. I'd also used the synch-tool's dangerous nondefault settings to clone the drive's contents. It's the setting I normally use when setting up a new PC, not for catching up after a trip. Instead of the laptop sending new and changed files to the desktop PC, my desktop PC was busily overwriting the new files on the laptop with the old files from before I left on the trip. Doh!

I could blame jet lag or any number of other factors, but it was just a plain, unadorned, ugly brain-fade: I wasn't thinking clearly, and had done too much, too fast.

I raced through the house to the office and aborted the synch -- I had no idea how far it had gotten, but it was enough so that I had a really bad feeling in the pit of my stomach. I then carefully -- very carefully -- re-synched in the proper direction, pulling files off the laptop to the desktop system, which is what I'd meant to do in the first place. This made no further changes to the laptop system, and did get whatever was left of the new files backed up onto the desktop PC.I then surveyed the damage. Fortunately, I have large hard drives with lots of files, so most of the original, aborted synch had been comparing files that were identical on both systems. And it turned out that I'd aborted the bad synch before any of the mail files were munged. Whew.

But once I waded through the file structures, I found there were at least four important files that had been reverted to their pre-trip state. It could have been much, much worse, but it still wasn't good news: I could recreate the four rolled-back files, but it would cost me probably half a day's work.

Then I got thinking: Maybe the files were recoverable. The files in question were overwritten, not simply deleted, so basic "unerase" or "undelete" tools weren't likely to help. If the new text existed anywhere, it would probably be outside of the active file areas, somewhere on the laptop's hard drive.

I'd used deep-geek recovery tools -- sometimes called "hex editors," "disk editors," "sector editors," or "programmers' editors" -- in the past, but the one I had on hand was an ancient DOS-based fossil. So I went looking for a newer version, and therein lies a tale.

Fred Langa. Courtesy of Information Week

In this context, "hex" stands for "hexadecimal," the low level, machine-friendly base-16 notation system used in many computer programs and codes.In theory, a "hex editor" can let you see and modify anything and everything anywhere on your hard drive, including any and all kinds of files and their contents, and even the disk's own fundamental data structures.

Some hex editors are file-oriented; you can easily use this kind of tool to change program code even in executable files, in DLLs, and in other usually inaccessible places. You can use this kind of hex editor to remove annoying branding on some software. For example, you could change or remove the "Microsoft Internet Explorer" that appears at the top part of every IE browser window. That, or any other plain text coded within EXE and similar files, is easily changed with a hex editor.

Hex editors also are useful for exploring mystery files that you can't open by any other means: A hex editor will let you see what's in almost any file, and sometimes can provide enough clues so you can figure out what an unknown or unopenable file is, or where it came from.

File-oriented hex editors also often are optimized for the recovery of accidentally deleted files; they can let you find, identify, rename, and save (as a new file) anything that was mistakenly erased.

Some hex editors are geared to other special purposes, such as manually sorting out problems with the boot process or with partitions and logical disks; including unformatting, unpartitioning, or finding/undeleting lost partitions.While task-specific hex editors can make certain tasks easier (mostly by pointing you in the right directions), general-purpose hex editors can do it all, letting you view -- and optionally modify -- anything that's anywhere on your hard drive. This kind of hex editor is often used in digital forensics and in heavy-duty file- and disk-recovery: It will show you absolutely everything on the hard drive -- including every file, every deleted file, and even bits or scraps of data left over outside the active, in-use file areas. This can include residual data from deletion or defragging operations; data in normally unviewable areas (such as the swapfile or pagefile); and data left in the "slack" space after an end-of-file marker. (If these concepts are unfamiliar to you, see the information here, here, or here.)

The flip side is that general-purpose hex editors show you so much "raw" data, they can be hard to use, especially if you've never used one before. The special-purpose editors may have simpler, easier-to-use interfaces, as long as you're using them for their more-limited intended purpose.

But the above three general categories aren't at all rigidly defined: under the skin, all hex editors share some basic similarities. The differences from one editor to the next reside mostly in what functions are being optimized and emphasized, and how the front ends or interfaces are built. When push comes to shove, a general-purpose editor can be used for something like editing boot records, for example; and a drive-oriented editor may be used for editing specific files.

One thing all the editors share in common is that they can be quite slow when you're searching today's huge hard drives. That's not the fault of the editor, but simply a reflection of the amount of data they may have to process. Plus, all hex editors can be dangerous and must be used with care -- they give you the power to modify almost anything on the hard drive, including things best left alone. Many hex editors come with some form of disk imaging built in; or at least come with the strong recommendation to make an image by some other means before attempting to use the editor. (With a fresh image, you'll be able to recover from any mistakes or errors.)

There are literally hundreds of editors available, as even a cursory Google search will show. I've selected three examples -- one from each of the broad categories above -- simply as exemplars, not necessarily as recommendations. Please read the text and follow the search links before making a choice as to which one to try."Active Undelete" (feature-limited free trial; $40 and up to purchase full version) is a file-oriented tool optimized for file recovery. Its preconfigured "scans" (whole-disk searches) are designed to sniff out all recoverable deleted files on your system, and present them for easy undeletion.

However, under the skin, it still is a hex editor, and the "Disk Hex Editor" tab reveals a basic interface for editing files, as Screen Two shows. Because this is only a demo, I've used the editor to open a completely nonessential, expendable file -- Freecell.Exe.

Here, you can see that Active Undelete's editor window shows the tripartite information common to most hex editors -- the blue numbers show the reference position or "offset" within the file; the black two-place numbers in the middle are the actual hexadecimal coding of the file; and the far right pane shows the plain-text equivalent of the hex code. (Where there is no plain-text equivalent, a dot is used as a placeholder.)Active Undelete's optimizations make it a good choice for its intended purpose -- deleted file recovery -- but the optimizations are extreme enough to that Active Undelete isn't very useful as a general purpose hex editor. For example, it can be hard to impossible to scan the unused portions of a hard drive for data scraps; and the hex editor interface has confidence-sapping defects, like labeling cluster as "cluser" in Screen Two. Still, Active Undelete does serve to illustrate the kinds of tradeoffs available to you in the spectrum from task-specific tools to more general-purpose ones.

The "Acronis Disk Editor" is another optimized, or special-purpose editor. It's part of their "Disk Director Suite" (a feature-limited free trial of the Disk Editor is available; the full suite is $60 and down; price depends on quantity purchased). As you can see in Screen Three, the Disk Director Suite is mainly a partition- and boot-management and repair tool.

The standalone Hex Editor -- Acronis calls it a Disk Editor -- bundled in the suite is likewise optimized for large-scale operations rather than anything as fine-grained as editing a specific file. For example, the Disk Editor's search does let you find data anywhere on the disk, whether in a file or not. But once found, you may be in for some head scratching because the tool simply isn't optimized for this kind of work. To continue the example, in Screen Four, I used Acronis Disk Editor to locate the text string "freecell" on the hard drive, and the editor did just as I asked. But look at the screen and tell me what you'd do next with the information shown.

What I did next was to click on Help and start wading through the text there.

So, like Active Undelete, Acronis' Disk Editor (indeed, the whole Disk Director Suite) is fine if you're looking for a special-purpose tool; in this case for manipulating and repairing partitions, book records, and the like. But it's so optimized that it's not very good at other hex editing tasks.

One of the best general-purpose hex editors I know of is WinHex (free trial; prices are in Euros, and at current exchange rates range from about $50 (US) for a personal-use license to about $500 (US) for a full "Forensics" version with extremely sophisticated features.)

I like this tool a lot. You'll find a full description at the Web site, but here's a snippet:

WinHex's interface isn't flashy, but it's extremely functional. (The help system's good, too.) WinHex will search anywhere on a hard drive, locating data in or out of active file areas. It will show you all the normal and deleted files on a hard drive, much like Active Undelete. It can manipulate partition tables and boot records, much like Acronis' tool (though not as colorfully or graphically). But unlike the special-purpose tools we looked at, WinHex is also an outstanding text-finder and manipulator.

In fact, WinHex makes it incredibly easy to get at almost anything on the hard drive -- if the information's there, you'll be able to get at it, manipulate it, and save it. As a general-purpose hex editor, I think it's an outstanding tool; and this one, I do recommend. Check out the Web site and grab yourself a demo copy.

To close the circle, my exploration of hex editors was driven by the dire need to try to recover the four files I'd accidentally overwritten.

WinHex came the closest to helping, finding several fragments of interim saves of the files here and there on the hard drive. But when a file is truly overwritten, it's usually beyond software recovery -- you need special hardware that can ignore the most recent and strongest magnetic signals on the disk, looking instead "beneath" (or actually beside, in some cases) the newer files. It's a slow and expensive process, and usually trashes the hard drive in the process.

If the files had merely been deleted, I have no doubt that either WinHex or Active Undelete would have been able to help. But my overwritten files were well and truly gone, and beyond the reach of software. Sigh.

But the good news is that I can reconstruct the lost files -- I've only lost a couple hours time. And I've found a great new tool -- WinHex -- to add to my collection against future need.Now, if I can just remember not to try file transfers when I'm tired and distracted... .

What's your experience with hex editors? Which have you used, and to what effect? Have you tried any of the narrow-focus, task-specific tools, or the general purpose editors? Join in the discussion!

To discuss this column with other readers, please visit Fred Langa's forum.

To find out more about Fred Langa, please visit his page