intel 540s SSD fail

My intel SSD failed. Hard. As in: its content got wiped. But before getting way too theatrical, let’s stick to the facts first.

I upgraded my Lenovo ThinkPad X1 Carbon with a bigger SSD in the late summer this year — a 1TB intel 540s (M.2).

The BIOS of ThinkPads (and probably other brands as well) offer to secure your drive with an ATA password. This feature is part of the ATA specification and was already implemented and used back in the old IDE times (remember the X-BOX 1?).

With such an ATA password set, all read/write commands to the drive will be ignored until the drive gets unlocked. There’s some discussion about whether ATA passwords should or shouldn’t be used — personally I like the idea of $person not being able to just pull out my drive, modify its unencrypted boot record and put it back into my computer without me noticing.

In regard of current SSDs the ATA password doesn’t just lock access to the drive but also plays part in the FDE (full disk encryption) featured by modern SSDs — but back to what actually happened…

As people say, it’s good practice to frequently(TM) change passwords. So I did with my ATA password.

And then it happened. My data was gone. All of it. I could still access the SSD with the newly set password but it only contained random data. Even the first couple of KB, which were supposed to contain the partition table as well as unencrypted boot code, magically seem to have been replaced with random data. Perfectly random data.

So, what happened? Back to FDE of recent SSDs: They perform encryption on data written to the drive (decryption on reads, respectively) — no matter if you want it or not.
Encrypted with a key stored on the device — with no easy way of reading it out (hence no backup). This is happening totally transparently; the computer the device is connected to doesn’t have to care about that at all.

And the ATA password is used to encrypt the key the actual data on the drive is encrypted with. Password encrypts key encrypts data.

Back to my case: No data, just garbage. Perfectly random garbage. First idea on what happened, as obvious as devastating: the data on the drive gets read and decrypted with a different key than it initially got written and encrypted with. If that’s indeed the case, my data is gone.

This behaviour is actually advertised as a feature. intel calls it “Secure Erase“. No need to override your drive dozens of times like in the old days — therewith ensuring the data is irreversible vanished in the end. No, just wipe the key your data is encrypted with and done. And exactly this seems to have happened to me. I am done.

Fortunately I made backups. Some time ago. Quite some time ago. Of a few directories. Very few. Swearing. Tears. I know, I know, I don’t deserve your sympathies (but I’d still appreciate!).

Anger! Whose fault is it?! Who to blame?!

Let’s check the docs on ATA passwords, which appear to be very clear — from the official Lenovo FAQ:

“Will changing the Master or User hard drive password change the FDE key?”
– “No. The hard drive passwords have no effect on the encryption key. The passwords can safely be changed without risking loss of data.”

Not my fault! Yes! Wait, another FAQ entry says:

“Can the encryption key be changed?”
– “The encryption key can be regenerated within the BIOS, however, doing so will make all data inaccessible, effectively wiping the drive. To generate a new key, use the option listed under Security -> Disk Encryption HDD in the system BIOS.”

Double-checking the BIOS if I unintentionally told my BIOS to change the FDE key. No, I wasn’t even able to find such a setting.

Okay — intermediate result: either buggy BIOS telling my SSD to (re)generate the encryption key (and therewith “Secure Erase” everything on it) or buggy SSD controller, deciding to alter the key at will.

Google! Nothing. Frightening reports about the disastrous “8MB”-bug on the earlier series 320 devices popped up. But nothing on series 540s.

If nothing helps and/or there’s nobody to blame: go on Twitter!

#ssd (@intel 540s) wiped after changing the ATA password in @lenovo #thinkpad BIOS. Either buggy BIOS or broken SSD controller. Be careful.

— mirko (@foobarblablub) October 25, 2016

Some Ping-Pong:

LSF031C

— mirko (@foobarblablub) October 25, 2016

Then…

https://twitter.com/IntelSSD/status/791299306892308480

Wait, what?! That’s a known issue? I didn’t find a damn thing in the whole internets! Tell me more!

And to my surprise – they did. For a minute. Shortly before having respected tweets deleted.

Let’s take a look on what my phone cached:

The deleted tweets contain a link http://intel.ly/2eRl73j which resolves to https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00055&languageid=en-fr which is an advisory seemingly describing exactly what happened to me:

“In systems with the SATA devsleep feature enabled, setting or resetting the user or master password by the ATA security feature set may cause data corruption.”

Later on:

“Intel became aware of this issue during early customer validation.”

I guess I just became aware of being part of the “early customer validation”-program. This issue: Personally validated. Check.

Ok, short recap:

• intel has a severe bug causing data loss on 540s SSD and – according to the advisory – other series as well
• intel knows about it (advisory dates to 1st of August)
• intel doesn’t seem to be eager to spread the word about it
• affected intel SSDs are sold with the vulnerable firmware version
• nobody knows a damn thing about it (recall the series 320 issue which was big)

Meanwhile, I could try to follow up on @lenovo’s tips:

Or from the Intel website https://t.co/sLiVTzQPcC or via the SSD Toolbox https://t.co/jnw2fxbDrj

Ray

— Lenovo (@Lenovo) October 26, 2016

Sounds good! Maybe, just maybe, that could bring my data back.

Let’s skip the second link, as it contains a dedicated Windows software I’d love to run, but my Windows installation just got wiped (and I’m not really keen of reinstalling and therewith overriding my precious maybe-still-not-yet-permamently-lost data).

The first link points to an ISO file. Works for me! Until it crashes. Reproducibly. This ISO reproducibly crashes my Lenovo X1 Carbon 3rd generation. Booting from USB thumb-drive (officially supported it says), as well as from CD. Hm.

For now I seem to have to conclude with the following questions:

• Why there’s not I can’t find a damn thing about this bug in the media?
• Why did intel delete its tweets referencing this bug?
• Why does the firmware-updater doesn’t do much despite crashing my computer?
• Why didn’t I do proper backups?!
• How do I get my data back?!?1ß11

 

PS: Before I clicked the Publish button I again set up a few search queries. Found my tweets.