My intel SSD failed. Hard. As in: its content got wiped. But before getting way too theatrical, let’s stick to the facts first.
I upgraded my Lenovo ThinkPad X1 Carbon with a bigger SSD in the late summer this year — a 1TB intel 540s (M.2).
The BIOS of ThinkPads (and probably other brands as well) offer to secure your drive with an ATA password. This feature is part of the ATA specification and was already implemented and used back in the old IDE times (remember the X-BOX 1?).
With such an ATA password set, all read/write commands to the drive will be ignored until the drive gets unlocked. There’s some discussion about whether ATA passwords should or shouldn’t be used — personally I like the idea of $person not being able to just pull out my drive, modify its unencrypted boot record and put it back into my computer without me noticing.
In regard of current SSDs the ATA password doesn’t just lock access to the drive but also plays part in the FDE (full disk encryption) featured by modern SSDs — but back to what actually happened…
As people say, it’s good practice to frequently(TM) change passwords. So I did with my ATA password.
And then it happened. My data was gone. All of it. I could still access the SSD with the newly set password but it only contained random data. Even the first couple of KB, which were supposed to contain the partition table as well as unencrypted boot code, magically seem to have been replaced with random data. Perfectly random data.
So, what happened? Back to FDE of recent SSDs: They perform encryption on data written to the drive (decryption on reads, respectively) — no matter if you want it or not.
Encrypted with a key stored on the device — with no easy way of reading it out (hence no backup). This is happening totally transparently; the computer the device is connected to doesn’t have to care about that at all.
And the ATA password is used to encrypt the key the actual data on the drive is encrypted with. Password encrypts key encrypts data.
Back to my case: No data, just garbage. Perfectly random garbage. First idea on what happened, as obvious as devastating: the data on the drive gets read and decrypted with a different key than it initially got written and encrypted with. If that’s indeed the case, my data is gone.
This behaviour is actually advertised as a feature. intel calls it “Secure Erase“. No need to override your drive dozens of times like in the old days — therewith ensuring the data is irreversible vanished in the end. No, just wipe the key your data is encrypted with and done. And exactly this seems to have happened to me. I am done.
Fortunately I made backups. Some time ago. Quite some time ago. Of a few directories. Very few. Swearing. Tears. I know, I know, I don’t deserve your sympathies (but I’d still appreciate!).
Anger! Whose fault is it?! Who to blame?!
Let’s check the docs on ATA passwords, which appear to be very clear — from the official Lenovo FAQ:
“Will changing the Master or User hard drive password change the FDE key?”
– “No. The hard drive passwords have no effect on the encryption key. The passwords can safely be changed without risking loss of data.”
Not my fault! Yes! Wait, another FAQ entry says:
“Can the encryption key be changed?”
– “The encryption key can be regenerated within the BIOS, however, doing so will make all data inaccessible, effectively wiping the drive. To generate a new key, use the option listed under Security -> Disk Encryption HDD in the system BIOS.”
Double-checking the BIOS if I unintentionally told my BIOS to change the FDE key. No, I wasn’t even able to find such a setting.
Okay — intermediate result: either buggy BIOS telling my SSD to (re)generate the encryption key (and therewith “Secure Erase” everything on it) or buggy SSD controller, deciding to alter the key at will.
Google! Nothing. Frightening reports about the disastrous “8MB”-bug on the earlier series 320 devices popped up. But nothing on series 540s.
If nothing helps and/or there’s nobody to blame: go on Twitter!
— mirko (@foobarblablub) October 25, 2016
— mirko (@foobarblablub) October 25, 2016
— Intel SSD (@IntelSSD) October 26, 2016
Wait, what?! That’s a known issue? I didn’t find a damn thing in the whole internets! Tell me more!
And to my surprise – they did. For a minute. Shortly before having respected tweets deleted.
Let’s take a look on what my phone cached:
The deleted tweets contain a link http://intel.ly/2eRl73j which resolves to https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00055&languageid=en-fr which is an advisory seemingly describing exactly what happened to me:
“In systems with the SATA devsleep feature enabled, setting or resetting the user or master password by the ATA security feature set may cause data corruption.”
“Intel became aware of this issue during early customer validation.”
I guess I just became aware of being part of the “early customer validation”-program. This issue: Personally validated. Check.
Ok, short recap:
- intel has a severe bug causing data loss on 540s SSD and – according to the advisory – other series as well
- intel knows about it (advisory dates to 1st of August)
- intel doesn’t seem to be eager to spread the word about it
- affected intel SSDs are sold with the vulnerable firmware version
- nobody knows a damn thing about it (recall the series 320 issue which was big)
Meanwhile, I could try to follow up on @lenovo’s tips:
— Lenovo (@lenovo) October 26, 2016
Sounds good! Maybe, just maybe, that could bring my data back.
Let’s skip the second link, as it contains a dedicated Windows software I’d love to run, but my Windows installation just got wiped (and I’m not really keen of reinstalling and therewith overriding my precious maybe-still-not-yet-permamently-lost data).
The first link points to an ISO file. Works for me! Until it crashes. Reproducibly. This ISO reproducibly crashes my Lenovo X1 Carbon 3rd generation. Booting from USB thumb-drive (officially supported it says), as well as from CD. Hm.
For now I seem to have to conclude with the following questions:
there’s notI can’t find a damn thing about this bug in the media?
- Why did intel delete its tweets referencing this bug?
- Why does the firmware-updater doesn’t do much despite crashing my computer?
- Why didn’t I do proper backups?!
- How do I get my data back?!?1ß11
PS: Before I clicked the Publish button I again set up a few search queries. Found my tweets.
The “radio controlled power sockets”-project finally got its very own name and project site:
Several improvements happened since my last post about this project:
- there’s finally an Android app now!
- shared-memory is now used for sharing the states of devices between several instances
- there’s now one more tested and working platform: the La Fonera router by Fon
- and – as usual at the end of changelogs: several bugs and timing issues got fixed 🙂
Since it was raining in Ubud most of the time I decided to get back to the coast, back to Sanur.
“My room” in the house of that English-Balinese family in Sanur got a bit, ehrm, “broken” meanwhile:
However the time the weather was reasonable, I tried to catch as much as possible – e.g. the legendary monkey forest: hell, 2500 monkeys trying to rip you off.
Trying to get everything which is not part of your body (cameras, necklaces, earrings, (sun)glasses, etc.) – waiting with your belongings a few meters in front of you for bananas or other food in exchange – seriously, you can buy your stuff back!
After two days in Sanur I moved to the “southern most point” of Bali, Uluwatu. It was my fault to not book a room or an apartment in advance, so I experienced one “sorry, we’re full” after another on site.
Finally some building workers put me into one of their unfinished apartments they were working on. It was overprized, however I was tired and happy I finally found an accommodation.
In Uluwatu, according surfers, there is a couple of best spots highly seeked by surfers – that’s why the few existing quarters down there are full and overpriced all the time.
Since I’m not surfing (and these spots are definitly not proper spots for beginning with) and nothing else was around (the last kind of “bar” around closes at 8PM), I just enjoyed the view, read something and left two days later.
Actually I wanted to go Seminyak and a few days further to Tanah Lot (one of the most adored temples in Bali), but everything there was full, as well (however this time I knew before, as I called them :)).
As I had to cross Kuta (the “Ballermann” of Bali, mentioned in the post before) – and since it was the only place around I was able to get a room reserved for this night – I decided to stay there overnight and move further the day after.
When I arrived in Kuta however, they told me, the guy who wanted to checkout is going to stay longer and there’s no more room available. After hours of walking around looking for and calling hotels / apartments (once again), some locals I met offered me to stay in there place where I am currently.
The reason by the way everything is fully booked is Ramadan.
Will go directly to Tanah Lot this afternoon, after confirming the room reserved there is still available when leaving 🙂
UPATE: I _am_ in Tanah Lot right now, however didn’t get post online in time 🙂
Okay, now some – really little – text I promised…
As you can I see Qt is running inside OpenWrt on the Ben NanoNote of qi-hardware. The device has only 32MB of RAM so this – especially this video I made (qt_openwrt_nanonote.ogm) with it’s coverflow-like 3d and mirroring-effects – shows the great potential of even such embedded hardware.
The Qt packages are not yet committed, I’ll do some cleanups and testing before.
However it’s almost ready to get its way into the OpenWrt packages repository.
Me and most of the other OpenWrt-guys are going to FOSDEM – the Free And Open Software Developement Meeting in Brussels.
See you there! 🙂