Is my NVME drive dying?
Is my NVME drive dying?
My laptop is working just fine. It's from 2018 and it has an NVME drive.
It has an EFI boot partition and other partition with LUKS and LVM on top of that.
Since this week I see these logs from time to time:
Mar 07 17:31:14 almendra kernel: pcieport 0000:00:1d.6: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Mar 07 17:31:14 almendra kernel: pcieport 0000:00:1d.6: device [8086:34b6] error status/mask=00000001/00002000
Mar 07 17:31:14 almendra kernel: pcieport 0000:00:1d.6: [ 0] RxErr (First)
Mar 07 17:31:14 almendra kernel: pcieport 0000:00:1d.6: AER: Error of this Agent is reported first
Mar 07 17:31:14 almendra kernel: nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Mar 07 17:31:14 almendra kernel: nvme 0000:02:00.0: device [8086:0975] error status/mask=00000001/00002000
Mar 07 17:31:14 almendra kernel: nvme 0000:02:00.0: [ 0] RxErr (First)
The devices are:
$ lspci -vv | grep 1d.6
00:1d.6 PCI bridge: Intel Corporation Device 34b6 (rev 30) (prog-if 00 [Normal decode])
$ lspci -vv | grep 02:00.0
02:00.0 Non-Volatile memory controller: Intel Corporation Optane NVME SSD H10 with Solid State Storage [Teton Glacier] (prog-if 02 [NVM Express])
The laptop works like always, but I have the impression that the NVME drive is telling me something bad.
It happens from time to time:
$ journalctl --since yesterday | grep -c "nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical"
9
Do you know what does it mean?
The good news is: the error shown there was a PCIe bus error, which means the error is somewhere between the NVME controller and your processor's PCIe interface. Also good news: the errors you experienced were fully corrected, so you probably lost no data.
So the flash memory in the drive isn't failing. That's good because if the flash memory starts failing, it's probably only going to fail more. In this case, your errors may be correctable: by replacing the motherboard, by replacing the processor, by reseating the NVME drive in its slot, by verifying that your power supply is reliable...
However, if your NVME controller actually does fail, it will be little consolation to tell you that your data is all still there on the flash chips, but with no way to get it. So now might be a good time to make a backup. Any time is a good time to make a backup, but now is an especially good time.
If you keep getting these errors at the same rate, then you probably don't need to do anything, since the errors are being corrected. If you're worried, you could use BTRFS and enable checksumming of data.
[...] by replacing the motherboard, by replacing the processor, by reseating the NVME drive in its slot, by verifying that your power supply is reliable…
I will start with the cheapest option 😅
I assume the power supply is reliable. Having a battery should make it more stable I guess.
And maybe clean the insides of your laptop, that's probably the first thing that could solve the issue. See if all cables are still locked in their connectors. Maybe take out the SSD, clean the contacts and you can use compressed air to clean the socket. But be careful, you want to do it right or you might cause damage. No dampness or water, it has to be either isopropyl alcohol or dry. And don't use a rag that introduces static electricity. And no workshop air compressor. Maybe something like a paintbrush is better suited. And don't just shove the vacuum in. I've done that and it might dislocate small components or key-caps and suck them in and it's a major annoyance to get them out of the vacuum cleaner bag 😆 Just be a bit careful. But I already had something like loose connectors/components cause random errors. Especially in equipment that is moved around or gets dropped occasionally. After 5 years, you might also find some dust inside. At least it used to be that way, It seems to be less of a problem with modern laptops. And more and more stuff gets soldered anyways.
And don't do too much if you're not comfortable with that. IMHO the SSD should be a safe thing to touch for most people. But it's really easy to break or bend some tiny contacts from other components or ribbon cables. And there are consumer devices that aren't really meant to be serviced. I wouldn't disassemble such a model without prior experience. If it's still working you might also leave it as is. Do backups. Storage devices often fail even without prior warning.
Regardless of what it is, make sure your backups are working, running often (daily or better is good), and test your restore process fully.
Smartctl works on nvme drives. Use it.
I did a short and a long test. It looks good
$ sudo smartctl -l selftest /dev/nvme0 smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.7.6-arch1-2] (local build) Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org === START OF SMART DATA SECTION === Self-test Log (NVMe Log 0x06) Self-test status: No self-test in progress Num Test_Description Status Power_on_Hours Failing_LBA NSID Seg SCT Code 0 Extended Completed without error 6334 - - - - - 1 Short Completed without error 6334 - - - - -
Given that it's just an interface error, you could try turning it all off, take the drive out and hit its contacts with electronics contact cleaner (I guess CRC brand is good as any). Work it a little bit, let it dry before putting it all back together.
Another possibility is that power management is being naughty. Fiddle with ASPM or APST.
Oh and do a btrfs/zfs scrub to check that your data is correct.
Doing a scrub on bad hardware will make corruption worse in many cases. When you have faulty hardware freeze everything
This person has had the same device for 6 years. If the drive was used heavily it probably just failed due to age
Yeah, you're probably right. I'm thinking in terms of "not a raid, no redundant copies available" scrub, where the main output would be a sanity check of data checksums.
I used a hand dust blower intended for photography gear. I opened the laptop, blew the dust, disconnected the SSD and blew the socket and it's surroundings.
Now I will monitor the logs and see if it helps.
Thanks.
Dont forget to blow on it
I'm not knowledgeable enough to tell you whether the drive is failing or not, but I just want to double check that you got rolling back ups on this drive right now. Because I'm just an idiot, put to me that drive looks unreliable.
Chances are it is. Always keep good backups.
Honestly its good practice to replace your drives every 5 years. That's not always necessary but it can save you some headaches
Look at smart errors