Hey so I bought some of these 118GB optane disks and had some trouble switching them to use 4k blocks. I thought I would post something here saying how I did it.
$ echo 0 >/sys/module/nvme_core/parameters/default_ps_max_latency_us
$ echo 0 >/proc/sys/kernel/hung_task_timeout_secs
$ time nvme format -fl1 --timeout=3600000 /dev/nvme2
Success formatting namespace:ffffffff
real 14m19.103s
user 0m0.002s
sys 0m0.000s
Basically formatting this thing takes too long and I had to keep turning off things that would time out the command. If a timeout was hit I would get a line in dmesg about the nvme controller or something then normal reboots would hang. Then I would have to use echo b >/proc/sysrq-trigger to reliably reboot without having to go and hold the power button.
Hope this helps someone.
On an unrelated note… it looks like blkdiscard doesn’t work on these things or the 900GB 905Ps…
Could you tell what distro and kernel version you were using. I’ve tried doing the same but I can’t reformat the drive.
I couldn’t do it quickly on gentoo because kernel lacked /proc/sys/kernel/hung_task_timeout_secs.
I’ve tried using ubuntu 22.10 and arch livecd.
On ubuntu kernel 5.15 I killed nvme format after 25 minutes. Drive was just dead and I had to reboot to check it’s state.
On arch 6.2.8 livecd kernel operation timed out despite writing 0 to default_ps_max_latency_us and hung_task_timeout_secs
Drive timed out, reseted itself and appeared back in nvme list
I did format over the weekend (NVMe) using the tool provided by BIOS. I am guessing it took around that long. But I let it run all the way to the end and reboot itself. It’s probably normal to take that long.
Not sure if it’s related but some 1733 firmware has 512/520 and 4k/4k+dif which is the data integrity field formatting option. It might or might not be that.
I’m almost sure it is. This is what I get when attempting to do the same on a P5800X: 5 formatting options, 2 for 512 and 3 for 4k sectors. The first options for each sector size are usually for 0 metadata per sector. Since it is a consumer motherboard, we are probably not given the metadata sizes for the other options.
Do you happen to have experience with flashing Samsung enterprise drives?
Thanks to a user from the ServeTheHome forums the actual firmware files seem to have been made “available”, but I’m a bit worried damaging the drives:
I’ve done it only from Samsungs bootable iso which is a Linux distro. Back up the drives. I did it one drive at a time on a 5 drive raid array once and made sure each drive had the same md5 sum after every every one. If ones md5 sum changed I was gonna let the raid rebuild as I was worried about it
To answer your question I am using ubuntu server 22.04 LTS. I don’t know what kernel I was using. I’m currently on 5.15.0-69-generic for what it’s worth.
As @wendell correctly intuited, two of the formats include extra bytes per sector for data integrity. My understanding is that the lowest relative performance value gets you the highest absolute performance.
On NAND, sure. It just seems to be an issue with 3D Xpoint.
I have read that it does pass some “hints” to the controller that can help it optimize… something… but I’m not sure how significant the effect is, or if it even is at all.
That’s not accurate. Here are the LBA formats for a P1600X 58GB (firmware version U5110550):
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 - 512 0 2
1 + 4096 0 0
And for a P1600X 118GB (firmware version U5110550):
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 - 512 0 2
1 + 4096 0 0
This is really helpful, thanks. I’ve also had a lot of trouble changing the format of P1600X drives, even using the official Intel tool. On every other model NVMe I’ve ever used it’s a simple nvme format --lbaf=1 or whatever and it returns immediately. It seems to take forever on the P1600X; sometimes I literally have to reboot the system after an hour or two of waiting and try again, and then it usually takes about 15 minutes. I’ll try your workaround next time.
I haven’t run the Intel tool in a while but maybe you’d have better luck with that? You can download the CLI packages from here and the command is something like:
I found the same thing on optane 4k sectors but does it have to do if the optane is connected via normal pcie lanes or the VMd/vroc lanes? Might be worth trying with and without VMd modes to see if that lets you format as 4k
In my case at least I’ve experienced the issue using CPU lanes to attach a P1600X on a SuperMicro C612 board, an ASRock DeskMini X300, and an Asus X570 motherboard.
I just wanted to add on my experience: I tried this same procedure on a few different machines, plus a couple others.
TrueNAS scale bluefin: Does not work, kernel resets the target device.
Ubuntu 22.04 live USB: Looked like it was working but never completes. Requires reset.
Windows using intelMAS: gives immediate error “operation not supported with this driver”. intelmas start -intelssd $index -nvmeformat LBAformat=1
I tried both nvme-cli and intelMAS, as root (admin). Nothing actually worked.
Here’s what worked the first time with zero issues:
Asus secure erase utility in a z690 mobo. It took around 15 minutes per drive.
Thanks for the idea /u/aBav.Normie-Pleb
So. I plan to do some digging. I imagine that something in the intel firmware is being picky about the order of operations. The various OS are interacting with the drive to get it into a usable state for the system, and this may preclude it from using the nvme commands without a controller reset, or some other intermediate steps.
I came back to this a short time later and found a reliable method to reformat these drives, I boot using the latest arch linux iso and used these commands.