Yup, moving quickly. At my work we are using E1.S NVMe sticks everywhere now with drives costing us less than €70/TB for 8 & 16TB. I’ve seen samples of 30TB E1.L coming in. I think within 5 years the 150-200TB of data I’m having so much trouble using and keeping backed up will be easily stored in a few NVMe drives.
QVOs are fine for me as it’s mostly just data at rest, rarely changed. I even have a few 8TB QVO already and could finance the remainder with the sale of the DS1821s (I get 40%+ Samsung discounts through work)
Currently I’m in a city apartment and I can’t have anything in the rack longer than 450mm. I don’t think there are any JBOD enclosures that meet that. Also most have loud fans/PSUs.
I will be moving to a large house on the edge of the city during the summer. But I’m adding the problem of fiancée/future wife acceptance for noise/space at the same time.
Currently this all fits beside my washing machine and dryer:
All trayless 3.5" bays/enclosures I’ve tried have failed long term or negatively affected hdd longevity.
Other than the rack mounting, I want atleast all the same.
Icydock has an 8 x EDSFF E1.S to 5.25" bay device that they may produce called the CP121.
5.25" bays are extremely versatile, I don’t understand why they are being removed from modern cases.
It’s the reads with the QVO’s that are what I’d be most worried about. NAND charge decay affects them more than TLC. Another worry about QVOs is that NAND charge decay happens faster the more PE cycles the NAND has been through and the QVOs have less PE cycles to begin with.
I wouldn’t trust them to be able to read the data faster than a hdd would for any block on the device written more than a year ago assuming the ssd was at 100% wear health.
This, I actually agree 100% with. While 3.5" will still be a thing for another couple of years, I think even the mighty 100 TB 3.5" drives will be no match for 32 TB m.2 form factor drives eventually. Now it’s just a matter of time for cost curves to do their thing. The last 6 months 2 TB m.2 drives in my region have dropped 20% on the consumer side, it is just insane.
With current speeds, it would not surprise me if I in 2030 could buy a 64 TB consumer m.2 drive for ~$500 or so. Time will tell, but I think HDDs are about to be pushed to obsolete status!
Then again, Sarah Connor and her Pops might have a thing or two to say about that!
I wish SSD manufactures would give us an option in firmware to periodically refresh NAND blocks with “old” data on them so performance/integrity can be maintained long term.
What surprises me is the only time this issue came to the mainstream was when the samsung 840 evos would have the data “age” super fast so that after only ~4 weeks the slowdown occurred.
The mg08s are doing pretty good, I’ve got 12 I’ve been running for 14 months now with no issues.
I went ahead and bought another carton of them a couple months ago in anticipation for building out my next NAS so I’m sitting on 34 of them total now.
I’m not exactly having buyers remorse but I saw that dual actuator 14TB exos 2x14’s just hit ebay for 165USD a piece. They can do over 500MiB/s read/write and are supposedly quieter during seeks because each arm assembly has roughly half the mass of a “normal” hdd arm.
The cons are that can use ~14 watts full tilt; are used drives; and are seagate branded (I’m a little prejudiced).
Be careful with those old Samsung EVO’s. I had a bunch of 850 EVO’s, little ones (250gb), and i killed them all. They started dying slowly first, then quickly. I found one, brand new in the box (I got a deal on a bunch), and I used it as a boot drive. It died quickly as well, but I am pretty sure my 13-year-old Seasonic blew it out when she died (she took all my drives and my AIO with her). My intel SSD’s were tougher, but they were more like Samsung pro models.
I thought they automatically do this now when voltage hits a certain level? We went through that with some earlier SSDs like you said that didnt auto-refresh cell data and people started having data loss so firmware was released that does this process automatically. I have always assumed that since that time, that because we don’t hear about the data loss problems all over anymore that every manufacturer simply performed the cell refresh automatically still. Is that not the case?
I barely know anything about the technical details, but I know this being a thing for dormant/cold storage. And problems that only occur after years are barely considered by most people, marketing certainly won’t tell you. It certainly doesn’t feel like DRAM discharging.
And about the Samsung drives…yeah, I heard of too many problems regarding firmware to buy their stuff. I usually get Micron chips if I have the choice. My very first crucial 120GB SATA SSD is still up and running as it was on day one with some minor SMART reportings.
I didn’t know about the dual-actuator thing. Good to see some innovation, although sequential reads/writes never really was the problem of HDDs (especially not in RAID configs).
But I’m totally fine with triple mirror and saturating my 10Gbit networking. Well-cached and well-designed pool. I may add disks if I need more space, but performance is plenty for my needs.
If you are looking for buyer’s remorse, you will always find it. There is always new stuff coming out, new offers and discounts. If the stuff is doing the job you bought it for and you payed a good price, that’s all you can ask for in a changing tech market driven by innovation. I’m glad I bought the MG08s and not going for a SATA SSD RAID. They gave me so much bang for the buck and I’m having a blast. Who cares what’s up in 5 years.
I’m interested to see what you come up with but suspect that the requirements vs. cost will make this fairly extreme niche.
You’re still going to need to power and cool the thing, (not just fans I mean, but actual ambient room cooling)
I agree with most of this post. If this is to play with storage for the sake of playing with storage (or personal hobby project purpose, etc.) go nuts, just be aware that the job market for that is pretty small, getting smaller and more focused on enterprise array certification.
If its to store stuff at home and/or provide for VM lab storage - you’d be amazed at just how far you can push a handful of SSDs for hot/active storage without needing so many hard drive spindles. For archive just buy a few 40TB SATA drives and call it a day imho. Surely you don’t have more than a couple of hundred terabytes of personal archive requirement?
Unfortunately not.
The problem isn’t so much that the SSDs go straight into loosing data, the problem is the “old” NAND cell voltages are different enough from what the controller is expecting that it goes into a kind of “safety mode” and runs a more granular check of the voltage and infers what levels should be with a look up table (TLC already has 8 different voltages a cell could legitimately be at) and runs through additional ECC routines; These actions cause the read speed to degrade significantly, perhaps up to two orders of magnitude.
If I recall correctly the problem with the old samsung drives quickly and drastically slowing down wasn’t that the cell voltages decayed too much but that the special “safety mode” was activated too liberally.
Here’s a breakdown of file age compared to read speed I just ran on a firecuda 520 with 76% of it’s PE cycles left:
The test isn’t the greatest because of variety of file sizes throwing off transfer rates, but there is a general trend of worse read performance as files age past a certain point. I think garbage collection has also helped out some of the old files too, throwing off the results. It’s hard to quantify the problem until you run across it because there are so many factors.
I think you’re right about this.
But as hdd size is going up, their read/write speeds are remaining the same without this technology, which becomes scary if/when rebuild times start stretching into days.
Another cool hdd technology on the horizon they talked about at the openzfs summit was NVME hdds, the presenters seemed to think these will become an option along with quad actuator hdds further down the line.
ZFS is going to need to catch up to hardware raid in the support of these new multi actuator drives, currently ZFS lets you easily configure a really failure prone setup with dual actuator hdds. Each dual actuator hdd is presented to the OS as two logical drives and their failure mode it to die at the same time so the two logical drives would need to be spread across different VDEVs which at the moment isn’t as easy as it should be with ZFS.
As you can see in the bottom righthand corner the average read speed for the entire disk went from 193 Mb/s to 2519MB/s. For the past 2 years GC had been running in the back ground on SSD and trim was run regularly so they do not factor in.
An interesting point is that I cleared off ~500GB of files before running the test in the hopes that it would run faster compared to the test in the previous post and the average read speed did improve, but perhaps that is just because the majority of the 500GB were old files.
This behavior is typical of every NAND SSD I have ever used (many dozens of different SKUs) except for some of the early Ultra-320 SCSI SSDs.
Bit rots happen in SSD drives when left used because the charges in NAND cells decay. However, the process is not super fast or very slow either from what I read. Often you could expect to find some flipped bits after a year of power off.
As an OS disk or primary storage with sensible usage. I don’t think people have to worry about bit rots or to lesser extent decayed charge level that prolonged read time. For example, a 250GB OS disk, with 10GB write daily, every month all NAND cells are reprogrammed to be fresh as ‘new’.
If people want to perform experiments to show some ‘evidence’ of charge decay behavior, assuming the tools used here good (personally I haven’t used or looked into), how about perform the following procedures:
wipe out the whole disk (new or used; doesnt’ matter) first and recreate the filesystem
create a large file, say 100GB of video; and only this file
at week 0 to 51 with 1 step interval, take the sequential read speed
take the disk offline (powered off) after each measurement
I expect the result will be of minimal variance i.e. week 51 as good as week 0, contrary to what have demonstrated online by these tools.
Bit rot would likely eventually happen as well, but on longer time scales, perhaps 4+ years for this particular drive depending on temperature and wear state.
What I’m more interested in is the fairly fast (<1 year) degradation in read speeds.
To be clear I’m referring to drives that remain under power at all times.
That is not true, while wear leveling and garbage collection would likely refresh some of the cells incidentally, it is very unlikely they would get to all the cells.
That 2TB SSD from the previous post is an OS drive that had at least 50GB of writes to it every day and it still suffers from decayed cells.
This would be interesting to see, but isn’t necessary. In that surface test graphic, the color of the block indicates it’s speed. The fact that there are slow dark blocks mixed in with the fast light blocks shows that some blocks are significantly slower than others, the only reason they would be that slow is cell charge decay. If I were to run that surface test again, all the blocks would be in their fast state because they were refreshed yesterday.
All this goes to show that it isn’t advisable to use flash storage for a file server you expect to have data at rest for years (I would expect that covers most of us home labbers), because by that time SSDs will become slower than HDDs.
All the more reason we should have cases with 3.5" bays.
I’ve started a redesign again, manufacturability and features are being refined. I realized supermicro has stopped development of their 11-slot motherboard standard because of PCIe 5.0, so it doesn’t really make sense to design a case around it (although silverstone demo’d a case for that motherboard form factor last month at computex).
Its a dual chamber design reminiscent of a dell xps 700. The bottom chamber will house 32 3.5" drives and the top chamber 6-7 5.25" bays plus either a matx or atx motherboard (haven’t deiced yet). The dimensions are very close to the original xps 700, a tad smaller.
I’m thinking I’m going to jump on the trend bandwagon and do a wood bottom front (maybe purpleheart?) and fabric/filter around the 5.25" bays.
Only real advice I can give ya at this point is KISS - Keep It Simple, Stupid! The simpler the mechanisms, the easier it will be to manufacture.
Personally I’m now leaning towards a hinged design at the back with a metric shitton of drives on the right and the motherboard + PSU on the left side. Basically, think “Rackmount” except vertically and it folds like a doll house, with 2U + 4U in one combined 6U unit. All you will need at that point extra is a 6U rack.
I’m going to be mechanically affixing the SFF-8639 connectors that come off of the raid card individually to each hdd slot to keep hot swapability and not have to create my own backplane pcb. Each SAS lane is actually about 40% faster/more bandwidth than a pcie gen4 lane so PCB design and signal integrity analysis would be non-trivial if I did try to spin my own.
I decided to copy the mechanism the Lian Li PC-Q26 uses to hold it’s hdds (the thumbscrew in the middle of the image that tightens a moveable locking bar that locks all the drives in place after insertion).
kind of like the supermicro cases that have hdd bays on a hinge?:
I’m about 95% done with the design and wanted to see if anyone has any advice/critiques/comments on the design before I pull the trigger. I’ve been in my own echo chamber while designing this.
The one thing I have yet to model are the front face plates that will have the filters on them, I’ve got ball detents designed into the case to facilitate pulling the front plates off and on already though.
2 120mm fans to ventilate the 5.25"/motherboard section
provisions for small dual 92mm radiator
“retractable” nylon webbing for handles (not modeled)
Some of the things I was on the fence about:
The hdd section fans can only be 25mm in depth in current design, if Phanteks comes out with a 140mm T30 and they’re larger depth I’m going to be kicking myself for not being able to run them… but I don’t want to add extra length to the case now based on a hunch for a fan that doesn’t exist.
Not sure on what the design of the front filter/face plates should look like, I was thinking about just doing vertical pleats of filter material attached to a 3d printed frame that snaps into the case.
Not sure about the 92mm radiator support, it’s such a weird size but there isn’t room for a CPU cooler taller than 70mm with the PSU situated where it is, so if I ever went with a high TDP processor it would pretty much necessitate water cooling.
