im in the process of building a new workstation. this will be used for stress testing multithreaded programs that use sql server, postgres and oracle. It will also be used to some extent to play games. will run windows server datacenter 2022 and could have dual/tripple boot to windows 11 pro and proxmox. some games play is anticipated but that is secondary.
this will replace my current system that is used as described above.
current hardware choices:
ASUS Pro WS WRX90E-SAGE SE EEB Workstation motherboard
AMD Ryzen Threadripper PRO 7965WX
Super Flower Leadex Titanium 1600W 80+ Titanium, ECO Fanless & Silent Mode, Full Modular Power Supply, Dual Ball Bearing Fan (1600W) (may consider other power supplies, currently use seasonic)
CORSAIR WS RDIMM Series 128GB (8 x 16GB) ECC Registered DDR5 5600 (PC5 44800) Server Memory
SilverStone Technology XE360-TR5 360mm All-in-One Liquid Cooler for AMD TR5 / SP6, SST-XE360-TR5
Fractal Design Define 7 XL
Kraken Elite 360 nzxt cpu cooler
G.Skill Zeta R5 Neo 128GB (4 x 32GB) DDR5-6400 PC5-51200 CL32 Quad Channel ECC Registered Desktop Memory Kit F5-6400R3239G32GQ4-ZR5NK
MSI NVIDIA GeForce RTX 4090 Gaming X Slim RGB Overclocked Triple Fan 24GB GDDR6X PCIe 4.0 Graphics Card
SAMSUNG 990 EVO PLUS SSD 4TB, PCIe Gen 4x4 | Gen 5x2 M.2 2280, Speeds Up-to 7,250 MB/s, Upgrade Storage for PC/Laptops, HMB Technology and Intelligent Turbowrite 2.0 (MZ-V9S4T0B/AM)
SAMSUNG 990 EVO PLUS SSD 2TB, PCIe Gen 4x4 | Gen 5x2 M.2 2280, Speeds Up-to 7,250 MB/s, Upgrade Storage for PC/Laptops, HMB Technology and Intelligent Turbowrite 2.0 (MZ-V9S2T0B/AM)
this is running tests, data integrity doesnt matter. i restore db, run test, restore db etc. and hba with batteries is an old tech. as for zfs file system the majority of my tests are for sql server. vms are hosted on hyperv, even if linux of some flavor so running zfs on top of hyper-v provided luns isnt need. if do deploy proxmox that may change. the above scenarios are what i have been running for years. company i work for provides servers at work that provide the next phase of stress testing after validation in home labs.
i would assume you would suggest u.2 drives. im currently looking at those with some confusion. the motherboard has two slimsas connectors. but ive read that connecting u2s drives via a cable can be very iffy. the alternative would be some sort of pcie card that allows directly attaching u2 drives. this is a new uknown for me (u.2s and how to connect) . i have run 15k cheetah SAS drives in the past and have an old adaptec SAS raid card but my experience with such systems is fairly old.
btw. i would very much like drive bay in the front with 1 or 2 slots for u.2s if i went this way. this is what i am used to, drive bays with hot swap disks (sata and sas). i use one of the slots for full metal backups for doing periodic backup and rotation of backup drives. once again the issue is the integrity of cabling from u.2 connectors to the bays.
Not necessarily, probably even a 990 Pro or a Crucial T500 will perform better.
But to be honest, I am currently not up to date when it comes to Server SSDs.
I only know that consumer SSDs often get slower when they canāt use the pseudo cache SLC and that consumer drives donāt have PLP which accelerates sync writes. That is why the numbers from data sheets are mostly meaningless, unless you actually have that best case scenario workload.
build is based off of a pugetsystems threadripper pro eatx including nactua fans and cpu cooler.
as to power (plp) not the same thing, but all of my workstations are connected to upses and i have a whole house generator that comes on automatically. in my setup, single power supply per workstation is the most likely failure point and of course i dont have dual pdus running to two upses, running to two power supplies.
but like i said, a database is crash and burn, it only matters during each test.
PLP isnāt about power loss, PLP is about making use of volatile DRAM as cache to cache sync writes. That is why consumer drives used to be very slow in that regard compared to server SSDs.
Again, I am not up to date about SSDs, but it used to be the case that consumer drives are extremely slow doing sync writes, while PLP can cache a sync write thanks to, well PLP protecting the DRAM
it IS possible for consumer drives to cache, but not have PLP, and it is possible for PLP drives to not do caching the way you expect. assuming PLP = performance caching is also incorrect.
This is what DiskSpd/CrystalDiskMark SEQ Q1 write and similar benches evaluate. The limited set of use cases aside, Iāve had no issues getting 6-7 GB/s sync write with the app code I write talking to consumer PCIe 4.0 x4 drives.
Decent current gen TLC consumer 4.0 x4s also have ~30% of available space pSLC utilizable and ~1.5 GB/s cache folding rates. If enterprise and datacenter SSDs were any better Iād expect their manufacturers to point that out. They donāt.
Where running high power SSDs does seem to have an advantage is taking a few tens of Ī¼s off latency. Useful for reducing core stalls on IOPSy read dependency chains, unlikely to be significant to writes.
Canāt comment on 5.0 x4 as Iām waiting on 4 TB DRAMed drive availability at reasonable power and pricing. Pretty sure 10-12 GB/s sequential real world write wonāt be an issue but might run out of CPU cores or two channel DDR5 bandwidth before 14 GB/s.
Yeah, Pugetās component choices usually arenāt all that great, particularly Fractal, Define, XL, Noctua, and Asus. Itās also unclear to me what the AM5 AIOās doing in the parts list.
You are wrong, but I understand where you are coming from. To understand why, we have to look at how ānormalā aka none PLP drives work.
You write something in async. That something could end up in any kind of volatile cache. If you now loose power or the system crashes, that data is lost.
That is why we have sync writes.
Thanks to sync writes, you can safely write data without ever loosing data due to power loss or system halt. Only when data really is on NAND, the drive will send back an ack.
As you can see, the safety part is already solved by sync.
Now what does PLP do? How are they different?
A PLP drive basically lies about where the data is. It says āyeah sure bud, I have written it down to none volatile NANDā while it is only in valotile DRAM cache yet. That makes the drive very fast for sync writes. Now what happens if you loose power or the system halts? Thanks to PLP, the drive has enough energy to really write down the data from DRAM to NAND.
So yeah,
PLP is ONLY about performance, never about security. For security we have sync.
You could say that PLP is some kind of protection but that is only true because you introduced an āinsecureā cache to begin with. We had this discussion recently in the Proxmox forum. Someone came up with a great analogy. āSaying PLP is added security, is like saying putting a turbo on your car and add a software so that the turbo does not blow up, is added securityā
these drives have extra capacitors, (THE YELLOW THINGS) to write out data (from cache) if they loose power. that is all that PLP actually is. everything you talk about is not a requirement of PLP. even though it usually exists.
Exactly. That is why PLP drives are usually faster than none PLP drives. They have yellow capacitors that enable the drive to send the ack even though the data is not on NAND but only on DRAM.
Both a PLP and a none PLP drive can have DRAM cache.
The difference is that a none PLP can not send ack for a sync write before data is written into NAND (because in case of a outage, data would be lost), while a PLP drive can (because it does not care about an outage thanks to the yellow things )
That is why sync writes (and only sync writes, not async writes) are faster on PLP drives.
Same is true for RAID controllers with cache and a battery backup unit (BBU).
RAID controllers with BBU can cache sync writes.
RAID controllers without BBU canāt cache sync writes.*
*some RAID controllers offered to cache even without BBU as an unsafe option.
And while we are at it, the same logic applies to SLOG for ZFS. SLOG moves the ZIL to a separate device. ZIL is only used for sync. That is why an async write, never ever can be accelerated by a SLOG. Only sync writes. And since a SLOG only ācachesā (ZFS people will rightfully scream that ZIL is not a chache) sync writes, the drive for SLOG needs to be able to write sync writes fast. Guess which drives this applies to. Correct, PLP drives that can ālieā about sync.
Someone else already wrote all of what I try to explain to you down in way better English (sorry, I am not a native speaker).
Most consumer drives use DRAM write cache to achieve high performance, but do not have this power loss protection. That is why their sync write speeds are generally low.
the thing to remember with this build is this is not a enterprise database server. its an HEDT than can be used to test performance of certain systems prior to moving the test into the enterprise, which proceeds a move into real enterprise environment.
a good example is i am using qlik oracle to sql server replication. if i load a 2 or 3 tb CDR db into oracle and then want to see how replication is impacted by a multi-threaded app modifying (inserts, updates deletes) the oracle db i create vms for the oracle server, the qlik server and the sql server. performance eval in this instance becomes an indicator of how it will perform in production. in such cases, i load the db, run the test, restore the db, run the test, restore the db, run the tests. i dont need all the enterprise cutting edge hardware to do this test.
for me, off-the-shelf m.2 drives is probably good enough though i am also interested in u.2 drives which i am investigating.
as to the build copying pugetsystems, that seemed like a good model for an HEDT. its not a good build for an enterprise db server. i will use this workstation for occasional games, email, browsing, compiling code also.
im curious about the comment about the am5 aio. thats a liquid cpu cooler. if there are issues with that setup and i should consider something else for some reason, i am interested in why. as to noctua fans, i have replaced all the fans in my current desktop system with noctuas and been quite happy. im currently running a liquid cooled solution with noctua fans and a radiator and its performed quite well over the years.
this system is not an production enterprise server. i dont need a full blown rack mount environment running vmware, veeam and SANS for what i am doing.
Itās awkward to use asynchronous synonymously with write back. Similarly, synchronous isnāt normally a synonym for write through. While terminologyās sometimes overloaded this way around zfs, more commonly sync-ansyc refers to whether or not the app thread blocks until the IO call to the kernel completes.
CrystalDiskMark uses diskspd -Sh, so commonly reported drive benches include write through. Just Q32T8ed a 2 TB and a 4 TB 990 Pro with -Su instead. IOPS are the same while latencyās higher than with write through enabled. Minimum latency the 990s is 17 Ī¼s, not too far from the 10-11 Ī¼s specs of enterprise SSDs. So arguments thereās a large write through penalty to avoid are weakly connected to actual drive behavior.
Write through and power loss protection reduce the likelihood of data loss but donāt guarantee it. If write throughs are composed as transactions the guarantee is fairly strong but narrowly bounded. Write through doesnāt apply prior to transaction commit and apps that arenāt doing SQL usually arenāt transacted.
Itās also not guaranteed a drive (or array controller) will honor write through, either because supportās lacking or because of firmware bugs. Also, if power loss or BSOD injection were included in testing, PLP drivesā ability to mask app issues increasing data loss likelihood in those cases would probably be desirable.
Yeah, Iād hope a PLP drive honors write through and is well tested by its manufacturer. But there seems to be very little in the way of third party validation and nobodyās going to do fault injection in production. Also, the PLP niche is pretty narrow. Like if youāre worried about orderly shutdown presumably thereās a UPS, generator switch over, redundant supplies, and backups. PLP might matter when those fail but itās still a race condition against app code, network traffic, and whatever else might be building transactions.
Pragmatically speaking, a test workstationās fine if it goes down and the test database gets gakked since the databaseās getting regularly reset to initial test conditions anyway. Or you go, hey, cool, and switch over to testing database recovery. Which may be more valuable coverage.
Yes. And if the intentās to put the Kraken on the CPU why is there an XE360-TR5?
Yeah, theyāre fine if you want to pay 2-5x more for average to trailing performance. Was different 5-8 years ago but Noctuaās let a slow engineering cadence get them pretty far behind, so they donāt really have anything in their current product lineup thatās cost or performance competitive.
Thereās some narrow exceptions around the NF-A14x25 G2 and NH-D15 G2 HBC but the A14 G2ās basically tied with the P14 Pro thatās 60% the price and thereās like a dozen 360 AIOs that cost the same or less and stomp the D15 G2.
how about SilverStone Technology XE360-TR5 360mm All-in-One Liquid Cooler for AMD TR5 / SP6, SST-XE360-TR5 which supposedly supports socket str5?
the silverstone was on there originally in a spreadsheet, i thought i had removed it when i entered the kraken. thats why there were two coolers in the original list.
Agree. But outside of one twitter thread (the user deleted his account) with Phison E18 controllers on some Patriot and ADATA SSD that had this firmware bug, this almost never happens, because it is very dangerous.
