I like the 4x U.2/U.3 connections - assuming that SM bothers to populate them (my H11DSi is missing them, even though they’re on the spec-sheet!) and you can find the cables for sensible money - that’s my next search task!
Yeah, speaking of : my ROMED8-2T is actually missing the “Thunderbolt” header. Not that I would need it. It’s only a regular 2.54mm pin header footprint though, so if you’re feeling courageous and have a proper soldering station you could soak-off the solder in those holes with some copper braid and solder in a header… but still, that’s a 650 € board and they couldn’t be bothered to mount a 0.1 cent header ?
Or maybe there were other considerations. I don’t know squat about TB but I understand it’s not supposed to exist on AMD machines, because it’s an Intel patent or something. Maybe they left it out to avoid legal issues.
@Nefastor sorry to hear about the lack of hibernate. I can totally see how a server board could be shipped without it. When googling around I find both positive and negative indications for various generations of Supermicro Server boards, at least some seem to support it, but nothing about the recent ones such as the H12SSL. For me this is not an issue, I never really considered using hibernate on a workstation. But it is probably another of these Workstation vs. Server issues, that @wendell talks about, to put on the list (of things to worry about).
For the fan control issue, the reddit link that @jtredux posted mentions that it since some BIOS revision, fan control moved from BIOS to IPMI (or whatever it is properly called nowadays). So that would explain why it is not in BIOS - but since you mention updating BMC, I suppose you have searched there too and not found it?
I’m also considering the H12SSL. What I don’t really get is the different storage connection options. There is a matrix in the manual looking like this:
suggesting that the NT version has no way to connect SATA devices out of the box. On the other hand, the spec sheet mentions a
- SATA/NVMe Hybrid - 2 SlimSAS x8 (each: 8 SATA3 or 2 NVMe)
Does this mean that SATA drives could nevertheless be connected given the appropriate cable?
I have no use for a SAS controller anymore, since I moved away from mechanical disks on desktop, but I still have reason to use SATA SSDs for some time ahead.
The SATA/U.2 is driven directly from the EPYC CPU, isn’t it? I think you just have to pick the cable you plug into your SlimSAS socket and the rest is magic!
@oegat and @jtredux I was just looking at this. Here’s my understanding, which appears to be supported by the photos of each variant of the motherboard on Amazon (if that can be trusted).
-
The “i” variant is the most basic (cheapest) of the lot. It’s all SATA and does have regular SATA connectors. Maybe the “i” stands for “internal SATA”.
-
If the SAS3008 is present, so are its two mini-SAS connectors. But that is mutually-exclusive with single-drive SATA connectors. Can’t have both.
-
The “NT” variant has two PCIe x8 connectors (SFF8654 with 74 pins, not to be confused with the SFF8654 with 38 pins which only carries 4 lanes). Those are driven straight by the EPYC’s PCIe lanes that can work both as PCIe and SATA lanes. Those connectors are also mutually exclusive with the single-drive SATA connectors.
Speaking as an electrical engineer, here’s what I think is happening behind the scenes : those 8 single-drive SATA connectors are each fed one PCIe lane, the same that would also go to one of the SFF8654. That’s why on the “i” variant you have one SFF8654 only. They did not use PCIe lane switches you might find on other motherboards, probably for cost or signal-integrity reasons.
Bottom line : the only board variant that has the single-drive SATA connectors should be the “i” variant. Again, assuming that Supermicro implants all the connectors that make sense.
Regarding SFF8654 x8 : it’s one of the least common interfaces there is. The cable are a bit pricy. I found this one :
It’s over a hundred dollars. Interestingly, it appears to carry power for the SSD’s though.
I guess it’s probably not worth buying any cables until you’re 100% sure what you want to plug them into - there seem to be SFF-8654 8i options for both U.2 (SFF-8639) and ‘mini-SAS HD’ connectors that look like they might work with an IcyDock U.2 hot-swap-bay.
I think for SlimSAS 8i → 2x U.2 there is a cheaper but longer option from Broadcom : https://docs.broadcom.com/doc/12354774
Seems to be 05-60005-00 which seems to cost approx £55 inc VAT. Pricey but £25-30/drive seems to be on-par with Oculink.
1 Like
Indeed, performance has a price and carrying PCIe 4.0 lanes any distance is not an easy feat.
Regarding OCuLink : since I’ve never had this type of hardware before I tested the cable that comes with the ROMED8-2T and let me tell you, this connector does not inspire confidence. The socket looks very flimsy. The plug has a locking mechanism (hooks) and you need to press the plug body to release it. This requires little force but because of the flimsy socket the operation made me really nervous. This looks like a “plug once and never touch again” type of connector.
Note that Broadcom (and Intel?) cables may not be compatible with vanilla motherboard ports, and might instead be meant only work with specific their HBA’s/raid cards, which may use pinout hacks to allow them to interface with sata, sas, and nvme (tri-mode)
1 Like
Good point - I’ve not tested, nor do I recommend any of the links I’ve shared, just wanted to show what I’d found.
I think this is the way the world is going though - looking at recent server reviews, even the PCIe slots are getting cabled to the MB - and you seem to be able to by boards from SM like that too : H12DSG-Q-CPU6 | Motherboards | Super Micro Computer, Inc.
TBH, this DIY server lark has to be a niche-upon-a-niche-upon-a- niche!
I have a little supermicro board, and last I recall, trying to set fan speed did not work like I expected. I think I ended up needing to use a CLI utility to directly manage BMC stuff, but I’m not clear on that anymore.
1 Like
Thanks @jtredux @Nefastor . That’s (the option to attach SATA devices to the CPU-fed storage ports) also what I assumed before finding that manual page, then I became unsure based on that matrix. But it makes sense as the more exotic cables are not included.
Interesting! So that cable corresponds to almost all the price difference between a board with vs. without the LSI 3008 - which would use cables for ~1/10th of the price 
(not the way I would choose to enable SATA ports anyway, I see no point in having another onboard chip that generates heat).
1 Like
I also found this page of SM cables for SlimSAS 8i but even they can’t tell you whether they’re compatible with the H12SSL boards in their compatability checker.
https://store.supermicro.com/cable/sas-sata.html/slimline-sas.html
1 Like
I like those right-angle connectors !
It’s probably safe to assume that SM cables on SM’s website are compatible with SM boards… except of course if the direction they are angled ends-up going towards the CPU socket instead of the drives 
Fun fact, ECC doesn’t really do anything anymore. Aside from like, space calculation stuff. It’s good for that.
I don’t think that is a fact - do you have any evidence you’d like to share?
By my undertsanding of the physics of DRAM-cells, smaller ones should be a bit easier to flip a bit in, and my next machine will have approximately 10,000,000 times as much memory as the first computer I ever owned, so I’d expect the probability of a bit-flip to be significant.
Whether or not that flipped bit will ever cause you any pain is a whole other matter, but for a computer that has 100’s of GB of RAM and is never switched off, I prefer to have memory that can fix those bit-flips.
1 Like
Oh at hundreds of gigs of capacity no,. Never mind I’m some retrd monkey.
VRMs and cooling
Going back to the VRM et al. cooling question. So there is the issue that things like VRMs on server boards rely on a high ambient airflow through the case, that we might not get as easily in a workstation if keeping noise levels under control. As pointed out by @wendell and others.
Supermicro H12SSL surely has a diminutive VRM heatsink, as has the ASRock ROMED8. But how much of an issue will it be?
I notice, looking at Supermicro’s new TR Pro board M12SWA-TF, it does not have much of VRM cooling. Now, the product picture for the MB have the heatsink for most of the VRMs removed, but in the product pictures for their upcoming Workstation based on it we can see it’s approx size. Not at all in the size-order of the Asus board, though perhaps a bit taller. The CPU cooler that sits right behind it is probably a high fanspeed one (for being a workstation) though.
The M12SWA still has seemingly much less VRM cooling than the TR Pro boards from Gigabyte and Asus. However, it looks like it has a slightly larger heatsink than the H12SSL server board, and importantly the M12SWA seems to have many more phases. So judging from this, some extra spot-cooling for the VRMs may still be needed for the server-board-in-desktop-case scenario.
@Nefastor Are you planning to do some load-testing with your current CPU and compare FLIR images of the VRMs? Since you have what is needed, including the measurement equipment, I think it would be quite helpful for many here! I think the most useful thing would be if you could compare idle with full load, the latter on both a low and a high cTDP setting. Then we would get a feel for the dynamics. (maybe put an extra fan in the vicinity of the VRMs for a first load-test, and then test without it only if it seems safe ). I’d be very curious to see the results of that!
Back to the topic of cooler orientation
VRMs are an additional reason to try to get front-to-back airflow.
This looks great! Hmm, if you would put the first fan (counting in the airflow direction) on the side facing towards us in that picture, would it fit between the cooler and the VRM heatsink on your board? It would be interesting if the front fan could sit a bit lower, as to pull air over the VRMs too, most of which will flow underneath the CPU cooler. Then the second fan could sit a bit higher on the backside, drawing air through the full height of the cooler fin package. Something like that configuration is what we see in the Lenovo P620 (proprietary cooler, unfortunately) - I suspect that VRM cooling is at least part of the reasons for that positioning of the fans.
One drawback with current Noctua coolers is that any 90 degree off airflow solution would have the heatpipes partially hindering the air. However, I don’t know how much of a real issue it would be. I would really like to see more coolers for SP3/sWRX8 with the heatpipes on the longside of the CPU, then more heatpipes would fit, each chiplet pair and the IO die could have “their own” heatpipe, and it would all be more suitable for front-to-back airflow.
I only have a 7281 in my EPYC box, so 155W TDP IIRC so I don’t think it’s too hard on my VRM’s, but from what I can see, that tall VRM heatsink is for both CPU sockets, and I don’t actually have much aiflow over it at all. There’s an 80mm Noctua at the front - some 300mm away and blowing straight into the drives in the mid-bay. There are 2 more 80mm Noctuas pulling air out of the back of the case inline with the CPUs, but way above the top of that VRM heatsink.
My Noctua cooler is approx 2x as wide (directions fans mount) as it is deep - you really wouldn’t get much cooling air-flow with the fans at 90 degrees - the welds/folds for the fins are also in the front->back direction.
If you look at the video for that lovely SM Threadripper Pro workstation (I’m trying to get a price quote on one, but Google suggests $1200 for bare-bones case/psu/mboard) I don’t see any intake fans at the top of the front of the case that would blow across the VRM’s either.
I’m not claiming to know more than Wendell, but if the VRM’s are in the mid 90’s percent efficiency, a 280W CPU is only going to burn 14-28W in the VRM. I don’t think you need a hurricane to cool that, but you do need to get the heat out of your chassis.
1 Like