My plan is to buy all components during Black Friday, other than the GPUs which will come from the second hand market, and the 9950X3D on release.
Budget ~ €1500-€2000 excluding GPUs and case (if required)
German market usually offers the best value in the EU.
Existing case: Phantekx Enthoo EVOLV ATX TG.
No peripherals required.
*Gaming important but second priority to financial trading backtesting, economic modelling, machine learning, LLMs.
An option to overclock or watercool in the future (if it offers a significant benefit or to reduce noise) would be great but I would prefer to not be dependent on it when I first build.
Ultra to high settings.
144Hz.
1440p.
All types of games.
Rust.
As I must put work before gaming this build is intended for 2 x 3090s for the VRAM rather than a single 4090. Ultra settings at high FPS would be great but not at the cost of VRAM.
Ideally the system will be capable of running a host and 3 VMs simultaneously with a third GPU (please recommend) and Looking Glass, though if I must I can move the basic VM to my laptop and reduce that to 2 VMs.
In terms of the PSU and case I am unsure if best practice is to plan for the possibility of upgrading to 2 x 4090s in the future, or save the money and build for now. If there is a reasonably priced case that will accommodate either configuration or that would just more easily accommodate 2 x 3090s and a third card, please let me know. Maybe something like the Lian Li O11 Dynamic EVO XL ATX Full Tower?
The motherboard, ECC RAM, and SSDs are based on what I have read here on the forums but if I have not built a PC for many years so please let me know if I should consider other options. I should probably add a good amount of storage for storing models and games.
Wendel used different SSDs for each VM in a build video but I am unsure how critical this is for performance or better for VFIO, or if the host running on a slower SSD will impact a VM using a faster SSD. First VFIO build.
Update: All components bar m.2 were priced higher on Black Friday in the EU market than at some point in the previous 6M/1Y so there is no urgency to pick them up before the 9950X3D has been released and some time allowed for any issues with it to be reported.
I’m gonna be the one that posts the general and not useful reply.
If your priority is work, picking up the 9950x3D on launch might not be the best way to go historically speaking.
instead of assuming that you’ll go mission critical with this machine from the day you get the CPU in socket I’m just curious to know what plans you have for checking that it will run as expected.
With two 3090 FE the third slot will be covered, unless you watercool or use a riser. The third card will also be limited to PCIe 4.0x2.
Why not 5200 or 5600 MT modules? The cost is not higher… Granted, with 4 modules you might not be able to run those speeds but they might hold up better in a future system or with a next gen CPU.
I’m not sure if it is announced yet if it will have 1 or 2 cache CCDs? And even if only one CCD has cache, the 9800X3D seems to have solved the frequency problem. So I would not worry too much about this.
But on launch day reviews should be there and you’ll be able to decide to get 9950x or 9950x3D.
If you don’t need 10 GbE down ASRock has full line ECC support and, generally speaking, is probably the best default for AM5, followed by MSI. Asus is, hmm, probably worse than Gigabyte at this point.
SN850X probably about as good as it gets for an all round 4.0 x4 drive but telling the same size two of the same drive apart can be a pain at times. Plan accordingly.
Dual GPUs make most M.2 sockets besides the 5.0 x4 reliant on motherboard armor, so expect thermal limitations with intensive IO, particularly once the dGPUs have pushed NVMe ambients up to 45-50+ °C. SN850Xes use 85 °C NAND, which helps some.
It’s not, so far as I know, and I figure probably won’t be until CES.
Personally I’m hesitant to call the clocks solved until they’re the same on vcache and non-vache SKUs. But, based on the 9800X3D, I think it’s reasonable to anticipate the 9900X3D and 9950X3D will be a good bit more frequency symmetric than the 7900X3D and 7950X3D.
Yeah, I built a 9900X six weeks after launch and that was still almost too early.
I can confirm I’m having a bunch of annoying issues with this board that get solved but often replaced again by other ones. I’m not sure what to blame though; linux kernel, amd drivers, Asus or AMD for the bios? Things like USB dropout (seems to be solved), amdgpu driver issues (a couple solved but currently it is not working any more), the Intel NIC eating its own firmware, sleep issues on linux, etc. But the Asus board is AFAIK the only one with 2 GPU slots, a third slot, and ECC support (taichi only has two slots).
Taichi does seem better for IOMMU; all NICs and the thunderbolt ports are isolated. On Asus (at least X670E) everything on the second chipset is clumped together (both NICs, wifi, TB4, …)
Yeah, me neither. OP BOM doesn’t require a third slot, though, and maybe the X870E Taichi Lite’s 5 GbE is good enough.
Also I’m unsure a three slot board is helpful. 3090 FEs are three slots and current gen x8/x8 boards have slots in positions { 2, 5, 7 } or { 2, 6, 7 } so a third card isn’t going to fit unless the second GPU’s risered off. With on board dGPUs, a { 2, 7 } like the Taichi is as good as it gets for allowing upper GPU intake and the O11D EVO XL has the below board space for the second GPU.
I’m not sure about mechanical compatibility of the O11D’s PCIe slot frame with a below board GPU but that’s a pretty easy mod if it’s an issue.
Maybe some of all of them? My sense of the issues threads here is they’re Asus dominated disproportionately to market share, though, which is the one controllable variable if you’re gonna do an AM5 Linux build.
It is useful, this is a significant investment for me, and it is worth thinking through.
I should clarify, my passion for financial markets and economics is something casual I would like to make professional. I have had some success and I want to explore it further, but I still have much to learn and I will not be placing trades or performing other financial operations in the very beginning using algorithms developed on the system - I just do not want to get to a point in the not too distant future where I find some limit because I made the wrong hardware choices for the build. I do consider this work but by that I am more broadly also referring to software development and productivity generally.
Are you suggesting that I wait until several months until after the CPU has been released, even beyond reviews, so that if there are any major issues I will be aware of them before I invest? This is something I have considered, and I am not ruling it out, it is just I have already been waiting years and if I can save money during a sales event I would like to. It is not difficult to imagine myself approaching Black Friday 2025 torn over buying or waiting for the next better hardware due to be announced at CES the following January.
Watercooling in the future maybe but a 3 GPU first VFIO build will keep me busy for awhile
My rough plan is to keep one 3090 pinned to the main VM, and use the other dynamically between that and the Windows VM as required, iGPU for host, and a third basic card for the third light VM (isolation, web browsing). My information is that PCIe 4.0 x2 should be adequate for the light VM @ 1440p?
If you have experience with them any recommendation on the riser cable?
I think I might have seen something recently that ‘DDR5 on die ECC is not real ECC’?
I have read a good rule of thumb for ML is system RAM twice the VRAM, and I have queried in the ML section if 96GB total on a system using some of that for the host and multiple VMs will still be enough, so I can avoid the issues using 4 DIMMs on AM5: < 96GB RAM on 48GB VRAM?
The latest rumour/leaks I have heard are v-cache only on one CCD. Are you talking about the core frequencies now maintained stable, or something else? Short of really bad reviews, what would it take to make you consider the 9950X over the X3D?
Don’t need it. ProArt x670e board suggestion based on reading how at least the B650 was so widely lauded as a great VFIO board, and Wendel’s x670e review where he mentions something like it is the board for linux support, but on watching it again today he says it is at least the Asus board for linux support. What are the main issues with it?
Not sure if I missed one or you were suggesting the ASRock Taichi, Steel Legend, or Phantom Gaming, as none of those have 3 x PCIe slots, which I assume I will need for the use case I describe: 3 GPUs for a host and 3 VMs. Very happy to be disabused of the notion I need 3 GPUs, and I can manage with just 2 x 3090s and the iGPU. The light VM could run on my laptop but I would prefer if could use my new build for everything.
If I can manage with just 2 slots, or I end up choosing to lose the light VM, which specific board are you recommending?
Is it the case that using the iGPU on a 9000 series for both the host and light VM is a non-runner? Either way, if buying again and 2 GPU PCIe slots were enough for your needs, is the Taichi better enough you would choose is over the ProArt?
Samsung 990 Pro will probably also be on sale, unless there is some other way to avoid the confusion? What are your thoughts on a separate SSD for the main VM and Windows VM, and then cheaping out on a more budget slower smaller SSD for the host and light VM to share, or would that slow all VMs?
Short of water cooling, which I would prefer not to do when I first build, are there any specific measures you would suggest to mitigate or safeguard from this? I plan to power limit the 3090s as apparently there is only a slight reduction in performance for the efficiency gained, somewhere around 275W:
Tech Yes City locked his 9950X cores to the same 5.2Ghz. Is it still worth doing something like this when splitting core across VMs with VFIO?
I can wait, I have an old system, it would be nice to have a new one sooner, but it does not make a huge difference whether I am using a new one January or June. I could keep planning, learning, and watching, and wait until then to buy all components. Not sure how much that may cost in terms of Black Friday discounts versus possible price reductions in the coming months, though there is always the peace of mind I would be building with the absolute most up do date information and after everything has been battle-tested. There may also be better AM5 DDR5 options then too. What would you do?
It’s real ECC and also enables refresh scrubbing. The distinction is DDR5 on-die ECC is EC2, and therefore provides SEC, rather than the SECDED of the EC4 characteristic of bus ECC. DDR5 RDIMMs can also support EC8 for chipkill mitigation.
Since DDR5 closes DDR4’s read CRC hole, non-ECC DDR5’s fault tolerance exceeds that of ECC DDR4 (which is all EC4) in some aspects. So, personally, I don’t see much of a use case for ECC DDR5 UDIUMMs.
There’s three basic options.
Spend for Threadripper, dual PSU support, and waterblocking three GPUs to solve the airflow and physical constraints.
Jam everything into a compromise desktop build still requiring electrical circuits and HVAC that can handle the power density.
Split the money and workloads across multiple builds.
At work we do the latter. Costs less for more cores, DDR, and dGPUs and is way less hassle (despite the larger machine count to admin) because we don’t have to rewire buildings and deal with custom loops. Only real disadvantage is the max workload size that fits on any one machine is smaller, but our workloads are mostly shard friendly. So the arrangement’s a net benefit as we can run more 1DPC 2R machines, need just a few 2DPC 2Rs, and can get way better NGFF thermals.
We run a bunch of a different mobos. Mostly various ASRock Steel Legends, Phantom Gaming flavors, and assorted Pros. They’re all fine within desktop platform constraints. I’m looking at another Zen 5 dual CCD build in maybe the February to May range but not planning to make any decisions until mid-January at the earliest. Right now I like B650 Steel for the below dGPU CPU 4.0 x4 M.2 but they’ve been stupid hard to get, so hoping for a reasonably priced B850 alternative.
Every launch is different. In general I figure about three months to get out of beta and for software support to catch up. But Windows 11’s at three years and doesn’t look like it’s ever getting out of beta. Seen enough demented scheduler behavior from 11 with the pretty symmetric 9900X I don’t have much confidence in its ability not to implode with real world workloads on a more asymmetric CPU.
PCIe 4x2 should be fine for general use I expect. I’m not sure if Looking Glass will work well (since that needs bandwidth too) but output to a monitor should be OK. ML applications probably too, insofar they don’t need much communication with CPU/main memory. I’m not too sure if the PCIe 4x2 slot on the proart is isolated in its own IOMMU group. I have that board but can’t test right this moment. Here are the groups btw:
IOMMU Group 0:
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 1:
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 2:
00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 3:
00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 4:
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 5:
00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 6:
00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 7:
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 8:
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 9:
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 10:
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
IOMMU Group 11:
00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
IOMMU Group 12:
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13:
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e0]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e1]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e2]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e3]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e4]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e5]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e6]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e7]
IOMMU Group 14:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] [10de:2504] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GA106 High Definition Audio Controller [10de:228e] (rev a1)
IOMMU Group 15:
02:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. KC3000/FURY Renegade NVMe SSD E18 [2646:5013] (rev 01)
IOMMU Group 16:
03:00.0 Ethernet controller [0200]: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:1003]
IOMMU Group 17:
04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port [1022:43f4] (rev 01)
IOMMU Group 18:
05:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
06:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. NV2 NVMe SSD SM2267XT (DRAM-less) [2646:5017] (rev 03)
IOMMU Group 19:
05:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
07:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port [1022:43f4] (rev 01)
08:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
08:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
08:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
08:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
08:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
08:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
08:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
08:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
09:00.0 Network controller [0280]: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter [14c3:0616]
0a:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)
0b:00.0 Ethernet controller [0200]: Aquantia Corp. AQtion AQC113CS NBase-T/IEEE 802.3an Ethernet Controller [Antigua 10G] [1d6a:94c0] (rev 03)
0d:00.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
0e:00.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
0e:01.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
0e:02.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
0e:03.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
0f:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 NHI [Maple Ridge 4C 2020] [8086:1137]
3b:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 USB Controller [Maple Ridge 4C 2020] [8086:1138]
68:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset USB 3.2 Controller [1022:43f7] (rev 01)
69:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller [1022:43f6] (rev 01)
IOMMU Group 20:
05:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
6a:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset USB 3.2 Controller [1022:43f7] (rev 01)
IOMMU Group 21:
05:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
6b:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller [1022:43f6] (rev 01)
IOMMU Group 22:
6c:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. NV2 NVMe SSD SM2267XT (DRAM-less) [2646:5017] (rev 03)
IOMMU Group 23:
6d:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c1)
IOMMU Group 24:
6d:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
IOMMU Group 25:
6d:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 19h PSP/CCP [1022:1649]
IOMMU Group 26:
6d:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b6]
IOMMU Group 27:
6d:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b7]
IOMMU Group 28:
6d:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]
IOMMU Group 29:
6e:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b8]
The taichi x670 does have everything in its own groups as does the proart B650 (search the forum – other users posted those). Drawback of the taichi is only 2 slots, drawback of the B650 is no PCIe 5 (and no usb4). There is the MSI carbon x670e but that one does not have ECC support (and I don’t know about IOMMU groups). No board has it all as far as I know. Pick your poison I guess?
I’m puzzled why you need 3 VMs with GPUs though. Do you need 4 independent systems with their own display output simultaneously? What for? I get needing a VM for windows, maybe one for testing/isolation, but on linux you can get very far with containers/distrobox and similar tools with less overhead and more flexibility.
Yeah, I think not in AM5. TR5 should tick all the feature boxes, within ATX form factor limits, but at a cost. Not having luck finding IOMMU documentation for the boards, though.
Not clear to me either. Game on one while the other two are chugging an LLM, maybe, but that feels oversubscribed for dual channel DDR. With a third dGPU I’d watch the chipset uplink allocation too.
The reason I think I may need 3 GPUs is that the dual 3090s will normally be used by VM1 but when I want to play games one of them will be used by Windows VM2. VM3 is for placing manual trades and other financial transactions and I want this to be completely isolated from everything else - this VM is not for ML, it just requires basic web access, a USB port, and to display at 1440p. VM1 and VM2 should always be running, so that whether I am gaming on Windows VM2, or doing anything else on VM1, I can at a glance to my second monitor or with a key combination switch to VM3 for monitioring or taking action.
Host, Proxmox (iGPU)
VM1, Linux, Main, ML, every day use (3090A + dynamic 3090B), always on
VM2, Windows, Games, (dynamic 3090B), sometimes on
VM3, Linux, Trading, (third GPU?), always on
For trading preference for a VM that I could in future use Proxmox to delta/incrementally backup and sync to another device but if it really is not possible I am open to something like containers/distrobox that will provide a GUI VM like experience on the system. Before we rule out VM3 (Running simultaneously with one or both of VM1 and VM2) can I take it that there would still be problems in an AM5 desktop with all of these approaches that I do not have experience with but came up when researching:
2 x 3090s with a powerlimit of 275W and 1 x light card like a 1650 (TDP 75W) or a 1030 (TDP 30W)
With 2 x 3090 only:
CPU Software rendering: VirtIO, spice/QXL
Virtual iGPU: passthrough, vGPU
Remote desktop/VNC to headless VM3 from host or VM1
I am interested in water cooling but I already see this as a massive project and WC always seemed like an undertaking I would broach in the future after my first VFIO build, unless some kind of AIO or easier solution that does not involve melting and bending pipes etc. is reasonable for a noob and would actually make this project more practical and the benefits just make it the way to go here from the beginning.
Also, I just added FE 3090s to the BOM as a placeholder but I will buy used (in the EU or eating import costs from the US) what the market determines unless I should only consider something like Dell 3090s as they are slimmer - without negating the value of buying used 3090s in the first place.
I would like to stick with AM5 desktop but if the compromises are too great falling back to splitting the workload by using a MiniPC/laptop/SBC instead of (or for) the trading VM/tasks.
I was not aware standard DDR5 was equivalent to ECC DDR4. Are you recommending instead of either the Kingston I suggested from the QVL/AM5 DDR5 thread or the Premier Server you mentioned, I just go with the best value highest rated RAM from the QVL/that others have confirmed works at high speeds (if 128GB rather than 96GB is required)? I understand 6000 is a good target for AM5 even if rated higher.
If you were building and buying for yourself and wanted to get the best value, and you planned to let 3 months pass after the 9950X3D release, so potentially May, would you pick up the board, RAM, PSU, and 3 or 4 SSDs during Black Friday or wait and buy it altogether in May?
One monitor has just DP the other has HDMI and DP. I was thinking VM3 (Trading) on PCIe 4x2, everything else on the VM1 (Main, ML, every day) with both 3090s running at x8. Thanks for sharing the groups, I guess when I am more clear on hardware requirements I can start to work out what groups are required.
Happy to save money on something like a B650 unless PCIe 4.0 versus PCI 5.0 x8 would bottleneck the 3090s, or there would be some other issue with a third card (if required) or some issue with multiple m.2.
When I am more clear on whether or not I really need a third GPU this should make is easier to decide on a board.
3090 is pcie 4.0, so they work at pcie 4.0 regardless of the board. It’s rather for 5000 series cards it might matter, assuming those are pcie 5.0
Third VM probably doesn’t need a GPU, it’s just simple display? A GPU will be smoother sure but a virtual GPU only lags barely noticeable on my AM5 system for a Linux desktop.
There is some guide on how to pass the iGPU to a VM on proxmox, leaving proxmox headless. It is a bit finicky but probably not much more than squeezing the third gpu using risers & co.
Which is fair, because it’s not. Different failure modes get different levels of mitigation. What I’d suggest is, unless there’s reason to be particularly worried about certain types of faults, it gets hard to assess differences in net overall potential problem badness.
Don’t really have anything to add to what I’ve already said. Did just pick up 12 TB of SN850Xes as pricing’s good right now, but those are for other things.
Yes but I thought there would be a performance issue using 2 x 3090 at x8 in PCIe 4.0 versus 5.0. I was not able to find anywhere that mentioned the actual maximum transfer rate in GB/s for a 3090 without directly referencing it was intended for PCIe 4.0 x16 or should work fine at PCIe 4.0 x8/PCIe 3.0 x16. TechPowerUp demonstrates where there is a difference it is only negligible NVIDIA GeForce RTX 3080 PCI-Express Scaling - Performance Summary | TechPowerUp and the often cited A Full Hardware Guide to Deep Learning — Tim Dettmers says not to be too concerned about PCIe. I will also power limit during the most demanding tasks so it matters even less.
Great news on Proxmox. Since you posted I have read the norm is to access Proxmox when necessary via the WebUI, or use the iGPU as required, or just during setup. If I am clear on the right board I will buy it and the other components Friday.
If we combine together what you have both said about Asrock versus Asus, DDR5, motherboards, adequate PCIe speed, space and HVAC, and 2 GPUs rather than 3, could we say that you both agree for the best VFIO experience and my use case the board to go with is the still the Asrock Taichi x670e, or as PCIe 5.0 is not required, would something like the Asrock b650e Taichi Lite (1 PCIe 5.0 x16, 1 PCIe 4.0 x16) or one of the other b650e boards offer all of the other advantages at reduced cost? I can complement this with as many m.2 drives as supported, and the best value fast 2 x 48GB (non ECC) DDR5, though the QVL for the ASRock > B650E Taichi only allows selecting memory for 7000 and 8000 processors, even though the very top of the same page mentions 9000 support. There is not much in the price difference right now between the b650e Lite and x670e Carrara.
PEG + chipset 4.0 x4. Not an especially uncommon slot arrangement, so nothing particularly special about the B650E Taichis. Alternatives are PEG + CPU 4.0 x4 or 3.0 x4, depending whether it’s more useful to have a second CPU NVMe or the second GPU on CPU lanes. B650 Live Mixer, B650 Carbon, X670E Tomahawk, and X670E Gaming Plus are some others to look at, though MSI doesn’t do block diagrams.
It’s workload dependent but, yeah, I don’t know of any readily available data. Nvidia has proxy performance counters but they don’t seem to get used much.
Cheers, I was not aware of motherboard block diagrams, and I will look at those available for all boards mentioned, but looking at the one for the Taichi x670e helps my understanding. I see that it would nicely accommodate the 2 GPUs with 2 x PEG 5.0 x8 slots directly connected to the CPU which I should prioritise, while allowing ample bandwidth for 3 x m.2s (~6GB/s) with PCIe 5.0 and 4.0 x4 slots, leaving 1 free to keep the SATA ports available for mechanical storage.
I checked that block diagram first because it has been mentioned:
I can check if PC Part Picker allows me to add 2 x 3090s to every board that has been mentioned though I am unsure how reliable that is as it is telling me the slots required for one of the 3090s in the updated OP list with the Taichi x670 will be covered. I wonder if the intake benefit, the good IOMMU groups, and that you and other reports I read today said Asrock are generally good for VFIO, should supersede considering other x670es or b650s?
Edit: In fact many unless I am missing something most of the boards do not meet the minimum requirement for dual 3090s, having at least 2 x PCIe 5.0 x4/PCIe 4.0 x8, so they can be ruled out immediately, e.g B650 Live Mixer, MSI Carbon B650, X670E Tomahawk, X670E Gaming Plus.
Meets requirements:
X670 Taichi (2 x PCIe 5.0 x8/x8)
X670E Steel Legend (1 x PCIe 5.0 x16, 1 x PCIe 3.0 x16)
The Steel Legend’s 2nd PCIe slot is wired to x4 lanes, since it goes to the chipset.
For what you want (dual GPUs running out of the CPU at x8/x8), look for motherboards that have the 2 top slots with reinforcement (like the Taichi), that’s a good way to know if it supports both slots out of the CPU lanes or not.
FWIW, the only B650 motherboard that supports said config is the ProArt.
There are 10 other x670(e) models that support your config, some of those are:
x670e Biostar Valkyrie
x670e Asrock Taichi
x670e Asus ProArt
x670e Asrock Taichi Carrara
x670e Asus ROG Crosshair Hero
Judging by some of what’s in this thread, probably not very.
PEG + CPU 3.0 x4.
Maybe. 20 5.0 lanes + 4 4.0s is ~86 GB/s potential and I’m seeing like 67 GB/s real world with DDR5-6000. Also I don’t know of any data on how effectively one or more NVMes, one or more SATA drives, NICs, and USB can share the chipset uplink. 6 GB/s unidirectional with NVMe’s fairly close to saturating the uplink lanes in that direction.
There’s reasons multiple folks have suggested Threadripper here.
In terms of RAM for the Taichi X670E, and if 6000 is the sweet spot, should the only 6000 rated option the Corsair Hynix M-Die be used, or would RAM rated higher or with EXPO rather than XMP mean that RAM should perform better at 6000?
I assume when there are 96GB kits rated at or above 6000 it would be a mistake to hope to bump the G.Skill rated at 5600 up to it with either XMP or custom timings.
How important is the speed of the drive the host is on? Is it the case that if you intend to run VMs on ~6 GB/s m.2s the host should either be on a drive of the same speed/partitioned on a VM drive, or once the host starts the VM on a faster m.2 the speed of the host drive does not matter?
Even if the speed of the drive the host is on does not matter, it may make sense in terms of conserving lanes and bandwidth to just partition one 6 GB/s m.2 for the host, and both linux VMs, and just use one other for Windows.
Thanks, that helps.
From what I am reading so far elsewhere on the forum comparing X670Es the Taichi is still the board to beat but I have more to read and I want to go through boards on PC Part Picker checking for 2 reinforced slots.
Cheers, lanes are making more sense. For optimal use of resources I need to use the number of lanes and components within 67 GB/s.
Does what you have said about the chipset uplink mean that a device on one lane or group of lanes for something like a m.2 is not confined and if it saturates it affects others? If you had not mentioned that from what you said before I would have thought I could just check everything connected like the GPUs or m.2s have enough bandwidth within the lanes they are connected to and the total is less than 67 GB/s.
Sure, but for this build I would prefer to modify my expectations or offload some of the work like the trading VM to a MiniPC or laptop as MaxChaos suggested in the other thread.
