Hello everyone, sharing some of the fun that has kept me busy in the past couple of months.
I have built a Proxmox server on top of an Asrock Epyc Micro-ATX board, the ROMED6U-2L2T.
Here is my part list:
- Asrock Rack ROMED6U-2L2T
- EPYC 7402P 24-Core Processor
- Noctua NH-U9 TR4-SP3 Air Cooler
- 128GB ECC Unbranded DRAM 2x32GB, 4x16GB, the ROMED6U has 6 dimm slots)
- Proxmox VE as main OS
- Radeon XT 5700 passed through my main Windows WM (for Gaming)
- Sapphire GPRO e9260 8 GB passed through my main OSX VM (for Working)
- 2x Fresco Logic FL1100 USB 3.0 Host Controllers, passed through either my main VMs, connected using two of the ROMED6U Slimline connectors to external PCIe 8x slots
- 1x Dual 2X NVME M.2 AHCI x8 adapter, bifurcated to 2x 4x NVME ssds
- 1x 4x NVME Pci Adapter
- OS running from the onboard NVME slot
a bunch of SSDs for storage of the non passtrhrough VMs
Case:
- 2x Thermaltake Core V21 Stacked ( I already had them from previous builds)
- 2x Noctua NF-A14 PWM in pull configuration
- 2x 200mm Thermaltake FANs in push configuration
Use case for the build: I work and game and run various projects from my home office, my daily drivers were two Core i5 6th and 7th gen running OSX plus Virtualbox and Windows exclusively for gaming.
The server will run these two workloads virtualized, plus a bunch of accessory VMs I use daily for y work/experiments
The Asrock MB is running Beta Bios 3.01, and BMC firmware 1.10.00!
The only issue I have encountered so far (other than spending way too much money for parts that are not readily available here in southern europe) is the fact that the BIOS sees only my 32GB DIMMS, while the 16GB DIMMS are not showing either in the BMC inventory or in the BIOS, but are seen and work in the OS
In order to get maximum perfomance and minimum latency out of the OSX and Windows VMs I have adapted some existing Proxmox init scripts to make sure the following happens when a VM that needs low latency is started:
- The VM emulator threads are moved to a set of cores that are shielded from normal OS operations, and pinned on a one to one ratio
- Each VM gets its own Video, Storage, USB, by using VFIO passthrough
- The Host interrupt handler affinity is changed to make sure only the pinned cores are processing the interrupts generate by the passed through hardware
- I try to keep the pinned cpu cores aggregated by CCX, in order to maximize l3 cache performance
- I change both the cpu governor and, for the windows VM, the scheduler used in order to make sure latency spikes occur as rarely as possible
The scripts I use to perform these config tasks automatically on VM start are available here:
In short, I augment the proxmox configuration file with the information that is needed to perform the optimizations, an example of which is:
#cpu_governor performance
#cpu_emulatorpin 20
#cpu_taskset 6,7,8,9,10,11,12,13,14,15,16,17
#assign_interrupts 6,7,8,9,10,11,12,13,14,15,16,17 "0000:8a:00.0"
#assign_interrupts 6,7,8,9,10,11,12,13,14,15,16,17 "0000:8a:00.1"
#assign_interrupts 6,7,8,9,10,11,12,13,14,15,16,17 "0000:01:00.0"
#assign_interrupts 6,7,8,9,10,11,12,13,14,15,16,17 "0000:81:00.0"
#cpu_chrt fifo 1
balloon: 0
bios: ovmf
boot: order=hostpci2
cores: 12
cpu: host,hidden=1,flags=+pcid;-ssbd;+ibpb;-virt-ssbd;+amd-no-ssb;+pdpe1gb;+aes
cpulimit: 12
efidisk0: local-lvm:vm-106-disk-0,size=4M
hookscript: local:snippets/taskset-hook.sh
hostpci0: 0000:8a:00.0,pcie=1,x-vga=1
hostpci1: 0000:81:00.0,pcie=1
hostpci2: 0000:01:00.0,pcie=1
hostpci3: 0000:8a:00.1,pcie=1
hugepages: 1024
kvm: 1
machine: pc-q35-5.2
memory: 16384
name: win10GamingTest
net0: virtio=D2:98:19:5F:EB:02,bridge=vmbr0,firewall=1
numa: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=8f5cff06-d005-4928-92e0-4e63d6be635b
Here is my processor topology, it looks like the 24 cores of the EPYC are distributed across 8CCXs, each with 3 cores:
I am using VFIO to pass through the needed hardware to the VMs, and the vendor-reset kernel module to be able to shut down/restart VMs with AMD cards passed through without having to reboot the host
Thermals are decent, the FANS are quiet (not silent) at what I consider rest (only my OSX vm running playing music through spotify):
ID | Name | Type | Reading | Units | Event
1 | 3VSB | Voltage | 3.36 | V | 'OK'
2 | 5VSB | Voltage | 5.01 | V | 'OK'
3 | VCPU | Voltage | 1.24 | V | 'OK'
4 | VSOC | Voltage | 0.86 | V | 'OK'
5 | VCCM ABC | Voltage | 1.21 | V | 'OK'
6 | VCCM EFG | Voltage | 1.21 | V | 'OK'
7 | VPPM ABC | Voltage | 2.56 | V | 'OK'
8 | VPPM EFG | Voltage | 2.56 | V | 'OK'
9 | LAN_1.88V | Voltage | 1.80 | V | 'OK'
10 | 1.8VSB | Voltage | 1.84 | V | 'OK'
11 | 1.8V | Voltage | 1.83 | V | 'OK'
12 | BAT | Voltage | 3.10 | V | 'OK'
13 | 3V | Voltage | 3.34 | V | 'OK'
14 | 5V | Voltage | 5.07 | V | 'OK'
15 | 12V | Voltage | 11.90 | V | 'OK'
16 | LAN_1.0V | Voltage | 0.99 | V | 'OK'
17 | PSU1 VIN | Voltage | N/A | V | N/A
18 | PSU2 VIN | Voltage | N/A | V | N/A
19 | PSU1 IOUT | Current | N/A | A | N/A
20 | PSU2 IOUT | Current | N/A | A | N/A
21 | MB_A Temp | Temperature | 36.00 | C | 'OK'
22 | MB_B Temp | Temperature | 38.00 | C | 'OK'
23 | Card Side Temp | Temperature | 44.00 | C | 'OK'
24 | X710 Temp | Temperature | 45.00 | C | 'OK'
25 | CPU Temp | Temperature | 50.00 | C | 'OK'
26 | CPU Power | Power Supply | 53.00 | W | 'OK'
27 | TR1 Temp | Temperature | N/A | C | N/A
28 | TEMP M_2_1 | Temperature | N/A | C | N/A
29 | TEMP M_2_2 | Temperature | N/A | C | N/A
30 | DDR4_A Temp | Temperature | 43.00 | C | 'OK'
31 | DDR4_C Temp | Temperature | 43.00 | C | 'OK'
32 | DDR4_D Temp | Temperature | 43.00 | C | 'OK'
33 | DDR4_E Temp | Temperature | 45.00 | C | 'OK'
34 | DDR4_G Temp | Temperature | 44.00 | C | 'OK'
35 | DDR4_H Temp | Temperature | 43.00 | C | 'OK'
36 | PSU1 Temp | Temperature | N/A | C | N/A
37 | PSU2 Temp | Temperature | N/A | C | N/A
38 | FAN1 | Fan | 1300.00 | RPM | 'OK'
39 | FAN2 | Fan | 1400.00 | RPM | 'OK'
40 | FAN3 | Fan | 800.00 | RPM | 'OK'
41 | FAN4 | Fan | 800.00 | RPM | 'OK'
42 | FAN5 | Fan | N/A | RPM | N/A
43 | FAN6 | Fan | 300.00 | RPM | 'OK'
44 | PSU1 PIN | Power Supply | N/A | W | N/A
45 | PSU2 PIN | Power Supply | N/A | W | N/A
46 | PSU1 POUT | Power Supply | N/A | W | N/A
47 | PSU2 POUT | Power Supply | N/A | W | N/A
48 | ChassisIntr | Physical Security | N/A | N/A | 'OK'
49 | CPU_PROCHOT | Processor | N/A | N/A | 'OK'
50 | CPU_THERMTRIP | Processor | N/A | N/A | 'OK'
51 | PSU1 Status | Power Supply | N/A | N/A | 'OK'
52 | PSU1 AC lost | Power Supply | N/A | N/A | N/A
53 | PSU2 Status | Power Supply | N/A | N/A | 'OK'
54 | PSU2 AC lost | Power Supply | N/A | N/A | N/A
Power draw is a mixed bag, around 150W with only the OSX VM running, (I blame thw AMD card for 60 Watts of it) and 170-190 with both VGA passthrough active, while not gaming
I am running headless on the Proxmox server, and control some of the core stats and the state of my main VMs using a Streamdeck (code still under development, will be available here when ready):
So far so good, perfomance is more than good, I get about 3GB/S in sequential read and write on the OSX VM using a 2TB MZ-V7S2T0 970 EVO Plus , and decent performance out of the 5700XT in windows (mainly Apex Legends)
Stability is excellent, as long as the 5700XT doesnāt crash while gaming, that earns me a full power cycle to get it started again, but that is expected.
I had to try a bunch of USB PCI cards and settled for the non SATA powered version of these cards: Inateck KTU3FR-4P 4 porte PCI Express card desktop usb Controller, Fresco FL1100 chipset, they are connected to a USB hub each on one port, and a Zoom audio interface on the other (Livetrak L-12 on the OSX side, Zoom U-24 on the windows side)
So far it has been an āinterstingā journey, definitely worth the time invested in knowledge acquired ā¦ did I really need a virtualized environment for my use case? Probably not, but building two separate systems in this period of components dearth would have been challenging anyway ā¦ and boring ā¦
Feel free to pitch in for details about specific parts of the configuration
Cheers
Mattia
