Just starting out learning about Linux and have been paying a little more attention to Linux stuff on L1T videos and keep searing about IOMMU groups and if they look good or not. Can someone give an ELI5 on what IOMMU groups are and what makes grouping good or not good?
Let’s say OS has some spicy sweet peppers he wants to ship his friend GPU. He needs his address to send his packages properly, but GPU lives in an apartment so he needs his apartment number as well, and wait what? He can’t just send a package to his friend GPU he sends it to the entire quadplex? They have a shared mailbox GPU tells him. OS tells him he’s fucking crazy and there’s no way this package is getting to him.
That’s IOMMU. An address map for your IO. Depending on how well they’re separated out you can do some really cool shit with it by reassiging hardware to VMs.
1 Like
Eh, not really.
IOMMU stands for Input/Output Memory Map Unit. It maps physical (electrical) IO to addresses in memory. Your OS does not need this to operate hardware because it’s got access to the hardware. This is used for one thing only: To attach physical devices to VMs. I’m sure OP’s seen some of the passthrough videos on L1L. Those should give a good base knowledge.
IOMMU isn’t about addressing devices so much as it is about allowing a VM to address the devices communicate directly with them.
IOMMU groups are due to PCIe ACS rules set up by the UEFI to allow PCIe P2P (peer to peer) communication. I know that’s not ELI5, but I’m going to break it down. The quick explanation is that IOMMU groups (normally) mirror ACS rules, unless you use an override patch.
Let’s use the example of a modern GPU with an onboard HDMI audio device. These two devices show up in the same ACS group because there’s a higher chance they’ll utilize P2P communication. Say you are a network device (in a different ACS group) that wants to talk to the GPU. Because you’re not in the same ACS group, you have to send your data all the way to the CPU, which will then dispatch the data to the GPU. Having devices in the same ACS group allow for more efficient communication, but means that they all have to be in the same VM (or all attached to the host) otherwise P2P communication between them can cause crashes.
Now, it’s important to understand the frequency of P2P communication. You’re more likely to get hit by a bus than your computer is to use P2P communication unless you have high performance enterprise equipment. That’s where the ACS override patch comes in. It lies to the OS about the ACS rules, allowing you to pass devices however you’d like to. This does come with a risk that if your devices use P2P communication, they could cause crashes, but again, the likelihood of that is very slim.
3 Likes