How to enable SEV-SNP on EPYC Milan for a Linux VM host?

Operating Systems & Open Source Linux
1

I have SME, SEV, SEV-ES and SR-IOV working fine for some VMs so I thought I’d go all the way and enable SEV-SNP and get remote attestation working too - mainly for learning but I like the idea of the security isolation.

I’ve enabled every SEV/SEV-SNP option I can find in BIOS, or anything related, either to “auto” or “manual” with various different settings, but what ever I try so far, the relevant bit in the SEV status MSR never gets turned on.

I’m using virtee/sevctl (GitHub - virtee/sevctl: Administrative utility for AMD SEV) to check the status:

# sevctl ok
[ PASS ] - AMD CPU
[ PASS ]   - Microcode support
[ PASS ]   - Secure Memory Encryption (SME)
[ PASS ]   - Secure Encrypted Virtualization (SEV)
[ PASS ]     - Encrypted State (SEV-ES)
[ FAIL ]     - Secure Nested Paging (SEV-SNP)
[ SKIP ]       - VM Permission Levels
[ SKIP ]         - Number of VMPLs
[ PASS ]     - Physical address bit reduction: 4
[ PASS ]     - C-bit location: 51
[ PASS ]     - Number of encrypted guests supported simultaneously: 253
[ PASS ]     - Minimum ASID value for SEV-enabled, SEV-ES disabled guest: 100
[ PASS ]     - SEV enabled in KVM: enabled
[ PASS ]     - SEV-ES enabled in KVM: enabled
[ PASS ]     - Reading /dev/sev: /dev/sev readable
[ PASS ]     - Writing /dev/sev: /dev/sev writable
[ PASS ]   - Page flush MSR: ENABLED
[ PASS ] - KVM supported: API version: 12
[ PASS ] - Memlock resource limit: Soft: 8388608 | Hard: 8388608
Error: One or more tests in sevctl-ok reported a failure

Determining the SNP status manually using the description of “3.2.1 System Configuration Register (SYSCFG)” from the AMD APM vol 2, yields the same result:

# echo "obase=2;$(rdmsr -d 0xc0010010)" | bc
10111101000000000000000000

→ bit 24 (SNPE / SecureNestedPagingEn) = 0

Kernel

Kernel version is 6.1.61-1-lts (Arch Linux).
kvm_amd is loaded with options sev, sev_es, npt, and nested enabled.
Only line in dmesg that matches “snp” (case-insensitive) is:

[    0.709397] AMD-Vi: Extended features (0x841f77e022094ace, 0x0): PPR X2APIC NX IA GA PC SNP

BIOS

BIOS settings (board: ASRock Rack ROMED8-2T), tried version 3.20 and 3.80 (latest) :

  • CPU
    • SVM Mode: enabled
  • Chipset
    • Above 4G Decoding: enabled
    • SR-IOV Support: enabled
  • CPU common options
    • SEV ASID count: 253 ASIDs
    • SEV-ES ASID space limit control: manual
    • SEV-ES ASID space limit: 100
    • SEV control: enable
    • SNP memory (RMP table) coverage: (tried two options)
      • auto
      • enable (help text = “ENTIRE system memory is covered”)
      • custom (which allows input of “Amount of memory to cover”, as hex * 1MiB, for which I entered “20000”, since 0x20000 * 1MiB = 128GiB = my RAM size).
    • SMEE: enabled
  • NBIO
    • SEV-SNP support: enable

BIOS photos in case I missed something (sorry for the wonky camera):

PXL_20231104_125220628
PXL_20231104_1252206281200×730 120 KB

PXL_20231104_125318755
PXL_20231104_1253187551200×763 145 KB

PXL_20231104_125333554
PXL_20231104_1253335541200×693 321 KB

PXL_20231104_125352360
PXL_20231104_1253523601200×787 255 KB

PXL_20231104_125442472
PXL_20231104_1254424721200×712 232 KB

PXL_20231104_125449749
PXL_20231104_1254497491200×728 236 KB

PXL_20231104_125516685
PXL_20231104_1255166851200×720 210 KB

Am I being silly and missing something?

2

I’m having the same issue on Rocky Linux. Have you made any progress?

1 Like
3

No progress here - tried maybe 20 different combinations of BIOS settings and the SEV-SNP flag was missing in every one.

It could be a kernel issue, I haven’t tried a different distro yet, but I don’t see any kernel options that might enable it or change behaviour - only CONFIG_KVM_AMD_SEV enables support from what I can tell by reading the kernel source code.

1 Like
4

Hello,

SEV-SNP has not yet been enabled for several upstream projects, so you will have to build an SNP kernel, QEMU and OVMF for your host. There is a repository that contains info on how to do this. Check out the repository AMDESE/AMDSEV on github.

If you’re using Ubuntu this script will automatize it for you, it always helps me save some time. Checkout the repository amd/sev-utils.

I would also recommend to have your bios updated to latest, since that way you’ll have the latest firmware and you will avoid many common SNP bugs.

Lastly if you’re into secure VMs, AMD is currently hosting a hackathon! This could be a cool opportunity to learn more about developing using secure VMs. Look up amd code jam on Devpost for more info!

3 Likes
5

upped your trust level so you could link if you want; I had been having a lot of fun with the ubuntu script a little bit ago but was curious how things had moved in milan/epyc.

I have the options for these features on threadripper in the bios, but then it seems not to work

1 Like
6

Hey Wendell, from what I can see from a Phoronix article where they listed all the feature flags I didn’t see the “SEV” flag. If you grep for in /proc/cpuinfo is it listed on your Threadripper 7000?

I’m wondering if it’s going to be a “Pro” CPU feature.

7

Nope
Bios options but nothing actually in proc CPU info

8

Hey, thanks for the replies. I was able to get the ubuntu script to work on my Milan machine - the snapshot I used was a bit broken (it tried to re-sign some .ko.zst modules), but there was nothing that couldn’t be fixed. I tried using a 6.6 Arch kernel as a guest, but had no luck (I thought there was guest support since 5.19?). The final result seems a bit fragile (e.g. kvm segfaults for slightly wrong parameters, etc.)

For those trying to get the ubuntu script working on a fresh 22.04 install, here are the packages required for the build environment (sorry if there’s some packages are subsumed in other groups - I don’t typically use Ubuntu): ninja-build, make, gcc, pkg-config, libglib2.0-dev, iasl, sparse, libpixman-1-dev, cmake, build-essential, python-is-python3, nasm, flex, bison, libssl-dev, libelf-dev, debhelper. Also, here’s a working qemu-kvm invocation (I installed the built qemu and ovmf to /usr/local, and I’m using a tap0 device):

  get_cbitpos()
  {
      modprobe cpuid
      EBX=$(dd if=/dev/cpu/0/cpuid ibs=16 count=32 skip=134217728 | tail -c 16 | od -An -t u4 -j 4 -N 4 | sed -re 's|^ *||')
      CBITPOS=$((EBX & 0x3f))
  }

  get_cbitpos
  MEM=4096

  sudo qemu-system-x86_64 -enable-kvm -cpu EPYC-Milan-v2 -nographic -m ${MEM}M,slots=5,maxmem=$((${MEM} + 8192))M \
  -drive if=pflash,format=raw,readonly=on,file=/usr/local/share/qemu/OVMF_CODE.fd,unit=0 -drive if=pflash,format=raw,file=./OVMF_VARS.fd,unit=1 \
  -drive format=raw,file=disk.img \
  -object memory-backend-memfd-private,id=ram1,size=${MEM}M,share=true \
  -object sev-snp-guest,id=sev0,cbitpos=${CBITPOS},reduced-phys-bits=1,discard=both \
  -machine q35,confidential-guest-support=sev0,kvm-type=protected,memory-backend=ram1,vmport=off \
  -device virtio-net,netdev=network0 -netdev tap,id=network0,ifname=tap0,script=no,downscript=no,vhost=on

Note that -cpu host doesn’t work for me. I needed to provided the cpu explicitly. Disk.img is a unified kernel/initramfs.

1 Like