Virtualization

Overview

AMA virtualization support allows for isolated interaction of clients' operating systems with AMA compatible cards, in Linux. Specifically, isolation is achieved by relying on I/O Memory Management Unit (IOMMU) and Single Root I/O virtualization (SR-IOV) technologies. As such, AMA virtualization support provides an efficient interaction path between client applications and AMA compatible cards. This section describes Linux virtualization setup.

Linux KVM

The following 2 sections, Linux Host and Linux VM Guest, describe both the host and VM guest required configurations.

Linux Host

1. Ensure that host system is setup as per On Premises.

2. Install CPU checker and KVM packages on the host. For an Ubuntu host, do the following:

   sudo apt install -y cpu-checker qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virt-manager
   

3. Ensure that virtualization is enabled on the host:

   $ lscpu | grep Virtualization
   Virtualization:                  AMD-V
   $ kvm-ok
   INFO: /dev/kvm exists
   KVM acceleration can be used
   

4. If /etc/modules-load.d/ama_transcoder.conf doesn't exists, execute the following:

   $ echo ama_transcoder | sudo tee /etc/modules-load.d/ama_transcoder.conf
   

5. If /etc/modprobe.d/ama_transcoder.conf doesn't exists, execute the following:

$ echo 'install ama_transcoder /sbin/modprobe --ignore-install ama_transcoder && /opt/amd/ama/ma35/bin/mamgmt --force -d <DBDF> numvfs -v 1 && <REPEAT>' | sudo tee /etc/modprobe.d/ama_transcoder.conf
$ echo 'remove ama_transcoder /sbin/modprobe -r --ignore-install ama_transcoder; /opt/amd/ama/ma35/bin/mamgmt --force -d <DBDF> numvfs -v 0; <REPEAT>' | sudo tee -a /etc/modprobe.d/ama_transcoder.conf

, where DBDF is PCIe Domain:Bus:Device.Function designation of the device(s) and REPEAT indicates that the /opt/amd/ama/ma35/bin/mamgmt commands are to be repeated for all devices that are meant to be passed to a VM.

As an example, in a system with 2 cards:

lspci -d 10ee:
01:00.0 Multimedia controller: Xilinx Corporation Device 5070
02:00.0 Multimedia controller: Xilinx Corporation Device 5070
03:00.0 Multimedia controller: Xilinx Corporation Device 5070
04:00.0 Multimedia controller: Xilinx Corporation Device 5070

, in order to make one device, of each card, available to 2 distinct VMs, /etc/modprobe.d/ama_transcoder.conf will have the following content:

install ama_transcoder /sbin/modprobe --ignore-install ama_transcoder && /opt/amd/ama/ma35/bin/mamgmt --force -d 0000:02:00.0 numvfs -v 1 && /opt/amd/ama/ma35/bin/mamgmt --force -d 0000:04:00.0 numvfs -v 1
remove  ama_transcoder /sbin/modprobe --ignore-install ama_transcoder;   /opt/amd/ama/ma35/bin/mamgmt --force -d 0000:02:00.0 numvfs -v 0;   /opt/amd/ama/ma35/bin/mamgmt --force -d 0000:04:00.0 numvfs -v 0

1. Reboot the host. After reboot execute and confirm the following:

   $ lsmod | grep ama_transcoder
     ama_transcoder        831488  0
   

   If the above returns empty, then insert the module manually, by sudo modprobe ama_transcoder. Otherwise, confirm that driver is properly configured:

   $ cat /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs
   1
   
   $ lspci -d 10ee:
   01:00.0 Multimedia controller: Xilinx Corporation Device 5070
   01:00.1 Multimedia controller: Xilinx Corporation Device 5071
   ...
   

   The first command confirms that VM guest's Virtual Function (VF) that is associated with host's Physical Function (PF) is enabled and the second one confirms its creation. Note that this snippet assumes target device is 01.00.0.

2. Setup KVM on the host and confirm that it is running:

   $ sudo systemctl enable libvirtd
   $ sudo systemctl start libvirtd
   $ systemctl status libvirtd
   libvirtd.service - Virtualization daemon
      Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
      Active: active (running) since Wed 2023-06-07 09:19:44 PDT; 7h ago
      ...
   

3. Obtain Ubuntu 20.04 server image, from Ubuntu Focal Fossa.

4. Instantiate the VM guest, by allocating it at least 8 CPU cores and 12GB of RAM.

   Check virt-install version:

   virt-install --version
   

   If it is 4.0.0 or above, execute the following:

   sudo virt-install --name <VM name> \
   --ram 12288 --vcpu 8 --virt-type kvm --machine q35 \
   --disk path=<Path to qcow2 disk file>,size=128 \
   --os-type linux --os-variant ubuntu20.04 --network network:default --graphics none --console pty,target_type=serial --extra-args "console=ttyS0" \
   --location <Path to Ubuntu 20.04 ISO file>,kernel=casper/vmlinuz,initrd=casper/initrd \
   --force --debug --features  apic=on,ioapic.driver=qemu --iommu model=intel,driver.intremap=on,driver.eim=on,driver.caching_mode=on --memorybacking hugepages.page.size=2,hugepages.page.unit=M,nosharepages=1,locked=1 \
   --hostdev 0<xx>:00.1,driver.name=vfio
   

   , where VM name is the name of the VM image and Path to qcow2 disk file is the path to VM's disk storage, which will be created. Note that this file will be owned by libvirt-qemu user and kvm group. Also, Path to Ubuntu 20.04 ISO file is the path to the downloaded ISO image and xx is part of DBDF designation as described above. Repeat this step for as many VMs that have been defined in /etc/modprobe.d/ama_transcoder.conf.

   Otherwise, first create the VM without ioapic.driver and --iommu parameters:

   sudo virt-install --name <VM name> \
   --ram 12288 --vcpu 8 --virt-type kvm --machine q35 \
   --disk path=<Path to qcow2 disk file>,size=128 \
   --os-type linux --os-variant ubuntu20.04 --network network:default --graphics none --console pty,target_type=serial --extra-args "console=ttyS0" \
   --location <Path to Ubuntu 20.04 ISO file>,kernel=casper/vmlinuz,initrd=casper/initrd \
   --force --debug --hostdev 0<xx>:00.1,driver.name=vfio
   

   and after VM creation add ioapic.driver and --iommu parameters, via virsh edit:

     virsh edit <VM name>
     ...
     <memoryBacking>
      <hugepages>
        <page size='2048' unit='KiB'/>
      </hugepages>
      <nosharepages/>
      <locked/>
    </memoryBacking>
     ...
     <features>
       <acpi/>
       <apic/>
       <vmport state='off'/>
       <ioapic driver='qemu'/>
     </features>
   ...
     </rng>
     <iommu model='intel'>
       <driver intremap='on' caching_mode='on'/>
     </iommu>
   </devices>
   ...
   

   During or after creation of a VM, it is helpful to install ssh server on the guest VM, in order to be able to connect to the instance:

   sudo apt-get install openssh-server
   

   (To find the IP address of the running VM, run sudo virsh net-dhcp-leases default or attach to the instance virsh --connect qemu:///system console <VM name> and issue the ip a command)

Linux VM Guest

On the running VM instance execute the following:

1. Set the huge table pages to 6GB, plus an extra 96 pages, i.e. 3072+96:

   sudo sh -c "echo 'vm.nr_hugepages=3168' >> /etc/sysctl.conf"
   

2. Note that if you are running 20.04 Server edition, you may need to install generic kernel 5.15, dkms and gcc-11:

sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
sudo apt install -y gcc-11 linux-image-5.15.0-60-generic dkms build-essential linux-headers-5.15.0-60-generic

3. Reboot the VM

4. Install the AMD AMA Video SDK and run through Tutorials and Examples