Virtualization

SR-IOV

AMA virtualization support allows for isolated interaction of clients' operating systems with AMA compatible cards, in Linux. Specifically, isolation is achieved by relying on I/O Memory Management Unit (IOMMU) and Single Root I/O virtualization (SR-IOV) technologies. As such, AMA virtualization support provides an efficient interaction path between client applications and AMA compatible cards. This section describes Linux virtualization setup.

Linux Host

1. Ensure that host system is setup as per On Premises.

2. Install CPU checker and KVM packages on the host. For an Ubuntu host, do the following:

   sudo apt install -y cpu-checker qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virt-manager
   

3. Ensure that virtualization is enabled on the host:

   lscpu | grep Virtualization
   Virtualization:                  AMD-V
   $ kvm-ok
   INFO: /dev/kvm exists
   KVM acceleration can be used
   

4. If /etc/modules-load.d/ama_transcoder.conf doesn't exists, execute the following:

   echo ama_transcoder | sudo tee /etc/modules-load.d/ama_transcoder.conf
   

5. If /etc/modprobe.d/ama_transcoder.conf doesn't exists, execute the following:

   echo 'install ama_transcoder /sbin/modprobe --ignore-install ama_transcoder && /opt/amd/ama/ma35/bin/mamgmt numvfs --num 1 --device <BDF> && <REPEAT>' | sudo tee /etc/modprobe.d/ama_transcoder.conf
   echo 'remove ama_transcoder /sbin/modprobe -r --ignore-install ama_transcoder; /opt/amd/ama/ma35/bin/mamgmt numvfs --num 0 --device <BDF>; <REPEAT>' | sudo tee -a /etc/modprobe.d/ama_transcoder.conf
   

   , where BDF is PCIe Bus:Device.Function designation of the device(s) and REPEAT indicates that the /opt/amd/ama/ma35/bin/mamgmt commands are to be repeated for all devices that are meant to be passed to a VM.

   As an example, in a system with 2 cards:

   lspci -d 10ee:
   01:00.0 Multimedia controller: Xilinx Corporation Device 5070
   02:00.0 Multimedia controller: Xilinx Corporation Device 5070
   03:00.0 Multimedia controller: Xilinx Corporation Device 5070
   04:00.0 Multimedia controller: Xilinx Corporation Device 5070
   

   , in order to make one device, of each card, available to 2 distinct VMs, /etc/modprobe.d/ama_transcoder.conf will have the following content:

   install ama_transcoder /sbin/modprobe --ignore-install ama_transcoder && /opt/amd/ama/ma35/bin/mamgmt numvfs --num 1 --device 02:00.0 && /opt/amd/ama/ma35/bin/mamgmt numvfs --num 1 --device 04:00.0
   remove  ama_transcoder /sbin/modprobe --ignore-install ama_transcoder;   /opt/amd/ama/ma35/bin/mamgmt numvfs --num 0 --device 02:00.0;   /opt/amd/ama/ma35/bin/mamgmt numvfs --num 0 --device 04:00.0
   

6. Reboot the host. After reboot execute and confirm the following:

   lsmod | grep ama_transcoder
     ama_transcoder        831488  0
   

   If the above returns empty, then insert the module manually, by sudo modprobe ama_transcoder. Otherwise, confirm that driver is properly configured:

   cat /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs
   1
   
   $ lspci -d 10ee:
   01:00.0 Multimedia controller: Xilinx Corporation Device 5070
   01:00.1 Multimedia controller: Xilinx Corporation Device 5071
   ...
   

   The first command confirms that VM guest's Virtual Function (VF) that is associated with host's Physical Function (PF) is enabled and the second one confirms its creation. Note that this snippet assumes target device is 01.00.0.

7. Setup KVM on the host and confirm that it is running:

   sudo systemctl enable libvirtd
   sudo systemctl start libvirtd
   systemctl status libvirtd
   libvirtd.service - Virtualization daemon
      Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
      Active: active (running) since Wed 2023-06-07 09:19:44 PDT; 7h ago
      ...
   

8. Obtain Ubuntu 20.04 server image, from Ubuntu Focal Fossa.

9. Instantiate the VM guest, by allocating it at least 8 CPU cores and 12GB of RAM.

   Check virt-install version:

   virt-install --version
   

   If it is 4.0.0 or above, execute the following:

   sudo virt-install --name <VM name> \
   --ram 12288 --vcpu 8 --virt-type kvm --machine q35 \
   --disk path=<Path to qcow2 disk file>,size=128 \
   --os-type linux --os-variant ubuntu20.04 --network network:default --graphics none --console pty,target_type=serial --extra-args "console=ttyS0" \
   --location <Path to Ubuntu 20.04 ISO file>,kernel=casper/vmlinuz,initrd=casper/initrd \
   --force --debug --features  apic=on,ioapic.driver=qemu --iommu model=intel,driver.intremap=on,driver.eim=on,driver.caching_mode=on --memorybacking hugepages.page.size=2,hugepages.page.unit=M,nosharepages=1,locked=1 \
   --hostdev <xx>:00.1,driver.name=vfio
   

   , where VM name is the name of the VM image and Path to qcow2 disk file is the path to VM's disk storage, which will be created. Note that this file will be owned by libvirt-qemu user and kvm group. Also, Path to Ubuntu 20.04 ISO file is the path to the downloaded ISO image and xx is part of BDF designation as described above. Repeat this step for as many VMs that have been defined in /etc/modprobe.d/ama_transcoder.conf.

   Otherwise, first create the VM without ioapic.driver and --iommu parameters:

   sudo virt-install --name <VM name> \
   --ram 12288 --vcpu 8 --virt-type kvm --machine q35 \
   --disk path=<Path to qcow2 disk file>,size=128 \
   --os-type linux --os-variant ubuntu20.04 --network network:default --graphics none --console pty,target_type=serial --extra-args "console=ttyS0" \
   --location <Path to Ubuntu 20.04 ISO file>,kernel=casper/vmlinuz,initrd=casper/initrd \
   --force --debug --hostdev <xx>:00.1,driver.name=vfio
   

   and after VM creation add ioapic.driver and --iommu parameters, via virsh edit:

   virsh edit <VM name>
   <domain type='kvm'>
      ...
      <memoryBacking>
       <hugepages>
         <page size='2048' unit='KiB'/>
       </hugepages>
       <nosharepages/>
       <locked/>
     </memoryBacking>
      ...
      <features>
        <acpi/>
        <apic/>
        <vmport state='off'/>
        <ioapic driver='qemu'/>
      </features>
    ...
      </rng>
      <iommu model='intel'>
        <driver intremap='on' caching_mode='on'/>
      </iommu>
    </devices>
   </domain>
   

   During or after creation of a VM, it is helpful to install ssh server on the guest VM, in order to be able to connect to the instance:

   sudo apt-get install openssh-server
   

   (To find the IP address of the running VM, run sudo virsh net-dhcp-leases default or attach to the instance virsh --connect qemu:///system console <VM name> and issue the ip a command)

Linux VM Guest

On the running VM instance execute the following:

1. Set the huge table pages to 6GB, plus an extra 96 pages, i.e. 3072+96:

   sudo sh -c "echo 'vm.nr_hugepages=3168' >> /etc/sysctl.conf"
   

2. Note that if you are running 20.04 Server edition, you may need to install generic kernel 5.15, dkms and gcc-11:

sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
sudo apt install -y gcc-11 linux-image-5.15.0-60-generic dkms build-essential linux-headers-5.15.0-60-generic

3. Reboot the VM

4. Install the AMD AMA Video SDK and run through Tutorials and Examples

PCIe Passthrough

Unlike the procedure delineated in SR-IOV, where administrative tasks can only be performed through the host, PCIe passthrough allows for complete control of assigned devices, within the guest VM. That is to say, for all intent and purposes, the guest VM behaves as the host bare-metal. However, note that nested VMs are not supported. As an example, the following steps create a Ubuntu Focal Fossa VM guest, with a single assigned device:

1. Proceed by following Steps 1-7 of Chassis Setup (Enabling SR-IOV is optional.)

2. Ensure virtualization is enabled and relevant packages are installed, by following Steps 2 & 3 of Linux Host

3. Execute the following command: (KVM 4.0.0 or higher is assumed.)

   sudo virt-install --name vsdk_vm_passthrough --ram 12288 --vcpu 8 --virt-type kvm \
   --machine q35 --disk path=<Path to qcow2 disk file>,size=128 \
   --os-type linux --os-variant ubuntu20.04  --network network:default --graphics none \
   --console pty,target_type=serial \
   --features  apic=on,ioapic.driver=qemu --iommu model=intel,driver.intremap=on,driver.eim=on,driver.caching_mode=on \
   --memorybacking hugepages.page.size=2,hugepages.page.unit=M,nosharepages=1,locked=1 \
   --hostdev <xx>:00.0 --location <Path to Ubuntu 20.04 ISO file> \
   --extra-args "console=ttyS0" --force --debug
   

, where Path to Ubuntu 20.04 ISO file is the path to the downloaded ISO image. Customize the installation packages as needed. It is useful to include development packages and ssh server.

1. Set huge pages as per Step 1 of Linux VM Guest.

2. Proceed by following Steps 3 & 4 of Linux VM Guest.

Note

Ignore messages pertaining to sriov_drivers_autoprobe file not found .