Using GStreamer

Introduction

GStreamer is a pipeline-based multimedia framework that links together a wide variety of media processing systems to complete complex workflows. The pipeline design serves as a base to create many types of multimedia applications such as video editors, transcoders, streaming media broadcasters and media players. The AMD AMA Video SDK includes an enhanced version of GStreamer which can communicate with the hardware accelerated transcode pipeline in AMD devices.

It is not within the scope of this document to provide an exhaustive guide on the usage of GStreamer. Various resources can be found online, for example:

Overview of the GStreamer Plugins

The AMD AMA Video SDK provides the following GStreamer plugins for building hardware-accelerated video pipelines using AMA compatible devices:

  • ama_av1dec av1 decoder

  • ama_av1enc av1 encoder

  • ama_h264dec h264 decoder

  • ama_h264enc h264 encoder

  • ama_h265dec h264 decoder

  • ama_h264enc h265 encoder

  • ama_vp9dec VP9 decoder

  • ama_scaler hardware accelerated scaler and cropper

  • ama_download data downloader, from device to host

  • ama_upload data uploader, from host to device

  • ama_compositor feature rich compositor

  • ama_jpegdec JPEG decoder

  • ama_jpegenc JPEG encoder

  • ama_ljpegenc Lossless JPEG encoder

  • ama_videoconvert accelerated pixel converter

The following sections describe the options used with GStreamer to configure and use the various hardware accelerators available on AMA compatible devices.


Decoder Plugin

The ama_av1dec, ama_h264dec, ama_h265dec, and ama_vp9dec plugins provide support for hardware-accelerated decoding using AMA compatible devices, for AV1, AVC, HEVC and VP9.

SDK also supports JPEG decoding through ama_jpegdec plugin.

Decoder Inputs and Outputs

Video Decoders:

  • Input: 8 or 10 bits AV1, AVC, HEVC and VP9 encoded stream, in OBU format for AV1, byte-stream format for AVC or HEVC, or VP9 stream format for VP9.

  • Output: One of NV12_4L4, NV12, I420, NV12_10LE32, P010_10E, I420_10LE, RGBP, or NV12_10LE_4L4 formats

JPEG Decoder:

  • Input: Standard JPEG file

  • Output: One of NV12_4L4, NV12,I420, or RGBP

Decoder Parameters

Property Name

Description
device
Numeric ID of an XMA device
Type: Integer
Default: -1 (auto)
Range -1 to 2147483647
low-latency
Whether to enable low latency decoding
Type: Boolean
Default: false
latency-logging
Log latency info to syslog
Type: Boolean
Default: false
allow-downscaling
Downscale through decoder
Allow the decoder to downscale the output if negotiated with downstream. When FALSE, the decoder always outputs video in its native resolution. Scaling RGBP format isn't supported. | Type: Boolean
Default: false

Decoder Example Pipelines

Refer to the Decoder Pipeline example for an illustration of how to use this plugin.

Encoder Plugin

The ama_av1enc, ama_h264enc and ama_h265enc plugins provide support for hardware-accelerated encoding using AMA compatible devices, for AV1, AVC and HEVC.

Encoder Inputs and Outputs

  • Input: NV12_4L4, I420, NV12_10LE32, P010_10LE, I420_10LE, or NV12_10LE_4L4

  • Output: One of AV1, AVC or HEVC 8-bit/10-bit encoded stream, in OBU format for AV1 and byte-stream for AVC and HEVC

Encoder Parameters

Property Name

Description
b-frames
Number of B-frames between two consecutive P-frames
Type: Integer
Range: -1 - 7, for AV1 Type 1, with default 7 for FPS greater than 30, and 3 otherwise.
Range: -1 - 3, for AVC, HEVC, and AV1 Type 2, with default 3.
Default: -1 (auto)
bitrate
Fixed output bitrate in bits per second
Type: Integer
Range: 0 - 4294967295
Default: 5000000
crf
Constant Rate Factor value
Type: Integer
Range -1 - 63
Default: -1
device
Numeric ID of an XMA device
Type: Integer
Range: -1 - 2147483647
Default: -1 (auto)
dynamic-gop
Enable Dynamic B Frame Insertion
Type: Integer
Range: -1 - 1
Default: -1 (auto)
device-type
AV1 device type to use
Type: Enum
(0): any
(1): Type 1
(2): Type 2
Default: 1
forced-idr
Encode all intra frames as IDR
Type: Boolean
Default: True
gop-length
Maximum distance between two consecutive I frames
Type: Integer
Range: -1 - 2147483647
Default: -1 (auto)
idr-interval
The periodicity of IDR frames
Type: Integer
Range: 0 - 4294967295
Default: 4294967295
lookahead-depth
Number of frames in the future of the currently encoded frame that the encoder will analyze and take into account
Type: Integer
Range: 0 - 47+Maximum number of B frames b-frames
Default: -1 (auto)
max-bitrate
Use to limit encoding bitrate if you have not specified bitrate parameter
Type: Integer
Range: 0 - 35000000000
Default: -1 (auto)
max-qp
Maximum QP value allowed for the rate control
Type: Integer
Range: 0 - 51, for AVC and HEVC
Range: 0 - 255, for AV1
Default: 51, for AVC and HEVC
Default: 255, for AV1
min-qp
Minimum QP value allowed for the rate control
Type: Integer
Range: 0 - 51, for AVC and HEVC
Range: 0 - 255, for AV1
Default: 0
qp
Fixed quantization value. When constant quality is enabled, bitrate must be set to 0
Type: Integer
Range: -1 - 255
Default: -1 (disable)
qp-b-offset
FQuantization offset value - P/B
Type: Integer
Range: -51 - 51
Auto selection mode is -1.
Default: 0
qp-i-offset
FQuantization offset value - I
Type: Integer
Range: -51 - 51
Auto selection mode is -1.
Default: 0
qp-mode
QP control mode.
Type: Enum
(0): auto - Auto
(1): relative-load - Relative load (spatial AQ + temporal AQ)
(2): uniform - Uniform (AQ off)
rate-control
Rate control mode to use for encoding
Type: Enum
(-1): auto - Auto
(0): cqp - Constant QP
(1): cbr - Constant bitrate
(2): vbr - Variable bitrate
(3): cvbr - Constrained variable bitrate
(4): cabr - DEPRECATED Content adaptive bitrate
(5): crf - RESERVED
slice
Which slice of the XMA device to use
Type: Integer
Range: -1 - 1
Default: -1
spatial-aq
Enable spatial AQ
Type: Boolean
Default: True
spatial-aq-gain
Spatial AQ gain scale factor
Type: Integer
Range: 0 -255
Default: 255
temporall-aq
Enable temporal AQ
Type: Boolean
Default: True
temporal-aq-gain
Temporal AQ gain scale factor
Type: Integer
Range: 0 -255
Default: 255
Default: 1
tier
Tier
Type: Boolean
(-1): auto - Auto
(0): main - Main
(1): high - High
preset
Encoder VQ-Density preset
Valid values: medium or slow. (default is medium)
(medium) Full density with medium VQ - default
(fast) Allows for increase in total throughput from 4kp60 to 4kp75 , when 2 or more channels are used. Does not apply to AV1 device-type 1 encoding. Not applicable under ULL setting. Note lower VQ than medium preset is to be expected.
(slow) High VQ at lower density. (For AV1 Type 1, improvements are observed in 10 bit contents only.)
Consult the Tuning Video Quality section for more details on how to use this option.
Note that this option has no effects on AV1 device-type 1 encoding.
tune-metrics
Tune VQ for objective metrics
Type: Enum
(1): vq - Best visual quality
(2): psnr - Best PSNR
(3): ssim - Best SSIM
(4): vmaf - Best VMA
encoding-params-file
Dynamic Configuration File
Path to dynamic file location (default is empty)
Consult the Dynamic Encoder Parameters section for more details on how to use this option.
latency-ms
Encoder latency in milliseconds that the encoder will analyze and take into account
Type: Integer
Range: -1 - 60000
Default: -1 (auto)
no-bll
No low latency B Frames
Type: Integer
Range: -1 - 1
Default: -1 (auto)

Encoder Example Pipelines

Refer to the Encoder Pipeline example for an illustration of how to use this plugin.

Scaler Plugin

The ama_scaler plugin provides support for hardware-accelerated resizing.

Scaler Inputs and Outputs

  • Input: NV12_4L4, NV12, I420, NV12_10LE32, P010_10LE, I420_10LE, I420_10BE, RGBP, ARGB, ABGR, RGBA, BGRA, RGB, BGR, and NV12_10LE_4L4

  • Output: NV12_4L4, NV12, I420, NV12_10LE32, P010_10LE, I420_10LE, I420_10BE, RGBP, ARGB, ABGR, RGBA, BGRA, RGB, BGR, and NV12_10LE_4L4

Scaler Parameters

Sink Pad

Property Name

Description
device
Numeric ID of an XMA device
Type: Integer
Range: -1 - 2147483647
Default: -1 (auto)

The output width, height and frame rate are set through the succeeding cap filter.

Pad Property

Description

width
Output width
Type: Integer
Range: 144 - 7580

height
Output height
Type: Integer
Range: 144 - 7580

framerate
Output frame rate
Type: Enum
Range: -1 - 1
(-1): auto - Full rate
(0): full - Full rate
(1): half - Half rate

Scaler Example Pipelines

Refer to the Transcode with Multiple-Resolution Outputs example for an illustration of how to use this plugin.

2D Engines

This section describes different hardware accelerated 2D processing engines that are available in AMA SDK. The following plugins may take core-id and device as configuration parameters to specify the target core and device, respectively.

Inputs and Outputs

  • Input: NV12_4L4, NV12, I420, NV12_10LE32, P010_10LE, I420_10LE, I420_10BE, RGBP, ARGB, ABGR, RGBA, BGRA, RGB, BGR, and NV12_10LE_4L4

  • Output: NV12_4L4, NV12, I420, NV12_10LE32, P010_10LE, I420_10LE, I420_10BE, RGBP, ARGB, ABGR, RGBA, BGRA, RGB, BGR, and NV12_10LE_4L4

2D Plugins

2D Accelerator Engines

Engine Name

Description

Options
ama_videocrop
Video crop
Takes coordinates.
bottom Pixels to crop at the bottom (from 0 to 2147483647) (default 0)
top Pixels to crop at the top (from 0 to 2147483647) (default 0)
left Pixels to crop on the left (from 0 to 2147483647) (default 0)
right Pixels to crop on the right (from 0 to 2147483647) (default 0)
ama_videopad
Video pad
Takes size, coordinates, and pad color.
bottom Pixels to add at the bottom (from 0 to 2147483647) (default 0)
top Pixels to add at the top (from 0 to 2147483647) (default 0)
left Pixels to add on the left (from 0 to 2147483647) (default 0)
right Pixels to add on the right (from 0 to 2147483647) (default 0)
color Padded area color in ARGB format (e.g. 0xFF000000 for black)
ama_drawbox
Draw box
Takes size, coordinates, and pad color.
height Height of the box (from 0 to 2147483647) (default 10)
width Width of the box (from 0 to 2147483647) (default 10)
top Number of pixels from the top of the video to the top of the box (from 0 to 2147483647) (default 0)
left Number of pixels from the left side of the video to the left side of the box (from 0 to 2147483647) (default 0)
thickness Box line thickness (from 2 to 16) (default 2)
color Box line color in ARGB format (e.g. 0xFF000000 for black)
ama_rotate
Rotate frame
Takes an angle.
angle one of 0, 90, 180 or 270 (default 0)
ama_videoconvert
Color space conversion ans subsampling
Converts between RGB and YUV.
Converts from I422 to I420

ama_overlay
Video overlay
Overlays 1 picture over another
alpha-blend Enable alpha blend (Boolean) default false
offset-x Horizontal offset of the overlay in pixels from the left of the main video (from 0 to 2147483647) (default 0)
offset-y Vertical offset of the overlay in pixels from the top of the main video (from 0 to 2147483647) (default 0)
ama_tile
Video tile
Maps input streams to specified layout
grid-x Number of video tiles per line of the grid (from 1 to 16) (default 2)
grid-y Number of lines in the video grid (from 1 to 16) (default 2)

Compositor Plugin

The ama_compositor plugin provides support for hardware-accelerated video composition operations such as alpha-belding, rotation, drawbox, etc.

Compositor Inputs and Outputs

  • Input: NV12_4L4, NV12, I420, NV12_10LE32, P010_10LE, I420_10LE, I420_10BE, RGBP, ARGB, ABGR, RGBA, BGRA, RGB, BGR, and NV12_10LE_4L4

  • Output: NV12_4L4, NV12, I420, NV12_10LE32, P010_10LE, I420_10LE, I420_10BE, RGBP, ARGB, ABGR, RGBA, BGRA, RGB, BGR, and NV12_10LE_4L4

Compositor Parameters

Sink Pad

Property Name

Description
alpha
Alpha blending of the picture.
Type: float
Rang: 0 - 1
Default: 1
border-color
Border color of the picture encoded as 0x00RRGGBB
Type: Integer
Range: 0 - 16777215
Default: 0
border-inner-width
Inner width of the picture's border in pixels
Type: Integer
Range: 0 - 4294967295
Default: 0
border-outer-width
Outer width of the picture's border in pixels
Type: Integer
Range: 0 - 4294967295
Default: 0
border-outer-width
Outer width of the picture's border in pixels
Type: Integer
Range: 0 - 4294967295
Default: 0
crop-bottom
Pixels to crop at the bottom
Type: Integer
Range: 0 - 4294967295
Default: 0
crop-left
Pixels to crop on the left
Type: Integer
Range: 0 - 4294967295
Default: 0
crop-right
Pixels to crop on the right
Type: Integer
Range: 0 - 4294967295
Default: 0
crop-top
Pixels to crop at the top
Type: Integer
Range: 0 - 4294967295
Default: 0
height
Height of the picture
Type: Integer
Range: 0 - 4294967295
Default: 0
video-direction
Video direction's rotation and flip
Type: Enum
(0): identity - No rotation
(1): 90r - Rotate clockwise 90 degrees
(2): 180 - 180 degrees
(3): 90l - Rotate counter-clockwise 90 degrees
(4): horiz - Horzontal flip
(5): vert - Vertical flip
(6): ul-lr - Flip across upper left/lower right diagonal
(7): ur-ll - Flip across upper right/lower left diagonal
(8): auto - Select flip method based on image-orientation tag
(9): custom - Current status depends on plugin internal setup
width
Width of the picture
Type: Integer
Range: 0 - 4294967295
Default: 0
xpos
X Position of the picture
Type: Integer
Range: 0 - 4294967295
Default: 0
ypos
Y Position of the picture
Type: Integer
Range: 0 - 4294967295
Default: 0
zorder
Z Order of the picture
Type: Integer
Range: 0 - 4294967295
Default: 0
Source Pad

Property Name

Description
background
Background type
Type: Enum
(0) transparent - Transparent Background to enable further compositing
(0) color - Solid Color Background
background-color
Color of the composition's background encoded as 0x00RRGGBB
Type: Unsigned Integer
Range: 0 - 16777215
Default: 0
core-id
ID of the device core to use
Type: Integer
Range: 0 - 1
Default: 0
device
Numeric ID of an XMA device (-1 = unspecified)
Type: Integer.
Range: -1 - 2147483647
Default: -1

Hardware Down and Up Loads

The ama_download and ama_upload plugins provide support for DMA transfer of data to and from the host, respectively. Note that transaction directions are with respect to the host.

Inputs and Outputs

  • Input: I420, NV12, I420_10LE, I420_10BE, ARGB, ABGR, RGBA, BGRA, RGB, BGR

  • Output: I420, NV12, I420_10LE, I420_10BE, ARGB, ABGR, RGBA, BGRA, RGB, BGR

DMA Parameters

Property Name

Description
device
Numeric ID of an XMA device
Type: Integer
Range: -1 - 2147483647
Default: -1 (auto)

Moving Data through the Pipeline

The GStreamer plugins included in the AMD AMA Video SDK take care of efficiently moving data through the video processing pipeline. For optimal performance, video buffers will be moved between the host and the appropriate device only if needed by plugins in the pipeline. This is known as "zero-copy".

  • The decoder plugin will only copy the output buffer from the device to the host if a downstream element (e.g. filesink) running on the host needs to access the decoded frame. This allows downstream hardware-accelerated plugins (e.g. scaler, encoder) running on the same AMD device to process the decoded frames without unnecessary data transfers.

  • The encoder plugin will only copy the input buffer from host to the device if it is allocated in user-space memory on the host or in a different device. This allows the encoder to process frames produced by upstream hardware-accelerated plugins (e.g. decoder, scaler) running on the same AMD device without unnecessary data transfers.

Working with Multiple Devices

By default, if no device identifier is specified, a job is submitted to device 0. When running large jobs or multiple jobs in parallel, device 0 is bound to run out of resources and additional jobs will error out due to insufficient resources. By using the device option of the GStreamer plugins, included in the AMD AMA Video SDK, different functionalities such as decode, scale, etc. of a pipeline can be individually submitted to a specific device. This allows for an easy and straightforward way to leverage the entire video acceleration capacity of your system.

Rebuilding GStreamer

Obtain the zipped version of SDK's Gstreamer code from ma35d_sdk_v1.2.1_gstreamer.zip. Unzip this file into ma35_gstreamer folder. This folder will contain the source code for AMA SDK plugins and a patch for Gstreamer.

You can rebuild the Gstreamer executable with optional plugins by following the instructions below.

  1. Install the prerequisite packages:

    sudo apt install ninja-build flex bison python3-pip unzip pkg-config libjansson-dev uuid-dev
sudo snap install cmake --classic
sudo pip3 install meson ninja

  2. Unzip the downloaded zip file:

    unzip ma35d_sdk_v1.2.1_gstreamer.zip -d ma35_gstreamer

  3. Get Gstreamer:

    git clone https://github.com/GStreamer/gstreamer.git
cd gstreamer
git checkout f6e672f27fcac06af58d7ac995551893d36944ce

  4. Apply the patch to Gstreamer:

    cp ../ma35_gstreamer/ama_gstreamer.patch .
git apply ama_gstreamer.patch

  5. Build Gstreamer and AMA SDK plugins:

    cd ../ma35_gstreamer
ln -s ../gstreamer/subprojects ./
export C_INCLUDE_PATH=/opt/amd/ama/ma35/include/xma/:/opt/amd/ama/ma35/include/xrm/:/opt/amd/ama/ma35/include/xrm_interface/:$C_INCLUDE_PATH
export LIBRARY_PATH=/opt/amd/ama/ma35/lib/:$LIBRARY_PATH
meson setup build
cd build
ninja

  6. Install Gstreamer and plugins:

    ninja install