Section 1 - An Operator on Its Own¶
iron/operators/ is the operator library: GEMM, GEMV, MHA, RMSNorm, RoPE,
activations and more, each a declared class. You will usually use them
inside a graph (Section 2), but an operator also works on its own: construct
it, call it with tensors, and it compiles once and runs on the device.
Walkthrough of axpy_example.py¶
AXPY computes scalar_factor * a + b.
from iron.operators import AXPY
op = AXPY(size=2048, num_aie_columns=1, tile_size=2048, scalar_factor=1.5)
Construction is keyword-only and does not touch the device. It only declares the shape and tunables.
sizeis aparam(): a dimension of the problem, fixed once the operator is made.num_aie_columnsandtile_sizearetunable()s: choices about how the work is spread across the array. Set them, as here, or leave them out and the operator'sresolve(dev)picks values for the device.scalar_factoris also aparam(). It is compiled into the image rather than read from a tensor on each call.
Each operator's docstring says what it computes. Its construction keywords
are what inspect.signature (or cls.declaration.params) shows.
Calling it¶
Tensor = aie_utils.DEFAULT_TENSOR_CLASS
a, b = Tensor(a_np), Tensor(b_np)
y = Tensor(a_np.shape, dtype=a_np.dtype)
op(a, b, y)
result = y.numpy()
The first call compiles the design and runs it. Later calls with the same shape reuse the compiled image.
Called on its own, an operator takes one device tensor per declared buffer,
in order: every input, then every output (a, b, y here; op.buffers
lists them). aie_utils.DEFAULT_TENSOR_CLASS is the device tensor type of
the current host runtime (XRTTensor under XRT). Wrap a numpy array to
upload it, or give a shape and dtype to allocate a buffer for the device to
write. Inside a graph (Section 2) this is different: Op.step(...) takes
handles and weights, and the graph returns plain numpy arrays.
Checking it against numpy¶
Every operator has reference(*inputs), a numpy computation of what the
device should produce, and tolerance(), how close a device run must come
to it. Neither needs an NPU:
from aie.utils.verify import compare
reference = op.reference(a_np, b_np)
verdict = compare(result, reference, op.resolved().tolerance())
assert verdict, verdict.detail
Every example in this guide checks its result with compare(...). The
tolerance comes from the kernel's contract, an aie.utils.verify.Tolerance
whose note explains the bound. For AXPY it is one bf16 ulp: the kernel
computes in fp32 and rounds once to bf16 on the store. A Verdict is truthy
when the check passes, and verdict.detail describes the result either way.
Exercises¶
- Run
python3 axpy_example.pyon an NPU. You should seePASS!. - Print
op.resolved().tolerance(), then do the same for aReLU.ReLU's isTolerance.exact():max(x, 0)has nothing to round, whileAXPY's multiply-add does. - Leave out
num_aie_columns=1when constructingop.op.num_aie_columnsis thenNone, andop.resolved().num_aie_columnsshows the valueresolve(dev)chose for your device.opitself is unchanged.