Section 2 - Your First Graph¶
Section 1 called one operator on its own. A real design chains several: a
normalization, a few projections, an activation. A
Graph records that chain once, and
the tensors passed between its operators stay in device memory instead of
returning to the host between calls.
Walkthrough of first_graph.py¶
import iron
from iron.operators import AXPY, ReLU
class ReluThenAxpy(iron.Graph):
"""``scalar_factor * relu(x) + y``, two operators as one compiled image."""
def __init__(self, scalar_factor=2.0):
self.scalar_factor = scalar_factor
def body(self, x, y):
activated = ReLU.step(x)
return AXPY.step(activated, y, scalar_factor=self.scalar_factor)
A graph is a subclass of iron.Graph with a body you write. The
positional parameters of body (x and y) are the graph's inputs, and
its return value is the graph's output. __init__ is plain Python that runs
once, at construction.
Inside body, Op.step(...) adds an operator to the graph. It records
a step instead of running one and returns a Handle, a stand-in for the
tensor that step will produce. ReLU.step(x) records a ReLU reading x,
and AXPY.step(activated, y, ...) reads the handle it returned. Nothing
runs on the device yet: body runs once per input shape, when the graph is
recorded.
Op.step(...) infers an operator's shape parameters from the handles it is
given, so AXPY's size comes from activated, just as GEMV.step(w, x)
reads M and K from its inputs. Other construction keywords, such as
AXPY's scalar_factor, go to .step(...) as they would to the class.
Compiling and calling it¶
compile(**shapes) runs body on handles of those shapes (the
recording) and lowers the recorded steps to an image for the current
device: one full ELF on NPU2, an xclbin per step on NPU1. The graph keeps
the resulting CompiledGraph, keyed by input shapes, and graph(x, y)
calls the version that matches.
If no version is loaded yet, calling the graph with a new shape compiles it
on the spot. Once a version is loaded, a new shape raises ValueError.
Compile every shape you need before the first call: all versions share
one device buffer arena, and a later compile may have to grow it.
Checking it against numpy¶
As with a single operator, graph.reference(*tensors) runs body on the
host, each step through its operator's reference(), with no NPU needed.
The guide's tests use it
to check this example without a device.
A device call is checked with aie.utils.verify.compare(result, reference,
tolerance), as in Section 1. Here only one step rounds: ReLU is exact in
bf16, so AXPY's tolerance covers the whole chain. first_graph.py takes
the AXPY step from the compiled graph's recording
(compiled.recording.steps) and uses its tolerance(). Section 6 shows how
to check a graph where several steps round.
Exercises¶
- Run
python3 first_graph.pyon an NPU. You should seePASS!. - Add a third operator, for example another
AXPY.stepthat adds a third inputz, and extendbody's signature andmain()'s tensors to match. One compile still covers the whole chain. - Print
graph.record(x=(2048,), y=(2048,)), aRecording, and look at its.steps: each is one operator and the handles it reads and writes.compilebuilds this before lowering it to an image. - Print
ReLU'sop.resolved().tolerance(), as Section 1 did forAXPY, and confirm it isTolerance.exact(), with nortol,atolorulps.