Skip to content

Performance Checks

CI measures every figure here on a Krackan (NPU2) and a Phoenix (NPU1) runner, keeps each run on the ci branch, and rebuilds this page after every run. Times are UTC.

About these figures

A test marked @pytest.mark.bench records its latency, bandwidth or throughput; a model's test records its time to first token (TTFT) and tokens per second (TPS), and its accuracy checks' figures beside them.

Suite Benchmarks Runs on History here
Pull requests Operators, and each model's shortest call every pull request, and each push to devel and main the pushes to devel and main; a pull request's own figures are in the comment CI posts on it
Extensive Operators, the pull request suite's and the extensive ones each night, and each push to devel and main every run
Nightly Models, every test on the real checkpoints each night every run

Each trend card follows one headline figure: each model checkpoint's decode TPS (with its TTFT) from the nightly suite, and each operator's throughput, or its bandwidth where it counts no operations, from the extensive suite, on one parametrization per device, one both devices run where there is one. A decoder's card draws a line for each form each device runs it in, styled as the key under the cards explains. The History section draws every parametrization and metric; opened from a model's card, it draws every context and form of that checkpoint, a colour per context and the cards' line style per form.

A move between a benchmark's last two runs is called better or worse only past the same two gates as the pull request comment: at least 5%, and more than twice the spread the runs measured. A run whose checks failed is drawn with a ✕; its figure is what the test measured, not a result to trust. Each benchmark links to its operator in the operator reference or to Models. The history keeps every run of the last 90 days and, before that, the newest run of each week, at most 400 runs.

CI asks the NPU for its turbo power mode before it measures, and each run records the mode it measured in. A timing is compared only with runs in the same mode, and its line breaks where the mode changed. The History charts draw each run's spread as a bar: a latency's median absolute deviation over its timed calls, and a bandwidth's or throughput's share of it.