This repository contains infrastructure for the benchmarking and performance profiling of the xDSL compiler framework.
airspeed velocity (asv) is a tool for benchmarking Python packages over their lifetime. Runtime, memory consumption and even custom-computed values may be tracked. The results are displayed in an interactive web frontend that requires only a basic static webserver to host.
Every day by the cron schedule 0 4 * * *
, a GitHub actions workflow is run
using ASV to benchmark the 15 most recent commits to the xDSL repository, and
commit the results to the .asv/results/github-action
directory of an
artefact repository. Then, the
interactive web frontend is built from these results and all previously
committed results from previous workflow runs, then finally deployed to GitHub
pages.
This web frontend can be found at https://xdsl.dev/xdsl-bench/.
## Profiling
In addition to running under ASV, all benchmarks can be profiled with a variety
of tools using the infrastructure in bench_utils.py
. This provides a simple
CLI when a benchmark file is directly run which through which the tool and
benchmark can be specified. The help page for this CLI is as follows:
uv run python3
BENCHMARK.py --help
usage: BENCHMARK.py [-h] [-o OUTPUT] [-q]
{BENCHMARK_NAME,...}
{timeit,snakeviz,viztracer,flameprof}
positional arguments:
{BENCHMARK_NAME,...}
name of the benchmark to run
{timeit,snakeviz,viztracer,flameprof}
profiler to use
options:
-h, --help show this help message and exit
-o OUTPUT, --output OUTPUT
the directory into which to write out the profile
files
-q, --quiet don't show the profiler's UI
To use viztracer
to profile the lexer on apply_pdl_extra_file.mlir
(the
"Lexer.apply_pdl_extra_fil" benchmark):
uv run python3 components.py Lexer.apply_pdl_extra_file viztracer
This infrastructure can be modified to support further profilers by adding
further subcommands and command implementations to the bench_utils.py
file.
For a holistic view of xDSL's performance, we provide three levels of
granularity for benchmarking. At the highest level, end-to-end tests such as
running xdsl-opt
on MLIR files capture the entire compiler pipeline. Below
this, component tests such as lexing MLIR files only capture individual stages
in the compiler pipeline. Finally, microbenchmarks evaluate properties of the
implementation, and are desiged to align with
existing tests of MLIR.
- End-to-end (
end_to_end.py
)- Constant folding (with and without verifier)
- Empty program
- Loop unrolling
- CIRCT workload
- ASL workload
- ...
- Component (
components.py
)- Lexer
- Parser
- Pattern rewriter
- Printer
- Verifier
- Microbenchmarks (
microbenchmarks.py
)- IR traversal (direct block iteration and walking)
- Dialect loading
- Import machinery
- Extensibility through interface/trait lookups
- Operation creation
- ...