87 lines
4.7 KiB
TeX
87 lines
4.7 KiB
TeX
\section{Benchmarking harness}\label{sec:bench_harness}
|
|
|
|
To compare full-kernel cycle measurements to throughput predictions on
|
|
individual basic blocks, we lift predictions by adding the weighted basic block
|
|
predictions:
|
|
|
|
\[
|
|
\text{lifted\_pred}(\mathcal{K}) =
|
|
\sum_{b \in \operatorname{BBs}(\mathcal{K})}
|
|
\operatorname{occurences}(b) \times \operatorname{pred}(b)
|
|
\]
|
|
|
|
Our benchmarking harness works in three successive stages. It first
|
|
extracts the basic blocks constituting a computation kernel, and instruments it
|
|
to retrieve their respective occurrences in the original context. It then runs
|
|
all the studied tools on each basic block, while also running measures on the
|
|
whole computation kernel. Finally, the block-level results are lifted to
|
|
kernel-level results thanks to the occurrences previously measured.
|
|
|
|
\subsection{Basic block extraction}\label{ssec:bb_extr}
|
|
|
|
Using the Capstone disassembler~\cite{tool:capstone}, we split the assembly
|
|
code at each control flow instruction (jump, call, return, \ldots) and each
|
|
jump site.
|
|
|
|
To accurately obtain the occurrences of each basic block in the whole kernel's
|
|
computation,
|
|
we then instrument it with \gdb{} by placing a break
|
|
point at each basic block's first instruction in order to count the occurrences
|
|
of each basic block between two calls to the \perf{} counters\footnote{We
|
|
assume the program under analysis to be deterministic.}. While this
|
|
instrumentation takes about 50 to 100$\times$ more time than a regular run, it
|
|
can safely be run in parallel, as the performance results are discarded.
|
|
|
|
\subsection{Throughput predictions and measures}\label{ssec:throughput_pred_meas}
|
|
|
|
The harness leverages a variety of tools: actual CPU measurement; the \bhive{}
|
|
basic block profiler~\cite{bhive}; \llvmmca~\cite{llvm-mca}, \uica~\cite{uica}
|
|
and \iaca~\cite{iaca}, which leverage microarchitectural
|
|
models to predict a block's throughput; \ithemal~\cite{ithemal}, a machine
|
|
learning model; and \gus~\cite{phd:gruber}, a dynamic analyzer based on \qemu{}
|
|
that works at the whole binary level.
|
|
|
|
The execution time of the full kernel is measured using Linux
|
|
\perf~\cite{tool:perf} CPU counters around the full computation kernel. The
|
|
measure is repeated four times and the smallest is kept; this ensures that the
|
|
cache is warm and compensates for context switching or other measurement
|
|
artifacts. \gus{} instruments the whole function body. The other tools included
|
|
all work at basic block level; these are run on each basic block of each
|
|
benchmark.
|
|
|
|
We emphasize the importance, throughout the whole evaluation chain, to keep the
|
|
exact same assembled binary. Indeed, recompiling the kernel from source
|
|
\emph{cannot} be assumed to produce the same assembly kernel. This is even more
|
|
important in the presence of slight changes: for instance, inserting \iaca{}
|
|
markers at the C-level ---~as is intended~--- around the kernel \emph{might}
|
|
change the compiled kernel, if only for alignment regions. We argue that, in
|
|
the case of \iaca{} markers, the problem is even more critical, as those
|
|
markers prevent a binary from being run by overwriting registers with arbitrary
|
|
values. This forces a user to run and measure a version which is different from
|
|
the analyzed one. In our harness, we circumvent this issue by adding markers
|
|
directly at the assembly level, editing the already compiled version. Our
|
|
\gdb{} instrumentation procedure also respects this principle of
|
|
single-compilation. As \qemu{} breaks the \perf{} interface, we have to run
|
|
\gus{} with a preloaded stub shared library to be able to instrument binaries
|
|
containing calls to \perf.
|
|
|
|
\subsection{Prediction lifting and filtering}\label{ssec:harness_lifting}
|
|
|
|
We finally lift single basic block predictions to a whole-kernel cycle
|
|
prediction by summing the block-level results, weighted by the occurrences of
|
|
the basic block in the original context (formula above). If an analyzer fails
|
|
on one of the basic blocks of a benchmark, the whole benchmark is discarded for
|
|
this analyzer.
|
|
|
|
In the presence of complex control flow, \eg{} with conditionals inside loops,
|
|
our approach based on basic block occurrences is arguably less precise than an
|
|
approach based on paths occurrences, as we have less information available
|
|
---~for instance, whether a branch is taken with a regular pattern, whether we
|
|
have constraints on register values, etc. We however chose this block-based
|
|
approach, as most throughput prediction tools work a basic block-level, and are
|
|
thus readily available and can be directly plugged into our harness.
|
|
|
|
Finally, we control the proportion of cache misses in the program's execution
|
|
using \texttt{Cachegrind}~\cite{tool:valgrind} and \gus; programs that have more
|
|
than 15\,\% of cache misses on a warm cache are not considered L1-resident and
|
|
are discarded.
|