193 lines
8.4 KiB
TeX
193 lines
8.4 KiB
TeX
\section{Evaluating \palmed{}}\label{sec:palmed_results}
|
|
|
|
Evaluating \palmed{} on the previously gathered basic blocks now requires, on
|
|
one hand, to define evaluation metrics and, on the other hand, an evaluation
|
|
harness to collect the throughput predictions from \palmed{} and the other
|
|
considered code analyzers, from which metrics will be derived.
|
|
|
|
\subsection{Evaluation harness}
|
|
|
|
We implement into \palmed{} an evaluation harness to evaluate it both against
|
|
native measurement and other code analyzers.
|
|
|
|
We first strip each basic block gathered of its dependencies to fall into the
|
|
use-case of \palmed{} using \pipedream{}, as we did previously. This yields
|
|
assembly code that can be run and measured natively. The body of the most
|
|
nested loop can also be used as an assembly basic block for other code
|
|
analyzers.
|
|
However, as \pipedream{}
|
|
does not support some instructions (control flow, x86-64 divisions, \ldots),
|
|
those are stripped from the original kernel, which might denature the original
|
|
basic block.
|
|
|
|
To evaluate \palmed{}, the same kernel is run:
|
|
|
|
\begin{enumerate}
|
|
|
|
\item{} natively on each CPU, using the \pipedream{} harness to measure its
|
|
execution time;
|
|
|
|
\item{} using the resource mapping \palmed{} produced on the evaluation machine;
|
|
|
|
\item{} using the \uopsinfo{}~\cite{uopsinfo} port mapping, converted to its
|
|
equivalent conjunctive resource mapping\footnote{When this evaluation was
|
|
made, \uica{}~\cite{uica} was not yet published. Since \palmed{} provides a
|
|
resource mapping, the comparison to \uopsinfo{} is fair.};
|
|
|
|
\item{} using \pmevo~\cite{PMEvo}, ignoring any instruction not supported by
|
|
its provided mapping;
|
|
|
|
\item{} using \iaca~\cite{iaca}, by inserting assembly markers around the
|
|
kernel and running the tool;
|
|
|
|
\item{} using \llvmmca~\cite{llvm-mca}, by inserting markers in the
|
|
\pipedream{}-generated assembly code and running the tool.
|
|
|
|
\end{enumerate}
|
|
|
|
The raw results are saved (as a Python \pymodule{pickle} file) for reuse and
|
|
archival.
|
|
|
|
\subsection{Metrics extracted}\label{ssec:palmed_eval_metrics}
|
|
|
|
As \palmed{} internally works with Instructions Per Cycle (IPC) metrics, and as
|
|
all these tools are also able to provide results in IPC, the most natural
|
|
metric to evaluate is the error on the predicted IPC. We measure this as a
|
|
Root-Mean-Square (RMS) error over all basic blocks considered, weighted by each
|
|
basic block's measured occurrences:
|
|
|
|
\[ \text{Err}_\text{RMS, tool} = \sqrt{\sum_{i \in \text{BBs}}
|
|
\frac{\text{weight}_i}{\sum_j \text{weight}_j} \left(
|
|
\frac{\text{IPC}_{i,\text{tool}} - \text{IPC}_{i,\text{native}}}{\text{IPC}_{i,\text{native}}}
|
|
\right)^2
|
|
}
|
|
\]
|
|
|
|
\medskip{}
|
|
|
|
This error metric measures the relative deviation of predictions with respect
|
|
to a baseline. However, depending on how this prediction is used, the relative
|
|
\emph{ordering} of predictions ---~that is, which basic block is faster~---
|
|
might be more important. For instance, a compiler might use such models for
|
|
code selection; here, the goal would not be to predict the performance of the
|
|
kernel selected, but to accurately pick the fastest.
|
|
|
|
For this, we also provide Kendall's $\tau$ coefficient~\cite{kendalltau}. This
|
|
coefficient varies between $-1$ (full anti-correlation) and $1$ (full
|
|
correlation), and measures how many pairs of basic blocks $(i, j)$ were
|
|
correctly ordered by a tool, that is, whether
|
|
|
|
\[
|
|
\text{IPC}_{i,\text{native}} \leq \text{IPC}_{j,\text{native}}
|
|
\iff
|
|
\text{IPC}_{i,\text{tool}} \leq \text{IPC}_{j,\text{tool}}
|
|
\]
|
|
|
|
\medskip{}
|
|
|
|
Finally, we also provide a \emph{coverage} metric for each tool; that is,
|
|
which proportion of basic blocks it was able to process.
|
|
|
|
The definition of \emph{able to process}, however, varies from tool to tool.
|
|
For \iaca{} and \llvmmca{}, this means that the analyzer crashed or ended
|
|
without yielding a result. For \uopsinfo{}, this means that one of the
|
|
instructions of the basic block is absent from the port mapping. \pmevo{},
|
|
however, is evaluated in degraded mode when instructions are not mapped, simply
|
|
ignoring them; it is considered as failed only when \emph{no instruction at
|
|
all} in the basic block was present in the model.
|
|
|
|
This notion of coverage is partial towards \palmed{}. As we use \pipedream{} as
|
|
a baseline measurement, instructions that cannot be benchmarked by \pipedream{}
|
|
are pruned from the benchmarks; hence, \palmed{} has a 100\,\% coverage
|
|
\emph{by construction} --- which does not mean that is supports all the
|
|
instructions found in the original basic blocks.
|
|
|
|
\subsection{Results}
|
|
|
|
\input{40-1_results_fig.tex}
|
|
|
|
We run the evaluation harness on three different machines:
|
|
\begin{itemize}
|
|
\item{} an x86-64 Intel \texttt{SKL-SP}-based machine, with two Intel Xeon Silver
|
|
4114 CPU, totalling 20 cores;
|
|
\item{} an x86-64 AMD \texttt{ZEN1}-based machine, with a single AMD EPYC 7401P
|
|
CPU with 24 cores;
|
|
\item{} an ARMv8a Raspberry Pi 4 with 4 Cortex A72 cores.
|
|
\end{itemize}
|
|
|
|
As \iaca{} only supports Intel CPUs, and \uopsinfo{} only supports x86-64
|
|
machines and gives only very rough
|
|
information for \texttt{ZEN} architectures ---~without port mapping~---, these
|
|
two tools were only tested on the \texttt{SKL-SP} machine.
|
|
|
|
\medskip{}
|
|
|
|
The evaluation metrics for all three architecture and all five tools are
|
|
presented in \autoref{table:palmed_eval}. We further represent IPC prediction
|
|
accuracy as heatmaps in \autoref{fig:palmed_heatmaps}. A dark area at
|
|
coordinate $(x, y)$ means that the selected tool has a prediction accuracy of
|
|
$y$ for a significant number of microkernels with a measured IPC of $x$. The
|
|
closer a prediction is to the red horizontal line, the more accurate it is.
|
|
|
|
These results are analyzed in the full article~\cite{palmed}.
|
|
|
|
\section{Other contributions}
|
|
|
|
\paragraph{Using a database to enhance reproducibility and usability.}
|
|
\palmed{}'s method is driven by a large number of \pipedream{} benchmarks. For
|
|
instance, generating a mapping for an x86-64 machine requires the execution of
|
|
about $10^6$ benchmarks on the CPU\@.
|
|
|
|
Each of these measures takes time: the multiset of instructions must be
|
|
transformed into an assembly code, including the register mapping phrase; this
|
|
assembly must be assembled and linked into an ELF file; and finally, the
|
|
benchmark must be actually executed, with multiple warm-up rounds and multiple
|
|
measures. On average, on the \texttt{SKL-SP} CPU, each benchmark requires half
|
|
to two-thirds of a second on a single core. The whole benchmarking phase, on
|
|
the \texttt{SKL-SP} processor, roughly took eight hours.
|
|
|
|
\medskip{}
|
|
|
|
As \palmed{} relies on the Gurobi optimizer, which is itself non-deterministic,
|
|
\palmed{} cannot be made truly reproducible. However, the slight fluctuations
|
|
in measured cycles between two executions of a benchmark are also a source of
|
|
non-determinism in the execution of Palmed.
|
|
|
|
\medskip{}
|
|
|
|
For both these reasons, we implemented into \palmed{} a database-backed storage of
|
|
measurements. Whenever \palmed{} needs to measure a kernel, it will first try
|
|
to find a corresponding measure in the database; if the measure does not exist
|
|
yet, it will be run, then stored in database.
|
|
|
|
For each measure, we further store for context:
|
|
the time and date at which the measure was made;
|
|
the machine on which the measure was made;
|
|
how many times the measure was repeated;
|
|
how many warm-up rounds were performed;
|
|
how many instructions were in the unrolled loop;
|
|
how many instructions were executed per repetition in total;
|
|
the parameters for \pipedream{}'s assembly generation procedure;
|
|
how the final result was aggregated from the repeated measures;
|
|
the variance of the set of measures;
|
|
how many CPU cores were active when the measure was made;
|
|
which CPU core was used for this measure;
|
|
whether the kernel's scheduler was set to FIFO mode.
|
|
|
|
\bigskip{}
|
|
|
|
We believe that, as a whole, the use of a database increases the usability of
|
|
\palmed{}: it is faster if some measures were already made in the past and
|
|
recovers better upon error.
|
|
|
|
This also gives us a better confidence towards our results: we can easily
|
|
archive and backup our experimental data, and we can easily trace the origin of
|
|
a measure if needed. We can also reuse the exact same measures between two runs
|
|
of \palmed{}, to ensure that the results are as consistent as possible.
|
|
|
|
|
|
\paragraph{General engineering contributions.} Apart from purely scientific
|
|
contributions, we worked on improving \palmed{} as a whole, from the
|
|
engineering point of view: code quality; reliable parallel measurements;
|
|
recovery upon error; logging; \ldots{} These improvements amount to about a
|
|
hundred merge-requests between \nderumig{} and myself.
|