README.md

@@ -1,9 +1,10 @@
 # Dwarf Assembly
 
-A compiler from DWARF unwinding data to native x86\_64 binaries.
+Some experiments around compiling the most used Dwarf informations (ELF debug
+data) directly into assembly.
 
-This repository also contains various experiments, tools, benchmarking scripts,
+This project is a big work in progress, don't expect anything to be stable for
-stats scripts, etc. to work on this compiler.
+now.
 
 ## Dependencies
 
@@ -16,8 +17,7 @@ As of now, this project relies on the following libraries:
     - [libsrk31cxx](https://github.com/stephenrkell/libsrk31cxx)
 
 These libraries are expected to be installed somewhere your compiler can find
-them. If you are using Archlinux, you can check
+them.
-[these `PKGBUILD`s](https://git.tobast.fr/m2-internship/pkgbuilds).
 
 ## Scripts and directories
 
@@ -26,40 +26,4 @@ them. If you are using Archlinux, you can check
 * `./compare_sizes.py`: compare the sizes of the `.eh_frame` of a binary (and
   its dependencies) with the sizes of the `.text` of the generated ELFs.
 * `./extract_pc.py`: extracts a list of valid program counters of an ELF and
-  produce a file as read by `dwarf-assembly`, **deprecated**.
+  produce a file as read by `dwarf-assembly`
-
-* `benching`: all about benchmarking
-* `env`: environment variables manager to ease the use of various `eh_elf`s in
-  parallel, for experiments.
-* `shared`: code shared between various subprojects
-* `src`: the compiler code itself
-* `stack_walker`: a primitive stack walker using `eh_elf`s
-* `stack_walker_libunwind`: a primitive stack walker using vanilla `libunwind`
-* `stats`: a statistics gathering module
-* `tests`: some tests regarding `eh_elf`s, **deprecated**.
-
-## How to use
-
-To compile `eh_elf`s for a given ELF file, say `foo.bin`, it is recommended to
-use `generate_eh_elf.py`. Help can be obtained with `--help`. A standard
-command is
-
-```bash
-./generate_eh_elf.py --deps --enable-deref-arg --global-switch -o eh_elfs foo.bin
-```
-
-This will compile `foo.bin` and all the shared objects it relies on into
-`eh_elf`s, in the directory `./eh_elfs`, using a dereferencing argument (which
-is necessary for `perf-eh_elfs`).
-
-## Generate the intermediary C file
-
-If you're curious about the intermediary C file generated for a given ELF file
-`foo.bin`, you must call `dwarf-assembly` directly. A parameter `--help` can be
-passed; a standard command is
-
-```bash
-./dwarf-assembly --global-switch --enable-deref-arg foo.bin
-```
-
-**Beware**! This will generate the C code on the standard output.

benching/README.md

@@ -1,92 +0,0 @@
-# Benching `eh_elfs`
-
-## Benchmark setup
-
-Pick some name for your `eh_elfs` directory. We will call it `$EH_ELF_DIR`.
-
-### Generate the `eh_elfs`
-
-```bash
-../../generate_eh_elf.py --deps -o "$EH_ELF_DIR" \
-  --keep-holes -O2 --global-switch --enable-deref-arg "$BENCHED_BINARY"
-```
-
-### Record a `perf` session
-
-```bash
-perf record --call-graph dwarf,4096 "$BENCHED_BINARY" [args]
-```
-
-### Set up the environment
-
-```bash
-source ../../env/apply [vanilla | vanilla-nocache | *eh_elf] [dbg | *release]
-```
-
-The first value selects the version of libunwind you will be running, the
-second selects whether you want to run in debug or release mode (use release to
-get readings, debug to check for errors).
-
-You can reset your environment to its previous state by running `deactivate`.
-
-If you pick the `eh_elf` flavour, you will also have to
-
-```bash
-export LD_LIBRARY_PATH="$EH_ELF_DIR:$LD_LIBRARY_PATH"
-```
-
-## Extract results
-
-### Base readings
-
-**In release mode** (faster), run
-
-```bash
-perf report 2>&1 >/dev/null
-```
-
-with both `eh_elf` and `vanilla` shells. Compare average time.
-
-### Getting debug output
-
-```bash
-UNW_DEBUG_LEVEL=5 perf report 2>&1 >/dev/null
-```
-
-### Total number of calls to `unw_step`
-
-```bash
-UNW_DEBUG_LEVEL=5 perf report 2>&1 >/dev/null | grep -c "step:.* returning"
-```
-
-### Total number of vanilla errors
-
-With the `vanilla` context,
-
-```bash
-UNW_DEBUG_LEVEL=5 perf report 2>&1 >/dev/null | grep -c "step:.* returning -"
-```
-
-### Total number of fallbacks to original DWARF
-
-With the `eh_elf` context,
-
-```bash
-UNW_DEBUG_LEVEL=5 perf report 2>&1 >/dev/null | grep -c "step:.* falling back"
-```
-
-### Total number of fallbacks to original DWARF that actually used DWARF
-
-With the `eh_elf` context,
-
-```bash
-UNW_DEBUG_LEVEL=5 perf report 2>&1 >/dev/null | grep -c "step:.* fallback with"
-```
-
-### Get succeeded fallback locations
-
-```bash
-UNW_DEBUG_LEVEL=5 perf report 2>&1 >/dev/null \
-  | grep "step: .* fallback with" -B 15 \
-  | grep "In memory map" | sort | uniq -c
-```

benching/csmith/.gitignore

@@ -1 +0,0 @@
-/tests

benching/gzip/.gitignore

@@ -1,3 +0,0 @@
-gzip
-gzip-1.10
-perf.data

benching/gzip/EVALUATION.md

@@ -1,49 +0,0 @@
-# gzip - evaluation
-
-Artifacts saved in `evaluation_artifacts`.
-
-## Performance
-
-Using the command line
-
-```bash
-for i in $(seq 1 100); do
-  perf report 2>&1 >/dev/null | tail -n 1 \
-    | python ../hackbench/to_report_fmt.py \
-    | sed 's/^.* & .* & \([0-9]*\) & .*$/\1/g'
-done
-```
-
-we save a sequence of 100 performance readings to some file.
-
-Samples:
-* `eh_elf`:  331134 unw/exec
-* `vanilla`: 331144 unw/exec
-
-Average time/unw:
-* `eh_elf`:    83 ns
-* `vanilla`: 1304 ns
-
-Standard deviation:
-* `eh_elf`:   2 ns
-* `vanilla`: 24 ns
-
-Average ratio: 15.7
-Ratio uncertainty: 0.8
-
-## Distibution of `unw_step` issues
-
-### `eh_elf` case
-
-* success:                              331134 (99.9%)
-* fallback to DWARF:                         2  (0.0%)
-* fallback to libunwind heuristics:          8  (0.0%)
-* fail to unwind:                          379  (0.1%)
-* total:                                331523
-
-### `vanilla` case
-
-* success:                              331136 (99.9%)
-* fallback to libunwind heuristics:          8  (0.0%)
-* fail to unwind:                          379  (0.1%)
-* total:                                331523

benching/hackbench/.gitignore

@@ -1,5 +0,0 @@
-/eh_elfs*
-/bench*
-/perf.data*
-/perfperf.data*
-/hackbench

benching/hackbench/EVALUATION.md

@@ -1,48 +0,0 @@
-# Hackbench - evaluation
-
-Artifacts saved in `evaluation_artifacts`.
-
-## Performance
-
-Using the command line
-
-```bash
-for i in $(seq 1 100); do
-  perf report 2>&1 >/dev/null | tail -n 1 \
-    | python to_report_fmt.py | sed 's/^.* & .* & \([0-9]*\) & .*$/\1/g'
-done
-```
-
-we save a sequence of 100 performance readings to some file.
-
-Samples:
-* `eh_elf`:  135251 unw/exec
-* `vanilla`: 138233 unw/exec
-
-Average time/unw:
-* `eh_elf`:   102 ns
-* `vanilla`: 2443 ns
-
-Standard deviation:
-* `eh_elf`:  2 ns
-* `vanilla`: 47 ns
-
-Average ratio: 24
-Ratio uncertainty: 1.0
-
-## Distibution of `unw_step` issues
-
-### `eh_elf` case
-
-* success:                              135251 (97.7%)
-* fallback to DWARF:                      1467  (1.0%)
-* fallback to libunwind heuristics:        329  (0.2%)
-* fail to unwind:                         1410  (1.0%)
-* total:                                138457
-
-### `vanilla` case
-
-* success:                              138201 (98.9%)
-* fallback to libunwind heuristics:         32  (0.0%)
-* fail to unwind:                         1411  (1.0%)
-* total:                                139644

benching/hackbench/README.md

@@ -1,44 +0,0 @@
-# Running the benchmarks
-
-Pick some name for your `eh_elfs` directory. We will call it `$EH_ELF_DIR`.
-
-## Generate the `eh_elfs`
-
-```bash
-../../generate_eh_elf.py --deps -o "$EH_ELF_DIR" \
-  --keep-holes -O2 --global-switch --enable-deref-arg hackbench
-```
-
-## Record a `perf` session
-
-```bash
-perf record --call-graph dwarf,4096 ./hackbench 10 process 100
-```
-
-You can arbitrarily increase the first number up to ~100 and the second to get
-a longer session. This will most probably take all your computer's resources
-while it is running.
-
-## Set up the environment
-
-```bash
-source ../../env/apply [vanilla | vanilla-nocache | *eh_elf] [dbg | *release]
-```
-
-The first value selects the version of libunwind you will be running, the
-second selects whether you want to run in debug or release mode (use release to
-get readings, debug to check for errors).
-
-You can reset your environment to its previous state by running `deactivate`.
-
-If you pick the `eh_elf` flavour, you will also have to
-
-```bash
-export LD_LIBRARY_PATH="$EH_ELF_DIR:$LD_LIBRARY_PATH"
-```
-
-### Actually get readings
-
-```bash
-perf report 2>&1 >/dev/null
-```

benching/hackbench/to_report_fmt.py

@@ -1,21 +0,0 @@
-#!/usr/bin/env python3
-
-import re
-import sys
-
-line = input()
-regex = \
-    re.compile(r'Total unwind time: ([0-9]*) s ([0-9]*) ns, ([0-9]*) calls')
-
-match = regex.match(line.strip())
-if not match:
-    print('Badly formatted line', file=sys.stderr)
-    sys.exit(1)
-
-sec = int(match.group(1))
-ns = int(match.group(2))
-calls = int(match.group(3))
-
-time = sec * 10**9 + ns
-
-print("{} & {} & {} & ??".format(calls, time, time // calls))

benching/python3.7/test.py

@@ -1,8 +0,0 @@
-def slow_fibo(n):
-    if n <= 1:
-        return 1
-    return slow_fibo(n - 1) + slow_fibo(n - 2)
-
-
-if __name__ == "__main__":
-    slow_fibo(35)

benching/tools/common.sh

@@ -1,18 +0,0 @@
-#!/bin/bash
-
-if [ "$#" -lt 1 ] ; then
-    >&2 echo "Missing argument: directory"
-    exit 1
-fi
-
-BENCH_DIR="$(echo $1 | sed 's@/$@@g')"
-ENV_APPLY="$(readlink -f "$(dirname $0)/../../env/apply")"
-
-if ! [ -f "$ENV_APPLY" ] ; then
-    >&2 echo "Cannot find helper scripts. Abort."
-    exit 1
-fi
-
-function status_report {
-    echo -e "\e[33;1m[$BENCH_DIR]\e[0m $1"
-}

benching/tools/errors.sh

@@ -1,101 +0,0 @@
-#!/bin/bash
-
-source "$(dirname $0)/common.sh"
-
-TMP_FILE=$(mktemp)
-if [ -z "$EH_ELFS_NAME" ]; then
-    EH_ELFS_NAME="eh_elfs"
-fi
-
-function get_perf_output {
-    envtype=$1
-    source $ENV_APPLY "$envtype" "dbg"
-    LD_LIBRARY_PATH="$BENCH_DIR/$EH_ELFS_NAME:$LD_LIBRARY_PATH" \
-        UNW_DEBUG_LEVEL=15 \
-        perf report -i "$BENCH_DIR/perf.data" 2>$TMP_FILE >/dev/null
-    deactivate
-}
-
-function count_successes {
-    cat $TMP_FILE | tail -n 1 | sed 's/^.*, \([0-9]*\) calls.*$/\1/g'
-}
-
-function count_total_calls {
-    cat $TMP_FILE | grep -c "^  >.*step:.* returning"
-}
-
-function count_errors {
-    cat $TMP_FILE | grep -c "^  >.*step:.* returning -"
-}
-
-function count_eh_fallbacks {
-    cat $TMP_FILE | grep -c "step:.* falling back"
-}
-
-function count_vanilla_fallbacks {
-    cat $TMP_FILE | grep -c "step:.* frame-chain"
-}
-
-function count_fallbacks_to_dwarf {
-    cat $TMP_FILE | grep -c "step:.* fallback with"
-}
-
-function count_fallbacks_failed {
-    cat $TMP_FILE | grep -c "step:.* dwarf_step also failed"
-}
-
-function count_fail_after_fallback_to_dwarf {
-    cat $TMP_FILE \
-        | "$(dirname $0)/line_patterns.py" \
-            "fallback with" \
-            "step:.* unw_step called" \
-            ~"step:.* unw_step called" \
-            "step:.* returning -" \
-        | grep Complete -c
-}
-
-function report {
-    flavour="$1"
-
-    status_report "$flavour issues distribution"
-
-    successes=$(count_successes)
-    failures=$(count_errors)
-    total=$(count_total_calls)
-
-    if [ "$flavour" = "eh_elf" ]; then
-        fallbacks=$(count_eh_fallbacks)
-        fallbacks_to_dwarf=$(count_fallbacks_to_dwarf)
-        fallbacks_to_dwarf_failed_after=$(count_fail_after_fallback_to_dwarf)
-        fallbacks_failed=$(count_fallbacks_failed)
-        fallbacks_to_heuristics="$(( $fallbacks \
-            - $fallbacks_to_dwarf \
-            - $fallbacks_failed))"
-        echo -e "* success:\t\t\t\t$successes"
-        echo -e "* fallback to DWARF:\t\t\t$fallbacks_to_dwarf"
-        echo -e "* …of which failed at next step:\t$fallbacks_to_dwarf_failed_after"
-        echo -e "* fallback to libunwind heuristics:\t$fallbacks_to_heuristics"
-        computed_sum=$(( $successes + $fallbacks - $fallbacks_failed + $failures ))
-    else
-        fallbacks=$(count_vanilla_fallbacks)
-        successes=$(( $successes - $fallbacks ))
-        echo -e "* success:\t\t\t\t$successes"
-        echo -e "* fallback to libunwind heuristics:\t$fallbacks"
-        computed_sum=$(( $successes + $fallbacks + $failures ))
-    fi
-    echo -e "* fail to unwind:\t\t\t$failures"
-    echo -e "* total:\t\t\t\t$total"
-    if [ "$computed_sum" -ne "$total" ] ; then
-        echo "-- WARNING: missing cases (computed sum $computed_sum != $total) --"
-    fi
-}
-
-# eh_elf stats
-get_perf_output "eh_elf"
-report "eh_elf"
-
-# Vanilla stats
-get_perf_output "vanilla"
-report "vanilla"
-
-rm "$TMP_FILE"

benching/tools/errors_new.sh

@@ -1,86 +0,0 @@
-#!/bin/bash
-
-source "$(dirname $0)/common.sh"
-
-TMP_FILE=$(mktemp)
-
-function get_perf_output {
-    envtype=$1
-    source $ENV_APPLY "$envtype" "dbg"
-    LD_LIBRARY_PATH="$BENCH_DIR/eh_elfs:$LD_LIBRARY_PATH" \
-        UNW_DEBUG_LEVEL=15 \
-        perf report -i "$BENCH_DIR/perf.data" 2>$TMP_FILE >/dev/null
-    deactivate
-}
-
-function count_successes {
-    cat $TMP_FILE | tail -n 1 | sed 's/^.*, \([0-9]*\) calls.*$/\1/g'
-}
-
-function count_total_calls {
-    cat $TMP_FILE | grep -c "^  >.*step:.* returning"
-}
-
-function count_errors {
-    cat $TMP_FILE | grep -c "^  >.*step:.* returning -"
-}
-
-function count_eh_fallbacks {
-    cat $TMP_FILE | grep -c "step:.* falling back"
-}
-
-function count_vanilla_fallbacks {
-    cat $TMP_FILE | grep -c "step:.* frame-chain"
-}
-
-function count_fallbacks_to_dwarf {
-    cat $TMP_FILE | grep -c "step:.* fallback with"
-}
-
-function count_fallbacks_failed {
-    cat $TMP_FILE | grep -c "step:.* dwarf_step also failed"
-}
-
-function report {
-    flavour="$1"
-
-    status_report "$flavour issues distribution"
-
-    successes=$(count_successes)
-    failures=$(count_errors)
-    total=$(count_total_calls)
-
-    if [ "$flavour" = "eh_elf" ]; then
-        fallbacks=$(count_eh_fallbacks)
-        fallbacks_to_dwarf=$(count_fallbacks_to_dwarf)
-        fallbacks_failed=$(count_fallbacks_failed)
-        fallbacks_to_heuristics="$(( $fallbacks \
-            - $fallbacks_to_dwarf \
-            - $fallbacks_failed))"
-        echo -e "* success:\t\t\t\t$successes"
-        echo -e "* fallback to DWARF:\t\t\t$fallbacks_to_dwarf"
-        echo -e "* fallback to libunwind heuristics:\t$fallbacks_to_heuristics"
-        computed_sum=$(( $successes + $fallbacks - $fallbacks_failed + $failures ))
-    else
-        fallbacks=$(count_vanilla_fallbacks)
-        successes=$(( $successes - $fallbacks ))
-        echo -e "* success:\t\t\t\t$successes"
-        echo -e "* fallback to libunwind heuristics:\t$fallbacks"
-        computed_sum=$(( $successes + $fallbacks + $failures ))
-    fi
-    echo -e "* fail to unwind:\t\t\t$failures"
-    echo -e "* total:\t\t\t\t$total"
-    if [ "$computed_sum" -ne "$total" ] ; then
-        echo "-- WARNING: missing cases (computed sum $computed_sum != $total) --"
-    fi
-}
-
-# eh_elf stats
-get_perf_output "eh_elf"
-report "eh_elf"
-
-# Vanilla stats
-get_perf_output "vanilla"
-report "vanilla"
-
-rm "$TMP_FILE"

benching/tools/gen_evals.sh

@@ -1,27 +0,0 @@
-OUTPUT="$1"
-NB_ITER=10
-
-if [ "$#" -lt 1 ] ; then
-    >&2 echo "Missing argument: output directory."
-    exit 1
-fi
-
-if [ -z "$EH_ELFS" ]; then
-    >&2 echo "Missing environment: EH_ELFS. Aborting."
-    exit 1
-fi
-
-mkdir -p "$OUTPUT"
-
-for flavour in 'eh_elf' 'vanilla' 'vanilla-nocache'; do
-    >&2 echo "$flavour..."
-    source "$(dirname "$0")/../../env/apply" "$flavour" release
-    for iter in $(seq 1 $NB_ITER); do
-        >&2 echo -e "\t$iter..."
-        LD_LIBRARY_PATH="$EH_ELFS:$LD_LIBRARY_PATH" \
-            perf report 2>&1 >/dev/null | tail -n 1 \
-            | python "$(dirname $0)/to_report_fmt.py" \
-            | sed 's/^.* & .* & \([0-9]*\) & .*$/\1/g'
-    done > "$OUTPUT/${flavour}_times"
-    deactivate
-done

benching/tools/gen_perf_stats.py

@@ -1,106 +0,0 @@
-#!/usr/bin/env python3
-
-""" Generates performance statistics for the eh_elf vs vanilla libunwind unwinding,
-based on time series generated beforehand
-
-Intended to be run from `statistics.sh`
-"""
-
-from collections import namedtuple
-import numpy as np
-import sys
-import os
-
-
-Datapoint = namedtuple("Datapoint", ["nb_frames", "total_time", "avg_time"])
-
-
-def read_series(path):
-    with open(path, "r") as handle:
-        for line in handle:
-            nb_frames, total_time, avg_time = map(int, line.strip().split())
-            yield Datapoint(nb_frames, total_time, avg_time)
-
-
-FLAVOURS = ["eh_elf", "vanilla"]
-WITH_NOCACHE = False
-
-if "WITH_NOCACHE" in os.environ:
-    WITH_NOCACHE = True
-    FLAVOURS.append("vanilla-nocache")
-
-path_format = os.path.join(sys.argv[1], "{}_times")
-datapoints = {}
-avg_times = {}
-total_times = {}
-avgs_total = {}
-avgs = {}
-std_deviations = {}
-unwound_frames = {}
-
-for flv in FLAVOURS:
-    datapoints[flv] = list(read_series(path_format.format(flv)))
-    avg_times[flv] = list(map(lambda x: x.avg_time, datapoints[flv]))
-    total_times[flv] = list(map(lambda x: x.total_time, datapoints[flv]))
-    avgs[flv] = sum(avg_times[flv]) / len(avg_times[flv])
-    avgs_total[flv] = sum(total_times[flv]) / len(total_times[flv])
-    std_deviations[flv] = np.sqrt(np.var(avg_times[flv]))
-
-    cur_unwound_frames = list(map(lambda x: x.nb_frames, datapoints[flv]))
-    unwound_frames[flv] = cur_unwound_frames[0]
-    for run_id, unw_frames in enumerate(cur_unwound_frames[1:]):
-        if unw_frames != unwound_frames[flv]:
-            print(
-                "{}, run {}: unwound {} frames, reference unwound {}".format(
-                    flv, run_id + 1, unw_frames, unwound_frames[flv]
-                ),
-                file=sys.stderr,
-            )
-
-avg_ratio = avgs["vanilla"] / avgs["eh_elf"]
-ratio_uncertainty = (
-    1
-    / avgs["eh_elf"]
-    * (
-        std_deviations["vanilla"]
-        + avgs["vanilla"] / avgs["eh_elf"] * std_deviations["eh_elf"]
-    )
-)
-
-
-def format_flv(flv_dict, formatter, alterator=None):
-    out = ""
-    for flv in FLAVOURS:
-        val = flv_dict[flv]
-        altered = alterator(val) if alterator else val
-        out += "* {}: {}\n".format(flv, formatter.format(altered))
-    return out
-
-
-def get_ratios(avgs):
-    def avg_of(flavour):
-        return avgs[flavour] / avgs["eh_elf"]
-
-    if WITH_NOCACHE:
-        return "\n\tcached: {}\n\tuncached: {}".format(
-            avg_of("vanilla"), avg_of("vanilla-nocache")
-        )
-    else:
-        return avg_of("vanilla")
-
-
-print(
-    "Unwound frames:\n{}\n"
-    "Average whole unwinding time (one run):\n{}\n"
-    "Average time to unwind one frame:\n{}\n"
-    "Standard deviation:\n{}\n"
-    "Average ratio: {}\n"
-    "Ratio uncertainty: {}".format(
-        format_flv(unwound_frames, "{}"),
-        format_flv(avgs_total, "{} μs", alterator=lambda x: x // 1000),
-        format_flv(avgs, "{} ns"),
-        format_flv(std_deviations, "{}"),
-        get_ratios(avgs),
-        ratio_uncertainty,
-    )
-)

benching/tools/line_patterns.py

@@ -1,69 +0,0 @@
-#!/usr/bin/env python3
-
-import sys
-import re
-
-
-class Match:
-    def __init__(self, re_str, negate=False):
-        self.re = re.compile(re_str)
-        self.negate = negate
-
-    def matches(self, line):
-        return self.re.search(line) is not None
-
-
-class Matcher:
-    def __init__(self, match_objs):
-        self.match_objs = match_objs
-        self.match_pos = 0
-        self.matches = 0
-
-        if not self.match_objs:
-            raise Exception("No match expressions provided")
-        if self.match_objs[-1].negate:
-            raise Exception("The last match object must be a positive expression")
-
-    def feed(self, line):
-        for cur_pos, exp in enumerate(self.match_objs[self.match_pos :]):
-            cur_pos = cur_pos + self.match_pos
-            if not exp.negate:  # Stops the for here, whether matching or not
-                if exp.matches(line):
-                    self.match_pos = cur_pos + 1
-                    print(
-                        "Passing positive {}, advance to {}".format(
-                            cur_pos, self.match_pos
-                        )
-                    )
-                    if self.match_pos >= len(self.match_objs):
-                        print("> Complete match, reset.")
-                        self.matches += 1
-                        self.match_pos = 0
-                return
-            else:
-                if exp.matches(line):
-                    print("Failing negative [{}] {}, reset".format(exp.negate, cur_pos))
-                    old_match_pos = self.match_pos
-                    self.match_pos = 0
-                    if old_match_pos != 0:
-                        print("> Refeed: ", end="")
-                        self.feed(line)
-                    return
-
-
-def get_args(args):
-    out_args = []
-    for arg in args:
-        negate = False
-        if arg[0] == "~":
-            negate = True
-            arg = arg[1:]
-        out_args.append(Match(arg, negate))
-    return out_args
-
-
-if __name__ == "__main__":
-    matcher = Matcher(get_args(sys.argv[1:]))
-    for line in sys.stdin:
-        matcher.feed(line)
-    print(matcher.matches)

benching/tools/statistics.sh

@@ -1,43 +0,0 @@
-#!/bin/bash
-
-source "$(dirname $0)/common.sh"
-
-TEMP_DIR="$(mktemp -d)"
-NB_RUNS=10
-
-function collect_perf_time_data {
-    envtype=$1
-    source $ENV_APPLY "$envtype" "release"
-    LD_LIBRARY_PATH="$BENCH_DIR/eh_elfs:$LD_LIBRARY_PATH" \
-        perf report -i "$BENCH_DIR/perf.data" 2>&1 >/dev/null \
-        | tail -n 1 \
-        | python "$(dirname $0)/to_report_fmt.py" \
-        | sed 's/^\([0-9]*\) & \([0-9]*\) & \([0-9]*\) & .*$/\1 \2 \3/g'
-    deactivate
-}
-
-function collect_perf_time_data_runs {
-    envtype=$1
-    outfile=$2
-    status_report "Collecting $envtype data over $NB_RUNS runs"
-    rm -f "$outfile"
-    for run in $(seq 1 $NB_RUNS); do
-        collect_perf_time_data "$envtype" >> "$outfile"
-    done
-}
-
-eh_elf_data="$TEMP_DIR/eh_elf_times"
-vanilla_data="$TEMP_DIR/vanilla_times"
-
-collect_perf_time_data_runs "eh_elf" "$eh_elf_data"
-collect_perf_time_data_runs "vanilla" "$vanilla_data"
-
-if [ -n "$WITH_NOCACHE" ]; then
-    vanilla_nocache_data="$TEMP_DIR/vanilla-nocache_times"
-    collect_perf_time_data_runs "vanilla-nocache" "$vanilla_nocache_data"
-fi
-
-status_report "benchmark statistics"
-python "$(dirname "$0")/gen_perf_stats.py" "$TEMP_DIR"
-
-rm -rf "$TEMP_DIR"

benching/tools/to_report_fmt.py

@@ -1,21 +0,0 @@
-#!/usr/bin/env python3
-
-import re
-import sys
-
-line = input()
-regex = \
-    re.compile(r'Total unwind time: ([0-9]*) s ([0-9]*) ns, ([0-9]*) calls')
-
-match = regex.match(line.strip())
-if not match:
-    print('Badly formatted line', file=sys.stderr)
-    sys.exit(1)
-
-sec = int(match.group(1))
-ns = int(match.group(2))
-calls = int(match.group(3))
-
-time = sec * 10**9 + ns
-
-print("{} & {} & {} & ??".format(calls, time, time // calls))

env/apply

@@ -2,6 +2,24 @@
 ## Source this file.
 ## Usage: apply [vanilla | vanilla-nocache | *eh_elf] [dbg | *release]
 
+# ==== DEFINE DEACTIVATE ====
+
+function deactivate {
+    if [ "$IS_EHELFSAVE_EVT" -eq 1 ] ; then
+        unset IS_EHELFSAVE_EVT
+
+        export CPATH="$CPATH_EHELFSAVE"
+        export LIBRARY_PATH="$LIBRARY_PATH_EHELFSAVE"
+        export LD_LIBRARY_PATH="$LD_LIBRARY_PATH_EHELFSAVE"
+        export PS1="$PS1_EHELFSAVE"
+
+        unset CPATH_EHELFSAVE
+        unset LIBRARY_PATH_EHELFSAVE
+        unset LD_LIBRARY_PATH_EHELFSAVE
+        unset PS1_EHELFSAVE
+    fi
+}
+
 # ==== INPUT ACQUISITION ====
 flavour="eh_elf"
 dbg="release"
@@ -21,29 +39,6 @@ while [ "$#" -gt 0 ] ; do
     shift
 done
 
-# ==== UNSET PREVIOUS ENVIRONMENT ====
-
-type -t deactivate
-[ -n "$(type -t deactivate)" ] && deactivate
-
-# ==== DEFINE DEACTIVATE ====
-
-function deactivate {
-    export CPATH="$CPATH_EHELFSAVE"
-    export LIBRARY_PATH="$LIBRARY_PATH_EHELFSAVE"
-    export LD_LIBRARY_PATH="$LD_LIBRARY_PATH_EHELFSAVE"
-    export PS1="$PS1_EHELFSAVE"
-    export PATH="$PATH_EHELFSAVE"
-
-    unset CPATH_EHELFSAVE
-    unset LIBRARY_PATH_EHELFSAVE
-    unset LD_LIBRARY_PATH_EHELFSAVE
-    unset PS1_EHELFSAVE
-    unset PATH_EHELFSAVE
-
-    unset deactivate
-}
-
 # ==== PREFIX ====
 export PERF_PREFIX="$HOME/local/perf-$flavour"
 
@@ -81,23 +76,16 @@ export LIBUNWIND_PREFIX
 function colon_prepend {
     if [ -z "$2" ]; then
         echo "$1"
-    elif [ -z "$1" ] ; then
-        echo "$2"
     else
+        >&2 echo ">$2<"
         echo "$1:$2"
     fi
 }
 
-function ifpath {
+export IS_EHELFSAVE_EVT=1
-    if [ -e "$1" ] ; then
-        echo "$1"
-    fi
-}
-
 export CPATH_EHELFSAVE="$CPATH"
 export LIBRARY_PATH_EHELFSAVE="$LIBRARY_PATH"
 export LD_LIBRARY_PATH_EHELFSAVE="$LD_LIBRARY_PATH"
-export PATH_EHELFSAVE="$PATH"
 export PS1_EHELFSAVE="$PS1"
 
 export CPATH="$(colon_prepend \
@@ -106,13 +94,5 @@ export LIBRARY_PATH="$(colon_prepend \
     "$LIBUNWIND_PREFIX/lib/:$PERF_PREFIX/lib" "$LIBRARY_PATH")"
 export LD_LIBRARY_PATH="$(colon_prepend \
     "$LIBUNWIND_PREFIX/lib/:$PERF_PREFIX/lib" "$LD_LIBRARY_PATH")"
-export PATH="$(colon_prepend \
-    "$(colon_prepend \
-        "$(ifpath "$LIBUNWIND_PREFIX/bin")" \
-        "$(ifpath "$PERF_PREFIX/bin")")" \
-    "$PATH")"
 
 export PS1="($flavour $dbg) $PS1"
-
-unset ifpath
-unset colon_prepend

generate_eh_elf.py

@@ -13,122 +13,120 @@ import tempfile
 import argparse
 from enum import Enum
 
-from shared_python import (
+from shared_python import \
-    elf_so_deps,
+    elf_so_deps, \
-    do_remote,
+    do_remote, \
-    is_newer,
+    is_newer, \
-    to_eh_elf_path,
+    to_eh_elf_path, \
-    find_eh_elf_dir,
+    find_eh_elf_dir, \
-    DEFAULT_AUX_DIRS,
+    DEFAULT_AUX_DIRS
-)
 from extract_pc import generate_pc_list
 
 
 DWARF_ASSEMBLY_BIN = os.path.join(
-    os.path.dirname(os.path.abspath(sys.argv[0])), "dwarf-assembly"
+    os.path.dirname(os.path.abspath(sys.argv[0])),
-)
+    'dwarf-assembly')
-C_BIN = "gcc" if "C" not in os.environ else os.environ["C"]
+C_BIN = (
+    'gcc' if 'C' not in os.environ
+    else os.environ['C'])
 
 
 class SwitchGenPolicy(Enum):
-    """ The various switch generation policies possible """
+    ''' The various switch generation policies possible '''
-
+    SWITCH_PER_FUNC = '--switch-per-func'
-    SWITCH_PER_FUNC = "--switch-per-func"
+    GLOBAL_SWITCH = '--global-switch'
-    GLOBAL_SWITCH = "--global-switch"
 
 
 class Config:
-    """ Holds the run's settings """
+    ''' Holds the run's settings '''
 
     default_aux = DEFAULT_AUX_DIRS
 
-    def __init__(
+    def __init__(self,
-        self,
+                 output,
-        output,
+                 aux,
-        aux,
+                 no_dft_aux,
-        no_dft_aux,
+                 objects,
-        objects,
+                 sw_gen_policy=SwitchGenPolicy.GLOBAL_SWITCH,
-        sw_gen_policy=SwitchGenPolicy.GLOBAL_SWITCH,
+                 force=False,
-        force=False,
+                 use_pc_list=False,
-        use_pc_list=False,
+                 c_opt_level='3',
-        c_opt_level="3",
+                 enable_deref_arg=False,
-        enable_deref_arg=False,
+                 cc_debug=False,
-        keep_holes=False,
+                 remote=None):
-        cc_debug=False,
+        self.output = '.' if output is None else output
-        remote=None,
+        self.aux = (
-    ):
+            aux
-        self.output = "." if output is None else output
+            + ([] if no_dft_aux else self.default_aux)
-        self.aux = aux + ([] if no_dft_aux else self.default_aux)
+        )
         self.objects = objects
         self.sw_gen_policy = sw_gen_policy
         self.force = force
         self.use_pc_list = use_pc_list
         self.c_opt_level = c_opt_level
         self.enable_deref_arg = enable_deref_arg
-        self.keep_holes = keep_holes
         self.cc_debug = cc_debug
         self.remote = remote
 
     @staticmethod
     def default_aux_str():
-        return ", ".join(Config.default_aux)
+        return ', '.join(Config.default_aux)
 
     def dwarf_assembly_args(self):
-        """ Arguments to `dwarf_assembly` """
+        ''' Arguments to `dwarf_assembly` '''
         out = []
         out.append(self.sw_gen_policy.value)
         if self.enable_deref_arg:
-            out.append("--enable-deref-arg")
+            out.append('--enable-deref-arg')
-        if self.keep_holes:
-            out.append("--keep-holes")
         return out
 
     def cc_opts(self):
-        """ Options to pass to the C compiler """
+        ''' Options to pass to the C compiler '''
-        out = ["-fPIC"]
+        out = ['-fPIC']
         if self.cc_debug:
-            out.append("-g")
+            out.append('-g')
         out.append(self.opt_level())
         return out
 
     def opt_level(self):
-        """ The optimization level to pass to gcc """
+        ''' The optimization level to pass to gcc '''
-        return "-O{}".format(self.c_opt_level)
+        return '-O{}'.format(self.c_opt_level)
 
     def aux_dirs(self):
-        """ Get the list of auxiliary directories """
+        ''' Get the list of auxiliary directories '''
         return self.aux
 
 
 def gen_dw_asm_c(obj_path, out_path, config, pc_list_path=None):
-    """ Generate the C code produced by dwarf-assembly from `obj_path`, saving
+    ''' Generate the C code produced by dwarf-assembly from `obj_path`, saving
-    it as `out_path` """
+    it as `out_path` '''
 
     dw_assembly_args = config.dwarf_assembly_args()
     if pc_list_path is not None:
-        dw_assembly_args += ["--pc-list", pc_list_path]
+        dw_assembly_args += ['--pc-list', pc_list_path]
 
     try:
-        with open(out_path, "w") as out_handle:
+        with open(out_path, 'w') as out_handle:
             # TODO enhance error handling
-            command_args = [DWARF_ASSEMBLY_BIN, obj_path] + dw_assembly_args
+            dw_asm_output = subprocess.check_output(
-            dw_asm_output = subprocess.check_output(command_args).decode("utf-8")
+                [DWARF_ASSEMBLY_BIN, obj_path] + dw_assembly_args) \
+                .decode('utf-8')
             out_handle.write(dw_asm_output)
     except subprocess.CalledProcessError as exn:
         raise Exception(
-            (
+            ("Cannot generate C code from object file {} using {}: process "
-                "Cannot generate C code from object file {} using {}: process "
+             "terminated with exit code {}.").format(
-                "terminated with exit code {}."
+                 obj_path,
-            ).format(obj_path, DWARF_ASSEMBLY_BIN, exn.returncode)
+                 DWARF_ASSEMBLY_BIN,
-        )
+                 exn.returncode))
 
 
 def resolve_symlink_chain(objpath):
-    """ Resolves a symlink chain. This returns a pair `(new_obj, chain)`,
+    ''' Resolves a symlink chain. This returns a pair `(new_obj, chain)`,
     `new_obj` being the canonical path for `objpath`, and `chain` being a list
     representing the path followed, eg. `[(objpath, a), (a, b), (b, new_obj)]`.
     The goal of this function is to allow reproducing symlink architectures at
-    the eh_elf level. """
+    the eh_elf level. '''
 
     chain = []
     out_path = objpath
@@ -144,16 +142,16 @@ def resolve_symlink_chain(objpath):
 
 
 def find_out_dir(obj_path, config):
-    """ Find the directory in which the eh_elf corresponding to `obj_path` will
+    ''' Find the directory in which the eh_elf corresponding to `obj_path` will
-    be outputted, among the output directory and the aux directories """
+    be outputted, among the output directory and the aux directories '''
 
     return find_eh_elf_dir(obj_path, config.aux_dirs(), config.output)
 
 
 def gen_eh_elf(obj_path, config):
-    """ Generate the eh_elf corresponding to `obj_path`, saving it as
+    ''' Generate the eh_elf corresponding to `obj_path`, saving it as
     `out_dir/$(basename obj_path).eh_elf.so` (or in the current working
-    directory if out_dir is None) """
+    directory if out_dir is None) '''
 
     out_dir = find_out_dir(obj_path, config)
     obj_path, link_chain = resolve_symlink_chain(obj_path)
@@ -163,20 +161,19 @@ def gen_eh_elf(obj_path, config):
     link_chain = map(
         lambda elt: (
             to_eh_elf_path(elt[0], out_dir),
-            os.path.basename(to_eh_elf_path(elt[1], out_dir)),
+            os.path.basename(to_eh_elf_path(elt[1], out_dir))),
-        ),
+        link_chain)
-        link_chain,
-    )
 
     out_base_name = to_eh_elf_path(obj_path, out_dir, base=True)
     out_so_path = to_eh_elf_path(obj_path, out_dir, base=False)
-    pc_list_dir = os.path.join(out_dir, "pc_list")
+    pc_list_dir = os.path.join(out_dir, 'pc_list')
 
     if is_newer(out_so_path, obj_path) and not config.force:
         return  # The object is recent enough, no need to recreate it
 
     if os.path.exists(out_dir) and not os.path.isdir(out_dir):
-        raise Exception("The output path {} is not a directory.".format(out_dir))
+        raise Exception("The output path {} is not a directory.".format(
+            out_dir))
     if not os.path.exists(out_dir):
         os.makedirs(out_dir, exist_ok=True)
 
@@ -184,38 +181,42 @@ def gen_eh_elf(obj_path, config):
         # Generate PC list
         pc_list_path = None
         if config.use_pc_list:
-            pc_list_path = os.path.join(pc_list_dir, out_base_name + ".pc_list")
+            pc_list_path = \
+                os.path.join(pc_list_dir, out_base_name + '.pc_list')
             os.makedirs(pc_list_dir, exist_ok=True)
-            print("\tGenerating PC list…")
+            print('\tGenerating PC list…')
             generate_pc_list(obj_path, pc_list_path)
 
         # Generate the C source file
         print("\tGenerating C…")
-        c_path = os.path.join(compile_dir, (out_base_name + ".c"))
+        c_path = os.path.join(compile_dir, (out_base_name + '.c'))
         gen_dw_asm_c(obj_path, c_path, config, pc_list_path)
 
         # Compile it into a .o
         print("\tCompiling into .o…")
-        o_path = os.path.join(compile_dir, (out_base_name + ".o"))
+        o_path = os.path.join(compile_dir, (out_base_name + '.o'))
         if config.remote:
             remote_out = do_remote(
                 config.remote,
-                [C_BIN, "-o", out_base_name + ".o", "-c", out_base_name + ".c"]
+                [
-                + config.cc_opts(),
+                    C_BIN,
+                    '-o', out_base_name + '.o',
+                    '-c', out_base_name + '.c'
+                ] + config.cc_opts(),
                 send_files=[c_path],
-                retr_files=[(out_base_name + ".o", o_path)],
+                retr_files=[(out_base_name + '.o', o_path)])
-            )
             call_rc = 1 if remote_out is None else 0
         else:
             call_rc = subprocess.call(
-                [C_BIN, "-o", o_path, "-c", c_path, config.opt_level(), "-fPIC"]
+                [C_BIN, '-o', o_path, '-c', c_path,
-            )
+                 config.opt_level(), '-fPIC'])
         if call_rc != 0:
             raise Exception("Failed to compile to a .o file")
 
         # Compile it into a .so
         print("\tCompiling into .so…")
-        call_rc = subprocess.call([C_BIN, "-o", out_so_path, "-shared", o_path])
+        call_rc = subprocess.call(
+            [C_BIN, '-o', out_so_path, '-shared', o_path])
         if call_rc != 0:
             raise Exception("Failed to compile to a .so file")
 
@@ -224,32 +225,40 @@ def gen_eh_elf(obj_path, config):
         if os.path.exists(elt[0]):
             if not os.path.islink(elt[0]):
                 raise Exception(
-                    "{}: file already exists and is not a symlink.".format(elt[0])
+                    "{}: file already exists and is not a symlink.".format(
-                )
+                        elt[0]))
             os.remove(elt[0])
         os.symlink(elt[1], elt[0])
 
 
 def gen_all_eh_elf(obj_path, config):
-    """ Call `gen_eh_elf` on obj_path and all its dependencies """
+    ''' Call `gen_eh_elf` on obj_path and all its dependencies '''
     deps = elf_so_deps(obj_path)
     deps.append(obj_path)
     for dep in deps:
         gen_eh_elf(dep, config)
 
 
-def gen_eh_elfs(obj_path, out_dir, global_switch=True, deps=True, remote=None):
+def gen_eh_elfs(obj_path,
-    """ Call gen{_all,}_eh_elf with args setup accordingly with the given
+                out_dir,
-    options """
+                global_switch=True,
+                deps=True,
+                remote=None):
+    ''' Call gen{_all,}_eh_elf with args setup accordingly with the given
+    options '''
 
     switch_gen_policy = (
-        SwitchGenPolicy.GLOBAL_SWITCH
+        SwitchGenPolicy.GLOBAL_SWITCH if global_switch
-        if global_switch
         else SwitchGenPolicy.SWITCH_PER_FUNC
     )
 
     config = Config(
-        out_dir, [], False, [obj_path], sw_gen_policy=switch_gen_policy, remote=remote
+        out_dir,
+        [],
+        False,
+        [obj_path],
+        sw_gen_policy=switch_gen_policy,
+        remote=remote,
     )
 
     if deps:
@@ -258,176 +267,106 @@ def gen_eh_elfs(obj_path, out_dir, global_switch=True, deps=True, remote=None):
 
 
 def process_args():
-    """ Process `sys.argv` arguments """
+    ''' Process `sys.argv` arguments '''
 
     parser = argparse.ArgumentParser(
-        description="Compile ELFs into their related eh_elfs"
+        description="Compile ELFs into their related eh_elfs",
     )
 
-    parser.add_argument(
+    parser.add_argument('--deps', action='store_const',
-        "--deps",
+                        const=gen_all_eh_elf, default=gen_eh_elf,
-        action="store_const",
+                        dest='gen_func',
-        const=gen_all_eh_elf,
+                        help=("Also generate eh_elfs for the shared objects "
-        default=gen_eh_elf,
+                              "this object depends on"))
-        dest="gen_func",
+    parser.add_argument('-o', '--output', metavar="path",
-        help=("Also generate eh_elfs for the shared objects " "this object depends on"),
+                        help=("Save the generated objects at the given path "
-    )
+                              "instead of the current working directory"))
-    parser.add_argument(
+    parser.add_argument('-a', '--aux', action='append', default=[],
-        "-o",
+                        help=("Alternative output directories. These "
-        "--output",
+                              "directories are searched for existing matching "
-        metavar="path",
+                              "eh_elfs, and if found, these files are updated "
-        help=(
+                              "instead of creating new files in the --output "
-            "Save the generated objects at the given path "
+                              "directory. By default, some aux directories "
-            "instead of the current working directory"
+                              "are always considered, unless -A is passed: "
-        ),
+                              "{}.").format(Config.default_aux_str()))
-    )
+    parser.add_argument('-A', '--no-dft-aux', action='store_true',
-    parser.add_argument(
+                        help=("Do not use the default auxiliary output "
-        "-a",
+                              "directories: {}.").format(
-        "--aux",
+                                  Config.default_aux_str()))
-        action="append",
+    parser.add_argument('--remote', metavar='ssh_args',
-        default=[],
+                        help=("Execute the heavyweight commands on the remote "
-        help=(
+                              "machine, using `ssh ssh_args`."))
-            "Alternative output directories. These "
+    parser.add_argument('--use-pc-list', action='store_true',
-            "directories are searched for existing matching "
+                        help=("Generate a PC list using `extract_pc.py` for "
-            "eh_elfs, and if found, these files are updated "
+                              "each processed ELF file, and call "
-            "instead of creating new files in the --output "
+                              "dwarf-assembly accordingly."))
-            "directory. By default, some aux directories "
+    parser.add_argument('--force', '-f', action='store_true',
-            "are always considered, unless -A is passed: "
+                        help=("Force re-generation of the output files, even "
-            "{}."
+                              "when those files are newer than the target "
-        ).format(Config.default_aux_str()),
+                              "ELF."))
-    )
+    parser.add_argument('--enable-deref-arg', action='store_true',
-    parser.add_argument(
+                        help=("Pass the `--enable-deref-arg` to "
-        "-A",
+                              "dwarf-assembly, enabling an extra `deref` "
-        "--no-dft-aux",
+                              "argument for each lookup function, allowing "
-        action="store_true",
+                              "to work on remote address spaces."))
-        help=("Do not use the default auxiliary output " "directories: {}.").format(
+    parser.add_argument("-g", "--cc-debug", action='store_true',
-            Config.default_aux_str()
+                        help=("Compile the source file with -g for easy "
-        ),
+                              "debugging"))
-    )
-    parser.add_argument(
-        "--remote",
-        metavar="ssh_args",
-        help=(
-            "Execute the heavyweight commands on the remote "
-            "machine, using `ssh ssh_args`."
-        ),
-    )
-    parser.add_argument(
-        "--use-pc-list",
-        action="store_true",
-        help=(
-            "Generate a PC list using `extract_pc.py` for "
-            "each processed ELF file, and call "
-            "dwarf-assembly accordingly."
-        ),
-    )
-    parser.add_argument(
-        "--force",
-        "-f",
-        action="store_true",
-        help=(
-            "Force re-generation of the output files, even "
-            "when those files are newer than the target "
-            "ELF."
-        ),
-    )
-    parser.add_argument(
-        "--enable-deref-arg",
-        action="store_true",
-        help=(
-            "Pass the `--enable-deref-arg` to "
-            "dwarf-assembly, enabling an extra `deref` "
-            "argument for each lookup function, allowing "
-            "to work on remote address spaces."
-        ),
-    )
-    parser.add_argument(
-        "--keep-holes",
-        action="store_true",
-        help=(
-            "Keep holes between FDEs instead of filling "
-            "them with junk. More accurate, less compact."
-        ),
-    )
-    parser.add_argument(
-        "-g",
-        "--cc-debug",
-        action="store_true",
-        help=("Compile the source file with -g for easy " "debugging"),
-    )
     # c_opt_level
     opt_level_grp = parser.add_mutually_exclusive_group()
-    opt_level_grp.add_argument(
+    opt_level_grp.add_argument('-O0', action='store_const', const='0',
-        "-O0",
+                               dest='c_opt_level',
-        action="store_const",
+                               help=("Compile C file with this optimization "
-        const="0",
+                                     "level."))
-        dest="c_opt_level",
+    opt_level_grp.add_argument('-O1', action='store_const', const='1',
-        help=("Compile C file with this optimization " "level."),
+                               dest='c_opt_level',
-    )
+                               help=("Compile C file with this optimization "
-    opt_level_grp.add_argument(
+                                     "level."))
-        "-O1",
+    opt_level_grp.add_argument('-O2', action='store_const', const='2',
-        action="store_const",
+                               dest='c_opt_level',
-        const="1",
+                               help=("Compile C file with this optimization "
-        dest="c_opt_level",
+                                     "level."))
-        help=("Compile C file with this optimization " "level."),
+    opt_level_grp.add_argument('-O3', action='store_const', const='3',
-    )
+                               dest='c_opt_level',
-    opt_level_grp.add_argument(
+                               help=("Compile C file with this optimization "
-        "-O2",
+                                     "level."))
-        action="store_const",
+    opt_level_grp.add_argument('-Os', action='store_const', const='s',
-        const="2",
+                               dest='c_opt_level',
-        dest="c_opt_level",
+                               help=("Compile C file with this optimization "
-        help=("Compile C file with this optimization " "level."),
+                                     "level."))
-    )
+    opt_level_grp.set_defaults(c_opt_level='3')
-    opt_level_grp.add_argument(
-        "-O3",
-        action="store_const",
-        const="3",
-        dest="c_opt_level",
-        help=("Compile C file with this optimization " "level."),
-    )
-    opt_level_grp.add_argument(
-        "-Os",
-        action="store_const",
-        const="s",
-        dest="c_opt_level",
-        help=("Compile C file with this optimization " "level."),
-    )
-    opt_level_grp.set_defaults(c_opt_level="3")
 
-    switch_gen_policy = parser.add_mutually_exclusive_group(required=True)
+    switch_gen_policy = \
-    switch_gen_policy.add_argument(
+        parser.add_mutually_exclusive_group(required=True)
-        "--switch-per-func",
+    switch_gen_policy.add_argument('--switch-per-func',
-        dest="sw_gen_policy",
+                                   dest='sw_gen_policy',
-        action="store_const",
+                                   action='store_const',
-        const=SwitchGenPolicy.SWITCH_PER_FUNC,
+                                   const=SwitchGenPolicy.SWITCH_PER_FUNC,
-        help=("Passed to dwarf-assembly."),
+                                   help=("Passed to dwarf-assembly."))
-    )
+    switch_gen_policy.add_argument('--global-switch',
-    switch_gen_policy.add_argument(
+                                   dest='sw_gen_policy',
-        "--global-switch",
+                                   action='store_const',
-        dest="sw_gen_policy",
+                                   const=SwitchGenPolicy.GLOBAL_SWITCH,
-        action="store_const",
+                                   help=("Passed to dwarf-assembly."))
-        const=SwitchGenPolicy.GLOBAL_SWITCH,
+    parser.add_argument('object', nargs='+',
-        help=("Passed to dwarf-assembly."),
+                        help="The ELF object(s) to process")
-    )
-    parser.add_argument("object", nargs="+", help="The ELF object(s) to process")
     return parser.parse_args()
 
 
 def main():
     args = process_args()
     config = Config(
-        output=args.output,
+        args.output,
-        aux=args.aux,
+        args.aux,
-        no_dft_aux=args.no_dft_aux,
+        args.no_dft_aux,
-        objects=args.object,
+        args.object,
-        sw_gen_policy=args.sw_gen_policy,
+        args.sw_gen_policy,
-        force=args.force,
+        args.force,
-        use_pc_list=args.use_pc_list,
+        args.use_pc_list,
-        c_opt_level=args.c_opt_level,
+        args.c_opt_level,
-        enable_deref_arg=args.enable_deref_arg,
+        args.enable_deref_arg,
-        keep_holes=args.keep_holes,
+        args.cc_debug,
-        cc_debug=args.cc_debug,
+        args.remote,
-        remote=args.remote,
     )
 
     for obj in args.object:

src/CodeGenerator.cpp

@@ -271,16 +271,6 @@ void CodeGenerator::gen_of_reg(const SimpleDwarf::DwRegister& reg,
             }
             break;
         }
-        case SimpleDwarf::DwRegister::REG_PLT_EXPR: {
-            /*
-            if(settings::enable_deref_arg)
-                stream << "(deref(";
-            else
-                stream << "*((uintptr_t*)(";
-            */
-            stream << "(((ctx.rip & 15) >= 11) ? 8 : 0) + ctx.rsp";
-            break;
-        }
         case SimpleDwarf::DwRegister::REG_NOT_IMPLEMENTED:
             stream << "0";
             throw UnhandledRegister();

src/ConseqEquivFilter.cpp

@@ -2,7 +2,7 @@
 
 using namespace std;
 
-ConseqEquivFilter::ConseqEquivFilter(bool enable): SimpleDwarfFilter(enable) {}
+ConseqEquivFilter::ConseqEquivFilter() {}
 
 static bool equiv_reg(
         const SimpleDwarf::DwRegister& r1,

src/ConseqEquivFilter.hpp

@@ -9,7 +9,7 @@
 
 class ConseqEquivFilter: public SimpleDwarfFilter {
     public:
-        ConseqEquivFilter(bool enable=true);
+        ConseqEquivFilter();
 
     private:
         SimpleDwarf do_apply(const SimpleDwarf& dw) const;

src/DwarfReader.cpp

@@ -1,7 +1,5 @@
 #include "DwarfReader.hpp"
 
-#include "plt_std_expr.hpp"
-
 #include <fstream>
 #include <fileno.hpp>
 #include <set>
@@ -9,25 +7,14 @@
 using namespace std;
 using namespace dwarf;
 
+typedef std::set<std::pair<int, core::FrameSection::register_def> >
+    dwarfpp_row_t;
+
 DwarfReader::DwarfReader(const string& path):
     root(fileno(ifstream(path)))
 {}
 
-// Debug function -- dumps an expression
+SimpleDwarf DwarfReader::read() const {
-static void dump_expr(const core::FrameSection::register_def& reg) {
-    assert(reg.k == core::FrameSection::register_def::SAVED_AT_EXPR
-           || reg.k == core::FrameSection::register_def::VAL_OF_EXPR);
-
-    const encap::loc_expr& expr = reg.saved_at_expr_r();
-
-    for(const auto& elt: expr) {
-        fprintf(stderr, "(%02x, %02llx, %02llx, %02llx) :: ",
-                elt.lr_atom, elt.lr_number, elt.lr_number2, elt.lr_offset);
-    }
-    fprintf(stderr, "\n");
-}
-
-SimpleDwarf DwarfReader::read() {
     const core::FrameSection& fs = root.get_frame_section();
     SimpleDwarf output;
 
@@ -39,119 +26,57 @@ SimpleDwarf DwarfReader::read() {
     return output;
 }
 
-void DwarfReader::add_cell_to_row(
+SimpleDwarf::Fde DwarfReader::read_fde(const core::Fde& fde) const {
-        const dwarf::core::FrameSection::register_def& reg,
-        int reg_id,
-        int ra_reg,
-        SimpleDwarf::DwRow& cur_row)
-{
-    if(reg_id == DW_FRAME_CFA_COL3) {
-        cur_row.cfa = read_register(reg);
-    }
-    else {
-        try {
-            SimpleDwarf::MachineRegister reg_type =
-                from_dwarfpp_reg(reg_id, ra_reg);
-            switch(reg_type) {
-                case SimpleDwarf::REG_RBP:
-                    cur_row.rbp = read_register(reg);
-                    break;
-                case SimpleDwarf::REG_RBX:
-                    cur_row.rbx = read_register(reg);
-                    break;
-                case SimpleDwarf::REG_RA:
-                    cur_row.ra = read_register(reg);
-                    break;
-                default:
-                    break;
-            }
-        }
-        catch(const UnsupportedRegister&) {} // Just ignore it.
-    }
-}
-
-void DwarfReader::append_row_to_fde(
-        const dwarfpp_row_t& row,
-        uintptr_t row_addr,
-        int ra_reg,
-        SimpleDwarf::Fde& output)
-{
-    SimpleDwarf::DwRow cur_row;
-
-    cur_row.ip = row_addr;
-
-    for(const auto& cell: row) {
-        add_cell_to_row(cell.second, cell.first, ra_reg, cur_row);
-    }
-
-    if(cur_row.cfa.type == SimpleDwarf::DwRegister::REG_UNDEFINED)
-    {
-        // Not set
-        throw InvalidDwarf();
-    }
-
-    output.rows.push_back(cur_row);
-}
-
-template<typename Key, typename Value>
-static std::set<std::pair<Key, Value> > map_to_setpair(
-        const std::map<Key, Value>& src_map)
-{
-    std::set<std::pair<Key, Value> > out;
-    for(const auto map_it: src_map) {
-        out.insert(map_it);
-    }
-    return out;
-}
-
-void DwarfReader::append_results_to_fde(
-        const dwarf::core::FrameSection::instrs_results& results,
-        int ra_reg,
-        SimpleDwarf::Fde& output)
-{
-    for(const auto row_pair: results.rows) {
-        append_row_to_fde(
-                row_pair.second,
-                row_pair.first.lower(),
-                ra_reg,
-                output);
-    }
-    if(results.unfinished_row.size() > 0) {
-        try {
-            append_row_to_fde(
-                    map_to_setpair(results.unfinished_row),
-                    results.unfinished_row_addr,
-                    ra_reg,
-                    output);
-        } catch(const InvalidDwarf&) {
-            // Ignore: the unfinished_row can be undefined
-        }
-    }
-}
-
-SimpleDwarf::Fde DwarfReader::read_fde(const core::Fde& fde) {
     SimpleDwarf::Fde output;
     output.fde_offset = fde.get_fde_offset();
     output.beg_ip = fde.get_low_pc();
     output.end_ip = fde.get_low_pc() + fde.get_func_length();
 
+    auto rows = fde.decode().rows;
     const core::Cie& cie = *fde.find_cie();
     int ra_reg = cie.get_return_address_register_rule();
 
-    // CIE rows
+    for(const auto row_pair: rows) {
-    core::FrameSection cie_fs(root.get_dbg(), true);
+        SimpleDwarf::DwRow cur_row;
-    auto cie_rows = cie_fs.interpret_instructions(
-            cie,
-            fde.get_low_pc(),
-            cie.get_initial_instructions(),
-            cie.get_initial_instructions_length());
 
-    // FDE rows
+        cur_row.ip = row_pair.first.lower();
-    auto fde_rows = fde.decode();
 
-    // instrs
+        const dwarfpp_row_t& row = row_pair.second;
-    append_results_to_fde(cie_rows, ra_reg, output);
+
-    append_results_to_fde(fde_rows, ra_reg, output);
+        for(const auto& cell: row) {
+            if(cell.first == DW_FRAME_CFA_COL3) {
+                cur_row.cfa = read_register(cell.second);
+            }
+            else {
+                try {
+                    SimpleDwarf::MachineRegister reg_type =
+                        from_dwarfpp_reg(cell.first, ra_reg);
+                    switch(reg_type) {
+                        case SimpleDwarf::REG_RBP:
+                            cur_row.rbp = read_register(cell.second);
+                            break;
+                        case SimpleDwarf::REG_RBX:
+                            cur_row.rbx = read_register(cell.second);
+                            break;
+                        case SimpleDwarf::REG_RA:
+                            cur_row.ra = read_register(cell.second);
+                            break;
+                        default:
+                            break;
+                    }
+                }
+                catch(const UnsupportedRegister&) {} // Just ignore it.
+            }
+        }
+
+        if(cur_row.cfa.type == SimpleDwarf::DwRegister::REG_UNDEFINED)
+        {
+            // Not set
+            throw InvalidDwarf();
+        }
+
+        output.rows.push_back(cur_row);
+    }
 
     return output;
 }
@@ -182,13 +107,6 @@ SimpleDwarf::DwRegister DwarfReader::read_register(
                 output.type = SimpleDwarf::DwRegister::REG_UNDEFINED;
                 break;
 
-            case core::FrameSection::register_def::SAVED_AT_EXPR:
-                if(is_plt_expr(reg))
-                    output.type = SimpleDwarf::DwRegister::REG_PLT_EXPR;
-                else if(!interpret_simple_expr(reg, output))
-                    output.type = SimpleDwarf::DwRegister::REG_NOT_IMPLEMENTED;
-                break;
-
             default:
                 output.type = SimpleDwarf::DwRegister::REG_NOT_IMPLEMENTED;
                 break;
@@ -221,70 +139,3 @@ SimpleDwarf::MachineRegister DwarfReader::from_dwarfpp_reg(
             throw UnsupportedRegister();
     }
 }
-
-static bool compare_dw_expr(
-        const encap::loc_expr& e1,
-        const encap::loc_expr& e2)
-{
-    const std::vector<encap::expr_instr>& e1_vec =
-        static_cast<const vector<encap::expr_instr>&>(e1);
-    const std::vector<encap::expr_instr>& e2_vec =
-        static_cast<const vector<encap::expr_instr>&>(e2);
-
-    return e1_vec == e2_vec;
-}
-
-bool DwarfReader::is_plt_expr(
-        const core::FrameSection::register_def& reg) const
-{
-    if(reg.k != core::FrameSection::register_def::SAVED_AT_EXPR)
-        return false;
-    const encap::loc_expr& expr = reg.saved_at_expr_r();
-
-    bool res = compare_dw_expr(expr, REFERENCE_PLT_EXPR);
-    return res;
-}
-
-bool DwarfReader::interpret_simple_expr(
-        const dwarf::core::FrameSection::register_def& reg,
-        SimpleDwarf::DwRegister& output
-        ) const
-{
-    bool deref = false;
-    if(reg.k == core::FrameSection::register_def::SAVED_AT_EXPR)
-        deref = true;
-    else if(reg.k == core::FrameSection::register_def::VAL_OF_EXPR)
-        deref = false;
-    else
-        return false;
-
-    const encap::loc_expr& expr = reg.saved_at_expr_r();
-    if(expr.size() > 2 || expr.empty())
-        return false;
-
-    const auto& exp_reg = expr[0];
-    if(0x70 <= exp_reg.lr_atom && exp_reg.lr_atom <= 0x8f) { // DW_OP_breg<n>
-        int reg_id = exp_reg.lr_atom - 0x70;
-        try {
-            output.reg = from_dwarfpp_reg(reg_id, -1); // Cannot be CFA anyway
-            output.offset = exp_reg.lr_number;
-        } catch(const UnsupportedRegister& /* exn */) {
-            return false; // Unsupported register
-        }
-    }
-
-    if(expr.size() == 2) { // OK if deref
-        if(expr[1].lr_atom == 0x06) { // deref
-            if(deref)
-                return false;
-            deref = true;
-        }
-        else
-            return false;
-    }
-
-    if(deref)
-        return false; // TODO try stats? Mabye it's worth implementing
-    output.type = SimpleDwarf::DwRegister::REG_REGISTER;
-    return true;
-}

src/DwarfReader.hpp

@@ -13,9 +13,6 @@
 
 #include "SimpleDwarf.hpp"
 
-typedef std::set<std::pair<int, dwarf::core::FrameSection::register_def> >
-    dwarfpp_row_t;
-
 class DwarfReader {
     public:
         class InvalidDwarf: public std::exception {};
@@ -24,44 +21,19 @@ class DwarfReader {
         DwarfReader(const std::string& path);
 
         /** Actually read the ELF file, generating a `SimpleDwarf` output. */
-        SimpleDwarf read();
+        SimpleDwarf read() const;
 
     private: //meth
-        SimpleDwarf::Fde read_fde(const dwarf::core::Fde& fde);
+        SimpleDwarf::Fde read_fde(const dwarf::core::Fde& fde) const;
-
-        void append_results_to_fde(
-                const dwarf::core::FrameSection::instrs_results& results,
-                int ra_reg,
-                SimpleDwarf::Fde& output);
 
         SimpleDwarf::DwRegister read_register(
                 const dwarf::core::FrameSection::register_def& reg) const;
 
-        void add_cell_to_row(
-                const dwarf::core::FrameSection::register_def& reg,
-                int reg_id,
-                int ra_reg,
-                SimpleDwarf::DwRow& cur_row);
-
-        void append_row_to_fde(
-                const dwarfpp_row_t& row,
-                uintptr_t row_addr,
-                int ra_reg,
-                SimpleDwarf::Fde& output);
-
         SimpleDwarf::MachineRegister from_dwarfpp_reg(
                 int reg_id,
                 int ra_reg=-1
                 ) const;
 
-        bool is_plt_expr(
-                const dwarf::core::FrameSection::register_def& reg) const;
-
-        bool interpret_simple_expr(
-                const dwarf::core::FrameSection::register_def& reg,
-                SimpleDwarf::DwRegister& output
-                ) const;
-
         class UnsupportedRegister: public std::exception {};
 
     private:

src/EmptyFdeDeleter.cpp

@@ -1,21 +0,0 @@
-#include "EmptyFdeDeleter.hpp"
-
-#include <algorithm>
-#include <cstdio>
-
-using namespace std;
-
-EmptyFdeDeleter::EmptyFdeDeleter(bool enable): SimpleDwarfFilter(enable) {}
-
-SimpleDwarf EmptyFdeDeleter::do_apply(const SimpleDwarf& dw) const {
-    SimpleDwarf out(dw);
-
-    auto fde = out.fde_list.begin();
-    while(fde != out.fde_list.end()) {
-        if(fde->rows.empty())
-            fde = out.fde_list.erase(fde);
-        else
-            ++fde;
-    }
-    return out;
-}

src/EmptyFdeDeleter.hpp

@@ -1,15 +0,0 @@
-/** Deletes empty FDEs (that is, FDEs with no rows) from the FDEs collection.
- * This is used to ensure they do not interfere with PcHoleFiller and such. */
-
-#pragma once
-
-#include "SimpleDwarf.hpp"
-#include "SimpleDwarfFilter.hpp"
-
-class EmptyFdeDeleter: public SimpleDwarfFilter {
-    public:
-        EmptyFdeDeleter(bool enable=true);
-
-    private:
-        SimpleDwarf do_apply(const SimpleDwarf& dw) const;
-};

src/FactoredSwitchCompiler.cpp

@@ -2,7 +2,6 @@
 
 #include <sstream>
 #include <string>
-#include <iostream>
 using namespace std;
 
 FactoredSwitchCompiler::FactoredSwitchCompiler(int indent):
@@ -13,32 +12,12 @@ FactoredSwitchCompiler::FactoredSwitchCompiler(int indent):
 void FactoredSwitchCompiler::to_stream(
         std::ostream& os, const SwitchStatement& sw)
 {
-    if(sw.cases.empty()) {
-        std::cerr << "WARNING: empty unwinding data!\n";
-        os
-            << indent_str("/* WARNING: empty unwinding data! */\n")
-            << indent_str(sw.default_case) << "\n";
-        return;
-    }
     JumpPointMap jump_points;
 
-    uintptr_t low_bound = sw.cases.front().low_bound,
-              high_bound = sw.cases.back().high_bound;
-
-    os << indent() << "if("
-       << "0x" << hex << low_bound << " <= " << sw.switch_var
-       << " && " << sw.switch_var << " <= 0x" << high_bound << dec << ") {\n";
-    indent_count++;
-
     gen_binsearch_tree(os, jump_points, sw.switch_var,
-            sw.cases.begin(), sw.cases.end(),
+            sw.cases.begin(), sw.cases.end());
-            make_pair(low_bound, high_bound));
 
-    indent_count--;
+    os << indent_str(sw.default_case) << "\n"
-    os << indent() << "}\n";
-
-    os << indent() << "_factor_default:\n"
-       << indent_str(sw.default_case) << "\n"
        << indent() << "/* ===== LABELS  ============================== */\n\n";
 
     gen_jump_points_code(os, jump_points);
@@ -86,8 +65,7 @@ void FactoredSwitchCompiler::gen_binsearch_tree(
         FactoredSwitchCompiler::JumpPointMap& jump_map,
         const std::string& sw_var,
         const FactoredSwitchCompiler::case_iterator_t& begin,
-        const FactoredSwitchCompiler::case_iterator_t& end,
+        const FactoredSwitchCompiler::case_iterator_t& end)
-        const loc_range_t& loc_range)
 {
     size_t iter_delta = end - begin;
     if(iter_delta == 0)
@@ -95,19 +73,6 @@ void FactoredSwitchCompiler::gen_binsearch_tree(
     else if(iter_delta == 1) {
         FactorJumpPoint jump_point = get_jump_point(
                 jump_map, begin->content);
-        if(loc_range.first < begin->low_bound) {
-            os << indent() << "if(" << sw_var << " < 0x"
-               << hex << begin->low_bound << dec
-               << ") goto _factor_default; "
-               << "// IP=0x" << hex << loc_range.first << " ... 0x"
-               << begin->low_bound - 1 << "\n";
-        }
-        if(begin->high_bound + 1 < loc_range.second) {
-            os << indent() << "if(0x" << hex << begin->high_bound << dec
-               << " < " << sw_var << ") goto _factor_default; "
-               << "// IP=0x" << hex << begin->high_bound + 1 << " ... 0x"
-               << loc_range.second - 1 << "\n";
-        }
         os << indent() << "// IP=0x" << hex << begin->low_bound
                        << " ... 0x" << begin->high_bound << dec << "\n"
            << indent() << "goto " << jump_point << ";\n";
@@ -118,15 +83,11 @@ void FactoredSwitchCompiler::gen_binsearch_tree(
         os << indent() << "if(" << sw_var << " < 0x"
            << hex << mid->low_bound << dec << ") {\n";
         indent_count++;
-        gen_binsearch_tree(
+        gen_binsearch_tree(os, jump_map, sw_var, begin, mid);
-                os, jump_map, sw_var, begin, mid,
-                make_pair(loc_range.first, mid->low_bound));
         indent_count--;
         os << indent() << "} else {\n";
         indent_count++;
-        gen_binsearch_tree(
+        gen_binsearch_tree(os, jump_map, sw_var, mid, end);
-                os, jump_map, sw_var, mid, end,
-                make_pair(mid->low_bound, loc_range.second));
         indent_count--;
         os << indent() << "}\n";
     }

src/FactoredSwitchCompiler.hpp

@@ -24,7 +24,6 @@ class FactoredSwitchCompiler: public AbstractSwitchCompiler {
             JumpPointMap;
         typedef std::vector<SwitchStatement::SwitchCase>::const_iterator
             case_iterator_t;
-        typedef std::pair<uintptr_t, uintptr_t> loc_range_t;
 
     private:
         virtual void to_stream(std::ostream& os, const SwitchStatement& sw);
@@ -40,9 +39,7 @@ class FactoredSwitchCompiler: public AbstractSwitchCompiler {
                 JumpPointMap& jump_map,
                 const std::string& sw_var,
                 const case_iterator_t& begin,
-                const case_iterator_t& end,
+                const case_iterator_t& end);
-                const loc_range_t& loc_range // [beg, end[
-                );
 
         size_t cur_label_id;
 

src/Makefile

@@ -2,7 +2,7 @@ CXX=g++
 CXXLOCS?=-L. -I.
 CXXFL?=
 CXXFLAGS=$(CXXLOCS) -Wall -Wextra -std=c++14 -O2 -g $(CXXFL)
-CXXLIBS=-ldwarf -ldwarfpp -lsrk31c++ -lc++fileno -lelf
+CXXLIBS=-lelf -ldwarf -ldwarfpp -lsrk31c++ -lc++fileno
 
 TARGET=dwarf-assembly
 OBJS=\
@@ -12,9 +12,7 @@ OBJS=\
 	PcListReader.o \
 	SimpleDwarfFilter.o \
 	PcHoleFiller.o \
-	EmptyFdeDeleter.o \
 	ConseqEquivFilter.o \
-	OverriddenRowFilter.o \
 	SwitchStatement.o \
 	NativeSwitchCompiler.o \
 	FactoredSwitchCompiler.o \

src/OverriddenRowFilter.cpp

@@ -1,31 +0,0 @@
-#include "OverriddenRowFilter.hpp"
-
-OverriddenRowFilter::OverriddenRowFilter(bool enable)
-    : SimpleDwarfFilter(enable)
-{}
-
-SimpleDwarf OverriddenRowFilter::do_apply(const SimpleDwarf& dw) const {
-    SimpleDwarf out;
-
-    for(const auto& fde: dw.fde_list) {
-        out.fde_list.push_back(SimpleDwarf::Fde());
-        SimpleDwarf::Fde& cur_fde = out.fde_list.back();
-        cur_fde.fde_offset = fde.fde_offset;
-        cur_fde.beg_ip = fde.beg_ip;
-        cur_fde.end_ip = fde.end_ip;
-
-        if(fde.rows.empty())
-            continue;
-
-        for(size_t pos=0; pos < fde.rows.size(); ++pos) {
-            const auto& row = fde.rows[pos];
-            if(pos == fde.rows.size() - 1
-                    || row.ip != fde.rows[pos+1].ip)
-            {
-                cur_fde.rows.push_back(row);
-            }
-        }
-    }
-
-    return out;
-}

src/OverriddenRowFilter.hpp

@@ -1,15 +0,0 @@
-/** SimpleDwarfFilter to remove the first `n-1` rows of a block of `n`
- * contiguous rows that have the exact same address. */
-
-#pragma once
-
-#include "SimpleDwarf.hpp"
-#include "SimpleDwarfFilter.hpp"
-
-class OverriddenRowFilter: public SimpleDwarfFilter {
-    public:
-        OverriddenRowFilter(bool enable=true);
-
-    private:
-        SimpleDwarf do_apply(const SimpleDwarf& dw) const;
-};

src/PcHoleFiller.cpp

@@ -5,7 +5,7 @@
 
 using namespace std;
 
-PcHoleFiller::PcHoleFiller(bool enable): SimpleDwarfFilter(enable) {}
+PcHoleFiller::PcHoleFiller() {}
 
 SimpleDwarf PcHoleFiller::do_apply(const SimpleDwarf& dw) const {
     SimpleDwarf out(dw);

src/PcHoleFiller.hpp

@@ -8,7 +8,7 @@
 
 class PcHoleFiller: public SimpleDwarfFilter {
     public:
-        PcHoleFiller(bool enable=true);
+        PcHoleFiller();
 
     private:
         SimpleDwarf do_apply(const SimpleDwarf& dw) const;

src/SimpleDwarf.hpp

@@ -32,10 +32,6 @@ struct SimpleDwarf {
                              defined at some later IP in the same DIE) */
             REG_REGISTER, ///< Value of a machine register plus offset
             REG_CFA_OFFSET, ///< Value stored at some offset from CFA
-            REG_PLT_EXPR, /**< Value is the evaluation of the standard PLT
-                            expression, ie `((rip & 15) >= 11) >> 3 + rsp`
-                            This is hardcoded because it's the only expression
-                            found so far, thus worth implementing. */
             REG_NOT_IMPLEMENTED ///< This type of register is not supported
         };
 

src/SimpleDwarfFilter.cpp

@@ -1,11 +1,10 @@
 #include "SimpleDwarfFilter.hpp"
 
-SimpleDwarfFilter::SimpleDwarfFilter(bool enable): enable(enable)
+SimpleDwarfFilter::SimpleDwarfFilter()
 {}
 
 SimpleDwarf SimpleDwarfFilter::apply(const SimpleDwarf& dw) const {
-    if(!enable)
+    // For convenience of future enhancements
-        return dw;
     return do_apply(dw);
 }
 

src/SimpleDwarfFilter.hpp

@@ -9,11 +9,7 @@
 
 class SimpleDwarfFilter {
     public:
-        /** Constructor
+        SimpleDwarfFilter();
-         *
-         * @param apply set to false to disable this filter. This setting is
-         * convenient for compact filter-chaining code. */
-        SimpleDwarfFilter(bool enable=true);
 
         /// Applies the filter
         SimpleDwarf apply(const SimpleDwarf& dw) const;
@@ -23,6 +19,4 @@ class SimpleDwarfFilter {
 
     private:
         virtual SimpleDwarf do_apply(const SimpleDwarf& dw) const = 0;
-
-        bool enable;
 };

src/main.cpp

@@ -11,9 +11,7 @@
 #include "NativeSwitchCompiler.hpp"
 #include "FactoredSwitchCompiler.hpp"
 #include "PcHoleFiller.hpp"
-#include "EmptyFdeDeleter.hpp"
 #include "ConseqEquivFilter.hpp"
-#include "OverriddenRowFilter.hpp"
 
 #include "settings.hpp"
 
@@ -67,10 +65,6 @@ MainOptions options_parse(int argc, char** argv) {
         else if(option == "--enable-deref-arg") {
             settings::enable_deref_arg = true;
         }
-
-        else if(option == "--keep-holes") {
-            settings::keep_holes = true;
-        }
     }
 
     if(!seen_switch_gen_policy) {
@@ -90,7 +84,6 @@ MainOptions options_parse(int argc, char** argv) {
              << argv[0]
              << " [--switch-per-func | --global-switch]"
              << " [--enable-deref-arg]"
-             << " [--keep-holes]"
              << " [--pc-list PC_LIST_FILE] elf_path"
              << endl;
     }
@@ -105,11 +98,9 @@ int main(int argc, char** argv) {
     SimpleDwarf parsed_dwarf = DwarfReader(opts.elf_path).read();
 
     SimpleDwarf filtered_dwarf =
-        PcHoleFiller(!settings::keep_holes)(
+        PcHoleFiller()(
-        EmptyFdeDeleter()(
-        OverriddenRowFilter()(
         ConseqEquivFilter()(
-            parsed_dwarf))));
+            parsed_dwarf));
 
     FactoredSwitchCompiler* sw_compiler = new FactoredSwitchCompiler(1);
     CodeGenerator code_gen(

src/plt_std_expr.hpp

@@ -1,63 +0,0 @@
-#pragma once
-
-#include <dwarfpp/expr.hpp>
-
-static const dwarf::encap::loc_expr REFERENCE_PLT_EXPR(
-        std::vector<dwarf::encap::expr_instr> {
-            {
-               {
-                   .lr_atom  = 0x77,
-                   .lr_number = 8,
-                   .lr_number2 = 0,
-                   .lr_offset = 0
-               },
-               {
-                   .lr_atom  = 0x80,
-                   .lr_number = 0,
-                   .lr_number2 = 0,
-                   .lr_offset = 2
-               },
-               {
-                   .lr_atom  = 0x3f,
-                   .lr_number = 15,
-                   .lr_number2 = 0,
-                   .lr_offset = 4
-               },
-               {
-                   .lr_atom  = 0x1a,
-                   .lr_number = 0,
-                   .lr_number2 = 0,
-                   .lr_offset = 5
-               },
-               {
-                   .lr_atom  = 0x3b,
-                   .lr_number = 11,
-                   .lr_number2 = 0,
-                   .lr_offset = 6
-               },
-               {
-                   .lr_atom  = 0x2a,
-                   .lr_number = 0,
-                   .lr_number2 = 0,
-                   .lr_offset = 7
-               },
-               {
-                   .lr_atom  = 0x33,
-                   .lr_number = 3,
-                   .lr_number2 = 0,
-                   .lr_offset = 8
-               },
-               {
-                   .lr_atom  = 0x24,
-                   .lr_number = 0,
-                   .lr_number2 = 0,
-                   .lr_offset = 9
-               },
-               {
-                   .lr_atom  = 0x22,
-                   .lr_number = 0,
-                   .lr_number2 = 0,
-                   .lr_offset = 10
-               }
-            }
-            });

src/settings.cpp

@@ -4,5 +4,4 @@ namespace settings {
     SwitchGenerationPolicy switch_generation_policy = SGP_SwitchPerFunc;
     std::string pc_list = "";
     bool enable_deref_arg = false;
-    bool keep_holes = false;
 }

src/settings.hpp

@@ -16,6 +16,4 @@ namespace settings {
     extern SwitchGenerationPolicy switch_generation_policy;
     extern std::string pc_list;
     extern bool enable_deref_arg;
-    extern bool keep_holes; /**< Keep holes between FDEs. Larger eh_elf files,
-                              but more accurate unwinding. */
 }

stats/.gitignore

@@ -1,3 +0,0 @@
-venv
-elf_data*
-gathered

stats/README.md

@@ -1,11 +0,0 @@
-# Statistical scripts
-
-Computes stats about a whole lot of stuff.
-
-## Setup
-
-```sh
-  virtualenv -p python3 venv  # Do this only once
-  source venv/bin/activate  # Do this for every new shell working running the script
-  pip install -r requirements.txt  # Do this only once
-```

stats/fde_stats.py

@@ -1,106 +0,0 @@
-#!/usr/bin/env python3
-
-from stats_accu import StatsAccumulator
-import gather_stats
-
-import argparse
-import sys
-
-
-class Config:
-    def __init__(self):
-        args = self.parse_args()
-        self._cores = args.cores
-        self.feature = args.feature
-
-        if args.feature == 'gather':
-            self.output = args.output
-
-        elif args.feature == 'sample':
-            self.size = int(args.size)
-            self.output = args.output
-
-        elif args.feature == 'analyze':
-            self.data_file = args.data_file
-
-    @property
-    def cores(self):
-        if self._cores <= 0:
-            return None
-        return self._cores
-
-    def parse_args(self):
-        parser = argparse.ArgumentParser(
-            description="Gather statistics about system-related ELFs")
-
-        parser.add_argument('--cores', '-j', default=1, type=int,
-                            help=("Use N cores for processing. Defaults to "
-                                  "1. 0 to use up all cores."))
-
-        subparsers = parser.add_subparsers(help='Subcommands')
-
-        # Sample stats
-        parser_sample = subparsers.add_parser(
-            'sample',
-            help='Same as gather, but for a random subset of files')
-        parser_sample.set_defaults(feature='sample')
-        parser_sample.add_argument('--size', '-n',
-                                   default=1000,
-                                   help=('Pick this number of files'))
-        parser_sample.add_argument('--output', '-o',
-                                   default='elf_data',
-                                   help=('Output data to this file. Defaults '
-                                         'to "elf_data"'))
-
-        # Gather stats
-        parser_gather = subparsers.add_parser(
-            'gather',
-            help=('Gather system data into a file, to allow multiple '
-                  'analyses without re-scanning the whole system.'))
-        parser_gather.set_defaults(feature='gather')
-        parser_gather.add_argument('--output', '-o',
-                                   default='elf_data',
-                                   help=('Output data to this file. Defaults '
-                                         'to "elf_data"'))
-
-        # Analyze stats
-        parser_analyze = subparsers.add_parser(
-            'analyze',
-            help='Analyze data gathered by a previous run.')
-        parser_analyze.set_defaults(feature='analyze')
-        parser_analyze.add_argument('data_file',
-                                    default='elf_data',
-                                    help=('Analyze this data file. Defaults '
-                                          'to "elf_data".'))
-        # TODO histogram?
-
-        out = parser.parse_args()
-        if 'feature' not in out:
-            print("No subcommand specified.", file=sys.stderr)
-            parser.print_usage(file=sys.stderr)
-            sys.exit(1)
-
-        return out
-
-
-def main():
-    config = Config()
-
-    if config.feature == 'gather':
-        stats_accu = gather_stats.gather_system_files(config)
-        stats_accu.dump(config.output)
-
-    elif config.feature == 'sample':
-        stats_accu = gather_stats.gather_system_files(
-            config,
-            sample_size=config.size)
-        stats_accu.dump(config.output)
-
-    elif config.feature == 'analyze':
-        print("Not implemented", file=sys.stderr)
-        stats_accu = StatsAccumulator.load(config.data_file)
-        sys.exit(1)
-
-
-if __name__ == '__main__':
-    main()

stats/gather_stats.py

@@ -1,147 +0,0 @@
-from elftools.common.exceptions import DWARFError
-from pyelftools_overlay import system_elfs, get_cfi
-from elftools.dwarf import callframe
-import concurrent.futures
-import random
-
-
-from stats_accu import \
-    StatsAccumulator, SingleFdeData, FdeData, DwarfInstr
-
-
-class ProcessWrapper:
-    def __init__(self, fct):
-        self._fct = fct
-
-    def __call__(self, elf_descr):
-        try:
-            path, elftype = elf_descr
-
-            print("Processing {}…".format(path))
-
-            cfi = get_cfi(path)
-            if not cfi:
-                return None
-
-            return self._fct(path, elftype, cfi)
-        except DWARFError:
-            return None
-
-
-def process_wrapper(fct):
-    return ProcessWrapper(fct)
-
-
-@process_wrapper
-def process_elf(path, elftype, cfi):
-    ''' Process a single file '''
-
-    data = FdeData()
-
-    for entry in cfi:
-        if isinstance(entry, callframe.CIE):  # Is a CIE
-            process_cie(entry, data)
-        elif isinstance(entry, callframe.FDE):  # Is a FDE
-            process_fde(entry, data)
-
-    return SingleFdeData(path, elftype, data)
-
-
-def incr_cell(table, key):
-    ''' Increments table[key], or sets it to 1 if unset '''
-    if key in table:
-        table[key] += 1
-    else:
-        table[key] = 1
-
-
-def process_cie(cie, data):
-    ''' Process a CIE '''
-    pass  # Nothing needed from a CIE
-
-
-def process_fde(fde, data):
-    ''' Process a FDE '''
-    data.fde_count += 1
-
-    decoded = fde.get_decoded()
-    row_count = len(decoded.table)
-    incr_cell(data.fde_with_lines, row_count)
-
-    for row in decoded.table:
-        process_reg(data.regs.cfa, row['cfa'])
-        for entry in row:
-            if isinstance(entry, int):
-                process_reg(data.regs.regs[entry], row[entry])
-
-
-def process_reg(out_reg, reg_def):
-    ''' Process a register '''
-    if isinstance(reg_def, callframe.CFARule):
-        if reg_def.reg is not None:
-            out_reg.regs[reg_def.reg] += 1
-        else:
-            pass  # TODO exprs
-    else:
-        incr_cell(out_reg.instrs, DwarfInstr.of_pyelf(reg_def.type))
-        if reg_def.type == callframe.RegisterRule.REGISTER:
-            out_reg.regs[reg_def.arg] += 1
-        elif (reg_def.type == callframe.RegisterRule.EXPRESSION) \
-                or (reg_def.type == callframe.RegisterRule.VAL_EXPRESSION):
-            pass  # TODO exprs
-
-
-def gather_system_files(config, sample_size=None):
-    stats_accu = StatsAccumulator()
-
-    elf_list = []
-    for elf_path in system_elfs():
-        elf_list.append(elf_path)
-
-    if sample_size is not None:
-        elf_list_sampled = random.sample(elf_list, sample_size)
-        elf_list = elf_list_sampled
-
-    if config.cores > 1:
-        with concurrent.futures.ProcessPoolExecutor(max_workers=config.cores)\
-                as executor:
-            for fde in executor.map(process_elf, elf_list):
-                stats_accu.add_fde(fde)
-    else:
-        for elf in elf_list:
-            stats_accu.add_fde(process_elf(elf))
-
-    return stats_accu
-
-
-def map_system_files(mapper, sample_size=None, cores=None, include=None,
-                     elflist=None):
-    ''' `mapper` must take (path, elf_type, cfi) '''
-    if cores is None:
-        cores = 1
-    if include is None:
-        include = []
-
-    mapper = process_wrapper(mapper)
-
-    if elflist is None:
-        elf_list = []
-        for elf_path in system_elfs():
-            elf_list.append(elf_path)
-
-        if sample_size is not None:
-            elf_list_sampled = random.sample(elf_list, sample_size)
-            elf_list = elf_list_sampled
-
-        elf_list += list(map(lambda x: (x, None), include))
-    else:
-        elf_list = elflist
-
-    if cores > 1:
-        with concurrent.futures.ProcessPoolExecutor(max_workers=cores)\
-                as executor:
-            out = executor.map(mapper, elf_list)
-    else:
-        out = map(mapper, elf_list)
-
-    return out, elf_list

stats/helpers.py

@@ -1,228 +0,0 @@
-from elftools.dwarf import callframe
-import gather_stats
-import itertools
-import functools
-
-REGS_IDS = {
-    'RAX': 0,
-    'RDX': 1,
-    'RCX': 2,
-    'RBX': 3,
-    'RSI': 4,
-    'RDI': 5,
-    'RBP': 6,
-    'RSP': 7,
-    'R8':  8,
-    'R9':  9,
-    'R10': 10,
-    'R11': 11,
-    'R12': 12,
-    'R13': 13,
-    'R14': 14,
-    'R15': 15,
-    'RIP': 16
-}
-
-ID_TO_REG = [
-    'RAX',
-    'RDX',
-    'RCX',
-    'RBX',
-    'RSI',
-    'RDI',
-    'RBP',
-    'RSP',
-    'R8',
-    'R9',
-    'R10',
-    'R11',
-    'R12',
-    'R13',
-    'R14',
-    'R15',
-    'RIP',
-]
-
-HANDLED_REGS = list(map(lambda x: REGS_IDS[x], [
-    'RIP',
-    'RSP',
-    'RBP',
-    'RBX',
-]))
-
-ONLY_HANDLED_REGS = True  # only analyzed handled regs columns
-
-PLT_EXPR = [119, 8, 128, 0, 63, 26, 59, 42, 51, 36, 34]  # Handled exp
-
-
-def accumulate_regs(reg_list):
-    out = [0] * 17
-    for lst in reg_list:
-        for pos in range(len(lst)):
-            out[pos] += lst[pos]
-
-    return out
-
-
-def filter_none(lst):
-    for x in lst:
-        if x:
-            yield x
-
-
-def deco_filter_none(fct):
-    def wrap(lst):
-        return fct(filter_none(lst))
-    return wrap
-
-
-class FdeProcessor:
-    def __init__(self, fct, reducer=None):
-        self._fct = fct
-        self._reducer = reducer
-
-    def __call__(self, path, elftype, cfi):
-        out = []
-        for entry in cfi:
-            if isinstance(entry, callframe.FDE):
-                decoded = entry.get_decoded()
-                out.append(self._fct(path, entry, decoded))
-        if self._reducer is not None and len(out) >= 2:
-            out = [self._reducer(out)]
-        return out
-
-
-class FdeProcessorReduced:
-    def __init__(self, reducer):
-        self._reducer = reducer
-
-    def __call__(self, fct):
-        return FdeProcessor(fct, self._reducer)
-
-
-def fde_processor(fct):
-    return FdeProcessor(fct)
-
-
-def fde_processor_reduced(reducer):
-    return FdeProcessorReduced(reducer)
-
-
-def is_handled_expr(expr):
-    if expr == PLT_EXPR:
-        return True
-
-    if len(expr) == 2 and 0x70 <= expr[0] <= 0x89:
-        if expr[0] - 0x70 in HANDLED_REGS:
-            return True
-    return False
-
-
-# @fde_processor
-def find_non_cfa(path, fde, decoded):
-    regs_seen = 0
-    non_handled_regs = 0
-    non_handled_exp = 0
-    cfa_dat = [0, 0]  # Seen, expr
-    rule_type = {
-        callframe.RegisterRule.UNDEFINED: 0,
-        callframe.RegisterRule.SAME_VALUE: 0,
-        callframe.RegisterRule.OFFSET: 0,
-        callframe.RegisterRule.VAL_OFFSET: 0,
-        callframe.RegisterRule.REGISTER: 0,
-        callframe.RegisterRule.EXPRESSION: 0,
-        callframe.RegisterRule.VAL_EXPRESSION: 0,
-        callframe.RegisterRule.ARCHITECTURAL: 0,
-    }
-    problematic_paths = set()
-
-    for row in decoded.table:
-        for entry in row:
-            reg_def = row[entry]
-
-            if entry == 'cfa':
-                cfa_dat[0] += 1
-                if reg_def.expr:
-                    cfa_dat[1] += 1
-                    if not is_handled_expr(reg_def.expr):
-                        non_handled_exp += 1
-                        problematic_paths.add(path)
-                elif reg_def:
-                    if reg_def.reg not in HANDLED_REGS:
-                        non_handled_regs += 1
-                        problematic_paths.add(path)
-            if not isinstance(entry, int):  # CFA or PC
-                continue
-
-            if ONLY_HANDLED_REGS and entry not in HANDLED_REGS:
-                continue
-
-            rule_type[reg_def.type] += 1
-            reg_rule = reg_def.type
-
-            if reg_rule in [callframe.RegisterRule.OFFSET,
-                            callframe.RegisterRule.VAL_OFFSET]:
-                regs_seen += 1  # CFA
-            elif reg_rule == callframe.RegisterRule.REGISTER:
-                regs_seen += 1
-                if reg_def.arg not in HANDLED_REGS:
-                    problematic_paths.add(path)
-                    non_handled_regs += 1
-            elif reg_rule in [callframe.RegisterRule.EXPRESSION,
-                              callframe.RegisterRule.VAL_EXPRESSION]:
-                expr = reg_def.arg
-                if not is_handled_expr(reg_def.arg):
-                    problematic_paths.add(path)
-                    with open('/tmp/exprs', 'a') as handle:
-                        handle.write('[{} - {}] {}\n'.format(
-                            path, fde.offset,
-                            ', '.join(map(lambda x: hex(x), expr))))
-                    non_handled_exp += 1
-
-    return (regs_seen, non_handled_regs, non_handled_exp, rule_type, cfa_dat,
-            problematic_paths)
-
-
-def reduce_non_cfa(lst):
-    def merge_dict(d1, d2):
-        for x in d1:
-            d1[x] += d2[x]
-        return d1
-
-    def merge_list(l1, l2):
-        out = []
-        for pos in range(len(l1)):  # Implicit assumption len(l1) == len(l2)
-            out.append(l1[pos] + l2[pos])
-        return out
-
-    def merge_elts(accu, elt):
-        accu_regs, accu_nh, accu_exp, accu_rt, accu_cfa, accu_paths = accu
-        elt_regs, elt_nh, elt_exp, elt_rt, elt_cfa, elf_paths = elt
-        return (
-            accu_regs + elt_regs,
-            accu_nh + elt_nh,
-            accu_exp + elt_exp,
-            merge_dict(accu_rt, elt_rt),
-            merge_list(accu_cfa, elt_cfa),
-            accu_paths.union(elf_paths),
-        )
-
-    return functools.reduce(merge_elts, lst)
-
-
-@deco_filter_none
-def flatten_non_cfa(result):
-    flat = itertools.chain.from_iterable(result)
-    out = reduce_non_cfa(flat)
-    out_cfa = {
-        'seen': out[4][0],
-        'expr': out[4][1],
-        'offset': out[4][0] - out[4][1],
-    }
-    out = (out[0],
-           (out[1], out[0] + out_cfa['offset']),
-           (out[2], out[3]['EXPRESSION'] + out_cfa['expr']),
-           out[3],
-           out_cfa,
-           out[5])
-    return out

stats/pyelftools_overlay.py

@@ -1,110 +0,0 @@
-""" Overlay of PyElfTools for quick access to what we want here """
-
-from elftools.elf.elffile import ELFFile
-from elftools.common.exceptions import ELFError, DWARFError
-from stats_accu import ElfType
-import os
-
-
-ELF_BLACKLIST = [
-    '/usr/lib/libavcodec.so',
-]
-
-
-def get_cfi(path):
-    ''' Get the CFI entries from the ELF at the provided path '''
-
-    try:
-        with open(path, 'rb') as file_handle:
-            elf_file = ELFFile(file_handle)
-
-            if not elf_file.has_dwarf_info():
-                print("No DWARF")
-                return None
-
-            dw_info = elf_file.get_dwarf_info()
-            if dw_info.has_CFI():
-                cfis = dw_info.CFI_entries()
-            elif dw_info.has_EH_CFI():
-                cfis = dw_info.EH_CFI_entries()
-            else:
-                print("No CFI")
-                return None
-    except ELFError:
-        print("ELF Error")
-        return None
-    except DWARFError:
-        print("DWARF Error")
-        return None
-    except PermissionError:
-        print("Permission Error")
-        return None
-    except KeyError:
-        print("Key Error")
-        return None
-
-    return cfis
-
-
-def system_elfs():
-    ''' Iterator over system libraries '''
-
-    def readlink_rec(path):
-        if not os.path.islink(path):
-            return path
-
-        return readlink_rec(
-            os.path.join(os.path.dirname(path),
-                         os.readlink(path)))
-
-    sysbin_dirs = [
-        ('/lib', ElfType.ELF_LIB),
-        ('/usr/lib', ElfType.ELF_LIB),
-        ('/usr/local/lib', ElfType.ELF_LIB),
-        ('/bin', ElfType.ELF_BINARY),
-        ('/usr/bin', ElfType.ELF_BINARY),
-        ('/usr/local/bin', ElfType.ELF_BINARY),
-        ('/sbin', ElfType.ELF_BINARY),
-    ]
-    to_explore = sysbin_dirs
-
-    seen_elfs = set()
-
-    while to_explore:
-        bindir, elftype = to_explore.pop()
-
-        if not os.path.isdir(bindir):
-            continue
-
-        for direntry in os.scandir(bindir):
-            if not direntry.is_file():
-                if direntry.is_dir():
-                    to_explore.append((direntry.path, elftype))
-                continue
-
-            canonical_name = readlink_rec(direntry.path)
-            for blacked in ELF_BLACKLIST:
-                if canonical_name.startswith(blacked):
-                    continue
-            if canonical_name in seen_elfs:
-                continue
-
-            valid_elf = True
-            try:
-                with open(canonical_name, 'rb') as handle:
-                    magic_bytes = handle.read(4)
-                    if magic_bytes != b'\x7fELF':
-                        valid_elf = False
-                    elf_class = handle.read(1)
-                    if elf_class != b'\x02':  # ELF64
-                        valid_elf = False
-            except Exception:
-                continue
-            if not valid_elf:
-                continue
-
-            if not os.path.isfile(canonical_name):
-                continue
-
-            seen_elfs.add(canonical_name)
-            yield (canonical_name, elftype)

stats/requirements.txt

@@ -1 +0,0 @@
-git+https://github.com/eliben/pyelftools

stats/stats_accu.py

@@ -1,263 +0,0 @@
-from elftools.dwarf import callframe
-import enum
-import subprocess
-import re
-import json
-import collections
-
-from math import ceil
-
-
-class ProportionFinder:
-    ''' Finds figures such as median, etc. on the original structure of a
-    dictionnary mapping a value to its occurrence count '''
-
-    def __init__(self, count_per_value):
-        self.cumulative = []
-        prev_count = 0
-        for key in sorted(count_per_value.keys()):
-            n_count = prev_count + count_per_value[key]
-            self.cumulative.append(
-                (key, n_count))
-            prev_count = n_count
-
-        self.elem_count = prev_count
-
-    def find_at_proportion(self, proportion):
-        if not self.cumulative:  # Empty list
-            return None
-
-        low_bound = ceil(self.elem_count * proportion)
-
-        def binsearch(beg, end):
-            med = ceil((beg + end) / 2)
-
-            if beg + 1 == end:
-                return self.cumulative[beg][0]
-
-            if self.cumulative[med - 1][1] < low_bound:
-                return binsearch(med, end)
-            return binsearch(beg, med)
-
-        return binsearch(0, len(self.cumulative))
-
-
-def elf_so_deps(path):
-    ''' Get the list of shared objects dependencies of the given ELF object.
-    This is obtained by running `ldd`. '''
-
-    deps_list = []
-
-    try:
-        ldd_output = subprocess.check_output(['/usr/bin/ldd', path]) \
-            .decode('utf-8')
-        ldd_re = re.compile(r'^.* => (.*) \(0x[0-9a-fA-F]*\)$')
-
-        ldd_lines = ldd_output.strip().split('\n')
-        for line in ldd_lines:
-            line = line.strip()
-            match = ldd_re.match(line)
-            if match is None:
-                continue  # Just ignore that line — it might be eg. linux-vdso
-            deps_list.append(match.group(1))
-
-        return deps_list
-
-    except subprocess.CalledProcessError as exn:
-        raise Exception(
-            ("Cannot get dependencies for {}: ldd terminated with exit code "
-             "{}.").format(path, exn.returncode))
-
-
-class ElfType(enum.Enum):
-    ELF_LIB = enum.auto()
-    ELF_BINARY = enum.auto()
-
-
-class DwarfInstr(enum.Enum):
-    @staticmethod
-    def of_pyelf(val):
-        _table = {
-            callframe.RegisterRule.UNDEFINED: DwarfInstr.INSTR_UNDEF,
-            callframe.RegisterRule.SAME_VALUE: DwarfInstr.INSTR_SAME_VALUE,
-            callframe.RegisterRule.OFFSET: DwarfInstr.INSTR_OFFSET,
-            callframe.RegisterRule.VAL_OFFSET: DwarfInstr.INSTR_VAL_OFFSET,
-            callframe.RegisterRule.REGISTER: DwarfInstr.INSTR_REGISTER,
-            callframe.RegisterRule.EXPRESSION: DwarfInstr.INSTR_EXPRESSION,
-            callframe.RegisterRule.VAL_EXPRESSION:
-                DwarfInstr.INSTR_VAL_EXPRESSION,
-            callframe.RegisterRule.ARCHITECTURAL:
-                DwarfInstr.INSTR_ARCHITECTURAL,
-        }
-        return _table[val]
-
-    INSTR_UNDEF = enum.auto()
-    INSTR_SAME_VALUE = enum.auto()
-    INSTR_OFFSET = enum.auto()
-    INSTR_VAL_OFFSET = enum.auto()
-    INSTR_REGISTER = enum.auto()
-    INSTR_EXPRESSION = enum.auto()
-    INSTR_VAL_EXPRESSION = enum.auto()
-    INSTR_ARCHITECTURAL = enum.auto()
-
-
-def intify_dict(d):
-    out = {}
-    for key in d:
-        try:
-            nKey = int(key)
-        except Exception:
-            nKey = key
-
-        try:
-            out[nKey] = int(d[key])
-        except ValueError:
-            out[nKey] = d[key]
-    return out
-
-
-class RegData:
-    def __init__(self, instrs=None, regs=None, exprs=None):
-        if instrs is None:
-            instrs = {}
-        if regs is None:
-            regs = [0]*17
-        if exprs is None:
-            exprs = {}
-        self.instrs = intify_dict(instrs)
-        self.regs = regs
-        self.exprs = intify_dict(exprs)
-
-    @staticmethod
-    def map_dict_keys(fnc, dic):
-        out = {}
-        for key in dic:
-            out[fnc(key)] = dic[key]
-        return out
-
-    def dump(self):
-        return {
-            'instrs': RegData.map_dict_keys(lambda x: x.value, self.instrs),
-            'regs': self.regs,
-            'exprs': self.exprs,
-        }
-
-    @staticmethod
-    def load(data):
-        return RegData(
-            instrs=RegData.map_dict_keys(
-                lambda x: DwarfInstr(int(x)),
-                data['instrs']),
-            regs=data['regs'],
-            exprs=data['exprs'],
-        )
-
-
-class RegsList:
-    def __init__(self, cfa=None, regs=None):
-        if cfa is None:
-            cfa = RegsList.fresh_reg()
-        if regs is None:
-            regs = [RegsList.fresh_reg() for _ in range(17)]
-        self.cfa = cfa
-        self.regs = regs
-
-    @staticmethod
-    def fresh_reg():
-        return RegData()
-
-    def dump(self):
-        return {
-            'cfa': RegData.dump(self.cfa),
-            'regs': [RegData.dump(r) for r in self.regs],
-        }
-
-    @staticmethod
-    def load(data):
-        return RegsList(
-            cfa=RegData.load(data['cfa']),
-            regs=[RegData.load(r) for r in data['regs']],
-        )
-
-
-class FdeData:
-    def __init__(self, fde_count=0, fde_with_lines=None, regs=None):
-        if fde_with_lines is None:
-            fde_with_lines = {}
-        if regs is None:
-            regs = RegsList()
-
-        self.fde_count = fde_count
-        self.fde_with_lines = intify_dict(fde_with_lines)
-        self.regs = regs
-
-    def dump(self):
-        return {
-            'fde_count': self.fde_count,
-            'fde_with_lines': self.fde_with_lines,
-            'regs': self.regs.dump(),
-        }
-
-    @staticmethod
-    def load(data):
-        return FdeData(
-            fde_count=int(data['fde_count']),
-            fde_with_lines=data['fde_with_lines'],
-            regs=RegsList.load(data['regs']))
-
-
-class SingleFdeData:
-    def __init__(self, path, elf_type, data):
-        self.path = path
-        self.elf_type = elf_type
-        self.data = data  # < of type FdeData
-
-        self.gather_deps()
-
-    def gather_deps(self):
-        """ Collect ldd data on the binary """
-        # self.deps = elf_so_deps(self.path)
-        self.deps = []
-
-    def dump(self):
-        return {
-            'path': self.path,
-            'elf_type': self.elf_type.value,
-            'data': self.data.dump()
-        }
-
-    @staticmethod
-    def load(data):
-        return SingleFdeData(
-            data['path'],
-            ElfType(int(data['elf_type'])),
-            FdeData.load(data['data']))
-
-
-class StatsAccumulator:
-    def __init__(self):
-        self.fdes = []
-
-    def add_fde(self, fde_data):
-        if fde_data:
-            self.fdes.append(fde_data)
-
-    def get_fdes(self):
-        return self.fdes
-
-    def add_stats_accu(self, stats_accu):
-        for fde in stats_accu.get_fdes():
-            self.add_fde(fde)
-
-    def dump(self, path):
-        dict_form = [fde.dump() for fde in self.fdes]
-        with open(path, 'w') as handle:
-            handle.write(json.dumps(dict_form))
-
-    @staticmethod
-    def load(path):
-        with open(path, 'r') as handle:
-            text = handle.read()
-        out = StatsAccumulator()
-        out.fdes = [SingleFdeData.load(data) for data in json.loads(text)]
-        return out