DwAsm: check global bounds

Before, the space between FDEs was abstracted away, thought as dead
space that produced an error when queried. This is not the case, though:
empty FDEs indicate undefined DWARF

README.md

@@ -1,10 +1,9 @@
 # Dwarf Assembly
 
-Some experiments around compiling the most used Dwarf informations (ELF debug
-data) directly into assembly.
+A compiler from DWARF unwinding data to native x86\_64 binaries.
 
-This project is a big work in progress, don't expect anything to be stable for
-now.
+This repository also contains various experiments, tools, benchmarking scripts,
+stats scripts, etc. to work on this compiler.
 
 ## Dependencies
 
@@ -17,7 +16,8 @@ As of now, this project relies on the following libraries:
     - [libsrk31cxx](https://github.com/stephenrkell/libsrk31cxx)
 
 These libraries are expected to be installed somewhere your compiler can find
-them.
+them. If you are using Archlinux, you can check
+[these `PKGBUILD`s](https://git.tobast.fr/m2-internship/pkgbuilds).
 
 ## Scripts and directories
 
@@ -26,4 +26,40 @@ them.
 * `./compare_sizes.py`: compare the sizes of the `.eh_frame` of a binary (and
   its dependencies) with the sizes of the `.text` of the generated ELFs.
 * `./extract_pc.py`: extracts a list of valid program counters of an ELF and
-  produce a file as read by `dwarf-assembly`
+  produce a file as read by `dwarf-assembly`, **deprecated**.
+
+* `benching`: all about benchmarking
+* `env`: environment variables manager to ease the use of various `eh_elf`s in
+  parallel, for experiments.
+* `shared`: code shared between various subprojects
+* `src`: the compiler code itself
+* `stack_walker`: a primitive stack walker using `eh_elf`s
+* `stack_walker_libunwind`: a primitive stack walker using vanilla `libunwind`
+* `stats`: a statistics gathering module
+* `tests`: some tests regarding `eh_elf`s, **deprecated**.
+
+## How to use
+
+To compile `eh_elf`s for a given ELF file, say `foo.bin`, it is recommended to
+use `generate_eh_elf.py`. Help can be obtained with `--help`. A standard
+command is
+
+```bash
+./generate_eh_elf.py --deps --enable-deref-arg --global-switch -o eh_elfs foo.bin
+```
+
+This will compile `foo.bin` and all the shared objects it relies on into
+`eh_elf`s, in the directory `./eh_elfs`, using a dereferencing argument (which
+is necessary for `perf-eh_elfs`).
+
+## Generate the intermediary C file
+
+If you're curious about the intermediary C file generated for a given ELF file
+`foo.bin`, you must call `dwarf-assembly` directly. A parameter `--help` can be
+passed; a standard command is
+
+```bash
+./dwarf-assembly --global-switch --enable-deref-arg foo.bin
+```
+
+**Beware**! This will generate the C code on the standard output.

benching/README.md

@@ -0,0 +1,92 @@
+# Benching `eh_elfs`
+
+## Benchmark setup
+
+Pick some name for your `eh_elfs` directory. We will call it `$EH_ELF_DIR`.
+
+### Generate the `eh_elfs`
+
+```bash
+../../generate_eh_elf.py --deps -o "$EH_ELF_DIR" \
+  --keep-holes -O2 --global-switch --enable-deref-arg "$BENCHED_BINARY"
+```
+
+### Record a `perf` session
+
+```bash
+perf record --call-graph dwarf,4096 "$BENCHED_BINARY" [args]
+```
+
+### Set up the environment
+
+```bash
+source ../../env/apply [vanilla | vanilla-nocache | *eh_elf] [dbg | *release]
+```
+
+The first value selects the version of libunwind you will be running, the
+second selects whether you want to run in debug or release mode (use release to
+get readings, debug to check for errors).
+
+You can reset your environment to its previous state by running `deactivate`.
+
+If you pick the `eh_elf` flavour, you will also have to
+
+```bash
+export LD_LIBRARY_PATH="$EH_ELF_DIR:$LD_LIBRARY_PATH"
+```
+
+## Extract results
+
+### Base readings
+
+**In release mode** (faster), run
+
+```bash
+perf report 2>&1 >/dev/null
+```
+
+with both `eh_elf` and `vanilla` shells. Compare average time.
+
+### Getting debug output
+
+```bash
+UNW_DEBUG_LEVEL=5 perf report 2>&1 >/dev/null
+```
+
+### Total number of calls to `unw_step`
+
+```bash
+UNW_DEBUG_LEVEL=5 perf report 2>&1 >/dev/null | grep -c "step:.* returning"
+```
+
+### Total number of vanilla errors
+
+With the `vanilla` context,
+
+```bash
+UNW_DEBUG_LEVEL=5 perf report 2>&1 >/dev/null | grep -c "step:.* returning -"
+```
+
+### Total number of fallbacks to original DWARF
+
+With the `eh_elf` context,
+
+```bash
+UNW_DEBUG_LEVEL=5 perf report 2>&1 >/dev/null | grep -c "step:.* falling back"
+```
+
+### Total number of fallbacks to original DWARF that actually used DWARF
+
+With the `eh_elf` context,
+
+```bash
+UNW_DEBUG_LEVEL=5 perf report 2>&1 >/dev/null | grep -c "step:.* fallback with"
+```
+
+### Get succeeded fallback locations
+
+```bash
+UNW_DEBUG_LEVEL=5 perf report 2>&1 >/dev/null \
+  | grep "step: .* fallback with" -B 15 \
+  | grep "In memory map" | sort | uniq -c
+```

benching/csmith/.gitignore

@@ -0,0 +1 @@
+/tests

benching/gzip/.gitignore

@@ -0,0 +1,3 @@
+gzip
+gzip-1.10
+perf.data

benching/gzip/EVALUATION.md

@@ -0,0 +1,49 @@
+# gzip - evaluation
+
+Artifacts saved in `evaluation_artifacts`.
+
+## Performance
+
+Using the command line
+
+```bash
+for i in $(seq 1 100); do
+  perf report 2>&1 >/dev/null | tail -n 1 \
+    | python ../hackbench/to_report_fmt.py \
+    | sed 's/^.* & .* & \([0-9]*\) & .*$/\1/g'
+done
+```
+
+we save a sequence of 100 performance readings to some file.
+
+Samples:
+* `eh_elf`:  331134 unw/exec
+* `vanilla`: 331144 unw/exec
+
+Average time/unw:
+* `eh_elf`:    83 ns
+* `vanilla`: 1304 ns
+
+Standard deviation:
+* `eh_elf`:   2 ns
+* `vanilla`: 24 ns
+
+Average ratio: 15.7
+Ratio uncertainty: 0.8
+
+## Distibution of `unw_step` issues
+
+### `eh_elf` case
+
+* success:                              331134 (99.9%)
+* fallback to DWARF:                         2  (0.0%)
+* fallback to libunwind heuristics:          8  (0.0%)
+* fail to unwind:                          379  (0.1%)
+* total:                                331523
+
+### `vanilla` case
+
+* success:                              331136 (99.9%)
+* fallback to libunwind heuristics:          8  (0.0%)
+* fail to unwind:                          379  (0.1%)
+* total:                                331523

benching/hackbench/.gitignore

@@ -0,0 +1,5 @@
+/eh_elfs*
+/bench*
+/perf.data*
+/perfperf.data*
+/hackbench

benching/hackbench/EVALUATION.md

@@ -0,0 +1,48 @@
+# Hackbench - evaluation
+
+Artifacts saved in `evaluation_artifacts`.
+
+## Performance
+
+Using the command line
+
+```bash
+for i in $(seq 1 100); do
+  perf report 2>&1 >/dev/null | tail -n 1 \
+    | python to_report_fmt.py | sed 's/^.* & .* & \([0-9]*\) & .*$/\1/g'
+done
+```
+
+we save a sequence of 100 performance readings to some file.
+
+Samples:
+* `eh_elf`:  135251 unw/exec
+* `vanilla`: 138233 unw/exec
+
+Average time/unw:
+* `eh_elf`:   102 ns
+* `vanilla`: 2443 ns
+
+Standard deviation:
+* `eh_elf`:  2 ns
+* `vanilla`: 47 ns
+
+Average ratio: 24
+Ratio uncertainty: 1.0
+
+## Distibution of `unw_step` issues
+
+### `eh_elf` case
+
+* success:                              135251 (97.7%)
+* fallback to DWARF:                      1467  (1.0%)
+* fallback to libunwind heuristics:        329  (0.2%)
+* fail to unwind:                         1410  (1.0%)
+* total:                                138457
+
+### `vanilla` case
+
+* success:                              138201 (98.9%)
+* fallback to libunwind heuristics:         32  (0.0%)
+* fail to unwind:                         1411  (1.0%)
+* total:                                139644

benching/hackbench/README.md

@@ -0,0 +1,44 @@
+# Running the benchmarks
+
+Pick some name for your `eh_elfs` directory. We will call it `$EH_ELF_DIR`.
+
+## Generate the `eh_elfs`
+
+```bash
+../../generate_eh_elf.py --deps -o "$EH_ELF_DIR" \
+  --keep-holes -O2 --global-switch --enable-deref-arg hackbench
+```
+
+## Record a `perf` session
+
+```bash
+perf record --call-graph dwarf,4096 ./hackbench 10 process 100
+```
+
+You can arbitrarily increase the first number up to ~100 and the second to get
+a longer session. This will most probably take all your computer's resources
+while it is running.
+
+## Set up the environment
+
+```bash
+source ../../env/apply [vanilla | vanilla-nocache | *eh_elf] [dbg | *release]
+```
+
+The first value selects the version of libunwind you will be running, the
+second selects whether you want to run in debug or release mode (use release to
+get readings, debug to check for errors).
+
+You can reset your environment to its previous state by running `deactivate`.
+
+If you pick the `eh_elf` flavour, you will also have to
+
+```bash
+export LD_LIBRARY_PATH="$EH_ELF_DIR:$LD_LIBRARY_PATH"
+```
+
+### Actually get readings
+
+```bash
+perf report 2>&1 >/dev/null
+```

benching/hackbench/to_report_fmt.py

@@ -0,0 +1,21 @@
+#!/usr/bin/env python3
+
+import re
+import sys
+
+line = input()
+regex = \
+    re.compile(r'Total unwind time: ([0-9]*) s ([0-9]*) ns, ([0-9]*) calls')
+
+match = regex.match(line.strip())
+if not match:
+    print('Badly formatted line', file=sys.stderr)
+    sys.exit(1)
+
+sec = int(match.group(1))
+ns = int(match.group(2))
+calls = int(match.group(3))
+
+time = sec * 10**9 + ns
+
+print("{} & {} & {} & ??".format(calls, time, time // calls))

benching/python3.7/test.py

@@ -0,0 +1,8 @@
+def slow_fibo(n):
+    if n <= 1:
+        return 1
+    return slow_fibo(n - 1) + slow_fibo(n - 2)
+
+
+if __name__ == "__main__":
+    slow_fibo(35)

benching/tools/common.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+
+if [ "$#" -lt 1 ] ; then
+    >&2 echo "Missing argument: directory"
+    exit 1
+fi
+
+BENCH_DIR="$(echo $1 | sed 's@/$@@g')"
+ENV_APPLY="$(readlink -f "$(dirname $0)/../../env/apply")"
+
+if ! [ -f "$ENV_APPLY" ] ; then
+    >&2 echo "Cannot find helper scripts. Abort."
+    exit 1
+fi
+
+function status_report {
+    echo -e "\e[33;1m[$BENCH_DIR]\e[0m $1"
+}

benching/tools/errors.sh

@@ -0,0 +1,101 @@
+#!/bin/bash
+
+source "$(dirname $0)/common.sh"
+
+TMP_FILE=$(mktemp)
+if [ -z "$EH_ELFS_NAME" ]; then
+    EH_ELFS_NAME="eh_elfs"
+fi
+
+function get_perf_output {
+    envtype=$1
+    source $ENV_APPLY "$envtype" "dbg"
+    LD_LIBRARY_PATH="$BENCH_DIR/$EH_ELFS_NAME:$LD_LIBRARY_PATH" \
+        UNW_DEBUG_LEVEL=15 \
+        perf report -i "$BENCH_DIR/perf.data" 2>$TMP_FILE >/dev/null
+    deactivate
+}
+
+function count_successes {
+    cat $TMP_FILE | tail -n 1 | sed 's/^.*, \([0-9]*\) calls.*$/\1/g'
+}
+
+function count_total_calls {
+    cat $TMP_FILE | grep -c "^  >.*step:.* returning"
+}
+
+function count_errors {
+    cat $TMP_FILE | grep -c "^  >.*step:.* returning -"
+}
+
+function count_eh_fallbacks {
+    cat $TMP_FILE | grep -c "step:.* falling back"
+}
+
+function count_vanilla_fallbacks {
+    cat $TMP_FILE | grep -c "step:.* frame-chain"
+}
+
+function count_fallbacks_to_dwarf {
+    cat $TMP_FILE | grep -c "step:.* fallback with"
+}
+
+function count_fallbacks_failed {
+    cat $TMP_FILE | grep -c "step:.* dwarf_step also failed"
+}
+
+function count_fail_after_fallback_to_dwarf {
+    cat $TMP_FILE \
+        | "$(dirname $0)/line_patterns.py" \
+            "fallback with" \
+            "step:.* unw_step called" \
+            ~"step:.* unw_step called" \
+            "step:.* returning -" \
+        | grep Complete -c
+}
+
+function report {
+    flavour="$1"
+
+    status_report "$flavour issues distribution"
+
+    successes=$(count_successes)
+    failures=$(count_errors)
+    total=$(count_total_calls)
+
+    if [ "$flavour" = "eh_elf" ]; then
+        fallbacks=$(count_eh_fallbacks)
+        fallbacks_to_dwarf=$(count_fallbacks_to_dwarf)
+        fallbacks_to_dwarf_failed_after=$(count_fail_after_fallback_to_dwarf)
+        fallbacks_failed=$(count_fallbacks_failed)
+        fallbacks_to_heuristics="$(( $fallbacks \
+            - $fallbacks_to_dwarf \
+            - $fallbacks_failed))"
+        echo -e "* success:\t\t\t\t$successes"
+        echo -e "* fallback to DWARF:\t\t\t$fallbacks_to_dwarf"
+        echo -e "* …of which failed at next step:\t$fallbacks_to_dwarf_failed_after"
+        echo -e "* fallback to libunwind heuristics:\t$fallbacks_to_heuristics"
+        computed_sum=$(( $successes + $fallbacks - $fallbacks_failed + $failures ))
+    else
+        fallbacks=$(count_vanilla_fallbacks)
+        successes=$(( $successes - $fallbacks ))
+        echo -e "* success:\t\t\t\t$successes"
+        echo -e "* fallback to libunwind heuristics:\t$fallbacks"
+        computed_sum=$(( $successes + $fallbacks + $failures ))
+    fi
+    echo -e "* fail to unwind:\t\t\t$failures"
+    echo -e "* total:\t\t\t\t$total"
+    if [ "$computed_sum" -ne "$total" ] ; then
+        echo "-- WARNING: missing cases (computed sum $computed_sum != $total) --"
+    fi
+}
+
+# eh_elf stats
+get_perf_output "eh_elf"
+report "eh_elf"
+
+# Vanilla stats
+get_perf_output "vanilla"
+report "vanilla"
+
+rm "$TMP_FILE"

benching/tools/errors_new.sh

@@ -0,0 +1,86 @@
+#!/bin/bash
+
+source "$(dirname $0)/common.sh"
+
+TMP_FILE=$(mktemp)
+
+function get_perf_output {
+    envtype=$1
+    source $ENV_APPLY "$envtype" "dbg"
+    LD_LIBRARY_PATH="$BENCH_DIR/eh_elfs:$LD_LIBRARY_PATH" \
+        UNW_DEBUG_LEVEL=15 \
+        perf report -i "$BENCH_DIR/perf.data" 2>$TMP_FILE >/dev/null
+    deactivate
+}
+
+function count_successes {
+    cat $TMP_FILE | tail -n 1 | sed 's/^.*, \([0-9]*\) calls.*$/\1/g'
+}
+
+function count_total_calls {
+    cat $TMP_FILE | grep -c "^  >.*step:.* returning"
+}
+
+function count_errors {
+    cat $TMP_FILE | grep -c "^  >.*step:.* returning -"
+}
+
+function count_eh_fallbacks {
+    cat $TMP_FILE | grep -c "step:.* falling back"
+}
+
+function count_vanilla_fallbacks {
+    cat $TMP_FILE | grep -c "step:.* frame-chain"
+}
+
+function count_fallbacks_to_dwarf {
+    cat $TMP_FILE | grep -c "step:.* fallback with"
+}
+
+function count_fallbacks_failed {
+    cat $TMP_FILE | grep -c "step:.* dwarf_step also failed"
+}
+
+function report {
+    flavour="$1"
+
+    status_report "$flavour issues distribution"
+
+    successes=$(count_successes)
+    failures=$(count_errors)
+    total=$(count_total_calls)
+
+    if [ "$flavour" = "eh_elf" ]; then
+        fallbacks=$(count_eh_fallbacks)
+        fallbacks_to_dwarf=$(count_fallbacks_to_dwarf)
+        fallbacks_failed=$(count_fallbacks_failed)
+        fallbacks_to_heuristics="$(( $fallbacks \
+            - $fallbacks_to_dwarf \
+            - $fallbacks_failed))"
+        echo -e "* success:\t\t\t\t$successes"
+        echo -e "* fallback to DWARF:\t\t\t$fallbacks_to_dwarf"
+        echo -e "* fallback to libunwind heuristics:\t$fallbacks_to_heuristics"
+        computed_sum=$(( $successes + $fallbacks - $fallbacks_failed + $failures ))
+    else
+        fallbacks=$(count_vanilla_fallbacks)
+        successes=$(( $successes - $fallbacks ))
+        echo -e "* success:\t\t\t\t$successes"
+        echo -e "* fallback to libunwind heuristics:\t$fallbacks"
+        computed_sum=$(( $successes + $fallbacks + $failures ))
+    fi
+    echo -e "* fail to unwind:\t\t\t$failures"
+    echo -e "* total:\t\t\t\t$total"
+    if [ "$computed_sum" -ne "$total" ] ; then
+        echo "-- WARNING: missing cases (computed sum $computed_sum != $total) --"
+    fi
+}
+
+# eh_elf stats
+get_perf_output "eh_elf"
+report "eh_elf"
+
+# Vanilla stats
+get_perf_output "vanilla"
+report "vanilla"
+
+rm "$TMP_FILE"

benching/tools/gen_evals.sh

@@ -0,0 +1,27 @@
+OUTPUT="$1"
+NB_ITER=10
+
+if [ "$#" -lt 1 ] ; then
+    >&2 echo "Missing argument: output directory."
+    exit 1
+fi
+
+if [ -z "$EH_ELFS" ]; then
+    >&2 echo "Missing environment: EH_ELFS. Aborting."
+    exit 1
+fi
+
+mkdir -p "$OUTPUT"
+
+for flavour in 'eh_elf' 'vanilla' 'vanilla-nocache'; do
+    >&2 echo "$flavour..."
+    source "$(dirname "$0")/../../env/apply" "$flavour" release
+    for iter in $(seq 1 $NB_ITER); do
+        >&2 echo -e "\t$iter..."
+        LD_LIBRARY_PATH="$EH_ELFS:$LD_LIBRARY_PATH" \
+            perf report 2>&1 >/dev/null | tail -n 1 \
+            | python "$(dirname $0)/to_report_fmt.py" \
+            | sed 's/^.* & .* & \([0-9]*\) & .*$/\1/g'
+    done > "$OUTPUT/${flavour}_times"
+    deactivate
+done

benching/tools/gen_perf_stats.py

@@ -0,0 +1,106 @@
+#!/usr/bin/env python3
+
+""" Generates performance statistics for the eh_elf vs vanilla libunwind unwinding,
+based on time series generated beforehand
+
+Intended to be run from `statistics.sh`
+"""
+
+from collections import namedtuple
+import numpy as np
+import sys
+import os
+
+
+Datapoint = namedtuple("Datapoint", ["nb_frames", "total_time", "avg_time"])
+
+
+def read_series(path):
+    with open(path, "r") as handle:
+        for line in handle:
+            nb_frames, total_time, avg_time = map(int, line.strip().split())
+            yield Datapoint(nb_frames, total_time, avg_time)
+
+
+FLAVOURS = ["eh_elf", "vanilla"]
+WITH_NOCACHE = False
+
+if "WITH_NOCACHE" in os.environ:
+    WITH_NOCACHE = True
+    FLAVOURS.append("vanilla-nocache")
+
+path_format = os.path.join(sys.argv[1], "{}_times")
+datapoints = {}
+avg_times = {}
+total_times = {}
+avgs_total = {}
+avgs = {}
+std_deviations = {}
+unwound_frames = {}
+
+for flv in FLAVOURS:
+    datapoints[flv] = list(read_series(path_format.format(flv)))
+    avg_times[flv] = list(map(lambda x: x.avg_time, datapoints[flv]))
+    total_times[flv] = list(map(lambda x: x.total_time, datapoints[flv]))
+    avgs[flv] = sum(avg_times[flv]) / len(avg_times[flv])
+    avgs_total[flv] = sum(total_times[flv]) / len(total_times[flv])
+    std_deviations[flv] = np.sqrt(np.var(avg_times[flv]))
+
+    cur_unwound_frames = list(map(lambda x: x.nb_frames, datapoints[flv]))
+    unwound_frames[flv] = cur_unwound_frames[0]
+    for run_id, unw_frames in enumerate(cur_unwound_frames[1:]):
+        if unw_frames != unwound_frames[flv]:
+            print(
+                "{}, run {}: unwound {} frames, reference unwound {}".format(
+                    flv, run_id + 1, unw_frames, unwound_frames[flv]
+                ),
+                file=sys.stderr,
+            )
+
+avg_ratio = avgs["vanilla"] / avgs["eh_elf"]
+ratio_uncertainty = (
+    1
+    / avgs["eh_elf"]
+    * (
+        std_deviations["vanilla"]
+        + avgs["vanilla"] / avgs["eh_elf"] * std_deviations["eh_elf"]
+    )
+)
+
+
+def format_flv(flv_dict, formatter, alterator=None):
+    out = ""
+    for flv in FLAVOURS:
+        val = flv_dict[flv]
+        altered = alterator(val) if alterator else val
+        out += "* {}: {}\n".format(flv, formatter.format(altered))
+    return out
+
+
+def get_ratios(avgs):
+    def avg_of(flavour):
+        return avgs[flavour] / avgs["eh_elf"]
+
+    if WITH_NOCACHE:
+        return "\n\tcached: {}\n\tuncached: {}".format(
+            avg_of("vanilla"), avg_of("vanilla-nocache")
+        )
+    else:
+        return avg_of("vanilla")
+
+
+print(
+    "Unwound frames:\n{}\n"
+    "Average whole unwinding time (one run):\n{}\n"
+    "Average time to unwind one frame:\n{}\n"
+    "Standard deviation:\n{}\n"
+    "Average ratio: {}\n"
+    "Ratio uncertainty: {}".format(
+        format_flv(unwound_frames, "{}"),
+        format_flv(avgs_total, "{} μs", alterator=lambda x: x // 1000),
+        format_flv(avgs, "{} ns"),
+        format_flv(std_deviations, "{}"),
+        get_ratios(avgs),
+        ratio_uncertainty,
+    )
+)

benching/tools/line_patterns.py

@@ -0,0 +1,69 @@
+#!/usr/bin/env python3
+
+import sys
+import re
+
+
+class Match:
+    def __init__(self, re_str, negate=False):
+        self.re = re.compile(re_str)
+        self.negate = negate
+
+    def matches(self, line):
+        return self.re.search(line) is not None
+
+
+class Matcher:
+    def __init__(self, match_objs):
+        self.match_objs = match_objs
+        self.match_pos = 0
+        self.matches = 0
+
+        if not self.match_objs:
+            raise Exception("No match expressions provided")
+        if self.match_objs[-1].negate:
+            raise Exception("The last match object must be a positive expression")
+
+    def feed(self, line):
+        for cur_pos, exp in enumerate(self.match_objs[self.match_pos :]):
+            cur_pos = cur_pos + self.match_pos
+            if not exp.negate:  # Stops the for here, whether matching or not
+                if exp.matches(line):
+                    self.match_pos = cur_pos + 1
+                    print(
+                        "Passing positive {}, advance to {}".format(
+                            cur_pos, self.match_pos
+                        )
+                    )
+                    if self.match_pos >= len(self.match_objs):
+                        print("> Complete match, reset.")
+                        self.matches += 1
+                        self.match_pos = 0
+                return
+            else:
+                if exp.matches(line):
+                    print("Failing negative [{}] {}, reset".format(exp.negate, cur_pos))
+                    old_match_pos = self.match_pos
+                    self.match_pos = 0
+                    if old_match_pos != 0:
+                        print("> Refeed: ", end="")
+                        self.feed(line)
+                    return
+
+
+def get_args(args):
+    out_args = []
+    for arg in args:
+        negate = False
+        if arg[0] == "~":
+            negate = True
+            arg = arg[1:]
+        out_args.append(Match(arg, negate))
+    return out_args
+
+
+if __name__ == "__main__":
+    matcher = Matcher(get_args(sys.argv[1:]))
+    for line in sys.stdin:
+        matcher.feed(line)
+    print(matcher.matches)

benching/tools/statistics.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+source "$(dirname $0)/common.sh"
+
+TEMP_DIR="$(mktemp -d)"
+NB_RUNS=10
+
+function collect_perf_time_data {
+    envtype=$1
+    source $ENV_APPLY "$envtype" "release"
+    LD_LIBRARY_PATH="$BENCH_DIR/eh_elfs:$LD_LIBRARY_PATH" \
+        perf report -i "$BENCH_DIR/perf.data" 2>&1 >/dev/null \
+        | tail -n 1 \
+        | python "$(dirname $0)/to_report_fmt.py" \
+        | sed 's/^\([0-9]*\) & \([0-9]*\) & \([0-9]*\) & .*$/\1 \2 \3/g'
+    deactivate
+}
+
+function collect_perf_time_data_runs {
+    envtype=$1
+    outfile=$2
+    status_report "Collecting $envtype data over $NB_RUNS runs"
+    rm -f "$outfile"
+    for run in $(seq 1 $NB_RUNS); do
+        collect_perf_time_data "$envtype" >> "$outfile"
+    done
+}
+
+eh_elf_data="$TEMP_DIR/eh_elf_times"
+vanilla_data="$TEMP_DIR/vanilla_times"
+
+collect_perf_time_data_runs "eh_elf" "$eh_elf_data"
+collect_perf_time_data_runs "vanilla" "$vanilla_data"
+
+if [ -n "$WITH_NOCACHE" ]; then
+    vanilla_nocache_data="$TEMP_DIR/vanilla-nocache_times"
+    collect_perf_time_data_runs "vanilla-nocache" "$vanilla_nocache_data"
+fi
+
+status_report "benchmark statistics"
+python "$(dirname "$0")/gen_perf_stats.py" "$TEMP_DIR"
+
+rm -rf "$TEMP_DIR"

benching/tools/to_report_fmt.py

@@ -0,0 +1,21 @@
+#!/usr/bin/env python3
+
+import re
+import sys
+
+line = input()
+regex = \
+    re.compile(r'Total unwind time: ([0-9]*) s ([0-9]*) ns, ([0-9]*) calls')
+
+match = regex.match(line.strip())
+if not match:
+    print('Badly formatted line', file=sys.stderr)
+    sys.exit(1)
+
+sec = int(match.group(1))
+ns = int(match.group(2))
+calls = int(match.group(3))
+
+time = sec * 10**9 + ns
+
+print("{} & {} & {} & ??".format(calls, time, time // calls))

env/apply

@@ -2,24 +2,6 @@
 ## Source this file.
 ## Usage: apply [vanilla | vanilla-nocache | *eh_elf] [dbg | *release]
 
-# ==== DEFINE DEACTIVATE ====
-
-function deactivate {
-    if [ "$IS_EHELFSAVE_EVT" -eq 1 ] ; then
-        unset IS_EHELFSAVE_EVT
-
-        export CPATH="$CPATH_EHELFSAVE"
-        export LIBRARY_PATH="$LIBRARY_PATH_EHELFSAVE"
-        export LD_LIBRARY_PATH="$LD_LIBRARY_PATH_EHELFSAVE"
-        export PS1="$PS1_EHELFSAVE"
-
-        unset CPATH_EHELFSAVE
-        unset LIBRARY_PATH_EHELFSAVE
-        unset LD_LIBRARY_PATH_EHELFSAVE
-        unset PS1_EHELFSAVE
-    fi
-}
-
 # ==== INPUT ACQUISITION ====
 flavour="eh_elf"
 dbg="release"
@@ -39,6 +21,29 @@ while [ "$#" -gt 0 ] ; do
     shift
 done
 
+# ==== UNSET PREVIOUS ENVIRONMENT ====
+
+type -t deactivate
+[ -n "$(type -t deactivate)" ] && deactivate
+
+# ==== DEFINE DEACTIVATE ====
+
+function deactivate {
+    export CPATH="$CPATH_EHELFSAVE"
+    export LIBRARY_PATH="$LIBRARY_PATH_EHELFSAVE"
+    export LD_LIBRARY_PATH="$LD_LIBRARY_PATH_EHELFSAVE"
+    export PS1="$PS1_EHELFSAVE"
+    export PATH="$PATH_EHELFSAVE"
+
+    unset CPATH_EHELFSAVE
+    unset LIBRARY_PATH_EHELFSAVE
+    unset LD_LIBRARY_PATH_EHELFSAVE
+    unset PS1_EHELFSAVE
+    unset PATH_EHELFSAVE
+
+    unset deactivate
+}
+
 # ==== PREFIX ====
 export PERF_PREFIX="$HOME/local/perf-$flavour"
 
@@ -76,16 +81,23 @@ export LIBUNWIND_PREFIX
 function colon_prepend {
     if [ -z "$2" ]; then
         echo "$1"
+    elif [ -z "$1" ] ; then
+        echo "$2"
     else
-        >&2 echo ">$2<"
         echo "$1:$2"
     fi
 }
 
-export IS_EHELFSAVE_EVT=1
+function ifpath {
+    if [ -e "$1" ] ; then
+        echo "$1"
+    fi
+}
+
 export CPATH_EHELFSAVE="$CPATH"
 export LIBRARY_PATH_EHELFSAVE="$LIBRARY_PATH"
 export LD_LIBRARY_PATH_EHELFSAVE="$LD_LIBRARY_PATH"
+export PATH_EHELFSAVE="$PATH"
 export PS1_EHELFSAVE="$PS1"
 
 export CPATH="$(colon_prepend \
@@ -94,5 +106,13 @@ export LIBRARY_PATH="$(colon_prepend \
     "$LIBUNWIND_PREFIX/lib/:$PERF_PREFIX/lib" "$LIBRARY_PATH")"
 export LD_LIBRARY_PATH="$(colon_prepend \
     "$LIBUNWIND_PREFIX/lib/:$PERF_PREFIX/lib" "$LD_LIBRARY_PATH")"
+export PATH="$(colon_prepend \
+    "$(colon_prepend \
+        "$(ifpath "$LIBUNWIND_PREFIX/bin")" \
+        "$(ifpath "$PERF_PREFIX/bin")")" \
+    "$PATH")"
 
 export PS1="($flavour $dbg) $PS1"
+
+unset ifpath
+unset colon_prepend

generate_eh_elf.py

@@ -13,120 +13,122 @@ import tempfile
 import argparse
 from enum import Enum
 
-from shared_python import \
-    elf_so_deps, \
-    do_remote, \
-    is_newer, \
-    to_eh_elf_path, \
-    find_eh_elf_dir, \
-    DEFAULT_AUX_DIRS
+from shared_python import (
+    elf_so_deps,
+    do_remote,
+    is_newer,
+    to_eh_elf_path,
+    find_eh_elf_dir,
+    DEFAULT_AUX_DIRS,
+)
 from extract_pc import generate_pc_list
 
 
 DWARF_ASSEMBLY_BIN = os.path.join(
-    os.path.dirname(os.path.abspath(sys.argv[0])),
-    'dwarf-assembly')
-C_BIN = (
-    'gcc' if 'C' not in os.environ
-    else os.environ['C'])
+    os.path.dirname(os.path.abspath(sys.argv[0])), "dwarf-assembly"
+)
+C_BIN = "gcc" if "C" not in os.environ else os.environ["C"]
 
 
 class SwitchGenPolicy(Enum):
-    ''' The various switch generation policies possible '''
-    SWITCH_PER_FUNC = '--switch-per-func'
-    GLOBAL_SWITCH = '--global-switch'
+    """ The various switch generation policies possible """
+
+    SWITCH_PER_FUNC = "--switch-per-func"
+    GLOBAL_SWITCH = "--global-switch"
 
 
 class Config:
-    ''' Holds the run's settings '''
+    """ Holds the run's settings """
 
     default_aux = DEFAULT_AUX_DIRS
 
-    def __init__(self,
-                 output,
-                 aux,
-                 no_dft_aux,
-                 objects,
-                 sw_gen_policy=SwitchGenPolicy.GLOBAL_SWITCH,
-                 force=False,
-                 use_pc_list=False,
-                 c_opt_level='3',
-                 enable_deref_arg=False,
-                 cc_debug=False,
-                 remote=None):
-        self.output = '.' if output is None else output
-        self.aux = (
-            aux
-            + ([] if no_dft_aux else self.default_aux)
-        )
+    def __init__(
+        self,
+        output,
+        aux,
+        no_dft_aux,
+        objects,
+        sw_gen_policy=SwitchGenPolicy.GLOBAL_SWITCH,
+        force=False,
+        use_pc_list=False,
+        c_opt_level="3",
+        enable_deref_arg=False,
+        keep_holes=False,
+        cc_debug=False,
+        remote=None,
+    ):
+        self.output = "." if output is None else output
+        self.aux = aux + ([] if no_dft_aux else self.default_aux)
         self.objects = objects
         self.sw_gen_policy = sw_gen_policy
         self.force = force
         self.use_pc_list = use_pc_list
         self.c_opt_level = c_opt_level
         self.enable_deref_arg = enable_deref_arg
+        self.keep_holes = keep_holes
         self.cc_debug = cc_debug
         self.remote = remote
 
     @staticmethod
     def default_aux_str():
-        return ', '.join(Config.default_aux)
+        return ", ".join(Config.default_aux)
 
     def dwarf_assembly_args(self):
-        ''' Arguments to `dwarf_assembly` '''
+        """ Arguments to `dwarf_assembly` """
         out = []
         out.append(self.sw_gen_policy.value)
         if self.enable_deref_arg:
-            out.append('--enable-deref-arg')
+            out.append("--enable-deref-arg")
+        if self.keep_holes:
+            out.append("--keep-holes")
         return out
 
     def cc_opts(self):
-        ''' Options to pass to the C compiler '''
-        out = ['-fPIC']
+        """ Options to pass to the C compiler """
+        out = ["-fPIC"]
         if self.cc_debug:
-            out.append('-g')
+            out.append("-g")
         out.append(self.opt_level())
         return out
 
     def opt_level(self):
-        ''' The optimization level to pass to gcc '''
-        return '-O{}'.format(self.c_opt_level)
+        """ The optimization level to pass to gcc """
+        return "-O{}".format(self.c_opt_level)
 
     def aux_dirs(self):
-        ''' Get the list of auxiliary directories '''
+        """ Get the list of auxiliary directories """
         return self.aux
 
 
 def gen_dw_asm_c(obj_path, out_path, config, pc_list_path=None):
-    ''' Generate the C code produced by dwarf-assembly from `obj_path`, saving
-    it as `out_path` '''
+    """ Generate the C code produced by dwarf-assembly from `obj_path`, saving
+    it as `out_path` """
 
     dw_assembly_args = config.dwarf_assembly_args()
     if pc_list_path is not None:
-        dw_assembly_args += ['--pc-list', pc_list_path]
+        dw_assembly_args += ["--pc-list", pc_list_path]
 
     try:
-        with open(out_path, 'w') as out_handle:
+        with open(out_path, "w") as out_handle:
             # TODO enhance error handling
-            dw_asm_output = subprocess.check_output(
-                [DWARF_ASSEMBLY_BIN, obj_path] + dw_assembly_args) \
-                .decode('utf-8')
+            command_args = [DWARF_ASSEMBLY_BIN, obj_path] + dw_assembly_args
+            dw_asm_output = subprocess.check_output(command_args).decode("utf-8")
             out_handle.write(dw_asm_output)
     except subprocess.CalledProcessError as exn:
         raise Exception(
-            ("Cannot generate C code from object file {} using {}: process "
-             "terminated with exit code {}.").format(
-                 obj_path,
-                 DWARF_ASSEMBLY_BIN,
-                 exn.returncode))
+            (
+                "Cannot generate C code from object file {} using {}: process "
+                "terminated with exit code {}."
+            ).format(obj_path, DWARF_ASSEMBLY_BIN, exn.returncode)
+        )
 
 
 def resolve_symlink_chain(objpath):
-    ''' Resolves a symlink chain. This returns a pair `(new_obj, chain)`,
+    """ Resolves a symlink chain. This returns a pair `(new_obj, chain)`,
     `new_obj` being the canonical path for `objpath`, and `chain` being a list
     representing the path followed, eg. `[(objpath, a), (a, b), (b, new_obj)]`.
     The goal of this function is to allow reproducing symlink architectures at
-    the eh_elf level. '''
+    the eh_elf level. """
 
     chain = []
     out_path = objpath
@@ -142,16 +144,16 @@ def resolve_symlink_chain(objpath):
 
 
 def find_out_dir(obj_path, config):
-    ''' Find the directory in which the eh_elf corresponding to `obj_path` will
-    be outputted, among the output directory and the aux directories '''
+    """ Find the directory in which the eh_elf corresponding to `obj_path` will
+    be outputted, among the output directory and the aux directories """
 
     return find_eh_elf_dir(obj_path, config.aux_dirs(), config.output)
 
 
 def gen_eh_elf(obj_path, config):
-    ''' Generate the eh_elf corresponding to `obj_path`, saving it as
+    """ Generate the eh_elf corresponding to `obj_path`, saving it as
     `out_dir/$(basename obj_path).eh_elf.so` (or in the current working
-    directory if out_dir is None) '''
+    directory if out_dir is None) """
 
     out_dir = find_out_dir(obj_path, config)
     obj_path, link_chain = resolve_symlink_chain(obj_path)
@@ -161,19 +163,20 @@ def gen_eh_elf(obj_path, config):
     link_chain = map(
         lambda elt: (
             to_eh_elf_path(elt[0], out_dir),
-            os.path.basename(to_eh_elf_path(elt[1], out_dir))),
-        link_chain)
+            os.path.basename(to_eh_elf_path(elt[1], out_dir)),
+        ),
+        link_chain,
+    )
 
     out_base_name = to_eh_elf_path(obj_path, out_dir, base=True)
     out_so_path = to_eh_elf_path(obj_path, out_dir, base=False)
-    pc_list_dir = os.path.join(out_dir, 'pc_list')
+    pc_list_dir = os.path.join(out_dir, "pc_list")
 
     if is_newer(out_so_path, obj_path) and not config.force:
         return  # The object is recent enough, no need to recreate it
 
     if os.path.exists(out_dir) and not os.path.isdir(out_dir):
-        raise Exception("The output path {} is not a directory.".format(
-            out_dir))
+        raise Exception("The output path {} is not a directory.".format(out_dir))
     if not os.path.exists(out_dir):
         os.makedirs(out_dir, exist_ok=True)
 
@@ -181,42 +184,38 @@ def gen_eh_elf(obj_path, config):
         # Generate PC list
         pc_list_path = None
         if config.use_pc_list:
-            pc_list_path = \
-                os.path.join(pc_list_dir, out_base_name + '.pc_list')
+            pc_list_path = os.path.join(pc_list_dir, out_base_name + ".pc_list")
             os.makedirs(pc_list_dir, exist_ok=True)
-            print('\tGenerating PC list…')
+            print("\tGenerating PC list…")
             generate_pc_list(obj_path, pc_list_path)
 
         # Generate the C source file
         print("\tGenerating C…")
-        c_path = os.path.join(compile_dir, (out_base_name + '.c'))
+        c_path = os.path.join(compile_dir, (out_base_name + ".c"))
         gen_dw_asm_c(obj_path, c_path, config, pc_list_path)
 
         # Compile it into a .o
         print("\tCompiling into .o…")
-        o_path = os.path.join(compile_dir, (out_base_name + '.o'))
+        o_path = os.path.join(compile_dir, (out_base_name + ".o"))
         if config.remote:
             remote_out = do_remote(
                 config.remote,
-                [
-                    C_BIN,
-                    '-o', out_base_name + '.o',
-                    '-c', out_base_name + '.c'
-                ] + config.cc_opts(),
+                [C_BIN, "-o", out_base_name + ".o", "-c", out_base_name + ".c"]
+                + config.cc_opts(),
                 send_files=[c_path],
-                retr_files=[(out_base_name + '.o', o_path)])
+                retr_files=[(out_base_name + ".o", o_path)],
+            )
             call_rc = 1 if remote_out is None else 0
         else:
             call_rc = subprocess.call(
-                [C_BIN, '-o', o_path, '-c', c_path,
-                 config.opt_level(), '-fPIC'])
+                [C_BIN, "-o", o_path, "-c", c_path, config.opt_level(), "-fPIC"]
+            )
         if call_rc != 0:
             raise Exception("Failed to compile to a .o file")
 
         # Compile it into a .so
         print("\tCompiling into .so…")
-        call_rc = subprocess.call(
-            [C_BIN, '-o', out_so_path, '-shared', o_path])
+        call_rc = subprocess.call([C_BIN, "-o", out_so_path, "-shared", o_path])
         if call_rc != 0:
             raise Exception("Failed to compile to a .so file")
 
@@ -225,40 +224,32 @@ def gen_eh_elf(obj_path, config):
         if os.path.exists(elt[0]):
             if not os.path.islink(elt[0]):
                 raise Exception(
-                    "{}: file already exists and is not a symlink.".format(
-                        elt[0]))
+                    "{}: file already exists and is not a symlink.".format(elt[0])
+                )
             os.remove(elt[0])
         os.symlink(elt[1], elt[0])
 
 
 def gen_all_eh_elf(obj_path, config):
-    ''' Call `gen_eh_elf` on obj_path and all its dependencies '''
+    """ Call `gen_eh_elf` on obj_path and all its dependencies """
     deps = elf_so_deps(obj_path)
     deps.append(obj_path)
     for dep in deps:
         gen_eh_elf(dep, config)
 
 
-def gen_eh_elfs(obj_path,
-                out_dir,
-                global_switch=True,
-                deps=True,
-                remote=None):
-    ''' Call gen{_all,}_eh_elf with args setup accordingly with the given
-    options '''
+def gen_eh_elfs(obj_path, out_dir, global_switch=True, deps=True, remote=None):
+    """ Call gen{_all,}_eh_elf with args setup accordingly with the given
+    options """
 
     switch_gen_policy = (
-        SwitchGenPolicy.GLOBAL_SWITCH if global_switch
+        SwitchGenPolicy.GLOBAL_SWITCH
+        if global_switch
         else SwitchGenPolicy.SWITCH_PER_FUNC
     )
 
     config = Config(
-        out_dir,
-        [],
-        False,
-        [obj_path],
-        sw_gen_policy=switch_gen_policy,
-        remote=remote,
+        out_dir, [], False, [obj_path], sw_gen_policy=switch_gen_policy, remote=remote
     )
 
     if deps:
@@ -267,106 +258,176 @@ def gen_eh_elfs(obj_path,
 
 
 def process_args():
-    ''' Process `sys.argv` arguments '''
+    """ Process `sys.argv` arguments """
 
     parser = argparse.ArgumentParser(
-        description="Compile ELFs into their related eh_elfs",
+        description="Compile ELFs into their related eh_elfs"
     )
 
-    parser.add_argument('--deps', action='store_const',
-                        const=gen_all_eh_elf, default=gen_eh_elf,
-                        dest='gen_func',
-                        help=("Also generate eh_elfs for the shared objects "
-                              "this object depends on"))
-    parser.add_argument('-o', '--output', metavar="path",
-                        help=("Save the generated objects at the given path "
-                              "instead of the current working directory"))
-    parser.add_argument('-a', '--aux', action='append', default=[],
-                        help=("Alternative output directories. These "
-                              "directories are searched for existing matching "
-                              "eh_elfs, and if found, these files are updated "
-                              "instead of creating new files in the --output "
-                              "directory. By default, some aux directories "
-                              "are always considered, unless -A is passed: "
-                              "{}.").format(Config.default_aux_str()))
-    parser.add_argument('-A', '--no-dft-aux', action='store_true',
-                        help=("Do not use the default auxiliary output "
-                              "directories: {}.").format(
-                                  Config.default_aux_str()))
-    parser.add_argument('--remote', metavar='ssh_args',
-                        help=("Execute the heavyweight commands on the remote "
-                              "machine, using `ssh ssh_args`."))
-    parser.add_argument('--use-pc-list', action='store_true',
-                        help=("Generate a PC list using `extract_pc.py` for "
-                              "each processed ELF file, and call "
-                              "dwarf-assembly accordingly."))
-    parser.add_argument('--force', '-f', action='store_true',
-                        help=("Force re-generation of the output files, even "
-                              "when those files are newer than the target "
-                              "ELF."))
-    parser.add_argument('--enable-deref-arg', action='store_true',
-                        help=("Pass the `--enable-deref-arg` to "
-                              "dwarf-assembly, enabling an extra `deref` "
-                              "argument for each lookup function, allowing "
-                              "to work on remote address spaces."))
-    parser.add_argument("-g", "--cc-debug", action='store_true',
-                        help=("Compile the source file with -g for easy "
-                              "debugging"))
+    parser.add_argument(
+        "--deps",
+        action="store_const",
+        const=gen_all_eh_elf,
+        default=gen_eh_elf,
+        dest="gen_func",
+        help=("Also generate eh_elfs for the shared objects " "this object depends on"),
+    )
+    parser.add_argument(
+        "-o",
+        "--output",
+        metavar="path",
+        help=(
+            "Save the generated objects at the given path "
+            "instead of the current working directory"
+        ),
+    )
+    parser.add_argument(
+        "-a",
+        "--aux",
+        action="append",
+        default=[],
+        help=(
+            "Alternative output directories. These "
+            "directories are searched for existing matching "
+            "eh_elfs, and if found, these files are updated "
+            "instead of creating new files in the --output "
+            "directory. By default, some aux directories "
+            "are always considered, unless -A is passed: "
+            "{}."
+        ).format(Config.default_aux_str()),
+    )
+    parser.add_argument(
+        "-A",
+        "--no-dft-aux",
+        action="store_true",
+        help=("Do not use the default auxiliary output " "directories: {}.").format(
+            Config.default_aux_str()
+        ),
+    )
+    parser.add_argument(
+        "--remote",
+        metavar="ssh_args",
+        help=(
+            "Execute the heavyweight commands on the remote "
+            "machine, using `ssh ssh_args`."
+        ),
+    )
+    parser.add_argument(
+        "--use-pc-list",
+        action="store_true",
+        help=(
+            "Generate a PC list using `extract_pc.py` for "
+            "each processed ELF file, and call "
+            "dwarf-assembly accordingly."
+        ),
+    )
+    parser.add_argument(
+        "--force",
+        "-f",
+        action="store_true",
+        help=(
+            "Force re-generation of the output files, even "
+            "when those files are newer than the target "
+            "ELF."
+        ),
+    )
+    parser.add_argument(
+        "--enable-deref-arg",
+        action="store_true",
+        help=(
+            "Pass the `--enable-deref-arg` to "
+            "dwarf-assembly, enabling an extra `deref` "
+            "argument for each lookup function, allowing "
+            "to work on remote address spaces."
+        ),
+    )
+    parser.add_argument(
+        "--keep-holes",
+        action="store_true",
+        help=(
+            "Keep holes between FDEs instead of filling "
+            "them with junk. More accurate, less compact."
+        ),
+    )
+    parser.add_argument(
+        "-g",
+        "--cc-debug",
+        action="store_true",
+        help=("Compile the source file with -g for easy " "debugging"),
+    )
     # c_opt_level
     opt_level_grp = parser.add_mutually_exclusive_group()
-    opt_level_grp.add_argument('-O0', action='store_const', const='0',
-                               dest='c_opt_level',
-                               help=("Compile C file with this optimization "
-                                     "level."))
-    opt_level_grp.add_argument('-O1', action='store_const', const='1',
-                               dest='c_opt_level',
-                               help=("Compile C file with this optimization "
-                                     "level."))
-    opt_level_grp.add_argument('-O2', action='store_const', const='2',
-                               dest='c_opt_level',
-                               help=("Compile C file with this optimization "
-                                     "level."))
-    opt_level_grp.add_argument('-O3', action='store_const', const='3',
-                               dest='c_opt_level',
-                               help=("Compile C file with this optimization "
-                                     "level."))
-    opt_level_grp.add_argument('-Os', action='store_const', const='s',
-                               dest='c_opt_level',
-                               help=("Compile C file with this optimization "
-                                     "level."))
-    opt_level_grp.set_defaults(c_opt_level='3')
+    opt_level_grp.add_argument(
+        "-O0",
+        action="store_const",
+        const="0",
+        dest="c_opt_level",
+        help=("Compile C file with this optimization " "level."),
+    )
+    opt_level_grp.add_argument(
+        "-O1",
+        action="store_const",
+        const="1",
+        dest="c_opt_level",
+        help=("Compile C file with this optimization " "level."),
+    )
+    opt_level_grp.add_argument(
+        "-O2",
+        action="store_const",
+        const="2",
+        dest="c_opt_level",
+        help=("Compile C file with this optimization " "level."),
+    )
+    opt_level_grp.add_argument(
+        "-O3",
+        action="store_const",
+        const="3",
+        dest="c_opt_level",
+        help=("Compile C file with this optimization " "level."),
+    )
+    opt_level_grp.add_argument(
+        "-Os",
+        action="store_const",
+        const="s",
+        dest="c_opt_level",
+        help=("Compile C file with this optimization " "level."),
+    )
+    opt_level_grp.set_defaults(c_opt_level="3")
 
-    switch_gen_policy = \
-        parser.add_mutually_exclusive_group(required=True)
-    switch_gen_policy.add_argument('--switch-per-func',
-                                   dest='sw_gen_policy',
-                                   action='store_const',
-                                   const=SwitchGenPolicy.SWITCH_PER_FUNC,
-                                   help=("Passed to dwarf-assembly."))
-    switch_gen_policy.add_argument('--global-switch',
-                                   dest='sw_gen_policy',
-                                   action='store_const',
-                                   const=SwitchGenPolicy.GLOBAL_SWITCH,
-                                   help=("Passed to dwarf-assembly."))
-    parser.add_argument('object', nargs='+',
-                        help="The ELF object(s) to process")
+    switch_gen_policy = parser.add_mutually_exclusive_group(required=True)
+    switch_gen_policy.add_argument(
+        "--switch-per-func",
+        dest="sw_gen_policy",
+        action="store_const",
+        const=SwitchGenPolicy.SWITCH_PER_FUNC,
+        help=("Passed to dwarf-assembly."),
+    )
+    switch_gen_policy.add_argument(
+        "--global-switch",
+        dest="sw_gen_policy",
+        action="store_const",
+        const=SwitchGenPolicy.GLOBAL_SWITCH,
+        help=("Passed to dwarf-assembly."),
+    )
+    parser.add_argument("object", nargs="+", help="The ELF object(s) to process")
     return parser.parse_args()
 
 
 def main():
     args = process_args()
     config = Config(
-        args.output,
-        args.aux,
-        args.no_dft_aux,
-        args.object,
-        args.sw_gen_policy,
-        args.force,
-        args.use_pc_list,
-        args.c_opt_level,
-        args.enable_deref_arg,
-        args.cc_debug,
-        args.remote,
+        output=args.output,
+        aux=args.aux,
+        no_dft_aux=args.no_dft_aux,
+        objects=args.object,
+        sw_gen_policy=args.sw_gen_policy,
+        force=args.force,
+        use_pc_list=args.use_pc_list,
+        c_opt_level=args.c_opt_level,
+        enable_deref_arg=args.enable_deref_arg,
+        keep_holes=args.keep_holes,
+        cc_debug=args.cc_debug,
+        remote=args.remote,
     )
 
     for obj in args.object:

src/CodeGenerator.cpp

@@ -271,6 +271,16 @@ void CodeGenerator::gen_of_reg(const SimpleDwarf::DwRegister& reg,
             }
             break;
         }
+        case SimpleDwarf::DwRegister::REG_PLT_EXPR: {
+            /*
+            if(settings::enable_deref_arg)
+                stream << "(deref(";
+            else
+                stream << "*((uintptr_t*)(";
+            */
+            stream << "(((ctx.rip & 15) >= 11) ? 8 : 0) + ctx.rsp";
+            break;
+        }
         case SimpleDwarf::DwRegister::REG_NOT_IMPLEMENTED:
             stream << "0";
             throw UnhandledRegister();

src/ConseqEquivFilter.cpp

@@ -2,7 +2,7 @@
 
 using namespace std;
 
-ConseqEquivFilter::ConseqEquivFilter() {}
+ConseqEquivFilter::ConseqEquivFilter(bool enable): SimpleDwarfFilter(enable) {}
 
 static bool equiv_reg(
         const SimpleDwarf::DwRegister& r1,

src/ConseqEquivFilter.hpp

@@ -9,7 +9,7 @@
 
 class ConseqEquivFilter: public SimpleDwarfFilter {
     public:
-        ConseqEquivFilter();
+        ConseqEquivFilter(bool enable=true);
 
     private:
         SimpleDwarf do_apply(const SimpleDwarf& dw) const;

src/DwarfReader.cpp

@@ -1,5 +1,7 @@
 #include "DwarfReader.hpp"
 
+#include "plt_std_expr.hpp"
+
 #include <fstream>
 #include <fileno.hpp>
 #include <set>
@@ -7,14 +9,25 @@
 using namespace std;
 using namespace dwarf;
 
-typedef std::set<std::pair<int, core::FrameSection::register_def> >
-    dwarfpp_row_t;
-
 DwarfReader::DwarfReader(const string& path):
     root(fileno(ifstream(path)))
 {}
 
-SimpleDwarf DwarfReader::read() const {
+// Debug function -- dumps an expression
+static void dump_expr(const core::FrameSection::register_def& reg) {
+    assert(reg.k == core::FrameSection::register_def::SAVED_AT_EXPR
+           || reg.k == core::FrameSection::register_def::VAL_OF_EXPR);
+
+    const encap::loc_expr& expr = reg.saved_at_expr_r();
+
+    for(const auto& elt: expr) {
+        fprintf(stderr, "(%02x, %02llx, %02llx, %02llx) :: ",
+                elt.lr_atom, elt.lr_number, elt.lr_number2, elt.lr_offset);
+    }
+    fprintf(stderr, "\n");
+}
+
+SimpleDwarf DwarfReader::read() {
     const core::FrameSection& fs = root.get_frame_section();
     SimpleDwarf output;
 
@@ -26,57 +39,119 @@ SimpleDwarf DwarfReader::read() const {
     return output;
 }
 
-SimpleDwarf::Fde DwarfReader::read_fde(const core::Fde& fde) const {
+void DwarfReader::add_cell_to_row(
+        const dwarf::core::FrameSection::register_def& reg,
+        int reg_id,
+        int ra_reg,
+        SimpleDwarf::DwRow& cur_row)
+{
+    if(reg_id == DW_FRAME_CFA_COL3) {
+        cur_row.cfa = read_register(reg);
+    }
+    else {
+        try {
+            SimpleDwarf::MachineRegister reg_type =
+                from_dwarfpp_reg(reg_id, ra_reg);
+            switch(reg_type) {
+                case SimpleDwarf::REG_RBP:
+                    cur_row.rbp = read_register(reg);
+                    break;
+                case SimpleDwarf::REG_RBX:
+                    cur_row.rbx = read_register(reg);
+                    break;
+                case SimpleDwarf::REG_RA:
+                    cur_row.ra = read_register(reg);
+                    break;
+                default:
+                    break;
+            }
+        }
+        catch(const UnsupportedRegister&) {} // Just ignore it.
+    }
+}
+
+void DwarfReader::append_row_to_fde(
+        const dwarfpp_row_t& row,
+        uintptr_t row_addr,
+        int ra_reg,
+        SimpleDwarf::Fde& output)
+{
+    SimpleDwarf::DwRow cur_row;
+
+    cur_row.ip = row_addr;
+
+    for(const auto& cell: row) {
+        add_cell_to_row(cell.second, cell.first, ra_reg, cur_row);
+    }
+
+    if(cur_row.cfa.type == SimpleDwarf::DwRegister::REG_UNDEFINED)
+    {
+        // Not set
+        throw InvalidDwarf();
+    }
+
+    output.rows.push_back(cur_row);
+}
+
+template<typename Key, typename Value>
+static std::set<std::pair<Key, Value> > map_to_setpair(
+        const std::map<Key, Value>& src_map)
+{
+    std::set<std::pair<Key, Value> > out;
+    for(const auto map_it: src_map) {
+        out.insert(map_it);
+    }
+    return out;
+}
+
+void DwarfReader::append_results_to_fde(
+        const dwarf::core::FrameSection::instrs_results& results,
+        int ra_reg,
+        SimpleDwarf::Fde& output)
+{
+    for(const auto row_pair: results.rows) {
+        append_row_to_fde(
+                row_pair.second,
+                row_pair.first.lower(),
+                ra_reg,
+                output);
+    }
+    if(results.unfinished_row.size() > 0) {
+        try {
+            append_row_to_fde(
+                    map_to_setpair(results.unfinished_row),
+                    results.unfinished_row_addr,
+                    ra_reg,
+                    output);
+        } catch(const InvalidDwarf&) {
+            // Ignore: the unfinished_row can be undefined
+        }
+    }
+}
+
+SimpleDwarf::Fde DwarfReader::read_fde(const core::Fde& fde) {
     SimpleDwarf::Fde output;
     output.fde_offset = fde.get_fde_offset();
     output.beg_ip = fde.get_low_pc();
     output.end_ip = fde.get_low_pc() + fde.get_func_length();
 
-    auto rows = fde.decode().rows;
     const core::Cie& cie = *fde.find_cie();
     int ra_reg = cie.get_return_address_register_rule();
 
-    for(const auto row_pair: rows) {
-        SimpleDwarf::DwRow cur_row;
+    // CIE rows
+    core::FrameSection cie_fs(root.get_dbg(), true);
+    auto cie_rows = cie_fs.interpret_instructions(
+            cie,
+            fde.get_low_pc(),
+            cie.get_initial_instructions(),
+            cie.get_initial_instructions_length());
 
-        cur_row.ip = row_pair.first.lower();
+    // FDE rows
+    auto fde_rows = fde.decode();
 
-        const dwarfpp_row_t& row = row_pair.second;
-
-        for(const auto& cell: row) {
-            if(cell.first == DW_FRAME_CFA_COL3) {
-                cur_row.cfa = read_register(cell.second);
-            }
-            else {
-                try {
-                    SimpleDwarf::MachineRegister reg_type =
-                        from_dwarfpp_reg(cell.first, ra_reg);
-                    switch(reg_type) {
-                        case SimpleDwarf::REG_RBP:
-                            cur_row.rbp = read_register(cell.second);
-                            break;
-                        case SimpleDwarf::REG_RBX:
-                            cur_row.rbx = read_register(cell.second);
-                            break;
-                        case SimpleDwarf::REG_RA:
-                            cur_row.ra = read_register(cell.second);
-                            break;
-                        default:
-                            break;
-                    }
-                }
-                catch(const UnsupportedRegister&) {} // Just ignore it.
-            }
-        }
-
-        if(cur_row.cfa.type == SimpleDwarf::DwRegister::REG_UNDEFINED)
-        {
-            // Not set
-            throw InvalidDwarf();
-        }
-
-        output.rows.push_back(cur_row);
-    }
+    // instrs
+    append_results_to_fde(cie_rows, ra_reg, output);
+    append_results_to_fde(fde_rows, ra_reg, output);
 
     return output;
 }
@@ -107,6 +182,13 @@ SimpleDwarf::DwRegister DwarfReader::read_register(
                 output.type = SimpleDwarf::DwRegister::REG_UNDEFINED;
                 break;
 
+            case core::FrameSection::register_def::SAVED_AT_EXPR:
+                if(is_plt_expr(reg))
+                    output.type = SimpleDwarf::DwRegister::REG_PLT_EXPR;
+                else if(!interpret_simple_expr(reg, output))
+                    output.type = SimpleDwarf::DwRegister::REG_NOT_IMPLEMENTED;
+                break;
+
             default:
                 output.type = SimpleDwarf::DwRegister::REG_NOT_IMPLEMENTED;
                 break;
@@ -139,3 +221,70 @@ SimpleDwarf::MachineRegister DwarfReader::from_dwarfpp_reg(
             throw UnsupportedRegister();
     }
 }
+
+static bool compare_dw_expr(
+        const encap::loc_expr& e1,
+        const encap::loc_expr& e2)
+{
+    const std::vector<encap::expr_instr>& e1_vec =
+        static_cast<const vector<encap::expr_instr>&>(e1);
+    const std::vector<encap::expr_instr>& e2_vec =
+        static_cast<const vector<encap::expr_instr>&>(e2);
+
+    return e1_vec == e2_vec;
+}
+
+bool DwarfReader::is_plt_expr(
+        const core::FrameSection::register_def& reg) const
+{
+    if(reg.k != core::FrameSection::register_def::SAVED_AT_EXPR)
+        return false;
+    const encap::loc_expr& expr = reg.saved_at_expr_r();
+
+    bool res = compare_dw_expr(expr, REFERENCE_PLT_EXPR);
+    return res;
+}
+
+bool DwarfReader::interpret_simple_expr(
+        const dwarf::core::FrameSection::register_def& reg,
+        SimpleDwarf::DwRegister& output
+        ) const
+{
+    bool deref = false;
+    if(reg.k == core::FrameSection::register_def::SAVED_AT_EXPR)
+        deref = true;
+    else if(reg.k == core::FrameSection::register_def::VAL_OF_EXPR)
+        deref = false;
+    else
+        return false;
+
+    const encap::loc_expr& expr = reg.saved_at_expr_r();
+    if(expr.size() > 2 || expr.empty())
+        return false;
+
+    const auto& exp_reg = expr[0];
+    if(0x70 <= exp_reg.lr_atom && exp_reg.lr_atom <= 0x8f) { // DW_OP_breg<n>
+        int reg_id = exp_reg.lr_atom - 0x70;
+        try {
+            output.reg = from_dwarfpp_reg(reg_id, -1); // Cannot be CFA anyway
+            output.offset = exp_reg.lr_number;
+        } catch(const UnsupportedRegister& /* exn */) {
+            return false; // Unsupported register
+        }
+    }
+
+    if(expr.size() == 2) { // OK if deref
+        if(expr[1].lr_atom == 0x06) { // deref
+            if(deref)
+                return false;
+            deref = true;
+        }
+        else
+            return false;
+    }
+
+    if(deref)
+        return false; // TODO try stats? Mabye it's worth implementing
+    output.type = SimpleDwarf::DwRegister::REG_REGISTER;
+    return true;
+}

src/DwarfReader.hpp

@@ -13,6 +13,9 @@
 
 #include "SimpleDwarf.hpp"
 
+typedef std::set<std::pair<int, dwarf::core::FrameSection::register_def> >
+    dwarfpp_row_t;
+
 class DwarfReader {
     public:
         class InvalidDwarf: public std::exception {};
@@ -21,19 +24,44 @@ class DwarfReader {
         DwarfReader(const std::string& path);
 
         /** Actually read the ELF file, generating a `SimpleDwarf` output. */
-        SimpleDwarf read() const;
+        SimpleDwarf read();
 
     private: //meth
-        SimpleDwarf::Fde read_fde(const dwarf::core::Fde& fde) const;
+        SimpleDwarf::Fde read_fde(const dwarf::core::Fde& fde);
+
+        void append_results_to_fde(
+                const dwarf::core::FrameSection::instrs_results& results,
+                int ra_reg,
+                SimpleDwarf::Fde& output);
 
         SimpleDwarf::DwRegister read_register(
                 const dwarf::core::FrameSection::register_def& reg) const;
 
+        void add_cell_to_row(
+                const dwarf::core::FrameSection::register_def& reg,
+                int reg_id,
+                int ra_reg,
+                SimpleDwarf::DwRow& cur_row);
+
+        void append_row_to_fde(
+                const dwarfpp_row_t& row,
+                uintptr_t row_addr,
+                int ra_reg,
+                SimpleDwarf::Fde& output);
+
         SimpleDwarf::MachineRegister from_dwarfpp_reg(
                 int reg_id,
                 int ra_reg=-1
                 ) const;
 
+        bool is_plt_expr(
+                const dwarf::core::FrameSection::register_def& reg) const;
+
+        bool interpret_simple_expr(
+                const dwarf::core::FrameSection::register_def& reg,
+                SimpleDwarf::DwRegister& output
+                ) const;
+
         class UnsupportedRegister: public std::exception {};
 
     private:

src/EmptyFdeDeleter.cpp

@@ -0,0 +1,21 @@
+#include "EmptyFdeDeleter.hpp"
+
+#include <algorithm>
+#include <cstdio>
+
+using namespace std;
+
+EmptyFdeDeleter::EmptyFdeDeleter(bool enable): SimpleDwarfFilter(enable) {}
+
+SimpleDwarf EmptyFdeDeleter::do_apply(const SimpleDwarf& dw) const {
+    SimpleDwarf out(dw);
+
+    auto fde = out.fde_list.begin();
+    while(fde != out.fde_list.end()) {
+        if(fde->rows.empty())
+            fde = out.fde_list.erase(fde);
+        else
+            ++fde;
+    }
+    return out;
+}

src/EmptyFdeDeleter.hpp

@@ -0,0 +1,15 @@
+/** Deletes empty FDEs (that is, FDEs with no rows) from the FDEs collection.
+ * This is used to ensure they do not interfere with PcHoleFiller and such. */
+
+#pragma once
+
+#include "SimpleDwarf.hpp"
+#include "SimpleDwarfFilter.hpp"
+
+class EmptyFdeDeleter: public SimpleDwarfFilter {
+    public:
+        EmptyFdeDeleter(bool enable=true);
+
+    private:
+        SimpleDwarf do_apply(const SimpleDwarf& dw) const;
+};

src/FactoredSwitchCompiler.cpp

@@ -2,6 +2,7 @@
 
 #include <sstream>
 #include <string>
+#include <iostream>
 using namespace std;
 
 FactoredSwitchCompiler::FactoredSwitchCompiler(int indent):
@@ -12,12 +13,32 @@ FactoredSwitchCompiler::FactoredSwitchCompiler(int indent):
 void FactoredSwitchCompiler::to_stream(
         std::ostream& os, const SwitchStatement& sw)
 {
+    if(sw.cases.empty()) {
+        std::cerr << "WARNING: empty unwinding data!\n";
+        os
+            << indent_str("/* WARNING: empty unwinding data! */\n")
+            << indent_str(sw.default_case) << "\n";
+        return;
+    }
     JumpPointMap jump_points;
 
-    gen_binsearch_tree(os, jump_points, sw.switch_var,
-            sw.cases.begin(), sw.cases.end());
+    uintptr_t low_bound = sw.cases.front().low_bound,
+              high_bound = sw.cases.back().high_bound;
 
-    os << indent_str(sw.default_case) << "\n"
+    os << indent() << "if("
+       << "0x" << hex << low_bound << " <= " << sw.switch_var
+       << " && " << sw.switch_var << " <= 0x" << high_bound << dec << ") {\n";
+    indent_count++;
+
+    gen_binsearch_tree(os, jump_points, sw.switch_var,
+            sw.cases.begin(), sw.cases.end(),
+            make_pair(low_bound, high_bound));
+
+    indent_count--;
+    os << indent() << "}\n";
+
+    os << indent() << "_factor_default:\n"
+       << indent_str(sw.default_case) << "\n"
        << indent() << "/* ===== LABELS  ============================== */\n\n";
 
     gen_jump_points_code(os, jump_points);
@@ -65,7 +86,8 @@ void FactoredSwitchCompiler::gen_binsearch_tree(
         FactoredSwitchCompiler::JumpPointMap& jump_map,
         const std::string& sw_var,
         const FactoredSwitchCompiler::case_iterator_t& begin,
-        const FactoredSwitchCompiler::case_iterator_t& end)
+        const FactoredSwitchCompiler::case_iterator_t& end,
+        const loc_range_t& loc_range)
 {
     size_t iter_delta = end - begin;
     if(iter_delta == 0)
@@ -73,6 +95,19 @@ void FactoredSwitchCompiler::gen_binsearch_tree(
     else if(iter_delta == 1) {
         FactorJumpPoint jump_point = get_jump_point(
                 jump_map, begin->content);
+        if(loc_range.first < begin->low_bound) {
+            os << indent() << "if(" << sw_var << " < 0x"
+               << hex << begin->low_bound << dec
+               << ") goto _factor_default; "
+               << "// IP=0x" << hex << loc_range.first << " ... 0x"
+               << begin->low_bound - 1 << "\n";
+        }
+        if(begin->high_bound + 1 < loc_range.second) {
+            os << indent() << "if(0x" << hex << begin->high_bound << dec
+               << " < " << sw_var << ") goto _factor_default; "
+               << "// IP=0x" << hex << begin->high_bound + 1 << " ... 0x"
+               << loc_range.second - 1 << "\n";
+        }
         os << indent() << "// IP=0x" << hex << begin->low_bound
                        << " ... 0x" << begin->high_bound << dec << "\n"
            << indent() << "goto " << jump_point << ";\n";
@@ -83,11 +118,15 @@ void FactoredSwitchCompiler::gen_binsearch_tree(
         os << indent() << "if(" << sw_var << " < 0x"
            << hex << mid->low_bound << dec << ") {\n";
         indent_count++;
-        gen_binsearch_tree(os, jump_map, sw_var, begin, mid);
+        gen_binsearch_tree(
+                os, jump_map, sw_var, begin, mid,
+                make_pair(loc_range.first, mid->low_bound));
         indent_count--;
         os << indent() << "} else {\n";
         indent_count++;
-        gen_binsearch_tree(os, jump_map, sw_var, mid, end);
+        gen_binsearch_tree(
+                os, jump_map, sw_var, mid, end,
+                make_pair(mid->low_bound, loc_range.second));
         indent_count--;
         os << indent() << "}\n";
     }

src/FactoredSwitchCompiler.hpp

@@ -24,6 +24,7 @@ class FactoredSwitchCompiler: public AbstractSwitchCompiler {
             JumpPointMap;
         typedef std::vector<SwitchStatement::SwitchCase>::const_iterator
             case_iterator_t;
+        typedef std::pair<uintptr_t, uintptr_t> loc_range_t;
 
     private:
         virtual void to_stream(std::ostream& os, const SwitchStatement& sw);
@@ -39,7 +40,9 @@ class FactoredSwitchCompiler: public AbstractSwitchCompiler {
                 JumpPointMap& jump_map,
                 const std::string& sw_var,
                 const case_iterator_t& begin,
-                const case_iterator_t& end);
+                const case_iterator_t& end,
+                const loc_range_t& loc_range // [beg, end[
+                );
 
         size_t cur_label_id;
 

src/Makefile

@@ -2,7 +2,7 @@ CXX=g++
 CXXLOCS?=-L. -I.
 CXXFL?=
 CXXFLAGS=$(CXXLOCS) -Wall -Wextra -std=c++14 -O2 -g $(CXXFL)
-CXXLIBS=-lelf -ldwarf -ldwarfpp -lsrk31c++ -lc++fileno
+CXXLIBS=-ldwarf -ldwarfpp -lsrk31c++ -lc++fileno -lelf
 
 TARGET=dwarf-assembly
 OBJS=\
@@ -12,7 +12,9 @@ OBJS=\
 	PcListReader.o \
 	SimpleDwarfFilter.o \
 	PcHoleFiller.o \
+	EmptyFdeDeleter.o \
 	ConseqEquivFilter.o \
+	OverriddenRowFilter.o \
 	SwitchStatement.o \
 	NativeSwitchCompiler.o \
 	FactoredSwitchCompiler.o \

src/OverriddenRowFilter.cpp

@@ -0,0 +1,31 @@
+#include "OverriddenRowFilter.hpp"
+
+OverriddenRowFilter::OverriddenRowFilter(bool enable)
+    : SimpleDwarfFilter(enable)
+{}
+
+SimpleDwarf OverriddenRowFilter::do_apply(const SimpleDwarf& dw) const {
+    SimpleDwarf out;
+
+    for(const auto& fde: dw.fde_list) {
+        out.fde_list.push_back(SimpleDwarf::Fde());
+        SimpleDwarf::Fde& cur_fde = out.fde_list.back();
+        cur_fde.fde_offset = fde.fde_offset;
+        cur_fde.beg_ip = fde.beg_ip;
+        cur_fde.end_ip = fde.end_ip;
+
+        if(fde.rows.empty())
+            continue;
+
+        for(size_t pos=0; pos < fde.rows.size(); ++pos) {
+            const auto& row = fde.rows[pos];
+            if(pos == fde.rows.size() - 1
+                    || row.ip != fde.rows[pos+1].ip)
+            {
+                cur_fde.rows.push_back(row);
+            }
+        }
+    }
+
+    return out;
+}

src/OverriddenRowFilter.hpp

@@ -0,0 +1,15 @@
+/** SimpleDwarfFilter to remove the first `n-1` rows of a block of `n`
+ * contiguous rows that have the exact same address. */
+
+#pragma once
+
+#include "SimpleDwarf.hpp"
+#include "SimpleDwarfFilter.hpp"
+
+class OverriddenRowFilter: public SimpleDwarfFilter {
+    public:
+        OverriddenRowFilter(bool enable=true);
+
+    private:
+        SimpleDwarf do_apply(const SimpleDwarf& dw) const;
+};

src/PcHoleFiller.cpp

@@ -5,7 +5,7 @@
 
 using namespace std;
 
-PcHoleFiller::PcHoleFiller() {}
+PcHoleFiller::PcHoleFiller(bool enable): SimpleDwarfFilter(enable) {}
 
 SimpleDwarf PcHoleFiller::do_apply(const SimpleDwarf& dw) const {
     SimpleDwarf out(dw);

src/PcHoleFiller.hpp

@@ -8,7 +8,7 @@
 
 class PcHoleFiller: public SimpleDwarfFilter {
     public:
-        PcHoleFiller();
+        PcHoleFiller(bool enable=true);
 
     private:
         SimpleDwarf do_apply(const SimpleDwarf& dw) const;

src/SimpleDwarf.hpp

@@ -32,6 +32,10 @@ struct SimpleDwarf {
                              defined at some later IP in the same DIE) */
             REG_REGISTER, ///< Value of a machine register plus offset
             REG_CFA_OFFSET, ///< Value stored at some offset from CFA
+            REG_PLT_EXPR, /**< Value is the evaluation of the standard PLT
+                            expression, ie `((rip & 15) >= 11) >> 3 + rsp`
+                            This is hardcoded because it's the only expression
+                            found so far, thus worth implementing. */
             REG_NOT_IMPLEMENTED ///< This type of register is not supported
         };
 

src/SimpleDwarfFilter.cpp

@@ -1,10 +1,11 @@
 #include "SimpleDwarfFilter.hpp"
 
-SimpleDwarfFilter::SimpleDwarfFilter()
+SimpleDwarfFilter::SimpleDwarfFilter(bool enable): enable(enable)
 {}
 
 SimpleDwarf SimpleDwarfFilter::apply(const SimpleDwarf& dw) const {
-    // For convenience of future enhancements
+    if(!enable)
+        return dw;
     return do_apply(dw);
 }
 

src/SimpleDwarfFilter.hpp

@@ -9,7 +9,11 @@
 
 class SimpleDwarfFilter {
     public:
-        SimpleDwarfFilter();
+        /** Constructor
+         *
+         * @param apply set to false to disable this filter. This setting is
+         * convenient for compact filter-chaining code. */
+        SimpleDwarfFilter(bool enable=true);
 
         /// Applies the filter
         SimpleDwarf apply(const SimpleDwarf& dw) const;
@@ -19,4 +23,6 @@ class SimpleDwarfFilter {
 
     private:
         virtual SimpleDwarf do_apply(const SimpleDwarf& dw) const = 0;
+
+        bool enable;
 };

src/main.cpp

@@ -11,7 +11,9 @@
 #include "NativeSwitchCompiler.hpp"
 #include "FactoredSwitchCompiler.hpp"
 #include "PcHoleFiller.hpp"
+#include "EmptyFdeDeleter.hpp"
 #include "ConseqEquivFilter.hpp"
+#include "OverriddenRowFilter.hpp"
 
 #include "settings.hpp"
 
@@ -65,6 +67,10 @@ MainOptions options_parse(int argc, char** argv) {
         else if(option == "--enable-deref-arg") {
             settings::enable_deref_arg = true;
         }
+
+        else if(option == "--keep-holes") {
+            settings::keep_holes = true;
+        }
     }
 
     if(!seen_switch_gen_policy) {
@@ -84,6 +90,7 @@ MainOptions options_parse(int argc, char** argv) {
              << argv[0]
              << " [--switch-per-func | --global-switch]"
              << " [--enable-deref-arg]"
+             << " [--keep-holes]"
              << " [--pc-list PC_LIST_FILE] elf_path"
              << endl;
     }
@@ -98,9 +105,11 @@ int main(int argc, char** argv) {
     SimpleDwarf parsed_dwarf = DwarfReader(opts.elf_path).read();
 
     SimpleDwarf filtered_dwarf =
-        PcHoleFiller()(
+        PcHoleFiller(!settings::keep_holes)(
+        EmptyFdeDeleter()(
+        OverriddenRowFilter()(
         ConseqEquivFilter()(
-            parsed_dwarf));
+            parsed_dwarf))));
 
     FactoredSwitchCompiler* sw_compiler = new FactoredSwitchCompiler(1);
     CodeGenerator code_gen(

src/plt_std_expr.hpp

@@ -0,0 +1,63 @@
+#pragma once
+
+#include <dwarfpp/expr.hpp>
+
+static const dwarf::encap::loc_expr REFERENCE_PLT_EXPR(
+        std::vector<dwarf::encap::expr_instr> {
+            {
+               {
+                   .lr_atom  = 0x77,
+                   .lr_number = 8,
+                   .lr_number2 = 0,
+                   .lr_offset = 0
+               },
+               {
+                   .lr_atom  = 0x80,
+                   .lr_number = 0,
+                   .lr_number2 = 0,
+                   .lr_offset = 2
+               },
+               {
+                   .lr_atom  = 0x3f,
+                   .lr_number = 15,
+                   .lr_number2 = 0,
+                   .lr_offset = 4
+               },
+               {
+                   .lr_atom  = 0x1a,
+                   .lr_number = 0,
+                   .lr_number2 = 0,
+                   .lr_offset = 5
+               },
+               {
+                   .lr_atom  = 0x3b,
+                   .lr_number = 11,
+                   .lr_number2 = 0,
+                   .lr_offset = 6
+               },
+               {
+                   .lr_atom  = 0x2a,
+                   .lr_number = 0,
+                   .lr_number2 = 0,
+                   .lr_offset = 7
+               },
+               {
+                   .lr_atom  = 0x33,
+                   .lr_number = 3,
+                   .lr_number2 = 0,
+                   .lr_offset = 8
+               },
+               {
+                   .lr_atom  = 0x24,
+                   .lr_number = 0,
+                   .lr_number2 = 0,
+                   .lr_offset = 9
+               },
+               {
+                   .lr_atom  = 0x22,
+                   .lr_number = 0,
+                   .lr_number2 = 0,
+                   .lr_offset = 10
+               }
+            }
+            });

src/settings.cpp

@@ -4,4 +4,5 @@ namespace settings {
     SwitchGenerationPolicy switch_generation_policy = SGP_SwitchPerFunc;
     std::string pc_list = "";
     bool enable_deref_arg = false;
+    bool keep_holes = false;
 }

src/settings.hpp

@@ -16,4 +16,6 @@ namespace settings {
     extern SwitchGenerationPolicy switch_generation_policy;
     extern std::string pc_list;
     extern bool enable_deref_arg;
+    extern bool keep_holes; /**< Keep holes between FDEs. Larger eh_elf files,
+                              but more accurate unwinding. */
 }

stats/.gitignore

@@ -0,0 +1,3 @@
+venv
+elf_data*
+gathered

stats/README.md

@@ -0,0 +1,11 @@
+# Statistical scripts
+
+Computes stats about a whole lot of stuff.
+
+## Setup
+
+```sh
+  virtualenv -p python3 venv  # Do this only once
+  source venv/bin/activate  # Do this for every new shell working running the script
+  pip install -r requirements.txt  # Do this only once
+```

stats/fde_stats.py

@@ -0,0 +1,106 @@
+#!/usr/bin/env python3
+
+from stats_accu import StatsAccumulator
+import gather_stats
+
+import argparse
+import sys
+
+
+class Config:
+    def __init__(self):
+        args = self.parse_args()
+        self._cores = args.cores
+        self.feature = args.feature
+
+        if args.feature == 'gather':
+            self.output = args.output
+
+        elif args.feature == 'sample':
+            self.size = int(args.size)
+            self.output = args.output
+
+        elif args.feature == 'analyze':
+            self.data_file = args.data_file
+
+    @property
+    def cores(self):
+        if self._cores <= 0:
+            return None
+        return self._cores
+
+    def parse_args(self):
+        parser = argparse.ArgumentParser(
+            description="Gather statistics about system-related ELFs")
+
+        parser.add_argument('--cores', '-j', default=1, type=int,
+                            help=("Use N cores for processing. Defaults to "
+                                  "1. 0 to use up all cores."))
+
+        subparsers = parser.add_subparsers(help='Subcommands')
+
+        # Sample stats
+        parser_sample = subparsers.add_parser(
+            'sample',
+            help='Same as gather, but for a random subset of files')
+        parser_sample.set_defaults(feature='sample')
+        parser_sample.add_argument('--size', '-n',
+                                   default=1000,
+                                   help=('Pick this number of files'))
+        parser_sample.add_argument('--output', '-o',
+                                   default='elf_data',
+                                   help=('Output data to this file. Defaults '
+                                         'to "elf_data"'))
+
+        # Gather stats
+        parser_gather = subparsers.add_parser(
+            'gather',
+            help=('Gather system data into a file, to allow multiple '
+                  'analyses without re-scanning the whole system.'))
+        parser_gather.set_defaults(feature='gather')
+        parser_gather.add_argument('--output', '-o',
+                                   default='elf_data',
+                                   help=('Output data to this file. Defaults '
+                                         'to "elf_data"'))
+
+        # Analyze stats
+        parser_analyze = subparsers.add_parser(
+            'analyze',
+            help='Analyze data gathered by a previous run.')
+        parser_analyze.set_defaults(feature='analyze')
+        parser_analyze.add_argument('data_file',
+                                    default='elf_data',
+                                    help=('Analyze this data file. Defaults '
+                                          'to "elf_data".'))
+        # TODO histogram?
+
+        out = parser.parse_args()
+        if 'feature' not in out:
+            print("No subcommand specified.", file=sys.stderr)
+            parser.print_usage(file=sys.stderr)
+            sys.exit(1)
+
+        return out
+
+
+def main():
+    config = Config()
+
+    if config.feature == 'gather':
+        stats_accu = gather_stats.gather_system_files(config)
+        stats_accu.dump(config.output)
+
+    elif config.feature == 'sample':
+        stats_accu = gather_stats.gather_system_files(
+            config,
+            sample_size=config.size)
+        stats_accu.dump(config.output)
+
+    elif config.feature == 'analyze':
+        print("Not implemented", file=sys.stderr)
+        stats_accu = StatsAccumulator.load(config.data_file)
+        sys.exit(1)
+
+
+if __name__ == '__main__':
+    main()

stats/gather_stats.py

@@ -0,0 +1,147 @@
+from elftools.common.exceptions import DWARFError
+from pyelftools_overlay import system_elfs, get_cfi
+from elftools.dwarf import callframe
+import concurrent.futures
+import random
+
+
+from stats_accu import \
+    StatsAccumulator, SingleFdeData, FdeData, DwarfInstr
+
+
+class ProcessWrapper:
+    def __init__(self, fct):
+        self._fct = fct
+
+    def __call__(self, elf_descr):
+        try:
+            path, elftype = elf_descr
+
+            print("Processing {}…".format(path))
+
+            cfi = get_cfi(path)
+            if not cfi:
+                return None
+
+            return self._fct(path, elftype, cfi)
+        except DWARFError:
+            return None
+
+
+def process_wrapper(fct):
+    return ProcessWrapper(fct)
+
+
+@process_wrapper
+def process_elf(path, elftype, cfi):
+    ''' Process a single file '''
+
+    data = FdeData()
+
+    for entry in cfi:
+        if isinstance(entry, callframe.CIE):  # Is a CIE
+            process_cie(entry, data)
+        elif isinstance(entry, callframe.FDE):  # Is a FDE
+            process_fde(entry, data)
+
+    return SingleFdeData(path, elftype, data)
+
+
+def incr_cell(table, key):
+    ''' Increments table[key], or sets it to 1 if unset '''
+    if key in table:
+        table[key] += 1
+    else:
+        table[key] = 1
+
+
+def process_cie(cie, data):
+    ''' Process a CIE '''
+    pass  # Nothing needed from a CIE
+
+
+def process_fde(fde, data):
+    ''' Process a FDE '''
+    data.fde_count += 1
+
+    decoded = fde.get_decoded()
+    row_count = len(decoded.table)
+    incr_cell(data.fde_with_lines, row_count)
+
+    for row in decoded.table:
+        process_reg(data.regs.cfa, row['cfa'])
+        for entry in row:
+            if isinstance(entry, int):
+                process_reg(data.regs.regs[entry], row[entry])
+
+
+def process_reg(out_reg, reg_def):
+    ''' Process a register '''
+    if isinstance(reg_def, callframe.CFARule):
+        if reg_def.reg is not None:
+            out_reg.regs[reg_def.reg] += 1
+        else:
+            pass  # TODO exprs
+    else:
+        incr_cell(out_reg.instrs, DwarfInstr.of_pyelf(reg_def.type))
+        if reg_def.type == callframe.RegisterRule.REGISTER:
+            out_reg.regs[reg_def.arg] += 1
+        elif (reg_def.type == callframe.RegisterRule.EXPRESSION) \
+                or (reg_def.type == callframe.RegisterRule.VAL_EXPRESSION):
+            pass  # TODO exprs
+
+
+def gather_system_files(config, sample_size=None):
+    stats_accu = StatsAccumulator()
+
+    elf_list = []
+    for elf_path in system_elfs():
+        elf_list.append(elf_path)
+
+    if sample_size is not None:
+        elf_list_sampled = random.sample(elf_list, sample_size)
+        elf_list = elf_list_sampled
+
+    if config.cores > 1:
+        with concurrent.futures.ProcessPoolExecutor(max_workers=config.cores)\
+                as executor:
+            for fde in executor.map(process_elf, elf_list):
+                stats_accu.add_fde(fde)
+    else:
+        for elf in elf_list:
+            stats_accu.add_fde(process_elf(elf))
+
+    return stats_accu
+
+
+def map_system_files(mapper, sample_size=None, cores=None, include=None,
+                     elflist=None):
+    ''' `mapper` must take (path, elf_type, cfi) '''
+    if cores is None:
+        cores = 1
+    if include is None:
+        include = []
+
+    mapper = process_wrapper(mapper)
+
+    if elflist is None:
+        elf_list = []
+        for elf_path in system_elfs():
+            elf_list.append(elf_path)
+
+        if sample_size is not None:
+            elf_list_sampled = random.sample(elf_list, sample_size)
+            elf_list = elf_list_sampled
+
+        elf_list += list(map(lambda x: (x, None), include))
+    else:
+        elf_list = elflist
+
+    if cores > 1:
+        with concurrent.futures.ProcessPoolExecutor(max_workers=cores)\
+                as executor:
+            out = executor.map(mapper, elf_list)
+    else:
+        out = map(mapper, elf_list)
+
+    return out, elf_list

stats/helpers.py

@@ -0,0 +1,228 @@
+from elftools.dwarf import callframe
+import gather_stats
+import itertools
+import functools
+
+REGS_IDS = {
+    'RAX': 0,
+    'RDX': 1,
+    'RCX': 2,
+    'RBX': 3,
+    'RSI': 4,
+    'RDI': 5,
+    'RBP': 6,
+    'RSP': 7,
+    'R8':  8,
+    'R9':  9,
+    'R10': 10,
+    'R11': 11,
+    'R12': 12,
+    'R13': 13,
+    'R14': 14,
+    'R15': 15,
+    'RIP': 16
+}
+
+ID_TO_REG = [
+    'RAX',
+    'RDX',
+    'RCX',
+    'RBX',
+    'RSI',
+    'RDI',
+    'RBP',
+    'RSP',
+    'R8',
+    'R9',
+    'R10',
+    'R11',
+    'R12',
+    'R13',
+    'R14',
+    'R15',
+    'RIP',
+]
+
+HANDLED_REGS = list(map(lambda x: REGS_IDS[x], [
+    'RIP',
+    'RSP',
+    'RBP',
+    'RBX',
+]))
+
+ONLY_HANDLED_REGS = True  # only analyzed handled regs columns
+
+PLT_EXPR = [119, 8, 128, 0, 63, 26, 59, 42, 51, 36, 34]  # Handled exp
+
+
+def accumulate_regs(reg_list):
+    out = [0] * 17
+    for lst in reg_list:
+        for pos in range(len(lst)):
+            out[pos] += lst[pos]
+
+    return out
+
+
+def filter_none(lst):
+    for x in lst:
+        if x:
+            yield x
+
+
+def deco_filter_none(fct):
+    def wrap(lst):
+        return fct(filter_none(lst))
+    return wrap
+
+
+class FdeProcessor:
+    def __init__(self, fct, reducer=None):
+        self._fct = fct
+        self._reducer = reducer
+
+    def __call__(self, path, elftype, cfi):
+        out = []
+        for entry in cfi:
+            if isinstance(entry, callframe.FDE):
+                decoded = entry.get_decoded()
+                out.append(self._fct(path, entry, decoded))
+        if self._reducer is not None and len(out) >= 2:
+            out = [self._reducer(out)]
+        return out
+
+
+class FdeProcessorReduced:
+    def __init__(self, reducer):
+        self._reducer = reducer
+
+    def __call__(self, fct):
+        return FdeProcessor(fct, self._reducer)
+
+
+def fde_processor(fct):
+    return FdeProcessor(fct)
+
+
+def fde_processor_reduced(reducer):
+    return FdeProcessorReduced(reducer)
+
+
+def is_handled_expr(expr):
+    if expr == PLT_EXPR:
+        return True
+
+    if len(expr) == 2 and 0x70 <= expr[0] <= 0x89:
+        if expr[0] - 0x70 in HANDLED_REGS:
+            return True
+    return False
+
+
+# @fde_processor
+def find_non_cfa(path, fde, decoded):
+    regs_seen = 0
+    non_handled_regs = 0
+    non_handled_exp = 0
+    cfa_dat = [0, 0]  # Seen, expr
+    rule_type = {
+        callframe.RegisterRule.UNDEFINED: 0,
+        callframe.RegisterRule.SAME_VALUE: 0,
+        callframe.RegisterRule.OFFSET: 0,
+        callframe.RegisterRule.VAL_OFFSET: 0,
+        callframe.RegisterRule.REGISTER: 0,
+        callframe.RegisterRule.EXPRESSION: 0,
+        callframe.RegisterRule.VAL_EXPRESSION: 0,
+        callframe.RegisterRule.ARCHITECTURAL: 0,
+    }
+    problematic_paths = set()
+
+    for row in decoded.table:
+        for entry in row:
+            reg_def = row[entry]
+
+            if entry == 'cfa':
+                cfa_dat[0] += 1
+                if reg_def.expr:
+                    cfa_dat[1] += 1
+                    if not is_handled_expr(reg_def.expr):
+                        non_handled_exp += 1
+                        problematic_paths.add(path)
+                elif reg_def:
+                    if reg_def.reg not in HANDLED_REGS:
+                        non_handled_regs += 1
+                        problematic_paths.add(path)
+            if not isinstance(entry, int):  # CFA or PC
+                continue
+
+            if ONLY_HANDLED_REGS and entry not in HANDLED_REGS:
+                continue
+
+            rule_type[reg_def.type] += 1
+            reg_rule = reg_def.type
+
+            if reg_rule in [callframe.RegisterRule.OFFSET,
+                            callframe.RegisterRule.VAL_OFFSET]:
+                regs_seen += 1  # CFA
+            elif reg_rule == callframe.RegisterRule.REGISTER:
+                regs_seen += 1
+                if reg_def.arg not in HANDLED_REGS:
+                    problematic_paths.add(path)
+                    non_handled_regs += 1
+            elif reg_rule in [callframe.RegisterRule.EXPRESSION,
+                              callframe.RegisterRule.VAL_EXPRESSION]:
+                expr = reg_def.arg
+                if not is_handled_expr(reg_def.arg):
+                    problematic_paths.add(path)
+                    with open('/tmp/exprs', 'a') as handle:
+                        handle.write('[{} - {}] {}\n'.format(
+                            path, fde.offset,
+                            ', '.join(map(lambda x: hex(x), expr))))
+                    non_handled_exp += 1
+
+    return (regs_seen, non_handled_regs, non_handled_exp, rule_type, cfa_dat,
+            problematic_paths)
+
+
+def reduce_non_cfa(lst):
+    def merge_dict(d1, d2):
+        for x in d1:
+            d1[x] += d2[x]
+        return d1
+
+    def merge_list(l1, l2):
+        out = []
+        for pos in range(len(l1)):  # Implicit assumption len(l1) == len(l2)
+            out.append(l1[pos] + l2[pos])
+        return out
+
+    def merge_elts(accu, elt):
+        accu_regs, accu_nh, accu_exp, accu_rt, accu_cfa, accu_paths = accu
+        elt_regs, elt_nh, elt_exp, elt_rt, elt_cfa, elf_paths = elt
+        return (
+            accu_regs + elt_regs,
+            accu_nh + elt_nh,
+            accu_exp + elt_exp,
+            merge_dict(accu_rt, elt_rt),
+            merge_list(accu_cfa, elt_cfa),
+            accu_paths.union(elf_paths),
+        )
+
+    return functools.reduce(merge_elts, lst)
+
+
+@deco_filter_none
+def flatten_non_cfa(result):
+    flat = itertools.chain.from_iterable(result)
+    out = reduce_non_cfa(flat)
+    out_cfa = {
+        'seen': out[4][0],
+        'expr': out[4][1],
+        'offset': out[4][0] - out[4][1],
+    }
+    out = (out[0],
+           (out[1], out[0] + out_cfa['offset']),
+           (out[2], out[3]['EXPRESSION'] + out_cfa['expr']),
+           out[3],
+           out_cfa,
+           out[5])
+    return out

stats/pyelftools_overlay.py

@@ -0,0 +1,110 @@
+""" Overlay of PyElfTools for quick access to what we want here """
+
+from elftools.elf.elffile import ELFFile
+from elftools.common.exceptions import ELFError, DWARFError
+from stats_accu import ElfType
+import os
+
+
+ELF_BLACKLIST = [
+    '/usr/lib/libavcodec.so',
+]
+
+
+def get_cfi(path):
+    ''' Get the CFI entries from the ELF at the provided path '''
+
+    try:
+        with open(path, 'rb') as file_handle:
+            elf_file = ELFFile(file_handle)
+
+            if not elf_file.has_dwarf_info():
+                print("No DWARF")
+                return None
+
+            dw_info = elf_file.get_dwarf_info()
+            if dw_info.has_CFI():
+                cfis = dw_info.CFI_entries()
+            elif dw_info.has_EH_CFI():
+                cfis = dw_info.EH_CFI_entries()
+            else:
+                print("No CFI")
+                return None
+    except ELFError:
+        print("ELF Error")
+        return None
+    except DWARFError:
+        print("DWARF Error")
+        return None
+    except PermissionError:
+        print("Permission Error")
+        return None
+    except KeyError:
+        print("Key Error")
+        return None
+
+    return cfis
+
+
+def system_elfs():
+    ''' Iterator over system libraries '''
+
+    def readlink_rec(path):
+        if not os.path.islink(path):
+            return path
+
+        return readlink_rec(
+            os.path.join(os.path.dirname(path),
+                         os.readlink(path)))
+
+    sysbin_dirs = [
+        ('/lib', ElfType.ELF_LIB),
+        ('/usr/lib', ElfType.ELF_LIB),
+        ('/usr/local/lib', ElfType.ELF_LIB),
+        ('/bin', ElfType.ELF_BINARY),
+        ('/usr/bin', ElfType.ELF_BINARY),
+        ('/usr/local/bin', ElfType.ELF_BINARY),
+        ('/sbin', ElfType.ELF_BINARY),
+    ]
+    to_explore = sysbin_dirs
+
+    seen_elfs = set()
+
+    while to_explore:
+        bindir, elftype = to_explore.pop()
+
+        if not os.path.isdir(bindir):
+            continue
+
+        for direntry in os.scandir(bindir):
+            if not direntry.is_file():
+                if direntry.is_dir():
+                    to_explore.append((direntry.path, elftype))
+                continue
+
+            canonical_name = readlink_rec(direntry.path)
+            for blacked in ELF_BLACKLIST:
+                if canonical_name.startswith(blacked):
+                    continue
+            if canonical_name in seen_elfs:
+                continue
+
+            valid_elf = True
+            try:
+                with open(canonical_name, 'rb') as handle:
+                    magic_bytes = handle.read(4)
+                    if magic_bytes != b'\x7fELF':
+                        valid_elf = False
+                    elf_class = handle.read(1)
+                    if elf_class != b'\x02':  # ELF64
+                        valid_elf = False
+            except Exception:
+                continue
+            if not valid_elf:
+                continue
+
+            if not os.path.isfile(canonical_name):
+                continue
+
+            seen_elfs.add(canonical_name)
+            yield (canonical_name, elftype)

stats/requirements.txt

@@ -0,0 +1 @@
+git+https://github.com/eliben/pyelftools

stats/stats_accu.py

@@ -0,0 +1,263 @@
+from elftools.dwarf import callframe
+import enum
+import subprocess
+import re
+import json
+import collections
+
+from math import ceil
+
+
+class ProportionFinder:
+    ''' Finds figures such as median, etc. on the original structure of a
+    dictionnary mapping a value to its occurrence count '''
+
+    def __init__(self, count_per_value):
+        self.cumulative = []
+        prev_count = 0
+        for key in sorted(count_per_value.keys()):
+            n_count = prev_count + count_per_value[key]
+            self.cumulative.append(
+                (key, n_count))
+            prev_count = n_count
+
+        self.elem_count = prev_count
+
+    def find_at_proportion(self, proportion):
+        if not self.cumulative:  # Empty list
+            return None
+
+        low_bound = ceil(self.elem_count * proportion)
+
+        def binsearch(beg, end):
+            med = ceil((beg + end) / 2)
+
+            if beg + 1 == end:
+                return self.cumulative[beg][0]
+
+            if self.cumulative[med - 1][1] < low_bound:
+                return binsearch(med, end)
+            return binsearch(beg, med)
+
+        return binsearch(0, len(self.cumulative))
+
+
+def elf_so_deps(path):
+    ''' Get the list of shared objects dependencies of the given ELF object.
+    This is obtained by running `ldd`. '''
+
+    deps_list = []
+
+    try:
+        ldd_output = subprocess.check_output(['/usr/bin/ldd', path]) \
+            .decode('utf-8')
+        ldd_re = re.compile(r'^.* => (.*) \(0x[0-9a-fA-F]*\)$')
+
+        ldd_lines = ldd_output.strip().split('\n')
+        for line in ldd_lines:
+            line = line.strip()
+            match = ldd_re.match(line)
+            if match is None:
+                continue  # Just ignore that line — it might be eg. linux-vdso
+            deps_list.append(match.group(1))
+
+        return deps_list
+
+    except subprocess.CalledProcessError as exn:
+        raise Exception(
+            ("Cannot get dependencies for {}: ldd terminated with exit code "
+             "{}.").format(path, exn.returncode))
+
+
+class ElfType(enum.Enum):
+    ELF_LIB = enum.auto()
+    ELF_BINARY = enum.auto()
+
+
+class DwarfInstr(enum.Enum):
+    @staticmethod
+    def of_pyelf(val):
+        _table = {
+            callframe.RegisterRule.UNDEFINED: DwarfInstr.INSTR_UNDEF,
+            callframe.RegisterRule.SAME_VALUE: DwarfInstr.INSTR_SAME_VALUE,
+            callframe.RegisterRule.OFFSET: DwarfInstr.INSTR_OFFSET,
+            callframe.RegisterRule.VAL_OFFSET: DwarfInstr.INSTR_VAL_OFFSET,
+            callframe.RegisterRule.REGISTER: DwarfInstr.INSTR_REGISTER,
+            callframe.RegisterRule.EXPRESSION: DwarfInstr.INSTR_EXPRESSION,
+            callframe.RegisterRule.VAL_EXPRESSION:
+                DwarfInstr.INSTR_VAL_EXPRESSION,
+            callframe.RegisterRule.ARCHITECTURAL:
+                DwarfInstr.INSTR_ARCHITECTURAL,
+        }
+        return _table[val]
+
+    INSTR_UNDEF = enum.auto()
+    INSTR_SAME_VALUE = enum.auto()
+    INSTR_OFFSET = enum.auto()
+    INSTR_VAL_OFFSET = enum.auto()
+    INSTR_REGISTER = enum.auto()
+    INSTR_EXPRESSION = enum.auto()
+    INSTR_VAL_EXPRESSION = enum.auto()
+    INSTR_ARCHITECTURAL = enum.auto()
+
+
+def intify_dict(d):
+    out = {}
+    for key in d:
+        try:
+            nKey = int(key)
+        except Exception:
+            nKey = key
+
+        try:
+            out[nKey] = int(d[key])
+        except ValueError:
+            out[nKey] = d[key]
+    return out
+
+
+class RegData:
+    def __init__(self, instrs=None, regs=None, exprs=None):
+        if instrs is None:
+            instrs = {}
+        if regs is None:
+            regs = [0]*17
+        if exprs is None:
+            exprs = {}
+        self.instrs = intify_dict(instrs)
+        self.regs = regs
+        self.exprs = intify_dict(exprs)
+
+    @staticmethod
+    def map_dict_keys(fnc, dic):
+        out = {}
+        for key in dic:
+            out[fnc(key)] = dic[key]
+        return out
+
+    def dump(self):
+        return {
+            'instrs': RegData.map_dict_keys(lambda x: x.value, self.instrs),
+            'regs': self.regs,
+            'exprs': self.exprs,
+        }
+
+    @staticmethod
+    def load(data):
+        return RegData(
+            instrs=RegData.map_dict_keys(
+                lambda x: DwarfInstr(int(x)),
+                data['instrs']),
+            regs=data['regs'],
+            exprs=data['exprs'],
+        )
+
+
+class RegsList:
+    def __init__(self, cfa=None, regs=None):
+        if cfa is None:
+            cfa = RegsList.fresh_reg()
+        if regs is None:
+            regs = [RegsList.fresh_reg() for _ in range(17)]
+        self.cfa = cfa
+        self.regs = regs
+
+    @staticmethod
+    def fresh_reg():
+        return RegData()
+
+    def dump(self):
+        return {
+            'cfa': RegData.dump(self.cfa),
+            'regs': [RegData.dump(r) for r in self.regs],
+        }
+
+    @staticmethod
+    def load(data):
+        return RegsList(
+            cfa=RegData.load(data['cfa']),
+            regs=[RegData.load(r) for r in data['regs']],
+        )
+
+
+class FdeData:
+    def __init__(self, fde_count=0, fde_with_lines=None, regs=None):
+        if fde_with_lines is None:
+            fde_with_lines = {}
+        if regs is None:
+            regs = RegsList()
+
+        self.fde_count = fde_count
+        self.fde_with_lines = intify_dict(fde_with_lines)
+        self.regs = regs
+
+    def dump(self):
+        return {
+            'fde_count': self.fde_count,
+            'fde_with_lines': self.fde_with_lines,
+            'regs': self.regs.dump(),
+        }
+
+    @staticmethod
+    def load(data):
+        return FdeData(
+            fde_count=int(data['fde_count']),
+            fde_with_lines=data['fde_with_lines'],
+            regs=RegsList.load(data['regs']))
+
+
+class SingleFdeData:
+    def __init__(self, path, elf_type, data):
+        self.path = path
+        self.elf_type = elf_type
+        self.data = data  # < of type FdeData
+
+        self.gather_deps()
+
+    def gather_deps(self):
+        """ Collect ldd data on the binary """
+        # self.deps = elf_so_deps(self.path)
+        self.deps = []
+
+    def dump(self):
+        return {
+            'path': self.path,
+            'elf_type': self.elf_type.value,
+            'data': self.data.dump()
+        }
+
+    @staticmethod
+    def load(data):
+        return SingleFdeData(
+            data['path'],
+            ElfType(int(data['elf_type'])),
+            FdeData.load(data['data']))
+
+
+class StatsAccumulator:
+    def __init__(self):
+        self.fdes = []
+
+    def add_fde(self, fde_data):
+        if fde_data:
+            self.fdes.append(fde_data)
+
+    def get_fdes(self):
+        return self.fdes
+
+    def add_stats_accu(self, stats_accu):
+        for fde in stats_accu.get_fdes():
+            self.add_fde(fde)
+
+    def dump(self, path):
+        dict_form = [fde.dump() for fde in self.fdes]
+        with open(path, 'w') as handle:
+            handle.write(json.dumps(dict_form))
+
+    @staticmethod
+    def load(path):
+        with open(path, 'r') as handle:
+            text = handle.read()
+        out = StatsAccumulator()
+        out.fdes = [SingleFdeData.load(data) for data in json.loads(text)]
+        return out