Skip to content

Timing

Time a simulation across core counts and report the speed-up (Chapter 7).

simstreams.benchmark

Timing helpers for speed-up experiments (Chapter 7).

Time the whole simulation call, including pool start-up, discard warm-up runs, and summarize with the median. These are the functions behind the timing tables in Chapters 4 and 5.

benchmark 

benchmark(func, repeats=10, warmup=1)

Benchmark a function that takes no arguments.

Parameters:

Name Type Description Default
func callable

Function containing the complete simulation call.

 required
repeats int

Number of measured runs.

 10
warmup int

Number of unmeasured warm-up runs.

 1

Returns:

Type Description
dict

Summary statistics (median, mean, stdev, minimum, maximum, q1, q3) and all measured elapsed times in seconds.
Source code in src/simstreams/benchmark.py 
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
def benchmark(func, repeats=10, warmup=1):
    """Benchmark a function that takes no arguments.

    Parameters
    ----------
    func : callable
        Function containing the complete simulation call.
    repeats : int
        Number of measured runs.
    warmup : int
        Number of unmeasured warm-up runs.

    Returns
    -------
    dict
        Summary statistics (``median``, ``mean``, ``stdev``, ``minimum``,
        ``maximum``, ``q1``, ``q3``) and all measured elapsed ``times`` in
        seconds.
    """
    # Warm-up runs are not included in the results.
    for _ in range(warmup):
        func()

    times = []

    for _ in range(repeats):
        start = perf_counter()
        func()
        elapsed = perf_counter() - start
        times.append(elapsed)

    return {
        "median": statistics.median(times),
        "mean": statistics.mean(times),
        "stdev": statistics.stdev(times) if repeats > 1 else 0.0,
        "minimum": min(times),
        "maximum": max(times),
        "q1": float(np.percentile(times, 25)),
        "q3": float(np.percentile(times, 75)),
        "times": times,
    }

benchmark_cores 

benchmark_cores(
    run_simulation, core_counts, repeats=10, warmup=1
)

Benchmark a simulation over several core counts.

Parameters:

Name Type Description Default
run_simulation callable

A function accepting n_cores as its only argument that runs the complete simulation. Typically a thin wrapper around simstreams.sim_template with every other argument fixed.

 required
core_counts iterable

Core counts to benchmark. The first one is the baseline for the speed-up.

 required
repeats int

Number of measured runs at each core count.

 10
warmup int

Number of warm-up runs at each core count.

 1

Returns:

Type Description
list of dict

One benchmark summary per core count, with n_cores, speedup (baseline median over this median), and efficiency (speedup / n_cores) added.
Source code in src/simstreams/benchmark.py 
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
def benchmark_cores(run_simulation, core_counts, repeats=10, warmup=1):
    """Benchmark a simulation over several core counts.

    Parameters
    ----------
    run_simulation : callable
        A function accepting ``n_cores`` as its only argument that runs the
        complete simulation. Typically a thin wrapper around
        ``simstreams.sim_template`` with every other argument fixed.
    core_counts : iterable
        Core counts to benchmark. The first one is the baseline for the
        speed-up.
    repeats : int
        Number of measured runs at each core count.
    warmup : int
        Number of warm-up runs at each core count.

    Returns
    -------
    list of dict
        One ``benchmark`` summary per core count, with ``n_cores``,
        ``speedup`` (baseline median over this median), and ``efficiency``
        (``speedup / n_cores``) added.
    """
    results = []

    for n_cores in core_counts:
        summary = benchmark(
            partial(run_simulation, n_cores),
            repeats=repeats,
            warmup=warmup,
        )

        summary["n_cores"] = n_cores
        results.append(summary)

    serial_time = results[0]["median"]

    for result in results:
        result["speedup"] = serial_time / result["median"]
        result["efficiency"] = result["speedup"] / result["n_cores"]

    return results