Benchmarking Running Experiments

This tutorial shows how to design and run reproducible experiments that compare solvers across multiple scenes and parameter configurations. You’ll learn to collect structured results, save them as CSV files, and process large batches efficiently.

Goal

By the end of this tutorial, you will:

• Design experiment scenarios (solver + scene + parameters).

• Run batch experiments and collect results.

• Use CSV output for analysis and performance comparison.

• Scale experiments across multiple chunks for parallel processing.

What is an experiment?

An experiment is a repeatable comparison of one or more solvers on one or more scenes. Instead of manually clicking “Solve” in the Solver Viewer for each combination, you define a list of scenarios — each one specifies a solver, a scene, and parameter settings — and the framework runs them all, collects metrics, and saves the results to CSV.

This is essential when you want to:

• Compare two solvers on the same scene.

• Test parameter sensitivity (e.g., does increasing num_landmarks actually help?).

• Run the same benchmark across different hardware.

• Generate publication-ready data tables.

Understanding scenarios

A Scenario is a single experiment case that combines:

1. A solver class or instance.

2. A scene file (JSON path).

3. A dictionary of solver parameters.

When you run a batch of scenarios, each one executes independently, and the results are stored in separate CSV files.

Default metrics recorded

Every scenario automatically records:

• total_path_length — sum of all robot path lengths in the solution.

• makespan — total time taken by the solution (maximum time of any single robot).

• calc_time — wall-clock time spent computing the solution.

You can add custom metrics via result handlers.

Custom result handlers

Sometimes you need measurements beyond the defaults. Use result handlers to compute any metric you want from the solver’s roadmap or path collection.

Each handler is a function that receives:

• The path collection (solution).

• The solver object (so you can access internal state like roadmap size).

It should return a single numeric value.

Common examples:

• Roadmap size (number of nodes or edges).

• Graph density or connectivity measures.

• Distance from start to goal in the roadmap.

Step 1: Create a scenarios file

Start by creating a Python file that defines your experiments. Here’s a minimal example:

from discopygal.experiments.scenarios_runner import Scenario
from discopygal.solvers.rrt.drrt_star import dRRT_star

SCENARIOS = [
    Scenario(
        dRRT_star,
        "examples/scenes/2_discs_corridor.json",
        {"num_landmarks": 150}
    ),
]

That’s it! One scenario: run dRRT_star on the corridor scene with 150 landmarks.

Scaling up: multiple scenes and parameters

For a real benchmark, you probably want to test many combinations. Use itertools.product to create the cross-product of solvers, scenes, and parameters:

import itertools

from discopygal.experiments.scenarios_runner import Scenario
from discopygal.solvers.rrt.drrt_star import dRRT_star
from discopygal.solvers.rrt.rrt_star import RRT_star

# Define parameter values to test
landmark_counts = [50, 100, 150, 200]

scenes = [
    "examples/scenes/2_discs_corridor.json",
    "examples/scenes/2_pocket_maze.json",
    "examples/scenes/coffee_shop/coffee_shop.json",
]

solvers = [dRRT_star, RRT_star]

# Create all combinations
SCENARIOS = [
    Scenario(solver, scene, {"num_landmarks": landmarks})
    for solver, scene, landmarks in itertools.product(solvers, scenes, landmark_counts)
]

This creates 2 solvers × 3 scenes × 4 parameter values = 24 scenarios.

Adding custom metrics

Extend your scenarios file with a RESULT_HANDLERS dictionary:

RESULT_HANDLERS = {
    "roadmap_nodes": lambda path_collection, solver: len(solver.roadmap.nodes),
    "roadmap_edges": lambda path_collection, solver: len(solver.roadmap.edges),
    "solution_robots": lambda path_collection, solver: len(path_collection.paths),
}

These handlers run after every scenario and their results are added to the CSV output.

Real-world example

Here’s a complete, realistic scenarios file:

import itertools

from discopygal.experiments.scenarios_runner import Scenario
from discopygal.solvers.rrt.drrt_star import dRRT_star

# Test different parameter configurations
random_sample_counters = [0, 1, 10, 20, 50, 100]
landmark_counts = [100, 500, 1000]

scenes = [
    "examples/scenes/tunnels_disc.json",
    "examples/scenes/2_discs_corridor.json",
    "examples/scenes/2_pocket_maze_tight.json",
]

SCENARIOS = [
    Scenario(
        dRRT_star,
        scene,
        {
            "prm_num_landmarks": landmarks,
            "num_landmarks": 100,
            "random_sample_counter": random_sample_counter,
        },
    )
    for random_sample_counter, scene, landmarks in itertools.product(
        random_sample_counters, scenes, landmark_counts
    )
]

RESULT_HANDLERS = {
    "roadmap_size": lambda _, solver: len(solver.roadmap.nodes),
    "solution_is_optimal": lambda paths, _: 1 if paths.is_optimal() else 0,
}

This creates 6 × 3 × 3 = 54 scenarios. When run, it produces detailed CSV results showing how the random sampling counter affects planning performance.

Step 2: Run scenarios from Python

If you’re working in a Jupyter notebook or a standalone script, run experiments directly:

from discopygal.experiments.scenarios_runner import run_scenarios
from my_scenarios import SCENARIOS, RESULT_HANDLERS

run_scenarios(
    SCENARIOS,
    result_root_dir="results",
    result_handlers=RESULT_HANDLERS
)

This creates a results/ directory with timestamped subdirectories, each containing:

• all.csv — Summary of all scenarios.

• Per-scenario result files.

• Log files with solver output.

Step 3: Run from the command line

The CLI entry point is scenarios_runner. First, activate your environment, then:

Linux / macOSWindowsFrom source (any OS)

scenarios_runner results my_scenarios.py

This runs all scenarios defined in my_scenarios.py and saves results to the results/ directory.

Step 4: Advanced workload management

Resuming interrupted runs

If an experiment crashes halfway through, resume it without re-running completed scenarios:

scenarios_runner results my_scenarios.py resume

The runner skips scenarios that already have output files and continues from where it left off.

Splitting large experiments across machines

If you have 1000 scenarios and only one machine, break the work into chunks and run them in parallel:

# Split into 10 chunks, run chunk 3 on this machine
scenarios_runner results my_scenarios.py 10 3

# (On other machines, run chunks 1, 2, 4, 5, ..., 10)

Each machine produces results in its own subdirectory.

Merging chunk results

After all chunks finish, merge them into a single results directory:

scenarios_runner results my_scenarios.py 10 end

This consolidates all chunk CSV files into one all.csv summary.

Understanding output

CSV structure

Each run creates an all.csv with one row per scenario. Columns include:

• scenario_id — unique identifier for this scenario.

• solver — solver class name.

• scene — path to the scene file.

• total_path_length, makespan, calc_time — default metrics.

• Custom handler columns (e.g., roadmap_size) if you defined them.

You can load this in Pandas, Excel, or any analysis tool:

import pandas as pd

results = pd.read_csv("results/20240427_103020/all.csv")
print(results.groupby("solver")["calc_time"].mean())  # average compute time per solver

Tips and tricks

• Start small: Run 5–10 scenarios first to catch configuration errors before launching 1000.

• Parameter sweeps: Use itertools.product to explore high-dimensional parameter spaces systematically.

• Named scenarios: Add a name field to scenarios for clearer CSV output (if supported by your version).

• Version control: Commit your scenarios file to git so results are always reproducible.

• Parallel runs: Use the chunk mode on multiple machines or CPU cores to speed up large experiments.