.. _tutorial-benchmarks:

Benchmarking Running Experiments
================================

This tutorial shows how to design and run reproducible experiments that compare solvers across multiple scenes and parameter configurations. You'll learn to collect structured results, save them as CSV files, and process large batches efficiently.

Goal
----

By the end of this tutorial, you will:

* Design experiment scenarios (solver + scene + parameters).
* Run batch experiments and collect results.
* Use CSV output for analysis and performance comparison.
* Scale experiments across multiple chunks for parallel processing.

What is an experiment?
----------------------

An experiment is a repeatable comparison of one or more solvers on one or more scenes.
Instead of manually clicking "Solve" in the Solver Viewer for each combination, you define a list
of **scenarios** — each one specifies a solver, a scene, and parameter settings — and the
framework runs them all, collects metrics, and saves the results to CSV.

This is essential when you want to:

* Compare two solvers on the same scene.
* Test parameter sensitivity (e.g., does increasing ``num_landmarks`` actually help?).
* Run the same benchmark across different hardware.
* Generate publication-ready data tables.

Understanding scenarios
------------------------

A **Scenario** is a single experiment case that combines:

1. A solver class or instance.
2. A scene file (JSON path).
3. A dictionary of solver parameters.

When you run a batch of scenarios, each one executes independently, and the results are stored
in separate CSV files.

Default metrics recorded
~~~~~~~~~~~~~~~~~~~~~~~~

Every scenario automatically records:

* ``total_path_length`` — sum of all robot path lengths in the solution.
* ``makespan`` — total time taken by the solution (maximum time of any single robot).
* ``calc_time`` — wall-clock time spent computing the solution.

You can add custom metrics via **result handlers**.

Custom result handlers
~~~~~~~~~~~~~~~~~~~~~~

Sometimes you need measurements beyond the defaults. Use result handlers to compute any
metric you want from the solver's roadmap or path collection.

Each handler is a function that receives:

* The path collection (solution).
* The solver object (so you can access internal state like roadmap size).

It should return a single numeric value.

Common examples:

* Roadmap size (number of nodes or edges).
* Graph density or connectivity measures.
* Distance from start to goal in the roadmap.

Step 1: Create a scenarios file
-------------------------------

Start by creating a Python file that defines your experiments. Here's a minimal example:

.. code-block:: python

   from discopygal.experiments.scenarios_runner import Scenario
   from discopygal.solvers.rrt.drrt_star import dRRT_star

   SCENARIOS = [
       Scenario(
           dRRT_star,
           "examples/scenes/2_discs_corridor.json",
           {"num_landmarks": 150}
       ),
   ]

That's it! One scenario: run ``dRRT_star`` on the corridor scene with 150 landmarks.

Scaling up: multiple scenes and parameters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For a real benchmark, you probably want to test many combinations. Use ``itertools.product``
to create the cross-product of solvers, scenes, and parameters:

.. code-block:: python

   import itertools

   from discopygal.experiments.scenarios_runner import Scenario
   from discopygal.solvers.rrt.drrt_star import dRRT_star
   from discopygal.solvers.rrt.rrt_star import RRT_star

   # Define parameter values to test
   landmark_counts = [50, 100, 150, 200]

   scenes = [
       "examples/scenes/2_discs_corridor.json",
       "examples/scenes/2_pocket_maze.json",
       "examples/scenes/coffee_shop/coffee_shop.json",
   ]

   solvers = [dRRT_star, RRT_star]

   # Create all combinations
   SCENARIOS = [
       Scenario(solver, scene, {"num_landmarks": landmarks})
       for solver, scene, landmarks in itertools.product(solvers, scenes, landmark_counts)
   ]

This creates 2 solvers × 3 scenes × 4 parameter values = **24 scenarios**.

Adding custom metrics
~~~~~~~~~~~~~~~~~~~~~

Extend your scenarios file with a ``RESULT_HANDLERS`` dictionary:

.. code-block:: python

   RESULT_HANDLERS = {
       "roadmap_nodes": lambda path_collection, solver: len(solver.roadmap.nodes),
       "roadmap_edges": lambda path_collection, solver: len(solver.roadmap.edges),
       "solution_robots": lambda path_collection, solver: len(path_collection.paths),
   }

These handlers run after every scenario and their results are added to the CSV output.

Real-world example
~~~~~~~~~~~~~~~~~~

Here's a complete, realistic scenarios file:

.. code-block:: python

   import itertools

   from discopygal.experiments.scenarios_runner import Scenario
   from discopygal.solvers.rrt.drrt_star import dRRT_star

   # Test different parameter configurations
   random_sample_counters = [0, 1, 10, 20, 50, 100]
   landmark_counts = [100, 500, 1000]

   scenes = [
       "examples/scenes/tunnels_disc.json",
       "examples/scenes/2_discs_corridor.json",
       "examples/scenes/2_pocket_maze_tight.json",
   ]

   SCENARIOS = [
       Scenario(
           dRRT_star,
           scene,
           {
               "prm_num_landmarks": landmarks,
               "num_landmarks": 100,
               "random_sample_counter": random_sample_counter,
           },
       )
       for random_sample_counter, scene, landmarks in itertools.product(
           random_sample_counters, scenes, landmark_counts
       )
   ]

   RESULT_HANDLERS = {
       "roadmap_size": lambda _, solver: len(solver.roadmap.nodes),
       "solution_is_optimal": lambda paths, _: 1 if paths.is_optimal() else 0,
   }

This creates 6 × 3 × 3 = **54 scenarios**. When run, it produces detailed CSV results showing
how the random sampling counter affects planning performance.

Step 2: Run scenarios from Python
---------------------------------

If you're working in a Jupyter notebook or a standalone script, run experiments directly:

.. code-block:: python

   from discopygal.experiments.scenarios_runner import run_scenarios
   from my_scenarios import SCENARIOS, RESULT_HANDLERS

   run_scenarios(
       SCENARIOS,
       result_root_dir="results",
       result_handlers=RESULT_HANDLERS
   )

This creates a ``results/`` directory with timestamped subdirectories, each containing:

* ``all.csv`` — Summary of all scenarios.
* Per-scenario result files.
* Log files with solver output.

Step 3: Run from the command line
---------------------------------

The CLI entry point is ``scenarios_runner``. First, activate your environment, then:

.. tabs::

   .. tab:: Linux / macOS

      .. code-block:: bash

         scenarios_runner results my_scenarios.py

   .. tab:: Windows

      .. code-block:: powershell

         scenarios_runner.exe results my_scenarios.py

   .. tab:: From source (any OS)

      .. code-block:: bash

         python src/discopygal/experiments/run_experiment.py results my_scenarios.py

This runs all scenarios defined in ``my_scenarios.py`` and saves results to the ``results/`` directory.

Step 4: Advanced workload management
------------------------------------

Resuming interrupted runs
~~~~~~~~~~~~~~~~~~~~~~~~~

If an experiment crashes halfway through, resume it without re-running completed scenarios:

.. code-block:: bash

   scenarios_runner results my_scenarios.py resume

The runner skips scenarios that already have output files and continues from where it left off.

Splitting large experiments across machines
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you have 1000 scenarios and only one machine, break the work into chunks and run them in parallel:

.. code-block:: bash

   # Split into 10 chunks, run chunk 3 on this machine
   scenarios_runner results my_scenarios.py 10 3

   # (On other machines, run chunks 1, 2, 4, 5, ..., 10)

Each machine produces results in its own subdirectory.

Merging chunk results
~~~~~~~~~~~~~~~~~~~~~

After all chunks finish, merge them into a single results directory:

.. code-block:: bash

   scenarios_runner results my_scenarios.py 10 end

This consolidates all chunk CSV files into one ``all.csv`` summary.

Understanding output
--------------------

CSV structure
~~~~~~~~~~~~~

Each run creates an ``all.csv`` with one row per scenario. Columns include:

* ``scenario_id`` — unique identifier for this scenario.
* ``solver`` — solver class name.
* ``scene`` — path to the scene file.
* ``total_path_length``, ``makespan``, ``calc_time`` — default metrics.
* Custom handler columns (e.g., ``roadmap_size``) if you defined them.

You can load this in Pandas, Excel, or any analysis tool:

.. code-block:: python

   import pandas as pd

   results = pd.read_csv("results/20240427_103020/all.csv")
   print(results.groupby("solver")["calc_time"].mean())  # average compute time per solver

Tips and tricks
---------------

* **Start small**: Run 5–10 scenarios first to catch configuration errors before launching 1000.
* **Parameter sweeps**: Use ``itertools.product`` to explore high-dimensional parameter spaces systematically.
* **Named scenarios**: Add a ``name`` field to scenarios for clearer CSV output (if supported by your version).
* **Version control**: Commit your scenarios file to git so results are always reproducible.
* **Parallel runs**: Use the chunk mode on multiple machines or CPU cores to speed up large experiments.