# How to benchmark Beluga

## Run rosbag benchmark parameterizing the number of particles

This script will run the benchmark with the specified number of particles and record:

- The `timem` output: CPU usage, RSS memory, virtual memory.
- A rosbag with the reference and estimated trajectories.
- Perf events (optionally).

To run, use:

```bash
ros2 run beluga_benchmark parameterized_run <NUMBER_OF_PARTICLES_EXPERIMENT_1> <NUMBER_OF_PARTICLES_EXPERIMENT_2>
```

The results of the different runs will be stored in folders named `benchmark_${N_PARTICLES}_particles_output`, where `N_PARTICLES` are the numbers specified in the above command.

To run the same experiment using another AMCL node, e.g. nav2, use:

```bash
ros2 run beluga_benchmark parameterized_run <NUMBER_OF_PARTICLES_EXPERIMENT_1> <NUMBER_OF_PARTICLES_EXPERIMENT_2> --package nav2_amcl --executable amcl
```

For other options, e.g. using a different rosbag, see:

```bash
ros2 run beluga_benchmark parameterized_run --help
```

## Visualizing timem results of one run

Use the following command:

```
ros2 run beluga_benchmark timem_results <PATH_TO_OUTPUT_DIR_OF_RUN>
```

Where the specified directory needs to have a `timem-output.json` file, e.g. the output directory generated by `parameterized_run`.
The script will print cpu usage, peak rss and elapsed time.
It will also plot virtual and RSS memory over time.

## Visualizing estimated trajectory APE of one run

Use the following command:

```
evo_ape bag2 <PATH_TO_BAGFILE> /odometry/ground_truth /pose -p
```

For `nav2_amcl`, replace `/pose` with `/amcl_pose`.
The bagfiles generated by `parameterized_run` can be found in the generated output directories for each run.
This will print APE metrics (mean, median, max, std, rmse, etc), and also plot APE over time.

## Comparing parameterized runs

The following command allows to compare the results of different benchmarking runs in a single plot for each metric being measured.

This can be used to compare different `beluga_amcl` and/or `nav2_amcl` runs, or to compare the same node with different base configuration settings.

The command is
```
ros2 run beluga_benchmark compare_results -s <PATH1> -l <LABEL1> -s <PATH2> -l <LABEL2> ...
```
where `PATH1` and `PATH2` are the paths to the output directories of the benchmarking runs to compare, and `LABEL1` and `LABEL2` are the labels to use in the plot for each of them. Any number of runs can be added in the same plot by providing additional `-s <PATH> -l <LABEL>` pairs.

Additionally, the Y axis of the plot can be configured to be plotted in log scale using the `--use-logy` flag. If not specificied, the plot will default to linear scale.

Notice that the result paths passed to `compare_results` must be the ones where `parameterized_run` was run, i.e. the one containing the `benchmark_*_particles_output` directories.

The script will plot the following metrics vs the number of particles:

- RSS memory
- CPU usage
- APE mean
- APE median
- APE max
- APE rmse

## Saving benchmarking results to a CSV file

The following command allows to save the results of different benchmarking runs to a CSV file, assuming that
beam data has been stored in the `beam_beluga_seq` and `beam_nav2_amcl` directories, and
likelihood data has been stored in the `likelihood_beluga_seq` and `likelihood_nav2_amcl` directories.

```bash
ros2 run beluga_benchmark compare_results \
    -s beam_beluga_seq -l beluga          \
    -s beam_nav2_amcl  -l nav2            \
    --save-csv beam_model_data.csv

ros2 run beluga_benchmark compare_results \
    -s likelihood_beluga_seq -l beluga    \
    -s likelihood_nav2_amcl  -l nav2      \
    --save-csv likelihood_model_data.csv
```

The files `beam_model_data.csv` and `likelihood_model_data.csv` will be created in the current directory,
containing all of the data captured in the benchmark runs, where columns will have been prefixed with the
corresponding label passed to `compare_results` (e.g. `beluga_rss`, `nav2_rss`, etc).

## References

- https://github.com/NERSC/timemory
- https://github.com/MichaelGrupp/evo