Update ADACS: Scaling and Profiling notes authored by Conrad Chan's avatar Conrad Chan
......@@ -121,9 +121,15 @@ To start this script (e.g. saved as `pbilby.py`) as a Slurm job, simply call `py
mpirun python pbilby.py outdir/data/GW150914_data_dump.pickle --nact 10 --min-eff 3.0 --label GW150914_0 --outdir outdir/result --sampling-seed 1234
```
To perform manual MPI timing (hardcoded into the modified Schwimmbad library in the ADACS branch), add the argument `--mpi-timing 1`, which will give a printout at the end of the run in the log file. To record MPI timings at regular 60 seconds intervals, add `--mpi-timing-interval 60`.
Note: The MPI timing printout at the end of the run is a sum of all worker tasks, while the periodic timing intervals are only the timings for **one** worker task. This is to prevent necessary MPI communication during the run.
Using excessively short intervals will result in large overheads. The timing results for each interval are saved in a JSON file `mpi_worker_timing.json` at the end of the run.
## Modifications to the Schwimmbad library
There is some misunderstanding of how the library should be called. The Schwimmbad documentation itself contains incorrect instructions.
There is some misunderstanding of how the library should be called in the original implementation. The Schwimmbad documentation itself contains incorrect instructions.
The lines of code for the worker tasks
```
......
......