ADACS:-Scaling-and-Profiling-notes.md

@@ -40,6 +40,87 @@ The effect of changing nlive from 1000 to 2000 and 4000 is measured, to determin
 
 # Profiling
 
+## Getting started
+
+Modifications to pBilby are required for detailed profiling. The necessary modifications have been applied in the `ADACS` branch. Three methods for profiling pBilby are used:
+
+1. cProfile
+2. pyInstrument
+3. Manual MPI timers
+
+See https://pythonspeed.com/articles/beyond-cprofile/ for a detailed comparison between cProfile and pyInstrument, and a setup guide. While these profilers are typically used to profile serial Python applications, they can also be used to profile MPI threads. This is done by initialising the profiler on every MPI task (or just the ones being profiled).
+
+The following Python script (a copy of `parallel_bilby_analysis`) is used to profile the code using pyInstrument:
+
+```
+__requires__ = 'parallel-bilby==0.1.4'
+import re
+import sys
+from pkg_resources import load_entry_point
+
+from mpi4py.MPI import COMM_WORLD as comm
+from pyinstrument import Profiler
+
+# Start profiler
+profiler = Profiler()
+profiler.start()
+
+print(f'MPI Rank {comm.rank} starting')
+
+# Start pbilby
+sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
+load_entry_point('parallel-bilby==0.1.4', 'console_scripts', 'parallel_bilby_analysis')()
+# Finish pbilby
+
+print(f'MPI Rank {comm.rank} finishing')
+
+# Stop profiler
+profiler.stop()
+
+# Write output
+output = profiler.output_html()
+with open(f'prof/{comm.rank:03}.html', 'w') as f:
+  f.write(output)
+```
+Each task will output the profiling results as an HTML file to the `prof` directory on completion. If the run crashes or is cancelled early, the results will not be saved.
+
+For cProfile, a similar script is used:
+```
+__requires__ = 'parallel-bilby==0.1.4'
+import re
+import sys
+from pkg_resources import load_entry_point
+
+from mpi4py.MPI import COMM_WORLD as comm
+import cProfile import Profile
+
+# Create and start profiler
+pr = Profile()
+pr.enable()
+
+print(f'MPI Rank {comm.rank} starting')
+
+# Start pbilby
+sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
+load_entry_point('parallel-bilby==0.1.4', 'console_scripts', 'parallel_bilby_analysis')()
+# Finish pbilby
+
+print(f'MPI Rank {comm.rank} finishing')
+
+# Finish profiling
+pr.disable()
+
+# Write profiling output
+filename = f'prof/pbilby.{comm.rank}.prof'
+pr.dump_stats(filename)
+```
+
+To start this script (e.g. saved as `pbilby.py`) as a Slurm job, simply call `python pbilby.py` instead of `parallel_bilby_analysis`, using the same arguments. For example:
+
+```
+mpirun python pbilby.py outdir/data/GW150914_data_dump.pickle --nact 10 --min-eff 3.0 --label GW150914_0 --outdir outdir/result --sampling-seed 1234
+```
+
 ## Modifications to the Schwimmbad library
 
 There is some misunderstanding of how the library should be called. The Schwimmbad documentation itself contains incorrect instructions.