Skip to content

Detailed MPI timing and improved communication

Conrad Chan requested to merge conrad.chan/parallel_bilby:overlapping-communication into dev

Timing routine for MPI

  • In Schwimmbad, measure time on workers spent in serial code, MPI recv, MPI send, barriers, and actual compute time
  • Print out cumulative wall time at the end
  • Enable using --mpi-timing 1
  • Output periodic timing snapshots from a single worker (write to .json) using --mpi-timing-interval INTERVAL (for INTERVAL seconds)
  • No measured effect on performance when enabled on a 32 core run on OzSTAR skylake nodes

Improved MPI communication

  • Create buffer for each worker to use MPI isend, allowing tasks to be sent in parallel
  • Enable using --fast-mpi 1
  • Provides infrastructure for future improvements to jobs with short sampling times

Merge request reports