dynesty-parallization.md

@@ -210,7 +210,7 @@ phase = Uniform(name='phase', minimum=0, maximum=2 * np.pi, boundary='periodic')
 </details>
 </p>
 
-### Scaling test
+### Scaling test using any node on CIT
 I ran this bash script using the files above
 ```
 for i in {1..20};
@@ -218,4 +218,43 @@ do
     bilby_pipe config.ini --request-cpus $i --outdir outdir_${i}
 done
 ```
-For the cases `request_cpus > 16`, I manually modified the `.submit` files to only request 16 cpus. In this sense, everything above `16` is instructing the sampler to use more cores than there are available.
+For the cases `request_cpus > 16`, I manually modified the `.submit` files to only request 16 cpus. In this sense, everything above `16` is instructing the sampler to use more cores than requested.
\ No newline at end of file
+
+This is the speedup (the effective time per likelihood evaluation relative to the 1-core job).
+ 
+![image](uploads/17e37d27e0e2b774dd1ce70e7d3b7751/image.png)
+
+Notes 
+* The speed-up continue above n=16, clearly there are more cores available and bilby is stealing resources it did not request.
+* There is some variability between results which is likely due to variance in the performance of the randomly-selected nodes
+
+### Scaling test using node 1997 on CIT
+
+At Stuart's recommendation, I ran the job above again, but restricting the submit to only node1997, which has 8 available cores. The run script to generate the jobs and overwrite the submit files with the requirements etc is:
+
+```
+for i in {1..8};
+do
+    bilby_pipe config.ini --request-cpus $i --outdir outdir_${i}
+    sed -i 's/queue/+Optimization_SkyHough = True\nRequirements = (TARGET.Optimization_SkyHough =?= True)\nqueue/g' outdir_${i}/submit/*par*submit
+done
+
+for i in {9..14};
+do
+    bilby_pipe config.ini --request-cpus $i --outdir outdir_${i}
+    sed -i 's/queue/+Optimization_SkyHough = True\nRequirements = (TARGET.Optimization_SkyHough =?= True)\nqueue/g' outdir_${i}/submit/*par*submit
+    sed -i "s/request_cpus = ${i}/request_cpus = 8/g" outdir_${i}/submit/*par*submit
+done
+```
+
+Here is the timing plot (both the total timing and the per-likelihood call timing)
+
+![image](uploads/b72a35a9e1753e30453f07d02b9121dc/image.png)
+
+Here is the speed-up relative to the 1-core job
+
+![image](uploads/a1d2590be795fc7c03af91029f46d78d/image.png)
+
+Notes
+* This demonstrates the linear scaling
+* This demonstrates that if the code is instructed to use more cores than the available number, we reach a plateue
\ No newline at end of file