O4-review/relative-binning.md

@@ -2,12 +2,41 @@
 
 Compute likelihood difference for
 
-- 64s NSBH pp-test
 - Good BNS analysis
 - Bad BNS analysis
 
 ## [Unit testing](https://git.ligo.org/lscsoft/bilby/-/blob/master/test/gw/likelihood/relative_binning_test.py)
 
-As part of the `Bilby` CI unit testing, we verify that the binned likelihood agrees with the regular likelihood the reference point for a range of cases.
+As part of the `Bilby` CI unit testing, we verify that the binned likelihood agrees with the regular likelihood as the reference point for a range of cases.
 We also verify that the likelihood is close to the regular model for 100 points drawn from a prior distribution.
 The final test is the optimization to find the reference parameters gives a good likelihood match after the optimization.
+
+## Large-scale performance
+
+The relative binning likelihood is used as part of the sampler review as we don't have a viable ROQ basis for `IMRPhenomNSBH`.
+This allows testing the approximation over a broad range of potential NSBH systems.
+For these analyses, we choose the fiducial parameters to match the injected values.
+We extract the following general trends:
+
+- reweighting to the regular likelihood correctly identifies failures of the method.
+- the approximation is very good for systems with SNR > 10 when initialized near the peak of the likelihood and rapidly degrades at higher SNRs.
+
+
+### Efficiency against SNR
+
+The two plots below indicate the SNR dependence of the reweighting efficiency (equivalent to approximation fidelity.)
+
+![image](uploads/2e1aa8e0419299afa93ab5c8c993d61d/image.png)
+
+![image](uploads/d6f8b2dd52d54e942bf5b49f64ba59d8/image.png)
+
+
+### Likelihood mismatches
+
+Below are the absolute likelihood mismatches for all of the pp-test analyses.
+the blue histograms correspond to the points above with efficiencies < 0.9 and the yellow have efficiencies > 0.9.
+We note that while for many of the blue traces, the differences are comparatively large, the important quantity is the width of each distribution and the cases that fail often have large tails in the mismatches.
+
+For some successful cases, the mean ln likelihood is > 0.1 in the tails, however, this can easily be corrected using likelihood reweighting.
+
+![image](uploads/9f2075efcca8e13f0d7b3ba7e2831bfc/image.png)
\ No newline at end of file