... | ... | @@ -26,6 +26,10 @@ The results can be seen below. The displayed errorbars are taken to be standard |
|
|
The sampling packages quote evidence uncertainties by calculating a K-L divergence. We want to test whether this quoted uncertainty is truly Gaussian, i.e. is the true evidence covered by the 1(2)-sigma interval ~68(95)% of the time, etc.
|
|
|
We can display these results most conveniently by creating percentile-percentile style plots. Specifically, we look at the results of dynesty and also Polychord as a reference.
|
|
|
|
|
|
With dynesty we see that the evidences for a low number of live points is significantly biased. This is obviously the case because the evidence estimates are systematically biased in this regime and the true value is therefore rarely covered by the uncertainty interval. As we increase the number of live points to 1024, we see that the curves generally stay inside the grey 95% confidence band. Interestingly, the curves tend to lie a bit below the band for low values of the confidence interval (CI), and overshoot for high values of the CI. This indicates that the errors quoted by `dynesty` are not truly Gaussian. Instead, they do in fact tail off faster than a Gaussian distribution, which means that there are very rarely any outliers past twice the quoted error. Because of that `dynesty` at and above 1000 live points can in general be used confidently to report log evidences and the associated errors.
|
|
|
|
|
|
Polychord in the unimodal case performs extremely well even for a low number of live points. The uncertainties generally appear to be truly Gaussian. In the bimodal case,
|
|
|
|
|
|
![dynesty_review_unimodal_summary_pp](uploads/da9cff0a6c9d6128b7bae9afd571c3d0/dynesty_review_unimodal_summary_pp.png)
|
|
|
![dynesty_review_bimodal_summary_pp](uploads/751a53cd96288a0f893e53805572937e/dynesty_review_bimodal_summary_pp.png)
|
|
|
|
... | ... | |