diff --git a/.gitignore b/.gitignore index 06e70e79a611a00f28e82c516a5aae10f1eaa5b6..2a75b9b1d44425e4a7e1038e6a513c35d4d12180 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,4 @@ GW*/output/*.txt GW*/output/*.png .ipynb_checkpoints *.err +site diff --git a/README.md b/README.md deleted file mode 100644 index 02262ae24de44f85e4eb2bc2824c7145465506e7..0000000000000000000000000000000000000000 --- a/README.md +++ /dev/null @@ -1,69 +0,0 @@ -# Acceptance tests for [ligo.skymap](https://git.ligo.org/lscsoft/ligo.skymap) - -This repository provides nightly acceptance testing for the BAYESTAR and its] -parent Python package, `ligo.skymap`. The pipeline is triggered nightly. For -implementation details, see [.gitlab-ci.yml](.gitlab-ci.yml). - -The baseline results are from BAYESTAR in LALInference 1.9.4.1 (lscsoft/lalsuite@b05391e18175aa2496de895ae46cacd8906f6a32). - -## First Two Years - -### Test description - -Run [`bayestar-localize-coincs`](https://lscsoft.docs.ligo.org/ligo.skymap/ligo/skymap/tool/bayestar_localize_coincs.html) on all 500 events from the ["First Two Years" study](https://www.ligo.org/scientists/first2years/). -These events *do not* have SNR time series, so the test exercises BAYESTAR's code path for synthesizing fake SNR time series using the template autocorrelation function. - -### Acceptance criteria - -In the five summary figures below, there is no visible difference between the baseline version and the version under test. - -1. [ ] P-P plot in sky position -2. [ ] P-P plot in distance -3. [ ] P-P plot jointly in sky position and distance (volume) -4. [ ] Histogram of searched area -5. [ ] Histogram of angle offset between injected position and maximum a posteriori position - -### Results - -| 2015 | 2016 | -| ----------------------------------------- | ----------------------------------------- | -| ![comparison][f2y/15/searched_prob] | ![comparison][f2y/16/searched_prob] | -| ![comparison][f2y/15/searched_prob_dist] | ![comparison][f2y/16/searched_prob_dist] | -| ![comparison][f2y/15/searched_prob_vol] | ![comparison][f2y/16/searched_prob_vol] | -| ![comparison][f2y/15/searched_area_hist] | ![comparison][f2y/16/searched_area_hist] | -| ![comparison][f2y/15/offset_hist] | ![comparison][f2y/16/offset_hist] | - -## Historical Events - -### Test description - -Run [`bayestar-localize-lvalert`](https://lscsoft.docs.ligo.org/ligo.skymap/ligo/skymap/tool/bayestar_localize_lvalert.html) on the original input data from GraceDB for GW170814 and GW170817. -These events *do* have SNR time series, so the test exercises BAYESTAR's code path for using existing SNR time series. - -### Acceptance criteria - -1. [ ] The Mollweide plots are pixel-for-pixel identical to the baseline versions (as measured using [`matplotlib.testing.compare.compare_images`](https://matplotlib.org/api/testing_api.html#matplotlib.testing.compare.compare_images)). - -### Results - -| Event | Baseline | Latest | Comparison | -| --------- | ------------------------------ | ----------------------------- | ----------------------------- | -| GW170814 | ![baseline][GW170814/baseline] | ![latest][GW170814/latest] | ![latest][GW170814/compare] | -| GW170817 | ![baseline][GW170817/baseline] | ![latest][GW170817/latest] | ![latest][GW170817/compare] | - -[f2y/15/searched_prob]: https://lscsoft.docs.ligo.org/ligo-skymap-acceptance-tests/first2years/comparison/2015/searched_prob.svg -[f2y/16/searched_prob]: https://lscsoft.docs.ligo.org/ligo-skymap-acceptance-tests/first2years/comparison/2016/searched_prob.svg -[f2y/15/searched_prob_dist]: https://lscsoft.docs.ligo.org/ligo-skymap-acceptance-tests/first2years/comparison/2015/searched_prob_dist.svg -[f2y/16/searched_prob_dist]: https://lscsoft.docs.ligo.org/ligo-skymap-acceptance-tests/first2years/comparison/2016/searched_prob_dist.svg -[f2y/15/searched_prob_vol]: https://lscsoft.docs.ligo.org/ligo-skymap-acceptance-tests/first2years/comparison/2015/searched_prob_vol.svg -[f2y/16/searched_prob_vol]: https://lscsoft.docs.ligo.org/ligo-skymap-acceptance-tests/first2years/comparison/2016/searched_prob_vol.svg -[f2y/15/searched_area_hist]: https://lscsoft.docs.ligo.org/ligo-skymap-acceptance-tests/first2years/comparison/2015/searched_area_hist.svg -[f2y/16/searched_area_hist]: https://lscsoft.docs.ligo.org/ligo-skymap-acceptance-tests/first2years/comparison/2016/searched_area_hist.svg -[f2y/15/offset_hist]: https://lscsoft.docs.ligo.org/ligo-skymap-acceptance-tests/first2years/comparison/2015/offset_hist.svg -[f2y/16/offset_hist]: https://lscsoft.docs.ligo.org/ligo-skymap-acceptance-tests/first2years/comparison/2016/offset_hist.svg -[GW170814/baseline]: https://lscsoft.docs.ligo.org/ligo-skymap-acceptance-tests/GW170814/baseline/bayestar.png -[GW170817/baseline]: https://lscsoft.docs.ligo.org/ligo-skymap-acceptance-tests/GW170817/baseline/bayestar.png -[GW170814/latest]: https://lscsoft.docs.ligo.org/ligo-skymap-acceptance-tests/GW170814/output/bayestar.png -[GW170817/latest]: https://lscsoft.docs.ligo.org/ligo-skymap-acceptance-tests/GW170817/output/bayestar.png -[GW170814/compare]: https://lscsoft.docs.ligo.org/ligo-skymap-acceptance-tests/GW170814/output/bayestar-failed-diff.png -[GW170817/compare]: https://lscsoft.docs.ligo.org/ligo-skymap-acceptance-tests/GW170817/output/bayestar-failed-diff.png diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000000000000000000000000000000000000..d6614310f9745a72eb5f281053bb48ce0d2d7a98 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,65 @@ +# Acceptance tests for [ligo.skymap](https://git.ligo.org/lscsoft/ligo.skymap) + +The baseline results are from BAYESTAR in LALInference 1.9.4.1 (lscsoft/lalsuite@b05391e18175aa2496de895ae46cacd8906f6a32). + +## First Two Years + +### Test description + +Run [`bayestar-localize-coincs`](https://lscsoft.docs.ligo.org/ligo.skymap/tool/bayestar_localize_coincs.html) on all 500 events from the ["First Two Years" study](https://www.ligo.org/scientists/first2years/). +These events *do not* have SNR time series, so the test exercises BAYESTAR's code path for synthesizing fake SNR time series using the template autocorrelation function. + +### Acceptance criteria + +In the five summary figures below, there is no visible difference between the baseline version and the version under test. + +1. [ ] P-P plot in sky position +2. [ ] P-P plot in distance +3. [ ] P-P plot jointly in sky position and distance (volume) +4. [ ] Histogram of searched area +5. [ ] Histogram of angle offset between injected position and maximum a posteriori position + +### Results + +| 2015 | 2016 | +| ----------------------------------------- | ----------------------------------------- | +| ![comparison][f2y/15/searched_prob] | ![comparison][f2y/16/searched_prob] | +| ![comparison][f2y/15/searched_prob_dist] | ![comparison][f2y/16/searched_prob_dist] | +| ![comparison][f2y/15/searched_prob_vol] | ![comparison][f2y/16/searched_prob_vol] | +| ![comparison][f2y/15/searched_area_hist] | ![comparison][f2y/16/searched_area_hist] | +| ![comparison][f2y/15/offset_hist] | ![comparison][f2y/16/offset_hist] | + +## Historical Events + +### Test description + +Run [`bayestar-localize-lvalert`](https://lscsoft.docs.ligo.org/ligo.skymap/tool/bayestar_localize_lvalert.html) on the original input data from GraceDB for GW170814 and GW170817. +These events *do* have SNR time series, so the test exercises BAYESTAR's code path for using existing SNR time series. + +### Acceptance criteria + +1. [ ] The Mollweide plots are pixel-for-pixel identical to the baseline versions (as measured using [`matplotlib.testing.compare.compare_images`](https://matplotlib.org/api/testing_api.html#matplotlib.testing.compare.compare_images)). + +### Results + +| Event | Baseline | Latest | Comparison | +| --------- | ------------------------------ | ----------------------------- | ----------------------------- | +| GW170814 | ![baseline][GW170814/baseline] | ![latest][GW170814/latest] | ![latest][GW170814/compare] | +| GW170817 | ![baseline][GW170817/baseline] | ![latest][GW170817/latest] | ![latest][GW170817/compare] | + +[f2y/15/searched_prob]: first2years/comparison/2015/searched_prob.svg +[f2y/16/searched_prob]: first2years/comparison/2016/searched_prob.svg +[f2y/15/searched_prob_dist]: first2years/comparison/2015/searched_prob_dist.svg +[f2y/16/searched_prob_dist]: first2years/comparison/2016/searched_prob_dist.svg +[f2y/15/searched_prob_vol]: first2years/comparison/2015/searched_prob_vol.svg +[f2y/16/searched_prob_vol]: first2years/comparison/2016/searched_prob_vol.svg +[f2y/15/searched_area_hist]: first2years/comparison/2015/searched_area_hist.svg +[f2y/16/searched_area_hist]: first2years/comparison/2016/searched_area_hist.svg +[f2y/15/offset_hist]: first2years/comparison/2015/offset_hist.svg +[f2y/16/offset_hist]: first2years/comparison/2016/offset_hist.svg +[GW170814/baseline]: GW170814/baseline/bayestar.png +[GW170817/baseline]: GW170817/baseline/bayestar.png +[GW170814/latest]: GW170814/output/bayestar.png +[GW170817/latest]: GW170817/output/bayestar.png +[GW170814/compare]: GW170814/output/bayestar-failed-diff.png +[GW170817/compare]: GW170817/output/bayestar-failed-diff.png diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000000000000000000000000000000000000..e37b8e8ecfa750407b3deddc87e0aaed50c310ac --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,3 @@ +site_name: ligo.skymap Acceptance Tests +theme: + name: material