MDC04-Count-Tracker-Results.md

+## Recap
+* The count tracker records all the counts inside a pre-defined box in a 10s window around every GraceDB upload done by Mario
+* The box is defined by SNR > 6 and chisq/SNRsq < 0.04
+* The FAR threshold for Mario was set to 1/hour (not sure, can someone confirm?)
+* Counts can only be stored after burn-in
+* Mario has 921 SVD bins
+
+## Procedure
+
+* Started by making a list of all the dist_stats files that had a non-zero count tracker (282 such bins)
+* Made background plots for all of them and checked for signal contamination (142 such bins)
+* The list of contaminated bins can be found here: /home/prathamesh.joshi/lr_background_estimation/MDC04/try2/contaminated_bins, with the background plots for those bins located [here](https://ldas-jobs.ligo.caltech.edu/~prathamesh.joshi/bg_count_tracker/MDC04/try2/check/)
+* Most of this contamination was pretty minor, possibly not even caused by signals
+* Observed that contamination was very frequently spread across multiple neighboring bins (for example, bins 0808 - 0820 in L1)
+* Removed count from 9 events while the analysis was running
+* gstlal_ll_inspiral_remove_counts --gps-time 1336991936 --gps-time 1337209407 --gps-time 1338325229 --gps-time 1338325231 --gps-time 1338428116 --gps-time 1338805411 --gps-time 1339314095 --gps-time 1339347995 --gps-time 1339385711 --gps-time 1339385709 --gps-time 1339473792 *noninj*
+* For every combination of gps-time and registry file, get a message of the form
+http://node934:37375/remove_counts.txt:
+        b'error'
+but it's not really an error (it's actually an error if it says "failed")
+* Can check that the command was successful by using the GET method:
+* vi http://node1245:33508/remove_counts.txt
+gps_time=1336991936.0,1337209407.0,1338325229.0,1338325231.0,1338428116.0,1338805411.0,1339314095.0,1339347995.0,1339385711.0,1339385709.0,1339473792.0
+* Wait for 4 hours for the next snapshot, and then create the background plots again
+* They can be found [here](https://ldas-jobs.ligo.caltech.edu/~prathamesh.joshi/bg_count_tracker/MDC04/try2/convert/)
+
+## Results
+
+* All the major contamination was effectively removed
+* Examples are [here](https://ldas-jobs.ligo.caltech.edu/~prathamesh.joshi/bg_count_tracker/MDC04/try2/convert/L1/0818.png) and [here](https://ldas-jobs.ligo.caltech.edu/~prathamesh.joshi/bg_count_tracker/MDC04/try2/convert/L1/0834.png) as well as around 10 bins around them.
+* Minor contaminations were mostly not removed
+* Example is [here](https://ldas-jobs.ligo.caltech.edu/~prathamesh.joshi/bg_count_tracker/MDC04/try2/convert/L1/0215.png)
+
+## Removing all counts
+
+* For each dist_stats file, removed counts from all the times stored in its count tracker. (The Mario files were not changed for this)
+* The background plots can be found [here](https://ldas-jobs.ligo.caltech.edu/~prathamesh.joshi/bg_count_tracker/MDC04/try2/remove_all/)
+* Slightly more effective at removing minor contamination (example [here](https://ldas-jobs.ligo.caltech.edu/~prathamesh.joshi/bg_count_tracker/MDC04/try2/remove_all/L1/0303.png))
+* However, apart from contamination removal, no bins were affected by this, so it seems to be a safe thing to do
+
+## Discussion
+
+* Minor contamination was not effectively removed, probably due to a high FAR thresold
+* In an analysis with FAR threshold of 1/10min, minor contamination was much more effectively removed
\ No newline at end of file