More calibration developments

This adds error estimates for loglike(rank) along with exposing some API for simple variance estimates instead of just quantiles. The estimates for P(loglike|rank) are supported via a monte carlo sampling of P(loglike|rank) along with storing the ordered samples for interpolation later. This process appears to scale well and is dominated by the cost of performing the monte carlo, which we only do once. Interpolation remains fast, as is reading stored CalibrationMap objects off disk.

Sanity checks pass with flying colors. Using the same data set I used to check !13 (merged), I checked that the quantiles stored here are reasonable. I'll argue that they are.

pdf estimates with errors (same as shown in !13 (merged))

loglike estimate with errors. Black shading are the monte carlo samples drawn as part of CalibrationMap.compute_loglike_cdf. Grey lines are the 10%, 20%, 30%, ..., 80%, 90% quantiles for loglike as a function of rank. The blue and red lines are labeled in the legend; blue corresponds to the log of the expected values and red is the expected value of the logs. The difference between these two is expected.

We note that, for this particular (small) training set, the classifier produces a peak in loglike near rank~0.5. For well behaved classifiers, we'd expect loglike to increase monotonically with rank, meaning a re-ordering based on loglike will produce the same ordering as using rank alone. This is likely associated with the small number of samples than anything inherent in the classifier. More importantly, it looks like all of CalibrationMap is functioning properly.

Merging this is a relatively low priority and we can chat about it on this week's call and/or address it later when you have some free time. Even if not used immediately in idq-timeseries, this will provide very nice plots for idq-report.