CalibrationMap size issue
CalibrationMaps can be quite large, large enough to make the associated I/O problematic. In particular, they are big enough that we have to be careful when sending them through Kafka connections (max buffer size ~10 MB).
This is thought to be associated with the monte carlo sampling of the loglike distributions (from sampling uncertainty associated with finite numbers of samples). Possible solutions include
- fewer monte carlo samples
- storing a vector representation of the monte carlo sampled distribution, rather than the samples directly (a CDF?)
- finding a parameteric representation of the distribution (instead of just a vector) analogous to the beta distributions used in FixedBandwidth1DKDE.
Target object sizes would be 10-100 KB (reduced from current "default" CalibrationMap sizes by a factor of ~100), and these are thought to be achievable without undue effort.
/cc @patrick.godwin