LALPulsar: point PyPI/Conda packages to use new Solar System ephemerides package

@matthew-pitkin has put together a new Solar System ephemerides package for distributing the Earth, Sun, and time standard ephemerides.

Once this package is available in both PyPI and Conda, the LALSuite/LALPulsar PyPI/Conda packages will not longer include ephemeris files, but instead will pull in the new Solar System ephemerides package. This should save ~18 MB from the PyPI package which hopefully helps alleviate package size constraints.

I imagine the Solar System ephemerides package should be an optional dependency of LALSuite, if possible, so that non-CW users don't need to pull in a large package they don't need. This may require updating of various how-to guides which use LALPulsar from PyPI/Conda.

To-dos:

• Get the Solar System ephemerides package on both PyPI and Conda.
• Update LALPulsar configuration options to allow not including any ephemerides (currently must include at least a minimum set): !2156 (merged)
• Update PyPI packaging, add tests to make sure ephemerides are still being found: !2156 (merged)
• Refactor fallback path passed to XLALFileResolvePathLong() to all per-library fallback paths (set at ./configure time) and both absolute and relative paths
• Update Conda packaging, add tests to make sure ephemerides are still being found

cc @matthew-pitkin @david-keitel

assigned to @karl-wette and @matthew-pitkin

I've started the process of getting this into conda-forge here.

How is this going to effect the RPMs and Debian packages?

@adam-mercer This won't affect the RPM/Debian packages, they'll continue to distribute the existing ephemerides for backward compatibility. I imagine new ephemerides will only be added to Matt's package.

(FYI: the main customer for the ephemerides is the known pulsar search, so that we can exactly match the timing solutions we get from pulsar astronomers. Going forward Matt's Python-based CWInPy code is going to be used for those analyses, so it makes sense for new ephemerides to also be distributed in a Python ecosystem package. For other CW analysis codes the particular choice of ephemeris is generally not important, and they can continue using the existing files in LALSuite.)

OK, sounds good. I was basically concerned about the files in LALSuite and this new package being out of sync and leading to potential differences in output which could be a pain to debug, but it sounds like that won't be an issue.

added lalpulsar prioritywishlist labels

added pkg_conda pkg_wheel labels

Update LALPulsar configuration options to allow not including any ephemerides (currently must include at least a minimum set)

Do you mean not requiring any internally provided files, but it would fail if neither those nor the package dependency are found? Or to actually succeed in the absence of any ephems souce, even though the result won't be of much use?

The former. LALPulsar currently has a configuration option to include either a minimal set of ephemeris files (basically just the default DE405 ephemerides) or all of them. I'd just need to add a 3rd option to include no ephemeris files, and instead rely on the SSE package being installed.

Ok, sounds good!

I imagine the Solar System ephemerides package should be an optional dependency of LALSuite, if possible, so that non-CW users don't need to pull in a large package they don't need. This may require updating of various how-to guides which use LALPulsar from PyPI/Conda.

For PyPI, this would be pip install lalsuite[ephemerides]?

For conda, the last time I tried to research it, the answer seemed to be "this is not currently possible" - has something similar been implemented in the meantime?

Alternatively, since subpackages are already split up on the conda side, it could just be a fixed requirement of lalpulsar, and users installing the all-in-one lalsuite and complaining about too big data files can just be told to instead install the subpackages they want?

For PyPI, this would be pip install lalsuite[ephemerides]?

I presume that something like that would be possible.

For conda, the last time I tried to research it, the answer seemed to be "this is not currently possible" - has something similar been implemented in the meantime?

Alternatively, since subpackages are already split up on the conda side, it could just be a fixed requirement of lalpulsar, and users installing the all-in-one lalsuite and complaining about too big data files can just be told to instead install the subpackages they want?

I imagine this would just be a fixed dependency of the LALPulsar Conda package. (I wasn't even aware there was a catch-all LALSuite meta-Conda package.)

From that link, I've used only the setup.py method so far, not the .toml or .cfg, but yes either should work for PyPI.

(I wasn't even aware there was a catch-all LALSuite meta-Conda package.)

I also only noticed it relatively recently, but the files history goes back 4 years, so either Duncan used a time machine at some point or we just overlooked it for a long time. ;)

Looks like it only exists in conda-forge, not in LALSuite itself, so I wouldn't have come across it that way. I imagine anyone installing all LALSuite packages isn't going to worry too much about space constraints. In case the actual on-disk size of (LALSuite+ephemerides) is going to be similar, it's just the files will come from a different package. (And Matt's only allowed to add another ~100 MB worth of files, so that's an upper limit )

Just for info, my request for the PyPI file size limit upgrade for the solar-system-ephemerides package has been granted, so it can now contain up to 200 Mb.

The solar-system-ephemerides package is now available on conda-forge. I plan to release v0.2.0 (or I might even bump straight up to v1.0.0!) containing an additional ephemeris that I'd not included in the last release to keep the size below 100Mb. I'll hopefully get round to this at some point over the next week (depending whether I can force myself to write a test suite for the package in a reasonable time or not).

Thanks @matthew-pitkin I think that's all that's needed from your end. I'll keep this assigned to me to start making the LALPulsar changes.

I've released v1.0.0, which now actually includes a test suite!

unassigned @matthew-pitkin

marked the checklist item Get the Solar System ephemerides package on both PyPI and Conda. as completed

marked the checklist item Update LALPulsar configuration options to allow not including any ephemerides (currently must include at least a minimum set) as completed

changed title from LALPulsar: point PyPI/Conda packages to use new Solar System ephemerides package to LALPulsar: point PyPI packages to use new Solar System ephemerides package

changed the description

@matthew-pitkin @david-keitel I've opened !2156 (merged) to point the LALSuite PyPI package to solar-system-ephemerides to get the ephemeris files.

Note that I haven't implemented this for Conda, because the option to specify a hard-coded fallback path for LAL_DATA_PATH is in LAL, not LALPulsar; i.e. there's no way to specify a fallback just for LALPulsar files. I don't have time to retool this, so the LALPulsar Conda package will stay as is. If anyone needs the extra files available only in solar-system-ephemerides they'll need to point LALPulsar to them manually, i.e. by setting LAL_DATA_PATH.

Currently we have XLALReadEphemerisFile call XLALPulsarFileResolvePath which is a 1-line wrapper to XLALFileResolvePathLong passing along the PKG_DATA_DIR fallback. So shouldn't it actually be trivial to change the fallback on the lalpulsar side?

It's a little more complicated. The PKG_DATA_DIR is always a hard-coded absolute path (e.g. /usr/share/...) passed through from Automake. This is fine for RPMs/debs which know where they'll be installed, but not for PyPI/Conda packages which could end up anywhere. For those we need to search a relative path, but that's currently set by the --with-relative-data-path configure option, which is only available in LAL.

It's not impossible to re-tool this, I just don't have the time currently, so I'll leave it until someone says they really need this. (I imagine @matthew-pitkin will be handling the ephemeris file paths in CWInPy on the Python side, and so probably won't be needing a C-based solution.)

Ok, so in the C lalpulsar code we could just swap in another variable there, but the problem is to provide that variable, which can only be done at configure stage. And for conda lal and lalpulsar are configured separately, so by "retool" you mean a similar logic would need to be newly implemented for lalpulsar. Did I follow that correctly?

Something like that. Essentially we'd want to merge the relative path and PKG_DATA_DIR fallback features into a single fallback path which could handle either absolute or relative directories, and which could be configured at a per-library level.

I've added back porting the Conda packages as a future goal for this issue; this won't be done immediately but I'll leave the issue open as a reminder.

mentioned in merge request !2156 (merged)

changed title from LALPulsar: point PyPI packages to use new Solar System ephemerides package to LALPulsar: point PyPI/Conda packages to use new Solar System ephemerides package

changed the description

marked the checklist item Update Conda packaging, add tests to make sure ephemerides are still being found as incomplete