3-level CI build system for LALSuite

Thoughts on a 3-level CI build system for LALSuite:

• jobs that run on push events: aimed at developers, focus on basic build & code quality checks, minimise runtime/resource usage
  • current lint jobs (except lint:depends), documentation
  • toplevel builds (except toplevel:pulsar)
  • a basic package-level build:
    • lal*:pkg just runs make distcheck then make install into a local prefix, which is them passed on to dependent jobs as artifacts.
    • advantages over current Conda/Debian/RPM builds:
      • use 3x fewer resources, given that push events are more frequent
      • should run quicker without extra packaging tasks, minimising runtime/resource usage
      • should be more resilient to intermittent failures (e.g. from package builders unable to connect to servers to download needed packages), can avoid using retry so that build errors are found earlier
      • clearer output: the package builders output lots & lots of cruft, which can make it difficult to find the actual build error message in the job logs, whereas a simple make distcheck build will be a lot clearer
• jobs that run on merge request events: aimed at maintainers, perform comprehensive tests, automate as much as possible
  • start with all jobs currently run as part of nightly build
    • for package-level build, would run Conda/Debian/RPM builds instead of basic package-level build described above
    • maybe make package-level builds as close to production package build process as possible, e.g. use Koji for RPM builds?
  • possible new jobs:
    • automatic ABI checking using https://github.com/lvc/abi-compliance-checker
    • anything else?
• jobs that run on nightly schedule: deployment tasks
  • should only run a strict subset of merge request jobs needed for deployment, e.g. wheels, documentation builds
  • i.e. should not perform any new tests not part of merge request level
• other thoughts:
  • if Conda/Debian/RPM package builds are very close to production package build processes, could these completely automate LALSuite package production?
    • i.e. run package builds on release tag, deploy jobs then upload packages to where they need to be

cc @duncanmmacleod @adam-mercer