CI: use retry() macro instead of GitLab `retry` job option

The CI currently retries failed jobs at least once (default:retry: 1 in .gitlab-ci.yml). This doesn't distinguish between build failures (i.e. compile error) and transient failures (e.g. network outages). (retry:when doesn't help as that only distinguishes runner failures.) Retrying a CI job on build failures wastes resources (particularly for resource-scarce runners e.g. macOS) and delays pipeline completion.

Using the retry() macro instead gives more control on which commands need to be retried (i.e. anything that downloads packages), how many times, etc. Retrying a single command within a CI job is more efficient that failing and the restarting the whole job. Jobs that fail on e.g. a compile error will fail immediately and not be rerun.

Todo:

• Use retry() macro in front of all package-installing commands: conda, apt-get, yum, others?
  • Relevant MRs: !2395 (merged), !2421 (merged)
• Copy retry() macro upstream to https://git.ligo.org/computing/gitlab-ci-templates so it can also be used in the templates.
  • Relevant MRs: computing/gitlab-ci-templates!105 (closed)
• Use retry() from upstream template
  • Relevant MRs:
• Set retry:when to only retry jobs with runner failures (https://docs.gitlab.com/ci/yaml/#retrywhen)
  • Relevant MRs:

cc @duncanmmacleod @adam-mercer