Skip to content

Major overhauls to pipeline infrastructure

James Alexander Clark PhD requested to merge james-clark/bayeswave:pipe-dev into master

This is collection of 25 commits which overhaul the pipeline infrastructure for OSG deployment. There are some minor user-level changes:

  • Output directory names: results are now dumped into directories called trigtime_<GPS>_<slide> to make it clearer what is in each directory
  • Logging: stderr and stdout files now live in a separate log directory, following recommendations from HTCondor experts and sysadmins. This makes errors far easier to debug and reduces unnecessary clutter and network traffic.
  • Log file names: log file names now clearly indicate the output directory for that job, as well as the condor ID. Allows to grep for non-zero error files and instantly identify the problematic directory, as well as the actual job in the DAGMan file / shell script.
  • Major changes to OSG operability including PRE_Scripts for local output directory creation and individual job GEO location files.
  • I also simplified the running-BayesWave examples and have a new student coming online who i'm using to write up a dedicated examples page on use-cases

Commit messages:

Eliminating wrapper from singularity

all jobs now transfer under singularity

renaming output dirs and centralizing logs

OSG requirements only used for OSG jobs

stream stdout and stderr when file transfers enabled

Turn off transfer empty outputDir for singularity-based jobs

Test commit message in readme for webhooks

Test commit message in readme for webhooks

Test commit message in readme for webhooks

Cleaner UI for singularity usage, updated singularity requirments

Eliminating wrapper from singularity

all jobs now transfer under singularity

renaming output dirs and centralizing logs

OSG requirements only used for OSG jobs

Test commit message in readme for webhooks

Test commit message in readme for webhooks

adding exception to allow accounting-group OR accounting_group to be used in config files

moving to python-based geolocation using OSG school example

updatging image paths

fixing email notifications

Revert to meta equality in HAS_LIGO_FRAMES requirement

added license info to pre-proc script

added license info to pre-proc script

propgate new PreCmd to all routines

Mail notification. Fix typo in determination of notification type.

Overhauled some of the pipeline examples, removed others which no longer work

Merge request reports