Advanced Monitoring
The condor_chirp
utility allows us to inject whatever information we like into condor classads.
For humans: with the right wrapper script, we can tell condor about various diagnostics and progress information, so we can keep an eye on job progress and health via commands like condor_q albert.einstein Cmd='"/usr/bin/BayesWave"' -af Niterations -af Model -af PercentComplete
. Further, that information could also be piped into your favorite monitoring/plotting software for real-time analytics of BayesWave workflows.
This issue tracks details of this work, starting with information we'd like to grep from existing log files and output.