Skip to content

When creating temp files, check if _CONDOR_SCRATCH_DIR is set before falling back to TMPDIR

Patrick Godwin requested to merge tmpdir_condor into master

Some folks have been seeing this message in the logs recently from jobs submitted via Condor:

error (gst-resource-error-quark:14 'No space left on the resource.'): gstmultifilesink.c(787): gst_multi_file_sink_write_buffer (): /GstPipeline:pipeline/FRAMECPPFilesink:framecppfilesink0/GstMultiFileSink:multifilesink

Some temporary files being generated (mostly gstlal_inspiral) appear to rely on $TMPDIR == $_CONDOR_SCRATCH_DIR when submitting jobs via condor, but this assumption appears to have been broken recently. It's not a problem by itself but can be a problem if wherever $TMPDIR is pointed to has very little space allocated and/or isn't being tracked by Condor for cleanup. Specifically for the local frame caching where we expect to write O(GB) of frames, this can be a real problem. It's a good idea anyways to avoid indirectly relying on specific quirks of the system's environment to make our workflows more portable.

For the various places where it expects to create files via tempfile, it first checks whether $_CONDOR_SCRATCH_DIR is set before falling back on tempfile to find a suitable temporary directory (see https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir).

Merge request reports