When creating temp files, check if _CONDOR_SCRATCH_DIR is set before falling back to TMPDIR
Some folks have been seeing this message in the logs recently from jobs submitted via Condor:
error (gst-resource-error-quark:14 'No space left on the resource.'): gstmultifilesink.c(787): gst_multi_file_sink_write_buffer (): /GstPipeline:pipeline/FRAMECPPFilesink:framecppfilesink0/GstMultiFileSink:multifilesink
Some temporary files being generated (mostly gstlal_inspiral
) appear to rely on $TMPDIR
== $_CONDOR_SCRATCH_DIR
when submitting jobs via condor, but this assumption appears to have been broken recently. It's not a problem by itself but can be a problem if wherever $TMPDIR
is pointed to has very little space allocated and/or isn't being tracked by Condor for cleanup. Specifically for the local frame caching where we expect to write O(GB) of frames, this can be a real problem. It's a good idea anyways to avoid indirectly relying on specific quirks of the system's environment to make our workflows more portable.
For the various places where it expects to create files via tempfile
, it first checks whether $_CONDOR_SCRATCH_DIR
is set before falling back on tempfile
to find a suitable temporary directory (see https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir).