Node failed to restart "GUARD_MODULE_PATH" did not match any files
I was restarting a node that had crashed due to a systemd watchdog time out (a separate issue I think), when this error showed up. New to me. I don't believe that the user code was the cause since a second restart worked.
2020-06-16_20:34:02.734450Z Starting Advanced LIGO Guardian service: SEI_CS...
2020-06-16_20:34:03.598546Z ifo: H1
2020-06-16_20:34:03.598546Z name: SEI_CS
2020-06-16_20:34:03.617206Z CA prefix: SEI-CS
2020-06-16_20:34:03.617206Z module:
2020-06-16_20:34:03.617206Z /opt/rtcds/userapps/release/isi/h1/guardian/SEI_CS.py
2020-06-16_20:34:03.617206Z usercode:
2020-06-16_20:34:03.617206Z /opt/rtcds/userapps/release/isi/h1/guardian/sei_config/sensor_correction/__init__.py
2020-06-16_20:34:03.617206Z /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/sensor_correction/states.py
2020-06-16_20:34:03.617206Z /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/sensor_correction/__init__.py
2020-06-16_20:34:03.617206Z /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/sensor_correction/fader.py
2020-06-16_20:34:03.617206Z /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/__init__.py
2020-06-16_20:34:03.617206Z /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/sensor_correction/edges.py
2020-06-16_20:34:03.619888Z /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/const.py
2020-06-16_20:34:03.619888Z /opt/rtcds/userapps/release/isi/h1/guardian/sei_config/__init__.py
2020-06-16_20:34:03.619888Z /opt/rtcds/userapps/release/isi/h1/guardian/sei_config/sensor_correction/const.py
2020-06-16_20:34:03.619888Z /opt/rtcds/userapps/release/isi/h1/guardian/sei_config/sensor_correction/states.py
2020-06-16_20:34:03.619888Z nominal state: NONE
2020-06-16_20:34:03.619888Z initial request: NONE
2020-06-16_20:34:03.619888Z states (*=requestable):
2020-06-16_20:34:03.619888Z 40 * CONFIG_SC_OFF
2020-06-16_20:34:03.619888Z 30 * CONFIG_EQ
2020-06-16_20:34:03.619888Z 20 * CONFIG_WINDY
2020-06-16_20:34:03.619888Z 10 * CONFIG_SWARM
2020-06-16_20:34:03.619888Z 5 * DOWN
2020-06-16_20:34:03.619888Z -10 ENGAGING_CONFIG_EQ
2020-06-16_20:34:03.619888Z -11 ENGAGING_CONFIG_SC_OFF
2020-06-16_20:34:03.619888Z -12 ENGAGING_CONFIG_SWARM
2020-06-16_20:34:03.619888Z -13 ENGAGING_CONFIG_WINDY
2020-06-16_20:34:03.619888Z -14 RAMP_OFF_STS_MATRIX
2020-06-16_20:34:03.619888Z -15 RAMP_ON_STS_MATRIX_EQ
2020-06-16_20:34:03.619888Z -16 RAMP_ON_STS_MATRIX_SWARM
2020-06-16_20:34:03.619888Z -17 RAMP_ON_STS_MATRIX_WINDY
2020-06-16_20:34:03.619888Z 0 * INIT
2020-06-16_20:34:04.477048Z SEI_CS Guardian v1.3.4
2020-06-16_20:34:04.481403Z cas warning: Configured TCP port was unavailable.
2020-06-16_20:34:04.481403Z cas warning: Using dynamically assigned TCP port 41107,
2020-06-16_20:34:04.481403Z cas warning: but now two or more servers share the same UDP port.
2020-06-16_20:34:04.481403Z cas warning: Depending on your IP kernel this server may not be
2020-06-16_20:34:04.481403Z cas warning: reachable with UDP unicast (a host's IP in EPICS_CA_ADDR_LIST)
2020-06-16_20:34:04.485959Z SEI_CS EPICS control prefix: H1:GRD-SEI_CS_
2020-06-16_20:34:04.486339Z SEI_CS system archive: /ligo/cds/lho/h1/guardian/archive/SEI_CS
2020-06-16_20:34:12.886254Z Traceback (most recent call last):
2020-06-16_20:34:12.886254Z File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
2020-06-16_20:34:12.887273Z "__main__", fname, loader, pkg_name)
2020-06-16_20:34:12.887273Z File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
2020-06-16_20:34:12.887273Z exec code in run_globals
2020-06-16_20:34:12.887273Z File "/usr/lib/python2.7/dist-packages/guardian/__main__.py", line 291, in <module>
2020-06-16_20:34:12.887273Z main()
2020-06-16_20:34:12.887273Z File "/usr/lib/python2.7/dist-packages/guardian/__main__.py", line 270, in main
2020-06-16_20:34:12.887273Z archive=archive,
2020-06-16_20:34:12.887273Z File "/usr/lib/python2.7/dist-packages/guardian/daemon.py", line 111, in __init__
2020-06-16_20:34:12.887273Z self.archive_commit('DAEMON INIT')
2020-06-16_20:34:12.887273Z File "/usr/lib/python2.7/dist-packages/guardian/daemon.py", line 158, in archive_commit
2020-06-16_20:34:12.887273Z if archive.commit(self.system, message):
2020-06-16_20:34:12.887273Z File "/usr/lib/python2.7/dist-packages/guardian/archive.py", line 92, in commit
2020-06-16_20:34:12.906997Z repo.index.remove([entry.path])
2020-06-16_20:34:12.906997Z File "/usr/lib/python2.7/dist-packages/git/index/util.py", line 58, in post_clear_cache_if_not_rai
2020-06-16_20:34:12.915884Z rval = func(self, *args, **kwargs)
2020-06-16_20:34:12.915884Z File "/usr/lib/python2.7/dist-packages/git/index/util.py", line 76, in check_default_index
2020-06-16_20:34:12.916600Z return func(self, *args, **kwargs)
2020-06-16_20:34:12.916600Z File "/usr/lib/python2.7/dist-packages/git/index/base.py", line 868, in remove
2020-06-16_20:34:12.916600Z removed_paths = self.repo.git.rm(args, paths, **kwargs).splitlines()
2020-06-16_20:34:12.916600Z File "/usr/lib/python2.7/dist-packages/git/cmd.py", line 424, in <lambda>
2020-06-16_20:34:12.917854Z return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
2020-06-16_20:34:12.917854Z File "/usr/lib/python2.7/dist-packages/git/cmd.py", line 873, in _call_process
2020-06-16_20:34:12.917854Z return self.execute(call, **_kwargs)
2020-06-16_20:34:12.917854Z File "/usr/lib/python2.7/dist-packages/git/cmd.py", line 687, in execute
2020-06-16_20:34:12.917854Z raise GitCommandError(command, status, stderr_value, stdout_value)
2020-06-16_20:34:12.917854Z git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
2020-06-16_20:34:12.917854Z cmdline: git rm --cached -- GUARD_MODULE_PATH
2020-06-16_20:34:12.917854Z stderr: 'fatal: pathspec 'GUARD_MODULE_PATH' did not match any files'
2020-06-16_20:34:12.972406Z guardian@SEI_CS.service: Main process exited, code=exited, status=1/FAILURE
2020-06-16_20:34:12.972736Z Failed to start Advanced LIGO Guardian service: SEI_CS.
2020-06-16_20:34:12.972775Z guardian@SEI_CS.service: Unit entered failed state.
2020-06-16_20:34:12.972782Z guardian@SEI_CS.service: Failed with result 'exit-code'.
Edited by Jameson Rollins