handle git archive locks more gracefully.
Right now python-git throws an exception that is not being caught. Given that the code is reloaded in the daemon as well, and not just the worker, it may be hard to catch it in a reasonable way:
2019-01-23_02:27:07.733906Z Traceback (most recent call last):
2019-01-23_02:27:07.733906Z File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
2019-01-23_02:27:07.734576Z "__main__", fname, loader, pkg_name)
2019-01-23_02:27:07.734576Z File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
2019-01-23_02:27:07.734576Z exec code in run_globals
2019-01-23_02:27:07.734576Z File "/usr/lib/python2.7/dist-packages/guardian/__main__.py", line 287, in <module>
2019-01-23_02:27:07.734576Z main()
2019-01-23_02:27:07.734576Z File "/usr/lib/python2.7/dist-packages/guardian/__main__.py", line 266, in main
2019-01-23_02:27:07.734576Z archive=archive,
2019-01-23_02:27:07.734576Z File "/usr/lib/python2.7/dist-packages/guardian/daemon.py", line 111, in __init__
2019-01-23_02:27:07.734576Z self.archive_commit('DAEMON INIT')
2019-01-23_02:27:07.734576Z File "/usr/lib/python2.7/dist-packages/guardian/daemon.py", line 155, in archive_commit
2019-01-23_02:27:07.734576Z with self.archive as archive:
2019-01-23_02:27:07.734576Z File "/usr/lib/python2.7/dist-packages/guardian/archive.py", line 53, in __enter__
2019-01-23_02:27:07.734576Z self.repo = git.Repo.init(self.root)
2019-01-23_02:27:07.734576Z File "/usr/lib/python2.7/dist-packages/git/repo/base.py", line 851, in init
2019-01-23_02:27:07.734576Z git.init(**kwargs)
2019-01-23_02:27:07.734576Z File "/usr/lib/python2.7/dist-packages/git/cmd.py", line 424, in <lambda>
2019-01-23_02:27:07.734576Z return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
2019-01-23_02:27:07.734576Z File "/usr/lib/python2.7/dist-packages/git/cmd.py", line 873, in _call_process
2019-01-23_02:27:07.734576Z return self.execute(call, **_kwargs)
2019-01-23_02:27:07.734576Z File "/usr/lib/python2.7/dist-packages/git/cmd.py", line 687, in execute
2019-01-23_02:27:07.734576Z raise GitCommandError(command, status, stderr_value, stdout_value)
2019-01-23_02:27:07.734576Z git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
2019-01-23_02:27:07.734576Z cmdline: git init
2019-01-23_02:27:07.734576Z stderr: 'error: could not lock config file /ligo/cds/llo/l1/guardian/archive/ISC_LOCK/.git/config: File exists
2019-01-23_02:27:07.734576Z fatal: could not set 'core.repositoryformatversion' to '0''