make dmt2 actually redundant with dmt1
We would like to make the dmt2 machines into true redundant backups of the dmt1 machines. As far as I can tell these are the issues to be addressed:
-
The lowlatency machines receive data from dmt1 via a framelink process. There are two links, one for the detchar frames, and one for the hoft frames. In order for lowlatency to receive data from dmt2 as well, we would need to setup two additional framelink receivers on lowlatency, listening on different ports, and the dmt2 framelink senders would need to be configured to send their data to those alternate ports.
-
Various services write output to the shared /gds filesystem. The two machines would need to be configured to write their output to different locations in /gds:
-
The dmt2 machines are the only ones writing the 1-second frames to /gds. We would need to turn on the /gds frame writing on dmt1, but configure it's output location.
-
sensmon trend frames are written to /gds.
-
Segments generated by seg-gener are written to /gds.
-
dmt_wstream omega triggers are written to /gds.
-
-
Segment are also uploaded to the segdb servers. We would need to figure out how to upload the segments from the two different seggen instances, and have those resolved on the segdb side.
This is just a couple of small config tweaks, that I think are or could be covered in the L?O_MonitorDQ_Initial.in
files, e.g.:
- dmt1
DMTFRAMEDIR=/gds/dmt
FRAMELINK_DETCHAR_PORT=32104
FRAMELINK_HOFT_PORT=32105
DMTSEGSERVER=segments.ligo.org
- dmt2
DMTFRAMEDIR=/gds/dmt-backup
FRAMELINK_DETCHAR_PORT=32114
FRAMELINK_HOFT_PORT=32115
DMTSEGSERVER=segments-backup.ligo.org