No other processes other than things like `bash`, `systemd`, `ssh`,
and `sd-pam` should be running. This may take a few minutes.
Specifically, you should not see either `procmgt` or `sleep`.
3. Start the `dmt-procmgt` process
```shell
$ sudo systemctl start dmt-procmgt
```
4. Check that all the expected processes are running, e.g. on `l1dmt3`:
```shell
$ dmt status --all --node l1dmt3
node: l1dmt3
AlarmManager_T OK
DMTDQ_L1_T OK
DMTGate_L1_T OK
DpushM-l1dmt3a OK
DpushM-l1dmt3b OK
FrTest OK
FrTest_1 OK
FrTest_2 OK
FrTest_3 OK
FrTest_4 OK
FrTest_5 OK
FrTest_6 OK
htCalib_L1_T OK
L1_ht_Frames_T OK
L1_rename_Frames_T OK
link_idq_T OK
LLO_TestDQ-l1dmt3 OK
mux_l1dmt3 OK
NameServer_T OK
Omega_HOFT_L1_T OK
Science_Segs_L1_T OK
SenseMonitor_HOFT_L1_T OK
trender_L1_T OK
TriggerManager_T OK
```
4. After a few minutes, you can verify that the data is flowing:
Run smlist a few seconds apart, e.g.
```shell
$ smlist
id name nBuf lBuf nTotal
32 LLO_Receive1 8 11000000 10286
33 LLO_Receive2 8 11000000 10286
34 LLO_Online 128 11000000 174588
35 LLO_DQGATE 64 100000 10285
36 LLO_hoft 16 2000000 194
37 LLO_IDQ 32 100000 10285
38 LLO_DMTDQ 64 2000000 0
$ smlist
id name nBuf lBuf nTotal
32 LLO_Receive1 8 11000000 10327
33 LLO_Receive2 8 11000000 10327
34 LLO_Online 128 11000000 174629
35 LLO_DQGATE 64 100000 10326
36 LLO_hoft 16 2000000 199 <--- Check that this changes
37 LLO_IDQ 32 100000 10327
38 LLO_DMTDQ 64 2000000 0
```
This should show an increasing number of frames.
You can also look at the most recent directory and file in
`/gds/dmt/frames/hoft/L1/` and verify the most recent timestamps
there.
### Configuration files
This repo will install the following file tree on `--prefix`
```
dmtspi/cache
pars
|- asq_set.filter
|- blrmsconfig_LLO.conf
|- blrms_ifo_LLO.conf
|- data
|- pycbc_generate_sensemonitor_integrand.py
|- SenseMonitor_BBH_30_30.txt
|- dq-gating-LLO.txt
|- dq-stream-LLO_dmtdq.txt
|- dq-stream-LLO_gate.txt
|- dq-stream-LLO_hoft.txt
|- dq-stream-LLO_idq.txt
|- dq-stream-LLO_Online.txt
|- dvTest_O3_LLO.cfg
|- EndTimes_L1.cfg
|- half_decade_set_2.filter
|- half_decade_set_3.filter
|- hoft_dq_flags.osc
|- L1-HOFT_OMEGA.cfg
|- L1-hoftStreamList.txt
|- L1-htpart_in.txt
|- l1_locked.osc
|- L-DQ+Gate_O3.json
|- L-DQmod_O3.json
|- L-GateMonitor_O4.js
|- LLO_SEI_blrms.conf
|- LockLoss.osc
|- PlaneMon_LLO.conf
|- ReferenceCalibrationDarmCal_L1.xml
|- ReferenceCalibration_hoft_L1.xml
|- Science_Segs_L1.cnf
|- SegGener_GRD_LLO.cnf
|- SegGener_GRD_LLO.osc
|- SegGener_HWINJ_LLO.cnf
|- SegGener_HWINJ_LLO.osc
|- SegGener_IOS_LLO.cnf
|- SegGener_IOS_LLO.osc
|- SegGener_LLO.cnf
|- SegGener_LLO.osc
|- SegGener_LSC_LLO.cnf
|- SegGener_LSC_LLO.osc
|- SegGener_PEM_LLO.cnf
|- SegGener_PEM_LLO.osc
|- SenseMonitor_BBH_30_30_L1.conf
|- SenseMonitor_CAL_L1.conf
|- SenseMonitor_hoft_L1.conf
|- STS_Seis_Blrms.filter
|- TrigMgr_LLO.conf
|- TrigStream_LLO.cfg
procmgt
|- LLO_MonitorDQ_Initial
|- LLO_MonitorDQ_Run.l1dmt0
|- LLO_MonitorDQ_Run.l1dmt1
|- LLO_MonitorDQ_Start
|- LLO_MonitorDQ_Start.l1dmt0
scripts
|- condense_segments_LLO
|- condense_test_LLO
status
|- all_files-llo
|- empty-file.json
|- file_ok.json
|- last-file-old.json
|- no-new-in-part.json
|- no-partition.json
|- no-such-file.json
|- partition-ok.json
|- part-stats-too-old.json
|- smsource_test.sh
|- status-monitor-l1dmt0
|- status-monitor-l1dmt1
|- status-monitor-l1dmt2
|- too-few-in-part.json
|- too-late-in-part.json
test/dmtspi/cache
logs
```
### Calibration pipeline updates
When an update to the calibration configurations is made, these new configurations will need to be pulled from the `calibration` home directory using
```shell
$ cd /home/calibration/ifo/[H1,L1]
$ git pull
```
This must be done on each of the machines that run the calibration process (`[h1,l1]dmt[1-3]`).
Once the new configurations have been pulled, the calibration pipeline needs to be restarted as the `dmtexec` user.
On `[h1,l1]dmt1` this is done with
```shell
$ dmt restart htCalib_[H1,L1]
```
On `[h1,l1]dmt2` this is done with
```shell
$ dmt restart htCalibR_[H1,L1]
```
FIXME: The name for the redundant process might be different on l1dmt2 than h1dmt2. One might be `htCalib_[H1,L1]_R` and the other might be `htCalibR_[H1,L1]`. This should be made consistent if it is different.
On `[h1,l1]dmt3` this is done with
```shell
$ dmt restart htCalib_[H1,L1]_T
```
After all of the processes are restarted, wait a few minutes and then verify that data is flowing again using
```shell
$ smlist
id name nBuf lBuf nTotal
32 LLO_Receive1 8 11000000 10286
33 LLO_Receive2 8 11000000 10286
34 LLO_Online 128 11000000 174588
35 LLO_DQGATE 64 100000 10285
36 LLO_hoft 16 2000000 194
37 LLO_IDQ 32 100000 10285
38 LLO_DMTDQ 64 2000000 0
$ smlist
id name nBuf lBuf nTotal
32 LLO_Receive1 8 11000000 10327
33 LLO_Receive2 8 11000000 10327
34 LLO_Online 128 11000000 174629
35 LLO_DQGATE 64 100000 10326
36 LLO_hoft 16 2000000 199 <--- Check that this changes
37 LLO_IDQ 32 100000 10327
38 LLO_DMTDQ 64 2000000 0
```
## useful commands3
watch -d smlist
FrameDump -p LLO_hoft -chanlist
dir=$(ls -t /gds/dmt/frames/hoft/L1/ | head -1) && frame=$(ls -t /gds/dmt/frames/hoft/L1/$dir/ | head -1) && FrameDump -i /gds/dmt/frames/hoft/L1/$dir/$frame -chanlist
