advLigoRTS issueshttps://git.ligo.org/cds/software/advligorts/-/issues2023-06-06T16:35:45Zhttps://git.ligo.org/cds/software/advligorts/-/issues/575Add automated test to check locally controlled filters don't indicate remote ...2023-06-06T16:35:45ZErik von ReisAdd automated test to check locally controlled filters don't indicate remote filter requestswhen an fmc or fmc2 filter section is locally controlled, remote requests to change the state of the filter section should not be indicated as a requested state (because the request won't be honored).
This should be tested automatical...when an fmc or fmc2 filter section is locally controlled, remote requests to change the state of the filter section should not be indicated as a requested state (because the request won't be honored).
This should be tested automatically in the fmctest.pl and fmc2test.pl automated tests run on x2ats.https://git.ligo.org/cds/software/advligorts/-/issues/573awgtpman segfaults when the model was built with a bad hostname2023-06-01T21:36:09ZEzekiel Dohmenawgtpman segfaults when the model was built with a bad hostnameEzekiel DohmenEzekiel Dohmenhttps://git.ligo.org/cds/software/advligorts/-/issues/572Older IO chassis duotone calculation becomes invalid after time 5+ min.2023-06-01T16:44:59ZEzekiel DohmenOlder IO chassis duotone calculation becomes invalid after time 5+ min.Older IO chassis duotone calculation becomes invalid after time 5+ min.
Might be cause by duotone calculation updates that were done to support new LIGO firmware, not being fully backwards compatible with older IO chassis.Older IO chassis duotone calculation becomes invalid after time 5+ min.
Might be cause by duotone calculation updates that were done to support new LIGO firmware, not being fully backwards compatible with older IO chassis.https://git.ligo.org/cds/software/advligorts/-/issues/571IOP models now constrained to have iop in the name2023-05-18T17:17:27ZErik von ReisIOP models now constrained to have iop in the nameIOP model names now must start with "nniop" where nn is the IFO identifier. The constraint was introduced accidentally by direct use of the CDS_EPICS::iop in fe/controlleriop.c to support improved duotone feedback in EPICS. It turns o...IOP model names now must start with "nniop" where nn is the IFO identifier. The constraint was introduced accidentally by direct use of the CDS_EPICS::iop in fe/controlleriop.c to support improved duotone feedback in EPICS. It turns out this struct name is derived from the model name.
Possible solutions:
1. keep the constraint, but issue a better error when an iop is renamed
2. Remove the constraint by forcing the struct name to be iop for iops.
3. Remove the constraint by moving the duotone
4. Remove the constraint by creating a name-agnostic way to access the system specific structurehttps://git.ligo.org/cds/software/advligorts/-/issues/568DAQ file archive should have subdirectory for each model. Install/vs restart ...2023-05-10T16:19:39ZEzekiel DohmenDAQ file archive should have subdirectory for each model. Install/vs restart should have distinguishable archives.`/chans/daq/archive/``/chans/daq/archive/`https://git.ligo.org/cds/software/advligorts/-/issues/567The RCG should provide a live count of the running excitations in a channel.2023-05-05T15:50:03ZEzekiel DohmenThe RCG should provide a live count of the running excitations in a channel.https://git.ligo.org/cds/software/advligorts/-/issues/565Add "Clear Test Points" Button (and command line writtable EPICs momentary) T...2023-04-28T18:21:15ZJeffrey KisselAdd "Clear Test Points" Button (and command line writtable EPICs momentary) To GDS_TP ScreenAs far as I know, at the moment, in order to clear stale test points on a given front-end model, one has to remember how to use the cryptic, multi-step process with the command line "diag" interface and the DCUID of that model, e.g. what...As far as I know, at the moment, in order to clear stale test points on a given front-end model, one has to remember how to use the cryptic, multi-step process with the command line "diag" interface and the DCUID of that model, e.g. what was done in [LHO:69086](https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=69086).
As we continue to push towards automating measurement processes, graceful exits upon lock loss are critical, which includes clearing test points. That pushes me towards the creation of a more straight-forward, one-command-line, way to clear test points on a given model seems necessary (if it doesn't exist already). Also, for the less "I'll just pop open a command line" oriented staff, having a button on an MEDM screen that triggers the same clearing process would also be extremely helpful (so that one doesn't have to find out what the DCUID number is).
The natural place, to me, would be on the RCG, automatically-generated GDS_TP screen, near the relatively new "GDS TABLE" where all this info seems readily available.
If there's already a one-command-line way to interface with diag, then I'd appreciate a re-education since I've lost that institutional memory.https://git.ligo.org/cds/software/advligorts/-/issues/564Filter modules and filters integrated into parts have their specification spr...2023-04-13T17:03:57ZEzekiel DohmenFilter modules and filters integrated into parts have their specification spread over the model header.Instead, they should be consolidated into a single structure.Instead, they should be consolidated into a single structure.https://git.ligo.org/cds/software/advligorts/-/issues/563H1 FPU Bit Tracking2023-04-12T15:27:52ZEzekiel DohmenH1 FPU Bit Tracking## h1susprocpi
### Suspicious Channels
```
H1:SUS-PI_PROC_COMPUTE_MODE30_NORMLOG10RMSMON -inf
H1:SUS-PI_PROC_COMPUTE_MODE30_NORMLOG10AMPMON -nan
H1:SUS-PI_PROC_COMPUTE_MODE21_NORMLOG10RMSMON -inf
H1:SUS-PI_PROC_COMPUTE_MODE21_NORMLOG10A...## h1susprocpi
### Suspicious Channels
```
H1:SUS-PI_PROC_COMPUTE_MODE30_NORMLOG10RMSMON -inf
H1:SUS-PI_PROC_COMPUTE_MODE30_NORMLOG10AMPMON -nan
H1:SUS-PI_PROC_COMPUTE_MODE21_NORMLOG10RMSMON -inf
H1:SUS-PI_PROC_COMPUTE_MODE21_NORMLOG10AMPMON -nan
```
### Model
Signal Data IPCs are fed to MuxMatrix:
![image](/uploads/e16b4d604b15ef17bd8f20adf008688b/image.png)
Outputs 21 and 30 are always zero, these outputs are fed to pretty standard -> RMS -> log10(x). Log of 0 causes the invalid math operation.
### Solution
TBD
## h1susproc
### Suspicious Channels
```
H1:SUS-ETMX_L2_DAMP_MODE10_RMSLP_LOG10_OUTMON -inf
H1:SUS-ETMX_L2_DAMP_MODE19_RMSLP_LOG10_OUTMON -inf
H1:SUS-ETMX_L2_DAMP_MODE20_RMSLP_LOG10_OUTMON -inf
H1:SUS-ETMY_L2_DAMP_MODE16_RMSLP_LOG10_OUTMON -inf
H1:SUS-ETMY_L2_DAMP_MODE17_RMSLP_LOG10_OUTMON -inf
H1:SUS-ITMX_L2_DAMP_MODE10_RMSLP_LOG10_OUTMON -inf
H1:SUS-ITMX_L2_DAMP_MODE18_RMSLP_LOG10_OUTMON -inf
```
### Model
![image](/uploads/af531db6576110027c25491bb11314cf/image.png)
Various filter modules have gains of 0.0, inputs/outputs off causing a 0 to be passed to log.
### Solution
- Configure these filter modules to pass signal data.
-
## h1oaf
### Suspicious Channels
None, might be hard to debug. Reported error div/0 and invalid math.
### Model
### Solution
## h1bos
### Suspicious Channels
None, might be hard to debug. Reported error div/0 and invalid math.
### Model
### Solution
## h1omc
### Suspicious Channels
None, might be hard to debug. Reported error div/0 and invalid math.
### Model
### Solution
## h1lsc
Usually clears, but comes back after some time.
### Suspicious Channels
None, might be hard to debug. Reported error div/0 and invalid math.
### Model
### Solution
## h1sqz
Usually clears, but comes back after some time.
### Suspicious Channels
None, might be hard to debug. Reported error div/0 and invalid math.
### Model
### Solutionhttps://git.ligo.org/cds/software/advligorts/-/issues/560Permissions error and changes pending message overlap in SDF table.2023-04-11T16:31:07ZErik von ReisPermissions error and changes pending message overlap in SDF table.When the changes are pending and there are permissions problems with safe.snap, both messages appear on the SDF screen and both are unreadable because of the overlap.When the changes are pending and there are permissions problems with safe.snap, both messages appear on the SDF screen and both are unreadable because of the overlap.https://git.ligo.org/cds/software/advligorts/-/issues/559Detection of invalid floating point calculations.2023-03-31T20:05:44ZEzekiel DohmenDetection of invalid floating point calculations.# Two Implemented Methods
## Query the FPU
Periodically query and reset the FPU exceptions to detect invalid math operations and division by 0.
## Check for non-finite numbers filter histories.
Periodically check one value in each fil...# Two Implemented Methods
## Query the FPU
Periodically query and reset the FPU exceptions to detect invalid math operations and division by 0.
## Check for non-finite numbers filter histories.
Periodically check one value in each filter history for non-finite numbers.
# Test Stand Errors
## x2sqz Example
Whole error resolved by setting non-zero initial values.
![image](/uploads/39f24a19d7fb381413d3512d3de9ab6c/image.png)
## x2omcpi
Has a lot of FFT output being fed to log10 functions. Because the ADC data is zero, the FFT is zero and we try to calc log10(0)
## x2susproc
0s being fed to FFT and then log10. Probably not an issue on production.
## x2susprocpi
More 0s being passed around to log10. Probably not an issue on production.
## x2iopasc0
Filter `ASC0_AS_A_RF45_I_SUM` and `ASC0_AS_A_RF45_Q_SUM` get an +inf value in their histories on startup. `./cds/common/src/PHASE_WITH_CONTROL.c` does a division by sin result. When IPCs are just starting sin(0) = 0 -> div by zero.
```
sincos(R_radian,&sinR,&cosR);
sincos(D_radian,&sinD,&cosD);
sincos(R_radian+D_radian,&sinRD,&cosRD);
argout[0] = in1*(sinRD/sinD) - in2*(sinR/sinD);
argout[1] = in2*(cosR/sinD) - in1*(cosRD/sinD);
```
D_radian is fed from IPC receiver, so will be zero on startup. Probably need to check for zero before division.https://git.ligo.org/cds/software/advligorts/-/issues/558GDS Broadcaster DAQ does not always restart properly after DC restart2023-03-24T19:16:22ZKeith ThorneGDS Broadcaster DAQ does not always restart properly after DC restartBoth sites have the problem that when DC (data concentrator) is restarted with rts-transport@cps_recv service restart, the rts-daqd process on the GDS broadcaster properly restarts but still using an out-of-date channel list. A second (m...Both sites have the problem that when DC (data concentrator) is restarted with rts-transport@cps_recv service restart, the rts-daqd process on the GDS broadcaster properly restarts but still using an out-of-date channel list. A second (manual) restart is then neededJonathan HanksJonathan Hankshttps://git.ligo.org/cds/software/advligorts/-/issues/557NDS server rts-daqd does not always restart after rts-transport@cps_recv rest...2023-03-24T19:13:07ZKeith ThorneNDS server rts-daqd does not always restart after rts-transport@cps_recv restart on DCBoth sites have a continuing problem where sometimes the NDS server rts-daqd does not restart when rts-transport@cps_recv service is restarted on the data concentrator (DC)Both sites have a continuing problem where sometimes the NDS server rts-daqd does not restart when rts-transport@cps_recv service is restarted on the data concentrator (DC)Jonathan HanksJonathan Hankshttps://git.ligo.org/cds/software/advligorts/-/issues/556Daniel suggests a railing part2023-03-22T17:20:17ZErik von ReisDaniel suggests a railing partPart would force output to be within specified range. Infinite values would be railed. How NaNs are handled is an open question.
We could let the user select NaN handling from: set to min or max, set mid range, or last non NaN value...Part would force output to be within specified range. Infinite values would be railed. How NaNs are handled is an open question.
We could let the user select NaN handling from: set to min or max, set mid range, or last non NaN value.
Should have outputs showing high rail, low rail, and NaN errors.https://git.ligo.org/cds/software/advligorts/-/issues/555File /etc/advligorts/env replaced by a blank file after every upgrade2023-03-22T18:31:46ZArtemiy DmitrievFile /etc/advligorts/env replaced by a blank file after every upgradeWe see in all Birmingham cymac setups that the contents of `/etc/advligorts/env` are erased after every upgrade of advligorts-* packages. I couldn't find the source of the problem quickly, but suspect that it might have something to do w...We see in all Birmingham cymac setups that the contents of `/etc/advligorts/env` are erased after every upgrade of advligorts-* packages. I couldn't find the source of the problem quickly, but suspect that it might have something to do with `postinst` instructions for the `advligorts-common` package. On our machines, `/etc/advligorts/systemd_env` is a symlink for `/etc/advligorts/env` -- not sure if it might affect the installation scripts.
For reference, typical contents of `/etc/advligorts/env` on our machines:
```
$ cat /etc/advligorts/env
SITE=BHAM
IFO=B1
OPTRTCDS=/opt/rtcds/bham/b1
RCG_LIB_PATH=/home/controls/test_models/:/home/controls/models
local_dc_args='--systab=/etc/advligorts/rtsystab -b local_dc -m 100 -d /opt/rtcds/bham/b1/target/gds/param'
```Ezekiel DohmenEzekiel Dohmenhttps://git.ligo.org/cds/software/advligorts/-/issues/551resctrl for improved isolation of realtime tasks?2023-03-09T21:24:07ZChristopher Wipfresctrl for improved isolation of realtime tasks?Recently both [Intel](https://www.intel.com/content/www/us/en/architecture-and-technology/resource-director-technology.html) and [AMD](https://www.amd.com/system/files/TechDocs/56375_1.03_PUB.pdf) have added hardware features for allocat...Recently both [Intel](https://www.intel.com/content/www/us/en/architecture-and-technology/resource-director-technology.html) and [AMD](https://www.amd.com/system/files/TechDocs/56375_1.03_PUB.pdf) have added hardware features for allocating scarce CPU resources, like cache space and memory bandwidth. Linux has support for this under the banner of [resctrl](https://docs.kernel.org/x86/resctrl.html). Perhaps it's worth trying to see if this can help with isolating realtime tasks in advligorts?
Note that [a recent study](https://doi.org/10.1145/3534879.3534882) reported some promising results, but also some inconsistencies with Intel's implementation of this technology.https://git.ligo.org/cds/software/advligorts/-/issues/550Future Readiness: The RCG Will need to be capable of driving a DAC at 2^17 Hz...2023-03-06T17:59:39ZJeffrey KisselFuture Readiness: The RCG Will need to be capable of driving a DAC at 2^17 Hz (131072 Hz)LIGO custom built 32-bit DACs ([E2200440](https://dcc.ligo.org/LIGO-E2200440)) are slated for use in O5. As a part of their use, we want to explore driving the test inputs to the OMC DCPD sensing chain (e.g. like those in [LHO:62653](htt...LIGO custom built 32-bit DACs ([E2200440](https://dcc.ligo.org/LIGO-E2200440)) are slated for use in O5. As a part of their use, we want to explore driving the test inputs to the OMC DCPD sensing chain (e.g. like those in [LHO:62653](https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=62653) and [LHO:67760](https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=67760)) at as high a sampling frequency as 2^17 Hz (131072 Hz, some might colloquially refer to this as "128 kHz"), but at least at the native rate of the current IO chassis (2^16 Hz = 65536 Hz).
@erik.vonreis suggests that prepping the RCG for such a drive may not be that hard, but it needs some eyeballs on it at least.
FYI: This is NOT a priority before O4 starts. However, I might imagine @daniel.sigg and @marc.pirello might want this RCG change to support their prototyping efforts on test stands.https://git.ligo.org/cds/software/advligorts/-/issues/548Add EPICS channel for the timing card firmware rev in IOP models.2023-03-02T17:05:54ZEzekiel DohmenAdd EPICS channel for the timing card firmware rev in IOP models.https://git.ligo.org/cds/software/advligorts/-/issues/547Allow higher data rates for some front ends.2023-04-07T15:29:18ZErik von ReisAllow higher data rates for some front ends.@jeffrey-kissel requests we allow more than 8 MB/s from some models. In particular, he'd like to be able to open at least 8 but ideally 16 test points on h1iopomc0, a 524kHz model that outputs 2 MB/s per test point.
He would also like ...@jeffrey-kissel requests we allow more than 8 MB/s from some models. In particular, he'd like to be able to open at least 8 but ideally 16 test points on h1iopomc0, a 524kHz model that outputs 2 MB/s per test point.
He would also like to be able to add some number of these channels to the frames.
This plays into the idea of have a network-wide bandwidth control (or at least front end bandwidth control). larger, or non-statically sized DAQ data buffers would be needed.
Number of testpoints on OMC0 is expected to double for O5.https://git.ligo.org/cds/software/advligorts/-/issues/538RCG: add feature for checking the source has changed for a model2023-02-14T18:28:19ZErik von ReisRCG: add feature for checking the source has changed for a modelShow which file has changed. Don't actually build model. Something we could run nightly on all models to show which models are different.Show which file has changed. Don't actually build model. Something we could run nightly on all models to show which models are different.