guardian issueshttps://git.ligo.org/cds/software/guardian/-/issues2022-05-19T22:02:01Zhttps://git.ligo.org/cds/software/guardian/-/issues/34Manager still makes request to node manually unmanaged2022-05-19T22:02:01ZThomas ShafferManager still makes request to node manually unmanagedAs of Guardian version 1.2.0 this is a feature of the "loose" management style. The problem comes when a user is working on a node and manually UNmanages a node. It may seem obvious to the user that the manager will no longer make reques...As of Guardian version 1.2.0 this is a feature of the "loose" management style. The problem comes when a user is working on a node and manually UNmanages a node. It may seem obvious to the user that the manager will no longer make requests, but this is not true.
Example:
While working on the SQZ_MANAGER, Sheila had unmonitored this node to prevent ISC_LOCK (the manager) from moving it around. ISC_LOCK went to DOWN and was still able to make a request to SQZ_MANAGER.
ISC_LOCK log:
```
2019-06-10_20:42:18.700866Z ISC_LOCK executing state: DOWN (10)
2019-06-10_20:42:18.709163Z ISC_LOCK [DOWN.main] USERMSG 0: SQZ_MANAGER: REQUEST CHANGED (was: SQZ_READY_IFO, now: LOCKED_SEED_LOW)
2019-06-10_20:42:18.709833Z ISC_LOCK [DOWN.main] USERMSG 1: SQZ_MANAGER: NOT IN MANAGED MODE
2019-06-10_20:42:18.710262Z ISC_LOCK [DOWN.main] USERMSG 2: LASER_PWR: REQUEST CHANGED (was: POWER_35W, now: POWER_37W)
2019-06-10_20:42:18.713993Z ISC_LOCK [DOWN.main] Unstalling VIOLIN_DAMPING
2019-06-10_20:42:18.747731Z ISC_LOCK [DOWN.main] ezca: H1:SUS-ETMY_M0_DARM_DAMP_V_GAIN => 0
2019-06-10_20:42:18.791187Z ISC_LOCK [DOWN.main] ezca: H1:GRD-ALS_COMM_REQUEST => DOWN
2019-06-10_20:42:18.792761Z ISC_LOCK [DOWN.main] ezca: H1:GRD-ALS_DIFF_REQUEST => DOWN
2019-06-10_20:42:18.794219Z ISC_LOCK [DOWN.main] ezca: H1:GRD-ISC_DRMI_REQUEST => DOWN
2019-06-10_20:42:18.795783Z ISC_LOCK [DOWN.main] ezca: H1:GRD-OMC_LOCK_REQUEST => DOWN
2019-06-10_20:42:18.797656Z ISC_LOCK [DOWN.main] ezca: H1:GRD-IMC_LOCK_REQUEST => LOCKED
2019-06-10_20:42:18.799524Z ISC_LOCK [DOWN.main] ezca: H1:GRD-ALS_XARM_REQUEST => UNLOCKED
2019-06-10_20:42:18.801094Z ISC_LOCK [DOWN.main] ezca: H1:GRD-ALS_YARM_REQUEST => UNLOCKED
2019-06-10_20:42:18.802933Z ISC_LOCK [DOWN.main] ezca: H1:GRD-LASER_PWR_REQUEST => POWER_2W
2019-06-10_20:42:18.804507Z ISC_LOCK [DOWN.main] ezca: H1:GRD-PSL_FSS_REQUEST => READY_FOR_MC_LOCK
2019-06-10_20:42:18.807749Z ISC_LOCK [DOWN.main] ezca: H1:GRD-FAST_SHUTTER_REQUEST => DOWN
2019-06-10_20:42:18.809209Z ISC_LOCK [DOWN.main] ezca: H1:GRD-VIOLIN_DAMPING_REQUEST => TURN_OFF_DAMPING_ALL
2019-06-10_20:42:18.810686Z ISC_LOCK [DOWN.main] ezca: H1:GRD-SQZ_MANAGER_REQUEST => SQZ_READY_IFO
```
SQZ_MANAGER log:
```
2019-06-10_20:42:18.846154Z SQZ_MANAGER REQUEST: SQZ_READY_IFO
2019-06-10_20:42:18.846154Z SQZ_MANAGER calculating path: LOCKED_SEED_LOW->SQZ_READY_IFO
2019-06-10_20:42:18.846154Z SQZ_MANAGER new target: DOWN
2019-06-10_20:42:18.846154Z SQZ_MANAGER GOTO REDIRECT
```
H1:GRD-SQZ_MANAGER_MODE was 0 (Auto) at that time.guardian 1.5Jameson Rollinsjameson.rollins@ligo.orgJameson Rollinsjameson.rollins@ligo.orghttps://git.ligo.org/cds/software/guardian/-/issues/48State number of subordinate is not an available attribute from manager2020-05-01T03:45:02ZThomas ShafferState number of subordinate is not an available attribute from managerShould only be a rare use case, but it would be nice.
Current attrs:
('OP',
'MODE',
'MANAGER',
'REQUEST',
'STATE_S',
'TARGET_S',
'REQUEST_S',
'STATUS',
'STALLED',
'ERROR',
'NOTIFICATION')Should only be a rare use case, but it would be nice.
Current attrs:
('OP',
'MODE',
'MANAGER',
'REQUEST',
'STATE_S',
'TARGET_S',
'REQUEST_S',
'STATUS',
'STALLED',
'ERROR',
'NOTIFICATION')guardian 1.5https://git.ligo.org/cds/software/guardian/-/issues/45guardutil command to resolve state index2020-05-14T18:23:16ZJameson Rollinsjameson.rollins@ligo.orgguardutil command to resolve state indexguardian 1.5https://git.ligo.org/cds/software/guardian/-/issues/42add "nominal manager" setting2020-05-14T18:22:38ZThomas Shafferadd "nominal manager" settingNOMINAL_MANAGER would be specified in the system module, similar to NOMINAL_STATE. This would allow users or other nodes to take over management of a subordinate, which would prevent the nominal manager from issuing requests, but then a...NOMINAL_MANAGER would be specified in the system module, similar to NOMINAL_STATE. This would allow users or other nodes to take over management of a subordinate, which would prevent the nominal manager from issuing requests, but then allow for slotting it back into the nominal management hierarchy when done.guardian 1.5https://git.ligo.org/cds/software/guardian/-/issues/36Remove "Manage" button from state selection screen2020-05-01T03:47:46ZThomas ShafferRemove "Manage" button from state selection screenThis button is used more with the erroneous thought that the node will regain its previous manager. The actual intent of management by the "USER" is rarely, if ever, used.
An epics record of the Manager for the node on this screen could...This button is used more with the erroneous thought that the node will regain its previous manager. The actual intent of management by the "USER" is rarely, if ever, used.
An epics record of the Manager for the node on this screen could also be a useful addition.guardian 1.5https://git.ligo.org/cds/software/guardian/-/issues/33guardctrl log limiting lines when date range is specified2020-05-14T18:21:57ZJameson Rollinsjameson.rollins@ligo.orgguardctrl log limiting lines when date range is specifiedBy default guardctrl log limits the number of lines it outputs. However, if the user specifies a bounded date range, this can cause confusing truncation of the logs. E.g. if the range A->B is requested, and there are 200 lines in that ...By default guardctrl log limits the number of lines it outputs. However, if the user specifies a bounded date range, this can cause confusing truncation of the logs. E.g. if the range A->B is requested, and there are 200 lines in that range, guardctrl log will only output the first 100 lines in that range, which is confusing.
See https://git.ligo.org/jameson.rollins/locklost/issues/95guardian 1.5https://git.ligo.org/cds/software/guardian/-/issues/28per-node manager overview MEDM screen2020-05-14T18:21:42ZJameson Rollinsjameson.rollins@ligo.orgper-node manager overview MEDM screenEach node should have a MANAGEMENT overview screen, similar to the USERMSG screen that lists all subordinate nodes, their state/status, and the NodeManager's internal state in regards to the node (such as whether the node is tightly or l...Each node should have a MANAGEMENT overview screen, similar to the USERMSG screen that lists all subordinate nodes, their state/status, and the NodeManager's internal state in regards to the node (such as whether the node is tightly or loosely managed (although I guess this can just be determined by looking at the MANAGED and MANAGER state of the subordinate itself)).guardian 1.5https://git.ligo.org/cds/software/guardian/-/issues/27user feedback about why node not READY, OK2020-05-01T03:47:26ZJameson Rollinsjameson.rollins@ligo.orguser feedback about why node not READY, OKNow that the OK state of subordinate nodes affects the READY, and therefore OK, bits, there needs to be better feedback to the user about why a node is not READY/OK.
A status bar with indicators for all the pre-conditions would be one w...Now that the OK state of subordinate nodes affects the READY, and therefore OK, bits, there needs to be better feedback to the user about why a node is not READY/OK.
A status bar with indicators for all the pre-conditions would be one way. But there also needs to be an indicator of which subordinates are showing issues. Maybe a manager overview screen?guardian 1.5https://git.ligo.org/cds/software/guardian/-/issues/25update channel lists on upgrade2020-05-01T03:47:23ZJameson Rollinsjameson.rollins@ligo.orgupdate channel lists on upgradeGuardian upgrades can change the list of desired acquired channels. Somehow upgrades of the guardctrl package should initiate an update of the channel list.
At the very least there should be a guardctrl command that does the update, e....Guardian upgrades can change the list of desired acquired channels. Somehow upgrades of the guardctrl package should initiate an update of the channel list.
At the very least there should be a guardctrl command that does the update, e.g. "guardctrl update-channels" or the like.guardian 1.5https://git.ligo.org/cds/software/guardian/-/issues/20indicate NOMINAL state in graph2020-05-01T03:47:21ZJameson Rollinsjameson.rollins@ligo.orgindicate NOMINAL state in graphwith border, background halo, etc.with border, background halo, etc.guardian 1.5https://git.ligo.org/cds/software/guardian/-/issues/19guardutil archive-clone throws error with no arguments2020-05-01T03:47:19ZJameson Rollinsjameson.rollins@ligo.orgguardutil archive-clone throws error with no arguments```
jameson.rollins@opsportal:~ 0$ guardutil archive-clone
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.7/...```
jameson.rollins@opsportal:~ 0$ guardutil archive-clone
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/lib/python2.7/site-packages/guardian/guardutil.py", line 523, in <module>
main()
File "/usr/lib/python2.7/site-packages/guardian/guardutil.py", line 520, in main
args.func(args)
File "/usr/lib/python2.7/site-packages/guardian/guardutil.py", line 367, in archive_clone
archive_path = os.path.join(ARCHIVE_ROOT, args.system)
File "/usr/lib64/python2.7/posixpath.py", line 75, in join
if b.startswith('/'):
AttributeError: 'NoneType' object has no attribute 'startswith'
jameson.rollins@opsportal:~ 1$
```guardian 1.5https://git.ligo.org/cds/software/guardian/-/issues/80guardlog should never limit number of output log lines2022-05-23T17:39:49ZJameson Rollinsjameson.rollins@ligo.orgguardlog should never limit number of output log linesThis should just be a matter of adjusting the options sent to journalctl to specify not limiting the log lines.This should just be a matter of adjusting the options sent to journalctl to specify not limiting the log lines.https://git.ligo.org/cds/software/guardian/-/issues/79guardlog should produce informative message if there are no logs available fo...2022-05-23T17:40:15ZJameson Rollinsjameson.rollins@ligo.orgguardlog should produce informative message if there are no logs available for the given searchThis is prompted by an issue that came up at LLO where the guardlog window was closing immediately, which is probably because there are no logs between the specified times:
https://services1.ligo-la.caltech.edu/FRS/show_bug.cgi?id=23791This is prompted by an issue that came up at LLO where the guardlog window was closing immediately, which is probably because there are no logs between the specified times:
https://services1.ligo-la.caltech.edu/FRS/show_bug.cgi?id=23791https://git.ligo.org/cds/software/guardian/-/issues/91Guardian raises ModuleNotFoundError on Python 3.122024-03-19T08:35:12ZDuncan Macleodduncan.macleod@ligo.orgGuardian raises ModuleNotFoundError on Python 3.12The guardian code is attempting to import the `imp` module, which was removed in Python 3.12, and so raises a `ModuleNotFoundError`:
```console
$ python3 --version
Python 3.12.0
$ git describe HEAD
1.5.1-8-g6d76afa
$ PYTHONPATH=lib pyth...The guardian code is attempting to import the `imp` module, which was removed in Python 3.12, and so raises a `ModuleNotFoundError`:
```console
$ python3 --version
Python 3.12.0
$ git describe HEAD
1.5.1-8-g6d76afa
$ PYTHONPATH=lib python3 -c "import guardian"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/duncan/git/guardian/lib/guardian/__init__.py", line 14, in <module>
from .system import GuardSystem
File "/home/duncan/git/guardian/lib/guardian/system.py", line 5, in <module>
import imp
ModuleNotFoundError: No module named 'imp'
```https://git.ligo.org/cds/software/guardian/-/issues/90Add pem to the userapps directory search2024-03-18T21:52:04ZThomas ShafferAdd pem to the userapps directory searchNow that there are pem guardians, it would be good to have guardian search in these directories in userapps. Currently, the users are creating sym links to paths in the guardian search paths.Now that there are pem guardians, it would be good to have guardian search in these directories in userapps. Currently, the users are creating sym links to paths in the guardian search paths.https://git.ligo.org/cds/software/guardian/-/issues/89filter setting fails if front end offline2023-04-27T00:28:45ZKarla Ramirezfilter setting fails if front end offlineThis was supposed to be using EPICS_NOFAIL, but it still failed with a filter channel that wasn't available:
```
2023-04-27T00:18:20.317Z ISC_LOCK [DOWN.main] ezca: L1:SUS-ETMY_L2_OLDAMP_Y_RSET => 2
2023-04-27T00:18:21.951Z ISC_LOCK [DO...This was supposed to be using EPICS_NOFAIL, but it still failed with a filter channel that wasn't available:
```
2023-04-27T00:18:20.317Z ISC_LOCK [DOWN.main] ezca: L1:SUS-ETMY_L2_OLDAMP_Y_RSET => 2
2023-04-27T00:18:21.951Z ISC_LOCK [DOWN.main] ezca: L1:SUS-ETMX_L1_DRIVEALIGN_L2P_RSET => 2
2023-04-27T00:18:21.960Z ISC_LOCK [DOWN.main] ezca: L1:SUS-ETMX_L1_DRIVEALIGN_L2Y_RSET => 2
2023-04-27T00:18:22.098Z ISC_LOCK [DOWN.main] ezca: L1:GRD-SQZ_MANAGER_REQUEST => INIT
2023-04-27T00:18:34.579Z ISC_LOCK [DOWN.main] ezca: L1:ASC-DHARD_P => OFF: OUTPUT
2023-04-27T00:18:38.580Z ISC_LOCK W: Traceback (most recent call last):
File "/opt/conda/base/envs/cds-py39-2023030301/lib/python3.9/site-packages/guardian/worker.py", line 494, in run
retval = statefunc()
File "/opt/rtcds/userapps/release/isc/l1/guardian/ISC_LOCK.py", line 558, in main
ezca.switch('ASC-' + stiffness + '_' + rot, 'OUTPUT', 'OFF')
File "/opt/conda/base/envs/cds-py39-2023030301/lib/python3.9/site-packages/ezca/ezca.py", line 521, in switch
return LIGOFilter(sfm_name, self).switch(*args, **kwargs)
File "/opt/conda/base/envs/cds-py39-2023030301/lib/python3.9/site-packages/ezca/ligofilter.py", line 489, in switch
mask |= self.__get_mask('on', switch_actions['ON'])
File "/opt/conda/base/envs/cds-py39-2023030301/lib/python3.9/site-packages/ezca/ligofilter.py", line 600, in __get_mask
for sw_name, current_value in self.get_current_switch_dict().items():
File "/opt/conda/base/envs/cds-py39-2023030301/lib/python3.9/site-packages/ezca/ligofilter.py", line 242, in get_current_switch_dict
return {sw_name: int(getattr(self, '_'+sw_name+'R').get()) for sw_name in const.FILTER_SW_NAMES}
File "/opt/conda/base/envs/cds-py39-2023030301/lib/python3.9/site-packages/ezca/ligofilter.py", line 242, in <dictcomp>
return {sw_name: int(getattr(self, '_'+sw_name+'R').get()) for sw_name in const.FILTER_SW_NAMES}
ValueError: cannot convert float NaN to integer
```https://git.ligo.org/cds/software/guardian/-/issues/88NodeManager should be able to access node index2023-02-17T23:58:46ZThomas ShafferNodeManager should be able to access node indexIt would be very helpful for a manager to get the index of a subordinate.It would be very helpful for a manager to get the index of a subordinate.Thomas ShafferThomas Shafferhttps://git.ligo.org/cds/software/guardian/-/issues/87Node needed a restart after the name of nominal state was changed.2023-02-17T19:23:25ZCamilla ComptonNode needed a restart after the name of nominal state was changed.![image](/uploads/c0ff344e4bf401a87324b3e886952a33/image.png)
This is the error once we reloaded the VIOLIN_DAMPING guardian after the old nominal state DAMPING_ON_DC was removed and renamed to DAMP_VIOLINS_FULL_POWER. This was solved ...![image](/uploads/c0ff344e4bf401a87324b3e886952a33/image.png)
This is the error once we reloaded the VIOLIN_DAMPING guardian after the old nominal state DAMPING_ON_DC was removed and renamed to DAMP_VIOLINS_FULL_POWER. This was solved by command `guardctrl restart VIOLIN_DAMPING`. [Svn commit 24745](https://redoubt.ligo-wa.caltech.edu/viewvc/cds_user_apps?view=revision&revision=24745).
@thomas-shaffer thinks this shouldn't have been needed.https://git.ligo.org/cds/software/guardian/-/issues/86Guardian returned True in the middle of main state?2023-01-24T17:43:51ZJenne DriggersGuardian returned True in the middle of main state?![OMC_guardian_more](/uploads/8d1b314809784dea1e6956a2d6569082/OMC_guardian_more.png)
From LHO control room chat: https://chat.ligo.org/ligo/pl/84zwqfdd83r4dfki4qwtz8bj3w
Sheila: @Jameson Rollins @Thomas Shaffer Here is a screenshot of ...![OMC_guardian_more](/uploads/8d1b314809784dea1e6956a2d6569082/OMC_guardian_more.png)
From LHO control room chat: https://chat.ligo.org/ligo/pl/84zwqfdd83r4dfki4qwtz8bj3w
Sheila: @Jameson Rollins @Thomas Shaffer Here is a screenshot of the OMC gaurdian, when it gets a redirect request while in the OMC_LSC_ON state. It does some but not all of the steps in OMC_LSC_ON, but then returns the next state, OMC_LOCKED before completing main.
1:09 PM
So the IFO transitioned to DC readout even though the OMC wasn't locked. We only noticed this because by luck the OMC was staying close enough to on resonance to keep the IFO locked.
Jenne: One thing I wonder (haven't figured out yet) is if ISC_LOCK was requesting the OMC_LOCK to DOWN, but then OMC_LOCK was able to jump from halfway through the main state it was in to the next in sequence, rather than to DOWN.
Sheila: yeah, I think there's no situation where it should have returned true halfway through main. And it has to return true to move onto the next state in the graph. But there might be a bug in the way the redirect is handled that allows this to happen?https://git.ligo.org/cds/software/guardian/-/issues/85guardctrl not able to show pre-update log entries2022-12-14T17:23:08ZErik von Reisguardctrl not able to show pre-update log entriesguardctrl log command cannot show logs from before the update
At LHO I upgraded Guardian to deb 11. A new log directory was automatically created. The log files have a new (seemingly random) hash embedded in their name. Old log files...guardctrl log command cannot show logs from before the update
At LHO I upgraded Guardian to deb 11. A new log directory was automatically created. The log files have a new (seemingly random) hash embedded in their name. Old log files are still preserved in their original directory.