... | ... | @@ -2,31 +2,35 @@ |
|
|
|
|
|
Guardian comes with `guardctrl`, which is an interface for controlling and supervising guardian nodes on a host system. It is a wrapper around [systemd](https://www.freedesktop.org/wiki/Software/systemd/), which is the built-in init and service supervision system standard on all major linux distributions. systemd handles stopping, starting, and logging of the guardian daemons. `guardctrl` is essentially a convenient wrapper around systemctl and journalctl which allows specifying nodes by name, as opposed to the underlying systemd service names.
|
|
|
|
|
|
Each guardian node is handled by a systemd "unit", which describes how the process should be handled. We use "templated" units, which is a common way to describe supervision for a set of common processes.
|
|
|
Each guardian node is handled by a systemd templated service unit, `guardian@.service`, which describes how the processes should be handled.
|
|
|
|
|
|
## host setup
|
|
|
|
|
|
The `guardctrl` package is available through the [LIGO Debian apt archives](http://apt.ligo-wa.caltech.edu/debian/), so once that archive needs to be enabled we install the package:
|
|
|
The `guardctrl` package is available through the [LIGO Debian apt archives](http://apt.ligo-wa.caltech.edu/debian/), so once that archive is enabled it can be installed directly:
|
|
|
```shell
|
|
|
# apt install guardctrl
|
|
|
```
|
|
|
The `guardctrl` package depends on `guardian` package, so you'll automatically get them both.
|
|
|
|
|
|
guardctrl assumes a systemd --user instance. For the LIGO site installations we therefore create a `guardian` user:
|
|
|
`guardctrl` uses a `systemd --user` instance. This means that `guardctrl` should always be invoked as the same user so that processes are managed in a unified way. The `guardctrl` interface knows that it's running as the correct user by the presence of a `~/.guardctrl-home` file.
|
|
|
|
|
|
For the LIGO site installations we therefore create a `guardian` user:
|
|
|
```shell
|
|
|
# adduser --gecos '' --uid 1010 --ingroup controls --disabled-password guardian
|
|
|
```
|
|
|
We use `uid=1010` to not collide with any of the other standard system users, and we add it to the `controls` group so that it can write archive and channel info to locations owned by the controls group.
|
|
|
|
|
|
NOTE: For a site setup where guardctrl will be accessed through a ~passwordless-SSH-interface, the guardian user should not have a password. Otherwise the guardian user can have a password as usual.
|
|
|
We use `uid=1010` to not collide with any of the other standard system users, and we add it to the `controls` group so that it can write archive and channel info to locations owned by the controls group. (NOTE: For a site setup where guardctrl will be accessed through a ~passwordless-SSH-interface, the guardian user should not have a password. Otherwise the guardian user can have a password as usual.)
|
|
|
|
|
|
### systemd and journald config
|
|
|
Once we've got the user that will handle supervision, we touch the `~/.guardctrl-home` file in the user's home directory:
|
|
|
```shell
|
|
|
# su guardian -c "touch ~/.guardctrl-home"
|
|
|
```
|
|
|
|
|
|
The guardian systemd --user setup needs to be configured to be "persistent" so that it won't be shut down if the guardian user is not logged in. This is handled with loginctl:
|
|
|
We need to inform the system systemd that the `guardctrl` user is "persistent", so that it's `systemd --user` process won't be shut down if the user is not logged in. We do this with `loginctl enable-linger`. So if we're running under the `guardian` user the command is:
|
|
|
```shell
|
|
|
# loginctl enable-linger guardian
|
|
|
```
|
|
|
|
|
|
The system journald needs to be configured to store all logs indefinitely. This is done by setting `Storage=persistent` in `/etc/systemd/journald.conf`:
|
|
|
For the LIGO sites, we want to store logs from all guardian processes in perpetuity. To do this, the journald system logger needs to be configured to store all logs indefinitely. This is done by setting `Storage=persistent` in `/etc/systemd/journald.conf`:
|
|
|
```
|
|
|
[Journal]
|
|
|
Storage=persistent
|
... | ... | @@ -35,14 +39,7 @@ Storage=persistent |
|
|
# systemctl force-reload systemd-journald
|
|
|
```
|
|
|
|
|
|
### guardctrl setup
|
|
|
|
|
|
The `guardctrl` interface knows that it's running as the correct user by the presence of a `~/.guardctrl-home` file. Touch this file in the `guardian` user home directory:
|
|
|
```shell
|
|
|
# su guardian -c "touch ~/.guardctrl-home"
|
|
|
```
|
|
|
|
|
|
The guardian systemd service unit expects an `/etc/guardian/local-env` environment file to exist, for providing any needed environment variables to the supervised guardian processes. Here's an example of the file for H1 at LHO:
|
|
|
The `guardian@.service` expects an `/etc/guardian/local-env` environment file to exist, for providing any needed environment variables to the supervised guardian processes. Here's an example of the file for H1 at LHO:
|
|
|
```shell
|
|
|
# cat /etc/guardian/local-env
|
|
|
IFO=H1
|
... | ... | |