lvalert issueshttps://git.ligo.org/lscsoft/lvalert/-/issues2023-06-22T07:44:21Zhttps://git.ligo.org/lscsoft/lvalert/-/issues/20sleekxmpp dropped from Debian Bookworm - replace with slixmpp?2023-06-22T07:44:21ZSteffen Grunewaldsleekxmpp dropped from Debian Bookworm - replace with slixmpp?See Debian bug [1028067](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028067) - while Bullseye was released with `(python3-)sleekxmpp`, this package has been dropped from Bookworm in Jan 2023. The suggestion is to replace it with a...See Debian bug [1028067](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028067) - while Bullseye was released with `(python3-)sleekxmpp`, this package has been dropped from Bookworm in Jan 2023. The suggestion is to replace it with a derivative, `slixmpp`.https://git.ligo.org/lscsoft/lvalert/-/issues/19lvalert_admin broken on both conda igwn-py2 and igwn-py37 environments2020-03-11T17:04:37ZFranco Carbognanilvalert_admin broken on both conda igwn-py2 and igwn-py37 environmentsAt the moment lvalert_admin is broken on both igwn-py27 and igwn-py37
(igwn-py27) [carbognani@ui01-virgo ~]$ which lvalert_admin
/cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/envs/igwn-py27/bin/lvalert_admin
(igwn-py27) [carbognani@ui...At the moment lvalert_admin is broken on both igwn-py27 and igwn-py37
(igwn-py27) [carbognani@ui01-virgo ~]$ which lvalert_admin
/cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/envs/igwn-py27/bin/lvalert_admin
(igwn-py27) [carbognani@ui01-virgo ~]$ lvalert_admin
Traceback (most recent call last):
File "/cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/envs/igwn-py27/bin/lvalert_admin", line 23, in <module>
import libxml2
ImportError: No module named libxml2
(igwn-py37) [carbognani@ui01-virgo ~]$ lvalert_admin
Traceback (most recent call last):
File "/cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/envs/igwn-py37/bin/lvalert_admin", line 23, in <module>
import libxml2
ModuleNotFoundError: No module named 'libxml2'https://git.ligo.org/lscsoft/lvalert/-/issues/18Links in README file are broken2019-12-06T18:28:19ZLeo P. SingerLinks in README file are brokenThese links in the README file are broken:
* https://wiki.ligo.org/DASWG/LVAlert
* https://bugs.ligo.org/redmine/projects/lvalertThese links in the README file are broken:
* https://wiki.ligo.org/DASWG/LVAlert
* https://bugs.ligo.org/redmine/projects/lvalerthttps://git.ligo.org/lscsoft/lvalert/-/issues/17ligo-lvalert packages uninstallable on Debian Buster2020-09-04T14:56:10ZSteffen Grunewaldligo-lvalert packages uninstallable on Debian BusterAn attempt to install `python-ligo-lvalert` and `python3-ligo-lvalert` on Debian Buster reveals that, while the packages built fine against python{,3}-pyasn1 0.4.2-3 and python{,3}-pyasn1-modules 0.2.1-0.2, for some reason the dependenci...An attempt to install `python-ligo-lvalert` and `python3-ligo-lvalert` on Debian Buster reveals that, while the packages built fine against python{,3}-pyasn1 0.4.2-3 and python{,3}-pyasn1-modules 0.2.1-0.2, for some reason the dependencies have been limited to the version range (>= 0.1.8 <= 0.3.7) for *-pyasn1 and (>= 0.0.5 <= 0.1.5) for *-pyasn1-modules.
These ranges are hardcoded in `ligo_lvalert.egg-info/requires.txt` and `setup.cfg` - is there a hard reason to do so?
Any workaround to make this package installable asap? Just patch away the upper limits, and rebuild?https://git.ligo.org/lscsoft/lvalert/-/issues/16Openfire breaks after Java/Mariadb update2019-06-28T20:10:02ZAlexander PaceOpenfire breaks after Java/Mariadb updateNot sure if this is the best place to put this (new wiki page maybe?). I encountered a server-side issue when testing software updates on the `lvalert-test` box before trying the same procedure on production and playground.
The list of...Not sure if this is the best place to put this (new wiki page maybe?). I encountered a server-side issue when testing software updates on the `lvalert-test` box before trying the same procedure on production and playground.
The list of packages from `apt-get upgrade`:
```
The following packages were automatically installed and are no longer required:
linux-image-4.9.0-3-amd64 linux-image-4.9.0-4-amd64
Use 'apt autoremove' to remove them.
The following packages have been kept back:
default-jre default-jre-headless icedtea-netx linux-image-amd64 mariadb-client-10.1
mariadb-server mariadb-server-10.1 mariadb-server-core-10.1
The following packages will be upgraded:
apt apt-transport-https apt-utils base-files bind9-host ca-certificates-java certbot curl dbus
dirmngr dnsutils gnupg gnupg-agent gpgv icedtea-netx-common java-common libapt-inst2.0
libapt-pkg5.0 libbind9-140 libc-bin libc-dev-bin libc-l10n libc6 libc6-dev libcups2 libcurl3
libcurl3-gnutls libdbus-1-3 libdns-export162 libdns162 libisc-export160 libisc160 libisccc140
libisccfg140 libjs-jquery libldb1 liblwres141 libmariadbclient-dev libmariadbclient18
libpam-systemd libpng16-16 libpq5 libsmbclient libssh2-1 libssl1.0.2 libsystemd0 libudev1
libwayland-client0 libwayland-cursor0 libwayland-server0 libwbclient0 libxapian30 linux-libc-dev
locales mariadb-client-core-10.1 mariadb-common multiarch-support openjdk-8-jre
openjdk-8-jre-headless openssh-client openssh-server openssh-sftp-server osg-ca-certs
puppet-agent python-ldb python-samba python3-acme python3-certbot python3-cryptography
python3-josepy rsync samba-common samba-common-bin samba-libs smbclient systemd systemd-sysv
tzdata udev unzip vim vim-common vim-runtime vim-tiny wget xxd
86 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.
Need to get 105 MB of archives.
After this operation, 15.0 MB of additional disk space will be used.
Do you want to continue? [Y/n]
```
and from `apt-get dist-upgrade`:
```
The following packages were automatically installed and are no longer required:
icedtea-netx icedtea-netx-common linux-image-4.9.0-3-amd64 linux-image-4.9.0-4-amd64
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
default-java-plugin icedtea-8-plugin
The following NEW packages will be installed:
libconfig-inifiles-perl linux-image-4.9.0-9-amd64
The following packages will be upgraded:
default-jre default-jre-headless icedtea-netx linux-image-amd64 mariadb-client-10.1
mariadb-server mariadb-server-10.1 mariadb-server-core-10.1
8 upgraded, 2 newly installed, 2 to remove and 0 not upgraded.
Need to get 55.8 MB of archives.
After this operation, 193 MB of additional disk space will be used.
Do you want to continue? [Y/n]
```
Everything upgraded fine, but I noticed this in the output from apt-get:
```
Current default time zone: 'America/Chicago'
Local time is now: Thu Jun 27 14:04:36 CDT 2019.
Universal Time is now: Thu Jun 27 19:04:36 UTC 2019.
Run 'dpkg-reconfigure tzdata' if you wish to change it.
```
Important to note that for later. After the upgrade, `lvalert-test` because unresponsive to clients and I couldn't access the web interface. `service openfire status` still showed it running, but there was this in the error log:
```
2019.06.27 14:13:44 org.jivesoftware.openfire.auth.DefaultAuthProvider - User SQL failure:
java.sql.SQLException: ConnectionManager.getConnection() failed to obtain a connection after 11 retries. The exception from the last attempt is as follows: java.sql.SQLException: The server time zone value 'CDT' is unrecognized or represents more than one time zone. You must configure either the server or JDBC driver (via the serverTimezone configuration property) to use a more specifc time zone value if you want to utilize time zone support.
```
I googled around and found the following posts with similar issues:
https://bugs.mysql.com/bug.php?id=85816
https://stackoverflow.com/questions/26515700/mysql-jdbc-driver-5-1-33-time-zone-issue
https://stackoverflow.com/questions/52906554/server-time-zone-value-cdt-is-unrecognized-or-represents-more-than-one-time-zo
I manually opened the `openfire.xml` configuration file and edited the mysql connection string to look like this:
```
<serverURL>jdbc:mysql://localhost:3306/openfire?rewriteBatchedStatements=true&useLegacyDatetimeCode=false&serverTimezone=America/Chicago</serverURL>
```
And then i restarted openfire, and tests have been passing since. Leaving this ticket open until I patch the production servers.Alexander PaceAlexander Pacehttps://git.ligo.org/lscsoft/lvalert/-/issues/14Debian packaging: Please use proper build numbers2022-01-07T10:53:12ZSteffen GrunewaldDebian packaging: Please use proper build numbersFor non-native Debian packages, a build number should be given. Please change (1.5.5) to (1.5.5-1) etc. in debian/changelog upstream.
(Currently a -2 has been built. No need for a re-release as debian/* files are being kept in a separate...For non-native Debian packages, a build number should be given. Please change (1.5.5) to (1.5.5-1) etc. in debian/changelog upstream.
(Currently a -2 has been built. No need for a re-release as debian/* files are being kept in a separate file belonging to the source package.)https://git.ligo.org/lscsoft/lvalert/-/issues/13Implement TLSv1.1+ encryption2019-02-14T14:48:10ZAlexander PaceImplement TLSv1.1+ encryptionI'm (@alexander\-pace) starting this ticket to track some changes I had to make server-side and client-side changes to get it to work.
**Client-side:** This was pretty easy. [This commit](https://git.ligo.org/lscsoft/lvalert/commit/5a6...I'm (@alexander\-pace) starting this ticket to track some changes I had to make server-side and client-side changes to get it to work.
**Client-side:** This was pretty easy. [This commit](https://git.ligo.org/lscsoft/lvalert/commit/5a6c2ece9513542b55d1b2fa8da338010b17089e) changes the ssl protocol to TLSv1.1. Curiously, even though ssl was being imported, it wasn't being used to set the protocol anywhere in the code. Also, SleekXMPP had the default protocol set to [ssl.PROTOCOL_SSLv23](https://github.com/fritzy/SleekXMPP/blob/develop/sleekxmpp/xmlstream/xmlstream.py#L133), and according to the docs:
```
#: Default ssl_version is ``PROTOCOL_SSLv23``, but despite the name,
#: ``PROTOCOL_SSLv23`` means to enable all possible SSL/TLS versions.
#: In addition, SSLv2 & SSLv3 is insecure, we have explictly disabled
#: them during the connections, which is considered as the best practice.
#: Thus, ironically, ``PROTOCOL_SSLv23`` enables everything except SSLv2/3.
```
But, despite that, when disabling TLSv1 in the server settings (see below), LVAlert with sleekxmpp would not connect unless explicitly setting the ssl.protocol to TLSV1_1.
**Server-side:** As far as I can tell, there are two settings that need to be changed. The first is under [Advanced security settings](http://lvalert-test.cgca.uwm.edu:9090/connection-settings-advanced.jsp?connectionType=SOCKET_C2S&connectionMode=plain); uncheck every protocol except TLSv1.1 and TLSv1.2:
![Screen_Shot_2019-02-13_at_4.42.36_PM](/uploads/c8f8b40de62a30ab245fea629124071f/Screen_Shot_2019-02-13_at_4.42.36_PM.png)
And then change the system property `xmpp.socket.ssl.client.certificate.accept-selfsigned` to true:
![Screen_Shot_2019-02-13_at_4.44.25_PM](/uploads/dc994357a2a7c8d28d4a4510a66f8e48/Screen_Shot_2019-02-13_at_4.44.25_PM.png)
It was initially set to false by default. I have no idea how this was working before now.
Either way, forcing encryption >= TLSv1.1 allows the sleekxmpp clients to work, but breaks the overseer. So, I disabled this for now until I figure out how to fix that.Alexander PaceAlexander Pacehttps://git.ligo.org/lscsoft/lvalert/-/issues/12Request release 1.5.52019-01-23T13:40:59ZDuncan Macleodduncan.macleod@ligo.orgRequest release 1.5.5The current `ligo-lvalert` release is unusable because of the bug fixed in !17, it would be great to get a new release on pypi that includes the fix. Thanks.The current `ligo-lvalert` release is unusable because of the bug fixed in !17, it would be great to get a new release on pypi that includes the fix. Thanks.https://git.ligo.org/lscsoft/lvalert/-/issues/11pyxmpp.exceptions.HostMismatch when forwarding to new hosts.2018-08-24T16:26:58ZAlexander Pacepyxmpp.exceptions.HostMismatch when forwarding to new hosts.This is a migration of [Redmine issue #6163](https://bugs.ligo.org/redmine/issues/6163).
**Original Description (2018-06-22):**
>>>
I encountered this issue when using a cname to forward traffic from (for example) lvalert.ligo.org to l...This is a migration of [Redmine issue #6163](https://bugs.ligo.org/redmine/issues/6163).
**Original Description (2018-06-22):**
>>>
I encountered this issue when using a cname to forward traffic from (for example) lvalert.ligo.org to lvalert.cgca.ligo.org:
```
[alexander.pace@ldas-pcdev10 ~]$ lvalert_admin -a alexander.pace -s lvalert.ligo.org -m -v
connecting...
*** State changed: resolving srv (u'lvalert.ligo.org', 'xmpp-client') ***
*** State changed: resolving u'lvalert.ligo.org' ***
*** State changed: connecting ('129.89.61.153', 5222) ***
*** State changed: connected ('129.89.61.153', 5222) ***
build pubsub stanza...
Getting nodes
sending message...
Traceback (most recent call last):
File "/bin/lvalert_admin", line 299, in <module>
s.loop(1)
File "/usr/lib64/python2.7/site-packages/pyxmpp/client.py", line 242, in loop
act=stream.loop_iter(timeout)
File "/usr/lib64/python2.7/site-packages/pyxmpp/streambase.py", line 598, in loop_iter
return self._loop_iter(timeout)
File "/usr/lib64/python2.7/site-packages/pyxmpp/streambase.py", line 616, in _loop_iter
self._process()
File "/usr/lib64/python2.7/site-packages/pyxmpp/streamtls.py", line 179, in _process
StreamBase._process(self)
File "/usr/lib64/python2.7/site-packages/pyxmpp/streambase.py", line 637, in _process
self._read()
File "/usr/lib64/python2.7/site-packages/pyxmpp/streamtls.py", line 174, in _read
StreamBase._read(self)
File "/usr/lib64/python2.7/site-packages/pyxmpp/streambase.py", line 661, in _read
self._feed_reader(r)
File "/usr/lib64/python2.7/site-packages/pyxmpp/streambase.py", line 677, in _feed_reader
r=self._reader.feed(data)
File "/usr/lib64/python2.7/site-packages/pyxmpp/xmlextra.py", line 537, in feed
return self.reader.feed(s)
File "/usr/lib64/python2.7/site-packages/pyxmpp/xmlextra.py", line 46, in _stream_start
self.stream_start(doc)
File "/usr/lib64/python2.7/site-packages/pyxmpp/streambase.py", line 366, in stream_start
raise HostMismatch
pyxmpp.exceptions.HostMismatch
It also occurred when using an A Record to forward traffic from *.cgca.ligo.org to *.ligo.uwm.edu.
```
Unfortunately, this makes the server upgrade/transition not as seamless as I would have hoped. I think the course of action should be to: * Take down the LVAlert servers for an extended period on Tuesday (June 26) * Change the DNS entries to so that *.cgca.uwm.edu point to the new server's IP's. * Ensure the system hostname and puppet configuration are consistent with the new domain.
After that, when I do a final database override with the old server's openfire database, the xmpp.fqdn and xmpp.hostname should be reset to the old value, and should be consistent with the (now) current hostname, and openfire should work.
After the MDC, I can modify the client software to relax this exception as we transition servers and domains.
As a side note, when the new domains get stood up, their .netrc files will have to be modified to reflect the change. (put here as more of a mental reminder to myself).
>>>
**Update:**
Leaving this ticket open, since it will eventually become an issue once LVAlert gets fully migrated to the *.ligo.uwm.edu domain.Alexander PaceAlexander Pacehttps://git.ligo.org/lscsoft/lvalert/-/issues/10Duplicate subscriptions; can't unsubscribe2018-08-24T16:21:44ZAlexander PaceDuplicate subscriptions; can't unsubscribeThis is a migration of [Redmine issue #5576](https://bugs.ligo.org/redmine/issues/5576).
**Original Description (2017-08-21):**
>>>
You can subscribe two or more times to an LVAlert node and you definitely shouldn't be able to do that....This is a migration of [Redmine issue #5576](https://bugs.ligo.org/redmine/issues/5576).
**Original Description (2017-08-21):**
>>>
You can subscribe two or more times to an LVAlert node and you definitely shouldn't be able to do that. And once you have multiple subscriptions, you can't unsubscribe at all. Here's a long example:
Subscribe to a node:
(gracedb)[tanner.prestegard@tanner-test:~/lvalert_testing]$ lvalert_admin -s lvalert-test.cgca.uwm.edu --subscriptions
Node: tanner_test1 [ subscribed subid=VO5QO2G7tSd85qN98yU32nacN4gvtKfTfJ10ukjE]
(gracedb)[tanner.prestegard@tanner-test:~/lvalert_testing]$ lvalert_admin -s lvalert-test.cgca.uwm.edu --subscribe --node external_snews
Successfully subscribed to node external_snews
(gracedb)[tanner.prestegard@tanner-test:~/lvalert_testing]$ lvalert_admin -s lvalert-test.cgca.uwm.edu --subscriptions
Node: tanner_test1 [ subscribed subid=VO5QO2G7tSd85qN98yU32nacN4gvtKfTfJ10ukjE]
Node: external_snews [ subscribed subid=AMrdDCV36Nj2SzIy5Kgh6FI83LwJrYewGEaE6L6r]
Unsubscribe from a node:
(gracedb)[tanner.prestegard@tanner-test:~/lvalert_testing]$ lvalert_admin -s lvalert-test.cgca.uwm.edu --unsubscribe --node external_snews
Successfully completed operation
(gracedb)[tanner.prestegard@tanner-test:~/lvalert_testing]$ lvalert_admin -s lvalert-test.cgca.uwm.edu --subscriptions
Node: tanner_test1 [ subscribed subid=VO5QO2G7tSd85qN98yU32nacN4gvtKfTfJ10ukjE]
Subscribe to a node twice:
(gracedb)[tanner.prestegard@tanner-test:~/lvalert_testing]$ lvalert_admin -s lvalert-test.cgca.uwm.edu --subscribe --node external_snews
Successfully subscribed to node external_snews
(gracedb)[tanner.prestegard@tanner-test:~/lvalert_testing]$ lvalert_admin -s lvalert-test.cgca.uwm.edu --subscriptions
Node: tanner_test1 [ subscribed subid=VO5QO2G7tSd85qN98yU32nacN4gvtKfTfJ10ukjE]
Node: external_snews [ subscribed subid=ieP1IdWTRkG4bHd6g3XfjmOtm5PMG3z2Wy36PswL]
(gracedb)[tanner.prestegard@tanner-test:~/lvalert_testing]$ lvalert_admin -s lvalert-test.cgca.uwm.edu --subscribe --node external_snews
Successfully subscribed to node external_snews
(gracedb)[tanner.prestegard@tanner-test:~/lvalert_testing]$ lvalert_admin -s lvalert-test.cgca.uwm.edu --subscriptions
Node: tanner_test1 [ subscribed subid=VO5QO2G7tSd85qN98yU32nacN4gvtKfTfJ10ukjE]
Node: external_snews [ subscribed subid=ieP1IdWTRkG4bHd6g3XfjmOtm5PMG3z2Wy36PswL]
Node: external_snews [ subscribed subid=98rC5asLPv7uLJUm4Oc5mJONzWeMrHBpyGiiyJ27]
Can't unsubscribe:
(gracedb)[tanner.prestegard@tanner-test:~/lvalert_testing]$ lvalert_admin -s lvalert-test.cgca.uwm.edu --unsubscribe --node external_snews
Traceback (most recent call last):
File "/home/tanner.prestegard/gracedb/bin/lvalert_admin", line 299, in <module>
s.loop(1)
File "/usr/lib/python2.7/dist-packages/pyxmpp/client.py", line 241, in loop
act=stream.loop_iter(timeout)
File "/usr/lib/python2.7/dist-packages/pyxmpp/streambase.py", line 619, in loop_iter
return self._loop_iter(timeout)
File "/usr/lib/python2.7/dist-packages/pyxmpp/streambase.py", line 640, in _loop_iter
self._process()
File "/usr/lib/python2.7/dist-packages/pyxmpp/streamtls.py", line 194, in _process
StreamBase._process(self)
File "/usr/lib/python2.7/dist-packages/pyxmpp/streambase.py", line 661, in _process
self._read()
File "/usr/lib/python2.7/dist-packages/pyxmpp/streamtls.py", line 187, in _read
self._read_tls()
File "/usr/lib/python2.7/dist-packages/pyxmpp/streamtls.py", line 181, in _read_tls
self._feed_reader(r)
File "/usr/lib/python2.7/dist-packages/pyxmpp/streambase.py", line 701, in _feed_reader
r=self._reader.feed(data)
File "/usr/lib/python2.7/dist-packages/pyxmpp/xmlextra.py", line 555, in feed
return self.reader.feed(s)
File "/usr/lib/python2.7/dist-packages/pyxmpp/xmlextra.py", line 59, in _stanza
self.stanza(doc,node)
File "/usr/lib/python2.7/dist-packages/pyxmpp/streambase.py", line 420, in stanza
self._process_node(node)
File "/usr/lib/python2.7/dist-packages/pyxmpp/stream.py", line 106, in _process_node
StreamBase._process_node(self,xmlnode)
File "/usr/lib/python2.7/dist-packages/pyxmpp/streambase.py", line 734, in _process_node
self.process_stanza(stanza)
File "/home/tanner.prestegard/gracedb/local/lib/python2.7/site-packages/ligo/lvalert/lvstanzaprocessor.py", line 281, in process_stanza
if self.process_iq(stanza):
File "/home/tanner.prestegard/gracedb/local/lib/python2.7/site-packages/ligo/lvalert/lvstanzaprocessor.py", line 127, in process_iq
response = err_handler(stanza)
File "/home/tanner.prestegard/gracedb/local/lib/python2.7/site-packages/ligo/lvalert/pubsub.py", line 235, in generic_error
raise FatalClientError("Failed to establish a session: "+msg)
pyxmpp.exceptions.FatalClientError: Failed to establish a session: Bad request
There is an option to "unsubscribe by subid", and although the code doesn't break, it doesn't remove the subscription:
(gracedb)[tanner.prestegard@tanner-test:~/lvalert_testing]$ lvalert_admin -s lvalert-test.cgca.uwm.edu --subscriptions
Node: tanner_test1 [ subscribed subid=VO5QO2G7tSd85qN98yU32nacN4gvtKfTfJ10ukjE]
Node: external_snews [ subscribed subid=ieP1IdWTRkG4bHd6g3XfjmOtm5PMG3z2Wy36PswL]
Node: external_snews [ subscribed subid=98rC5asLPv7uLJUm4Oc5mJONzWeMrHBpyGiiyJ27]
(gracedb)[tanner.prestegard@tanner-test:~/lvalert_testing]$ lvalert_admin -s lvalert-test.cgca.uwm.edu --unsubscribe-subid=98rC5asLPv7uLJUm4Oc5mJONzWeMrHBpyGiiyJ27
(gracedb)[tanner.prestegard@tanner-test:~/lvalert_testing]$ lvalert_admin -s lvalert-test.cgca.uwm.edu --subscriptions
Node: tanner_test1 [ subscribed subid=VO5QO2G7tSd85qN98yU32nacN4gvtKfTfJ10ukjE]
Node: external_snews [ subscribed subid=ieP1IdWTRkG4bHd6g3XfjmOtm5PMG3z2Wy36PswL]
Node: external_snews [ subscribed subid=98rC5asLPv7uLJUm4Oc5mJONzWeMrHBpyGiiyJ27]
This option doesn't seem to work at all, as I also tried it with a node I was only subscribed to once, and I was not unsubscribed.
>>>
And follow-up comments:
>>>
Updated by Tanner Prestegard
Note that multiple subscriptions does not seem to have any effect on your ability to receive messages which are pushed to the node.
Updated by Tanner Prestegard
Update: you actually can unsubscribe, by using --unsubscribe and --unsubscribe-subid together:
lvalert_admin -s lvalert-test.cgca.uwm.edu --unsubscribe --node external_snews --unsubscribe-subid=98rC5asLPv7uLJUm4Oc5mJONzWeMrHBpyGiiyJ27
But you should still not be able to subscribe multiple times in the first place...
>>>
**Update:**
I've seen this same behavior with the SleekXMPP library in some initial testing, the subscriber API call should be smart enough to recognize that a subscription already exists before it makes a new subscription.Alexander PaceAlexander Pacehttps://git.ligo.org/lscsoft/lvalert/-/issues/9Incorporate LVAlert Heartbeat into nagios monitors2019-02-02T00:16:08ZAlexander PaceIncorporate LVAlert Heartbeat into nagios monitorsThis is a migration of [Redmine issue #5416](https://bugs.ligo.org/redmine/issues/5416).
**Original Description (2017-04-19):**
lvalert-heartbeat was packaged and deployed. The request from Reed Essick is below:
"Hi Patrick;
We've fi...This is a migration of [Redmine issue #5416](https://bugs.ligo.org/redmine/issues/5416).
**Original Description (2017-04-19):**
lvalert-heartbeat was packaged and deployed. The request from Reed Essick is below:
"Hi Patrick;
We've finally got lvalert-heartbeat installed on the clusters and hooked up to most (but not all) mission-critical listeners. I've confirmed that I can query the production listeners and get responses back in a timely manner (less than 5 seconds per query). The next step is to begin integrating this with Nagios so that we can leverage that notification scheme to warn developers of unresponsive code.
Could you please tell me what you need from me to make this happen? I believe `lvalert_heartbeat-server` should return the correct format for Nagios to interpret the result. However, I'm afraid I'm at a bit of loss as to how to test and deploy this on monitor.ligo.org.
For your reference, the production lvalert-heartbeat repo lives here:
`https://git.ligo.org/lscsoft/lvalert-heartbeat`
and I've got a test environment of what Nagios might run set up on CIT here:
`/home/reed.essick/lvalert-dev/lvalert-heartbeat/test/`
In particular, I imagine the plugin would be run with something like:
`lvalert_heartbeat-server -V /home/reed.essick/lvalert-dev/lvalert-heartbeat/test/test_config.ini`
We may want to break that config up into separate queries for each node (ie, each section in test_config.ini would become a separate file), but that's really a matter of convenience.
Any thoughts or suggestions you have would be much appreciated.
cheers"
This ticket is to track progress on this issue.
**Update:**
The comments on this ticket are as follows:
```
Updated by Patrick Brockill over 1 year ago
Comment Edit
After discussion with Reed and Alex, if I understand correctly I think this is what is needed:
(*) This is a simple Nagios script (prints line of output to stderr, returns numerical value for OK, WARNING, CRITICAL or UNKNOWN);
(*) The service has to be able to connect to lvalert.cgca.uwm.edu;
(*) This script only needs to be run on one machine. The preference is that it not be run on lvalert.cgca.uwm.edu (apparently: it should be run "far away from all online follow-up processes", including "somewhere besides CIT, LHO and LLO"). So UWM is a good fit (just not on the lvalert machine);
(*) Reed has written the script to be called from Nagios and use few system resources. He has profiled it and it uses less than 40MB of memory. It does not use Condor, does not have to be run continuously, doesn't require user home directories to be mounted, is not CPU intensive, not network intensive, not disk intensive. It just requires access to a few netrc files and a config file.
(*) It would be nice if Reed could get access to the machine for irregular access.
#2 Updated by Patrick Brockill over 1 year ago
Comment Edit
This service is in the process of being tested by Reed and Alex on lowlatency0@uwm and can be found under the "emfollow" group on monitor. As lvalert_heartbeat-server may take some time to complete, it is currently being run with a timeout of 180 seconds.
#3 Updated by Patrick Brockill over 1 year ago
Comment Edit
We ran into more issues with timeouts. The heartbeat monitor usually completes in just less than 60 seconds, but now and then just over 60 seconds (e.g. 61 seconds). This requires three changes in Nagios:
(1) Change "service_check_timeout=60" to "service_check_timeout=75" in /etc/nagios3/nagios.cfg on dashboard;
(2) Change "command_timeout=60" to "command_timeout=75" in /etc/nagios/nrpe.cfg on lowlatency0;
(3) Use "/usr/lib/nagios/plugins/check_nrpe -t 75" in the service definitions.
```
I'm leaving this ticket open, as I would like the new status monitor to get tied into Nagios. This also has overlap with https://git.ligo.org/lscsoft/lvalert/issues/2.Alexander PaceAlexander Pacehttps://git.ligo.org/lscsoft/lvalert/-/issues/8Warning message from lvalert_send: No handlers could be found for logger "pyx...2018-08-24T16:13:04ZAlexander PaceWarning message from lvalert_send: No handlers could be found for logger "pyxmpp.resolver"This is a migration of [Redmine issue #4862](https://bugs.ligo.org/redmine/issues/4862).
**Original Description (2016-11-18):**
The warning message described by Branson (https://bugs.ligo.org/redmine/issues/1783) for lvalert_admin and ...This is a migration of [Redmine issue #4862](https://bugs.ligo.org/redmine/issues/4862).
**Original Description (2016-11-18):**
The warning message described by Branson (https://bugs.ligo.org/redmine/issues/1783) for lvalert_admin and lvalert_listen still occurs in lvalert_send, but only in Debian, not in Scientific Linux.
**Update:**
Leaving this ticket open, but I haven't seen it yet with the SleekXMPP prototypes. I'll update this after testing, packaging, and distribution.Alexander PaceAlexander Pacehttps://git.ligo.org/lscsoft/lvalert/-/issues/7incorporate lvalertTest into standard LVAlert and GraceDb packaging2018-08-24T16:15:13ZAlexander Paceincorporate lvalertTest into standard LVAlert and GraceDb packagingThis is a migration of [Redmine issue #4686](https://bugs.ligo.org/redmine/issues/4686).
**Original Description (2015-09-27):**
a testing suite for interactions with GraceDb via LVAlert has been developed in a personal github repo. The...This is a migration of [Redmine issue #4686](https://bugs.ligo.org/redmine/issues/4686).
**Original Description (2015-09-27):**
a testing suite for interactions with GraceDb via LVAlert has been developed in a personal github repo. The code should be moved into more standard packaging at some point.
The repo is available at https://github.com/reedessick/lvalertTest and provides several different interfaces and methods for testing follow-up analyses.
**Update:**
There's been no momentum for this since the ticket was opened. We can reopen it if there's interest, but in the interim, I'm going to close the ticket.Alexander PaceAlexander Pacehttps://git.ligo.org/lscsoft/lvalert/-/issues/6incorporate lvalertMP into standard LVAlert packaging2018-08-24T15:44:27ZAlexander Paceincorporate lvalertMP into standard LVAlert packagingThis is a migration of [Redmine issue #4685](https://bugs.ligo.org/redmine/issues/4685).
**Original Description (2015-09-27):**
An extension to LVAlert has been developed in a personal github repository for a while now. However, the co...This is a migration of [Redmine issue #4685](https://bugs.ligo.org/redmine/issues/4685).
**Original Description (2015-09-27):**
An extension to LVAlert has been developed in a personal github repository for a while now. However, the code is stable enough that it should be reviewed and migrated into the existing default packaging.
The lvalertMP repo can be found here: https://github.com/reedessick/lvalertMP
**Update:**
lvalertMP and event_supervisor have been (as far as I can tell) depreciated in favor of GWCelery. I think it's safe to close this ticket.Alexander PaceAlexander Pacehttps://git.ligo.org/lscsoft/lvalert/-/issues/5Set up off-site backups of lvalert database2018-08-24T15:39:37ZAlexander PaceSet up off-site backups of lvalert databaseThis is a migration of [Redmine issue #2209](https://bugs.ligo.org/redmine/issues/2209).
**Original Description (2015-06-12):**
Just use mysqldump and logrotate like on GraceDB. And then make sure the directory gets picked up by the ba...This is a migration of [Redmine issue #2209](https://bugs.ligo.org/redmine/issues/2209).
**Original Description (2015-06-12):**
Just use mysqldump and logrotate like on GraceDB. And then make sure the directory gets picked up by the backup machine.
**Update:**
I (@alexander\-pace) had made copies of the databases on lvalert and lvalert-test for the purposes of migrating to new Debian 9 servers. They're not backed up automatically, or at an offsite location. Currently they're backed up on gracedb-test, but this should change.Alexander PaceAlexander Pacehttps://git.ligo.org/lscsoft/lvalert/-/issues/4Support for python3 - move to sleekxmpp2018-08-28T19:06:40ZDuncan Macleodduncan.macleod@ligo.orgSupport for python3 - move to sleekxmppI'm wondering about python3 support for lvalert. I have no idea what `xmpp` is or how to use it, but it seems that `pyxmpp` isn't available for python3, which unfortunately suggests moving to a new framework. [`SleekXMPP`](https://pypi.o...I'm wondering about python3 support for lvalert. I have no idea what `xmpp` is or how to use it, but it seems that `pyxmpp` isn't available for python3, which unfortunately suggests moving to a new framework. [`SleekXMPP`](https://pypi.org/project/sleekxmpp/) seems to be the most popular alternative, have the maintainers of LVAlert considered a path for python3 support?Alexander PaceAlexander Pacehttps://git.ligo.org/lscsoft/lvalert/-/issues/3investigate possible dropped messages2018-08-24T16:13:03ZReed Essickreed.essick@ligo.orginvestigate possible dropped messagesAnecdotal evidence from O2 suggested that packets were occasionally dropped, meaning lvalert messages were not received by all listeners. It is not clear where the fault lies here, as I'll try to explain below. We should determine whethe...Anecdotal evidence from O2 suggested that packets were occasionally dropped, meaning lvalert messages were not received by all listeners. It is not clear where the fault lies here, as I'll try to explain below. We should determine whether dropped messages are actually an issue within the lvalert back-end. If that can be ruled out, then we know it was due to specific users' use patterns.
My recollection is that the majority of these issues occurred with ApprovalProcessor, which was built on top of lvalertMP. Because of the way lvalertMP processed alerts (fundamentally in series), time-consuming delegations to `parse_alert` could cause messages to back-up in the Python multiprocessing.Pipe connection used to pass lvalert messages to separate subprocesses. If the Pipe had a limited size, this could have cause messages to be dropped. Alternatively, and what I believe to be more likely, it would cause delays in processing alerts. This could be interpreted as "dropping alerts" because of incomplete logging within ApprovalProcessor, etc.
As an example, ApprovalProcessor often queried GraceDb or attempted to annotate events as part of the call to `parse_alert`. When it hit time-out errors, it would block for several minutes at a time and effectively stop digesting new alerts even though they were delivered to the listener without issue.
My belief that the fundamental issue lay with how ApprovalProcessor blocked when processing alerts (exacerbated by time-out errors from GraceDb) is also motivated by the fact that "regular" lvalert_listen instances did not seem to run into this issue, or at least not as much. Those listeners simply forked a subprocess, often orphaning it, and then processed the next alert immediately. Therefore, if the subprocess blocked for a long time due to a time-out from GraceDb, this did not affect the listener's ability to process newer alerts.
I also do not remember EventSupervisor suffering from dropped messages. EventSupervisor was built on top of lvalertMP like ApprovalProcessor, but it's calls to `parse_alert` were quick and did not block for extended periods of time. This meant that it may have been able to process alerts without the same delays as ApprovalProcessor, even though other parts of EventSupervisor may have blocked for long periods of time.
/cc @alexander\-pace @patrick\-bradyhttps://git.ligo.org/lscsoft/lvalert/-/issues/2determine whether we still need to monitor known "silent failure mechanism" w...2018-08-24T16:09:35ZReed Essickreed.essick@ligo.orgdetermine whether we still need to monitor known "silent failure mechanism" with lvalert-heartbeatDuring O1/2, we occasionally observed "silent failures" in which listeners would stop receiving alerts but the processes would not fail. This motivated the development of [lvalert-heartbeat](https://git.ligo.org/lscsoft/lvalert-heartbeat...During O1/2, we occasionally observed "silent failures" in which listeners would stop receiving alerts but the processes would not fail. This motivated the development of [lvalert-heartbeat](https://git.ligo.org/lscsoft/lvalert-heartbeat). Although that did not solve the underlying problem, it at least prevented it from being silent.
Because the underlying software libraries within lvalert have changed, this failure mechanism may no longer be relevant. However, we may want to stand up a few *extremely long-lived* listeners and monitor them to confirm that is the case.
/cc @alexander\-pace @patrick\-brady https://git.ligo.org/lscsoft/lvalert/-/issues/1Production server's certificate has the wrong hostname2018-08-24T15:27:32ZLeo P. SingerProduction server's certificate has the wrong hostnameThe production server `lvalert.cgca.uwm.edu` is providing me with a certificate for the wrong hostname, `lvalert.ligo.uwm.edu`. I think that it is only possible to connect to the production server at all because this client is not even c...The production server `lvalert.cgca.uwm.edu` is providing me with a certificate for the wrong hostname, `lvalert.ligo.uwm.edu`. I think that it is only possible to connect to the production server at all because this client is not even checking the certificates, so we are vulnerable to a MITM attack every time we use lvalert.
CC @alexander\-pace, @tanner.prestegard.
```
$ openssl x509 -text <<EOF
-----BEGIN CERTIFICATE-----
MIIDMTCCAhmgAwIBAgIIPh7wQIhBDF4wDQYJKoZIhvcNAQELBQAwHzEdMBsGA1UE
AwwUbHZhbGVydC5saWdvLnV3bS5lZHUwHhcNMTgwNjIxMTQ0MjAzWhcNMjMwNjIw
MTQ0MjAzWjAfMR0wGwYDVQQDDBRsdmFsZXJ0LmxpZ28udXdtLmVkdTCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAI00CXvfhFYeY7D6fU5PgIsYOsKmC6d3
BI7N32sxV0bUJwQ1q1kJDBG/HcBSVk+MJDW0Cx9SeunaDJrUlAtN0Hb4rwLrCGlP
QbxaaJSdVCTEzfQc1pIVDQckIqD2+rC81W49gWOUdfB/uwrvC1iwRApsvXf2PtgL
RTRHDpQFJdrqQbQCIaahucjrV0hyM5RkL0di9AJBVLZ/cgEiyccAr7RduHdX2QS/
8HFz2QDJH2OVdf8pkBcheeh2ZQ6Cz7bMPAtThrqCGdru+TpfWvBG+KgFuo/hW5Vt
VTUO96eHSUYIi5Vt0mg1x3Y3jSN/SLywcB1wgh9wFll4YWuA+zN7Ef0CAwEAAaNx
MG8wLQYDVR0RBCYwJKAiBggrBgEFBQcIBaAWDBRsdmFsZXJ0LmxpZ28udXdtLmVk
dTAdBgNVHQ4EFgQUXlMpPYgeQcZIEP7bT/urCuGhfB0wHwYDVR0jBBgwFoAUXlMp
PYgeQcZIEP7bT/urCuGhfB0wDQYJKoZIhvcNAQELBQADggEBAIbCC4n9yrwsZZcw
KsFUi/OGUCuSVxNrRJQsz3Oj0ibvnmGfrLxL9JUnhW+AN8OK2wK2IgFUUk7+wYdk
0jz4SyZHVmUwW6KvADXrDf5oV+0RXhikp2PchXlCIaUBKF5HynvvT1F9UfN4uNK2
dI35HQKnILs76kQgDET7d4Dg+/0EpZxwiRRqvTuLr8Uts9K6zy+lzx1qWl0FFjvq
iZ2fLubvVPy4W0PA3EJ8+6gyaI0Oik495lMIjCm35AuxQWmKEHZmlcPrEVrNAe+M
BwvgCsPDPoy8fymOEgm3UfWCHStBNUNX1xD0Cm6fXi1KkqaEiKFNMMomN6phwQB+
lo+2Zb8=
-----END CERTIFICATE-----
EOF
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 4476279239607389278 (0x3e1ef04088410c5e)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=lvalert.ligo.uwm.edu
Validity
Not Before: Jun 21 14:42:03 2018 GMT
Not After : Jun 20 14:42:03 2023 GMT
Subject: CN=lvalert.ligo.uwm.edu
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:8d:34:09:7b:df:84:56:1e:63:b0:fa:7d:4e:4f:
80:8b:18:3a:c2:a6:0b:a7:77:04:8e:cd:df:6b:31:
57:46:d4:27:04:35:ab:59:09:0c:11:bf:1d:c0:52:
56:4f:8c:24:35:b4:0b:1f:52:7a:e9:da:0c:9a:d4:
94:0b:4d:d0:76:f8:af:02:eb:08:69:4f:41:bc:5a:
68:94:9d:54:24:c4:cd:f4:1c:d6:92:15:0d:07:24:
22:a0:f6:fa:b0:bc:d5:6e:3d:81:63:94:75:f0:7f:
bb:0a:ef:0b:58:b0:44:0a:6c:bd:77:f6:3e:d8:0b:
45:34:47:0e:94:05:25:da:ea:41:b4:02:21:a6:a1:
b9:c8:eb:57:48:72:33:94:64:2f:47:62:f4:02:41:
54:b6:7f:72:01:22:c9:c7:00:af:b4:5d:b8:77:57:
d9:04:bf:f0:71:73:d9:00:c9:1f:63:95:75:ff:29:
90:17:21:79:e8:76:65:0e:82:cf:b6:cc:3c:0b:53:
86:ba:82:19:da:ee:f9:3a:5f:5a:f0:46:f8:a8:05:
ba:8f:e1:5b:95:6d:55:35:0e:f7:a7:87:49:46:08:
8b:95:6d:d2:68:35:c7:76:37:8d:23:7f:48:bc:b0:
70:1d:70:82:1f:70:16:59:78:61:6b:80:fb:33:7b:
11:fd
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Alternative Name:
othername:<unsupported>
X509v3 Subject Key Identifier:
5E:53:29:3D:88:1E:41:C6:48:10:FE:DB:4F:FB:AB:0A:E1:A1:7C:1D
X509v3 Authority Key Identifier:
keyid:5E:53:29:3D:88:1E:41:C6:48:10:FE:DB:4F:FB:AB:0A:E1:A1:7C:1D
Signature Algorithm: sha256WithRSAEncryption
86:c2:0b:89:fd:ca:bc:2c:65:97:30:2a:c1:54:8b:f3:86:50:
2b:92:57:13:6b:44:94:2c:cf:73:a3:d2:26:ef:9e:61:9f:ac:
bc:4b:f4:95:27:85:6f:80:37:c3:8a:db:02:b6:22:01:54:52:
4e:fe:c1:87:64:d2:3c:f8:4b:26:47:56:65:30:5b:a2:af:00:
35:eb:0d:fe:68:57:ed:11:5e:18:a4:a7:63:dc:85:79:42:21:
a5:01:28:5e:47:ca:7b:ef:4f:51:7d:51:f3:78:b8:d2:b6:74:
8d:f9:1d:02:a7:20:bb:3b:ea:44:20:0c:44:fb:77:80:e0:fb:
fd:04:a5:9c:70:89:14:6a:bd:3b:8b:af:c5:2d:b3:d2:ba:cf:
2f:a5:cf:1d:6a:5a:5d:05:16:3b:ea:89:9d:9f:2e:e6:ef:54:
fc:b8:5b:43:c0:dc:42:7c:fb:a8:32:68:8d:0e:8a:4e:3d:e6:
53:08:8c:29:b7:e4:0b:b1:41:69:8a:10:76:66:95:c3:eb:11:
5a:cd:01:ef:8c:07:0b:e0:0a:c3:c3:3e:8c:bc:7f:29:8e:12:
09:b7:51:f5:82:1d:2b:41:35:43:57:d7:10:f4:0a:6e:9f:5e:
2d:4a:92:a6:84:88:a1:4d:30:ca:26:37:aa:61:c1:00:7e:96:
8f:b6:65:bf
-----BEGIN CERTIFICATE-----
MIIDMTCCAhmgAwIBAgIIPh7wQIhBDF4wDQYJKoZIhvcNAQELBQAwHzEdMBsGA1UE
AwwUbHZhbGVydC5saWdvLnV3bS5lZHUwHhcNMTgwNjIxMTQ0MjAzWhcNMjMwNjIw
MTQ0MjAzWjAfMR0wGwYDVQQDDBRsdmFsZXJ0LmxpZ28udXdtLmVkdTCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAI00CXvfhFYeY7D6fU5PgIsYOsKmC6d3
BI7N32sxV0bUJwQ1q1kJDBG/HcBSVk+MJDW0Cx9SeunaDJrUlAtN0Hb4rwLrCGlP
QbxaaJSdVCTEzfQc1pIVDQckIqD2+rC81W49gWOUdfB/uwrvC1iwRApsvXf2PtgL
RTRHDpQFJdrqQbQCIaahucjrV0hyM5RkL0di9AJBVLZ/cgEiyccAr7RduHdX2QS/
8HFz2QDJH2OVdf8pkBcheeh2ZQ6Cz7bMPAtThrqCGdru+TpfWvBG+KgFuo/hW5Vt
VTUO96eHSUYIi5Vt0mg1x3Y3jSN/SLywcB1wgh9wFll4YWuA+zN7Ef0CAwEAAaNx
MG8wLQYDVR0RBCYwJKAiBggrBgEFBQcIBaAWDBRsdmFsZXJ0LmxpZ28udXdtLmVk
dTAdBgNVHQ4EFgQUXlMpPYgeQcZIEP7bT/urCuGhfB0wHwYDVR0jBBgwFoAUXlMp
PYgeQcZIEP7bT/urCuGhfB0wDQYJKoZIhvcNAQELBQADggEBAIbCC4n9yrwsZZcw
KsFUi/OGUCuSVxNrRJQsz3Oj0ibvnmGfrLxL9JUnhW+AN8OK2wK2IgFUUk7+wYdk
0jz4SyZHVmUwW6KvADXrDf5oV+0RXhikp2PchXlCIaUBKF5HynvvT1F9UfN4uNK2
dI35HQKnILs76kQgDET7d4Dg+/0EpZxwiRRqvTuLr8Uts9K6zy+lzx1qWl0FFjvq
iZ2fLubvVPy4W0PA3EJ8+6gyaI0Oik495lMIjCm35AuxQWmKEHZmlcPrEVrNAe+M
BwvgCsPDPoy8fymOEgm3UfWCHStBNUNX1xD0Cm6fXi1KkqaEiKFNMMomN6phwQB+
lo+2Zb8=
-----END CERTIFICATE-----
```