determine whether we still need to monitor known "silent failure mechanism" with lvalert-heartbeat
During O1/2, we occasionally observed "silent failures" in which listeners would stop receiving alerts but the processes would not fail. This motivated the development of lvalert-heartbeat. Although that did not solve the underlying problem, it at least prevented it from being silent.
Because the underlying software libraries within lvalert have changed, this failure mechanism may no longer be relevant. However, we may want to stand up a few extremely long-lived listeners and monitor them to confirm that is the case.
/cc @alexander-pace @patrick-brady
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information