- p_astro reads mean values from emfollow/data to compute values

- mass gap category included

- unittests changed/added

This is a major rewrite of the VOEvent subsystem to improve
robustness and monitoring capabilites.

* We used to start the Comet broker as a subprocess inside an
  eternal task. Instead, start it as a thread in a custom *boot
  step*, which is a sort of Celery plugin for adding extra
  functionality to a worker. This avoids unnecessarily tying up
  a thread from the worker thread pool and makes it easier to
  tell the broker when to start and stop.
* Now that we are not relying on the celery beat periodic
  scheduler for keeping the VOEvent broker running, we can
  explore other forms of fault tolerance such as using the
  celery gossip feature to elect a new VOEvent broker by
  consensus among multiple Celery workers at physically separate
  sites.
* Do a better job of meshing together Celery and Twisted in terms
  of logging and signal handling.
* Because the Comet broker is running inside the same process as
  a worker, the `send` task can directly command the broker to
  send a VOEvent, and we don't have to maintain a dedicated local
  TCP port for this purpose. This makes it harder to contrive a
  scenario where a VOEvent could be dropped during the handoff
  from a Celery task to the broker.
* The new `send` task will fail and automatically retry later
  if there are no peers connected to the broker.
* Add monitoring functionality so that the addresses of all peers
  connected to the VOevent broker will appear in Flower.

Compare this ASCII art diagram of the old sending architecture:

    +--|worker 1|---------------------------------+
    |                                             |
    |   +--|send task|------------------------+   |
    |   |                                     |   |
    |   |   +--|comet process|------------+   |   |
    |   |   |                             |   |   |
    |   |   |   +--|sender|-------+       |   |   |
    |   |   |   |                 |       |   |   |
    |   |   |   |                 ----+   |   |   |
    |   |   |   |                 |   |   |   |   |
    |   |   |   +-----------------+   |   |   |   |
    |   |   |                         |   |   |   |
    |   |   +------------------------|||--+   |   |
    |   |                             |       |   |
    |   +----------------------------|||------+   |
    |                                 |           |
    +--------------------------------|||----------+
                                      |
                                      |
                                      |TCP
                                      |port
                                      |
                                      |
    +--|worker 2|--------------------|||----------+
    |                                 |           |
    |   +--|broker task|-------------|||------+   |
    |   |                             |       |   |
    |   |   +--|comet process|-------|||--+   |   |
    |   |   |                         |   |   |   |
    |   |   |   +--|receiver|-----+   |   |   |   |
    |   |   |   |                 |   |   |   |   |
    |   |   |   |                 <---+   |   |   |
    |   |   |   |                 |       |   |   |
    |   |   |   |                 ----+   |   |   |
    |   |   |   |                 |   |   |   |   |
    |   |   |   +-----------------+   |   |   |   |
    |   |   |                         |   |   |   |
    |   |   |   +--|broadcaster|--+   |   |   |   |
    |   |   |   |                 |   |   |   |   |
    |   |   |   |                 <---+   |   |   |
    |   |   |   |                 |       |   |   |
    |   |   |   |                 ----+   |   |   |
    |   |   |   |                 |   |   |   |   |
    |   |   |   +-----------------+   |   |   |   |
    |   |   |                         |   |   |   |
    |   |   +------------------------|||--+   |   |
    |   |                             |       |   |
    |   +----------------------------|||------+   |
    |                                 |           |
    +--------------------------------|||----------+
                                      |
                                      |
                                      |TCP
                                      |port
                                      |
                                      |
                                      +----> to GCN

to this diagram of the new architecture implemented by this patch:

    +--|worker 1|-------------------------+
    |                                     |
    |   +--|send task|----------------+   |
    |   |                             |   |
    |   |                             |   |
    |   |                             |   |
    |   +-------------------------|---+   |
    |                             |       |
    |                             |       |
    |   +--|twisted thread|------|||--+   |
    |   |                         |   |   |
    |   |   +--|receiver|-----+   |   |   |
    |   |   |                 |   |   |   |
    |   |   |                 <---+   |   |
    |   |   |                 |       |   |
    |   |   |                 ----+   |   |
    |   |   |                 |   |   |   |
    |   |   +-----------------+   |   |   |
    |   |                         |   |   |
    |   |   +--|broadcaster|--+   |   |   |
    |   |   |                 |   |   |   |
    |   |   |                 <---+   |   |
    |   |   |                 |       |   |
    |   |   |                 ----+   |   |
    |   |   |                 |   |   |   |
    |   |   +-----------------+   |   |   |
    |   |                         |   |   |
    |   +------------------------|||--+   |
    |                             |       |
    +----------------------------|||------+
                                  |
                                  |
                                  |TCP
                                  |port
                                  |
                                  |
                                  +----> to GCN

The handler log messages looked like this:

    [2019-02-22 08:27:42,630: INFO/ForkPoolWorker-32] ???[???]: calling handlers

This change removes the `???[???]`.

This will have the effect of requiring lalsuite <= 6.52 and
lscsoft-glue <= 1.6.0 in order to work around breaking changes in
LIGO-LW file handling.

tasks/lalinference.py and templates/online_pe.jinja2: Specify gps-start-time and gps-end-time in online_pe.jinja2

Although gps-start-time, gps-end-time, psd-start-time and psd-length are redundant, they are necessary for
lalinference_pipe to work properly.

Soichiro Morisaki authored 6 years ago and Leo P. Singer committed 6 years ago

task/lalinference.py and templates/online_pe.jinja2: Change variable names in onlin_pe.jinja2 so that the template is loaded

The recent redis-py update from 3.1.0 to 3.2.0 broke the following
unit test:

    _________________________________ test_nagios __________________________________

    capsys = <_pytest.capture.CaptureFixture object at 0x7f071357f400>
    monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f07133e9668>
    socket_enabled = None
    starter = <class 'gwcelery.tests.process.starter.<locals>.Starter'>
    tmpdir = local('/tmp/pytest-of-root/pytest-0/test_nagios0')

        def test_nagios(capsys, monkeypatch, socket_enabled,
                        starter, tmpdir):
            mock_lvalert_client = Mock()
            monkeypatch.setattr('sleek_lvalert.LVAlertClient', mock_lvalert_client)

            # no broker

            unix_socket = str(tmpdir / 'sock')
            monkeypatch.setenv('CELERY_BROKER_URL', 'redis+socket://' + unix_socket)

            with pytest.raises(SystemExit) as excinfo:
                app.start(['gwcelery', 'nagios'])
            assert excinfo.value.code == nagios.NagiosPluginStatus.CRITICAL
            out, err = capsys.readouterr()
            assert 'CRITICAL: No connection to broker' in out

            # broker, no worker

            redis_server = find_executable('redis-server')
            if redis_server is None:
                pytest.skip('redis-server is not installed')
            starter.exec_process(
                [redis_server, '--port', '0', '--unixsocket', unix_socket,
                 '--unixsocketperm', '700'], timeout=10,
                magic_words=b'The server is now ready to accept connections')

            with pytest.raises(SystemExit) as excinfo:
                app.start(['gwcelery', 'nagios'])
            assert excinfo.value.code == nagios.NagiosPluginStatus.CRITICAL
            out, err = capsys.readouterr()
            assert 'CRITICAL: Not all expected queues are active' in out

            # worker, no tasks

            starter.python_process(
                args=(['gwcelery', 'worker', '-l', 'info',
                       '-Q', 'celery,exttrig,openmp,superevent'],),
                target=app.start, timeout=10, magic_words=b'ready.')

            with pytest.raises(SystemExit) as excinfo:
                app.start(['gwcelery', 'nagios'])
            assert excinfo.value.code == nagios.NagiosPluginStatus.CRITICAL
            out, err = capsys.readouterr()
    >       assert 'CRITICAL: Not all expected tasks are active' in out
    E       AssertionError: assert 'CRITICAL: Not all expected tasks are active' in 'CRITICAL: Not all expected queues are active\nAssertionError: Missing queues: exttrig, openmp, celery, superevent\n\n'

    gwcelery/tests/test_tools_nagios.py:66: AssertionError

There is a patch for Redis broker support on master.