Kafka alert generator

added 8 commits

459dc937 - Initial commit for Kafka producer bootstep
cb0b0269 - Switch from python-kafka to confluent-python-kafka
fc2c0863 - Add fastavro to dependencies
e862c94f - avro_alerts.py: First commit
fefebfb8 - avro_alerts.py: Replace tabs with 4 spaces
3a600777 - avro.py: Renamed from avro_alerts.py
9508798e - avro.py:
ce4b7b78 - avro.py: First attempt at publishing avro packet to kafka topic

Compare with previous version

changed the description

added 11 commits

393f0708 - 1 commit from branch emfollow:main
9d29270e - Initial commit for Kafka producer bootstep
b7b89811 - Switch from python-kafka to confluent-python-kafka
033df7a8 - Use hop client for Kafka implementation
e13b34d9 - Add fastavro to dependencies
5253231e - avro_alerts.py: First commit
03f0702a - avro_alerts.py: Replace tabs with 4 spaces
78d857bb - avro.py: Renamed from avro_alerts.py
53da7728 - avro.py:
eafa5cf0 - avro.py: First attempt at publishing avro packet to kafka topic
608489dc - avro.py: Switch to using kombu to send avro packet to consumer set up by kafka bootstep

Compare with previous version

added 1 commit

93dcdd05 - poetry.lock: Update to resolve conflict with pyproject.toml

Compare with previous version

added 3 commits

e0887e64 - poetry.lock and pyproject.toml: Add hop-client dependency for kafka bootstep
b6f0048b - kafka/bootsteps.py: Comment out confluence_kafka import
8365dd21 - avro.py: Various fixes to make pytest run on local machine, wittle down a few

Compare with previous version

added 19 commits

8365dd21...a629ff07 - 5 commits from branch emfollow:main
b6a82c22 - Initial commit for Kafka producer bootstep
8d0d3d8e - Switch from python-kafka to confluent-python-kafka
e621e9f7 - Use hop client for Kafka implementation
cf4265a9 - Add fastavro to dependencies
d85ee1e7 - avro_alerts.py: First commit
1c9c119d - avro_alerts.py: Replace tabs with 4 spaces
29f178c1 - avro.py: Renamed from avro_alerts.py
20956540 - avro.py:
041d4259 - avro.py: First attempt at publishing avro packet to kafka topic
6891a0d8 - avro.py: Switch to using kombu to send avro packet to consumer set up by kafka bootstep
dd47e533 - poetry.lock: Update to resolve conflict with pyproject.toml
8d7d5f5b - poetry.lock and pyproject.toml: Add hop-client dependency for kafka bootstep
50c22b09 - kafka/bootsteps.py: Comment out confluence_kafka import
2eeeb75f - avro.py: Various fixes to make pytest run on local machine, wittle down a few

Compare with previous version

added 13 commits

697f9e37 - Initial commit for Kafka producer bootstep
c7c0aa0f - Switch from python-kafka to confluent-python-kafka
5ee79d32 - Use hop client for Kafka implementation
1f298def - Update kafka bootstep to be more modular
7d1b2106 - Add kafka to submit file
9d3f9fc5 - Fix bug with **kwargs and kafka kwarg
0c79054e - Give kafka its own worker and add its stop
41590535 - Add call to super()'s init in KafkaBootStep
21920ac8 - Remove super calls in KafkaBootStep
87bb31e5 - Change kafka queue exchange name
50a642b0 - Changes required to get kafka bootstep to work. Changes revolve around removing
b4356296 - Add fastavro as dependency
7a4fcc3c - Commit avro task, add placeholder function call to orchestrator

Compare with previous version

Just a note that I made an oopsie when rebasing, so I just burned it all to the ground and added commits to a branch starting at the kafka bootstep to get us back to where we were. I assumed the commit history didn't matter since we'll be squashing the commits when merging anyway.

added 22 commits

7a4fcc3c...c1a34bd1 - 6 commits from branch emfollow:main
5cb3c02f - Initial commit for Kafka producer bootstep
75de6699 - Switch from python-kafka to confluent-python-kafka
f7b34729 - Use hop client for Kafka implementation
a39b5ba5 - Update kafka bootstep to be more modular
b94f9807 - Add kafka to submit file
ac9abc77 - Fix bug with **kwargs and kafka kwarg
014fdcac - Give kafka its own worker and add its stop
9da54e76 - Add call to super()'s init in KafkaBootStep
fc1d7c65 - Remove super calls in KafkaBootStep
59337ca7 - Change kafka queue exchange name
0293cd9d - Changes required to get kafka bootstep to work. Changes revolve around removing
09a888f2 - Regenerate poetry.lock to fix problem introduced when rebasing against main
bb080b36 - Update kafka bootstep to be more modular
c2543970 - Add fastavro as dependency
6a1d2cfe - Commit avro task, add placeholder function call to orchestrator
1f94f450 - Update poetry.lock to fix problem introduced when rebasing

Compare with previous version

added 1 commit

a948006c - Fix mistake in kafka bootstep made when rebasing against kafka MR

Compare with previous version

added 2 commits

8d6af3b5 - Add two deps
e0414a1f - Fix bugs discovered when testing avro packet generation. Add avro packet

Compare with previous version

added 31 commits

e0414a1f...df35c1f1 - 12 commits from branch emfollow:main
7e040d17 - Initial commit for Kafka producer bootstep
3cd542c4 - Switch from python-kafka to confluent-python-kafka
1ee67b2f - Use hop client for Kafka implementation
216b6736 - Update kafka bootstep to be more modular
034e1575 - Add kafka to submit file
5ae4ac8b - Fix bug with **kwargs and kafka kwarg
40b9a040 - Give kafka its own worker and add its stop
6db0325a - Add call to super()'s init in KafkaBootStep
be0be84d - Remove super calls in KafkaBootStep
468df39a - Change kafka queue exchange name
71e17b98 - Changes required to get kafka bootstep to work. Changes revolve around removing
085a5956 - Regenerate poetry.lock to fix problem introduced when rebasing against main
eef387df - Update kafka bootstep to be more modular
da45c147 - Add fastavro as dependency
627424b8 - Commit avro task, add placeholder function call to orchestrator
44dc3ee9 - Update poetry.lock to fix problem introduced when rebasing
6b899a06 - Fix mistake in kafka bootstep made when rebasing against kafka MR
b70730fb - Add two deps
ac212f9e - Fix bugs discovered when testing avro packet generation. Add avro packet

Compare with previous version

marked the checklist item Test sending avro packet to kafka producer as completed

I've run up to ac212f9e on emfollow-test and have successfully produced an avro packet that I received through a kafka listener. I've attached three examples from one mock event.

MS220513s_01_PRELIMINARY.avro MS220513s_02_PRELIMINARY.avro MS220513s_03_RETRACTION.avro

For comparison, the voevents are

MS220513s-1-Preliminary.xml MS220513s-2-Preliminary.xml MS220513s-3-Retraction.xml

resolved all threads

added 1 commit

4f55b814 - Check for RAVEN_ALERT label in external_coinc logic when generating avro packet

Compare with previous version

added 1 commit

7ccd29a9 - Fix style issues

Compare with previous version

added 22 commits

077adc14 - Initial commit for Kafka producer bootstep
2533a8e5 - Switch from python-kafka to confluent-python-kafka
358a4165 - Use hop client for Kafka implementation
811ac9bb - Update kafka bootstep to be more modular
e4336744 - Add kafka to submit file
72891da9 - Fix bug with **kwargs and kafka kwarg
14ae7027 - Give kafka its own worker and add its stop
6f8b8bb0 - Add call to super()'s init in KafkaBootStep
73444ae5 - Remove super calls in KafkaBootStep
b7c0cdd6 - Change kafka queue exchange name
8387c6c1 - Changes required to get kafka bootstep to work. Changes revolve around removing
ae83ac1e - Regenerate poetry.lock to fix problem introduced when rebasing against main
b3bf64bc - Fix style issues, remove unused import
a82f688b - Update kafka bootstep to be more modular
69a575e0 - Add fastavro as dependency
222077df - Commit avro task, add placeholder function call to orchestrator
fa8d61c8 - Update poetry.lock to fix problem introduced when rebasing
2a47986f - Fix mistake in kafka bootstep made when rebasing against kafka MR
052b5ce4 - Add two deps
660c778a - Fix bugs discovered when testing avro packet generation. Add avro packet
b2e14f5e - Check for RAVEN_ALERT label in external_coinc logic when generating avro packet
f1f73293 - Fix style issues

Compare with previous version

added 1 commit

2263e1eb - Fix style issues

Compare with previous version

marked the checklist item Check that avro packet can be sent to kafka producer as completed

marked the checklist item Check that avro packet can be sent to kafka producer as incomplete

added 1 commit

d220e9be - Fix style issues

Compare with previous version

@leo-singer I was working on the avro validator and realized that we will need to listen for publicly released avro packets if we want to validate them the same way that we validate GCNs. I assume that is something that we want to do, but it seems like we can't implement a validate function until we've settled details about where avro packets are going to be sent. Should we punt this particular test down the road and drop it as a requirement for this MR?

Yes. And Avro alerts might not need validation at all, since no one is reformatting them.

changed the description

added 2 commits

93852b82 - Rename avro.py to gcn_translate.py
c4df6c7d - Add send_json function and send_avro_and_json function to gcn_translate. Update

Compare with previous version

I've changed this MR a bit to generate both avro and json packets. The task formerly named avro.py is now named gcn_translate.py, since its primary function is deconstruct a VOEvent and use that to generate a different type of alert, now either avro or json. There are now three functions that can be called, send_avro, send_json, and send_avro_and_json. Long term, I'd like to advocate using the last function so we send both avro and json packets and let our consumers decide what they prefer between those two and GCNs. In the short term, we could simply use the send_json function until the hop client (or anyone really) officially supports sending avro packets over kafka.

added 7 commits

d220e9be...4dafb433 - 5 commits from branch emfollow:main
b89c2e40 - Add kafka bootstep
d0a41fbe - Generate avro alerts in addition to GCNs

Compare with previous version

added 1 commit

5e6fe651 - Add avro tests to orchestrator unittest

Compare with previous version

changed the description

changed title from WIP: Avro alert generator to Avro alert generator

marked the checklist item Write unit tests to... as completed

High-level comment: populating the Kafka record from the VOEvent means that the Kafka alerts will be strictly higher latency than the VOEvents. It will also make it harder to eventually retire the VOEvents. It would be preferable to populate both the VOEvent and the Kafka alert from common data.