Kafka alert generator

Addresses #388 (closed), and depends on !851 (merged).

Feb 11 edit: First attempt at adding the code to publish to a kafka topic is done, needs testing. Alerts currently set to publish to StandinKafkaTopic

Aug 10 edit: Ready to be reviewed. Need new release of hop-client for a PR needed for GCN support (merged) and need to update URL of avro schema files once userguide!124 (merged) is merged.

Sep 21 edit: Peeled a lot of these changes off into other MRs, specifically !914 (merged), !915 (merged), and !917 (merged). This MR now primarily deals with the work needed to plug the kafka alerts into the orchestrator.

To-Do list:

• Test sending avro packet to kafka producer
• Decide where to store avro schema files for deployed instances to read
• Write unit tests to...
  • Validate that avro packet constructed by voevent is correct
  • Check that avro packet is sent to kafka producer

added 8 commits

• 459dc937 - Initial commit for Kafka producer bootstep
• cb0b0269 - Switch from python-kafka to confluent-python-kafka
• fc2c0863 - Add fastavro to dependencies
• e862c94f - avro_alerts.py: First commit
• fefebfb8 - avro_alerts.py: Replace tabs with 4 spaces
• 3a600777 - avro.py: Renamed from avro_alerts.py
• 9508798e - avro.py:
• ce4b7b78 - avro.py: First attempt at publishing avro packet to kafka topic

Compare with previous version

changed the description

added 11 commits

• 393f0708 - 1 commit from branch emfollow:main
• 9d29270e - Initial commit for Kafka producer bootstep
• b7b89811 - Switch from python-kafka to confluent-python-kafka
• 033df7a8 - Use hop client for Kafka implementation
• e13b34d9 - Add fastavro to dependencies
• 5253231e - avro_alerts.py: First commit
• 03f0702a - avro_alerts.py: Replace tabs with 4 spaces
• 78d857bb - avro.py: Renamed from avro_alerts.py
• 53da7728 - avro.py:
• eafa5cf0 - avro.py: First attempt at publishing avro packet to kafka topic
• 608489dc - avro.py: Switch to using kombu to send avro packet to consumer set up by kafka bootstep

Compare with previous version

added 1 commit

• 93dcdd05 - poetry.lock: Update to resolve conflict with pyproject.toml

Compare with previous version

added 3 commits

• e0887e64 - poetry.lock and pyproject.toml: Add hop-client dependency for kafka bootstep
• b6f0048b - kafka/bootsteps.py: Comment out confluence_kafka import
• 8365dd21 - avro.py: Various fixes to make pytest run on local machine, wittle down a few

Compare with previous version

added 19 commits

• 8365dd21...a629ff07 - 5 commits from branch emfollow:main
• b6a82c22 - Initial commit for Kafka producer bootstep
• 8d0d3d8e - Switch from python-kafka to confluent-python-kafka
• e621e9f7 - Use hop client for Kafka implementation
• cf4265a9 - Add fastavro to dependencies
• d85ee1e7 - avro_alerts.py: First commit
• 1c9c119d - avro_alerts.py: Replace tabs with 4 spaces
• 29f178c1 - avro.py: Renamed from avro_alerts.py
• 20956540 - avro.py:
• 041d4259 - avro.py: First attempt at publishing avro packet to kafka topic
• 6891a0d8 - avro.py: Switch to using kombu to send avro packet to consumer set up by kafka bootstep
• dd47e533 - poetry.lock: Update to resolve conflict with pyproject.toml
• 8d7d5f5b - poetry.lock and pyproject.toml: Add hop-client dependency for kafka bootstep
• 50c22b09 - kafka/bootsteps.py: Comment out confluence_kafka import
• 2eeeb75f - avro.py: Various fixes to make pytest run on local machine, wittle down a few

Compare with previous version

added 13 commits

• 697f9e37 - Initial commit for Kafka producer bootstep
• c7c0aa0f - Switch from python-kafka to confluent-python-kafka
• 5ee79d32 - Use hop client for Kafka implementation
• 1f298def - Update kafka bootstep to be more modular
• 7d1b2106 - Add kafka to submit file
• 9d3f9fc5 - Fix bug with **kwargs and kafka kwarg
• 0c79054e - Give kafka its own worker and add its stop
• 41590535 - Add call to super()'s init in KafkaBootStep
• 21920ac8 - Remove super calls in KafkaBootStep
• 87bb31e5 - Change kafka queue exchange name
• 50a642b0 - Changes required to get kafka bootstep to work. Changes revolve around removing
• b4356296 - Add fastavro as dependency
• 7a4fcc3c - Commit avro task, add placeholder function call to orchestrator

Compare with previous version

Just a note that I made an oopsie when rebasing, so I just burned it all to the ground and added commits to a branch starting at the kafka bootstep to get us back to where we were. I assumed the commit history didn't matter since we'll be squashing the commits when merging anyway.

added 22 commits

• 7a4fcc3c...c1a34bd1 - 6 commits from branch emfollow:main
• 5cb3c02f - Initial commit for Kafka producer bootstep
• 75de6699 - Switch from python-kafka to confluent-python-kafka
• f7b34729 - Use hop client for Kafka implementation
• a39b5ba5 - Update kafka bootstep to be more modular
• b94f9807 - Add kafka to submit file
• ac9abc77 - Fix bug with **kwargs and kafka kwarg
• 014fdcac - Give kafka its own worker and add its stop
• 9da54e76 - Add call to super()'s init in KafkaBootStep
• fc1d7c65 - Remove super calls in KafkaBootStep
• 59337ca7 - Change kafka queue exchange name
• 0293cd9d - Changes required to get kafka bootstep to work. Changes revolve around removing
• 09a888f2 - Regenerate poetry.lock to fix problem introduced when rebasing against main
• bb080b36 - Update kafka bootstep to be more modular
• c2543970 - Add fastavro as dependency
• 6a1d2cfe - Commit avro task, add placeholder function call to orchestrator
• 1f94f450 - Update poetry.lock to fix problem introduced when rebasing

Compare with previous version

added 1 commit

• a948006c - Fix mistake in kafka bootstep made when rebasing against kafka MR

Compare with previous version

added 2 commits

• 8d6af3b5 - Add two deps
• e0414a1f - Fix bugs discovered when testing avro packet generation. Add avro packet

Compare with previous version

added 31 commits

• e0414a1f...df35c1f1 - 12 commits from branch emfollow:main
• 7e040d17 - Initial commit for Kafka producer bootstep
• 3cd542c4 - Switch from python-kafka to confluent-python-kafka
• 1ee67b2f - Use hop client for Kafka implementation
• 216b6736 - Update kafka bootstep to be more modular
• 034e1575 - Add kafka to submit file
• 5ae4ac8b - Fix bug with **kwargs and kafka kwarg
• 40b9a040 - Give kafka its own worker and add its stop
• 6db0325a - Add call to super()'s init in KafkaBootStep
• be0be84d - Remove super calls in KafkaBootStep
• 468df39a - Change kafka queue exchange name
• 71e17b98 - Changes required to get kafka bootstep to work. Changes revolve around removing
• 085a5956 - Regenerate poetry.lock to fix problem introduced when rebasing against main
• eef387df - Update kafka bootstep to be more modular
• da45c147 - Add fastavro as dependency
• 627424b8 - Commit avro task, add placeholder function call to orchestrator
• 44dc3ee9 - Update poetry.lock to fix problem introduced when rebasing
• 6b899a06 - Fix mistake in kafka bootstep made when rebasing against kafka MR
• b70730fb - Add two deps
• ac212f9e - Fix bugs discovered when testing avro packet generation. Add avro packet

Compare with previous version

marked the checklist item Test sending avro packet to kafka producer as completed

I've run up to ac212f9e on emfollow-test and have successfully produced an avro packet that I received through a kafka listener. I've attached three examples from one mock event.

MS220513s_01_PRELIMINARY.avro MS220513s_02_PRELIMINARY.avro MS220513s_03_RETRACTION.avro

For comparison, the voevents are

MS220513s-1-Preliminary.xml MS220513s-2-Preliminary.xml MS220513s-3-Retraction.xml