Changes

Page history
Update home authored by Jameson Rollins's avatar Jameson Rollins
Hide whitespace changes
Inline Side-by-side
home.md
View page @ db2deb02
# Next Generation Data Delivery
This is the home of LIGO's Next Generation Data Delivery (NGDD) project, the aim of which is to develop a low-latency data distribution and stream processing platform to meet the needs of real-time searches for gravitational waves.
![concept_full](uploads/eecb16d304cc6f02c521271a359bc505/concept_full.png)
## talks and presentations
- [The next generation of low-latency data distribution and stream processing for LIGO](https://dcc.ligo.org/LIGO-G2200571)
- [NGDD architecture overview](https://docs.google.com/presentation/d/1ItEeNtkLCAzxBYc-dLtPY35C1EBXQwQBV2LmgEPn2LI/edit?usp=sharing)
- [NGDD serialization overview](https://docs.google.com/presentation/d/1OGFqV7TBqMlqXr9I_JoOokDyZGW6KcC_REuoMjg22Eg/edit?usp=sharing)
## deployment
- [Current deployments](Deployments)
## development
- [[todo]]
- [Live monitor of LHO infrastructure](https://grafana.ligo-wa.caltech.edu/d/HnFEtyB7z/kafka-metrics?orgId=1&from=now-15m&to=now&refresh=5s)
- [Requirements document (draft)](https://ngdd.docs.ligo.org/requirements-doc/ngdd-requirements.pdf)
- [Client Protocol Schema](Client-Protocol-Schema)
- [Publisher Protocol Schema](Publisher-Protocol-Schema)
- [NGDD dev workshop, Jan. 2022](https://dcc.ligo.org/cgi-bin/private/DocDB/DisplayMeeting?conferenceid=1128)
- [notes from the workshop](https://docs.google.com/document/d/1bUmt_wzKaU5cGEfGHEfJnSqlUb7348kcrOQYMydZPdk/edit#heading=h.qf7tvz41m6al)
- [Tech investigation notes](https://docs.google.com/document/d/1YyHJSrM5gTfuoJVp6U9gOntdmonaEqBqF5Pf5VaJfkI/edit#heading=h.bw1cizptefma):
- [low-latency data distribution](https://wiki.ligo.org/Computing/DASWG/LowLatencyDataDistribution)
- [distributed key/value stores](https://wiki.ligo.org/Computing/DASWG/KeyValueStores)
- [Kafka client](https://dcc.ligo.org/DocDB/0180/G2200107/001/Kafka%20Client%20Tests%20-%20NGDD%20Jan%202022%20F2F.pdf)
- [Materialize](https://docs.google.com/presentation/d/1F0l7TKwxD4BHX4JHTXBIywApURG-OA7VRmUjlbtGi_U/edit?usp=sharing)
- [Materialize/ClickHouse](https://docs.google.com/presentation/d/1D0Kyg694PlKnOgHIK2lJJvAHR4aWju9xGolUWwOHG7U/edit?usp=sharing)
- [profiling](profiling)
- [LIGO NDS server architecture](https://git.ligo.org/groups/nds/-/wikis/LHO-Distributed-NDS-Install)
- [Nomad Review](Nomad-Review)
- [Brockill's thoughts on replay infrastructure](https://wiki.ligo.org/Computing/DASWG/ReplayInfrastructure)
- [Authn/z](Auth)
## applications
- [online/low-latency application table](https://docs.google.com/spreadsheets/d/15JzZ1KEHscwXhny8L0lxZIYXkiv6LlkCgSLi_Iuruyw/edit#gid=1412826968)
- [open public alerts application table (searches)](https://docs.google.com/spreadsheets/d/1kLO84LNSj6B_PTlB35ujh-uYxXLdFc7L4P2lxY1-KXc/edit#gid=1943285589)
## technologies
- [gRPC](https://grpc.io/)
- [Arrow](https://arrow.apache.org/)
- [Arrow Flight](https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/)
- [Kafka](https://kafka.apache.org/)
- [influxDB](https://www.influxdata.com/)
- [gstreamer RTP/RTSP](https://gstreamer.freedesktop.org/documentation/additional/rtp.html?gi-language=c)
- [consul](https://www.consul.io/)
- [Materialize](https://materialize.com/)
- [ClickHouse](https://clickhouse.com/)
- [OpenTelemetry](https://opentelemetry.io/), [Jaeger OT server](https://www.jaegertracing.io/)
# Next Generation Data Delivery
This is the home of LIGO's Next Generation Data Delivery (NGDD) project, the aim of which is to develop a low-latency data distribution and stream processing platform to meet the needs of real-time searches for gravitational waves.
![concept_full](uploads/eecb16d304cc6f02c521271a359bc505/concept_full.png)
## talks and presentations
- [The next generation of low-latency data distribution and stream processing for LIGO](https://dcc.ligo.org/LIGO-G2200571)
- [NGDD architecture overview](https://docs.google.com/presentation/d/1ItEeNtkLCAzxBYc-dLtPY35C1EBXQwQBV2LmgEPn2LI/edit?usp=sharing)
- [NGDD serialization overview](https://docs.google.com/presentation/d/1OGFqV7TBqMlqXr9I_JoOokDyZGW6KcC_REuoMjg22Eg/edit?usp=sharing)
## meetings
* \[Joint NGDD+LDAS face-to-face, August 13-15, CIT\]\[f2f-202408\]
## deployment
- [Current deployments](Deployments)
## development
- [[todo]]
- [Live monitor of LHO infrastructure](https://grafana.ligo-wa.caltech.edu/d/HnFEtyB7z/kafka-metrics?orgId=1&from=now-15m&to=now&refresh=5s)
- [Requirements document (draft)](https://ngdd.docs.ligo.org/requirements-doc/ngdd-requirements.pdf)
- [Client Protocol Schema](Client-Protocol-Schema)
- [Publisher Protocol Schema](Publisher-Protocol-Schema)
- [NGDD dev workshop, Jan. 2022](https://dcc.ligo.org/cgi-bin/private/DocDB/DisplayMeeting?conferenceid=1128)
- [notes from the workshop](https://docs.google.com/document/d/1bUmt_wzKaU5cGEfGHEfJnSqlUb7348kcrOQYMydZPdk/edit#heading=h.qf7tvz41m6al)
- [Tech investigation notes](https://docs.google.com/document/d/1YyHJSrM5gTfuoJVp6U9gOntdmonaEqBqF5Pf5VaJfkI/edit#heading=h.bw1cizptefma):
- [low-latency data distribution](https://wiki.ligo.org/Computing/DASWG/LowLatencyDataDistribution)
- [distributed key/value stores](https://wiki.ligo.org/Computing/DASWG/KeyValueStores)
- [Kafka client](https://dcc.ligo.org/DocDB/0180/G2200107/001/Kafka%20Client%20Tests%20-%20NGDD%20Jan%202022%20F2F.pdf)
- [Materialize](https://docs.google.com/presentation/d/1F0l7TKwxD4BHX4JHTXBIywApURG-OA7VRmUjlbtGi_U/edit?usp=sharing)
- [Materialize/ClickHouse](https://docs.google.com/presentation/d/1D0Kyg694PlKnOgHIK2lJJvAHR4aWju9xGolUWwOHG7U/edit?usp=sharing)
- [profiling](profiling)
- [LIGO NDS server architecture](https://git.ligo.org/groups/nds/-/wikis/LHO-Distributed-NDS-Install)
- [Nomad Review](Nomad-Review)
- [Brockill's thoughts on replay infrastructure](https://wiki.ligo.org/Computing/DASWG/ReplayInfrastructure)
- [Authn/z](Auth)
## applications
- [online/low-latency application table](https://docs.google.com/spreadsheets/d/15JzZ1KEHscwXhny8L0lxZIYXkiv6LlkCgSLi_Iuruyw/edit#gid=1412826968)
- [open public alerts application table (searches)](https://docs.google.com/spreadsheets/d/1kLO84LNSj6B_PTlB35ujh-uYxXLdFc7L4P2lxY1-KXc/edit#gid=1943285589)
## technologies
- [gRPC](https://grpc.io/)
- [Arrow](https://arrow.apache.org/)
- [Arrow Flight](https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/)
- [Kafka](https://kafka.apache.org/)
- [influxDB](https://www.influxdata.com/)
- [gstreamer RTP/RTSP](https://gstreamer.freedesktop.org/documentation/additional/rtp.html?gi-language=c)
- [consul](https://www.consul.io/)
- [Materialize](https://materialize.com/)
- [ClickHouse](https://clickhouse.com/)
- [OpenTelemetry](https://opentelemetry.io/), [Jaeger OT server](https://www.jaegertracing.io/)
\ No newline at end of file