GstLAL issueshttps://git.ligo.org/lscsoft/gstlal/-/issues2021-10-29T20:45:38Zhttps://git.ligo.org/lscsoft/gstlal/-/issues/91Refactor to use gsttools for pipeline creation and execution2021-10-29T20:45:38ZJames KenningtonRefactor to use gsttools for pipeline creation and executionWith the introduction of basic pipeline utilities in #88, now refactor the gstlal python package to use these utilities wherever possible to avoid code duplication.With the introduction of basic pipeline utilities in #88, now refactor the gstlal python package to use these utilities wherever possible to avoid code duplication.https://git.ligo.org/lscsoft/gstlal/-/issues/90Add tutorial for gstreamer pipeline creation and execution2021-10-29T20:43:20ZJames KenningtonAdd tutorial for gstreamer pipeline creation and executionAdd a tutorial, both markdown via sphinx and as Jupyter notebook, that shows how to construct a simple gstreamer pipeline, add some transformation elements, and execute the pipeline.Add a tutorial, both markdown via sphinx and as Jupyter notebook, that shows how to construct a simple gstreamer pipeline, add some transformation elements, and execute the pipeline.Documentationhttps://git.ligo.org/lscsoft/gstlal/-/issues/89Add pipeline graph visualization tools2021-10-29T20:01:47ZJames KenningtonAdd pipeline graph visualization toolsLightweight utilities that consume DOT formatted output from GStreamer and relabel / format for use with pydot. Can be output as cleaned svg or image files, as well as rendered directly in a Jupyter environmentLightweight utilities that consume DOT formatted output from GStreamer and relabel / format for use with pydot. Can be output as cleaned svg or image files, as well as rendered directly in a Jupyter environmenthttps://git.ligo.org/lscsoft/gstlal/-/issues/87Add pydot to env2021-11-05T16:09:04ZJames KenningtonAdd pydot to envThe `pydot` package is useful for interacting with graph representations of Gstreamer pipelinesThe `pydot` package is useful for interacting with graph representations of Gstreamer pipelineshttps://git.ligo.org/lscsoft/gstlal/-/issues/85Inspiral Microservice Zoo Map2021-10-29T19:55:31ZJames KenningtonInspiral Microservice Zoo MapCreate documentation that describes the complete set of micro services and interdependencies for the inspiral pipeline. This documentation can take the form of a YAML file, but the information is essentially a graph where:
- Services ar...Create documentation that describes the complete set of micro services and interdependencies for the inspiral pipeline. This documentation can take the form of a YAML file, but the information is essentially a graph where:
- Services are nodes
- Dependencies are labeled, directed edges
- Bottle (HTTP) deps
- Kafka deps, labeled by topic
Initial diagram from @patrick.godwin [here](tinyurl.com/zzmrkw84)https://git.ligo.org/lscsoft/gstlal/-/issues/84Add code coverage to CI2021-10-28T19:48:50ZJames KenningtonAdd code coverage to CIAdd code coverage report to CIAdd code coverage report to CIhttps://git.ligo.org/lscsoft/gstlal/-/issues/83Job identifiers used to encode meaningful information2021-10-28T02:01:51ZJames KenningtonJob identifiers used to encode meaningful informationJob IDs are being used beyond the scope of the files in which they are defined to encode a variety of information.
Known usages:
- Determine injection vs non injection job (using constant 2000 offset)
- Was proposed to pass informatio...Job IDs are being used beyond the scope of the files in which they are defined to encode a variety of information.
Known usages:
- Determine injection vs non injection job (using constant 2000 offset)
- Was proposed to pass information with grafana (need more description of intended use)https://git.ligo.org/lscsoft/gstlal/-/issues/80Keep python env synced across conda and el7 containers2021-09-20T17:19:29ZJames KenningtonKeep python env synced across conda and el7 containersThere are two docker containers used in the build suite:
- one built from a conda lock file
- one built from an e7 dockerfile
How should we syncronize python env changes across both containers? The conda lock container has `conda-flow`...There are two docker containers used in the build suite:
- one built from a conda lock file
- one built from an e7 dockerfile
How should we syncronize python env changes across both containers? The conda lock container has `conda-flow` for instance, but a separate, matching change would need to be made in the el7 containerhttps://git.ligo.org/lscsoft/gstlal/-/issues/76Consolidate and improve documentation2021-10-29T19:47:57ZChad HannaConsolidate and improve documentationWe need to improve our documentation. We know there are several components of documentation: see e.g.,
https://documentation.divio.com
- [x] Step 1: Figure out where our documentation is
Existing docs:
- GstLAL O3 Online: https://l...We need to improve our documentation. We know there are several components of documentation: see e.g.,
https://documentation.divio.com
- [x] Step 1: Figure out where our documentation is
Existing docs:
- GstLAL O3 Online: https://ldas-jobs.ligo.caltech.edu/~gstlalcbc/doc.html
- Source: https://docs.google.com/document/u/1/d/e/2PACX-1vS1014eAe587cIVnoP_KcsaSYxCa6BH0i_w489jBW4sErrAwv3glZI3Gj7wJAWcrfo6xHA-vCyuH_dP/pub?urp=gmail_link&gxids=7757#h.n8pcsdk1dn6k or https://docs.google.com/document/d/1ueap7Gj7x8qLmcdjumtbwZ_ue_kWocc3PRBrQtiBwQY/edit?skip_itp2_check=true#heading=h.q1zeppegd1qe
- Gitlab docs: https://lscsoft.docs.ligo.org/gstlal/
- https://git.ligo.org/gstlal/osg-workflow#osg-workflow
- https://git.ligo.org/gstlal/online-analysis#quickstart
- https://dcc.ligo.org/cgi-bin/private/DocDB/Search?.submit=Title&titlesearch=gstlal&titlesearchmode=allsub
- https://git.ligo.org/lscsoft/gstlal/-/blob/master/README.md
- https://arxiv.org/abs/2010.05082
- https://wiki.ligo.org/Computing/DASWG/GstLAL
Operational : git.ligo.org/gstlal ???
- [ ] Step 2: Figure out our local community's highest needs
- Want a working gstlal installation (support containers, conda, src, something for everyone). This boils down to: does this work: https://lscsoft.docs.ligo.org/gstlal/installation.html
- Want to know how to use the software for real stuff
- PSD generation
- frame generation (recoloring, add injections, etc)
- SVD/template bank generation
- mass models/dtdphi
- event followup
- offline CBC analysis (filtering/rerank/injection runs)
- generate segments/vetoes
- data discovery/access for data products (e.g. how to get all triggers from run below some FAR threshold)
- write gstreamer pipelines
- plotting
- We want to nurture a culture of contribution.
- Developer documentation
- should have lower level tutorials that can help with development
- How to setup an effective and successful development lifestyle
- How to debug? (should gather e.g., gstreamer debug stuff)
- Add logging support throughout the codeDocumentationhttps://git.ligo.org/lscsoft/gstlal/-/issues/73Add bankcorrelation mask functionality in itacac2021-09-17T19:43:08ZRyan MageeAdd bankcorrelation mask functionality in itacachttps://git.ligo.org/lscsoft/gstlal/-/merge_requests/54#note_404719https://git.ligo.org/lscsoft/gstlal/-/merge_requests/54#note_404719Ryan MageeRyan Mageehttps://git.ligo.org/lscsoft/gstlal/-/issues/72gstlal-ugly make fails on osx-64 when using gds-framexmit conda package2022-11-22T09:33:31ZJames Kenningtongstlal-ugly make fails on osx-64 when using gds-framexmit conda packageOn osx-64, adding gds-framexmit to the conda env causes gstlal-ugly `make` to fail with the error below. For a reproducible environment, see the [lock file](https://git.ligo.org/lscsoft/gstlal/-/snippets/128).
```
libtool: link: x86_64...On osx-64, adding gds-framexmit to the conda env causes gstlal-ugly `make` to fail with the error below. For a reproducible environment, see the [lock file](https://git.ligo.org/lscsoft/gstlal/-/snippets/128).
```
libtool: link: x86_64-apple-darwin13.4.0-clang++ -o .libs/libgstgds.so -bundle .libs/libgstgds_la-gds_plugin.o .libs/libgstgds_la-framexmitsink.o .libs/libgstgds_la-framexmitsrc.o .libs/libgstgds_la-lvshmsrc.o .libs/libgstgds_la-lvshmsink.o -L/Users/jim/repos/research/deps/lib -L/Users/jim/miniconda/envs/gstlal-dev-env/lib /Users/jim/repos/research/deps/lib/libgstlal.dylib -L/opt/local/lib -lfftw3 -lfftw3f /Users/jim/repos/research/deps/lib/libgstlaltags.dylib /Users/jim/repos/research/deps/lib/libgstlaltypes.dylib -llalmetaio -llalsupport -llalsimulation -llalburst -llalinspiral -lgstallocators-1.0 -lgstcontroller-1.0 -lz -lgsl -lgslcblas -lcblas -lgstaudio-1.0 -lgsttag-1.0 -llal -lgstbase-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 -lintl -llsmp -lframexmit -lsockutil -lpthread -lm -pthread -march=core2 -mtune=haswell -mssse3 -fstack-protector-strong -O2 -stdlib=libc++ -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath -Wl,/Users/jim/miniconda/envs/gstlal-dev-env/lib -pthread -fopenmp -Wl,-exported_symbols_list,.libs/libgstgds-symbols.expsym
ld: warning: -pie being ignored. It is only used when linking a main executable
Undefined symbols for architecture x86_64:
"_TAInow", referenced from:
create(_GstBaseSrc*, unsigned long long, unsigned int, _GstBuffer**) in libgstgds_la-framexmitsrc.o
create(_GstBaseSrc*, unsigned long long, unsigned int, _GstBuffer**) in libgstgds_la-lvshmsrc.o
ld: symbol(s) not found for architecture x86_64
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [Makefile:550: libgstgds.la] Error 1
make[2]: Leaving directory '/Users/jim/repos/research/gstlal/gstlal-ugly/gst/gds'
make[1]: *** [Makefile:475: all-recursive] Error 1
make[1]: Leaving directory '/Users/jim/repos/research/gstlal/gstlal-ugly/gst'
make: *** [Makefile:509: all-recursive] Error 1
```
<br>
Note: Even though framexmit datasources can't be run on osx-64, the gstlal suite and python package should be able to build on osx-64. We intend to add a warning for anyone building pipelines on osx that the framexmit source will only run on linux.https://git.ligo.org/lscsoft/gstlal/-/issues/69Incorrect coinc_inspiral.end_time_ns while producing snr time series especial...2021-09-17T19:41:09ZSurabhi SachdevIncorrect coinc_inspiral.end_time_ns while producing snr time series especially in subthreshold trigggersWhile making snr time series, sometimes the coinc_inspiral.end_time_ns is incorrectly recorded (does not match the sngl_inspiral.end_time_ns). While producing snr time series for events down to 2/day, this happened 1-2 times every chunk....While making snr time series, sometimes the coinc_inspiral.end_time_ns is incorrectly recorded (does not match the sngl_inspiral.end_time_ns). While producing snr time series for events down to 2/day, this happened 1-2 times every chunk. These usually get fixed on rerunning. An example is on ICS:
On ICS:
$ cd /ligo/home/ligo.org/gstlalcbc/observing/3/final/a/rerank/final_runs/final_rerank/coinc_files/chunk8_1243161494_1243786890
$ ligolw_print gracedb_uploads/12433/H1L1V1-GSTLAL_5_089_CBC_AllSky-1243305426-1.xml -t coinc_inspiral -c end_time -c end_time_ns
1243305426,**958782764**
$ ligolw_print gracedb_uploads/12433/H1L1V1-GSTLAL_5_089_CBC_AllSky-1243305426-1.xml -t sngl_inspiral -c end_time -c end_time_ns
1243305426,**325683594**
1243305426,**330472764**
1243305426,**317871094**
there is a file runme2.sh that contains the job to run. It is not always reproducible though, sometimes on rerunning, the end times are fixed.Chad HannaCody MessickPatrick GodwinChad Hannahttps://git.ligo.org/lscsoft/gstlal/-/issues/68gstreamer version for O42021-09-17T19:52:10ZStuart Andersongstreamer version for O4What version of GStreamer will GstLAL use for O4? Stick with the current 1.14.4, upgrade to 1.16.x (e.g., RHEL8 1.16.1-patch or further upstream 1.16.3), or update to 1.18.x (e.g., 1.18.3 rpm from OpenSuSE Tumbleweed)? Note, I don't know...What version of GStreamer will GstLAL use for O4? Stick with the current 1.14.4, upgrade to 1.16.x (e.g., RHEL8 1.16.1-patch or further upstream 1.16.3), or update to 1.18.x (e.g., 1.18.3 rpm from OpenSuSE Tumbleweed)? Note, I don't know how practical those options are for the system installed version if still needed in O4 (given the dependencies of other OS packages, or how easily different source RPM might build), of if there are any significant benefits to updating--just curious.https://git.ligo.org/lscsoft/gstlal/-/issues/67Issues casting array/matrix GStreamer properties to GValueArray on newer vers...2023-04-04T16:33:20ZPatrick GodwinIssues casting array/matrix GStreamer properties to GValueArray on newer versions of pygobject```
reading 'L1_psd.xml.gz' ...
Traceback (most recent call last):
File "/home/patrick.godwin/.conda/envs/gstlal-dev/bin/gstlal_fake_frames", line 307, in <module>
head = pipeparts.mkfirbank(pipeline, head, latency = latency, fir_m...```
reading 'L1_psd.xml.gz' ...
Traceback (most recent call last):
File "/home/patrick.godwin/.conda/envs/gstlal-dev/bin/gstlal_fake_frames", line 307, in <module>
head = pipeparts.mkfirbank(pipeline, head, latency = latency, fir_matrix = [fir_matrix], block_stride = 1 * options.sample_rate)
File "/home/patrick.godwin/.conda/envs/gstlal-dev/lib/python3.7/site-packages/gstlal/pipeparts/__init__.py", line 571, in mkfirbank
return mkgeneric(pipeline, src, "lal_firbank", **properties)
File "/home/patrick.godwin/.conda/envs/gstlal-dev/lib/python3.7/site-packages/gstlal/pipeparts/__init__.py", line 100, in mkgeneric
elem.set_property(name.replace("_", "-"), value)
TypeError: could not convert [array([-1.86826653e-56, -7.32817040e-31, -2.93145269e-30, ...,
-2.93145269e-30, -7.32817040e-31, -1.86826653e-56])] to type 'GValueArray' when setting property 'GSTLALFIRBank.fir-matrix'
```
My current suspicion is that it was caused from this commit, but have not confirmed this: https://gitlab.gnome.org/GNOME/pygobject/-/commit/b5eab39ebcae061a46c186beac47f7e936ce57c2https://git.ligo.org/lscsoft/gstlal/-/issues/65gstlal_inspiral_destagger_injections: port to ligo-lw dbtables.workingcopy2021-09-17T19:43:46ZPatrick Godwingstlal_inspiral_destagger_injections: port to ligo-lw dbtables.workingcopyThis program was not ported as part of 6280612b4eb88effb1a5e53776f0c2ddc89e2ea0, with the following rationale:
> those two codes are missing the required clean-up function calls, and leak working copies out into scratch disks. since I d...This program was not ported as part of 6280612b4eb88effb1a5e53776f0c2ddc89e2ea0, with the following rationale:
> those two codes are missing the required clean-up function calls, and leak working copies out into scratch disks. since I don't know what behaviour was intended, I can't correctly migrate them to the new mechanism.
It will be required to convert this before the next version of python-ligo-lw is released. That commit can be used as a conversion guide.
I have assigned this to who I believe is the code owner. Feel free to reassign this if I have assigned this to the wrong person.Cody MessickCody Messickhttps://git.ligo.org/lscsoft/gstlal/-/issues/64gstlal_inspiral_coinc_extractor_dag: port to ligo-lw dbtables.workingcopy2021-09-17T19:43:58ZPatrick Godwingstlal_inspiral_coinc_extractor_dag: port to ligo-lw dbtables.workingcopyThis program was not ported as part of 6280612b4eb88effb1a5e53776f0c2ddc89e2ea0, with the following rationale:
> those two codes are missing the required clean-up function calls, and leak working copies out into scratch disks. since I ...This program was not ported as part of 6280612b4eb88effb1a5e53776f0c2ddc89e2ea0, with the following rationale:
> those two codes are missing the required clean-up function calls, and leak working copies out into scratch disks. since I don't know what behaviour was intended, I can't correctly migrate them to the new mechanism.
It will be required to convert this before the next version of python-ligo-lw is released. That commit can be used as a conversion guide.
I have assigned this to who I believe is the code owner. Feel free to reassign this if I have assigned this to the wrong person.Debnandini MukherjeeDebnandini Mukherjeehttps://git.ligo.org/lscsoft/gstlal/-/issues/55gstlal tests raise SyntaxError on python3.62021-09-17T19:41:55ZDuncan Macleodduncan.macleod@ligo.orggstlal tests raise SyntaxError on python3.6The `gstlal/tests/test_common.py` script raises a `SyntaxError` on python3.6:
```console
$ python3.6 tests/test_common.py
File "tests/test_common.py", line 148
def need_data(elem, arg, (input_arrays, rate)):
...The `gstlal/tests/test_common.py` script raises a `SyntaxError` on python3.6:
```console
$ python3.6 tests/test_common.py
File "tests/test_common.py", line 148
def need_data(elem, arg, (input_arrays, rate)):
^
SyntaxError: invalid syntax
```https://git.ligo.org/lscsoft/gstlal/-/issues/51UI related to gstlal_itacac.c2021-09-17T19:53:28ZKoh UenoUI related to gstlal_itacac.cI attempted to run a dag for the O3GK all-sky search while I expected gstlal_inspiral to fail at the itacac part since it hasn't had the G1 detector yet. However, it didn't fail. So I ran one of the gstlal_inspiral jobs separately with t...I attempted to run a dag for the O3GK all-sky search while I expected gstlal_inspiral to fail at the itacac part since it hasn't had the G1 detector yet. However, it didn't fail. So I ran one of the gstlal_inspiral jobs separately with the GST_DEBUG option and found that gstlal_itacac.c actually detected the issue as follows
0:00:21.795395471 151352 0x3be8ed0 ERROR lal_itacac gstlal_itacac.c:208:get_instrument_pad_pointer:<gstlalitacac0:sink_0> pad has unknown detector, G1
at get_instrument_pad_pointer, but it isn't treated as a fatal error. I think we should make the issue fatal with g_assert or in some other appropriate ways so that the main program is immediately aborted in detection of this error. I would like to hear other people's opinions about this.Cody MessickCody Messickhttps://git.ligo.org/lscsoft/gstlal/-/issues/50Possibly undesired behavior from gst_audioadapter_copy_samples gap management2021-09-17T19:41:09ZCody MessickPossibly undesired behavior from gst_audioadapter_copy_samples gap managementIf you end up in a situation where you have gap buffers with zero duration being passed to the audioadapter, the function gst_audioadapter_copy_samples may incorrectly mark that the copied samples include a gap. I'm not sure if this is s...If you end up in a situation where you have gap buffers with zero duration being passed to the audioadapter, the function gst_audioadapter_copy_samples may incorrectly mark that the copied samples include a gap. I'm not sure if this is something that should be "fixed" in this function, by e.g. checking that the gap buffer contains more than 0 samples before setting the _copied_nongap boolean to TRUE, or if the "fix" is to simply not push any zero-length gap buffers to the adapter.Cody MessickCody Messickhttps://git.ligo.org/lscsoft/gstlal/-/issues/45gstlal-inspiral TESTS not included in tarballs2021-09-17T19:53:56ZDuncan Macleodduncan.macleod@ligo.orggstlal-inspiral TESTS not included in tarballs`gstlal-inspiral`'s python `TESTS` scripts (from `/gstlal-inspiral/tests/Makefile.am`) aren't included in the `EXTRA_DIST` list, which means they aren't being included in the release distributions, which in turn results in build failures...`gstlal-inspiral`'s python `TESTS` scripts (from `/gstlal-inspiral/tests/Makefile.am`) aren't included in the `EXTRA_DIST` list, which means they aren't being included in the release distributions, which in turn results in build failures when executing `make check` from a source build:
```
make check-TESTS
make[2]: Entering directory '$SRC_DIR/_build/tests'
make[3]: *** No rule to make target 'stats_horizonhistory_verify.py', needed by 'stats_horizonhistory_verify.py.log'. Stop.
```