python-ligo-lw issueshttps://git.ligo.org/kipp/python-ligo-lw/-/issues2022-08-16T06:25:25Zhttps://git.ligo.org/kipp/python-ligo-lw/-/issues/24ligolw_segments raises AttributeError due to removal of set_process_end_time2022-08-16T06:25:25ZDuncan Macleodduncan.macleod@ligo.orgligolw_segments raises AttributeError due to removal of set_process_end_timeAs of 1.8.0, `ligolw_segments` raises an `AttributeError` due to 40a1c07d8db89c094d71db35aa378ba222c89412:
```console
$ echo "1 2
> 3 4
> 5 7
> " | ligolw_segments
Traceback (most recent call last):
File "/home/duncan/opt/mambaforge/e...As of 1.8.0, `ligolw_segments` raises an `AttributeError` due to 40a1c07d8db89c094d71db35aa378ba222c89412:
```console
$ echo "1 2
> 3 4
> 5 7
> " | ligolw_segments
Traceback (most recent call last):
File "/home/duncan/opt/mambaforge/envs/py310/bin/ligolw_segments", line 212, in <module>
ligolw_process.set_process_end_time(process)
AttributeError: module 'ligo.lw.utils.process' has no attribute 'set_process_end_time'
```https://git.ligo.org/kipp/python-ligo-lw/-/issues/23Add optional timeout for ligo.lw.utils.load_url2021-12-08T10:06:06ZPatrick GodwinAdd optional timeout for ligo.lw.utils.load_urlIf passing in a URL which would be parsed by `urlopen`, this can potentially hang forever without being able to specify a timeout to it (https://git.ligo.org/kipp.cannon/python-ligo-lw/-/blob/master/ligo/lw/utils/__init__.py#L439).
This...If passing in a URL which would be parsed by `urlopen`, this can potentially hang forever without being able to specify a timeout to it (https://git.ligo.org/kipp.cannon/python-ligo-lw/-/blob/master/ligo/lw/utils/__init__.py#L439).
This failure mode was seen in an online gstlal job (`gstlal_inspiral_marginalize_likelihoods_online`) when pinging one of the bottle routes, where the server was taking in requests but never sending a response. Ultimately I tracked the polling code to the line linked above.https://git.ligo.org/kipp/python-ligo-lw/-/issues/22python-ligo-lw incompatible with python 3.10: PyLong_FromUnicode has been rem...2022-07-15T13:02:45ZDuncan Macleodduncan.macleod@ligo.orgpython-ligo-lw incompatible with python 3.10: PyLong_FromUnicode has been removedpython-ligo-lw cannot be used with Python 3.10 as follows:
```console
$ git status
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
$ git rev-parse HEAD
40a1c07d8db89c094d71db35aa37...python-ligo-lw cannot be used with Python 3.10 as follows:
```console
$ git status
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
$ git rev-parse HEAD
40a1c07d8db89c094d71db35aa378ba222c89412
$ python3 --version
Python 3.10.0
$ CFLAGS="${CFLAGS} -Wfatal-errors -Werror=implicit-function-declaration" python3 setup.py build_ext --inplace
/home/duncan/git/python-ligo-lw/setup.py:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils.core import setup, Extension
/home/duncan/opt/mambaforge/envs/py310/lib/python3.10/distutils/dist.py:274: UserWarning: Unknown distribution option: 'namespace_packages'
warnings.warn(msg)
/home/duncan/opt/mambaforge/envs/py310/lib/python3.10/distutils/dist.py:274: UserWarning: Unknown distribution option: 'install_requires'
warnings.warn(msg)
/home/duncan/opt/mambaforge/envs/py310/lib/python3.10/distutils/command/build_ext.py:13: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
from distutils.sysconfig import customize_compiler, get_python_version
running build_ext
building 'ligo.lw.tokenizer' extension
gcc -pthread -B /home/duncan/opt/mambaforge/envs/py310/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/duncan/opt/mambaforge/envs/py310/include -fPIC -O2 -isystem /home/duncan/opt/mambaforge/envs/py310/include -Wfatal-errors -Werror=implicit-function-declaration -fPIC -Iligo/lw -I/home/duncan/opt/mambaforge/envs/py310/include/python3.10 -c ligo/lw/tokenizer.RowBuilder.c -o build/temp.linux-x86_64-3.10/ligo/lw/tokenizer.RowBuilder.o
ligo/lw/tokenizer.RowBuilder.c:230:39: warning: missing braces around initializer [-Wmissing-braces]
PyTypeObject ligolw_RowBuilder_Type = {
^
ligo/lw/tokenizer.RowBuilder.c:230:39: warning: missing braces around initializer [-Wmissing-braces]
gcc -pthread -B /home/duncan/opt/mambaforge/envs/py310/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/duncan/opt/mambaforge/envs/py310/include -fPIC -O2 -isystem /home/duncan/opt/mambaforge/envs/py310/include -Wfatal-errors -Werror=implicit-function-declaration -fPIC -Iligo/lw -I/home/duncan/opt/mambaforge/envs/py310/include/python3.10 -c ligo/lw/tokenizer.RowDumper.c -o build/temp.linux-x86_64-3.10/ligo/lw/tokenizer.RowDumper.o
ligo/lw/tokenizer.RowDumper.c: In function ‘__init__’:
ligo/lw/tokenizer.RowDumper.c:103:3: warning: ‘PyUnicode_FromUnicode’ is deprecated [-Wdeprecated-declarations]
rowdumper->delimiter = PyUnicode_FromUnicode(&default_delimiter, 1);
^~~~~~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.RowDumper.c:29:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:551:42: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_FromUnicode(
^~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.RowDumper.c: In function ‘next’:
ligo/lw/tokenizer.RowDumper.c:223:4: warning: ‘PyUnicode_FromUnicode’ is deprecated [-Wdeprecated-declarations]
token = PyUnicode_FromUnicode(NULL, 0); /* u"" */
^~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.RowDumper.c:29:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:551:42: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_FromUnicode(
^~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.RowDumper.c: At top level:
ligo/lw/tokenizer.RowDumper.c:280:38: warning: missing braces around initializer [-Wmissing-braces]
PyTypeObject ligolw_RowDumper_Type = {
^
ligo/lw/tokenizer.RowDumper.c:280:38: warning: missing braces around initializer [-Wmissing-braces]
gcc -pthread -B /home/duncan/opt/mambaforge/envs/py310/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/duncan/opt/mambaforge/envs/py310/include -fPIC -O2 -isystem /home/duncan/opt/mambaforge/envs/py310/include -Wfatal-errors -Werror=implicit-function-declaration -fPIC -Iligo/lw -I/home/duncan/opt/mambaforge/envs/py310/include/python3.10 -c ligo/lw/tokenizer.Tokenizer.c -o build/temp.linux-x86_64-3.10/ligo/lw/tokenizer.Tokenizer.o
ligo/lw/tokenizer.Tokenizer.c: In function ‘add_to_data’:
ligo/lw/tokenizer.Tokenizer.c:96:2: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
Py_ssize_t n = PyUnicode_GET_SIZE(unicode);
^~~~~~~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
^~~~~~~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:96:2: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
Py_ssize_t n = PyUnicode_GET_SIZE(unicode);
^~~~~~~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
^~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:96:2: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
Py_ssize_t n = PyUnicode_GET_SIZE(unicode);
^~~~~~~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
^~~~~~~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:138:3: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
memcpy(tokenizer->length, PyUnicode_AsUnicode(unicode), n * sizeof(*tokenizer->length));
^~~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
^~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c: In function ‘parse_error’:
ligo/lw/tokenizer.Tokenizer.c:200:2: warning: ‘PyUnicode_FromUnicode’ is deprecated [-Wdeprecated-declarations]
buffer_str = PyUnicode_FromUnicode(buffer, buffer_length);
^~~~~~~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:551:42: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_FromUnicode(
^~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:201:2: warning: ‘PyUnicode_FromUnicode’ is deprecated [-Wdeprecated-declarations]
pos_str = PyUnicode_FromUnicode(pos, 1);
^~~~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:551:42: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_FromUnicode(
^~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c: In function ‘__init__’:
ligo/lw/tokenizer.Tokenizer.c:514:2: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
if(PyUnicode_GET_SIZE(arg) != 1) {
^~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
^~~~~~~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:514:2: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
if(PyUnicode_GET_SIZE(arg) != 1) {
^~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
^~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:514:2: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
if(PyUnicode_GET_SIZE(arg) != 1) {
^~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
^~~~~~~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:519:2: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
tokenizer->delimiter = *PyUnicode_AS_UNICODE(arg);
^~~~~~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
^~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c: In function ‘next’:
ligo/lw/tokenizer.Tokenizer.c:584:3: warning: ‘PyUnicode_EncodeDecimal’ is deprecated [-Wdeprecated-declarations]
if(PyUnicode_EncodeDecimal(start, end - start, ascii_buffer, NULL))
^~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:952:36: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(int) PyUnicode_EncodeDecimal(
^~~~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:598:3: warning: ‘PyUnicode_FromUnicode’ is deprecated [-Wdeprecated-declarations]
token = PyUnicode_FromUnicode(start, end - start);
^~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:551:42: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_FromUnicode(
^~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:607:11: error: implicit declaration of function ‘PyLong_FromUnicode’; did you mean ‘PyLong_FromPid’? [-Werror=implicit-function-declaration]
token = PyLong_FromUnicode(start, end - start, 0);
^~~~~~~~~~~~~~~~~~
PyLong_FromPid
compilation terminated due to -Wfatal-errors.
cc1: some warnings being treated as errors
error: command '/usr/bin/gcc' failed with exit code 1
```
For details of the Python 3.10 change, see [here](https://docs.python.org/3/whatsnew/3.10.html#id4). @kipp.cannon, are you able to provide a patch for this?https://git.ligo.org/kipp/python-ligo-lw/-/issues/20Migrate `lalapps_run_sqlite` to python-ligo-lw?2021-12-08T11:12:31ZPatrick GodwinMigrate `lalapps_run_sqlite` to python-ligo-lw?While this is located in `lalapps` (https://git.ligo.org/lscsoft/lalsuite/-/blob/master/lalapps/src/inspiral/lalapps_run_sqlite.py), this program doesn't rely on lalsuite at all. It'd be nice to not need to have lalsuite installed to exe...While this is located in `lalapps` (https://git.ligo.org/lscsoft/lalsuite/-/blob/master/lalapps/src/inspiral/lalapps_run_sqlite.py), this program doesn't rely on lalsuite at all. It'd be nice to not need to have lalsuite installed to execute SQL on LIGOLW files with this utility.
What do you think, @kipp.cannon?https://git.ligo.org/kipp/python-ligo-lw/-/issues/19API change in ligo.lw.utils.process.register_to_xmldoc, intended?2021-03-08T08:59:52ZPatrick GodwinAPI change in ligo.lw.utils.process.register_to_xmldoc, intended?Running a gstlal job using ligo-lw 1.7.1 is throwing me an error in calling `register_to_xmldoc`:
```
Traceback (most recent call last):
File "/home/patrick.godwin/local/gcc_master_210203/opt/bin/gstlal_inspiral_create_prior_diststats...Running a gstlal job using ligo-lw 1.7.1 is throwing me an error in calling `register_to_xmldoc`:
```
Traceback (most recent call last):
File "/home/patrick.godwin/local/gcc_master_210203/opt/bin/gstlal_inspiral_create_prior_diststats", line 166, in <module>
process = ligolw_process.register_to_xmldoc(xmldoc, u"gstlal_inspiral_create_prior_diststats", paramdict = process_params, ifos = options.instrument)
File "/usr/lib64/python3.6/site-packages/ligo/lw/utils/process.py", line 109, in register_to_xmldoc
process = proctable.RowType.initialized(program = program, process_id = proctable.get_next_id(), **kwargs)
TypeError: initialized() got an unexpected keyword argument 'ifos'
```
* Before: https://git.ligo.org/kipp.cannon/python-ligo-lw/-/blob/5b8eaa7eaabffbe72fab2c46f696811f473a4a4f/ligo/lw/utils/process.py#L82
* After: https://git.ligo.org/kipp.cannon/python-ligo-lw/-/blob/5faced2e682833caa0165f2a0bbcc7eaea99c4a5/ligo/lw/lsctables.py#L483
It looks like the API was changed during the `ligo.lw.utils.process` rework, so that it now takes `instruments` rather than the `ifos` kwarg:
I wanted to double check that this was intended since this shows up in a number of places in gstlal and may cause some issues unintended issues when it's deployed on SL7. I didn't see this in the changelogs but could have missed it.https://git.ligo.org/kipp/python-ligo-lw/-/issues/18ligolw_segments using python2 'file' keyword on python32021-02-16T09:04:58ZDuncan Macleodduncan.macleod@ligo.orgligolw_segments using python2 'file' keyword on python3`ligolw_segments` fails with a `NameError` on python3 because of the use of the python2 `file` keyword
```console
+ ligolw_segments -o python-ligo-lw-segments.xml.gz --insert-from-segwizard=X1=python-ligo-lw-segments.txt
Traceback (most...`ligolw_segments` fails with a `NameError` on python3 because of the use of the python2 `file` keyword
```console
+ ligolw_segments -o python-ligo-lw-segments.xml.gz --insert-from-segwizard=X1=python-ligo-lw-segments.txt
Traceback (most recent call last):
File "/home/duncan/opt/miniconda3/envs/py39/bin/ligolw_segments", line 168, in <module>
segments_tables.insert_from_segwizard(file(filename), set([instrument]), options.name, version = options.segments_version, comment = options.comment)
NameError: name 'file' is not defined
```https://git.ligo.org/kipp/python-ligo-lw/-/issues/17ValueError: invalid Name 'Sn(f)' when writing lal FrequencySeries2021-02-06T07:11:02ZPatrick GodwinValueError: invalid Name 'Sn(f)' when writing lal FrequencySeriesWhile this error originates from lalsuite/gstlal, it appears that this issue is a regression in behavior in ligo-lw in not allowing parentheses in names. Let me know if it's more appropriate for this issue to be moved to a downstream lib...While this error originates from lalsuite/gstlal, it appears that this issue is a regression in behavior in ligo-lw in not allowing parentheses in names. Let me know if it's more appropriate for this issue to be moved to a downstream library.
This was also found in `python-ligo-lw 1.6.0` so feel free to close if this has already been fixed.
Example command and traceback:
```
$ gstlal_psd_xml_from_asd_txt --instrument H1 --output H1_psd.xml.gz gstlal/share/early_aligo_asd.txt
Traceback (most recent call last):
File "/usr/bin/gstlal_psd_xml_from_asd_txt", line 88, in <module>
reference_psd.write_psd(options.output, {options.instrument: psdseries})
File "/usr/lib64/python3.6/site-packages/gstlal/reference_psd.py", line 207, in write_psd
ligolw_utils.write_filename(lal.series.make_psd_xmldoc(psddict), filename, gz = (filename or "stdout").endswith(".gz"), verbose = verbose, trap_signals = trap_signals)
File "/usr/lib64/python3.6/site-packages/lal/series.py", line 215, in make_psd_xmldoc
fs = lw.appendChild(build_REAL8FrequencySeries(psd))
File "/usr/lib64/python3.6/site-packages/lal/series.py", line 128, in build_REAL8FrequencySeries
return _build_series(series, (u"Frequency,Real", u"Frequency"), comment, 'deltaF', 's^-1')
File "/usr/lib64/python3.6/site-packages/lal/series.py", line 63, in _build_series
a = ligolw_array.Array.build(series.name, data, dim_names=dim_names)
File "/usr/lib64/python3.6/site-packages/ligo/lw/array.py", line 206, in build
elem.Name = name
File "/usr/lib64/python3.6/site-packages/ligo/lw/ligolw.py", line 163, in setter
self.setAttribute(name, enc(value))
File "/usr/lib64/python3.6/site-packages/ligo/lw/ligolw.py", line 472, in enc
raise ValueError("invalid Name '%s'" % name)
ValueError: invalid Name 'Sn(f)'
```https://git.ligo.org/kipp/python-ligo-lw/-/issues/16merge request 23 has broken install2023-03-16T21:07:02ZKipp Cannonmerge request 23 has broken install@leo-singer Since merging https://git.ligo.org/kipp.cannon/python-ligo-lw/-/merge_requests/23, the unit tests fail with
```
Traceback (most recent call last):
File "/home/kipp/local/bin/ligolw_print", line 4, in <module>
__import...@leo-singer Since merging https://git.ligo.org/kipp.cannon/python-ligo-lw/-/merge_requests/23, the unit tests fail with
```
Traceback (most recent call last):
File "/home/kipp/local/bin/ligolw_print", line 4, in <module>
__import__('pkg_resources').run_script('python-ligo-lw==1.7.0', 'ligolw_print')
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 665, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1451, in run_script
raise ResolutionError(
pkg_resources.ResolutionError: Script 'scripts/ligolw_print' not found in metadata at None
```
This does not correspond to line 4 of that script, but when I look inside $(which ligolw_print) it's nearly empty, and this error message does correspond to what is there. What happened to the programs in bin? Why are they empty, why does whatever python has done to them not work?https://git.ligo.org/kipp/python-ligo-lw/-/issues/15CoincInspiral.set_ifos() method compatibility2020-08-22T00:46:58ZTito Dal CantonCoincInspiral.set_ifos() method compatibilityIn glue.ligolw, with a `CoincInspiral` instance one could do
```
coinc_inspiral_instance.set_ifos(ifos)
```
It looks like the same method is not available in python-ligo-lw. Adding it would simplify porting code from glue to python-ligo-lw.In glue.ligolw, with a `CoincInspiral` instance one could do
```
coinc_inspiral_instance.set_ifos(ifos)
```
It looks like the same method is not available in python-ligo-lw. Adding it would simplify porting code from glue to python-ligo-lw.https://git.ligo.org/kipp/python-ligo-lw/-/issues/14tmp_path kwarg in dbtables.get_connection_filename not working for python 32021-03-05T02:15:06ZPatrick Godwintmp_path kwarg in dbtables.get_connection_filename not working for python 3This appears to be leveraging `temporary_file.unlink()`, which was an undocumented feature in python 2 and doesn't exist in python 3. Some more information in https://stackoverflow.com/a/28067461.
A traceback from a gstlal job failing w...This appears to be leveraging `temporary_file.unlink()`, which was an undocumented feature in python 2 and doesn't exist in python 3. Some more information in https://stackoverflow.com/a/28067461.
A traceback from a gstlal job failing when using this functionality:
```
Traceback (most recent call last):
File "/usr/bin/gstlal_inspiral_plotsummary", line 1814, in <module>
working_filename = dbtables.get_connection_filename(filename, tmp_path = options.tmp_space, verbose = options.verbose)
File "/usr/lib64/python3.6/site-packages/ligo/lw/dbtables.py", line 253, in get_connection_filename
target = mktmp(tmp_path, suffix = ".".join(os.path.split(filename)[-1].split(".")[1:]), verbose = verbose)
File "/usr/lib64/python3.6/site-packages/ligo/lw/dbtables.py", line 196, in mktmp
def new_unlink(self, orig_unlink = temporary_file.unlink):
File "/usr/lib64/python3.6/tempfile.py", line 480, in __getattr__
a = getattr(file, name)
AttributeError: '_io.BufferedRandom' object has no attribute 'unlink'
```https://git.ligo.org/kipp/python-ligo-lw/-/issues/13ligolw segments CLI utils fail on ImportErrors2020-02-21T04:52:48ZPatrick Godwinligolw segments CLI utils fail on ImportErrorsAs an example,
```
[patrick.godwin@ldas-pcdev5 ~] $ ligolw_segment_diff --help
Traceback (most recent call last):
File "/usr/bin/ligolw_segment_diff", line 44, in <module>
from glue.ligolw import utils
ModuleNotFoundError: No modu...As an example,
```
[patrick.godwin@ldas-pcdev5 ~] $ ligolw_segment_diff --help
Traceback (most recent call last):
File "/usr/bin/ligolw_segment_diff", line 44, in <module>
from glue.ligolw import utils
ModuleNotFoundError: No module named 'glue.ligolw'
```
Now, these are actually part of the lscsoft-glue library rather than ligo-lw but at its current state, I believe this has been broken since the ligolw module in glue was removed. It's not clear to me if the right thing to do is move these utilities over to ligolw or to fix the imports in the glue library.https://git.ligo.org/kipp/python-ligo-lw/-/issues/12python-ligo-lw 1.6.0 on debian stretch missing some scripts2019-12-17T09:23:06ZPatrick Godwinpython-ligo-lw 1.6.0 on debian stretch missing some scriptsThis is after pulling the 1.6.0 package from the debian repo:
```/bin/bash: ligolw_no_ilwdchar: command not found```This is after pulling the 1.6.0 package from the debian repo:
```/bin/bash: ligolw_no_ilwdchar: command not found```https://git.ligo.org/kipp/python-ligo-lw/-/issues/11python-ligo-lw packaging changes to cover Bullseye (without python2)2020-02-25T07:48:10ZSteffen Grunewaldpython-ligo-lw packaging changes to cover Bullseye (without python2)The following covers the full set of changes made to the packaging provided with the 1.6.0 tarball:
```
diff -Nur old/debian/changelog new/debian/changelog
--- old/debian/changelog 2019-06-13 11:42:47.000000000 +0200
+++ new/debia...The following covers the full set of changes made to the packaging provided with the 1.6.0 tarball:
```
diff -Nur old/debian/changelog new/debian/changelog
--- old/debian/changelog 2019-06-13 11:42:47.000000000 +0200
+++ new/debian/changelog 2019-11-05 11:37:39.000000000 +0100
@@ -1,3 +1,17 @@
+python-ligo-lw (1.6.0-2.1) unstable; urgency=medium
+
+ * repackaging changes, to build also with python3 only
+
+ -- Steffen Grunewald <steffen.grunewald@ligo.org> Tue, 05 Nov 2019 11:11:00 +0100
+
+python-ligo-lw (1.6.0-2) unstable; urgency=medium
+
+ * repackaging:
+ move executables into ligo-lw-bin
+ attempt to run tests but ignore outcome
+
+ -- Steffen Grunewald <steffen.grunewald@ligo.org> Wed, 11 Sep 2019 09:25:00 +0200
+
python-ligo-lw (1.6.0-1) unstable; urgency=medium
* remove custom ProcessParamsTable from dbtables
diff -Nur old/debian/control new/debian/control
--- old/debian/control 2019-05-31 09:31:21.000000000 +0200
+++ new/debian/control 2019-11-05 11:37:39.000000000 +0100
@@ -5,7 +5,7 @@
Build-Depends:
debhelper (>= 9),
dh-python,
- python-dev,
+ python-all-dev,
python-ligo-common,
python-six,
python3-dev,
@@ -37,13 +37,16 @@
python-pylal
Replaces:
python-glue-ligolw-tools
-Provides:
- python-glue-ligolw-tools
-Description: LIGO Light-Weight XML I/O library
+#Provides:
+# python-glue-ligolw-tools
+# * moved to ligo-lw-bin
+Description: LIGO Light-Weight XML I/O library - Python 2
The LIGO Light-Weight XML format is used extensively by compact object
detection pipelines and associate tool sets. This package provides a
Python I/O library for reading, writing, and interacting with documents in
this format.
+ .
+ This package provides the Python2 modules.
Package: python3-ligo-lw
Architecture: any
@@ -66,10 +69,44 @@
python3-glue (<< 2.0)
Replaces:
python3-glue-ligolw-tools
-Provides:
- python3-glue-ligolw-tools
-Description: LIGO Light-Weight XML I/O library
+#Provides:
+# python3-glue-ligolw-tools
+# * moved to ligo-lw-bin
+Description: LIGO Light-Weight XML I/O library - Python 3
+ The LIGO Light-Weight XML format is used extensively by compact object
+ detection pipelines and associate tool sets. This package provides a
+ Python I/O library for reading, writing, and interacting with documents in
+ this format.
+ .
+ This package provides the Python3 modules.
+
+Package: ligo-lw-bin
+Architecture: any
+Depends:
+ ${shlibs:Depends},
+ ${misc:Depends},
+ ${python:Depends},
+ python-ligo-lw (= ${binary:Version}),
+ python3-ligo-lw (= ${binary:Version})
+Recommends:
+ python-lalburst,
+ python3-lalburst
+Conflicts:
+ python-glue-ligolw-tools,
+ python-glue (<< 2.0),
+ python-pylal,
+ python-ligo-lw (<< 1.6.0-2)
+Replaces:
+ python-glue-ligolw-tools,
+ python-ligo-lw (<< 1.6.0-2)
+#Provides:
+# python-glue-ligolw-tools,
+# python3-glue-ligolw-tools
+Description: LIGO Light-Weight XML I/O library - Tools
The LIGO Light-Weight XML format is used extensively by compact object
detection pipelines and associate tool sets. This package provides a
Python I/O library for reading, writing, and interacting with documents in
this format.
+ .
+ This package provides the executables.
+
diff -Nur old/debian/ligo-lw-bin.install new/debian/ligo-lw-bin.install
--- old/debian/ligo-lw-bin.install 1970-01-01 01:00:00.000000000 +0100
+++ new/debian/ligo-lw-bin.install 2019-09-11 09:15:20.000000000 +0200
@@ -0,0 +1 @@
+usr/bin/*
diff -Nur old/debian/python3-ligo-lw.install new/debian/python3-ligo-lw.install
--- old/debian/python3-ligo-lw.install 2017-03-23 07:14:16.000000000 +0100
+++ new/debian/python3-ligo-lw.install 2019-11-05 11:13:05.000000000 +0100
@@ -1,2 +1 @@
-#usr/bin/*
-usr/lib/python3*/dist-packages/ligo/lw
+usr/lib/python3*/*-packages/ligo/lw
diff -Nur old/debian/python-ligo-lw.install new/debian/python-ligo-lw.install
--- old/debian/python-ligo-lw.install 2017-03-23 07:14:16.000000000 +0100
+++ new/debian/python-ligo-lw.install 2019-11-05 11:12:57.000000000 +0100
@@ -1,2 +1 @@
-usr/bin/*
-usr/lib/python2*/dist-packages/ligo/lw
+usr/lib/python2*/*-packages/ligo/lw
diff -Nur old/debian/rules new/debian/rules
--- old/debian/rules 2019-02-14 18:44:58.000000000 +0100
+++ new/debian/rules 2019-11-05 11:37:39.000000000 +0100
@@ -1,13 +1,22 @@
#!/usr/bin/make -f
include /usr/share/dpkg/pkg-info.mk
-# Redirect Python 3 packages to a directory that we are going to ignore.
+# Redirect Python 3 packages to a directory that we are going to ignore/use.
# FIXME: in the future, we could put these in libexec or change their names.
export PYBUILD_INSTALL_ARGS_python3 := --install-scripts=/dont-install
%:
dh $@ --with=python2,python3 --buildsystem=pybuild
-# do not run tests at all.
+override_dh_auto_install:
+# installs to debian/tmp/
+ dh_auto_install
+# if python2, binaries go to debian/tmp/usr/bin - OK
+# if python3, binaries go to debian/tmp/dont-install (as above) - move them, but don't overwrite
+ mkdir -p $(CURDIR)/debian/tmp/usr/bin
+ mv -n $(CURDIR)/debian/tmp/dont-install/* $(CURDIR)/debian/tmp/usr/bin/ || true
+# now everything is in place for *.install files
+
+# attempt to run tests but ignore outcome
override_dh_auto_test:
- true
+ dh_auto_test || true
```https://git.ligo.org/kipp/python-ligo-lw/-/issues/10python-ligo-lw contains binaries (problem after phasing out python2)2021-03-03T09:06:55ZSteffen Grunewaldpython-ligo-lw contains binaries (problem after phasing out python2)Please see a month-old discussion over there: https://git.ligo.org/lscsoft/glue/issues/20 - is it true that @kipp.cannon is the only person taking care of packaging issues (not only for .deb, but also for .rpm)?Please see a month-old discussion over there: https://git.ligo.org/lscsoft/glue/issues/20 - is it true that @kipp.cannon is the only person taking care of packaging issues (not only for .deb, but also for .rpm)?https://git.ligo.org/kipp/python-ligo-lw/-/issues/9ligolw_print --ilwdchar-compat only shows first row of table2019-11-13T15:14:17ZLeo P. Singerligolw_print --ilwdchar-compat only shows first row of table`ligolw_print --ilwchar-compat` only prints the first row of a table. It prints blank lines for all of the other rows.
I am attaching an example file, [test.xml](/uploads/774877618cc3a569630283705fe9f35f/test.xml). Here's the command:
...`ligolw_print --ilwchar-compat` only prints the first row of a table. It prints blank lines for all of the other rows.
I am attaching an example file, [test.xml](/uploads/774877618cc3a569630283705fe9f35f/test.xml). Here's the command:
```
$ ligolw_print --ilwdchar-compat -t sim_inspiral -c simulation_id -c mass1 -c mass2 test.xml
sim_inspiral:simulation_id:0,1.3806658,1.4673961
... many more blank lines
```https://git.ligo.org/kipp/python-ligo-lw/-/issues/8missing conflict with python-pylal2019-04-10T08:42:08ZSteffen Grunewaldmissing conflict with python-pylalreported over there: https://git.ligo.org/sccb/requests/issues/79#note_79459reported over there: https://git.ligo.org/sccb/requests/issues/79#note_79459https://git.ligo.org/kipp/python-ligo-lw/-/issues/7ligolw_sqlite_test fails on python3.72019-11-13T12:07:47ZDuncan Macleodduncan.macleod@ligo.orgligolw_sqlite_test fails on python3.7The `ligolw_sqlite_test` test fails on python3.7 as follows:
```
$ make -C test ligolw_sqlite_test
=== start ligolw_sqlite_test ===>
sh ligolw_sqlite_test.sh && { echo "Pass" ; true ; } || { echo "Fail" ; false ; }
ligolw_sqlite test 1...The `ligolw_sqlite_test` test fails on python3.7 as follows:
```
$ make -C test ligolw_sqlite_test
=== start ligolw_sqlite_test ===>
sh ligolw_sqlite_test.sh && { echo "Pass" ; true ; } || { echo "Fail" ; false ; }
ligolw_sqlite test 1: detect missing input
--------------------------------------------------------------------
removed 'ligolw_sqlite_test.sqlite'
'ligolw_sqlite_test.sqlite': no such file
ligolw_sqlite test 1: success
ligolw_sqlite detected missing input and did not create an empty database
ligolw_sqlite test 2: merge .xml.gz files and compare to ligolw_add
--------------------------------------------------------------------
removed 'ligolw_sqlite_test_ref.xml'
removed 'ligolw_sqlite_test_output.xml'
1/2:reading 'ligolw_sqlite_test_input.xml.gz' ...
2/2:reading 'ligolw_sqlite_test_input.xml.gz' ...
reassigning row IDs▐███████████████████████████████████████████████████████████████████▌100.0%
merging elements ...
writing 'ligolw_sqlite_test_ref.xml' ...
1/2: reading 'file:///Users/duncan/git/python-ligo-lw-fork/test/ligolw_sqlite_test_input.xml.gz' ...
updating IDs: 100%
2/2: reading 'ligolw_sqlite_test_input.xml.gz' ...
updating IDs: 100%
indexing process_params table ...
indexing search_summary table ...
indexing coinc_event table ...
indexing coinc_event_map table ...
indexing coinc_definer table ...
writing 'ligolw_sqlite_test_output.xml' ...
done.
ligolw_sqlite_test_ref.xml ligolw_sqlite_test_output.xml differ: char 184, line 5
Fail
make: *** [ligolw_sqlite_test] Error 1
```
The fundamental problem is that the column tags are written differently:
```
$ head test/ligolw_sqlite_test_ref.xml test/ligolw_sqlite_test_output.xml
==> test/ligolw_sqlite_test_ref.xml <==
<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE LIGO_LW SYSTEM "http://ldas-sw.ligo.caltech.edu/doc/ligolwAPI/html/ligolw_dtd.txt">
<LIGO_LW>
<Table Name="process:table">
<Column Type="lstring" Name="program"/>
<Column Type="lstring" Name="version"/>
<Column Type="lstring" Name="cvs_repository"/>
<Column Type="int_4s" Name="cvs_entry_time"/>
<Column Type="lstring" Name="comment"/>
<Column Type="int_4s" Name="is_online"/>
==> test/ligolw_sqlite_test_output.xml <==
<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE LIGO_LW SYSTEM "http://ldas-sw.ligo.caltech.edu/doc/ligolwAPI/html/ligolw_dtd.txt">
<LIGO_LW>
<Table Name="process:table">
<Column Name="program" Type="lstring"/>
<Column Name="version" Type="lstring"/>
<Column Name="cvs_repository" Type="lstring"/>
<Column Name="cvs_entry_time" Type="int_4s"/>
<Column Name="comment" Type="lstring"/>
<Column Name="is_online" Type="int_4s"/>
```
which is presumably something to do with `dict` key ordering. The same failure occurs on python3.6, but not on python2.7.https://git.ligo.org/kipp/python-ligo-lw/-/issues/6lw.utils.load_url doesn't work for non-local files2019-02-17T17:10:04ZCody Messicklw.utils.load_url doesn't work for non-local filesI came across the traceback below when running gstlal_inspiral_marginalize_likelihoods_online from master at CIT, which downloads a rankingstat file from the running inspiral jobs through a wget request, set up on the inspiral side with ...I came across the traceback below when running gstlal_inspiral_marginalize_likelihoods_online from master at CIT, which downloads a rankingstat file from the running inspiral jobs through a wget request, set up on the inspiral side with bottle route. I'm using master of gstlal and python-ligo-lw.
```
Traceback (most recent call last):
File "/home/gstlalcbc/observing/3/online/code_test_itacac_master_181213/opt/bin/gstlal_inspiral_calc_rank_pdfs", line 102, in <module>
rankingstat = far.marginalize_pdf_urls(urls, "RankingStat", verbose = options.verbose)
File "/home/gstlalcbc/observing/3/online/code_test_itacac_master_181213/opt/lib64/python2.7/site-packages/gstlal/far.py", line 1055, in marginalize_pdf_urls
xmldoc = ligolw_utils.load_url(url, verbose = verbose, contenthandler = RankingStat.LIGOLWContentHandler)
File "/home/gstlalcbc/observing/3/online/code_test_itacac_master_181213/opt/lib64/python2.7/site-packages/ligo/lw/utils/__init__.py", line 408, in load_url
with fileobj() as fileobj:
AttributeError: addinfourl instance has no attribute '__exit__'
```
Doing some digging, it looks like the object returned by urllib.request.urlopen() on line 407 [1] doesn't have an ```__exit__``` function, but it does appear to have a ```__close__``` function.
[1] https://git.ligo.org/kipp.cannon/python-ligo-lw/blob/master/ligo/lw/utils/__init__.py#L407
I looked in the urllib.request documentation to see if they provide a function that has an ```__exit__``` function, but I didn't see anything obvious. I hacked in a temporary work around for myself, which I've put below in case anyone else wants to use it, but it's probably not the correct way to fix this.
```
diff --git a/ligo/lw/utils/__init__.py b/ligo/lw/utils/__init__.py
index a482c1db25..4855ca39cf 100644
--- a/ligo/lw/utils/__init__.py
+++ b/ligo/lw/utils/__init__.py
@@ -403,10 +403,12 @@ def load_url(url, verbose = False, **kwargs):
scheme, host, path = urllib.parse.urlparse(url)[:3]
if scheme.lower() in ("", "file") and host.lower() in ("", "localhost"):
fileobj = lambda: open(path, "rb")
+ with fileobj() as fileobj:
+ xmldoc = load_fileobj(fileobj, **kwargs)
else:
- fileobj = lambda: urllib.request.urlopen(url)
- with fileobj() as fileobj:
+ fileobj = urllib.request.urlopen(url)
xmldoc = load_fileobj(fileobj, **kwargs)
+ fileobj.close()
return xmldoc
```Kipp CannonKipp Cannonhttps://git.ligo.org/kipp/python-ligo-lw/-/issues/5python-ligo-lw.spec file built from spec.in contains illegal characters2019-02-17T16:36:25ZDuncan Macleodduncan.macleod@ligo.orgpython-ligo-lw.spec file built from spec.in contains illegal charactersThe auto-version replacement in `setup.py` ends up creating a spec file with illegal characters:
```shell
$ python setup.py sdist
...
$ rpmbuild -tb dist/python-ligo-lw-1.4.0.tar.gz
error: line 3: Illegal char '"' in: Version: "1.4.0"
`...The auto-version replacement in `setup.py` ends up creating a spec file with illegal characters:
```shell
$ python setup.py sdist
...
$ rpmbuild -tb dist/python-ligo-lw-1.4.0.tar.gz
error: line 3: Illegal char '"' in: Version: "1.4.0"
```
The quotes around the version are not allowed in the spec it seems.https://git.ligo.org/kipp/python-ligo-lw/-/issues/4setup.py install doesn't work for python-ligo-lw any more2019-02-17T16:37:54ZDuncan Macleodduncan.macleod@ligo.orgsetup.py install doesn't work for python-ligo-lw any moreSee the following example:
```shell
$ # create a virtualenv that picks up the dependencies from the system
$ python -m virtualenv venv --system-site-packages
$ # install python-ligo-lw into it
$ ./venv/bin/python setup.py install
# list...See the following example:
```shell
$ # create a virtualenv that picks up the dependencies from the system
$ python -m virtualenv venv --system-site-packages
$ # install python-ligo-lw into it
$ ./venv/bin/python setup.py install
# list files
$ find venv/lib64/python*/site-packages/ligo
venv/lib64/python2.7/site-packages/ligo
venv/lib64/python2.7/site-packages/ligo/__init__.py
venv/lib64/python2.7/site-packages/ligo/lw
venv/lib64/python2.7/site-packages/ligo/lw/_ilwd.so
venv/lib64/python2.7/site-packages/ligo/lw/tokenizer.so
venv/lib64/python2.7/site-packages/ligo/__init__.pyc
```
This is because the `packages` keyword in the call to `distutils.setup` is incomplete. Since this package uses the `ligo` namespace_package, the correct value for this is
```diff
diff --git a/setup.py b/setup.py
index b17897c97b..82c479fa52 100644
--- a/setup.py
+++ b/setup.py
@@ -43,7 +43,8 @@ setup(
"ligo",
],
packages = [
- "ligo",
+ "ligo.lw",
+ "ligo.lw.utils",
],
ext_modules = [
Extension(
```
(as it was before 49a05805a03c3ea32bda19cc5cff6e1d1c5ddc71) I can post this as a merge request if that would be helpful.
`setuptools` also has a helpful `find_packages` function that will populate the `packages` entry automatically, which might be easier than manually listing things.