python-ligo-lw issueshttps://git.ligo.org/kipp/python-ligo-lw/-/issues2024-03-28T09:33:20Zhttps://git.ligo.org/kipp/python-ligo-lw/-/issues/29Build fails with Python 3.12 and clang 16 on macOS2024-03-28T09:33:20ZDuncan Macleodduncan.macleod@ligo.orgBuild fails with Python 3.12 and clang 16 on macOSThe build of `python-ligo-lw` fails when using Python 3.12.2 and clang 16.0.1:
```console
$ sw_vers
ProductName: macOS
ProductVersion: 11.7.10
BuildVersion: 20G1427
$ git describe --tags
1.8.0-83-gcfaddad282
$ python3 --version
Pyt...The build of `python-ligo-lw` fails when using Python 3.12.2 and clang 16.0.1:
```console
$ sw_vers
ProductName: macOS
ProductVersion: 11.7.10
BuildVersion: 20G1427
$ git describe --tags
1.8.0-83-gcfaddad282
$ python3 --version
Python 3.12.2
$ clang --version
clang version 16.0.6
Target: x86_64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Users/duncanmacleod/opt/mambaforge/envs/py312/bin
$ python3 setup.py build
/Users/duncanmacleod/opt/mambaforge/envs/py312/lib/python3.12/site-packages/setuptools/dist.py:700: SetuptoolsDeprecationWarning: The namespace_packages parameter is deprecated.
!!
********************************************************************************
Please replace its usage with implicit namespaces (PEP 420).
See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages for details.
********************************************************************************
!!
ep.load()(self, ep.name, value)
running build
running build_py
creating build
creating build/lib.macosx-10.9-x86_64-cpython-312
creating build/lib.macosx-10.9-x86_64-cpython-312/ligo
creating build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw
copying ligo/lw/ligolw.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw
copying ligo/lw/lsctables.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw
copying ligo/lw/__init__.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw
copying ligo/lw/param.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw
copying ligo/lw/types.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw
copying ligo/lw/dbtables.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw
copying ligo/lw/table.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw
copying ligo/lw/array.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw
creating build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw/utils
copying ligo/lw/utils/ligolw_add.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw/utils
copying ligo/lw/utils/search_summary.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw/utils
copying ligo/lw/utils/ilwd.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw/utils
copying ligo/lw/utils/segments.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw/utils
copying ligo/lw/utils/__init__.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw/utils
copying ligo/lw/utils/ligolw_sqlite.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw/utils
copying ligo/lw/utils/coincs.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw/utils
copying ligo/lw/utils/process.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw/utils
copying ligo/lw/utils/time_slide.py -> build/lib.macosx-10.9-x86_64-cpython-312/ligo/lw/utils
running build_ext
building 'ligo.lw.tokenizer' extension
creating build/temp.macosx-10.9-x86_64-cpython-312
creating build/temp.macosx-10.9-x86_64-cpython-312/ligo
creating build/temp.macosx-10.9-x86_64-cpython-312/ligo/lw
x86_64-apple-darwin13.4.0-clang -fno-strict-overflow -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /Users/duncanmacleod/opt/mambaforge/envs/py312/include -fPIC -O2 -isystem /Users/duncanmacleod/opt/mambaforge/envs/py312/include -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /Users/duncanmacleod/opt/mambaforge/envs/py312/include -D_FORTIFY_SOURCE=2 -isystem /Users/duncanmacleod/opt/mambaforge/envs/py312/include -Iligo/lw -I/Users/duncanmacleod/opt/mambaforge/envs/py312/include/python3.12 -c ligo/lw/tokenizer.RowBuilder.c -o build/temp.macosx-10.9-x86_64-cpython-312/ligo/lw/tokenizer.RowBuilder.o
ligo/lw/tokenizer.RowBuilder.c:231:2: warning: suggest braces around initialization of subobject [-Wmissing-braces]
PyObject_HEAD_INIT(NULL)
^~~~~~~~~~~~~~~~~~~~~~~~
/Users/duncanmacleod/opt/mambaforge/envs/py312/include/python3.12/object.h:141:11: note: expanded from macro 'PyObject_HEAD_INIT'
{ 1 }, \
^
ligo/lw/tokenizer.RowBuilder.c:231:2: error: incompatible pointer to integer conversion initializing 'Py_ssize_t' (aka 'long') with an expression of type 'void *' [-Wint-conversion]
PyObject_HEAD_INIT(NULL)
^~~~~~~~~~~~~~~~~~~~~~~~
/Users/duncanmacleod/opt/mambaforge/envs/py312/include/python3.12/object.h:142:9: note: expanded from macro 'PyObject_HEAD_INIT'
(type) \
^~~~~~
1 warning and 1 error generated.
error: command '/Users/duncanmacleod/opt/mambaforge/envs/py312/bin/x86_64-apple-darwin13.4.0-clang' failed with exit code 1
```
I think the `PyObject_HEAD_INIT(NULL)` needs to be modified to be `PyVarObject_HEAD_INIT(NULL, 0)`.https://git.ligo.org/kipp/python-ligo-lw/-/issues/282.0.0 release plans?2024-02-07T08:23:05ZSteffen Grunewald2.0.0 release plans?I just discovered the (11 months old) update of `debian/changelog` bumping the version to 2.0.0. Are there any plans to release this version, as a tarball and/or via PyPI?I just discovered the (11 months old) update of `debian/changelog` bumping the version to 2.0.0. Are there any plans to release this version, as a tarball and/or via PyPI?https://git.ligo.org/kipp/python-ligo-lw/-/issues/27update python interpreter2023-03-17T15:28:14ZLeo Tsukadaupdate python interpreterI tried running `ligolw_run_sqlite` and [this line](https://git.ligo.org/kipp.cannon/python-ligo-lw/-/blob/master/bin/ligolw_run_sqlite#L1) needs to be updated with `python3` as `python` interpreter is no longer available in cit. All oth...I tried running `ligolw_run_sqlite` and [this line](https://git.ligo.org/kipp.cannon/python-ligo-lw/-/blob/master/bin/ligolw_run_sqlite#L1) needs to be updated with `python3` as `python` interpreter is no longer available in cit. All other programs in `bin` needs to have this change.https://git.ligo.org/kipp/python-ligo-lw/-/issues/26API change / documentation request for changed use of process_params2023-03-03T17:21:01ZRichard O'ShaughnessyAPI change / documentation request for changed use of process_paramsOne of the RIFT [CI tests](https://git.ligo.org/rapidpe-rift/rift/-/blob/master/.travis/test-run.sh) is failing with the latest version of python-ligo-lw due to what seems to be an API/supported XML change.
```
./command-single.sh
<sni...One of the RIFT [CI tests](https://git.ligo.org/rapidpe-rift/rift/-/blob/master/.travis/test-run.sh) is failing with the latest version of python-ligo-lw due to what seems to be an API/supported XML change.
```
./command-single.sh
<snip standard messages>
====Loading injection XML: mdc.xml.gz 0 =======
Traceback (most recent call last):
File "/home/serguei.ossokine/.conda/envs/gwsignal_for_RIFT/bin/integrate_likelihood_extrinsic_batchmode", line 418, in <module>
P_list = lalsimutils.xml_to_ChooseWaveformParams_array(str(opts.sim_xml))
File "/home/serguei.ossokine/.conda/envs/gwsignal_for_RIFT/lib/python3.9/site-packages/RIFT/lalsimutils.py", line 1795, in xml_to_ChooseWaveformParams_array
[Ps[i-minrow].copy_lsctables_sim_inspiral(sim_insp[i]) for i in rng]
File "/home/serguei.ossokine/.conda/envs/gwsignal_for_RIFT/lib/python3.9/site-packages/RIFT/lalsimutils.py", line 1795, in <listcomp>
[Ps[i-minrow].copy_lsctables_sim_inspiral(sim_insp[i]) for i in rng]
File "/home/serguei.ossokine/.conda/envs/gwsignal_for_RIFT/lib/python3.9/site-packages/RIFT/lalsimutils.py", line 1724, in copy_lsctables_sim_inspiral
setattr(swigrow, simattr,0)
AttributeError: 'lalmetaio.SimInspiralTable' object has no attribute 'process:process_id'
```https://git.ligo.org/kipp/python-ligo-lw/-/issues/25Missing lalsuite dependency.2022-11-29T09:06:00ZPierre ChanialMissing lalsuite dependency.The python-ligo-lw dependency on lalsuite is not declared. And as a consequence, this package cannot be installed from PyPI with all its dependencies.The python-ligo-lw dependency on lalsuite is not declared. And as a consequence, this package cannot be installed from PyPI with all its dependencies.https://git.ligo.org/kipp/python-ligo-lw/-/issues/24ligolw_segments raises AttributeError due to removal of set_process_end_time2022-08-16T06:25:25ZDuncan Macleodduncan.macleod@ligo.orgligolw_segments raises AttributeError due to removal of set_process_end_timeAs of 1.8.0, `ligolw_segments` raises an `AttributeError` due to 40a1c07d8db89c094d71db35aa378ba222c89412:
```console
$ echo "1 2
> 3 4
> 5 7
> " | ligolw_segments
Traceback (most recent call last):
File "/home/duncan/opt/mambaforge/e...As of 1.8.0, `ligolw_segments` raises an `AttributeError` due to 40a1c07d8db89c094d71db35aa378ba222c89412:
```console
$ echo "1 2
> 3 4
> 5 7
> " | ligolw_segments
Traceback (most recent call last):
File "/home/duncan/opt/mambaforge/envs/py310/bin/ligolw_segments", line 212, in <module>
ligolw_process.set_process_end_time(process)
AttributeError: module 'ligo.lw.utils.process' has no attribute 'set_process_end_time'
```https://git.ligo.org/kipp/python-ligo-lw/-/issues/23Add optional timeout for ligo.lw.utils.load_url2021-12-08T10:06:06ZPatrick GodwinAdd optional timeout for ligo.lw.utils.load_urlIf passing in a URL which would be parsed by `urlopen`, this can potentially hang forever without being able to specify a timeout to it (https://git.ligo.org/kipp.cannon/python-ligo-lw/-/blob/master/ligo/lw/utils/__init__.py#L439).
This...If passing in a URL which would be parsed by `urlopen`, this can potentially hang forever without being able to specify a timeout to it (https://git.ligo.org/kipp.cannon/python-ligo-lw/-/blob/master/ligo/lw/utils/__init__.py#L439).
This failure mode was seen in an online gstlal job (`gstlal_inspiral_marginalize_likelihoods_online`) when pinging one of the bottle routes, where the server was taking in requests but never sending a response. Ultimately I tracked the polling code to the line linked above.https://git.ligo.org/kipp/python-ligo-lw/-/issues/22python-ligo-lw incompatible with python 3.10: PyLong_FromUnicode has been rem...2022-07-15T13:02:45ZDuncan Macleodduncan.macleod@ligo.orgpython-ligo-lw incompatible with python 3.10: PyLong_FromUnicode has been removedpython-ligo-lw cannot be used with Python 3.10 as follows:
```console
$ git status
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
$ git rev-parse HEAD
40a1c07d8db89c094d71db35aa37...python-ligo-lw cannot be used with Python 3.10 as follows:
```console
$ git status
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
$ git rev-parse HEAD
40a1c07d8db89c094d71db35aa378ba222c89412
$ python3 --version
Python 3.10.0
$ CFLAGS="${CFLAGS} -Wfatal-errors -Werror=implicit-function-declaration" python3 setup.py build_ext --inplace
/home/duncan/git/python-ligo-lw/setup.py:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils.core import setup, Extension
/home/duncan/opt/mambaforge/envs/py310/lib/python3.10/distutils/dist.py:274: UserWarning: Unknown distribution option: 'namespace_packages'
warnings.warn(msg)
/home/duncan/opt/mambaforge/envs/py310/lib/python3.10/distutils/dist.py:274: UserWarning: Unknown distribution option: 'install_requires'
warnings.warn(msg)
/home/duncan/opt/mambaforge/envs/py310/lib/python3.10/distutils/command/build_ext.py:13: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
from distutils.sysconfig import customize_compiler, get_python_version
running build_ext
building 'ligo.lw.tokenizer' extension
gcc -pthread -B /home/duncan/opt/mambaforge/envs/py310/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/duncan/opt/mambaforge/envs/py310/include -fPIC -O2 -isystem /home/duncan/opt/mambaforge/envs/py310/include -Wfatal-errors -Werror=implicit-function-declaration -fPIC -Iligo/lw -I/home/duncan/opt/mambaforge/envs/py310/include/python3.10 -c ligo/lw/tokenizer.RowBuilder.c -o build/temp.linux-x86_64-3.10/ligo/lw/tokenizer.RowBuilder.o
ligo/lw/tokenizer.RowBuilder.c:230:39: warning: missing braces around initializer [-Wmissing-braces]
PyTypeObject ligolw_RowBuilder_Type = {
^
ligo/lw/tokenizer.RowBuilder.c:230:39: warning: missing braces around initializer [-Wmissing-braces]
gcc -pthread -B /home/duncan/opt/mambaforge/envs/py310/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/duncan/opt/mambaforge/envs/py310/include -fPIC -O2 -isystem /home/duncan/opt/mambaforge/envs/py310/include -Wfatal-errors -Werror=implicit-function-declaration -fPIC -Iligo/lw -I/home/duncan/opt/mambaforge/envs/py310/include/python3.10 -c ligo/lw/tokenizer.RowDumper.c -o build/temp.linux-x86_64-3.10/ligo/lw/tokenizer.RowDumper.o
ligo/lw/tokenizer.RowDumper.c: In function ‘__init__’:
ligo/lw/tokenizer.RowDumper.c:103:3: warning: ‘PyUnicode_FromUnicode’ is deprecated [-Wdeprecated-declarations]
rowdumper->delimiter = PyUnicode_FromUnicode(&default_delimiter, 1);
^~~~~~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.RowDumper.c:29:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:551:42: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_FromUnicode(
^~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.RowDumper.c: In function ‘next’:
ligo/lw/tokenizer.RowDumper.c:223:4: warning: ‘PyUnicode_FromUnicode’ is deprecated [-Wdeprecated-declarations]
token = PyUnicode_FromUnicode(NULL, 0); /* u"" */
^~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.RowDumper.c:29:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:551:42: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_FromUnicode(
^~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.RowDumper.c: At top level:
ligo/lw/tokenizer.RowDumper.c:280:38: warning: missing braces around initializer [-Wmissing-braces]
PyTypeObject ligolw_RowDumper_Type = {
^
ligo/lw/tokenizer.RowDumper.c:280:38: warning: missing braces around initializer [-Wmissing-braces]
gcc -pthread -B /home/duncan/opt/mambaforge/envs/py310/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/duncan/opt/mambaforge/envs/py310/include -fPIC -O2 -isystem /home/duncan/opt/mambaforge/envs/py310/include -Wfatal-errors -Werror=implicit-function-declaration -fPIC -Iligo/lw -I/home/duncan/opt/mambaforge/envs/py310/include/python3.10 -c ligo/lw/tokenizer.Tokenizer.c -o build/temp.linux-x86_64-3.10/ligo/lw/tokenizer.Tokenizer.o
ligo/lw/tokenizer.Tokenizer.c: In function ‘add_to_data’:
ligo/lw/tokenizer.Tokenizer.c:96:2: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
Py_ssize_t n = PyUnicode_GET_SIZE(unicode);
^~~~~~~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
^~~~~~~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:96:2: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
Py_ssize_t n = PyUnicode_GET_SIZE(unicode);
^~~~~~~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
^~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:96:2: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
Py_ssize_t n = PyUnicode_GET_SIZE(unicode);
^~~~~~~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
^~~~~~~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:138:3: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
memcpy(tokenizer->length, PyUnicode_AsUnicode(unicode), n * sizeof(*tokenizer->length));
^~~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
^~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c: In function ‘parse_error’:
ligo/lw/tokenizer.Tokenizer.c:200:2: warning: ‘PyUnicode_FromUnicode’ is deprecated [-Wdeprecated-declarations]
buffer_str = PyUnicode_FromUnicode(buffer, buffer_length);
^~~~~~~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:551:42: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_FromUnicode(
^~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:201:2: warning: ‘PyUnicode_FromUnicode’ is deprecated [-Wdeprecated-declarations]
pos_str = PyUnicode_FromUnicode(pos, 1);
^~~~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:551:42: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_FromUnicode(
^~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c: In function ‘__init__’:
ligo/lw/tokenizer.Tokenizer.c:514:2: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
if(PyUnicode_GET_SIZE(arg) != 1) {
^~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
^~~~~~~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:514:2: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
if(PyUnicode_GET_SIZE(arg) != 1) {
^~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
^~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:514:2: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
if(PyUnicode_GET_SIZE(arg) != 1) {
^~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
^~~~~~~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:519:2: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
tokenizer->delimiter = *PyUnicode_AS_UNICODE(arg);
^~~~~~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
^~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c: In function ‘next’:
ligo/lw/tokenizer.Tokenizer.c:584:3: warning: ‘PyUnicode_EncodeDecimal’ is deprecated [-Wdeprecated-declarations]
if(PyUnicode_EncodeDecimal(start, end - start, ascii_buffer, NULL))
^~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:952:36: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(int) PyUnicode_EncodeDecimal(
^~~~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:598:3: warning: ‘PyUnicode_FromUnicode’ is deprecated [-Wdeprecated-declarations]
token = PyUnicode_FromUnicode(start, end - start);
^~~~~
In file included from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/unicodeobject.h:1046,
from /home/duncan/opt/mambaforge/envs/py310/include/python3.10/Python.h:96,
from ligo/lw/tokenizer.Tokenizer.c:32:
/home/duncan/opt/mambaforge/envs/py310/include/python3.10/cpython/unicodeobject.h:551:42: note: declared here
Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_FromUnicode(
^~~~~~~~~~~~~~~~~~~~~
ligo/lw/tokenizer.Tokenizer.c:607:11: error: implicit declaration of function ‘PyLong_FromUnicode’; did you mean ‘PyLong_FromPid’? [-Werror=implicit-function-declaration]
token = PyLong_FromUnicode(start, end - start, 0);
^~~~~~~~~~~~~~~~~~
PyLong_FromPid
compilation terminated due to -Wfatal-errors.
cc1: some warnings being treated as errors
error: command '/usr/bin/gcc' failed with exit code 1
```
For details of the Python 3.10 change, see [here](https://docs.python.org/3/whatsnew/3.10.html#id4). @kipp.cannon, are you able to provide a patch for this?https://git.ligo.org/kipp/python-ligo-lw/-/issues/21distutils is deprecated as of Python 3.102023-03-17T16:09:40ZDuncan Macleodduncan.macleod@ligo.orgdistutils is deprecated as of Python 3.10The `distutils` module is deprecated as of Python 3.10 and will be removed in Python 3.12. This project should proabably switch to `setuptools` as a drop-in replacement. For full details, see <https://docs.python.org/3/whatsnew/3.10.html...The `distutils` module is deprecated as of Python 3.10 and will be removed in Python 3.12. This project should proabably switch to `setuptools` as a drop-in replacement. For full details, see <https://docs.python.org/3/whatsnew/3.10.html#distutils-deprecated>.https://git.ligo.org/kipp/python-ligo-lw/-/issues/20Migrate `lalapps_run_sqlite` to python-ligo-lw?2021-12-08T11:12:31ZPatrick GodwinMigrate `lalapps_run_sqlite` to python-ligo-lw?While this is located in `lalapps` (https://git.ligo.org/lscsoft/lalsuite/-/blob/master/lalapps/src/inspiral/lalapps_run_sqlite.py), this program doesn't rely on lalsuite at all. It'd be nice to not need to have lalsuite installed to exe...While this is located in `lalapps` (https://git.ligo.org/lscsoft/lalsuite/-/blob/master/lalapps/src/inspiral/lalapps_run_sqlite.py), this program doesn't rely on lalsuite at all. It'd be nice to not need to have lalsuite installed to execute SQL on LIGOLW files with this utility.
What do you think, @kipp.cannon?https://git.ligo.org/kipp/python-ligo-lw/-/issues/19API change in ligo.lw.utils.process.register_to_xmldoc, intended?2021-03-08T08:59:52ZPatrick GodwinAPI change in ligo.lw.utils.process.register_to_xmldoc, intended?Running a gstlal job using ligo-lw 1.7.1 is throwing me an error in calling `register_to_xmldoc`:
```
Traceback (most recent call last):
File "/home/patrick.godwin/local/gcc_master_210203/opt/bin/gstlal_inspiral_create_prior_diststats...Running a gstlal job using ligo-lw 1.7.1 is throwing me an error in calling `register_to_xmldoc`:
```
Traceback (most recent call last):
File "/home/patrick.godwin/local/gcc_master_210203/opt/bin/gstlal_inspiral_create_prior_diststats", line 166, in <module>
process = ligolw_process.register_to_xmldoc(xmldoc, u"gstlal_inspiral_create_prior_diststats", paramdict = process_params, ifos = options.instrument)
File "/usr/lib64/python3.6/site-packages/ligo/lw/utils/process.py", line 109, in register_to_xmldoc
process = proctable.RowType.initialized(program = program, process_id = proctable.get_next_id(), **kwargs)
TypeError: initialized() got an unexpected keyword argument 'ifos'
```
* Before: https://git.ligo.org/kipp.cannon/python-ligo-lw/-/blob/5b8eaa7eaabffbe72fab2c46f696811f473a4a4f/ligo/lw/utils/process.py#L82
* After: https://git.ligo.org/kipp.cannon/python-ligo-lw/-/blob/5faced2e682833caa0165f2a0bbcc7eaea99c4a5/ligo/lw/lsctables.py#L483
It looks like the API was changed during the `ligo.lw.utils.process` rework, so that it now takes `instruments` rather than the `ifos` kwarg:
I wanted to double check that this was intended since this shows up in a number of places in gstlal and may cause some issues unintended issues when it's deployed on SL7. I didn't see this in the changelogs but could have missed it.https://git.ligo.org/kipp/python-ligo-lw/-/issues/18ligolw_segments using python2 'file' keyword on python32021-02-16T09:04:58ZDuncan Macleodduncan.macleod@ligo.orgligolw_segments using python2 'file' keyword on python3`ligolw_segments` fails with a `NameError` on python3 because of the use of the python2 `file` keyword
```console
+ ligolw_segments -o python-ligo-lw-segments.xml.gz --insert-from-segwizard=X1=python-ligo-lw-segments.txt
Traceback (most...`ligolw_segments` fails with a `NameError` on python3 because of the use of the python2 `file` keyword
```console
+ ligolw_segments -o python-ligo-lw-segments.xml.gz --insert-from-segwizard=X1=python-ligo-lw-segments.txt
Traceback (most recent call last):
File "/home/duncan/opt/miniconda3/envs/py39/bin/ligolw_segments", line 168, in <module>
segments_tables.insert_from_segwizard(file(filename), set([instrument]), options.name, version = options.segments_version, comment = options.comment)
NameError: name 'file' is not defined
```https://git.ligo.org/kipp/python-ligo-lw/-/issues/17ValueError: invalid Name 'Sn(f)' when writing lal FrequencySeries2021-02-06T07:11:02ZPatrick GodwinValueError: invalid Name 'Sn(f)' when writing lal FrequencySeriesWhile this error originates from lalsuite/gstlal, it appears that this issue is a regression in behavior in ligo-lw in not allowing parentheses in names. Let me know if it's more appropriate for this issue to be moved to a downstream lib...While this error originates from lalsuite/gstlal, it appears that this issue is a regression in behavior in ligo-lw in not allowing parentheses in names. Let me know if it's more appropriate for this issue to be moved to a downstream library.
This was also found in `python-ligo-lw 1.6.0` so feel free to close if this has already been fixed.
Example command and traceback:
```
$ gstlal_psd_xml_from_asd_txt --instrument H1 --output H1_psd.xml.gz gstlal/share/early_aligo_asd.txt
Traceback (most recent call last):
File "/usr/bin/gstlal_psd_xml_from_asd_txt", line 88, in <module>
reference_psd.write_psd(options.output, {options.instrument: psdseries})
File "/usr/lib64/python3.6/site-packages/gstlal/reference_psd.py", line 207, in write_psd
ligolw_utils.write_filename(lal.series.make_psd_xmldoc(psddict), filename, gz = (filename or "stdout").endswith(".gz"), verbose = verbose, trap_signals = trap_signals)
File "/usr/lib64/python3.6/site-packages/lal/series.py", line 215, in make_psd_xmldoc
fs = lw.appendChild(build_REAL8FrequencySeries(psd))
File "/usr/lib64/python3.6/site-packages/lal/series.py", line 128, in build_REAL8FrequencySeries
return _build_series(series, (u"Frequency,Real", u"Frequency"), comment, 'deltaF', 's^-1')
File "/usr/lib64/python3.6/site-packages/lal/series.py", line 63, in _build_series
a = ligolw_array.Array.build(series.name, data, dim_names=dim_names)
File "/usr/lib64/python3.6/site-packages/ligo/lw/array.py", line 206, in build
elem.Name = name
File "/usr/lib64/python3.6/site-packages/ligo/lw/ligolw.py", line 163, in setter
self.setAttribute(name, enc(value))
File "/usr/lib64/python3.6/site-packages/ligo/lw/ligolw.py", line 472, in enc
raise ValueError("invalid Name '%s'" % name)
ValueError: invalid Name 'Sn(f)'
```https://git.ligo.org/kipp/python-ligo-lw/-/issues/16merge request 23 has broken install2023-03-16T21:07:02ZKipp Cannonmerge request 23 has broken install@leo-singer Since merging https://git.ligo.org/kipp.cannon/python-ligo-lw/-/merge_requests/23, the unit tests fail with
```
Traceback (most recent call last):
File "/home/kipp/local/bin/ligolw_print", line 4, in <module>
__import...@leo-singer Since merging https://git.ligo.org/kipp.cannon/python-ligo-lw/-/merge_requests/23, the unit tests fail with
```
Traceback (most recent call last):
File "/home/kipp/local/bin/ligolw_print", line 4, in <module>
__import__('pkg_resources').run_script('python-ligo-lw==1.7.0', 'ligolw_print')
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 665, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1451, in run_script
raise ResolutionError(
pkg_resources.ResolutionError: Script 'scripts/ligolw_print' not found in metadata at None
```
This does not correspond to line 4 of that script, but when I look inside $(which ligolw_print) it's nearly empty, and this error message does correspond to what is there. What happened to the programs in bin? Why are they empty, why does whatever python has done to them not work?https://git.ligo.org/kipp/python-ligo-lw/-/issues/15CoincInspiral.set_ifos() method compatibility2020-08-22T00:46:58ZTito Dal CantonCoincInspiral.set_ifos() method compatibilityIn glue.ligolw, with a `CoincInspiral` instance one could do
```
coinc_inspiral_instance.set_ifos(ifos)
```
It looks like the same method is not available in python-ligo-lw. Adding it would simplify porting code from glue to python-ligo-lw.In glue.ligolw, with a `CoincInspiral` instance one could do
```
coinc_inspiral_instance.set_ifos(ifos)
```
It looks like the same method is not available in python-ligo-lw. Adding it would simplify porting code from glue to python-ligo-lw.https://git.ligo.org/kipp/python-ligo-lw/-/issues/14tmp_path kwarg in dbtables.get_connection_filename not working for python 32021-03-05T02:15:06ZPatrick Godwintmp_path kwarg in dbtables.get_connection_filename not working for python 3This appears to be leveraging `temporary_file.unlink()`, which was an undocumented feature in python 2 and doesn't exist in python 3. Some more information in https://stackoverflow.com/a/28067461.
A traceback from a gstlal job failing w...This appears to be leveraging `temporary_file.unlink()`, which was an undocumented feature in python 2 and doesn't exist in python 3. Some more information in https://stackoverflow.com/a/28067461.
A traceback from a gstlal job failing when using this functionality:
```
Traceback (most recent call last):
File "/usr/bin/gstlal_inspiral_plotsummary", line 1814, in <module>
working_filename = dbtables.get_connection_filename(filename, tmp_path = options.tmp_space, verbose = options.verbose)
File "/usr/lib64/python3.6/site-packages/ligo/lw/dbtables.py", line 253, in get_connection_filename
target = mktmp(tmp_path, suffix = ".".join(os.path.split(filename)[-1].split(".")[1:]), verbose = verbose)
File "/usr/lib64/python3.6/site-packages/ligo/lw/dbtables.py", line 196, in mktmp
def new_unlink(self, orig_unlink = temporary_file.unlink):
File "/usr/lib64/python3.6/tempfile.py", line 480, in __getattr__
a = getattr(file, name)
AttributeError: '_io.BufferedRandom' object has no attribute 'unlink'
```https://git.ligo.org/kipp/python-ligo-lw/-/issues/13ligolw segments CLI utils fail on ImportErrors2020-02-21T04:52:48ZPatrick Godwinligolw segments CLI utils fail on ImportErrorsAs an example,
```
[patrick.godwin@ldas-pcdev5 ~] $ ligolw_segment_diff --help
Traceback (most recent call last):
File "/usr/bin/ligolw_segment_diff", line 44, in <module>
from glue.ligolw import utils
ModuleNotFoundError: No modu...As an example,
```
[patrick.godwin@ldas-pcdev5 ~] $ ligolw_segment_diff --help
Traceback (most recent call last):
File "/usr/bin/ligolw_segment_diff", line 44, in <module>
from glue.ligolw import utils
ModuleNotFoundError: No module named 'glue.ligolw'
```
Now, these are actually part of the lscsoft-glue library rather than ligo-lw but at its current state, I believe this has been broken since the ligolw module in glue was removed. It's not clear to me if the right thing to do is move these utilities over to ligolw or to fix the imports in the glue library.https://git.ligo.org/kipp/python-ligo-lw/-/issues/12python-ligo-lw 1.6.0 on debian stretch missing some scripts2019-12-17T09:23:06ZPatrick Godwinpython-ligo-lw 1.6.0 on debian stretch missing some scriptsThis is after pulling the 1.6.0 package from the debian repo:
```/bin/bash: ligolw_no_ilwdchar: command not found```This is after pulling the 1.6.0 package from the debian repo:
```/bin/bash: ligolw_no_ilwdchar: command not found```https://git.ligo.org/kipp/python-ligo-lw/-/issues/11python-ligo-lw packaging changes to cover Bullseye (without python2)2020-02-25T07:48:10ZSteffen Grunewaldpython-ligo-lw packaging changes to cover Bullseye (without python2)The following covers the full set of changes made to the packaging provided with the 1.6.0 tarball:
```
diff -Nur old/debian/changelog new/debian/changelog
--- old/debian/changelog 2019-06-13 11:42:47.000000000 +0200
+++ new/debia...The following covers the full set of changes made to the packaging provided with the 1.6.0 tarball:
```
diff -Nur old/debian/changelog new/debian/changelog
--- old/debian/changelog 2019-06-13 11:42:47.000000000 +0200
+++ new/debian/changelog 2019-11-05 11:37:39.000000000 +0100
@@ -1,3 +1,17 @@
+python-ligo-lw (1.6.0-2.1) unstable; urgency=medium
+
+ * repackaging changes, to build also with python3 only
+
+ -- Steffen Grunewald <steffen.grunewald@ligo.org> Tue, 05 Nov 2019 11:11:00 +0100
+
+python-ligo-lw (1.6.0-2) unstable; urgency=medium
+
+ * repackaging:
+ move executables into ligo-lw-bin
+ attempt to run tests but ignore outcome
+
+ -- Steffen Grunewald <steffen.grunewald@ligo.org> Wed, 11 Sep 2019 09:25:00 +0200
+
python-ligo-lw (1.6.0-1) unstable; urgency=medium
* remove custom ProcessParamsTable from dbtables
diff -Nur old/debian/control new/debian/control
--- old/debian/control 2019-05-31 09:31:21.000000000 +0200
+++ new/debian/control 2019-11-05 11:37:39.000000000 +0100
@@ -5,7 +5,7 @@
Build-Depends:
debhelper (>= 9),
dh-python,
- python-dev,
+ python-all-dev,
python-ligo-common,
python-six,
python3-dev,
@@ -37,13 +37,16 @@
python-pylal
Replaces:
python-glue-ligolw-tools
-Provides:
- python-glue-ligolw-tools
-Description: LIGO Light-Weight XML I/O library
+#Provides:
+# python-glue-ligolw-tools
+# * moved to ligo-lw-bin
+Description: LIGO Light-Weight XML I/O library - Python 2
The LIGO Light-Weight XML format is used extensively by compact object
detection pipelines and associate tool sets. This package provides a
Python I/O library for reading, writing, and interacting with documents in
this format.
+ .
+ This package provides the Python2 modules.
Package: python3-ligo-lw
Architecture: any
@@ -66,10 +69,44 @@
python3-glue (<< 2.0)
Replaces:
python3-glue-ligolw-tools
-Provides:
- python3-glue-ligolw-tools
-Description: LIGO Light-Weight XML I/O library
+#Provides:
+# python3-glue-ligolw-tools
+# * moved to ligo-lw-bin
+Description: LIGO Light-Weight XML I/O library - Python 3
+ The LIGO Light-Weight XML format is used extensively by compact object
+ detection pipelines and associate tool sets. This package provides a
+ Python I/O library for reading, writing, and interacting with documents in
+ this format.
+ .
+ This package provides the Python3 modules.
+
+Package: ligo-lw-bin
+Architecture: any
+Depends:
+ ${shlibs:Depends},
+ ${misc:Depends},
+ ${python:Depends},
+ python-ligo-lw (= ${binary:Version}),
+ python3-ligo-lw (= ${binary:Version})
+Recommends:
+ python-lalburst,
+ python3-lalburst
+Conflicts:
+ python-glue-ligolw-tools,
+ python-glue (<< 2.0),
+ python-pylal,
+ python-ligo-lw (<< 1.6.0-2)
+Replaces:
+ python-glue-ligolw-tools,
+ python-ligo-lw (<< 1.6.0-2)
+#Provides:
+# python-glue-ligolw-tools,
+# python3-glue-ligolw-tools
+Description: LIGO Light-Weight XML I/O library - Tools
The LIGO Light-Weight XML format is used extensively by compact object
detection pipelines and associate tool sets. This package provides a
Python I/O library for reading, writing, and interacting with documents in
this format.
+ .
+ This package provides the executables.
+
diff -Nur old/debian/ligo-lw-bin.install new/debian/ligo-lw-bin.install
--- old/debian/ligo-lw-bin.install 1970-01-01 01:00:00.000000000 +0100
+++ new/debian/ligo-lw-bin.install 2019-09-11 09:15:20.000000000 +0200
@@ -0,0 +1 @@
+usr/bin/*
diff -Nur old/debian/python3-ligo-lw.install new/debian/python3-ligo-lw.install
--- old/debian/python3-ligo-lw.install 2017-03-23 07:14:16.000000000 +0100
+++ new/debian/python3-ligo-lw.install 2019-11-05 11:13:05.000000000 +0100
@@ -1,2 +1 @@
-#usr/bin/*
-usr/lib/python3*/dist-packages/ligo/lw
+usr/lib/python3*/*-packages/ligo/lw
diff -Nur old/debian/python-ligo-lw.install new/debian/python-ligo-lw.install
--- old/debian/python-ligo-lw.install 2017-03-23 07:14:16.000000000 +0100
+++ new/debian/python-ligo-lw.install 2019-11-05 11:12:57.000000000 +0100
@@ -1,2 +1 @@
-usr/bin/*
-usr/lib/python2*/dist-packages/ligo/lw
+usr/lib/python2*/*-packages/ligo/lw
diff -Nur old/debian/rules new/debian/rules
--- old/debian/rules 2019-02-14 18:44:58.000000000 +0100
+++ new/debian/rules 2019-11-05 11:37:39.000000000 +0100
@@ -1,13 +1,22 @@
#!/usr/bin/make -f
include /usr/share/dpkg/pkg-info.mk
-# Redirect Python 3 packages to a directory that we are going to ignore.
+# Redirect Python 3 packages to a directory that we are going to ignore/use.
# FIXME: in the future, we could put these in libexec or change their names.
export PYBUILD_INSTALL_ARGS_python3 := --install-scripts=/dont-install
%:
dh $@ --with=python2,python3 --buildsystem=pybuild
-# do not run tests at all.
+override_dh_auto_install:
+# installs to debian/tmp/
+ dh_auto_install
+# if python2, binaries go to debian/tmp/usr/bin - OK
+# if python3, binaries go to debian/tmp/dont-install (as above) - move them, but don't overwrite
+ mkdir -p $(CURDIR)/debian/tmp/usr/bin
+ mv -n $(CURDIR)/debian/tmp/dont-install/* $(CURDIR)/debian/tmp/usr/bin/ || true
+# now everything is in place for *.install files
+
+# attempt to run tests but ignore outcome
override_dh_auto_test:
- true
+ dh_auto_test || true
```https://git.ligo.org/kipp/python-ligo-lw/-/issues/10python-ligo-lw contains binaries (problem after phasing out python2)2021-03-03T09:06:55ZSteffen Grunewaldpython-ligo-lw contains binaries (problem after phasing out python2)Please see a month-old discussion over there: https://git.ligo.org/lscsoft/glue/issues/20 - is it true that @kipp.cannon is the only person taking care of packaging issues (not only for .deb, but also for .rpm)?Please see a month-old discussion over there: https://git.ligo.org/lscsoft/glue/issues/20 - is it true that @kipp.cannon is the only person taking care of packaging issues (not only for .deb, but also for .rpm)?