Skip to content

Add optional Zstandard compression

Leo P. Singer requested to merge leo-singer/igwn-ligolw:zstandard into master

Zstandard provides much faster compression and decompression than gzip at any compression ratio. The "zstandard" Python package is an optional dependency to enable this new feature.

I tried running ligolw_sqlite on a recent gstlal coinc database to convert it to XML. The uncompressed XML file took 32 seconds to write and was 300M in size. The gzip-compressed XML file took 52 seconds and was 98M. The Zstandard-compressed XML file took 35 seconds and was 87M in size.

In short, Zstandard compression will give comparable file sizes to gzip but with compression/decompression time that is negligible compared to LIGO-LW processing itself.

Merge request reports