Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
P
python-ligo-lw
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 2
    • Issues 2
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
    • Iterations
  • Merge Requests 5
    • Merge Requests 5
  • Requirements
    • Requirements
    • List
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Security & Compliance
    • Security & Compliance
    • Dependency List
    • License Compliance
  • Operations
    • Operations
    • Incidents
    • Environments
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • CI / CD
    • Code Review
    • Insights
    • Issue
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • Kipp Cannon
  • python-ligo-lw
  • Merge Requests
  • !13

Open
Opened Apr 10, 2020 by Leo Pound Singer@leo-singerContributor
  • Report abuse
Report abuse

Add optional Zstandard compression

  • Overview 2
  • Commits 1
  • Pipelines 1
  • Changes 6

Zstandard provides much faster compression and decompression than gzip at any compression ratio. The "zstandard" Python package is an optional dependency to enable this new feature.

I tried running ligolw_sqlite on a recent gstlal coinc database to convert it to XML. The uncompressed XML file took 32 seconds to write and was 300M in size. The gzip-compressed XML file took 52 seconds and was 98M. The Zstandard-compressed XML file took 35 seconds and was 87M in size.

In short, Zstandard compression will give comparable file sizes to gzip but with compression/decompression time that is negligible compared to LIGO-LW processing itself.

Assignee
Assign to
Reviewer
Request review from
None
Milestone
None
Assign milestone
Time tracking
Reference: kipp.cannon/python-ligo-lw!13
Source branch: zstandard