Multi-threaded compression of sky map files
Use a multi-threaded gzip (pigz) for faster compression of large sky map files on multi-threaded machines. For example, the 1.6GB bayestar.fits file from GW150914 compresses 5x faster with pigz than gzip using default options on emfollow-playground. Note, this test uses the /dev/shm filesystem to minimize filesystem/storage latencies:
[root@emfollow-playground shm]# ls -lh bayestar.fits -rw-r--r-- 1 root root 1.6G Aug 4 19:06 bayestar.fits [root@emfollow-playground shm]# md5sum bayestar.fits b788871843d0c0ac112b8b9076649b39 bayestar.fits [root@emfollow-playground shm]# time gzip --verbose bayestar.fits bayestar.fits: 99.7% -- replaced with bayestar.fits.gz real 0m8.665s user 0m8.181s sys 0m0.483s [root@emfollow-playground shm]# ls -lh bayestar.fits.gz -rw-r--r-- 1 root root 4.4M Aug 4 19:06 bayestar.fits.gz [root@emfollow-playground shm]# file bayestar.fits.gz bayestar.fits.gz: gzip compressed data, was "bayestar.fits", from Unix, last modified: Sun Aug 4 19:06:17 2019 [root@emfollow-playground shm]# gunzip bayestar.fits.gz [root@emfollow-playground shm]# md5sum bayestar.fits b788871843d0c0ac112b8b9076649b39 bayestar.fits [root@emfollow-playground shm]# time pigz bayestar.fits real 0m1.675s user 0m9.590s sys 0m0.776s [root@emfollow-playground shm]# ls -lh bayestar.fits.gz -rw-r--r-- 1 root root 4.7M Aug 4 19:06 bayestar.fits.gz [root@emfollow-playground shm]# file bayestar.fits.gz bayestar.fits.gz: gzip compressed data, was "bayestar.fits", from Unix, last modified: Sun Aug 4 19:06:17 2019 [root@emfollow-playground shm]# gunzip bayestar.fits.gz [root@emfollow-playground shm]# md5sum bayestar.fits b788871843d0c0ac112b8b9076649b39 bayestar.fits
Note, a further 30% reduction in wall clock time can be had at the expense of doubling the small output file size by using the --fast gzip option,
[root@emfollow-playground shm]# time pigz --fast bayestar.fits real 0m1.195s user 0m6.049s sys 0m0.802s [root@emfollow-playground shm]# ls -lh bayestar.fits.gz -rw-r--r-- 1 root root 11M Aug 4 19:06 bayestar.fits.gz
In this test the emfollow-playground VM was running with 8 Xeon Gold CPU cores,
[root@emfollow-playground shm]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1 Core(s) per socket: 2 Socket(s): 4 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz Stepping: 4 CPU MHz: 2992.938 BogoMIPS: 5985.87 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K L3 cache: 16384K NUMA node0 CPU(s): 0-7 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat umip pku ospke spec_ctrl intel_stibp arch_capabilities