Skip to content

Use ThreadPoolExecutor to speed up downloading data files

Duncan Macleod requested to merge duncanmmacleod/em-bright:threaded-download into main

This MR refactors __init__.py's downloading of data files to use concurrent.futures.ThreadPoolExecutor. This reduces the first-time startup transient from 80s to 40s on the CIT network (the difference will be larger on a typical home broadband connection).

$ rm -rf ~/.ligo.em_bright
$ time ./venv/bin/python3 -c "import ligo.em_bright"
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/equil_2H.dat
|=================================================================|  53k/ 53k (100.00%)         0s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/APR4_EPP.pickle
|=================================================================|  24M/ 24M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/BHF_BBB2.pickle
|=================================================================|  23M/ 23M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/H4.pickle
|=================================================================|  23M/ 23M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/HQC18.pickle
|=================================================================|  23M/ 23M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/KDE0V1.pickle
|=================================================================|  23M/ 23M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/KDE0V.pickle
|=================================================================|  23M/ 23M (100.00%)         2s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/MPA1.pickle
|=================================================================|  24M/ 24M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/MS1B_PP.pickle
|=================================================================|  24M/ 24M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/MS1_PP.pickle
|=================================================================|  24M/ 24M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/RS.pickle
|=================================================================|  23M/ 23M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/SK255.pickle
|=================================================================|  24M/ 24M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/SK272.pickle
|=================================================================|  24M/ 24M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/SKI2.pickle
|=================================================================|  24M/ 24M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/SKI3.pickle
|=================================================================|  24M/ 24M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/SKI4.pickle
|=================================================================|  24M/ 24M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/SKI5.pickle
|=================================================================|  24M/ 24M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/SKI6.pickle
|=================================================================|  24M/ 24M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/SKMP.pickle
|=================================================================|  23M/ 23M (100.00%)         2s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/SKOP.pickle
|=================================================================|  23M/ 23M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/SLY230A.pickle
|=================================================================|  23M/ 23M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/SLY2.pickle
|=================================================================|  23M/ 23M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/SLY9.pickle
|=================================================================|  24M/ 24M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/SLy.pickle
|=================================================================|  23M/ 23M (100.00%)         1s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/EOS_BAYES_FACTOR_MAP.json
|=================================================================| 404 /404  (100.00%)         0s
Downloading https://git.ligo.org/emfollow/em-properties/em-bright/-/raw/v1.0.3/ligo/em_bright/data/EOS_MAX_MASS_MAP.json
|=================================================================| 406 /406  (100.00%)         0s

real    1m19.299s
user    0m2.656s
sys     0m9.559s
$ rm -rf ~/.ligo.em_bright
$ git checkout threaded-download
Switched to branch 'threaded-download'
Your branch is up to date with 'origin/threaded-download'.
$ time ./venv/bin/python3 -c "import ligo.em_bright"
Downloading ligo.em_bright data files...
|=================================================================|  26 / 26  (100.00%)        37s

real    0m38.610s
user    0m2.936s
sys     0m18.822s

Merge request reports