Installation with conda can break multiprocessing
I have been trying to run bilby
sampling with dynesty
on a new cluster. When trying to run job with multiprocessing, the program freezes during 1st likelihood call by 'dynesty' (bilby generates live points without a problem, but when 'dynesty' tries to call the passed likelihood function, it with a pool, it waits forever for the results without raising any errors).
This problem does not arise if only 1 process is used. It is also partially solved by changing multiprocessing method from fork
to forkserver
or spawn
(The sampling happens, but now the program gets frozen after trying to close the Pool
).
The problem persists even after reinstalling conda. (I installed conda 23.3.1 for linux from official website).
This behaviour in unusual in that it only happens when installing bilby
with conda
- installation with pip
works fine. Ie.:
conda create -n <name> -c conda-forge python=3.10 bilby gwpy python-lalsimulation
creates environment where the problem occurs, while for:
conda create -n <name> python=3.10
conda activate <name>
pip install bilby gwpy lalsuite
everything works fine.
The problem also does not occur when creating environment with conda on CIT
and Hawk
clusters (which have conda
preinstalled for the user already).
Could someone check if the problem is replicable for them?