Add 'depends.py' script to help with dependent finding and conflict resolution

This MR adds the search/depends.py script which can be used to identify depends of a specific package. This can be useful in identifying which packages need to be updated if a migrated package is included in an update.

added 1 commit

• 451b9459 - depends.py: add a new script to identify dependents

Compare with previous version

added 1 commit

• 87f3218b - depends.py: add a new script to identify dependents

Compare with previous version

requested review from @rhys.poulton

requested review from @robert.bruntz

assigned to @duncanmmacleod

added infrastructure label

@rhys.poulton @robert.bruntz, please approve this new tool, which might help with identifying conflicts, the basic usage is here:

https://computing.docs.ligo.org/-/conda/-/jobs/2182106/artifacts/site/maintainer/workflow.html#debug-conflicts-identify

which demonstrates identifying which packages depend on an upstream library (eg root_base).

Once concrete example is gwollum, consider this sequence of events:

• gwollum is migrated to a new version of root_base
• a new version of gwollum is built (against the migrated version of root_base)
• the new version is requested for inclusion in IGWN Conda

Just updating gwollum.yaml will result in a conflict with the other packages that depend on root_base; the new tool will help identify which packages also depend on root_base and will need to be updated at the same time.

This looks really great and will be super useful in the future!

I was playing around with the script and when I run the command:

python3 ./scripts/depends.py whoneeds hdf5

I get the following error:

casacore :
Traceback (most recent call last):
  File "/users/poulton/packages/conda/./scripts/depends.py", line 115, in <module>
    sys.exit(main())
  File "/users/poulton/packages/conda/./scripts/depends.py", line 111, in main
    main_whoneeds(args.package, igwn_only=args.igwn_only)
  File "/users/poulton/packages/conda/./scripts/depends.py", line 77, in main_whoneeds
    for dep in sorted(pkgs[pkg], key=_sort_pkgs):
  File "/users/poulton/packages/conda/./scripts/depends.py", line 66, in _sort_pkgs
    return (pkgtup[0], int(pkgtup[1].split("_")[-1]), pkgtup[2])
ValueError: invalid literal for int() with base 10: '0h05be37c0'

This seems to be due to the fact that one of the builds of casacore has a strange build string:

('2.3.1a1', '0h05be37c0', 'hdf5 1.8.18|1.8.18.*')

could this be somehow checked or a warning given out if it is not able to parse the build string?

I also tried to see if this would help me understand the conflict encountered in !848 (merged). By using the following command:

 python ./scripts/depends.py whoneeds hdf5 --igwn-only

which gives the output:

gwollum :
    2.3.12-h6ad55ad_0 | hdf5 >=1.10.5,<1.10.6.0a0
    2.3.12-h1afb2d1_1 | hdf5 >=1.10.5,<1.10.6.0a0
    2.3.12-h281c2a4_2 | hdf5 >=1.10.5,<1.10.6.0a0
    2.3.12-hb07b48c_3 | hdf5 >=1.10.5,<1.10.6.0a0
    2.3.12-h5abd87d_4 | hdf5 >=1.10.5,<1.10.6.0a0
    2.3.12-h5abd87d_5 | hdf5 >=1.10.5,<1.10.6.0a0
    2.4.0-h9e007e1_0 | hdf5 >=1.10.5,<1.10.6.0a0
    2.4.0-h99f37a1_1 | hdf5 >=1.10.5,<1.10.6.0a0
    2.4.0-h4c1f761_1 | hdf5 >=1.10.6,<1.10.7.0a0
    2.4.1-h4c1f761_0 | hdf5 >=1.10.6,<1.10.7.0a0
    2.4.1-h735fe92_1 | hdf5 >=1.10.6,<1.10.7.0a0
    2.4.2-h735fe92_0 | hdf5 >=1.10.6,<1.10.7.0a0
    2.4.2-hf0d6e46_1 | hdf5 >=1.10.6,<1.10.7.0a0
    2.4.2-hec2da95_2 | hdf5 >=1.10.6,<1.10.7.0a0
    3.0.0-h73c9ba7_0 | hdf5 >=1.10.6,<1.10.7.0a0
    3.0.0-h96cbba9_1 | hdf5 >=1.10.6,<1.10.7.0a0
    3.0.0-h7347b9e_2 | hdf5 >=1.10.6,<1.10.7.0a0
    3.0.0-hf25a816_3 | hdf5 >=1.12.1,<1.12.2.0a0
    3.0.0-hf25a816_4 | hdf5 >=1.12.1,<1.12.2.0a0
    3.1.0-hf25a816_0 | hdf5 >=1.12.1,<1.12.2.0a0
    3.1.0-h1daf987_1 | hdf5 >=1.12.1,<1.12.2.0a0
    3.1.0-h75761a9_2 | hdf5 >=1.12.1,<1.12.2.0a0
    3.1.0-h1daf987_2 | hdf5 >=1.12.1,<1.12.2.0a0
    3.1.0-h75761a9_3 | hdf5 >=1.12.1,<1.12.2.0a0
    3.1.0-h254f8e2_4 | hdf5 >=1.12.1,<1.12.2.0a0
    3.1.0-h254f8e2_5 | hdf5 >=1.12.1,<1.12.2.0a0
    3.2.0-h254f8e2_0 | hdf5 >=1.12.1,<1.12.2.0a0
    3.2.0-h254f8e2_1 | hdf5 >=1.12.1,<1.12.2.0a0
    3.2.0-h254f8e2_2 | hdf5 >=1.12.1,<1.12.2.0a0
    3.2.0-h4fd6e6f_3 | hdf5 >=1.12.2,<1.12.3.0a0
    3.2.0-h4fd6e6f_4 | hdf5 >=1.12.2,<1.12.3.0a0
lal :
    6.18.0-h96c67fe_3 | hdf5 >=1.10.2,<1.10.3.0a0
    6.19.0-hc5f434a_1 | hdf5 >=1.10.2,<1.10.3.0a0
    6.19.0-hcdbed1a_2 | hdf5 >=1.10.3,<1.10.4.0a0
    6.19.0-hcdbed1a_3 | hdf5 >=1.10.3,<1.10.4.0a0
    6.19.0-h6ff7648_4 | hdf5 >=1.10.4,<1.10.5.0a0
    6.19.0-h7e9c209_1004 | hdf5 >=1.10.4,<1.10.5.0a0
    6.19.1-h6ff7648_0 | hdf5 >=1.10.4,<1.10.5.0a0
    6.19.1-h7e9c209_1000 | hdf5 >=1.10.4,<1.10.5.0a0
    6.19.2-h7e9c209_0 | hdf5 >=1.10.4,<1.10.5.0a0
    6.19.2-h7e9c209_1 | hdf5 >=1.10.4,<1.10.5.0a0
    6.19.2-haaac68a_2 | hdf5 >=1.10.4,<1.10.5.0a0
    6.19.2-mkl_h59e1d72_3 | hdf5 >=1.10.4,<1.10.5.0a0
    6.19.2-fftw_hccb7b1c_103 | hdf5 >=1.10.4,<1.10.5.0a0
    6.20.1-mkl_h59e1d72_0 | hdf5 >=1.10.4,<1.10.5.0a0
    6.20.1-mkl_hf3b3760_1 | hdf5 >=1.10.5,<1.10.6.0a0
    6.20.1-mkl_hfddc10a_2 | hdf5 >=1.10.5,<1.10.6.0a0
    6.20.1-fftw_hccb7b1c_100 | hdf5 >=1.10.4,<1.10.5.0a0
    6.20.1-fftw_h0d1b64c_101 | hdf5 >=1.10.5,<1.10.6.0a0
    6.20.1-fftw_h0710ca0_102 | hdf5 >=1.10.5,<1.10.6.0a0
    6.20.2-mkl_hfddc10a_0 | hdf5 >=1.10.5,<1.10.6.0a0
    6.20.2-mkl_h6555cc1_1 | hdf5 >=1.10.4,<1.10.5.0a0
    6.20.2-mkl_h6555cc1_2 | hdf5 >=1.10.4,<1.10.5.0a0
    6.20.2-mkl_hfddc10a_3 | hdf5 >=1.10.4,<1.10.5.0a0
    6.20.2-mkl_hfddc10a_4 | hdf5 >=1.10.4,<1.10.5.0a0
    6.20.2-mkl_hfddc10a_5 | hdf5 >=1.10.4,<1.10.5.0a0
    6.20.2-mkl_hfddc10a_6 | hdf5 >=1.10.5,<1.10.6.0a0
    6.20.2-mkl_h66c44d0_6 | hdf5 >=1.10.5,<1.10.6.0a0
    6.20.2-fftw_h0710ca0_100 | hdf5 >=1.10.5,<1.10.6.0a0
    6.20.2-fftw_hce47d02_101 | hdf5 >=1.10.4,<1.10.5.0a0
    6.20.2-fftw_hce47d02_102 | hdf5 >=1.10.4,<1.10.5.0a0
    6.20.2-fftw_h0710ca0_103 | hdf5 >=1.10.4,<1.10.5.0a0
    6.20.2-fftw_h0710ca0_104 | hdf5 >=1.10.4,<1.10.5.0a0
    6.20.2-fftw_h0710ca0_105 | hdf5 >=1.10.4,<1.10.5.0a0
    6.20.2-fftw_h0710ca0_106 | hdf5 >=1.10.5,<1.10.6.0a0
    6.20.2-fftw_h8e8928b_106 | hdf5 >=1.10.5,<1.10.6.0a0
    6.21.0-mkl_h66c44d0_0 | hdf5 >=1.10.5,<1.10.6.0a0
    6.21.0-mkl_hcd2bc93_1 | hdf5 >=1.10.5,<1.10.6.0a0
    6.21.0-mkl_hcd2bc93_2 | hdf5 >=1.10.5,<1.10.6.0a0
    6.21.0-fftw_h8e8928b_100 | hdf5 >=1.10.5,<1.10.6.0a0
    6.21.0-fftw_h9392de7_101 | hdf5 >=1.10.5,<1.10.6.0a0
    6.21.0-fftw_h9392de7_102 | hdf5 >=1.10.5,<1.10.6.0a0
    6.22.0-mkl_hcd2bc93_0 | hdf5 >=1.10.5,<1.10.6.0a0
    6.22.0-mkl_hcd2bc93_1 | hdf5 >=1.10.5,<1.10.6.0a0
    6.22.0-mkl_h360972e_2 | hdf5 >=1.10.6,<1.10.7.0a0
    6.22.0-fftw_h9392de7_100 | hdf5 >=1.10.5,<1.10.6.0a0
    6.22.0-fftw_h9392de7_101 | hdf5 >=1.10.5,<1.10.6.0a0
    6.22.0-fftw_h94c382a_102 | hdf5 >=1.10.6,<1.10.7.0a0
    7.0.0-mkl_h360972e_0 | hdf5 >=1.10.6,<1.10.7.0a0
    7.0.0-mkl_h360972e_1 | hdf5 >=1.10.6,<1.10.7.0a0
    7.0.0-fftw_h94c382a_100 | hdf5 >=1.10.6,<1.10.7.0a0
    7.0.0-fftw_h94c382a_101 | hdf5 >=1.10.6,<1.10.7.0a0
    7.1.0-mkl_hcd9884b_0 | hdf5 >=1.10.6,<1.10.7.0a0
    7.1.0-fftw_ha28fc39_100 | hdf5 >=1.10.6,<1.10.7.0a0
lalinference :
    1.10.3-he005d58_0 | hdf5 >=1.10.4,<1.10.5.0a0
    1.11.0-h8d081a0_0 | hdf5 >=1.10.4,<1.10.5.0a0
    1.11.1-h8d081a0_0 | hdf5 >=1.10.4,<1.10.5.0a0
    1.11.1-h9c975d8_1 | hdf5 >=1.10.4,<1.10.5.0a0
    1.11.2-h9ea3725_0 | hdf5 >=1.10.4,<1.10.5.0a0
    1.11.2-hd60b62b_1 | hdf5 >=1.10.5,<1.10.6.0a0
    1.11.2-hd60b62b_2 | hdf5 >=1.10.5,<1.10.6.0a0

But it does not show version 7.1.1-fftw_*_101 for lal which resolved the issue. Any idea why it is not?

Damn, this exposes a fairly large whole in the --igwn-only implementation, in that the specific lal package that requires hdf5 (these days) is liblal, which isn't actually ever specified in the IGWN Conda Distribution (because specifying lal gives you an exact pin down to liblal so specifying both is redundant).

I'm not quite sure how to handle this without a major restructuring (my only idea so far is to replace --igwn-only with a different argument that reads relevant package names from an environment YAML file.

added 1 commit

• c705a840 - depends.py: improve --igwn-only parsing

Compare with previous version