Skip to content

Add type=nonestr to injection-numbers parser

Lewis Lakerink requested to merge lewis.lakerink/bilby_pipe:fix-missing-nonestr-on-injection-numbers into master

Amongst the work we are doing with GWCloud, we need to programatically parse and manipulate ini files. One of the problems I've come across is that injection-numbers is missing a type=nonestr argument. This leads to some strange issues when parsing, resaving, and parsing an ini file again. I previously discussed this with @gregory.ashton on slack.

In depth details (Copy pasted from my conversation with Greg)

Other list fields like mode-array and scheduler-module have the nonestr type.

The reason I ask (why injection-numbers is missing type=nonestr), is I've had to add the nonestr type to injection-numbers otherwise it loads =None as a list that is ["None"] rather than [None]. The problem being that if you then take the parsed args and save them with parser.write_to_file it ends up in the ini file as injection-numbers=['None'] - if you then try to parse that again and feed it in to MainInput - it raises raise BilbyPipeError(f"Invalid injection numbers {injection_numbers}"). Here is a reproducer:-

from tempfile import NamedTemporaryFile
​
from bilby_pipe.main import MainInput
from bilby_pipe.parser import create_parser
from bilby_pipe.utils import parse_args
​
# Parse the ini file generated by the UI
test_ini = """
trigger-time=12345678
detectors=[H1, L1]
injection-numbers=None"""
​
# Create a bilby argument parser
parser = create_parser()
​
# Bilby pipe requires a real file in order to parse the ini file
with NamedTemporaryFile() as f:
    # Write the temporary ini file
    f.write(test_ini.encode('utf-8'))
​
    # Make sure the data is written to the temporary file
    f.flush()
​
    # Read the data from the ini file
    args, unknown_args = parse_args([f.name], parser)
​
# Modify the args for submission
args.scheduler = "slurm"
args.scheduler_env = "whatever.sh"
​
# Show the injection numbers
print(args.injection_numbers)
​
# Now save these parameters to a new ini file
with NamedTemporaryFile() as f:
    # Write the temporary ini file
    parser.write_to_file(f.name, args, overwrite=True)
​
    # Make sure the data is flushed
    f.flush()
​
    # Read the data from the ini file
    args, unknown_args = parse_args([f.name], parser)
​
    # Show the injection numbers
    print(args.injection_numbers)
​
    # Generate the Input object so that we can determine the correct ini file
    inputs = MainInput(args, unknown_args)

Without type=nonestr on injection-numbers

['None']
["'None'"]
10:57 bilby_pipe INFO    : Setting segment duration 4.0s
10:57 bilby_pipe INFO    : Setting prior-file to None
10:57 bilby_pipe INFO    : Setting trigger time 12345678.0
10:57 bilby_pipe INFO    : Setting n_simulation=0
Traceback (most recent call last):
  File "/home/lewis/.config/JetBrains/PyCharm2021.1/scratches/scratch.py", line 49, in <module>
    inputs = MainInput(args, unknown_args)
  File "/home/lewis/Projects/gwdc/gwcloud-bilby/src/bundle/venv/lib/python3.7/site-packages/bilby_pipe/main.py", line 105, in __init__
    self.injection_numbers = args.injection_numbers
  File "/home/lewis/Projects/gwdc/gwcloud-bilby/src/bundle/venv/lib/python3.7/site-packages/bilby_pipe/input.py", line 492, in injection_numbers
    raise BilbyPipeError(f"Invalid injection numbers {injection_numbers}")
bilby_pipe.utils.BilbyPipeError: Invalid injection numbers ["'None'"]

With type=nonestr on injection-numbers

[None]
[None]
10:58 bilby_pipe INFO    : Setting segment duration 4.0s
10:58 bilby_pipe INFO    : Setting prior-file to None
10:58 bilby_pipe INFO    : Setting trigger time 12345678.0
10:58 bilby_pipe INFO    : Setting n_simulation=0
10:58 bilby_pipe INFO    : Setting analysis request_memory=4.0GB
10:58 bilby_pipe INFO    : Setting request_memory_generation=8GB
10:58 bilby_pipe INFO    : Setting analysis request_cpus = 1
Process finished with exit code 0

However, I don't know bilby's config file well enough to know if type=nonestr is intentionally omitted from injection-numbers. That said - I'm depending on these bilby internals to behave in an expected manner, and so I've patched bilby_pipe to have type=nonestr on injection-numbers so that the code works for what I'm trying to do which is programatically manipulate ini args before job submission.

Merge request reports