Update DataSourceInfo API
Motivation
The datasource information utilities in gstlal.datasource were designed such that optparse command-line arg parsing was the primary means of initializing a GWDataSourceInfo object; a necessary step to build any pipeline using mkbasicsrc. The new api, via DataSourceInfo has several goals:
- Direct construction of datasources from Python literals
- Preserve compatibility with command-line args
- Provide convenience functions for constructing pipeline datasources directly in Python
Changes
This MR touches a lot of scripts, but should preserve existing functionality. The primary changes occur in a single module, gstlal.datasource, where the GWDataSourceInfo object and utilities have been deprecated in favor of the DataSourceInfo object and utilities. Feature-set is preserved but interface is improved, see examples below. Specific changes include:
- New
DataSourceInfoclass with improved interface and backwards compat - Generalized command-line parsing utilities, such as
datasource.parse_list_to_dict - New administrative utilities in
gstlal.utilities.adminfor:- Adding deprecation warnings to classes, functions, or methods
- Adding copyright stubs to modules
- Accompanying tests
Examples
Legacy: GWDataSourceInfo
args = [
'--data-source=frames',
'--frame-cache=/path/to/cache',
'--channel-name=H1=CHANNEL',
'--gps-start-time=1234567',
'--gps-end-time=1234568',
]
parser = optparse.OptionParser(description="test legacy mkbasicsrc")
datasource.append_options(parser)
options, filenames = parser.parse_args(args)
info = GWDataSourceInfo(options)
New api: DataSourceInfo
info = DataSourceInfo(data_source='frames',
frame_cache='path/to/cache',
gps_start_time=1234567,
gps_end_time=1234568,
channel_name={'H1': 'CHANNEL'})Refactor and Drop-In Replacement
What was:
gw_data_source_info = datasource.GWDataSourceInfo(options)Becomes:
gw_data_source_info = datasource.DataSourceInfo.from_optparse(options)Merge request reports
Activity
added 2 commits
added 10 commits
-
7a2f33f3 - 1 commit from branch
master - e1dcf319 - remove mocker fixture
- 9e3ae1cf - add init tests for each datasource to check parity between new and legacy api
- 2b7ffe96 - rename ALigo -> ALIGO
- 931b7906 - add legacy tests for basic source
- 05eec679 - Merge branch 'master' into feature-datasource-api
- ceacc5b0 - skip mkchecktimestamp
- 354a285c - add skip if on osx util
- 4569b2a3 - fix enum in statement
- b8639983 - fix tests and skip live on osx
Toggle commit list-
7a2f33f3 - 1 commit from branch
added 2 commits
- Resolved by James Kennington
@patrick.godwin There seems to be an env issue in the test suite runner. Is GST_PLUGIN_PATH being set properly during the build suite?
- Last reply by Patrick Godwin
added 2 commits
added 2 commits
added 1 commit
- 84c6e79c - mute doctests that output dicts (key order unstable)
added 1 commit
- 99423ad4 - fix channel-name vs state-channel-name behavior
added 9 commits
- 93adbfb9 - add admin utilities for deprecation warnings
- 13c46dd0 - add constants tests
- f209e832 - add admin utils to package
- 67a69197 - fully generalize old parsing utilities
- c13d46aa - add string cleaning test tool
- 5b4375a7 - add utility for adding copyright to modules
- 322b84f5 - update datasource module for copyright
- e32ef0f8 - shuffle and initial cleaning
- f447e6ba - migrate usage of GWDataSourceInfo to DataSourceInfo
Toggle commit listadded 1 commit
- becb5f2c - remove subdirs arg from new gstlal.utilities makefile
requested review from @patrick.godwin
-
Looks good, @james.kennington! Nice to see the extra test coverage and the extra functionality to allow building pipelines with the python API. Specific code review comments above. In addition, I see that some of the unit tests failed due to copyright-related functionality which would be good to address as well.
Edited by Patrick Godwin added 1 commit
- 75fb7d8a - add test for persistence of cache path if not specified