Skip to content

Revamp plugin architecture with pluggy

Patrick Godwin requested to merge plugin_architecture into master

This merge request replaces the 'plugin' architecture to discover various ClassifierData, Reporter, and Classifier objects with the pluggy library.

The main motivation for this is to enable plugins that are not part of the codebase that someone else (including ourselves) may develop to add on to existing functionality, such that idq-pytorch that add in various pytorch classifiers. This would also enable local development classifiers that previously, would need to edit the existing codebase to allow such classifiers to be discovered.

Besides this, the other main change is the splitting up of the idq.io module into various submodules for maintainability and also to allow the same optional plugin scheme as is done for the classifiers, such as the gwf series reporter. All the reporters now live in the idq.io.reporters submodule and split up into various file formats, such as hdf5, pickle, etc. All the classifier data objects now live in idq.io.triggers and split up into various pipelines, such as snax or omicron.

Finally, the lower test coverage reflects the "true" test coverage now that the tests/ folder is ignored.

Overview of changes:

  • Add pluggy dependency for plugin discovery/registration.
  • Split idq.io into various submodules.
  • Change canonical flavor names for various classes when registering plugins to be more user-friendly, e.g. HDF5SeriesReporter -> series:hdf5.
  • Change to relative imports throughout.
  • Consolidate idq/classifiers/__init__.py and idq/classifiers/base.py to module initialization.
  • Leverage factories more heavily in idq.batch and idq.stream.
  • Add hookimpl in __init__.py to allow hook (plugin) implementations as in pluggy's nomenclature, used throughout to register plugins.
  • Simplify package registration in setup.py.
  • Simplify generation of ClassifierData copies (rather than making a factory and generating a flavor name) within factories.QuiverFactory(). Instead, given a class instance, we can use classifier_data.__class__ to grab the underlying class to generate new class instances.
  • Add zip_safe=False in setup.py to avoid bad import headers issue when using older versions of setuptools upon reinstalling.
  • Remove all_subclasses() function in idq.utils as its use was to register plugins.
  • Update unit tests throughout for new module locations, etc.
  • Update gstlal to SNAX in various places.
  • Ignore tests/ when running unit tests, which was inflating the coverage reported.
Edited by Patrick Godwin

Merge request reports