Skip to content

python3 compatibility fixes

Patrick Godwin requested to merge python3_fixes into master

This merge request should get us python 3 compatibility, or at least something close to it. The unit tests all pass and imports work correctly as well as some basic runs through the pipeline. There's probably something I missed since I didn't test all combinations of Reporters, etc.

Changes:

  • dict.has_key(key) to key in dict
  • Error, 'message' to Error('message')
  • configparser.py: change ConfigParser import to from six.moves import configparser to support both easily.
  • added a function in utils.py called unpack(). This unpacks an argument tuple and calls the target function 'func', and is used as a decorator for any function that multiprocessing uses that needs to be passed in multiple arguments. The trick in python 2 was to wrap that all up in a tuple like: func((arg1, arg2)) but this doesn't work in python 3. I have the same function in ligo-scald for multiprocessing routines to get around this issue.
    • Example:
      @utils.unpack
      def _multiprocessing_fetch_triggers(data, args, kwargs):
      from
      def _multiprocessing_fetch_triggers((data, args, kwargs)):
  • PickleReporter: The open() context managers have been modified to handle pickle.dump() and pickle.load() properly, by changing the file modes from 'r' and 'w' to 'rb' and 'wb', respectively. This is required in python 3 but is done implicitly in python 2.
  • __setstate__ and __getstate__ for some classes for use in pickling have been modified. The same problem in multiprocessing not able to define tuples spanning multiple args is also affected here. Instead of setting the state as a tuple, we use a dict. I forget which thread I found this from, but it was a pattern I found on stackoverflow.
  • Some extra changes in the test suite that weren't issues before but with a newer numpy and pytest, were giving errors.

Merge request reports