Python3

Status

This code is as ready as I can get it. Here's what still needs to be done:

• Figure out issue with lvalert-overseer.
• Check for possible issues on uploading hardware injections (see here. I haven't been able to test it, but it uses StringIO which is definitely a cause for concern in terms of Python 2/3 compatibility.
• Tag a new version of lvalert-overseer, push it to PyPI, and update the requirements.txt file in this repository to use the new tag.

Suggested strategy for rolling it out

• Test the deployment with the new puppet configuration on gracedb-dev2 or dev1 (more below)
• Deploy on playground and see what happens
• Test this image on gracedb-test
• Tag it
• Deploy on production

To test on UWM VMs

• Delete /home/gracedb/djangoenv
• As root, run puppet and immediately turn it off so it doesn't revert to the default configuration:

  puppet agent -t --environment=gracedb_python3; systemctl stop puppet

• Make sure the repository in /home/gracedb/bin is up-to-date.
• Do some testing
• If everything looks good, proceed

Final deployment on UWM VMs

• Stop/disable puppet on gracedb-playground

  systemctl stop puppet
  systemctl disable puppet

• Someone at UWM should rebase and merge this MR.
• Setup the dev machines to use this branch and the new configuration.
  • As the gracedb user:

    cd /home/gracedb/gracedb_project
    rm -rf ../djangoenv
    git checkout --track origin/python3

  • As root:

    puppet agent -t
    systemctl restart gunicorn
    systemctl restart lvalert_overseer

• If all seems good, do the same for playground
• Iterate on any changes needed to the server code
• Once it seems stable on playground, test it on gracedb-test
• If that seems good, merge the code into master and tag it

NOTE: gracedb-dev2 is currently set up for Python 3 both in terms of its Puppet configuration and its code base. Puppet is currently turned off so that it doesn't revert to the "default" Puppet configuration with Python 2. Once you're ready to "officially" deploy with Python 3, and the Puppet configuration is merged, just do puppet agent -t as root on this server to restart Puppet.

Final deployment in AWS

Should be easy - it's all in a Docker image!

Potential issues

The parts of the code that I expect the most issues to occur are:

• VOEvent building (gracedb/annotations/voevent_utils.py) since it was completely rewritten
• VOEvent ingestion for external events (gracedb/events/translator.py) since it was completely rewritten
• General string parsing and encoding/decoding

changed the description

added 47 commits

• 2b761ba5 - 1 commit from branch master
• f13191b8 - gracedb.search: updated syntax for python3
• 8526d19f - gracedb: updated dict usage for python3
• 2b90df7b - updated exception syntax for python3
• 423e4cd2 - updated urllib imports for python3
• b9e13aaf - gracedb: updated mock imports for python3
• c8f547ae - gracedb.search: import reduce in python3
• d84f1269 - updated imports of StringIO for python3
• 206d8414 - gracedb.superevents: decode input to StringIO
• b6ecbfa2 - gracedb.events: updated Event.datadir for python3
• 23c74b23 - gracedb.core: updated octal for python3
• 3651979f - Fix urlencode imports
• 389a3026 - update print statements to function calls
• 932fc27e - WIP: get lvalert-overseer from gitlab for python3 compatibility
• 92bdcb3d - core: rework VersionedFile
• 4272f7f7 - events: python 3 compatibility for encoding utils
• 33bb0403 - python 3 compatibility: dict usage
• 70baab6f - python 3: rework __unicode__ -> __str__ methods
• c3137d40 - Python 3: remove use of unicode()
• d7ceb125 - Python 3: remove unneeded import of 'exceptions' module
• 27282757 - Python 3: fix lambda syntax
• 6d5782ba - requirements: fix pytest < 5 for python2/3 compatibility
• 524691f1 - Python 3: fix all StringIO usage
• 2ba4c585 - Import 'reduce' for Python 3
• df15ebfd - Python 3: cast outputs of map() to lists
• 0b00728e - Python 3: remove unnecessary use of long()
• de71ae77 - Python 3: fix b64encode usage in tests
• 42761c47 - Python 3: fix response content checks in unit tests
• 003248cf - Python 3: remove use of assertItemsEqual in unit tests
• abd58196 - Python 3: fix upload file tests
• 2ac3c8b1 - core: add handling for bytes file content in VersionedFiles
• 5cba419e - Python 3: remove use of basestring
• aec6810e - requirements: install VOEventLib from PyPI
• 8e0b4fc3 - fix usage of VersionedFile through events application
• e0abc4c9 - Python 3: fix a few tests with decoding response content
• bf8bc318 - requirements: remove unnecessary package (future)
• 4026f57b - requirements: fix voeventlib install to be only Python 3
• 019d2320 - requirements: fix pytest version for python 2 vs 3
• f47dfb23 - requirements: fix futures install to be python 2 only
• de8951be - core.vfile: fixed file writing for bytes
• d310ed69 - Python 3: remove Python 2 from CI config
• 0f4c7eb7 - Rework VOEvent building infrastructure
• 16db57a8 - core.time_utils: fix up and add some time handling utilities
• d689861a - events.translator: rework Fermi/Swift event parsing
• bf6c3d92 - docker: convert containers to use Python 3
• 3c633bd2 - api.v1.events.views.EventList.get: bugfix integer division calculation
• eccff84b - temp

Compare with previous version

changed the description

added 3 commits

• 6253b31f - Python 3: migrations
• 98895fbc - alerts.xmpp: fix encoding for LVAlerts in Python 3
• ac6910c8 - temp

Compare with previous version

changed the description

changed the description

changed the description

added 52 commits

• 7c16ed91 - 1 commit from branch master
• 2a4c1e3b - Bugfix migration which assigns permissions to groups
• 8461c207 - gracedb.search: updated syntax for python3
• 94a5d812 - gracedb: updated dict usage for python3
• 8e86f3ad - updated exception syntax for python3
• 6e3d8492 - updated urllib imports for python3
• 7841312f - gracedb: updated mock imports for python3
• 9bdb45fb - gracedb.search: import reduce in python3
• 8b4c5f8b - updated imports of StringIO for python3
• 05c02f83 - gracedb.superevents: decode input to StringIO
• 8d571b10 - gracedb.events: updated Event.datadir for python3
• de66251a - gracedb.core: updated octal for python3
• 99b2ccdc - Fix urlencode imports
• d520373e - update print statements to function calls
• 41446b75 - WIP: get lvalert-overseer from gitlab for python3 compatibility
• 44185f23 - core: rework VersionedFile
• 85352460 - events: python 3 compatibility for encoding utils
• 3b80b108 - python 3 compatibility: dict usage
• 5236e2f5 - python 3: rework __unicode__ -> __str__ methods
• 6a99c94b - Python 3: remove use of unicode()
• 8a0785da - Python 3: remove unneeded import of 'exceptions' module
• 28e4d210 - Python 3: fix lambda syntax
• 4fadf3a9 - requirements: fix pytest < 5 for python2/3 compatibility
• a1372b5f - Python 3: fix all StringIO usage
• 5d70a3cd - Import 'reduce' for Python 3
• 15fb3678 - Python 3: cast outputs of map() to lists
• bcdcfa32 - Python 3: remove unnecessary use of long()
• 19efbbcf - Python 3: fix b64encode usage in tests
• 2fa7eb10 - Python 3: fix response content checks in unit tests
• 2218851b - Python 3: remove use of assertItemsEqual in unit tests
• 6db0c2c7 - Python 3: fix upload file tests
• 2ca28a5f - core: add handling for bytes file content in VersionedFiles
• 7675bff4 - Python 3: remove use of basestring
• 6f8eb63f - requirements: install VOEventLib from PyPI
• f1015549 - fix usage of VersionedFile through events application
• 96319a00 - Python 3: fix a few tests with decoding response content
• 97f6c936 - requirements: remove unnecessary package (future)
• ce7d5f38 - requirements: fix voeventlib install to be only Python 3
• d78bf493 - requirements: fix pytest version for python 2 vs 3
• 4adbae8a - requirements: fix futures install to be python 2 only
• e5ffd2c6 - core.vfile: fixed file writing for bytes
• 75f34a84 - Python 3: remove Python 2 from CI config
• c1a78e70 - Rework VOEvent building infrastructure
• 70bbbde2 - Add newer version of numpy for compatibility with voeventparse
• 4b4c266f - core.time_utils: fix up and add some time handling utilities
• c877bcfc - events.translator: rework Fermi/Swift event parsing
• 2f0f53f2 - docker: convert containers to use Python 3
• 4c4f4426 - api.v1.events.views.EventList.get: bugfix integer division calculation
• a343b132 - Python 3: migrations
• 1728bf54 - alerts.xmpp: fix encoding for LVAlerts in Python 3
• f23bb98b - update user account getter to handle string encoding/decoding for Python 3
• 799b051d - Update manage.py to activate the virtualenv with Python 3 compatibility

Compare with previous version

Posting this here as well as on the Puppet config repo:

The current Puppet configuration is not executing correctly when installing the virtual environment. I think it is because the Puppet module for Python uses a hard-coded --no-binary :all: when installing packages to the virtualenv with pip. I've fixed that in the past by adding --no-binary :none: to the pip args in the Puppet configuration.

I think this complication is due to the fact that we are installing lvalert-overseer from its gitlab repo at present. I expect this issue to be resolved once a new version of that package is tagged and pushed to PyPI, and the requirements file in the gracedb code base is updated.

changed the description

added 6 commits

• 25e355e1 - Two migrations:
• faf19ffc - em_type and coinc_far added to superevent model. next step: add to superevent api requests
• 74c5cb81 - added em_type and coinc_far to api requests. next step: modify client to...
• a0decede - default for em_type and coinc_far = null
• 3700dcb9 - superevents/utils.py: one more edit for updating superevents.
• 87095575 - Adding support for new superevent fields

Compare with previous version

added 1 commit

• 9adb9cb0 - cleaning up migrations

Compare with previous version

added 10 commits

• 78c93144 - Add RAVEN VOEvent variant
• e2b06cb4 - Add processing to the superevent API
• fd83907e - Add processing and validation in the API
• 2f9d5221 - Fix bug accidently introduced
• aec2feb1 - Question on getting data
• b6eb2c42 - adding raven voevent fields
• b9444798 - fixed raven_coinc
• 08457eca - Build xml file from RAVEN entries
• 732a5ee4 - raven voevents in 'working' state.
• bbb5da21 - Merge branch 'brandon.piotrzkowski/gracedb-ravenvoevent' into HEAD

Compare with previous version

added 1 commit

• 88a440b5 - fixing voevent_utils.py

Compare with previous version

added 1 commit

• 21cdd867 - updating virgo_detchar's cert

Compare with previous version

added 1 commit

• b6fe8319 - Update 0053_update_virgodetchar_cert.py: missing slash

Compare with previous version

added 1 commit

• 0c893aeb - Adding latency to basic info table

Compare with previous version

added 1 commit

• 88590b96 - Adding a migration to update pycbclive's cert for O3b.

Compare with previous version