Skip to content
Snippets Groups Projects

Add retry for check_vectors if data is not yet available in the caches

Merged Add retry for check_vectors if data is not yet available in the caches
geoffrey.mo/gwcelery:retry_check_vectors into main
All threads resolved!
Merged Geoffrey Mo requested to merge geoffrey.mo/gwcelery:retry_check_vectors into main
All threads resolved!

Fixes #614 (closed) and #623 (closed).

check_vectors raises a ValueError when it tries to look for state vector data that's not yet available. This MR changes this so that it waits 5 seconds then tries again (up to four times), which should typically be enough for data to arrive at CIT.

Cases like this for this superevent should be handled by the retry. In this case, the early warning event with merger time 53.2 was uploaded at 53.0. Then, the check_vectors task is called at 54.306, asking for data to be checked up to 54.2.

Since the requested data end time (54.2) is before the time of the function call (54.306), the check built into check_vectors to skip for data after now passes. However, the 54.2 data hasn't arrived onto the cluster yet at 54.306, so the task runs into this ValueError.

This retry will now wait 5 seconds and try again, up to 4 times for a total of 20 seconds. This should be a long enough latency that any data will have arrived at CIT. If, after these 20 seconds the data is still not there, a ValueError will be raised and block the pipeline. This is the desired behaviour, since we want to make sure that we can check the data quality before an alert goes out.

Edited by Geoffrey Mo

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Geoffrey Mo changed the description

    changed the description

  • Brandon Piotrzkowski resolved all threads

    resolved all threads

  • Geoffrey Mo added 1 commit

    added 1 commit

    • fd8ca5a8 - Add retry for check_vectors if data is not yet available in the caches

    Compare with previous version

  • Geoffrey Mo added 4 commits

    added 4 commits

    • fd8ca5a8...9e121237 - 3 commits from branch emfollow:main
    • 00e0c6d0 - Add retry for check_vectors if data is not yet available in the caches

    Compare with previous version

  • Deep Chatterjee
  • Deep Chatterjee resolved all threads

    resolved all threads

  • Brandon Piotrzkowski added Needed for Release label

    added Needed for Release label

  • Deep Chatterjee
  • Deep Chatterjee
    Deep Chatterjee @deep.chatterjee ·
    Owner

    OK. One small suggestion above. But other than that, I am happy to give this a shot.

  • Geoffrey Mo resolved all threads

    resolved all threads

  • Geoffrey Mo added 1 commit

    added 1 commit

    • 386ef74e - Add unit test for check_vector retry

    Compare with previous version

  • Geoffrey Mo added 3 commits

    added 3 commits

    • e8128cfc - 1 commit from branch emfollow:main
    • 791c2158 - Add retry for check_vectors if data is not yet available in the caches
    • 6a80459a - Add unit test for check_vector retry

    Compare with previous version

  • Deep Chatterjee approved this merge request

    approved this merge request

  • Geoffrey Mo marked this merge request as ready

    marked this merge request as ready

  • Geoffrey Mo changed the description

    changed the description

  • Geoffrey Mo added 1 commit

    added 1 commit

    Compare with previous version

  • Geoffrey Mo enabled an automatic merge when the pipeline for 269f4297 succeeds

    enabled an automatic merge when the pipeline for 269f4297 succeeds

  • Geoffrey Mo merged

    merged

    • Please register or sign in to reply