Skip to content

Extended retrying to POST operations

Automatic retrying was enabled and has largely been a success at reconnecting connection issues and other select operations that fail, but I've noticed some POST operations falling through the cracks and not retrying automatically. I noticed this post which states:

POST is not considered IDEMPOTENT. You'll need to extend the method whitelist with POST to allow max_retries to function there.

Which is consistent with the behavior i'm seeing. As a test I wrote this script:

#!/bin/python3

from requests.adapters import HTTPAdapter, Retry
from requests.exceptions import HTTPError
from igwn_auth_utils.requests import Session
from requests.exceptions import RetryError

DEFAULT_RETRY_CODES = [500, 502, 503, 504, 408, 409]

retries = Retry(total=5,
                backoff_factor=0.1,
                status_forcelist=DEFAULT_RETRY_CODES)

server = 'https://httpbin.org/status/'

client = Session()
client.mount('https://', HTTPAdapter(max_retries=retries))

try:
    print('GETing a 502')
    client.get(server+'502')
except RetryError as e:
    print('--> This operation was retried and hit the maximum number of retries.')
except HTTPError as e:
    print('--> This operation returned a {} error without retrying'.format(e.response.status_code))

try:
    print('POSTing a 502')
    client.post(server+'502')
except RetryError as e:
    print('--> This operation was retried and hit the maximum number of retries.')
except HTTPError as e:
    print('--> This operation returned a {} error without retrying'.format(e.response.status_code))

try:
    print('PUTing a 502')
    client.put(server+'502')
except RetryError as e:
    print('--> This operation was retried and hit the maximum number of retries.')
except HTTPError as e:
    print('--> This operation returned a {} error without retrying'.format(e.response.status_code))

try:
    print('PATCHing a 502')
    client.patch(server+'502')
except RetryError as e:
    print('--> This operation was retried and hit the maximum number of retries.')
except HTTPError as e:
    print('--> This operation returned a {} error without retrying'.format(e.response.status_code))

Which for ligo-gracedb-2.11.0 returns:

GETing a 502
--> This operation was retried and hit the maximum number of retries.
POSTing a 502
--> This operation returned a 502 error without retrying
PUTing a 502
--> This operation was retried and hit the maximum number of retries.
PATCHing a 502
--> This operation returned a 502 error without retrying

The fix should be: extend the retry functionality for POST and PATCH and the list of allowed retry codes should be pared down for safety.

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information