Extended retrying to POST operations
Automatic retrying was enabled and has largely been a success at reconnecting connection issues and other select operations that fail, but I've noticed some POST
operations falling through the cracks and not retrying automatically. I noticed this post which states:
POST is not considered IDEMPOTENT. You'll need to extend the method whitelist with POST to allow max_retries to function there.
Which is consistent with the behavior i'm seeing. As a test I wrote this script:
#!/bin/python3
from requests.adapters import HTTPAdapter, Retry
from requests.exceptions import HTTPError
from igwn_auth_utils.requests import Session
from requests.exceptions import RetryError
DEFAULT_RETRY_CODES = [500, 502, 503, 504, 408, 409]
retries = Retry(total=5,
backoff_factor=0.1,
status_forcelist=DEFAULT_RETRY_CODES)
server = 'https://httpbin.org/status/'
client = Session()
client.mount('https://', HTTPAdapter(max_retries=retries))
try:
print('GETing a 502')
client.get(server+'502')
except RetryError as e:
print('--> This operation was retried and hit the maximum number of retries.')
except HTTPError as e:
print('--> This operation returned a {} error without retrying'.format(e.response.status_code))
try:
print('POSTing a 502')
client.post(server+'502')
except RetryError as e:
print('--> This operation was retried and hit the maximum number of retries.')
except HTTPError as e:
print('--> This operation returned a {} error without retrying'.format(e.response.status_code))
try:
print('PUTing a 502')
client.put(server+'502')
except RetryError as e:
print('--> This operation was retried and hit the maximum number of retries.')
except HTTPError as e:
print('--> This operation returned a {} error without retrying'.format(e.response.status_code))
try:
print('PATCHing a 502')
client.patch(server+'502')
except RetryError as e:
print('--> This operation was retried and hit the maximum number of retries.')
except HTTPError as e:
print('--> This operation returned a {} error without retrying'.format(e.response.status_code))
Which for ligo-gracedb-2.11.0
returns:
GETing a 502
--> This operation was retried and hit the maximum number of retries.
POSTing a 502
--> This operation returned a 502 error without retrying
PUTing a 502
--> This operation was retried and hit the maximum number of retries.
PATCHing a 502
--> This operation returned a 502 error without retrying
The fix should be: extend the retry functionality for POST
and PATCH
and the list of allowed retry codes should be pared down for safety.