Python server puts content in Python server log files for dates other than the current date
The Python server often puts entries into log files that are labeled with dates other than the current date, so a given date's log file will have entries from numerous dates, and entries from a given date are spread over multiple log files.
Example: A search for GPS time 1376296184 in /var/log/http/ssl_request_log on segments-backup at ~4:00 PM EDT = 13:00 PDT on 2023.08.25 returns the following 5 entries, in 3 clusters:
[root@segments-backup httpd]# grep 1376296184 ssl_request_log
[25/Aug/2023:01:29:29 -0700] [redacted] TLSv1.2 - "GET /dq/K1/K1:GRD_SCIENCE_MODE:1/1?s=1376296184.0&e=1376987384&include=known HTTP/1.1" 145
[25/Aug/2023:06:10:12 -0700] [redacted] - - "GET /dq/K1/K1:GRD_SCIENCE_MODE:1/1?s=1376296184.0&e=1376987384&include=known HTTP/1.1" 231
[25/Aug/2023:06:10:12 -0700] [redacted] TLSv1.2 - "GET /dq/K1/K1:GRD_SCIENCE_MODE:1/1?s=1376296184.0&e=1376987384&include=known HTTP/1.1" 150
[25/Aug/2023:06:55:22 -0700] [redacted] - - "GET /dq/K1/K1:GRD_SCIENCE_MODE:1/1?s=1376296184.0&e=1376987384&include=known HTTP/1.1" 231
[25/Aug/2023:06:55:22 -0700] [redacted] TLSv1.2 - "GET /dq/K1/K1:GRD_SCIENCE_MODE:1/1?s=1376296184.0&e=1376987384&include=known HTTP/1.1" 150
A search for the same GPS time in the Python server log files, in /opt/dqsegdb/python_server/logs/2023-08-*.log, returns 11 corresponding entries in 2 different log files, neither of which are for the current date:
[root@segments-backup httpd]# grep 1376296184 /opt/dqsegdb/python_server/logs/2023-08-*.log
/opt/dqsegdb/python_server/logs/2023-08-21.log:2023-08-25 01:29:30,051:DEBUG:GET /dq/K1/K1:GRD_SCIENCE_MODE:1/1?s=1376296184.0&e=1376987384&include=known - Attempt made
/opt/dqsegdb/python_server/logs/2023-08-21.log:2023-08-25 01:29:30,053:DEBUG:GET /dq/K1/K1:GRD_SCIENCE_MODE:1/1?s=1376296184.0&e=1376987384&include=known - Attempt made (token authorised for dqsegdb.read)
/opt/dqsegdb/python_server/logs/2023-08-21.log:2023-08-25 01:29:30,059:WARNING:GET /dq/K1/K1:GRD_SCIENCE_MODE:1/1?s=1376296184.0&e=1376987384&include=known - Flag does not exist in the database
/opt/dqsegdb/python_server/logs/2023-08-21.log:2023-08-25 06:10:12,502:DEBUG:GET /dq/K1/K1:GRD_SCIENCE_MODE:1/1?s=1376296184.0&e=1376987384&include=known - Attempt made
/opt/dqsegdb/python_server/logs/2023-08-21.log:2023-08-25 06:10:12,502:ERROR:GET /dq/K1/K1:GRD_SCIENCE_MODE:1/1?s=1376296184.0&e=1376987384&include=known - Authorisation failure (No Authorization header)
/opt/dqsegdb/python_server/logs/2023-08-21.log:2023-08-25 06:10:12,502:ERROR:GET /dq/K1/K1:GRD_SCIENCE_MODE:1/1?s=1376296184.0&e=1376987384&include=known - Authentication failure (Unable to load certificate)
/opt/dqsegdb/python_server/logs/2023-08-21.log:2023-08-25 06:10:12,502:ERROR:GET /dq/K1/K1:GRD_SCIENCE_MODE:1/1?s=1376296184.0&e=1376987384&include=known - Authorisation failure (Unauthorised)
/opt/dqsegdb/python_server/logs/2023-08-24.log:2023-08-25 06:55:22,862:DEBUG:GET /dq/K1/K1:GRD_SCIENCE_MODE:1/1?s=1376296184.0&e=1376987384&include=known - Attempt made
/opt/dqsegdb/python_server/logs/2023-08-24.log:2023-08-25 06:55:22,862:ERROR:GET /dq/K1/K1:GRD_SCIENCE_MODE:1/1?s=1376296184.0&e=1376987384&include=known - Authorisation failure (No Authorization header)
/opt/dqsegdb/python_server/logs/2023-08-24.log:2023-08-25 06:55:22,862:ERROR:GET /dq/K1/K1:GRD_SCIENCE_MODE:1/1?s=1376296184.0&e=1376987384&include=known - Authentication failure (Unable to load certificate)
/opt/dqsegdb/python_server/logs/2023-08-24.log:2023-08-25 06:55:22,863:ERROR:GET /dq/K1/K1:GRD_SCIENCE_MODE:1/1?s=1376296184.0&e=1376987384&include=known - Authorisation failure (Unauthorised)
Note that entries from 2023.08.25 are saved in files 2023-08-21.log and 2023-08-24.log, and as of 13:15 PDT on 2023.08.25, there is no file 2023-08-25.log.
Checking open files shows that there are still 3 log files open:
[root@segments-backup httpd]# lsof | grep 2023-08-.*log
httpd 3021 apache 17w REG 253,0 11225920 67190080 /opt/dqsegdb/python_server/logs/2023-08-21.log
httpd 3022 apache 17w REG 253,0 11225920 67190080 /opt/dqsegdb/python_server/logs/2023-08-21.log
httpd 3026 apache 17w REG 253,0 11225920 67190080 /opt/dqsegdb/python_server/logs/2023-08-21.log
httpd 3028 apache 17w REG 253,0 11225920 67190080 /opt/dqsegdb/python_server/logs/2023-08-21.log
httpd 3029 apache 17w REG 253,0 11225920 67190080 /opt/dqsegdb/python_server/logs/2023-08-21.log
httpd 10835 apache 17w REG 253,0 11225920 67190080 /opt/dqsegdb/python_server/logs/2023-08-21.log
httpd 10970 apache 17w REG 253,0 11225920 67190080 /opt/dqsegdb/python_server/logs/2023-08-21.log
httpd 20387 apache 17w REG 253,0 821442 67190082 /opt/dqsegdb/python_server/logs/2023-08-24.log
httpd 22125 apache 17w REG 253,0 821442 67190082 /opt/dqsegdb/python_server/logs/2023-08-24.log
httpd 27802 apache 17w REG 253,0 1154701 67190081 /opt/dqsegdb/python_server/logs/2023-08-22.log
The code should probably strictly write entries to the log file corresponding to the date of the transaction in that line, regardless of when the query was received or answered.
Note: it would also be good to expand the tools/commands available to the DQSegDB team to investigate why certain files are open, from the lsof output.