internal kafka error: KafkaError{code=_TIMED_OUT,val=-185,str="GroupCoordinator: kb-2.prod.hop.sc...

Error Details:

• Sentry event: https://sentry.io/ligo-caltech/gwcelery/issues/3646334060
• First seen: 2022-10-05T07:27:21+00:00
• Last seen: 2022-10-05T08:43:24+00:00
• Events: 24
• Users: 0

I'll leave a link to the conversation with Chris: https://github.com/astronomy-commons/adc-streaming/pull/59#issuecomment-1265941475

To add on to this, it wasn't obvious to me at first but I think you're right and this is connected to #441 (closed). Figured I'd tag it here so both issues have a record of each other.

I think this issue is something that will take time to work out, so I don't think we should hold up releasing v1.0.2 for this to be solved.

It will saturate the Sentry quota. If the client is automatically recovering, then at least we should catch and ignore this error with a FIXME comment.

Okay I'll submit a MR for that in a minute after I finish what I'm currently working on (another short MR).

can we at all put a try except for this error. There is no traceback. It's coming from the adc-streaming logger.

We can't catch the error with try/except, but we can ignore the noisy logger.

I modified the error callback function in !949 (merged), I'm waiting to close this issue though to make sure the error is now showing up in the log but not triggering an error in sentry.

Turns out this is still coming from the igwn alert client, not from the kafka producer bootstep. @deep.chatterjee do you have any idea why the try except you added before isn't working anymore? I'm wondering if it actually never worked and we just didn't notice. For example, the last instance of Timeout from kafka that I see in gwcelery-worker.log is 2022-09-02, 3 days after !907 (merged) was merged.

I should have checked before that these problems were coming from the kafka bootstep. Should we revert !949 (merged) since it's coming from the igwn alert client?

Are we certain it wasn't coming from both?

Sorry I realized after posting I hadn't stated why I know it's not coming from both, and was editing when you responded.

I confirmed it's only coming from the IGWN alert client by grepping the logs for _TIMED_OUT on playground. The only log file that has these is gwcelery-worker.log, and I noticed that all of the lines I could see came from the IGWNReceiverThread. I then removed all of those with grep -r '_TIMED_OUT' gwcelery-worker.log | grep -v 'IGWNReceiverThread'. The last line of that was adc.errors.KafkaException: Error communicating with Kafka: code=_TIMED_OUT GroupCoordinator: kb-0.prod.hop.scimma.org:9092: 99 request(s) timed out: disconnect (after 60045ms in state UP). The last time the pattern adc.errors.KafkaException showed up in the log was 2022-09-02, way before !852 (merged) was merged.

Should we just ignore the noisy logger for this release?

That's my suggestion, yes.

Okay, I'm currently working on both fixing the versioned filename issue and the updates you wanted in !945 (merged), do you have a minute to prepare a MR for this? If not I'll do it once I finish these

No, I'm pretty busy right now. Sorry, catching up on referee reports and other such busy work.

No worries, I'll get it done.

Should we revert !949 (merged)?

Yes! Good point.

mentioned in merge request !949 (merged)

mentioned in merge request !958 (merged)

mentioned in merge request !964 (merged)

mentioned in merge request !987 (merged)

mentioned in issue #686 (closed)

I think our workarounds make this not a problem anymore, or at least this doesn't crash our kafka consumer any more so I'm going to close this.

closed