Hello folks,
This is quite strange. We see a TM stop reporting metrics
to DataDog .The logs from that specific TM for every DataDog dispatch time
out with* java.net.SocketTimeoutException: timeout *and that seems to
repeat over every dispatch to DataDog. It seems it is on a 10 seconds
cadence per container. The TM remains humming, so does not seem to be under
memory/CPU distress. And the exception is *not* transient. It just stops
dead and from there on timeout.
Looking at SLA provided by DataDog any throttling exception should pretty
much not be a SocketTimeOut, till of course the reporting the specific
issue is off. This thus appears very much a n/w issue which appears weird
as other TMs with the same n/w just hum along, sending their metrics
successfully. The other issue could be just the amount of metrics and the
current volume for the TM is prohibitive. That said the exception is still
not helpful.
Any ideas from folks who have used DataDog reporter with Flink. I guess
even best practices may be a sufficient beginning.
Regards.
