Hello,
For the past number of days we've been investigating continuous
SocketTimeoutException like the following:
2021-01-12 20:53:42,105 DEBUG org.elasticsearch.client.RestClient
- request [POST
<our_host>:<port>/_bulk?timeout=1m
<http://graph-jvm-groups-es.empire/_bulk?timeout=1m>] failed
java.net.SocketTimeoutException: 30,000 milliseconds timeout on
connection http-outgoing-2 [ACTIVE]
at java.base/java.lang.Thread.run(Thread.java:829)
2021-01-12 20:53:42,106 DEBUG org.elasticsearch.client.RestClient
- added [[host=<our_host>:<port>
<http://graph-jvm-groups-es.empire/>]] to blacklist
It appears that we're hitting a socket timeout and then the client adds our
ES host to a blacklist which therefore fails any further request to ES and
eventually fails our job.
Following from information we've gathered over the internet, we should be
able to set the SocketTimeout to something higher than the HTTP timeout and
it may fix the behavior we're seeing. However, we have not found a way to
configure SocketTimeout from flink, what we want to set can be found here
[1]. How do we set this from the Elasticsearch SQL connector?
[1]
https://github.com/elastic/elasticsearch/blob/67be92c72386848ef52f029248b350ddaba0fd7e/client/rest/src/main/java/org/elasticsearch/client/RestClientBuilder.java#L213
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
