For further clarity, we're on Elasticsearch 7. On Tue, Jan 12, 2021 at 4:53 PM Rex Fenley <r...@remind101.com> wrote:
> Correction, by HTTP timeout I mean BulkProcessor timeout. > > On Tue, Jan 12, 2021 at 4:40 PM Rex Fenley <r...@remind101.com> wrote: > >> Hello, >> >> For the past number of days we've been investigating continuous >> SocketTimeoutException like the following: >> >> 2021-01-12 20:53:42,105 DEBUG org.elasticsearch.client.RestClient >> - request [POST <our_host>:<port>/_bulk?timeout=1m >> <http://graph-jvm-groups-es.empire/_bulk?timeout=1m>] failed >> java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection >> http-outgoing-2 [ACTIVE] >> at java.base/java.lang.Thread.run(Thread.java:829) >> 2021-01-12 20:53:42,106 DEBUG org.elasticsearch.client.RestClient >> - added [[host=<our_host>:<port> >> <http://graph-jvm-groups-es.empire/>]] to blacklist >> >> It appears that we're hitting a socket timeout and then the client adds >> our ES host to a blacklist which therefore fails any further request to ES >> and eventually fails our job. >> >> Following from information we've gathered over the internet, we should be >> able to set the SocketTimeout to something higher than the HTTP timeout and >> it may fix the behavior we're seeing. However, we have not found a way to >> configure SocketTimeout from flink, what we want to set can be found here >> [1]. How do we set this from the Elasticsearch SQL connector? >> >> [1] >> https://github.com/elastic/elasticsearch/blob/67be92c72386848ef52f029248b350ddaba0fd7e/client/rest/src/main/java/org/elasticsearch/client/RestClientBuilder.java#L213 >> >> Thanks! >> >> -- >> >> Rex Fenley | Software Engineer - Mobile and Backend >> >> >> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> >> | FOLLOW US <https://twitter.com/remindhq> | LIKE US >> <https://www.facebook.com/remindhq> >> > > > -- > > Rex Fenley | Software Engineer - Mobile and Backend > > > Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> | > FOLLOW US <https://twitter.com/remindhq> | LIKE US > <https://www.facebook.com/remindhq> > -- Rex Fenley | Software Engineer - Mobile and Backend Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> | FOLLOW US <https://twitter.com/remindhq> | LIKE US <https://www.facebook.com/remindhq>
