The cluster is not really overloaded and also couldn't find some ES errors logs (atleast something that is repeating) The job IS very busy (100% on the sink and backpressure) but I think its due to the reactor exceptions being thrown
what do you mean by incorrect batching? On 2021/10/13 07:43:38, Itamar Syn-Hershko <ita...@bigdataboutique.com> wrote: > Hi Oran, can you check your ES logs / metrics? > > Most issues we see with the ES sink are around incorrect batching and/or > overloaded clusters. Could it be your ES write queue is building up? > > On Wed, Oct 13, 2021 at 1:06 AM Oran Shuster <oran.shus...@houzz.com> wrote: > > > Flink version 1.13.1 > > ES Version 7.12.0 > > Flink deployment type - session cluster > > Scala version 2.12 > > > > Same job used to run with elasticsearch6 connector with about the same > > load and had no issues > > Since moving to elasticsearch7 more and more exceptions were being thrown > > about reactor being stopped in our logs > > > > [IllegalStateException] with message [Request cannot be executed; I/O > > reactor status: STOPPED] with rest status -1 > > > > Looking online for this type of error gave me some results from people > > instantiating the ES client on their own which is irrelevant for our case. > > Also found that it might be caused by an uncaught exception in the failure > > handler, so i wrapped all the code in try..catch but found no uncaught > > exceptions > > > > Next thing I tried was to force using the updated version of ES rest > > client with the same version as my cluster 7.12. That didn't seem to fix > > any issues > > > > Problem is that once this happens it does not fix itself. That instance of > > the sink will continuously get those exceptions until we restart the job. I > > tried to look for any logs before that but couldn't find anything > > > > > -- > [image: logo] <https://bigdataboutique.com/> > Itamar Syn-Hershko > CTO, Founder > Email: ita...@bigdataboutique.com > Website: https://bigdataboutique.com > <https://twitter.com/synhershko> <https://github.com/synhershko> > <https://www.linkedin.com/in/itamar-syn-hershko-78b25013/> > <https://www.youtube.com/channel/UCBHr7lM2u6SCWPJvcKug-Yg> >
