Hi Oran, could you check if smaller batches improve the situation?
On Sat, Oct 16, 2021 at 2:15 AM Oran Shuster <[email protected]> wrote: > The cluster is not really overloaded and also couldn't find some ES errors > logs (atleast something that is repeating) > The job IS very busy (100% on the sink and backpressure) but I think its > due to the reactor exceptions being thrown > > what do you mean by incorrect batching? > > On 2021/10/13 07:43:38, Itamar Syn-Hershko <[email protected]> > wrote: > > Hi Oran, can you check your ES logs / metrics? > > > > Most issues we see with the ES sink are around incorrect batching and/or > > overloaded clusters. Could it be your ES write queue is building up? > > > > On Wed, Oct 13, 2021 at 1:06 AM Oran Shuster <[email protected]> > wrote: > > > > > Flink version 1.13.1 > > > ES Version 7.12.0 > > > Flink deployment type - session cluster > > > Scala version 2.12 > > > > > > Same job used to run with elasticsearch6 connector with about the same > > > load and had no issues > > > Since moving to elasticsearch7 more and more exceptions were being > thrown > > > about reactor being stopped in our logs > > > > > > [IllegalStateException] with message [Request cannot be executed; I/O > > > reactor status: STOPPED] with rest status -1 > > > > > > Looking online for this type of error gave me some results from people > > > instantiating the ES client on their own which is irrelevant for our > case. > > > Also found that it might be caused by an uncaught exception in the > failure > > > handler, so i wrapped all the code in try..catch but found no uncaught > > > exceptions > > > > > > Next thing I tried was to force using the updated version of ES rest > > > client with the same version as my cluster 7.12. That didn't seem to > fix > > > any issues > > > > > > Problem is that once this happens it does not fix itself. That > instance of > > > the sink will continuously get those exceptions until we restart the > job. I > > > tried to look for any logs before that but couldn't find anything > > > > > > > > > -- > > [image: logo] <https://bigdataboutique.com/> > > Itamar Syn-Hershko > > CTO, Founder > > Email: [email protected] > > Website: https://bigdataboutique.com > > <https://twitter.com/synhershko> <https://github.com/synhershko> > > <https://www.linkedin.com/in/itamar-syn-hershko-78b25013/> > > <https://www.youtube.com/channel/UCBHr7lM2u6SCWPJvcKug-Yg> > > >
