Hi Dario, I'm not a Java developer so I don't have experience with Java clients or JavaBinCodec. But the principle remains the same: flushing 200 documents at a time instead of 10 is a 95% reduction in the number of requests fired.
If your documents are all similarly sized, I wouldn't bother with calculating the actual buffer size in bytes. Thomas Op di 13 mei 2025 om 19:51 schreef <dario.v...@coop.ch.invalid>: > Hi Thomas, > > As I understood it there is the ConcurrentUpdateSolrClient for this. Am I > better off doing this myself or should I just use the concurrent client? > Is it correct that I could reach your described scenario by setting > withQueueSize to 200 (default is 10) and setting > "solr.cloud.client.stallTime" to 60000 (default is 15 seconds). > I am not sure if this is really leading to the scenario you describe. > And if I want to measure the buffer size not in respect to the number of > documents in the buffer but in respect to the actual size that the buffer > would have when using JavaBinCodec I would need my own batcher anyway, > right? > > Dario > > From: Thomas Corthals <tho...@klascement.net> > Date: Tuesday, May 13, 2025 at 2:20?PM > To: users@solr.apache.org <users@solr.apache.org> > Subject: Re: GOAWAY signal > CAUTION: This is an external email. Do not click any links or open any > attachments unless you trust the sender and know the content is safe. > > > Hi Dario, > > Regardless of this GOAWAY signal, I would advise to buffer the documents on > the client side instead of indexing them individually for every event. > You'll have to try out which numbers work for your specific document size > and timeliness expectations. I usually start with a buffer size of 200 > documents, could be higher for very small documents or lower for very large > ones. When that amount is reached, flush them to Solr in a single request. > If you can't have documents show up "late", you can additionally flush your > buffer after e.g. 1 minute even if it has fewer documents. > > Thomas > > Op di 13 mei 2025 om 11:02 schreef <dario.v...@coop.ch.invalid>: > > > Dear Solr People > > > > On our system we have an indexer-service that indexes documents based on > > incoming events. Sometimes a lot of these events happen at the same time > > (or at least very close in time to each other). If that happens our > indexer > > receives GOAWAY signals from solr. > > We're using the HttpJdkSolrClient with HTTP/2. Solr Version is 9.5.0 > > > > When exactly do these GOAWAY signals show up. In other words, how many > > requests need to be made to solr in a certain amount of timespan that > solr > > starts emitting this signal. And how exactly should it be handled or > fixed. > > > > Should we switch to HTTP/1.1? This version of HTTP does not specify this > > signal, but this is probably not a solution? > > Using another client? > > > > Can there be done anything else about it? Will the documents that were in > > the request that caused the GOAWAY signal still get added to the index or > > do we need to resend this request? And If yes, how long should we wait to > > resend it? > > > > With kind regards, > > > > Dario > > >
