Hi Nikhilesh, It's not necessary to stop traffic, it's just that the update traffic will make replication more difficult, which is to be expected. Now, if you have A LOT of update traffic and recovery is slow because of it, it would help to pause the traffic for sure.
Best regards, Radu -- Elasticsearch/OpenSearch & Solr Consulting, Production Support & Training Sematext Cloud - Full Stack Observability https://sematext.com/ <http://sematext.com/> On Thu, Jul 7, 2022 at 6:41 AM Nikhilesh Jannu <nikhil...@predictspring.com> wrote: > Hi Radu, > > I will check again when this happens if the recovery is going on. So if the > recovery is going on should we be make sure there is no traffic sending > data to be indexed to the collection ? > > Regards, > Nikhilesh Jannu > > > > Nikhilesh Jannu Principal Software Engineer 405.609.4259 > <(405)%20741-9895> > > > On Wed, Jul 6, 2022 at 1:51 AM Radu Gheorghe <radu.gheor...@sematext.com> > wrote: > > > Hi Nikhilesh, > > > > Something sounds off, your transaction logs should be rotated unless > > there's a recovery going on at that time. Is it? > > > > If not, can you double check: > > - your solrconfig.xml - maybe you can post the relevant part here? > > - your config overlay - maybe there's something that was posted by the > > Config API that overwrote your solconfig settings? > > > > Best regards, > > Radu > > -- > > Elasticsearch/OpenSearch & Solr Consulting, Production Support & Training > > Sematext Cloud - Full Stack Observability > > http://sematext.com/ > > > > > > On Tue, Jul 5, 2022 at 9:46 PM Nikhilesh Jannu < > > nikhil...@predictspring.com> > > wrote: > > > > > Hi Radu, > > > > > > The issue happened again today and I see huge logs getting created even > > > with the config where we want smaller files and smaller intervals. > > > > > > This happens in a particular scenario where we are trying to index > > > documents. That API call is timing out and there is a connection > > > expectation. The Tlogs keep growing in size. The collection is healthy > > but > > > we are constantly adding documents and there are no errors. Only part I > > see > > > is the Tlogs are now at 4GB or so. > > > > > > Regards, > > > Nikhilesh Jannu > > > > > > Regards, > > > Nikhilesh Jannu > > > Nikhilesh Jannu Principal Software Engineer 405.609.4259 > > > <(405)%20741-9895> > > > > > > > > > On Wed, Jun 29, 2022 at 10:15 PM Radu Gheorghe < > > radu.gheor...@sematext.com > > > > > > > wrote: > > > > > > > You're welcome, Nikhilesh! > > > > > > > > Best wishes, > > > > Radu > > > > -- > > > > Elasticsearch/OpenSearch & Solr Consulting, Production Support & > > Training > > > > Sematext Cloud - Full Stack Observability > > > > http://sematext.com/ > > > > > > > > > > > > On Wed, Jun 29, 2022 at 10:49 AM Nikhilesh Jannu < > > > > nikhil...@predictspring.com> wrote: > > > > > > > > > Thank you Radu for the quick response. I have updated the values as > > > > > you suggested. > > > > > > > > > > > > > > > Nikhilesh Jannu Principal Software Engineer 405.609.4259 > > > > > <(405)%20741-9895> > > > > > > > > > > > > > > > On Wed, Jun 29, 2022 at 12:03 AM Radu Gheorghe < > > > > radu.gheor...@sematext.com > > > > > > > > > > > wrote: > > > > > > > > > > > Hi Nikhilesh, > > > > > > > > > > > > Try hard-committing more often. This way you'll have smaller tlog > > > files > > > > > and > > > > > > there will be less data to recover. My suggestion is to add a > > maxSize > > > > > > constraint to autoCommit. 100MB is a good rule of thumb, makes > sure > > > you > > > > > > don't replay more than 100MB worth of data (even if you have an > > > > indexing > > > > > > spike). > > > > > > > > > > > > Best regards, > > > > > > Radu > > > > > > -- > > > > > > Elasticsearch/OpenSearch & Solr Consulting, Production Support & > > > > Training > > > > > > Sematext Cloud - Full Stack Observability > > > > > > http://sematext.com/ > > > > > > > > > > > > > > > > > > On Wed, Jun 29, 2022 at 9:10 AM Nikhilesh Jannu < > > > > > > nikhil...@predictspring.com> > > > > > > wrote: > > > > > > > > > > > > > Dear Users, > > > > > > > > > > > > > > We are using the SOLR TRA collection for capturing the logs. > We > > > are > > > > > > > writing the logs to SOLR using the Rest API in a batch of 100 > and > > > > also > > > > > we > > > > > > > are using SOFT commit interval of 15000 and Hard commit > interval > > of > > > > > > 60000. > > > > > > > > > > > > > > Solr Version : 8.11.1. > > > > > > > > > > > > > > When we restart the SOLR node in the cloud the current day's > > > > collection > > > > > > > goes in recovery mode and we see the following logs. It takes a > > > long > > > > > time > > > > > > > for the recovery process to complete. Not sure how to avoid it. > > Any > > > > > > > suggestions ? > > > > > > > > > > > > > > Sample of the logs below. > > > > > > > > > > > > > > 2022-06-29 06:03:27.500 INFO > > > > > > > (recoveryExecutor-67-thread-1-processing-n:10.0.42.157:8983 > > _solr > > > > > > > x:logs__TRA__2022-06-29_shard1_replica_n1 > c:logs__TRA__2022-06-29 > > > > > > s:shard1 > > > > > > > r:core_node2) [c:logs__TRA__2022-06-29 s:shard1 r:core_node2 > > > > > > > x:logs__TRA__2022-06-29_shard1_replica_n1] o.a.s.u.UpdateLog > log > > > > replay > > > > > > > status > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tlog{file=/var/solr/data/logs__TRA__2022-06-29_shard1_replica_n1/data/tlog/tlog.0000000000000000243 > > > > > > > refcount=3} active=false starting pos=0 current pos=1119002110 > > > > current > > > > > > > size=3287152529 % read=34.0 > > > > > > > > > > > > > > Regards, > > > > > > > Nikhilesh Jannu > > > > > > > > > > > > > > > > > > > > > > > > > > > >
