There is no need to do a delete before an update. The update will replace the previous document.
The delete by query is an expensive operation that stops all updates. Stop doing that. See if the problem goes away when the delete by query is removed. You will also need to change to IDs which are stored in the database. Having Solr generate the IDs makes it impossible to update the documents. I’ve used Solr in production for over 15 years and I’ve never had Solr generate the IDs. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Apr 8, 2024, at 3:46 AM, Yadav, Rishabh <rishab.ya...@esko.com.INVALID> > wrote: > > Hi Team, > > We have built an Index Queue Mechanism where we store the Ids of the > Documents that needs reindexing as some of the data has been changed recently > by the user. > A Cron job runs in the background which keeps monitoring the queue every 5 > seconds and looks out for any new ids that are added to the queue. then it > picks up those ids and tries to reindex them to solr. > For Reindexing, first it deletes the existing documents from solr, then it > fetches the latest details from the database and then indexes it back to > solr. For deleting, we use the deleteByQuery method. We could not use > deleteById as fetching the Ids of the Docs is hard as they are uniquely > generated by Solr itself. We are Committing the changes manually by calling > the solrClient.commit(collectionName, true, true). Things were working pretty > fine up until a few days ago. recently they have started failing for prod > server. > > I have been recently Facing an issue with one of my prod instances where I am > constantly getting an error like > "Task queue processing has stalled for 20121 ms with 0 remaining elements to > process". > my application is not able to perform any kind of indexing after this and > even the search results are inconsistent now. The Same query is returning > different results > every time we hit it. > I am not able to see the same above issue in my other test environment where > I have a similar type of setup with the same amount of data. > > Below are the configuration details of our Solr setup > Solr Version : 8.11.2 > Solrj Version : 8.11.2 > Solr Is Running in Cloud mode with 3 shards and 2 Replica architecture. > > Some things that we noticed through logs and other forums is: > > 1. We are using deleteByQuery Method to delete the existing Solr Documents > 2. We have not implemented autoCommit, autoSoftCommit, idleTimeout, > socketTimeout, stallTimeout configurational settings. > 3. we are doing everything using a manual hard commit through the solrj > from my Application. This is done so that we can track the progress of how > many documents are indexed and how many are remaining. > > I saw a similar Issue existing before in solr 8.4 versions but that got > resolved with solr 8.4. But I can still see the issue happening. I understand > that I should not do manual commits but as this was our first release and we > are only improving the setup from here, i wanted to know if there is any > configuration that can be included to fix this error. Making the code change > is not possible right now as giving the code change can take around month to > reach the customer as the release but for now is there anything that we can > do to fix this issue so that the indexing can start again in solr. > > Thanks in Advance for the Help. > > > Rishabh Yadav > Software Engineer 1 > Esko Graphics Pvt Ind Ltd > > > Please be advised that this email may contain confidential information. If > you are not the intended recipient, please notify us by email by replying to > the sender and delete this message. The sender disclaims that the content of > this email constitutes an offer to enter into, or the acceptance of, any > agreement; provided that the foregoing does not invalidate the binding effect > of any digital or other electronic reproduction of a manual signature that is > included in any attachment. [0xC3D2] > > Confidential - Company Proprietary
