Commits need to happen or you'll get a memory leak. There are parts of the transaction log which are held in memory for real-time get. Commits release this memory as indexing occurs. I'm not aware of way to disable this behavior.
Joel Bernstein http://joelsolr.blogspot.com/ On Wed, Mar 17, 2021 at 6:47 PM Karl Stoney <karl.sto...@autotrader.co.uk.invalid> wrote: > Bit of a hacky workaround.... but: > > > * create a collection from the target config name before reindex > * set property on that (which writes overlay to zookeeper) > * delete the collection > * run reindex > * unset-property on the created collection > > ________________________________ > From: Karl Stoney <karl.sto...@autotrader.co.uk> > Sent: 17 March 2021 22:20 > To: users@solr.apache.org <users@solr.apache.org> > Subject: Re: Disable commits during a REINDEXCOLLECTION > > That's the source collection, I'm referring to the destination. > > @David - seeing as REINDEX creates the target collection, I had to start > the process and then attempt to set the config on the destination. > However setting the destination config kills the REINDEX: > > ``` > 22:18:40.389 > [DaemonStream-karl-test-20015-thread-1-processing-n:solr-0.search-solr.svc.cluster.local:80_solr > x:at-uk-002_shard1_replica_n1 c:at-uk-002 s:shard1 r:core_node2] ERROR > org.apache.solr.client.solrj.io.stream.DaemonStream - Err > or in DaemonStream: karl-test > java.io.IOException: org.apache.solr.common.SolrException: Could not find > a healthy node to handle the request. > at > org.apache.solr.client.solrj.io.stream.TopicStream.persistCheckpoints(TopicStream.java:472) > ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp > - 2021-02-16 15:52:04] > at > org.apache.solr.client.solrj.io.stream.TopicStream.close(TopicStream.java:342) > ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp > - 2021-02-16 15:52:04] > at > org.apache.solr.client.solrj.io.stream.PushBackStream.close(PushBackStream.java:75) > ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp > - 2021-02-16 15:52:04] > at > org.apache.solr.client.solrj.io.stream.UpdateStream.close(UpdateStream.java:147) > ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp > - 2021-02-16 15:52:04] > at > org.apache.solr.client.solrj.io.stream.CommitStream.close(CommitStream.java:155) > ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp > - 2021-02-16 15:52:04] > at > org.apache.solr.client.solrj.io.stream.DaemonStream$StreamRunner.stream(DaemonStream.java:380) > ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp > - 2021-02-16 15:52:04] > ``` > > If i remove the set-config; it's fine. > ________________________________ > From: Vivaldi <uyil...@vivaldi.net.INVALID> > Sent: 17 March 2021 21:59 > To: users@solr.apache.org <users@solr.apache.org> > Subject: Re: Disable commits during a REINDEXCOLLECTION > > I thought the REINDEXCOLLECTION puts the source collection in read-only > mode, am I wrong? Doesn’t that alsı disable commits? > > Sent from my iPhone > > > On 18 Mar 2021, at 00:28, Karl Stoney <karl.sto...@autotrader.co.uk.invalid> > wrote: > > > > Can you believe I had never come across that before! Thanks! > > > > Every day’s a school day > > > > Get Outlook for iOS< > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukef&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7Ce328a4cc735f49dc70a708d8e9900e49%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637516152316874603%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=hAgyg3QJVOFoukpXl6TlulQHhtOYASlxhiUvnY5O1Ac%3D&reserved=0 > > > > ________________________________ > > From: David Smiley <dsmi...@apache.org> > > Sent: Wednesday, March 17, 2021 8:48:23 PM > > To: users@solr.apache.org <users@solr.apache.org> > > Subject: Re: Disable commits during a REINDEXCOLLECTION > > > > Hi Karl, > > > > Look into the "config apI". Let us know how it goes! > > Ideally, this feature would do this automatically. > > > > ~ David Smiley > > Apache Lucene/Solr Search Developer > > > https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fdavidwsmiley&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7Ce328a4cc735f49dc70a708d8e9900e49%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637516152316874603%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=7fZ17ijzSk3rs34K01tJ4SjP4kV4XU6Md8yzpcnu1jw%3D&reserved=0 > > > > > > On Wed, Mar 17, 2021 at 2:02 PM Karl Stoney > > <karl.sto...@autotrader.co.uk.invalid> wrote: > > > >> Hi all, > >> We're wanting to use REINDEXCOLLECTION, but our config has a relatively > >> aggressive autoCommit interval configured by default (intentionally). > >> > >> Ideally I'd like to be able to disable hard commits for the duration of > >> the reindex, but can't see a way to do that without pushing a whole new > >> config and reloading that collection. > >> > >> Does anyone know of a ninja way at runtime to disable autoCommits on a > >> collection (solr cloud)? > >> > >> Thanks > >> Unless expressly stated otherwise in this email, this e-mail is sent on > >> behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place, > >> Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). > Auto > >> Trader Limited is part of the Auto Trader Group Plc group. This email > and > >> any files transmitted with it are confidential and may be legally > >> privileged, and intended solely for the use of the individual or entity > to > >> whom they are addressed. If you have received this email in error please > >> notify the sender. This email message has been swept for the presence of > >> computer viruses. > >> > > Unless expressly stated otherwise in this email, this e-mail is sent on > behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place, > Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto > Trader Limited is part of the Auto Trader Group Plc group. This email and > any files transmitted with it are confidential and may be legally > privileged, and intended solely for the use of the individual or entity to > whom they are addressed. If you have received this email in error please > notify the sender. This email message has been swept for the presence of > computer viruses. > > Unless expressly stated otherwise in this email, this e-mail is sent on > behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place, > Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto > Trader Limited is part of the Auto Trader Group Plc group. This email and > any files transmitted with it are confidential and may be legally > privileged, and intended solely for the use of the individual or entity to > whom they are addressed. If you have received this email in error please > notify the sender. This email message has been swept for the presence of > computer viruses. >
