Yes, I did. With the use of CloudClollection Utility. On Tue, 23 Mar 2021 at 12:01 AM, Dominique Bejean <dominique.bej...@eolya.fr> wrote:
> Hi, > > Did you try to force a hard commit in order to see the inpact on tlog ? > http://localhost:8983/solr/[collection_name]/update?commit=true > > Did you read this article ? > > https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ > > Regards > > Dominique > > > Le lun. 22 mars 2021 à 17:29, Ritvik Sharma <ritvik.s...@gmail.com> a > écrit : > >> Hi Dominique >> >> Any suggestions? >> >> On Mon, 22 Mar 2021 at 10:23, Ritvik Sharma <ritvik.s...@gmail.com> >> wrote: >> >>> HI Dominique >>> >>> softcommit=false is coming in logs, >>> >>> >>> INFO - 2021-03-22 07:40:58.129; [c:solrcollection s:shard2 r:core_node1 >>> x:solrcollection_shard2_replica1] >>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor; >>> [solrcollection_shard2_replica1] webapp=/solr path=/update >>> params={update.distrib=FROMLEADER&waitSearcher=true& >>> *openSearcher=true&commit=true&softCommit=false*&distrib.from= >>> http://xxxx:8983/solr/solrcollection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} >>> 0 676 >>> INFO - 2021-03-22 07:40:58.361; [c:solrcollection s:shard2 r:core_node1 >>> x:solrcollection_shard2_replica1] >>> org.apache.solr.update.DirectUpdateHandler2; start >>> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} >>> >>> >>> On Sun, 21 Mar 2021 at 21:06, Dominique Bejean < >>> dominique.bej...@eolya.fr> wrote: >>> >>>> Hi, >>>> >>>> If your tlogs grow, It looks like hard commit are never executed. >>>> >>>> Are you seeing hard commits in logs (grep 'softCommit=false') ? >>>> >>>> What is it happening if you force a hard commit ? Are tlog truncated ? >>>> http://localhost:8983/solr/[collection_name]/update?commit=true >>>> >>>> Did you read this article ? >>>> >>>> https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ >>>> >>>> Regards >>>> >>>> Dominique >>>> >>>> >>>> Le dim. 21 mars 2021 à 05:22, Ritvik Sharma <ritvik.s...@gmail.com> a >>>> écrit : >>>> >>>>> Hi Dominique >>>>> >>>>> Heavy query means high query rate on solr. >>>>> >>>>> Honestly for 2-3 days we have stopped the queries on solr only we are >>>>> doing indexing !!! >>>>> As you see we have enabled Hard commit also to decrease tlog log size >>>>> or more and the same is written on solr docs. Still the same behaviour is >>>>> occurring on replicas. In some replicas, tlog size is of 300 GB. >>>>> >>>>> We may increase softcommit timeout but as per my understanding it only >>>>> helps in searching availability prior to commit to indexes. >>>>> >>>>> On Sun, 21 Mar 2021 at 4:29 AM, Dominique Bejean < >>>>> dominique.bej...@eolya.fr> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> By heavy query, do you mean you have a high query rate and/or you need >>>>>> index update be available fast after few seconds (NRT search) ? >>>>>> >>>>>> Do you see the hard commits in logs ? >>>>>> Can you try to increase autoSoftCommit to 30 seconds or more ? >>>>>> >>>>>> Regards >>>>>> >>>>>> Dominique >>>>>> >>>>>> Le sam. 20 mars 2021 à 18:53, Ritvik Sharma <ritvik.s...@gmail.com> a >>>>>> écrit : >>>>>> >>>>>> > Hi Guys >>>>>> > >>>>>> > We have faced an issue where tlog size is increasing unnecessarily. >>>>>> We are >>>>>> > using a "heavy indexing, heavy query" approach. We enabled hard >>>>>> commit >>>>>> > also, >>>>>> > >>>>>> > solr cloud: 6.6 >>>>>> > zk: 3.4.10 >>>>>> > shards: 2, replication factor= 2 >>>>>> > >>>>>> > >>>>>> > solrconfig, >>>>>> > >>>>>> > <autoCommit> >>>>>> > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> >>>>>> > <maxDocs>10000</maxDocs> >>>>>> > <openSearcher>false</openSearcher> >>>>>> > >>>>>> > >>>>>> > </autoCommit> >>>>>> > >>>>>> > <!-- softAutoCommit is like autoCommit except it causes a >>>>>> > 'soft' commit which only ensures that changes are visible >>>>>> > but does not ensure that data is synced to disk. This is >>>>>> > faster and more near-realtime friendly than a hard commit. >>>>>> > --> >>>>>> > >>>>>> > <autoSoftCommit> >>>>>> > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> --> >>>>>> > </autoSoftCommit> >>>>>> > >>>>>> > In every replica, tlog is increasing for more than 200GB which >>>>>> exhaust disk >>>>>> > space. Please suggest someting >>>>>> > >>>>>> > On Fri, 19 Mar 2021 at 16:36, Ritvik Sharma <ritvik.s...@gmail.com> >>>>>> wrote: >>>>>> > >>>>>> > > Hi Guys >>>>>> > > >>>>>> > > We have faced an issue where tlog size is increasing >>>>>> unnecessarily. We >>>>>> > are >>>>>> > > using a "heavy indexing, heavy query" approach. We enabled hard >>>>>> commit >>>>>> > > also, >>>>>> > > >>>>>> > > solr cloud: 6.6 >>>>>> > > zk: 3.4.10 >>>>>> > > shards: 2, replication factor= 2 >>>>>> > > >>>>>> > > >>>>>> > > solrconfig, >>>>>> > > >>>>>> > > <autoCommit> >>>>>> > > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> >>>>>> > > <maxDocs>10000</maxDocs> >>>>>> > > <openSearcher>false</openSearcher> >>>>>> > > >>>>>> > > >>>>>> > > </autoCommit> >>>>>> > > >>>>>> > > <!-- softAutoCommit is like autoCommit except it causes a >>>>>> > > 'soft' commit which only ensures that changes are visible >>>>>> > > but does not ensure that data is synced to disk. This is >>>>>> > > faster and more near-realtime friendly than a hard >>>>>> commit. >>>>>> > > --> >>>>>> > > >>>>>> > > <autoSoftCommit> >>>>>> > > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> --> >>>>>> > > </autoSoftCommit> >>>>>> > > >>>>>> > > >>>>>> > >>>>>> >>>>>
