[
https://issues.apache.org/jira/browse/SOLR-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lehong.ding updated SOLR-15408:
-------------------------------
Description:
h4. *Background:*
Before we moving to solr 8.8.1 from 7.7.2, we performed some performance test
on solr 8.8.1. We met a lot of concurrent update error in solr log.
*Envrironment:*
{{solrCloud with 3 cluster nodes with 500 collections, each has about 1m
documents.}}
(1 shard, 3 replica)
h4. *Threads:*
30 update/add threads + 10 deleteByQuery threads
h4. *Results:*
During deleteByQuery thread runing, only one node (lead node) has update
transactions, but other two node has none .
h4. Errrors:
java.io.IOException: Request processing has stalled for 20091ms with 100
remaining elements in the queue.java.io.IOException: Request processing has
stalled for 20091ms with 100 remaining elements in the queue. at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:449)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
at
org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:338)
at
org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:244)
at
org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribAdd(DistributedZkUpdateProcessor.java:300)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:230)
at
org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:245)
at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:106)
at
org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:110)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:343)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:291)
at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:338)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283) at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:244)
h4. Temporary Solution:
adding -Dsolr.http1=1 in solr start parameters
There are still some error in error log but the number is much small.
h4. My Questions:
1 We found solr cluster will eventually get the data consistent. What’s the
concurrent update error mainly impacted?
2 Adding -Dsolr.http1=1 in solr start parameters can reduce the error number.
Do we realy need add this parameter? And does this parameter will be kept in
later version?
Many Thanks.
was:
h4. *Background:*
Before we moving to solr 8.8.1 from 7.7.2, we performed some performance test
on solr 8.8.1. We met a lot of concurrent update error in solr log.
*Envrironment:*
{{3 cluster nodes with 500 collections, each has about 1m documents.}}
h4. *Threads:*
30 update/add threads + 10 deleteByQuery threads
h4. *Results:*
During deleteByQuery thread runing, only one node (lead node) has update
transactions, but other two node has none .
h4. Errrors:
java.io.IOException: Request processing has stalled for 20091ms with 100
remaining elements in the queue.java.io.IOException: Request processing has
stalled for 20091ms with 100 remaining elements in the queue. at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:449)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
at
org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:338)
at
org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:244)
at
org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribAdd(DistributedZkUpdateProcessor.java:300)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:230)
at
org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:245)
at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:106)
at
org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:110)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:343)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:291)
at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:338)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283) at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:244)
h4. Temporary Solution:
adding -Dsolr.http1=1 in solr start parameters
There are still some error in error log but the number is much small.
h4. My Questions:
1 We found solr cluster will eventually get the data consistent. What’s the
concurrent update error mainly impacted?
2 Adding -Dsolr.http1=1 in solr start parameters can reduce the error number.
Do we realy need add this parameter? And does this parameter will be kept in
later version?
Many Thanks.
> ConcurrentUpdate issue of solr 8.8.1
> ------------------------------------
>
> Key: SOLR-15408
> URL: https://issues.apache.org/jira/browse/SOLR-15408
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Affects Versions: 8.8.1
> Reporter: lehong.ding
> Priority: Major
>
> h4. *Background:*
> Before we moving to solr 8.8.1 from 7.7.2, we performed some performance test
> on solr 8.8.1. We met a lot of concurrent update error in solr log.
> *Envrironment:*
> {{solrCloud with 3 cluster nodes with 500 collections, each has about 1m
> documents.}}
> (1 shard, 3 replica)
> h4. *Threads:*
> 30 update/add threads + 10 deleteByQuery threads
> h4. *Results:*
> During deleteByQuery thread runing, only one node (lead node) has update
> transactions, but other two node has none .
> h4. Errrors:
> java.io.IOException: Request processing has stalled for 20091ms with 100
> remaining elements in the queue.java.io.IOException: Request processing has
> stalled for 20091ms with 100 remaining elements in the queue. at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:449)
> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) at
> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
> at
> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:338)
> at
> org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:244)
> at
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribAdd(DistributedZkUpdateProcessor.java:300)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:230)
> at
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:245)
> at
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:106)
> at
> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:110)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:343)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:291)
> at
> org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:338) at
> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283) at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:244)
>
> h4. Temporary Solution:
> adding -Dsolr.http1=1 in solr start parameters
> There are still some error in error log but the number is much small.
>
> h4. My Questions:
> 1 We found solr cluster will eventually get the data consistent. What’s the
> concurrent update error mainly impacted?
> 2 Adding -Dsolr.http1=1 in solr start parameters can reduce the error
> number. Do we realy need add this parameter? And does this parameter will be
> kept in later version?
> Many Thanks.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
