[
https://issues.apache.org/jira/browse/SOLR-11200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115348#comment-16115348
]
Amrit Sarkar edited comment on SOLR-11200 at 8/7/17 4:10 PM:
-------------------------------------------------------------
Ah! I don't think this will serve our purpose in bulk indexing, logs ::
{code}
mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=5, maxMergeCount=15,
ioThrottle=false
2017-08-05 09:14:03.005 INFO (qtp1205044462-19) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][qtp1205044462-19]: updateMergeThreads ioThrottle=false
targetMBPerSec=10240.0 MB/sec
mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=5, maxMergeCount=15,
ioThrottle=false
2017-08-05 09:15:51.196 INFO (qtp1205044462-69) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][qtp1205044462-69]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
2017-08-05 09:15:56.711 INFO (Lucene Merge Thread #0) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][Lucene Merge Thread #0]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:10.752 INFO (qtp1205044462-17) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][qtp1205044462-17]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:18.229 INFO (Lucene Merge Thread #1) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][Lucene Merge Thread #1]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:26.516 INFO (qtp1205044462-69) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][qtp1205044462-69]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:35.551 INFO (Lucene Merge Thread #2) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][Lucene Merge Thread #2]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:38.580 INFO (qtp1205044462-18) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][qtp1205044462-18]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:49.397 INFO (Lucene Merge Thread #3) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][Lucene Merge Thread #3]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:56.630 INFO (qtp1205044462-15) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][qtp1205044462-15]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
{code}
See the {{targetMBPerSec}} is initialised to {{10gbps}}, but then it falls back
to default {{20mbps}}, instead of maintaining at 10gbps. Maybe
{{SolrIndexConfig#buildMergeSchedule}} is not the right place to do it. I will
look more.
was (Author: [email protected]):
Ah! I don't this will serve our purpose in bulk indexing, logs ::
{code}
mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=5, maxMergeCount=15,
ioThrottle=false
2017-08-05 09:14:03.005 INFO (qtp1205044462-19) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][qtp1205044462-19]: updateMergeThreads ioThrottle=false
targetMBPerSec=10240.0 MB/sec
mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=5, maxMergeCount=15,
ioThrottle=false
2017-08-05 09:15:51.196 INFO (qtp1205044462-69) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][qtp1205044462-69]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
2017-08-05 09:15:56.711 INFO (Lucene Merge Thread #0) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][Lucene Merge Thread #0]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:10.752 INFO (qtp1205044462-17) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][qtp1205044462-17]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:18.229 INFO (Lucene Merge Thread #1) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][Lucene Merge Thread #1]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:26.516 INFO (qtp1205044462-69) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][qtp1205044462-69]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:35.551 INFO (Lucene Merge Thread #2) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][Lucene Merge Thread #2]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:38.580 INFO (qtp1205044462-18) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][qtp1205044462-18]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:49.397 INFO (Lucene Merge Thread #3) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][Lucene Merge Thread #3]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:56.630 INFO (qtp1205044462-15) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][qtp1205044462-15]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
{code}
See the {{targetMBPerSec}} is initialised to {{10gbps}}, but then it falls back
to default {{20mbps}}, instead of maintaining at 10gbps. Maybe
{{SolrIndexConfig#buildMergeSchedule}} is not the right place to do it. I will
look more.
> provide a config to enable disable ConcurrentMergeSchedule.doAutoIOThrottle
> ---------------------------------------------------------------------------
>
> Key: SOLR-11200
> URL: https://issues.apache.org/jira/browse/SOLR-11200
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Nawab Zada Asad iqbal
> Priority: Minor
> Attachments: SOLR-11200.patch, SOLR-11200.patch
>
>
> This config can be useful while bulk indexing. Lucene introduced it
> https://issues.apache.org/jira/browse/LUCENE-6119 .
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
