Hi, Solr team
I am using Solr 8.11, and I want to ask some questions about query
for sharding core.
Now I have two Solr cloud clusters, one has two shards for core, and the
other has one shard.
Cluster sadhocb has 3 Solr instances. The core is creative, and has 2
shards and 3 replications. The router field is *network_id*.
Cluster sadhocc has 3 Solr instances. The core is creative, and has 1 shard
and 3 replications.
The query
is:
q=network_id:505334+AND+(is_delete:("0"))&network_id=505334&indent=off&fl=id&start=91100&collection=creative&sort=updated_at+asc,+id+desc&rows=50&wt=json&_route_=505334
And the query is deep paging.
And I use Solrj to send query.
1. For cluster sadhocc
When it requests the query, it only has one log in an instance, and the
query time is 81ms which is short.
The log is:
> INFO - 2023-02-15 07:01:48.003; org.apache.solr.core.SolrCore.Request;
> [creative_shard1_replica_n4] webapp=/solr path=/select
> params={q=network_id:505334+AND+(is_delete:("0"))&network_id=505334&indent=off&fl=id&start=91100&collection=creative&sort=updated_at+asc,+id+desc&rows=50&wt=json&version=2.2&_route_=505334}
> hits=245271 status=0 QTime=81
2. For cluster sadhocb
The time difference between multi queries is large.
*2.1 No other parameters*
For example, it only has one log in an instance, and the QTime is 71ms
which is short.
The log in dev-search-sadhocb-solr3:
> INFO - 2023-02-15 07:13:19.559; org.apache.solr.core.SolrCore.Request;
> [creative_shard2_replica_n8] webapp=/solr path=/select
> params={q=network_id:505334+AND+(is_delete:("0"))&network_id=505334&indent=off&fl=id&start=91100&collection=creative&sort=updated_at+asc,+id+desc&rows=50&wt=json&version=2.2&_route_=505334}
> hits=245215 status=0 QTime=71
But sometimes, it has two logs in different instances, and the
QTime is 2694ms which is large.
*The log in instance dev-search-sadhocb-solr1:*
> INFO - 2023-02-15 07:06:54.314; org.apache.solr.core.SolrCore.Request;
> [creative_shard2_replica_n6] webapp=/solr path=/select
> params={df=name&distrib=false&preferLocalShards=false&fl=id&shards.purpose=68&start=0&fsv=true&q.op=AND&collection=creative&sort=updated_at+asc,+id+desc&shard.url=
> http://dev-search-sadhocb-solr1.dev.ui.aws.fwmrm.net:8983/solr/creative_shard2_replica_n6/|http://dev-search-sadhocb-solr2.dev.ui.aws.fwmrm.net:8983/solr/creative_shard2_replica_n10/|http://dev-search-sadhocb-solr3.dev.ui.aws.fwmrm.net:8983/solr/creative_shard2_replica_n8/&rows=91150&rid=dev-search-sadhocb-solr2.dev.ui.aws.fwmrm.net-12707&version=2&q=network_id:505334+AND+(is_delete:("0"))&network_id=505334&omitHeader=false&NOW=1676444814205&isShard=true&timeAllowed=15000&wt=javabin&_route_=505334}
> hits=245215 status=0 QTime=104
*The log in instance dev-search-sadhocb-solr2:*
> INFO - 2023-02-15 07:06:56.900; org.apache.solr.core.SolrCore.Request;
> [creative_shard1_replica_n4] webapp=/solr path=/select
> params={q=network_id:505334+AND+(is_delete:("0"))&network_id=505334&indent=off&fl=id&start=91100&collection=creative&sort=updated_at+asc,+id+desc&rows=50&wt=json&version=2.2&_route_=505334}
> rid=dev-search-sadhocb-solr2.dev.ui.aws.fwmrm.net-12707 hits=245215
> status=0 QTime=2694
And I think the instance solr2 received the query and sent it to insance
solr1, and it can transfer network data from different instances.
*2.2 Add parameter shards.preference*
Then I use the parameter shards.preference=replica.location:local
https://solr.apache.org/guide/8_11/distributed-requests.html#preferlocalshards-parameter
And the query is:
q=network_id:505334+AND+(is_delete:("0"))&network_id=505334&indent=off&fl=id&start=91100&collection=creative&sort=updated_at+asc,+id+desc&rows=50&wt=json&_route_=505334&shards.preference=replica.location:local
Sometimes, it only has one log in an instance as above.
*The log in instance dev-search-sadhocb-solr1:*
> INFO - 2023-02-15 07:18:29.377; org.apache.solr.core.SolrCore.Request;
> [creative_shard2_replica_n6] webapp=/solr path=/select
> params={q=network_id:505334+AND+(is_delete:("0"))&network_id=505334&indent=off&fl=id&start=91100&collection=creative&sort=updated_at+asc,+id+desc&rows=50&shards.preference=replica.location:local&wt=json&version=2.2&_route_=505334}
> hits=245215 status=0 QTime=74
But sometimes, it has two logs in the same instance, and the QTime is
2660ms which is large.
*The log in instance dev-search-sadhocb-solr2:*
> INFO - 2023-02-15 07:16:12.355; org.apache.solr.core.SolrCore.Request;
> [creative_shard2_replica_n10] webapp=/solr path=/select
> params={df=name&distrib=false&preferLocalShards=false&fl=id&shards.purpose=68&start=0&fsv=true&q.op=AND&collection=creative&sort=updated_at+asc,+id+desc&shard.url=
> http://dev-search-sadhocb-solr2.dev.ui.aws.fwmrm.net:8983/solr/creative_shard2_replica_n10/|http://dev-search-sadhocb-solr1.dev.ui.aws.fwmrm.net:8983/solr/creative_shard2_replica_n6/|http://dev-search-sadhocb-solr3.dev.ui.aws.fwmrm.net:8983/solr/creative_shard2_replica_n8/&rows=91150&rid=dev-search-sadhocb-solr2.dev.ui.aws.fwmrm.net-12753&version=2&q=network_id:505334+AND+(is_delete:("0"))&network_id=505334&omitHeader=false&NOW=1676445372236&isShard=true&timeAllowed=15000&shards.preference=replica.location:local&wt=javabin&_route_=505334}
> hits=245215 status=0 QTime=114
> INFO - 2023-02-15 07:16:14.897; org.apache.solr.core.SolrCore.Request;
> [creative_shard1_replica_n4] webapp=/solr path=/select
> params={q=network_id:505334+AND+(is_delete:("0"))&network_id=505334&indent=off&fl=id&start=91100&collection=creative&sort=updated_at+asc,+id+desc&rows=50&shards.preference=replica.location:local&wt=json&version=2.2&_route_=505334}
> rid=dev-search-sadhocb-solr2.dev.ui.aws.fwmrm.net-12753 hits=245215
> status=0 QTime=2660
Through the log, it sends the query to itself, but it takes a long time.
When the query is deep paging, it sometimes transfers network data from
different instances and the query takes a long time.
Now I want to shard the core creative and reduce the query time for deep
paging. What can I do or how to set parameters to solve this problem.
Hope to get your response, thanks!
