About Query for sharding core

[Changcheng Shao]

Hi, Solr team

I am using Solr 8.11, and I want to ask some questions about query
for sharding core.

Now I have two Solr cloud clusters, one has two shards  for core, and the
other has one shard.
Cluster sadhocb has 3 Solr instances. The core is creative, and has 2
shards and 3 replications. The router field is *network_id*.
Cluster sadhocc has 3 Solr instances. The core is creative, and has 1 shard
and 3 replications.

The query
is: 
q=network_id:505334+AND+(is_delete:("0"))&network_id=505334&indent=off&fl=id&start=91100&collection=creative&sort=updated_at+asc,+id+desc&rows=50&wt=json&_route_=505334
And the query is deep paging.
And I use Solrj to send query.

1. For cluster sadhocc
When it requests the query, it only has one log in an instance, and the
query time is 81ms which is short.
The log is:

> INFO  - 2023-02-15 07:01:48.003; org.apache.solr.core.SolrCore.Request;
> [creative_shard1_replica_n4]  webapp=/solr path=/select
> params={q=network_id:505334+AND+(is_delete:("0"))&network_id=505334&indent=off&fl=id&start=91100&collection=creative&sort=updated_at+asc,+id+desc&rows=50&wt=json&version=2.2&_route_=505334}
> hits=245271 status=0 QTime=81


2. For cluster sadhocb
The time difference between multi queries is large.
*2.1 No other parameters*
For example, it only has one log in an instance, and the QTime is 71ms
which is short.
The log in dev-search-sadhocb-solr3:

> INFO  - 2023-02-15 07:13:19.559; org.apache.solr.core.SolrCore.Request;
> [creative_shard2_replica_n8]  webapp=/solr path=/select
> params={q=network_id:505334+AND+(is_delete:("0"))&network_id=505334&indent=off&fl=id&start=91100&collection=creative&sort=updated_at+asc,+id+desc&rows=50&wt=json&version=2.2&_route_=505334}
> hits=245215 status=0 QTime=71


But sometimes, it has two logs in different instances, and the
QTime is 2694ms which is large.
*The log in instance dev-search-sadhocb-solr1:*

> INFO  - 2023-02-15 07:06:54.314; org.apache.solr.core.SolrCore.Request;
> [creative_shard2_replica_n6]  webapp=/solr path=/select
> params={df=name&distrib=false&preferLocalShards=false&fl=id&shards.purpose=68&start=0&fsv=true&q.op=AND&collection=creative&sort=updated_at+asc,+id+desc&shard.url=
> http://dev-search-sadhocb-solr1.dev.ui.aws.fwmrm.net:8983/solr/creative_shard2_replica_n6/|http://dev-search-sadhocb-solr2.dev.ui.aws.fwmrm.net:8983/solr/creative_shard2_replica_n10/|http://dev-search-sadhocb-solr3.dev.ui.aws.fwmrm.net:8983/solr/creative_shard2_replica_n8/&rows=91150&rid=dev-search-sadhocb-solr2.dev.ui.aws.fwmrm.net-12707&version=2&q=network_id:505334+AND+(is_delete:("0"))&network_id=505334&omitHeader=false&NOW=1676444814205&isShard=true&timeAllowed=15000&wt=javabin&_route_=505334}
> hits=245215 status=0 QTime=104

*The log in instance dev-search-sadhocb-solr2:*

> INFO  - 2023-02-15 07:06:56.900; org.apache.solr.core.SolrCore.Request;
> [creative_shard1_replica_n4]  webapp=/solr path=/select
> params={q=network_id:505334+AND+(is_delete:("0"))&network_id=505334&indent=off&fl=id&start=91100&collection=creative&sort=updated_at+asc,+id+desc&rows=50&wt=json&version=2.2&_route_=505334}
> rid=dev-search-sadhocb-solr2.dev.ui.aws.fwmrm.net-12707 hits=245215
> status=0 QTime=2694


And I think the instance solr2 received the query and sent it to insance
solr1, and it can transfer network data from different instances.

*2.2 Add parameter shards.preference*
Then I use the parameter shards.preference=replica.location:local
https://solr.apache.org/guide/8_11/distributed-requests.html#preferlocalshards-parameter
And the query is:
q=network_id:505334+AND+(is_delete:("0"))&network_id=505334&indent=off&fl=id&start=91100&collection=creative&sort=updated_at+asc,+id+desc&rows=50&wt=json&_route_=505334&shards.preference=replica.location:local

Sometimes, it only has one log in an instance as above.
*The log in instance dev-search-sadhocb-solr1:*

> INFO  - 2023-02-15 07:18:29.377; org.apache.solr.core.SolrCore.Request;
> [creative_shard2_replica_n6]  webapp=/solr path=/select
> params={q=network_id:505334+AND+(is_delete:("0"))&network_id=505334&indent=off&fl=id&start=91100&collection=creative&sort=updated_at+asc,+id+desc&rows=50&shards.preference=replica.location:local&wt=json&version=2.2&_route_=505334}
> hits=245215 status=0 QTime=74


But sometimes, it has two logs in the same instance, and the QTime is
2660ms which is large.

*The log in instance dev-search-sadhocb-solr2:*

> INFO  - 2023-02-15 07:16:12.355; org.apache.solr.core.SolrCore.Request;
> [creative_shard2_replica_n10]  webapp=/solr path=/select
> params={df=name&distrib=false&preferLocalShards=false&fl=id&shards.purpose=68&start=0&fsv=true&q.op=AND&collection=creative&sort=updated_at+asc,+id+desc&shard.url=
> http://dev-search-sadhocb-solr2.dev.ui.aws.fwmrm.net:8983/solr/creative_shard2_replica_n10/|http://dev-search-sadhocb-solr1.dev.ui.aws.fwmrm.net:8983/solr/creative_shard2_replica_n6/|http://dev-search-sadhocb-solr3.dev.ui.aws.fwmrm.net:8983/solr/creative_shard2_replica_n8/&rows=91150

RE: SOLR security scan question

[Razvan Bolocan]

Any news on this?

From: Razvan Bolocan
Sent: Thursday, February 9, 2023 12:11 PM
To: users@solr.apache.org
Subject: SOLR security scan question

Hi,

We are using SOLR 8.11.2  both classic and containerised/docker.
We have an internal security scanner and it contains multiple types of scans. 
On the latest scans we have:

Critical
CVE-2015-1832 : org.apache.derby:derby 10.9.1.0
: org.apache.derby:derby 10.9.1.0
Critical
CVE-2017-15095 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
Critical
CVE-2018-11307 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
Critical
CVE-2018-14718 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
High
CVE-2018-5968 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
Critical
CVE-2018-7489 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
Critical
CVE-2019-14540 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
Critical
CVE-2019-14893 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
Critical
CVE-2019-16335 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
Critical
CVE-2019-16942 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
Critical
CVE-2019-16943 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
Critical
CVE-2019-17267 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
Critical
CVE-2019-17531 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
Critical
CVE-2019-20330 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
High
CVE-2020-10650 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
High
CVE-2020-35490 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
High
CVE-2020-35491 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
High
CVE-2020-36518 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
High
CVE-2021-22573 : com.google.oauth-client:google-oauth-client 1.32.1
: com.google.oauth-client:google-oauth-client 1.32.1
High
CVE-2021-33813 : org.jdom:jdom2 2.0.6
: org.jdom:jdom2 2.0.6
Critical
CVE-2021-37404 : org.apache.hadoop:hadoop-common 3.2.2
: org.apache.hadoop:hadoop-common 3.2.2
High
CVE-2022-2048 : org.eclipse.jetty.http2:http2-server 9.4.44.v20210927
: org.eclipse.jetty.http2:http2-server 9.4.44.v20210927
Critical
CVE-2022-25168 : org.apache.hadoop:hadoop-common 3.2.2
: org.apache.hadoop:hadoop-common 3.2.2
High
CVE-2022-25647 : com.google.code.gson:gson 2.7
: com.google.code.gson:gson 2.7
Critical
CVE-2022-26612 : org.apache.hadoop:hadoop-common 3.2.2
: org.apache.hadoop:hadoop-common 3.2.2
High
CVE-2022-3171 : com.google.protobuf:protobuf-java 3.11.0
: com.google.protobuf:protobuf-java 3.11.0
High
CVE-2022-36364 : org.apache.calcite.avatica:avatica-core 1.18.0
: org.apache.calcite.avatica:avatica-core 1.18.0
Critical
CVE-2022-39135 : org.apache.calcite:calcite-core 1.27.0
: org.apache.calcite:calcite-core 1.27.0
High
CVE-2022-40151 : com.fasterxml.woodstox:woodstox-core 6.2.4
: com.fasterxml.woodstox:woodstox-core 6.2.4
High
CVE-2022-40152 : com.fasterxml.woodstox:woodstox-core 6.2.4
: com.fasterxml.woodstox:woodstox-core 6.2.4
Critical
CVE-2022-41853 : org.hsqldb:hsqldb 2.4.0
: org.hsqldb:hsqldb 2.4.0
High
CVE-2022-42003 : com.fasterxml.jackson.core:jackson-databind 2.13.4
: com.fasterxml.jackson.core:jackson-databind 2.13.4
High
CVE-2022-42003 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
High
CVE-2022-42004 : com.fasterxml.jackson.core:jackson-databind 2.4.0
: com.fasterxml.jackson.core:jackson-databind 2.4.0
High
CVE-2022-47629 : libksba 1.3.5-8.el8_6
: libksba 1.3.5-8.el8_6


We know some of them are covered in 
https://solr.apache.org/security.html#cve-reports-for-apache-solr-dependencies 
but not all.
We have also seen the 
https://lists.apache.org/thread/539bkq8r11msjpl3yo1ssvy77kmdrps7
Can we have a resolution for the above?

Thanks,
Razvan Bolocan

Re: SOLR security scan question

[Andy Lester]

> Any news on this?
> 
> We know some of them are covered in 
> https://solr.apache.org/security.html#cve-reports-for-apache-solr-dependencies
>  but not all.
> We have also seen the 
> https://lists.apache.org/thread/539bkq8r11msjpl3yo1ssvy77kmdrps7
> Can we have a resolution for the above?


What sort of resolution are you looking for?

Re: SOLR security scan question

[Gus Heck]

Hi Razvan,

Have you looked at https://solr.apache.org/security.html yet? Some of the
CVE's in your list are already listed there. If you could eliminate the
CVE's from your list that are already dealt with on that page then you
might get more attention. As it stands, you seem to be asking us to do that
work for you.

Please note that there are two tables on that page, and one is near the
bottom. This is something I realize perhaps could be made clearer with a
table of contents or something.

Another thing to note is that in order for an actual vulnerability to
exist, the dependency must be used in the ways described in the CVE. I
notice you have many Jackson CVE's in your list and there are a large
number of Jackson CVE's that relate to features Solr does not use, and
therefore do not pose a threat. This is explained in the second table near
the bottom.

Let us know if you have found something not listed on that page (ctrl-f
find in your browser on the page for the CVE identifier may be quite
useful), or if you have questions about a specific explanation offered on
that page.

Best,
Gus

On Wed, Feb 15, 2023 at 12:20 PM Andy Lester  wrote:

> > Any news on this?
> >
> > We know some of them are covered in
> https://solr.apache.org/security.html#cve-reports-for-apache-solr-dependencies
> but not all.
> > We have also seen the
> https://lists.apache.org/thread/539bkq8r11msjpl3yo1ssvy77kmdrps7
> > Can we have a resolution for the above?
>
>
> What sort of resolution are you looking for?



-- 
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)

Re: Standalone Solo upgrade from 4.x to 8.x

[Natarajan, Rajeswari]

Thanks for the response. The link only has details from 6.x . Prior to 6.x 
versions there is no info.

-Rajeswari

On 2/14/23, 10:36 PM, "Jan Høydahl" mailto:jan@cominvent.com>> wrote:


It is possible, but I’d choose v9. Reference guide is your friend: 
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fguide%2Fsolr%2Flatest%2Fupgrade-notes%2Fsolr-upgrade-notes.html&data=05%7C01%7Crajeswari.natarajan%40sap.com%7Cb0660226c6a443e00a6908db0f1ef449%7C42f7676cf455423c82f6dc2d99791af7%7C0%7C0%7C638120397889351041%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fvuJmc2LrA9oU01ycK9OFXLk48Mdwiz72BAsEnIgbKE%3D&reserved=0
 



Jan Høydahl


> 15. feb. 2023 kl. 04:03 skrev Natarajan, Rajeswari 
>  lid>:
> 
> Hi,
> 
> Would like to know if it is possible to upgrade standalone solr from 4.x to 
> 8.x
> 
> Is there a wiki available. Please let me know.
> 
> 
> Thanks,
> Rajeswari

Re: Standalone Solo upgrade from 4.x to 8.x

[Shawn Heisey]

On 2/14/23 20:02, Natarajan, Rajeswari wrote:

Would like to know if it is possible to upgrade standalone solr from 4.x to 8.x

Is there a wiki available. Please let me know.


In most cases when jumping more than one major version you will have to 
completely reindex from scratch.


You can't upgrade it step-by-step to one major version at a time, 
because starting in one of the 6.x versions, the Lucene version that 
originally wrote each segment is recorded and never gets updated or 
removed.  If a segment has no version or the earliest version recorded 
is less than 7.0, then 8.x will refuse to open it.  9.x is similar for 
anything older than 8.0.


I have seen a tool that bypasses this version check, allowing upgrades 
that Lucene prevents.  Even if you did use that tool and managed to get 
it to work, it's a bad idea, which is why it's explicitly blocked.


Thanks,
Shawn

Re: Standalone Solo upgrade from 4.x to 8.x

[Eric Pugh]

If I were you, I would think about this as “a fresh implementation inspired by 
my old set up”….  So much has changed/evolved/gotten better.  I would start out 
with a fresh 9.x Setup, and then look at your old setup and compare 
differences….  

> On Feb 15, 2023, at 7:11 PM, Shawn Heisey  wrote:
> 
> On 2/14/23 20:02, Natarajan, Rajeswari wrote:
>> Would like to know if it is possible to upgrade standalone solr from 4.x to 
>> 8.x
>> Is there a wiki available. Please let me know.
> 
> In most cases when jumping more than one major version you will have to 
> completely reindex from scratch.
> 
> You can't upgrade it step-by-step to one major version at a time, because 
> starting in one of the 6.x versions, the Lucene version that originally wrote 
> each segment is recorded and never gets updated or removed.  If a segment has 
> no version or the earliest version recorded is less than 7.0, then 8.x will 
> refuse to open it.  9.x is similar for anything older than 8.0.
> 
> I have seen a tool that bypasses this version check, allowing upgrades that 
> Lucene prevents.  Even if you did use that tool and managed to get it to 
> work, it's a bad idea, which is why it's explicitly blocked.
> 
> Thanks,
> Shawn

___
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
http://www.opensourceconnections.com  | 
My Free/Busy   
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 


This e-mail and all contents, including attachments, is considered to be 
Company Confidential unless explicitly stated otherwise, regardless of whether 
attachments are marked as such.

Deleting shard in solr

[Neeraj giri]

Hi all,
I am trying to delete a shard in solr, but its saying shard is active and
cannot be deleted.
Is there a way to make a shard inactive ad then delete it? or any other way
to delete a shard
Regards,

Neeraj giri

Re: Deleting shard in solr

[Shawn Heisey]

On 2/15/23 18:42, Neeraj giri wrote:

I am trying to delete a shard in solr, but its saying shard is active and
cannot be deleted.
Is there a way to make a shard inactive ad then delete it? or any other way
to delete a shard


Which router is the collection using?  If it's the default compositeid, 
you can't delete a shard.  If you did, the docs that matched the hash 
range for that shard would have nowhere to go.  You also can't add a 
shard with that router.  About the only thing you can do along these 
lines is SPLITSHARD.


With the implicit router, you are 100 percent in control of which shards 
get which docs, so adding and deleting shards is permitted.


Thanks,
Shawn

Re: Deleting shard in solr

[Shawn Heisey]

On 2/15/23 20:10, Shawn Heisey wrote:
About the only thing you can do along these 
lines is SPLITSHARD.


After a split completes, you can delete the original shard, because at 
that point it will be inactive, and the newly created shards will cover 
its hash range.


Thanks,
Shawn

Re: Standalone Solr upgrade from 4.x to 8.x

[Natarajan, Rajeswari]

Thanks much for the reply.

On 2/15/23, 4:12 PM, "Shawn Heisey" mailto:apa...@elyograg.org>> wrote:


On 2/14/23 20:02, Natarajan, Rajeswari wrote:
> Would like to know if it is possible to upgrade standalone solr from 4.x to 
> 8.x
> 
> Is there a wiki available. Please let me know.


In most cases when jumping more than one major version you will have to 
completely reindex from scratch.


You can't upgrade it step-by-step to one major version at a time, 
because starting in one of the 6.x versions, the Lucene version that 
originally wrote each segment is recorded and never gets updated or 
removed. If a segment has no version or the earliest version recorded 
is less than 7.0, then 8.x will refuse to open it. 9.x is similar for 
anything older than 8.0.


I have seen a tool that bypasses this version check, allowing upgrades 
that Lucene prevents. Even if you did use that tool and managed to get 
it to work, it's a bad idea, which is why it's explicitly blocked.


Thanks,
Shawn