DNS Lookups during requests?
I ran into an interesting situation today with respect to latency due to failed DNS lookups. THE SETUP Using RHEL8, openjdk 11, solr 9.1.1. The solr instance was standalone in a mostly out of the box config. (No SSL, only http) I had 3 entries in /etc/resolv.conf and the first one in the list was "broken". It was timing out on requests. The situation - When I would issue a query - I'd have upwards of a 5 second pause prior to completion of the request. Which feels like a timeout being hit. For example, while on the server "10.0.0.200" curl -v ' http://10.0.0.200:8983/solr/mycore/select?q=*:*&fq=locale_s:en-us&rows=1' I then edited jetty.xml to remove "org.eclipse.jetty.server.handler.InetAccessHandler" (since I wasn't using an ACL anyways). From there - results seemed to get a little better. I tended to get instant responses, but occasionally, I'd still see the multi-second delay. I'm not sure how to track down this second piece of latency when DNS fails. Is this an opportunity to also remove these lookups to improve performance? -Tim
solr backups failing
Hi all, We are using solrcloud 8.11 with 10 shards. We've been getting errors trying to back up our index using the collections API. Here's the error we see in some of the nodes when we run the REQUESTSTATUS api. Failed to backup core=collection_02_replica_t21 because java.io.IOException: Input/output error We ran it successfully once, but since then at least one node, but not always the same one fails. It is a large index with almost 100 million documents and 2T total in size. What could cause this error? Thanks in advance
Multiple index combined boost using boost query
Hi All,
I want to apply multiple Index combined boost.
if(query({!terms f=au v='living' cache=false}),10.0,1) AND if(query({!terms
f=ti v='living' cache=false}),10.0,1)
My requirement is, if my term ‘living’ matching both field au and ti both in
same document then only add a boost to record.
Can anyone help my providing correct way to writing such boost query.
Thanks,
Jigar Gajjar
Terms Query Parser: escaping separator character in terms values
Hey, I tried to find how to escape the separator character in term values used
in Terms Query Parser but I could find it.
I check the documentation but it's not there:
https://solr.apache.org/guide/solr/latest/query-guide/other-parsers.html#terms-query-parser
Next, I tried to escape it with "\", or even quoting the value that contains
it. But it didn't work.
{!terms field=x}a,"b\,c"
gives me:
TermInSetQuery(x:("b a c")) !!
I tried to find the answer in the test [1] but it wasn't there.
Next, I looked into the implementation [2] and it looks like the values are
simply split with no support of any form of escaping/enquoting.
I think that the lack of support for escaping makes this query parser pretty
unusable for handling arbitrary input. Am I wrong?
I searched if maybe someone reported it already, but I couldn't find anything
in Solr's bug tracker. Should I open an issue for it?
[1]
https://github.com/apache/solr/blob/11253f05cfb31f9fb945c831d8889b3db1e607f1/solr/core/src/test/org/apache/solr/search/TestTermsQParserPlugin.java
[2]
https://github.com/apache/solr/blob/11253f05cfb31f9fb945c831d8889b3db1e607f1/solr/core/src/java/org/apache/solr/search/TermsQParserPlugin.java#L157-L158
Re: DNS Lookups during requests?
There are some settings which are common to apply to the JVM. You might try those first. If you do a search for JVM java DNS settings it will come up. On Wed, Mar 22, 2023 at 7:01 AM Tim Funk wrote: > I ran into an interesting situation today with respect to latency due to > failed DNS lookups. > > THE SETUP > Using RHEL8, openjdk 11, solr 9.1.1. The solr instance was standalone in a > mostly out of the box config. (No SSL, only http) > > I had 3 entries in /etc/resolv.conf and the first one in the list was > "broken". It was timing out on requests. > > The situation - When I would issue a query - I'd have upwards of a 5 second > pause prior to completion of the request. Which feels like a timeout being > hit. For example, while on the server "10.0.0.200" > curl -v ' > http://10.0.0.200:8983/solr/mycore/select?q=*:*&fq=locale_s:en-us&rows=1' > > I then edited jetty.xml to > remove "org.eclipse.jetty.server.handler.InetAccessHandler" (since I wasn't > using an ACL anyways). From there - results seemed to get a little better. > I tended to get instant responses, but occasionally, I'd still see the > multi-second delay. > > I'm not sure how to track down this second piece of latency when DNS fails. > Is this an opportunity to also remove these lookups to improve performance? > > > -Tim >
Re: DNS Lookups during requests?
I did try adding to SOLR_OPTS to be "-Djava.net.preferIPv4Stack=true -Dnetworkaddress.cache.ttl=3600" and they didn't seem to have any effect during the triage. -Tim On Wed, Mar 22, 2023 at 12:44 PM Tim Casey wrote: > There are some settings which are common to apply to the JVM. You might > try those first. > If you do a search for JVM java DNS settings it will come up. > > On Wed, Mar 22, 2023 at 7:01 AM Tim Funk wrote: > > > I ran into an interesting situation today with respect to latency due to > > failed DNS lookups. > > > > THE SETUP > > Using RHEL8, openjdk 11, solr 9.1.1. The solr instance was standalone in > a > > mostly out of the box config. (No SSL, only http) > > > > I had 3 entries in /etc/resolv.conf and the first one in the list was > > "broken". It was timing out on requests. > > > > The situation - When I would issue a query - I'd have upwards of a 5 > second > > pause prior to completion of the request. Which feels like a timeout > being > > hit. For example, while on the server "10.0.0.200" > > curl -v ' > > http://10.0.0.200:8983/solr/mycore/select?q=*:*&fq=locale_s:en-us&rows=1 > ' > > > > I then edited jetty.xml to > > remove "org.eclipse.jetty.server.handler.InetAccessHandler" (since I > wasn't > > using an ACL anyways). From there - results seemed to get a little > better. > > I tended to get instant responses, but occasionally, I'd still see the > > multi-second delay. > > > > I'm not sure how to track down this second piece of latency when DNS > fails. > > Is this an opportunity to also remove these lookups to improve > performance? > > > > > > -Tim > > >
Re: DNS Lookups during requests?
I did try adding to SOLR_OPTS this: "-Djava.net.preferIPv4Stack=true -Dnetworkaddress.cache.ttl=3600" and they didn't seem to have any effect during the triage. -Tim On Wed, Mar 22, 2023 at 12:44 PM Tim Casey wrote: > There are some settings which are common to apply to the JVM. You might > try those first. > If you do a search for JVM java DNS settings it will come up. > > On Wed, Mar 22, 2023 at 7:01 AM Tim Funk wrote: > > > I ran into an interesting situation today with respect to latency due to > > failed DNS lookups. > > > > THE SETUP > > Using RHEL8, openjdk 11, solr 9.1.1. The solr instance was standalone in > a > > mostly out of the box config. (No SSL, only http) > > > > I had 3 entries in /etc/resolv.conf and the first one in the list was > > "broken". It was timing out on requests. > > > > The situation - When I would issue a query - I'd have upwards of a 5 > second > > pause prior to completion of the request. Which feels like a timeout > being > > hit. For example, while on the server "10.0.0.200" > > curl -v ' > > http://10.0.0.200:8983/solr/mycore/select?q=*:*&fq=locale_s:en-us&rows=1 > ' > > > > I then edited jetty.xml to > > remove "org.eclipse.jetty.server.handler.InetAccessHandler" (since I > wasn't > > using an ACL anyways). From there - results seemed to get a little > better. > > I tended to get instant responses, but occasionally, I'd still see the > > multi-second delay. > > > > I'm not sure how to track down this second piece of latency when DNS > fails. > > Is this an opportunity to also remove these lookups to improve > performance? > > > > > > -Tim > > >
Re: DNS Lookups during requests?
Hello, You can check jstack or Thread Dump in Solr Admin to localize the clog. On Wed, Mar 22, 2023 at 5:01 PM Tim Funk wrote: > I ran into an interesting situation today with respect to latency due to > failed DNS lookups. > > THE SETUP > Using RHEL8, openjdk 11, solr 9.1.1. The solr instance was standalone in a > mostly out of the box config. (No SSL, only http) > > I had 3 entries in /etc/resolv.conf and the first one in the list was > "broken". It was timing out on requests. > > The situation - When I would issue a query - I'd have upwards of a 5 second > pause prior to completion of the request. Which feels like a timeout being > hit. For example, while on the server "10.0.0.200" > curl -v ' > http://10.0.0.200:8983/solr/mycore/select?q=*:*&fq=locale_s:en-us&rows=1' > > I then edited jetty.xml to > remove "org.eclipse.jetty.server.handler.InetAccessHandler" (since I wasn't > using an ACL anyways). From there - results seemed to get a little better. > I tended to get instant responses, but occasionally, I'd still see the > multi-second delay. > > I'm not sure how to track down this second piece of latency when DNS fails. > Is this an opportunity to also remove these lookups to improve performance? > > > -Tim > -- Sincerely yours Mikhail Khludnev https://t.me/MUST_SEARCH A caveat: Cyrillic!
Re: DNS Lookups during requests?
If you have a DNS out, you will get DNS timeout. Removing the specific entry might help, at least until it is resolved. On Wed, Mar 22, 2023 at 10:17 AM Tim Funk wrote: > I did try adding to SOLR_OPTS to be "-Djava.net.preferIPv4Stack=true > -Dnetworkaddress.cache.ttl=3600" and they didn't seem to have any effect > during the triage. > > -Tim > > On Wed, Mar 22, 2023 at 12:44 PM Tim Casey wrote: > > > There are some settings which are common to apply to the JVM. You might > > try those first. > > If you do a search for JVM java DNS settings it will come up. > > > > On Wed, Mar 22, 2023 at 7:01 AM Tim Funk wrote: > > > > > I ran into an interesting situation today with respect to latency due > to > > > failed DNS lookups. > > > > > > THE SETUP > > > Using RHEL8, openjdk 11, solr 9.1.1. The solr instance was standalone > in > > a > > > mostly out of the box config. (No SSL, only http) > > > > > > I had 3 entries in /etc/resolv.conf and the first one in the list was > > > "broken". It was timing out on requests. > > > > > > The situation - When I would issue a query - I'd have upwards of a 5 > > second > > > pause prior to completion of the request. Which feels like a timeout > > being > > > hit. For example, while on the server "10.0.0.200" > > > curl -v ' > > > > http://10.0.0.200:8983/solr/mycore/select?q=*:*&fq=locale_s:en-us&rows=1 > > ' > > > > > > I then edited jetty.xml to > > > remove "org.eclipse.jetty.server.handler.InetAccessHandler" (since I > > wasn't > > > using an ACL anyways). From there - results seemed to get a little > > better. > > > I tended to get instant responses, but occasionally, I'd still see the > > > multi-second delay. > > > > > > I'm not sure how to track down this second piece of latency when DNS > > fails. > > > Is this an opportunity to also remove these lookups to improve > > performance? > > > > > > > > > -Tim > > > > > >
Re: DNS Lookups during requests?
On 2023-03-22 12:16 PM, Tim Funk wrote: I did try adding to SOLR_OPTS this: "-Djava.net.preferIPv4Stack=true -Dnetworkaddress.cache.ttl=3600" and they didn't seem to have any effect during the triage. If you're querying http://10.0.0.200:8983/solr/mycore/select?q=*:*&fq=locale_s:en-us&rows=1, there shouldn't be either DNS nor IPv6 involved. Unless jetty is doing some funky internal redirects involving hostnames Dima
Re: Terms Query Parser: escaping separator character in terms values
I think it was made so for sake of simplicity. That's why it has separator
param. Query generator should just choose the right one absent
across terms.
On Wed, Mar 22, 2023 at 7:37 PM Tomasz Elendt
wrote:
> Hey, I tried to find how to escape the separator character in term values
> used in Terms Query Parser but I could find it.
>
> I check the documentation but it's not there:
>
> https://solr.apache.org/guide/solr/latest/query-guide/other-parsers.html#terms-query-parser
>
> Next, I tried to escape it with "\", or even quoting the value that
> contains it. But it didn't work.
>
> {!terms field=x}a,"b\,c"
>
> gives me:
>
> TermInSetQuery(x:("b a c")) !!
>
> I tried to find the answer in the test [1] but it wasn't there.
> Next, I looked into the implementation [2] and it looks like the values
> are simply split with no support of any form of escaping/enquoting.
>
> I think that the lack of support for escaping makes this query parser
> pretty unusable for handling arbitrary input. Am I wrong?
>
> I searched if maybe someone reported it already, but I couldn't find
> anything in Solr's bug tracker. Should I open an issue for it?
>
>
> [1]
> https://github.com/apache/solr/blob/11253f05cfb31f9fb945c831d8889b3db1e607f1/solr/core/src/test/org/apache/solr/search/TestTermsQParserPlugin.java
> [2]
> https://github.com/apache/solr/blob/11253f05cfb31f9fb945c831d8889b3db1e607f1/solr/core/src/java/org/apache/solr/search/TermsQParserPlugin.java#L157-L158
--
Sincerely yours
Mikhail Khludnev
https://t.me/MUST_SEARCH
A caveat: Cyrillic!
Re: Multiple index combined boost using boost query
Perhaps
if(query($qq),10.0,1)&qq=+{!terms f=au v='living' cache=false} +{!terms
f=ti v='living' cache=false}
Note, the syntax is really fragile, any subtle change may break it.
On Wed, Mar 22, 2023 at 6:04 PM Gajjar, Jigar wrote:
> Hi All,
>
>
> I want to apply multiple Index combined boost.
>
> if(query({!terms f=au v='living' cache=false}),10.0,1) AND
> if(query({!terms f=ti v='living' cache=false}),10.0,1)
>
> My requirement is, if my term ‘living’ matching both field au and ti both
> in same document then only add a boost to record.
>
> Can anyone help my providing correct way to writing such boost query.
>
> Thanks,
> Jigar Gajjar
>
>
>
--
Sincerely yours
Mikhail Khludnev
https://t.me/MUST_SEARCH
A caveat: Cyrillic!
Authentication for Solr delegating to Active Directory
I finally have a good grasp of exactly how rule based authorization in Solr works. It's exceptionally flexible once you figure it out. I've been trying to figure out whether there is any way to use Active Directory for authentication in Solr, and if so, how in the world I would manage mapping those users to Solr's authorization roles. I found a page that explains how to use JWT authentication to talk to Azure AD, but I suspect that won't work with on-prem AD. Also, the page says absolutely nothing about authorization. I do not know whether the AD in this environment I'm working in is Azure or not. I have asked, but the answer hasn't come yet. One mailing list thread I saw talks about possibly doing LDAP or Kerberos, but doesn't say how to do it, and also says nothing about authorization. So I'm asking the question again, since I haven't found answers yet. Can Solr use Active Directory for authentication? If it can, how do I set it up? What options are there for mapping users in AD to Solr's authorization roles? If I can work out how to do it, I will see about writing up instructions for the ref guide. Thanks, Shawn
Re: Authentication for Solr delegating to Active Directory
On 2023-03-22 7:24 PM, Shawn Heisey wrote: I finally have a good grasp of exactly how rule based authorization in Solr works. It's exceptionally flexible once you figure it out. I've been trying to figure out whether there is any way to use Active Directory for authentication in Solr, and if so, how in the world I would manage mapping those users to Solr's authorization roles. Looks like someone did it: https://stackoverflow.com/questions/17730950/jetty-jaas-and-active-directory-authentication-only On-prem AD is really just LDAP+SSL but I know next to nothing about jetty so I would be tempted to put it behind an apache or nginx proxy and off-load the auth to it. Those are well documented. my $.02 Dima
Live shard with no leader
I have a shard with replication factor 2. One shard is state:active, the other is state:down. The active shard is not a leader. Using the FORCELEADER command to try and get it elected leader doesn’t fix it. We tried adding another replica, but it is also in state:down, maybe because there isn’t a leader to replicate from. Any ideas on how to unwedge this? wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog)
Re: Authentication for Solr delegating to Active Directory
On 3/22/2023 6:56 PM, dmitri maziuk wrote: Looks like someone did it: https://stackoverflow.com/questions/17730950/jetty-jaas-and-active-directory-authentication-only On-prem AD is really just LDAP+SSL but I know next to nothing about jetty so I would be tempted to put it behind an apache or nginx proxy and off-load the auth to it. Those are well documented. I want to use Solr's authorization capability, so I don't think configuring auth in Jetty is what I want at all. Same goes for any kind of proxy layer. I could probably teach a proxy how to do the authorization I want, but I really don't want to reinvent the wheel that has already been built into Solr. It would be a TON of work. Thanks, Shawn
Solr Shard not coming up
Hi All, I have come across an issue where a few shards are not coming up after solr restart. Currently I don't have an error message stored. But I remember it was due to some index file corruption. Please share your suggestions to avoid such issues. -- Thanks and Regards, Hari Mobile:9790756568
