Missing highlights with AND-queries and unified highlighter (Solr 8.11.1 and 9.2.0)

[Morten Ernebjerg]

Hi there

I have a curious problem with the unified highlighter (both for Solr 8.11.1
and 9.2.0). Specifically, if I

1. do a Boolean AND-query on an indexed field filled by copyFields with
input from two underlying fields in the raw document, and
2. the two ANDed terms match entries from different source fields in the
same document, and
3. I request highlighting on the two source fields

then the unified matcher does not return highlights for the matching terms.
Interestingly, the original highlighter *does* return the expected
highlights, but going back to the original highlighter is not a real option
for other reasons, so I'm hoping someone will know a solution.

The simplest set-up to recreate the problem on using the _default configset
(tested on 9.2.0 but probably the same on 8.11.1) is this:

1. Define the following fields in the schema:
- "title": (field type=text_general, stored, tokenized, multivalued, not
indexed)
- "description": (same settings as "title")
- "search": (text type=text_general,  indexed, multivalued, uninvertible,
not stored)
- Copy fields from "title" and "description" to "search"

2. Upload the following document
{
"id": "1",
"title": "A fantastic title",
"description": "A wonderful description"
}

3. Do the search "search:fantastic AND search:wonderful" - as expected, the
document is returned (I'm using the edismax parser, but seems to be the
same for the Lucene one).

4. Switch on highlighting for the fields "title" and "description" (using
hl.fl). I expect one highlight in each field, one for each search term.
However, with the unified highlighter (default in 9.2), I get zero
highlights - more specifically, I get an empty entry in the "highlighting"
property in the JSON response:

"highlighting": {
"1": {}
}

If I choose the original highlighter (hl.method=original), I get the
expected two highlights. If I do a similar search where both matches are in
the *same* field (e.g. "search:fantastic AND search:title"), I get
highlights with both highlighters. The same is true if I do the original
query but with OR instead of AND ("search:fantastic OR search:wonderful").

I played around a bit with hl.requireFieldMatch and hl.queryFieldPattern
but did not find a combination that fixed the issue.

Cheers & thanks in advance,

Morten


--

*Morten Ernebjerg, Ph.D.*

Senior Developer


morten.ernebj...@data4life.care

D4L data4life gGmbH
c/o Digital Health Cluster (DHC) im Hasso-Plattner-Institut (HPI)
Rudolf-Breitscheid-Straße 187
14482 Potsdam, Germany

www.data4life.care

Amtsgericht Potsdam, HRB 30667

Managing Director: Christian-Cornelius Weiß


We are Data4Life. We've been certified by the German Federal Office for
Information Security (BSI) in accordance with ISO 27001 on the basis of
"IT-Grundschutz".

Help regarding solr request timeout because of spellcheck component performance.

[kumar gaurav]

HI Everyone

I am getting a solr socket timeout exception in the select search query
because of bad spellcheck performance.

I am using the spellcheck component in solr select request handler.
solrconfig



  
edismax
true
1
AND
100
true
25
false
true
true
true
false
10
150
100%
default
wordbreak
  
  
spellcheck
  



Do we have any time allowed parameter for spellcheck like query timeAllowed
parameter ?

how can i identify query timeout because of spellcheck component process ?

Please help. Thanks in advance.



-- 
Thanks & Regards
Kumar Gaurav

Re: [TOKYO Lucene/Solr meet-up] ~ Neural search: Fine-tuning a Multi-language model & Product retrieval based on online reviews ~

[kumar gaurav]

HI Mingchun

I have the same request. Can you please record the meeting and send it to
us?

It's very early in my country as well.

Lots of thanks 🙏

On Fri, 21 Apr 2023 at 22:20, Alessandro Benedetti <
benedetti.ale...@gmail.com> wrote:

> Great Mingchun!
> I am so happy of the various applications our neural search contribution
> has enabled! I'll try to attend remotely (but it's quite late in europe)
> and I wish you the best for the meetup!
>
> Cheers
>
> On Fri, 21 Apr 2023, 11:33 Mingchun Zhao, 
> wrote:
>
> > Hi everyone!
> >
> > We, the Solr community in Tokyo, will hold an online meet-up on 26 April
> > 2023 at 10:30 JST (25 April 2023 at 18:30 PDT, 26 April 2023 at 02:30
> CET).
> > The main topics of this presentation are the approaches for using Solo
> 9’s
> > Neural Search features.
> > Participation link: https://meet.google.com/cdh-zuun-uqu
> > There will be two sessions in this meeting.
> >
> > The first will focus on the topic of "Neural search the Solr Manual in
> > multiple languages Using a multi-language model".
> > You know The Solr Reference Guide:
> > https://solr.apache.org/guide/solr/latest/index.html is available only
> in
> > English at the moment, but we have made it possible to perform Neural
> > Search in Japanese by fine-tuning a multi-language model.
> > Please search for "security settings" or "customizing similarity
> function"
> > on the "Solr manual search demo site" as below.
> > https://demo.rondhuit.com/solr-manual
> >
> > The second will focus on the topic of "Searching for your favorite car
> > model".
> > Users provide diverse and valuable information on their online product
> > reviews. We employ this information to characterize products using a
> > Japanese language model and performing semantic search to adjust the
> > results to the users' needs.
> > In order to summarize multiple user reviews for one car model, We used
> some
> > OpenAI API(paid) to create a summary sentence and vectorize the
> > characteristics of the car model. We have prepared a demo, and we will
> also
> > share with you the feeling of using the OpenAI API and its cost.
> >
> > We usually don't announce our meet-up here because the speakers speak
> > Japanese,
> > but this time, 100% of speakers have their speeches in English, so we
> > wanted to announce it.
> > We look forward to your participation.
> >
> > The participation link: https://meet.google.com/cdh-zuun-uqu
> > The event page: https://solr.doorkeeper.jp/events/155335
> >
> > Thanks!
> >
>

Re: Help regarding solr request timeout because of spellcheck component performance.

[kumar gaurav]

++ users@solr.apache.org

On Mon, 24 Apr 2023 at 13:12, kumar gaurav  wrote:

> HI Everyone
>
> I am getting a solr socket timeout exception in the select search query
> because of bad spellcheck performance.
>
> I am using the spellcheck component in solr select request handler.
> solrconfig
>
> 
>
>   
> edismax
> true
> 1
> AND
> 100
> true
> 25
> false
> true
> true
> true
> false
> 10
> 150
> 100%
> default
> wordbreak
>   
>   
> spellcheck
>   
> 
>
>
> Do we have any time allowed parameter for spellcheck like query
> timeAllowed parameter ?
>
> how can i identify query timeout because of spellcheck component process ?
>
> Please help. Thanks in advance.
>
>
>
> --
> Thanks & Regards
> Kumar Gaurav
>

Disk usage not visible in solr admin UI

[Anjali Maurya]

Hi all,
We have seen that Disk usage is not visible on solr cloud UI in
cloud->nodes section. We observed this behavior when we created a
collection with implicit routing.
Why disk usage is not visible on UI and if is it related to implicit
routing then why it is happening for implicit routing?

Re: Disk usage not visible in solr admin UI

[Jan Høydahl]

Can you check http://localhost:8983/solr/admin/metrics on your node? There 
should be a key called "INDEX.size" for your core that shows disk usage.
Also, can you check whether there are errors in browser under "Developer Tools 
-> Network" or "Console"?

Jan

> 24. apr. 2023 kl. 10:15 skrev Anjali Maurya 
> :
> 
> Hi all,
> We have seen that Disk usage is not visible on solr cloud UI in
> cloud->nodes section. We observed this behavior when we created a
> collection with implicit routing.
> Why disk usage is not visible on UI and if is it related to implicit
> routing then why it is happening for implicit routing?

Re: [TOKYO Lucene/Solr meet-up] ~ Neural search: Fine-tuning a Multi-language model & Product retrieval based on online reviews ~

[Mingchun Zhao]

Hi Alessandro,

Thanks for your kind words and your such a wonderful contribution.
We will do our best to make this Meet-up a success.

Best regards,
Mingchun

On Sat, Apr 22, 2023 at 1:50 AM Alessandro Benedetti <
benedetti.ale...@gmail.com> wrote:

> Great Mingchun!
> I am so happy of the various applications our neural search contribution
> has enabled! I'll try to attend remotely (but it's quite late in europe)
> and I wish you the best for the meetup!
>
> Cheers
>
> On Fri, 21 Apr 2023, 11:33 Mingchun Zhao, 
> wrote:
>
> > Hi everyone!
> >
> > We, the Solr community in Tokyo, will hold an online meet-up on 26 April
> > 2023 at 10:30 JST (25 April 2023 at 18:30 PDT, 26 April 2023 at 02:30
> CET).
> > The main topics of this presentation are the approaches for using Solo
> 9’s
> > Neural Search features.
> > Participation link: https://meet.google.com/cdh-zuun-uqu
> > There will be two sessions in this meeting.
> >
> > The first will focus on the topic of "Neural search the Solr Manual in
> > multiple languages Using a multi-language model".
> > You know The Solr Reference Guide:
> > https://solr.apache.org/guide/solr/latest/index.html is available only
> in
> > English at the moment, but we have made it possible to perform Neural
> > Search in Japanese by fine-tuning a multi-language model.
> > Please search for "security settings" or "customizing similarity
> function"
> > on the "Solr manual search demo site" as below.
> > https://demo.rondhuit.com/solr-manual
> >
> > The second will focus on the topic of "Searching for your favorite car
> > model".
> > Users provide diverse and valuable information on their online product
> > reviews. We employ this information to characterize products using a
> > Japanese language model and performing semantic search to adjust the
> > results to the users' needs.
> > In order to summarize multiple user reviews for one car model, We used
> some
> > OpenAI API(paid) to create a summary sentence and vectorize the
> > characteristics of the car model. We have prepared a demo, and we will
> also
> > share with you the feeling of using the OpenAI API and its cost.
> >
> > We usually don't announce our meet-up here because the speakers speak
> > Japanese,
> > but this time, 100% of speakers have their speeches in English, so we
> > wanted to announce it.
> > We look forward to your participation.
> >
> > The participation link: https://meet.google.com/cdh-zuun-uqu
> > The event page: https://solr.doorkeeper.jp/events/155335
> >
> > Thanks!
> >
>

[Operator] [ANNOUNCE] Apache Solr Operator v0.7.0 released

[Houston Putman]

The Apache Solr PMC is pleased to announce the release of the Apache Solr
Operator v0.7.0.

The Apache Solr Operator is a safe and easy way of managing a Solr
ecosystem in Kubernetes.

This release contains numerous bug fixes, optimizations, and improvements,
some of which are highlighted below. The release is available for immediate
download at:

  

### Solr Operator v0.7.0 Release Highlights:

* The minimum Kubernetes version supported is now v1.21.
* The required Zookeeper Operator version has been upgraded to v0.2.15.
* Clusterwide PodDisruptionBudgets are now created for SolrClouds by
default.
* During a rolling restart/update, SolrCloud pods are now taken out of the
common service endpoint before they are decommissioned. This should improve
request reliability during rolling restarts.
* Many missing Provided Zookeeper options have been added.
* Using volume backupRepositories now initialize faster on pod creation and
work with custom user/fsGroup.
* Default SolrCloud and SolrPrometheusExporter Pod probes have been
improved.
* Bug fixed in non-recurring backups.
* Bug fixed with custom persistent data volume names.

A summary of important changes is published in the documentation at:

  

For the most exhaustive list, see the change log on ArtifactHub or view the
git history in the solr-operator repo.

  <
https://artifacthub.io/packages/helm/apache-solr/solr-operator?modal=changelog
>

  

Re: Query on Solr and ZK ports

[HariBabu kuruva]

Thank you for the information.

On Fri, Apr 21, 2023 at 4:53 AM Ishan Chattopadhyaya <
ichattopadhy...@gmail.com> wrote:

> ZK admin server port is 8080,
>
> https://zookeeper.apache.org/doc/r3.5.1-alpha/zookeeperAdmin.html#sc_adminserver
>
> In my usage in the past, it spits out useful metrics around number of times
> a znode was accessed.
>
> On Tue, 18 Apr, 2023, 1:50 pm Jan Høydahl,  wrote:
>
> > Hi,
> >
> > See https://unix.stackexchange.com/questions/684348/solr-ports-in-use
> for
> > an answer.
> > The STOP_PORT is the Jetty Servlet container's way to shutting down. It
> > will only listen on localhost, so you need to use bin/solr stop command
> on
> > the host, i.e. no need to expose this port to the outside.
> >
> > Wrt ZK ports, Only port 2181 needs to be open between Solr and ZK, but
> > between the Zookeepers, also ports 2888 and 3888 needs to be open for
> > internal communication, see
> >
> https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.0/bk_reference/content/zookeeper-ports.html
> > I'm not aware of other ports.
> >
> > Jan
> >
> > > 18. apr. 2023 kl. 05:54 skrev HariBabu kuruva <
> hari2708.kur...@gmail.com
> > >:
> > >
> > > Can someone please help with this information?
> > >
> > > On Tue, Apr 4, 2023 at 10:01 PM HariBabu kuruva <
> > hari2708.kur...@gmail.com>
> > > wrote:
> > >
> > >> Hi All,
> > >>
> > >> I could see the solr process is Listening on 7981 port along with the
> > >> normal solr port(8981). It is shown as DSTOP PORT , when I grep solr
> > >> process. Could you please give more details on this port, Can we
> disable
> > >> this ?
> > >>
> > >> With regards to Zookeeper I could see port 8080 as a ZK admin port,
> How
> > >> can I use this, can i disable it if I don't want it ?
> > >> Also I could see ZK is listening on some random port (43801) along
> with
> > >> the other ports. Please throw some light on this.
> > >>
> > >> --
> > >>
> > >> Thanks and Regards,
> > >> Hari
> > >> Mobile:9790756568
> > >>
> > >
> > >
> > > --
> > >
> > > Thanks and Regards,
> > > Hari
> > > Mobile:9790756568
> >
> >
>


-- 

Thanks and Regards,
 Hari
Mobile:9790756568