Re: Increase in response time in case of collapse queries.

[Florin Babes]

Hello,
First let's call the field you collapse on group_field
If group_field has a high cardinality you should make group_field null for
those documents that have a unique group_field and set nullPolicy=expand.
By doing that solr will use less memory for it's internal maps (so faster
gc) and the head selecting will be faster.
What is your head selecting strategy? Can you share your fq which you use
for collapsing?

Thanks,
Florin Babes



În lun., 8 mar. 2021 la 06:44, Parshant Kumar
 a scris:

> anyone please help
>
> On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> parshant.ku...@indiamart.com>
> wrote:
>
> > Hi all,
> >
> > We have implemented collapse queries in place of grouped queries on our
> > production solr. As mentioned in solr documentation collapse queries are
> > recommended in place of grouped queries in terms of performance . But
> after
> > switching to collapsed queries from grouped queries response time of
> > queries have increased. This is unexpected behaviour, the response time
> > should have been improved but results are opposites.
> > Please someone help why response time is increased for collapsed queries.
> >
> > Thanks
> > Parshant Kumar
> >
>
> --
>
>

Re: Increase in response time in case of collapse queries.

[Parshant Kumar]

Hi florin,

I am using below.

1) fq={!collapse field=parentglusrid}
2) expand.rows=4
3) expand=true

Size of index is around 100GB.
Solr version is 6.5

On Mon, Mar 8, 2021 at 1:46 PM Florin Babes  wrote:

> Hello,
> First let's call the field you collapse on group_field
> If group_field has a high cardinality you should make group_field null for
> those documents that have a unique group_field and set nullPolicy=expand.
> By doing that solr will use less memory for it's internal maps (so faster
> gc) and the head selecting will be faster.
> What is your head selecting strategy? Can you share your fq which you use
> for collapsing?
>
> Thanks,
> Florin Babes
>
>
>
> În lun., 8 mar. 2021 la 06:44, Parshant Kumar
>  a scris:
>
> > anyone please help
> >
> > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > parshant.ku...@indiamart.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > We have implemented collapse queries in place of grouped queries on our
> > > production solr. As mentioned in solr documentation collapse queries
> are
> > > recommended in place of grouped queries in terms of performance . But
> > after
> > > switching to collapsed queries from grouped queries response time of
> > > queries have increased. This is unexpected behaviour, the response time
> > > should have been improved but results are opposites.
> > > Please someone help why response time is increased for collapsed
> queries.
> > >
> > > Thanks
> > > Parshant Kumar
> > >
> >
> > --
> >
> >
>

-- 

Re: Unsubscribe me please

[serwah]

Please unsubscribe me

Thanks

On Mon, 8 Mar 2021 at 02:20, Li, Yi  wrote:

> Please Unsubscribe me
>
> On 3/7/21, 7:29 PM, "Yuval Dotan"  wrote:
>
> LEARN FAST: This email originated outside of HERE.
> Please do not click on links or open attachments unless you recognize
> the sender and know the content is safe. Thank you.
>
>
> Me too
>
> On Mon, Mar 8, 2021, 02:22 Iniyan  wrote:
>
> > --
> > Regards,
> > Iniyan P
> >
>
>

Re: Unsubscribe me please

[yave guadaño]

 Please unsubscribe me

En lunes, 8 de marzo de 2021 10:23:43 CET, serwah  
escribió:  
 
 Please unsubscribe me

Thanks

On Mon, 8 Mar 2021 at 02:20, Li, Yi  wrote:

> Please Unsubscribe me
>
> On 3/7/21, 7:29 PM, "Yuval Dotan"  wrote:
>
>    LEARN FAST: This email originated outside of HERE.
>    Please do not click on links or open attachments unless you recognize
> the sender and know the content is safe. Thank you.
>
>
>    Me too
>
>    On Mon, Mar 8, 2021, 02:22 Iniyan  wrote:
>
>    > --
>    > Regards,
>    > Iniyan P
>    >
>
>
  

Re: Unsubscribe me please

[Furkan KAMACI]

Hi All,

Please send a request to users-unsubscr...@solr.apache.org

Let me know if you cannot unsubscribe. Please do not send such emails into
the Solr user list.

Kind Regards,
Furkan KAMACI

On Mon, Mar 8, 2021 at 12:30 PM serwah  wrote:

> Please unsubscribe me
>
> Thanks
>
> On Mon, 8 Mar 2021 at 02:20, Li, Yi  wrote:
>
> > Please Unsubscribe me
> >
> > On 3/7/21, 7:29 PM, "Yuval Dotan"  wrote:
> >
> > LEARN FAST: This email originated outside of HERE.
> > Please do not click on links or open attachments unless you recognize
> > the sender and know the content is safe. Thank you.
> >
> >
> > Me too
> >
> > On Mon, Mar 8, 2021, 02:22 Iniyan  wrote:
> >
> > > --
> > > Regards,
> > > Iniyan P
> > >
> >
> >
>

Re: Test email to migrated list

[Furkan KAMACI]

Hi All,

Please send a request to users-unsubscr...@solr.apache.org

Let me know if you cannot unsubscribe. Please do not send such emails into
the Solr user list.

Kind Regards,
Furkan KAMACI

On Mon, Mar 8, 2021 at 4:05 AM Tiong Jeffrey 
wrote:

> Unsubscribe
>
> Get Outlook for Android
> 
> From: Manoj Bharadwaj 
> Sent: Monday, March 8, 2021 8:00:38 AM
> To: users@solr.apache.org 
> Subject: Re: Test email to migrated list
>
> unsubscribe
>
> On Fri, Mar 5, 2021 at 5:13 PM Anshum Gupta 
> wrote:
>
> > Testing the migrated mailing list users@solr.apache.org
> >
> > --
> > Anshum Gupta
> >
>

CustomBreakIterator Performance Issues

[df2832368_...@amberoad.de df2832368_...@amberoad.de]

Hello,

I am currently working on getting a custom BreakIterator for the Unified 
Highlighter to work, and struggle a bit performance wise.

I need a BreakIterator for getting nice highlights of passages. For this I want 
the start of the highlight to be a sentence-start and the end to be a word-end. 
There are also some weird edge cases.

I already coded the BreakIterator and integrated it to our custom 
UnifiedHighlighter class, but when I use this Iterator the qTime of all 
requests rise from ~1000 to 12000+ which is not acceptable for this application.

Here is a link to my implementation. I can't really find where I am horrible 
inefficient.(I know that these functions get called very often)

Any suggestions are welcome, also other approaches.

So are there some nice resources to learn more about BreakIterators and stuff, 
since digging into the code is really hard here.

Another approach I am considering next is to do this highlight "trimming", when 
the final highlights are found. This would reduce the amount of logic called, 
but I guess the scoring system of SOLR wouldn't be taken in to account the 
right way.

As I said all suggestions are welcome and thanks in advance.

Jan Ulrich Robens

Re: CustomBreakIterator Performance Issues

[df2832368_...@amberoad.de df2832368_...@amberoad.de]

And of cource the link broke : 
https://drive.google.com/file/d/1wfZFQD6loTeA9_-eGrdwi9YGtJcNjKli/view?usp=sharing

> df2832368_...@amberoad.de df2832368_...@amberoad.de  
> hat am 08.03.2021 11:05 geschrieben:
> 
> 
> Hello,
> 
> I am currently working on getting a custom BreakIterator for the Unified 
> Highlighter to work, and struggle a bit performance wise.
> 
> I need a BreakIterator for getting nice highlights of passages. For this 
> I want the start of the highlight to be a sentence-start and the end to be a 
> word-end. There are also some weird edge cases.
> 
> I already coded the BreakIterator and integrated it to our custom 
> UnifiedHighlighter class, but when I use this Iterator the qTime of all 
> requests rise from ~1000 to 12000+ which is not acceptable for this 
> application.
> 
> Here is a link to my implementation. I can't really find where I am 
> horrible inefficient.(I know that these functions get called very often)
> 
> Any suggestions are welcome, also other approaches.
> 
> So are there some nice resources to learn more about BreakIterators and 
> stuff, since digging into the code is really hard here.
> 
> Another approach I am considering next is to do this highlight 
> "trimming", when the final highlights are found. This would reduce the amount 
> of logic called, but I guess the scoring system of SOLR wouldn't be taken in 
> to account the right way.
> 
> As I said all suggestions are welcome and thanks in advance.
> 
> Jan Ulrich Robens
> 

Re: Increase in response time in case of collapse queries.

[Florin Babes]

Your group_field has a high cardinality?
Thanks,
Florin Babes

În lun., 8 mar. 2021 la 10:35, Parshant Kumar
 a scris:

> Hi florin,
>
> I am using below.
>
> 1) fq={!collapse field=parentglusrid}
> 2) expand.rows=4
> 3) expand=true
>
> Size of index is around 100GB.
> Solr version is 6.5
>
> On Mon, Mar 8, 2021 at 1:46 PM Florin Babes  wrote:
>
> > Hello,
> > First let's call the field you collapse on group_field
> > If group_field has a high cardinality you should make group_field null
> for
> > those documents that have a unique group_field and set nullPolicy=expand.
> > By doing that solr will use less memory for it's internal maps (so faster
> > gc) and the head selecting will be faster.
> > What is your head selecting strategy? Can you share your fq which you use
> > for collapsing?
> >
> > Thanks,
> > Florin Babes
> >
> >
> >
> > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> >  a scris:
> >
> > > anyone please help
> > >
> > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > parshant.ku...@indiamart.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > We have implemented collapse queries in place of grouped queries on
> our
> > > > production solr. As mentioned in solr documentation collapse queries
> > are
> > > > recommended in place of grouped queries in terms of performance . But
> > > after
> > > > switching to collapsed queries from grouped queries response time of
> > > > queries have increased. This is unexpected behaviour, the response
> time
> > > > should have been improved but results are opposites.
> > > > Please someone help why response time is increased for collapsed
> > queries.
> > > >
> > > > Thanks
> > > > Parshant Kumar
> > > >
> > >
> > > --
> > >
> > >
> >
>
> --
>
>

Re: Increase in response time in case of collapse queries.

[Parshant Kumar]

yes,group_field is having high cardinality.


Thanks
Parshant Kumar

On Mon, Mar 8, 2021 at 4:06 PM Florin Babes  wrote:

> Your group_field has a high cardinality?
> Thanks,
> Florin Babes
>
> În lun., 8 mar. 2021 la 10:35, Parshant Kumar
>  a scris:
>
> > Hi florin,
> >
> > I am using below.
> >
> > 1) fq={!collapse field=parentglusrid}
> > 2) expand.rows=4
> > 3) expand=true
> >
> > Size of index is around 100GB.
> > Solr version is 6.5
> >
> > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes 
> wrote:
> >
> > > Hello,
> > > First let's call the field you collapse on group_field
> > > If group_field has a high cardinality you should make group_field null
> > for
> > > those documents that have a unique group_field and set
> nullPolicy=expand.
> > > By doing that solr will use less memory for it's internal maps (so
> faster
> > > gc) and the head selecting will be faster.
> > > What is your head selecting strategy? Can you share your fq which you
> use
> > > for collapsing?
> > >
> > > Thanks,
> > > Florin Babes
> > >
> > >
> > >
> > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > >  a scris:
> > >
> > > > anyone please help
> > > >
> > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > parshant.ku...@indiamart.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > We have implemented collapse queries in place of grouped queries on
> > our
> > > > > production solr. As mentioned in solr documentation collapse
> queries
> > > are
> > > > > recommended in place of grouped queries in terms of performance .
> But
> > > > after
> > > > > switching to collapsed queries from grouped queries response time
> of
> > > > > queries have increased. This is unexpected behaviour, the response
> > time
> > > > > should have been improved but results are opposites.
> > > > > Please someone help why response time is increased for collapsed
> > > queries.
> > > > >
> > > > > Thanks
> > > > > Parshant Kumar
> > > > >
> > > >
> > > > --
> > > >
> > > >
> > >
> >
> > --
> >
> >
>

-- 

unsubscribe-us...@solr.apache.org

[jerome . dupont]

unsubscribe-us...@solr.apache.org

En raison des directives gouvernementales liées à la situation sanitaire, les 
expositions restent fermées jusqu'à nouvelle consigne. Les manifestations 
culturelles ne peuvent pas accueillir de public mais sont en grande partie  
diffusées en ligne . La bibliothèque tous publics est ouverte du mardi au 
vendredi de 10 h à 17 h. 
Les bibliothèques de recherche sont ouvertes, sur le site François-Mitterrand, 
le lundi de 14 h à 17 h et du mardi au vendredi de 10 h à 17 h, et, sur les 
sites Richelieu, Arsenal et Opéra, de 10 h à 17 h du lundi au vendredi.  
Consulter les modalités d'accès Avant d'imprimer, pensez à l'environnement.

Re: Increase in response time in case of collapse queries.

[Florin Babes]

We improved the performance of collapse by making the group_field null for
the documents that have an unique value for group_field. This might help/


În lun., 8 mar. 2021 la 12:40, Parshant Kumar
 a scris:

> yes,group_field is having high cardinality.
>
>
> Thanks
> Parshant Kumar
>
> On Mon, Mar 8, 2021 at 4:06 PM Florin Babes  wrote:
>
> > Your group_field has a high cardinality?
> > Thanks,
> > Florin Babes
> >
> > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> >  a scris:
> >
> > > Hi florin,
> > >
> > > I am using below.
> > >
> > > 1) fq={!collapse field=parentglusrid}
> > > 2) expand.rows=4
> > > 3) expand=true
> > >
> > > Size of index is around 100GB.
> > > Solr version is 6.5
> > >
> > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes 
> > wrote:
> > >
> > > > Hello,
> > > > First let's call the field you collapse on group_field
> > > > If group_field has a high cardinality you should make group_field
> null
> > > for
> > > > those documents that have a unique group_field and set
> > nullPolicy=expand.
> > > > By doing that solr will use less memory for it's internal maps (so
> > faster
> > > > gc) and the head selecting will be faster.
> > > > What is your head selecting strategy? Can you share your fq which you
> > use
> > > > for collapsing?
> > > >
> > > > Thanks,
> > > > Florin Babes
> > > >
> > > >
> > > >
> > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > >  a scris:
> > > >
> > > > > anyone please help
> > > > >
> > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > > parshant.ku...@indiamart.com>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We have implemented collapse queries in place of grouped queries
> on
> > > our
> > > > > > production solr. As mentioned in solr documentation collapse
> > queries
> > > > are
> > > > > > recommended in place of grouped queries in terms of performance .
> > But
> > > > > after
> > > > > > switching to collapsed queries from grouped queries response time
> > of
> > > > > > queries have increased. This is unexpected behaviour, the
> response
> > > time
> > > > > > should have been improved but results are opposites.
> > > > > > Please someone help why response time is increased for collapsed
> > > > queries.
> > > > > >
> > > > > > Thanks
> > > > > > Parshant Kumar
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > >
> > > >
> > >
> > > --
> > >
> > >
> >
>
> --
>
>

Re: Increase in response time in case of collapse queries.

[Gajendra Dadheech]

Hey Florin. What was incremental benefit of this optimization of have group
field null for unique docs.

On Mon, Mar 8, 2021, 4:41 PM Florin Babes  wrote:

> We improved the performance of collapse by making the group_field null for
> the documents that have an unique value for group_field. This might help/
>
>
> În lun., 8 mar. 2021 la 12:40, Parshant Kumar
>  a scris:
>
> > yes,group_field is having high cardinality.
> >
> >
> > Thanks
> > Parshant Kumar
> >
> > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes 
> wrote:
> >
> > > Your group_field has a high cardinality?
> > > Thanks,
> > > Florin Babes
> > >
> > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > >  a scris:
> > >
> > > > Hi florin,
> > > >
> > > > I am using below.
> > > >
> > > > 1) fq={!collapse field=parentglusrid}
> > > > 2) expand.rows=4
> > > > 3) expand=true
> > > >
> > > > Size of index is around 100GB.
> > > > Solr version is 6.5
> > > >
> > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes 
> > > wrote:
> > > >
> > > > > Hello,
> > > > > First let's call the field you collapse on group_field
> > > > > If group_field has a high cardinality you should make group_field
> > null
> > > > for
> > > > > those documents that have a unique group_field and set
> > > nullPolicy=expand.
> > > > > By doing that solr will use less memory for it's internal maps (so
> > > faster
> > > > > gc) and the head selecting will be faster.
> > > > > What is your head selecting strategy? Can you share your fq which
> you
> > > use
> > > > > for collapsing?
> > > > >
> > > > > Thanks,
> > > > > Florin Babes
> > > > >
> > > > >
> > > > >
> > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > > >  a scris:
> > > > >
> > > > > > anyone please help
> > > > > >
> > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > > > parshant.ku...@indiamart.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We have implemented collapse queries in place of grouped
> queries
> > on
> > > > our
> > > > > > > production solr. As mentioned in solr documentation collapse
> > > queries
> > > > > are
> > > > > > > recommended in place of grouped queries in terms of
> performance .
> > > But
> > > > > > after
> > > > > > > switching to collapsed queries from grouped queries response
> time
> > > of
> > > > > > > queries have increased. This is unexpected behaviour, the
> > response
> > > > time
> > > > > > > should have been improved but results are opposites.
> > > > > > > Please someone help why response time is increased for
> collapsed
> > > > > queries.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Parshant Kumar
> > > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > >
> > >
> >
> > --
> >
> >
>

Re: Increase in response time in case of collapse queries.

[Parshant Kumar]

How can we make group_field null? Using nullPolicy=expand ?

On Mon, Mar 8, 2021 at 4:41 PM Florin Babes  wrote:

> We improved the performance of collapse by making the group_field null for
> the documents that have an unique value for group_field. This might help/
>
>
> În lun., 8 mar. 2021 la 12:40, Parshant Kumar
>  a scris:
>
> > yes,group_field is having high cardinality.
> >
> >
> > Thanks
> > Parshant Kumar
> >
> > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes 
> wrote:
> >
> > > Your group_field has a high cardinality?
> > > Thanks,
> > > Florin Babes
> > >
> > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > >  a scris:
> > >
> > > > Hi florin,
> > > >
> > > > I am using below.
> > > >
> > > > 1) fq={!collapse field=parentglusrid}
> > > > 2) expand.rows=4
> > > > 3) expand=true
> > > >
> > > > Size of index is around 100GB.
> > > > Solr version is 6.5
> > > >
> > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes 
> > > wrote:
> > > >
> > > > > Hello,
> > > > > First let's call the field you collapse on group_field
> > > > > If group_field has a high cardinality you should make group_field
> > null
> > > > for
> > > > > those documents that have a unique group_field and set
> > > nullPolicy=expand.
> > > > > By doing that solr will use less memory for it's internal maps (so
> > > faster
> > > > > gc) and the head selecting will be faster.
> > > > > What is your head selecting strategy? Can you share your fq which
> you
> > > use
> > > > > for collapsing?
> > > > >
> > > > > Thanks,
> > > > > Florin Babes
> > > > >
> > > > >
> > > > >
> > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > > >  a scris:
> > > > >
> > > > > > anyone please help
> > > > > >
> > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > > > parshant.ku...@indiamart.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We have implemented collapse queries in place of grouped
> queries
> > on
> > > > our
> > > > > > > production solr. As mentioned in solr documentation collapse
> > > queries
> > > > > are
> > > > > > > recommended in place of grouped queries in terms of
> performance .
> > > But
> > > > > > after
> > > > > > > switching to collapsed queries from grouped queries response
> time
> > > of
> > > > > > > queries have increased. This is unexpected behaviour, the
> > response
> > > > time
> > > > > > > should have been improved but results are opposites.
> > > > > > > Please someone help why response time is increased for
> collapsed
> > > > > queries.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Parshant Kumar
> > > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > >
> > >
> >
> > --
> >
> >
>

-- 

Re: Increase in response time in case of collapse queries.

[Florin Babes]

@Gajendra Our response time dropped by 36% and our rps increased with 27%.

You have to reindex the core and the client should set to null the field if
it's unique.

În lun., 8 mar. 2021 la 13:18, Parshant Kumar
 a scris:

> How can we make group_field null? Using nullPolicy=expand ?
>
> On Mon, Mar 8, 2021 at 4:41 PM Florin Babes  wrote:
>
> > We improved the performance of collapse by making the group_field null
> for
> > the documents that have an unique value for group_field. This might help/
> >
> >
> > În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> >  a scris:
> >
> > > yes,group_field is having high cardinality.
> > >
> > >
> > > Thanks
> > > Parshant Kumar
> > >
> > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes 
> > wrote:
> > >
> > > > Your group_field has a high cardinality?
> > > > Thanks,
> > > > Florin Babes
> > > >
> > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > > >  a scris:
> > > >
> > > > > Hi florin,
> > > > >
> > > > > I am using below.
> > > > >
> > > > > 1) fq={!collapse field=parentglusrid}
> > > > > 2) expand.rows=4
> > > > > 3) expand=true
> > > > >
> > > > > Size of index is around 100GB.
> > > > > Solr version is 6.5
> > > > >
> > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes  >
> > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > > First let's call the field you collapse on group_field
> > > > > > If group_field has a high cardinality you should make group_field
> > > null
> > > > > for
> > > > > > those documents that have a unique group_field and set
> > > > nullPolicy=expand.
> > > > > > By doing that solr will use less memory for it's internal maps
> (so
> > > > faster
> > > > > > gc) and the head selecting will be faster.
> > > > > > What is your head selecting strategy? Can you share your fq which
> > you
> > > > use
> > > > > > for collapsing?
> > > > > >
> > > > > > Thanks,
> > > > > > Florin Babes
> > > > > >
> > > > > >
> > > > > >
> > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > > > >  a scris:
> > > > > >
> > > > > > > anyone please help
> > > > > > >
> > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > > > > parshant.ku...@indiamart.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > We have implemented collapse queries in place of grouped
> > queries
> > > on
> > > > > our
> > > > > > > > production solr. As mentioned in solr documentation collapse
> > > > queries
> > > > > > are
> > > > > > > > recommended in place of grouped queries in terms of
> > performance .
> > > > But
> > > > > > > after
> > > > > > > > switching to collapsed queries from grouped queries response
> > time
> > > > of
> > > > > > > > queries have increased. This is unexpected behaviour, the
> > > response
> > > > > time
> > > > > > > > should have been improved but results are opposites.
> > > > > > > > Please someone help why response time is increased for
> > collapsed
> > > > > > queries.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Parshant Kumar
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > >
> > > >
> > >
> > > --
> > >
> > >
> >
>
> --
>
>

Re: Solr backup no longer writes to a UNC path

[Jan Høydahl]

Hi,

That's right. UNC paths are always disallowed since 8.6. See 
https://solr.apache.org/guide/8_6/solr-upgrade-notes.html#solr-8-6 and 
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/core/SolrPaths.java#L64
Solution is to mount the remote server as a drive letter on the Solr hosts.

Jan


> 3. mar. 2021 kl. 12:34 skrev Gell-Holleron, Daniel 
> :
> 
> Hi there,
> 
> We've upgraded from Solr 7.7.1 to Solr 8.8.1 (running on Windows Operating 
> System) and I've noticed that when running a Solr backup, it will no longer 
> allow me to write to a UNC path. Is this something that has been purposely 
> changed?
> 
> I've noticed a new system property called SOLR_OPTS="%SOLR_OPTS% 
> -Dsolr.allowPath=S:\" which I've enabled. Is there a way to point this to a 
> remote server, rather than a different drive on the local server?
> 
> Thanks,
> 
> Daniel
> 

Re: CustomBreakIterator Performance Issues

[David Smiley]

The BreakIterator impls in the JDK (and likely IBM ICU) seem slow and can
sometimes dominate the performance of this highlighter.  I worked on a
large search project (which led to the creation of the UnifiedHighlighter)
and we used a technique of encoding the breaks directly into the text
a-priori.  It was just a special character.  Perhaps use a "vertical tab"?
On the Solr side, it then became a very trivial char based iterator which
is already in Lucene/Solr.  You might do this as well.  You could add a
custom Solr UpdateRequestProcessor (URP) that inserts these characters.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Mon, Mar 8, 2021 at 5:06 AM df2832368_...@amberoad.de
df2832368_...@amberoad.de  wrote:

> And of cource the link broke :
> https://drive.google.com/file/d/1wfZFQD6loTeA9_-eGrdwi9YGtJcNjKli/view?usp=sharing
>
> > df2832368_...@amberoad.de df2832368_...@amberoad.de 
> hat am 08.03.2021 11:05 geschrieben:
> >
> >
> > Hello,
> >
> > I am currently working on getting a custom BreakIterator for the
> Unified Highlighter to work, and struggle a bit performance wise.
> >
> > I need a BreakIterator for getting nice highlights of passages. For
> this I want the start of the highlight to be a sentence-start and the end
> to be a word-end. There are also some weird edge cases.
> >
> > I already coded the BreakIterator and integrated it to our custom
> UnifiedHighlighter class, but when I use this Iterator the qTime of all
> requests rise from ~1000 to 12000+ which is not acceptable for this
> application.
> >
> > Here is a link to my implementation. I can't really find where I am
> horrible inefficient.(I know that these functions get called very often)
> >
> > Any suggestions are welcome, also other approaches.
> >
> > So are there some nice resources to learn more about BreakIterators
> and stuff, since digging into the code is really hard here.
> >
> > Another approach I am considering next is to do this highlight
> "trimming", when the final highlights are found. This would reduce the
> amount of logic called, but I guess the scoring system of SOLR wouldn't be
> taken in to account the right way.
> >
> > As I said all suggestions are welcome and thanks in advance.
> >
> > Jan Ulrich Robens
> >
>

Re: Increase in response time in case of collapse queries.

[Parshant Kumar]

client should set to null the field if it's unique.

@florin @Gajendra can you please explain more .I am not clear how to
perform this.

On Mon, Mar 8, 2021 at 6:09 PM Florin Babes  wrote:

> @Gajendra Our response time dropped by 36% and our rps increased with 27%.
>
> You have to reindex the core and the client should set to null the field if
> it's unique.
>
> În lun., 8 mar. 2021 la 13:18, Parshant Kumar
>  a scris:
>
> > How can we make group_field null? Using nullPolicy=expand ?
> >
> > On Mon, Mar 8, 2021 at 4:41 PM Florin Babes 
> wrote:
> >
> > > We improved the performance of collapse by making the group_field null
> > for
> > > the documents that have an unique value for group_field. This might
> help/
> > >
> > >
> > > În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> > >  a scris:
> > >
> > > > yes,group_field is having high cardinality.
> > > >
> > > >
> > > > Thanks
> > > > Parshant Kumar
> > > >
> > > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes 
> > > wrote:
> > > >
> > > > > Your group_field has a high cardinality?
> > > > > Thanks,
> > > > > Florin Babes
> > > > >
> > > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > > > >  a scris:
> > > > >
> > > > > > Hi florin,
> > > > > >
> > > > > > I am using below.
> > > > > >
> > > > > > 1) fq={!collapse field=parentglusrid}
> > > > > > 2) expand.rows=4
> > > > > > 3) expand=true
> > > > > >
> > > > > > Size of index is around 100GB.
> > > > > > Solr version is 6.5
> > > > > >
> > > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <
> babesflo...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > > First let's call the field you collapse on group_field
> > > > > > > If group_field has a high cardinality you should make
> group_field
> > > > null
> > > > > > for
> > > > > > > those documents that have a unique group_field and set
> > > > > nullPolicy=expand.
> > > > > > > By doing that solr will use less memory for it's internal maps
> > (so
> > > > > faster
> > > > > > > gc) and the head selecting will be faster.
> > > > > > > What is your head selecting strategy? Can you share your fq
> which
> > > you
> > > > > use
> > > > > > > for collapsing?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Florin Babes
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > > > > >  a scris:
> > > > > > >
> > > > > > > > anyone please help
> > > > > > > >
> > > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > > > > > parshant.ku...@indiamart.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > We have implemented collapse queries in place of grouped
> > > queries
> > > > on
> > > > > > our
> > > > > > > > > production solr. As mentioned in solr documentation
> collapse
> > > > > queries
> > > > > > > are
> > > > > > > > > recommended in place of grouped queries in terms of
> > > performance .
> > > > > But
> > > > > > > > after
> > > > > > > > > switching to collapsed queries from grouped queries
> response
> > > time
> > > > > of
> > > > > > > > > queries have increased. This is unexpected behaviour, the
> > > > response
> > > > > > time
> > > > > > > > > should have been improved but results are opposites.
> > > > > > > > > Please someone help why response time is increased for
> > > collapsed
> > > > > > > queries.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Parshant Kumar
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > >
> > >
> >
> > --
> >
> >
>

-- 

RE: Idle timeout expired and Early Client Disconnect errors

[ufuk yılmaz]

How do you “register” something like a CloudSolrStream btw? Using Blob Store 
API?

Sent from Mail for Windows 10

From: Susmit
Sent: 06 March 2021 23:03
To: users@solr.apache.org
Subject: Re: Idle timeout expired and Early Client Disconnect errors

better to use solr 8.9 and configure http timeouts from solr.in.sh
workaround is bigger - need to extend cloudsolrstream , register it and install 
custom solrclientcache with overridden setcontext method 

Sent from my iPhone

> On Mar 6, 2021, at 9:25 AM, ufuk yılmaz  wrote:
> 
> How? O_O
> 
> Sent from Mail for Windows 10
> 
> From: Susmit
> Sent: 06 March 2021 18:35
> To: solr-u...@lucene.apache.org
> Subject: Re: Idle timeout expired and Early Client Disconnect errors
> 
> i have used a workaround to increase the default (hard coded) timeout of 2 
> min in solrclientcache. 
> i can run 9+ hour long streaming queries with no issues.
> 
> Sent from my iPhone
> 
>> On Mar 2, 2021, at 5:32 PM, ufuk yılmaz  wrote:
>> 
>> I divided the query to 1000 pieces and removed the parallel stream clause, 
>> it seems to be working without timeout so far, if it does I just can divide 
>> it to even smaller pieces I guess.
>> 
>> I tried to send all 1000 pieces in a “list” expression to be executed 
>> linearly, it didn’t work but I was just curious if it could handle such a 
>> large query 😃
>> 
>> Now I’m just generating expression strings from java code and sending them 
>> one by one. I tried to use SolrJ for this, but encountered a weird problem 
>> where even the simplest expression (echo) stops working after a few 
>> iterations in a loop. I’m guessing the underlying HttpClient is not closing 
>> connections timely, hitting the OS per-host connection limit. I asked a 
>> separate question about this. I was following the example on lucidworks: 
>> https://lucidworks.com/post/streaming-expressions-in-solrj/
>> 
>> I just modified my code to use regular REST calls using okhttp3, it’s a 
>> shame that I couldn’t use SolrJ since it truly streams every result 1 by 1 
>> continuously. REST just returns a single large response at the very end of 
>> the stream.
>> 
>> Thanks again for your help.
>> 
>> Sent from Mail for Windows 10
>> 
>> From: Joel Bernstein
>> Sent: 02 March 2021 00:19
>> To: solr-u...@lucene.apache.org
>> Subject: Re: Idle timeout expired and Early Client Disconnect errors
>> 
>> Also the parallel function builds hash partitioning filters that could lead
>> to timeouts if they take too long to build. Try the query without the
>> parallel function if you're still getting timeouts when making the query
>> smaller.
>> 
>> 
>> 
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>> 
>> 
 On Mon, Mar 1, 2021 at 4:03 PM Joel Bernstein  wrote:
>>> 
>>> The settings in your version are 30 seconds and 15 seconds for socket and
>>> connection timeouts.
>>> 
>>> Typically timeouts occur because one or more shards in the query are idle
>>> beyond the timeout threshold. This happens because lot's of data is being
>>> read from other shards.
>>> 
>>> Breaking the query into small parts would be a good strategy.
>>> 
>>> 
>>> 
>>> 
>>> Joel Bernstein
>>> http://joelsolr.blogspot.com/
>>> 
>>> 
>>> On Mon, Mar 1, 2021 at 3:30 PM ufuk yılmaz 
>>> wrote:
>>> 
 Hello Mr. Bernstein,
 
 I’m using version 8.4. So, if I understand correctly, I can’t increase
 timeouts and they are bound to happen in such a large stream. Should I just
 reduce the output of my search expressions?
 
 Maybe I can split my search results into ~100 parts and run the same
 query 100 times in series. Each part would emit ~3M documents so they
 should finish before timeout?
 
 Is this a reasonable solution?
 
 Btw how long is the default hard-coded timeout value? Because yesterday I
 ran another query which took more than 1 hour without any timeouts and
 finished successfully.
 
 Sent from Mail for Windows 10
 
 From: Joel Bernstein
 Sent: 01 March 2021 23:03
 To: solr-u...@lucene.apache.org
 Subject: Re: Idle timeout expired and Early Client Disconnect errors
 
 Oh wait, I misread your email. The idle timeout issue is configurable in:
 
 https://issues.apache.org/jira/browse/SOLR-14672
 
 This unfortunately missed the 8.8 release and will be 8.9.
 
 
 
 This i
 
 
 
 Joel Bernstein
 http://joelsolr.blogspot.com/
 
 
> On Mon, Mar 1, 2021 at 2:56 PM Joel Bernstein  wrote:
 
> What version are you using?
> 
> Solr 8.7 has changes that caused these errors to hit the logs. These
 used
> to be suppressed. This has been fixed in Solr 9.0 but it has not been
 back
> ported to Solr 8.x.
> 
> The errors are actually normal operational occurrences when doing joins
 so
> should be suppressed in the logs and were before the specific release.
> 
> It might make sense to do a release that specifica

High thread count while writing collection

[DAVID MARTIN NIETO]

Hello,

We have a cluster of solr 8.2 and sometimes when we have a high load of 
incoming queries about solr and we have to reindex at the same time, the 
following happens:
- One of the nodes or several that make up the cluster of solr sees how its 
processes are triggered in the order of 1000 until reaching 5000 and even 
sometimes up to more than 1 degrading the machine to the SO level with the 
consecuent loss of service, of that node.
- Looking at the detail of the thread dump we see that what appears in these 
cases are threads of this style:
qtp288994035-118554 (118554)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@471882ba

sun.misc.Unsafe.park (Native Method)
java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer $ 
ConditionObject.awaitNanos (AbstractQueuedSynchronizer.java:2078)
org.eclipse.jetty.util.BlockingArrayQueue.poll (BlockingArrayQueue.java:392)
org.eclipse.jetty.util.thread.QueuedThreadPool $ Runner.idleJobPoll 
(QueuedThreadPool.java:850)
org.eclipse.jetty.util.thread.QueuedThreadPool $ Runner.run 
(QueuedThreadPool.java:889)
java.lang.Thread.run (Thread.java:745)

Someone would know what could be due or how we could arrive at a specific 
cause. The solr or zookeeper logs do not show any problems at this time.

Example of one of this increasing:

[cid:5317ef95-a733-4218-b5d4-84b70324bc16]

The collection load batch that we use does not show any error either. 
Additionally, this upload of threads is not released after x minutes or hours 
but remains in time, descending in several days, which by thread timeouts I 
understand should disappear sooner.

Greetings and thank you for your help.


---
Este mensaje va dirigido únicamente a la(s) persona(s) y/o entidad(es) arriba 
relacionada(s). Puede contener información confidencial o legalmente protegida. 
Si no es usted el destinatario señalado, le rogamos borre del sistema 
inmediatamente el mensaje y sus copias. Asimismo le informamos que cualquier 
copia, divulgación, distribución o uso de los contenidos está prohibida.
---
This message is addressed only to the person (people) and / or entities listed 
above. It may contain confidential or legally protected information. If you are 
not the recipient indicated, please delete the message and its copies 
immediately from the system. We also inform that any copy, disclosure, 
distribution or use of the contents is forbidden
---
Viewnext, S.A. Domicilio Social: Avda. de Burgos 8-A 28036 de Madrid. telf: 
913834060, Fax: 913834090. Reg. M. Madrid: Tomo 3238, Libro:0, Folio: 78, 
Seccion: 8ª, Hoja M-55112, N.I.F.: A-80157746

Re: Increase in response time in case of collapse queries.

[Gajendra Dadheech]

@prashant Florin means to put null for parentglusrid in documents where
this field-value is only present in one document [Group has only one
document]. and then use nullPolicy to include/expand.



On Mon, Mar 8, 2021 at 6:55 PM Parshant Kumar
 wrote:

> client should set to null the field if it's unique.
>
> @florin @Gajendra can you please explain more .I am not clear how to
> perform this.
>
> On Mon, Mar 8, 2021 at 6:09 PM Florin Babes  wrote:
>
> > @Gajendra Our response time dropped by 36% and our rps increased with
> 27%.
> >
> > You have to reindex the core and the client should set to null the field
> if
> > it's unique.
> >
> > În lun., 8 mar. 2021 la 13:18, Parshant Kumar
> >  a scris:
> >
> > > How can we make group_field null? Using nullPolicy=expand ?
> > >
> > > On Mon, Mar 8, 2021 at 4:41 PM Florin Babes 
> > wrote:
> > >
> > > > We improved the performance of collapse by making the group_field
> null
> > > for
> > > > the documents that have an unique value for group_field. This might
> > help/
> > > >
> > > >
> > > > În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> > > >  a scris:
> > > >
> > > > > yes,group_field is having high cardinality.
> > > > >
> > > > >
> > > > > Thanks
> > > > > Parshant Kumar
> > > > >
> > > > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes  >
> > > > wrote:
> > > > >
> > > > > > Your group_field has a high cardinality?
> > > > > > Thanks,
> > > > > > Florin Babes
> > > > > >
> > > > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > > > > >  a scris:
> > > > > >
> > > > > > > Hi florin,
> > > > > > >
> > > > > > > I am using below.
> > > > > > >
> > > > > > > 1) fq={!collapse field=parentglusrid}
> > > > > > > 2) expand.rows=4
> > > > > > > 3) expand=true
> > > > > > >
> > > > > > > Size of index is around 100GB.
> > > > > > > Solr version is 6.5
> > > > > > >
> > > > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <
> > babesflo...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > > First let's call the field you collapse on group_field
> > > > > > > > If group_field has a high cardinality you should make
> > group_field
> > > > > null
> > > > > > > for
> > > > > > > > those documents that have a unique group_field and set
> > > > > > nullPolicy=expand.
> > > > > > > > By doing that solr will use less memory for it's internal
> maps
> > > (so
> > > > > > faster
> > > > > > > > gc) and the head selecting will be faster.
> > > > > > > > What is your head selecting strategy? Can you share your fq
> > which
> > > > you
> > > > > > use
> > > > > > > > for collapsing?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Florin Babes
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > > > > > >  a scris:
> > > > > > > >
> > > > > > > > > anyone please help
> > > > > > > > >
> > > > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > > > > > > parshant.ku...@indiamart.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > We have implemented collapse queries in place of grouped
> > > > queries
> > > > > on
> > > > > > > our
> > > > > > > > > > production solr. As mentioned in solr documentation
> > collapse
> > > > > > queries
> > > > > > > > are
> > > > > > > > > > recommended in place of grouped queries in terms of
> > > > performance .
> > > > > > But
> > > > > > > > > after
> > > > > > > > > > switching to collapsed queries from grouped queries
> > response
> > > > time
> > > > > > of
> > > > > > > > > > queries have increased. This is unexpected behaviour, the
> > > > > response
> > > > > > > time
> > > > > > > > > > should have been improved but results are opposites.
> > > > > > > > > > Please someone help why response time is increased for
> > > > collapsed
> > > > > > > > queries.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Parshant Kumar
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > >
> > > >
> > >
> > > --
> > >
> > >
> >
>
> --
>
>

Re: Test email to migrated list

[Abhishek Sharma]

I am still getting the emails from solr group.

On Mon, Mar 8, 2021 at 3:10 PM Furkan KAMACI  wrote:

> Hi All,
>
> Please send a request to users-unsubscr...@solr.apache.org
>
> Let me know if you cannot unsubscribe. Please do not send such emails into
> the Solr user list.
>
> Kind Regards,
> Furkan KAMACI
>
> On Mon, Mar 8, 2021 at 4:05 AM Tiong Jeffrey 
> wrote:
>
> > Unsubscribe
> >
> > Get Outlook for Android
> > 
> > From: Manoj Bharadwaj 
> > Sent: Monday, March 8, 2021 8:00:38 AM
> > To: users@solr.apache.org 
> > Subject: Re: Test email to migrated list
> >
> > unsubscribe
> >
> > On Fri, Mar 5, 2021 at 5:13 PM Anshum Gupta 
> > wrote:
> >
> > > Testing the migrated mailing list users@solr.apache.org
> > >
> > > --
> > > Anshum Gupta
> > >
> >
>

Re: Test email to migrated list

[Paweł Gdula]

unsubscribe

On Mon, 8 Mar 2021 at 17:53, Abhishek Sharma  wrote:

> I am still getting the emails from solr group.
>
> On Mon, Mar 8, 2021 at 3:10 PM Furkan KAMACI 
> wrote:
>
> > Hi All,
> >
> > Please send a request to users-unsubscr...@solr.apache.org
> >
> > Let me know if you cannot unsubscribe. Please do not send such emails
> into
> > the Solr user list.
> >
> > Kind Regards,
> > Furkan KAMACI
> >
> > On Mon, Mar 8, 2021 at 4:05 AM Tiong Jeffrey 
> > wrote:
> >
> > > Unsubscribe
> > >
> > > Get Outlook for Android
> > > 
> > > From: Manoj Bharadwaj 
> > > Sent: Monday, March 8, 2021 8:00:38 AM
> > > To: users@solr.apache.org 
> > > Subject: Re: Test email to migrated list
> > >
> > > unsubscribe
> > >
> > > On Fri, Mar 5, 2021 at 5:13 PM Anshum Gupta 
> > > wrote:
> > >
> > > > Testing the migrated mailing list users@solr.apache.org
> > > >
> > > > --
> > > > Anshum Gupta
> > > >
> > >
> >
>

zk upconfig does not recognize ZK_HOST style string

[Subhajit Das]

Hi There,

When I was trying to upload new configset to zookeeper using Solr control 
script, the -z parameter is not recognizing ZK_HOST style string.

Say, I use ,,/solr, then the config is uploaded to  
directly, instead of /solr znode.

Can anyone please help on this matter, if this is how it is supposed to work.

Note: /solr,/solr,/solr seems o work.

Re: Test email to migrated list

[Furkan KAMACI]

Hi All,

For whom who cannot unsubscribe via users-unsubscr...@solr.apache.org

Please send an email to users-ow...@solr.apache.org instead of writing to
the user mail list, so I can help you via manual unsubscribe from the mail
list.

Kind Regards,
Furkan KAMACI

On Mon, Mar 8, 2021 at 8:11 PM Paweł Gdula  wrote:

> unsubscribe
>
> On Mon, 8 Mar 2021 at 17:53, Abhishek Sharma  wrote:
>
> > I am still getting the emails from solr group.
> >
> > On Mon, Mar 8, 2021 at 3:10 PM Furkan KAMACI 
> > wrote:
> >
> > > Hi All,
> > >
> > > Please send a request to users-unsubscr...@solr.apache.org
> > >
> > > Let me know if you cannot unsubscribe. Please do not send such emails
> > into
> > > the Solr user list.
> > >
> > > Kind Regards,
> > > Furkan KAMACI
> > >
> > > On Mon, Mar 8, 2021 at 4:05 AM Tiong Jeffrey 
> > > wrote:
> > >
> > > > Unsubscribe
> > > >
> > > > Get Outlook for Android
> > > > 
> > > > From: Manoj Bharadwaj 
> > > > Sent: Monday, March 8, 2021 8:00:38 AM
> > > > To: users@solr.apache.org 
> > > > Subject: Re: Test email to migrated list
> > > >
> > > > unsubscribe
> > > >
> > > > On Fri, Mar 5, 2021 at 5:13 PM Anshum Gupta 
> > > > wrote:
> > > >
> > > > > Testing the migrated mailing list users@solr.apache.org
> > > > >
> > > > > --
> > > > > Anshum Gupta
> > > > >
> > > >
> > >
> >
>

solr.CurrencyFieldType not fully replicating it's sub-fields

[Steven Novotny]

Hi Everyone,

We currently have a Solr 7.7.2 solrcloud setup (been meaning to update, but
here we are).  Our collections are 2 shards with 2 replicas.  I'm not sure
how long this has been going on exactly, but we recently added a feature to
be able to query on one of the fields of solr.CurrencyFieldType which has
brought this issue up.  It appears that somehow some of our replicas do not
always build (or receive) the sub-fields required to correctly query on the
currency.

We're using the basic example in our schema as outlined in the docs:

  
  
  
  

For example, here is a query, with debug on that properly returns those
fields:

{
  "responseHeader":{
"zkConnected":true,
"status":0,
"QTime":3,
"params":{
  "q":"id: \"SaccAccount:1122\"",

"fl":"total_balance_c,total_balance_cs_ns,total_balance_cl_ns",
  "debugQuery":"on"}},
  "response":{"numFound":1,"start":0,"maxScore":10.844471,"docs":[
  {
"total_balance_c":"255.0,USD",
"total_balance_cl_ns":25500,
"total_balance_cs_ns":"USD"}]
  },
  "debug":{
"track":{
  "rid":"solr1.example.com-customer_shard2_replica_n3-1615226174296-63",
  "EXECUTE_QUERY":{
"
http://solr4.example.com:8080/solr/customer_shard1_replica_n1/|http://solr3.example.com:8080/solr/customer_shard1_replica_n2/
":{
  "QTime":"0",
  "ElapsedTime":"0",
  "RequestPurpose":"GET_TOP_IDS",
  "NumFound":"0",

"Response":"{responseHeader={zkConnected=true,status=0,QTime=0,params={df=_text_,distrib=false,debug=[false,
timing, track],fl=[id,
score],shards.purpose=4,start=0,fsv=true,q.op=AND,shard.url=
http://solr4.example.com:8080/solr/customer_shard1_replica_n1/|http://solr3.example.com:8080/solr/customer_shard1_replica_n2/,rows=10,rid=solr1.example.com-customer_shard2_replica_n3-1615226174296-63,version=2,q=id:
\"SaccAccount:1122\",requestPurpose=GET_TOP_IDS,NOW=1615226174296,isShard=true,wt=javabin,debugQuery=false}},response={numFound=0,start=0,maxScore=0.0,docs=[]},sort_values={},debug={timing={time=0.0,prepare={time=0.0,query={time=0.0},facet={time=0.0},facet_module={time=0.0},mlt={time=0.0},highlight={time=0.0},stats={time=0.0},expand={time=0.0},terms={time=0.0},debug={time=0.0}},process={time=0.0,query={time=0.0},facet={time=0.0},facet_module={time=0.0},mlt={time=0.0},highlight={time=0.0},stats={time=0.0},expand={time=0.0},terms={time=0.0},debug={time=0.0}"},
"
http://solr5.example.com:8080/solr/customer_shard2_replica_n5/|http://solr1.example.com:8080/solr/customer_shard2_replica_n3/
":{
  "QTime":"0",
  "ElapsedTime":"1",
  "RequestPurpose":"GET_TOP_IDS",
  "NumFound":"1",

"Response":"{responseHeader={zkConnected=true,status=0,QTime=0,params={df=_text_,distrib=false,debug=[false,
timing, track],fl=[id,
score],shards.purpose=4,start=0,fsv=true,q.op=AND,shard.url=
http://solr5.example.com:8080/solr/customer_shard2_replica_n5/|http://solr1.example.com:8080/solr/customer_shard2_replica_n3/,rows=10,rid=solr1.example.com-customer_shard2_replica_n3-1615226174296-63,version=2,q=id:
\"SaccAccount:1122\",requestPurpose=GET_TOP_IDS,NOW=1615226174296,isShard=true,wt=javabin,debugQuery=false}},response={numFound=1,start=0,maxScore=10.844471,docs=[SolrDocument{id=SaccAccount:1122,
score=10.844471}]},sort_values={},debug={timing={time=0.0,prepare={time=0.0,query={time=0.0},facet={time=0.0},facet_module={time=0.0},mlt={time=0.0},highlight={time=0.0},stats={time=0.0},expand={time=0.0},terms={time=0.0},debug={time=0.0}},process={time=0.0,query={time=0.0},facet={time=0.0},facet_module={time=0.0},mlt={time=0.0},highlight={time=0.0},stats={time=0.0},expand={time=0.0},terms={time=0.0},debug={time=0.0}"}},
  "GET_FIELDS":{
"
http://solr5.example.com:8080/solr/customer_shard2_replica_n5/|http://solr1.example.com:8080/solr/customer_shard2_replica_n3/
":{
  "QTime":"0",
  "ElapsedTime":"1",
  "RequestPurpose":"GET_FIELDS,GET_DEBUG",
  "NumFound":"1",

"Response":"{responseHeader={zkConnected=true,status=0,QTime=0,params={df=_text_,distrib=false,debug=[timing,
track],fl=[total_balance_c,total_balance_cs_ns,total_balance_cl_ns,
id],shards.purpose=320,q.op=AND,shard.url=
http://solr5.example.com:8080/solr/customer_shard2_replica_n5/|http://solr1.example.com:8080/solr/customer_shard2_replica_n3/,rows=10,rid=solr1.example.com-customer_shard2_replica_n3-1615226174296-63,version=2,q=id:
\"SaccAccount:1122\",requestPurpose=GET_FIELDS,GET_DEBUG,NOW=1615226174296,ids=SaccAccount:1122,isShard=true,wt=javabin,debugQuery=true}},response={numFound=1,start=0,docs=[SolrDocument{total_balance_c=255.0,USD,
total_balance_cl_ns=25500,
total_balance_cs_ns=USD}]},debug={rawquerystring=id:
\"SaccAccount:1122\",querystring=id:
\"SaccAccount:1122\",parsedquery=+id:SaccAccount:1122,parsedquery_toString=+id:SaccAccount:1122,explain={SaccAccount:1122=\n10.844471
= weight(id:SaccAccount:1122 in 4833) [SchemaSimilarit

Re: Increase in response time in case of collapse queries.

[Joel Bernstein]

Collapse is designed to outperform grouping in the following scenario:

There is high cardinality on the group field and group.ngroups is needed.
If either of these conditions is not satisfied grouping will typically be
faster.

You will need to provide some more information about your setup to get an
answer to the collapse performance question.

1) What are the response times for both methods. Saying one is faster is
not specific enough.
2) What is the cardinality of the collapse field, saying it's high is not
specific enough. What is the actual cardinality?
3) Is ngroups used in the grouping query.

















Joel Bernstein
http://joelsolr.blogspot.com/


On Mon, Mar 8, 2021 at 11:30 AM Gajendra Dadheech 
wrote:

> @prashant Florin means to put null for parentglusrid in documents where
> this field-value is only present in one document [Group has only one
> document]. and then use nullPolicy to include/expand.
>
>
>
> On Mon, Mar 8, 2021 at 6:55 PM Parshant Kumar
>  wrote:
>
> > client should set to null the field if it's unique.
> >
> > @florin @Gajendra can you please explain more .I am not clear how to
> > perform this.
> >
> > On Mon, Mar 8, 2021 at 6:09 PM Florin Babes 
> wrote:
> >
> > > @Gajendra Our response time dropped by 36% and our rps increased with
> > 27%.
> > >
> > > You have to reindex the core and the client should set to null the
> field
> > if
> > > it's unique.
> > >
> > > În lun., 8 mar. 2021 la 13:18, Parshant Kumar
> > >  a scris:
> > >
> > > > How can we make group_field null? Using nullPolicy=expand ?
> > > >
> > > > On Mon, Mar 8, 2021 at 4:41 PM Florin Babes 
> > > wrote:
> > > >
> > > > > We improved the performance of collapse by making the group_field
> > null
> > > > for
> > > > > the documents that have an unique value for group_field. This might
> > > help/
> > > > >
> > > > >
> > > > > În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> > > > >  a scris:
> > > > >
> > > > > > yes,group_field is having high cardinality.
> > > > > >
> > > > > >
> > > > > > Thanks
> > > > > > Parshant Kumar
> > > > > >
> > > > > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <
> babesflo...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Your group_field has a high cardinality?
> > > > > > > Thanks,
> > > > > > > Florin Babes
> > > > > > >
> > > > > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > > > > > >  a scris:
> > > > > > >
> > > > > > > > Hi florin,
> > > > > > > >
> > > > > > > > I am using below.
> > > > > > > >
> > > > > > > > 1) fq={!collapse field=parentglusrid}
> > > > > > > > 2) expand.rows=4
> > > > > > > > 3) expand=true
> > > > > > > >
> > > > > > > > Size of index is around 100GB.
> > > > > > > > Solr version is 6.5
> > > > > > > >
> > > > > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <
> > > babesflo...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > > First let's call the field you collapse on group_field
> > > > > > > > > If group_field has a high cardinality you should make
> > > group_field
> > > > > > null
> > > > > > > > for
> > > > > > > > > those documents that have a unique group_field and set
> > > > > > > nullPolicy=expand.
> > > > > > > > > By doing that solr will use less memory for it's internal
> > maps
> > > > (so
> > > > > > > faster
> > > > > > > > > gc) and the head selecting will be faster.
> > > > > > > > > What is your head selecting strategy? Can you share your fq
> > > which
> > > > > you
> > > > > > > use
> > > > > > > > > for collapsing?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Florin Babes
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > > > > > > >  a scris:
> > > > > > > > >
> > > > > > > > > > anyone please help
> > > > > > > > > >
> > > > > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > > > > > > > parshant.ku...@indiamart.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > We have implemented collapse queries in place of
> grouped
> > > > > queries
> > > > > > on
> > > > > > > > our
> > > > > > > > > > > production solr. As mentioned in solr documentation
> > > collapse
> > > > > > > queries
> > > > > > > > > are
> > > > > > > > > > > recommended in place of grouped queries in terms of
> > > > > performance .
> > > > > > > But
> > > > > > > > > > after
> > > > > > > > > > > switching to collapsed queries from grouped queries
> > > response
> > > > > time
> > > > > > > of
> > > > > > > > > > > queries have increased. This is unexpected behaviour,
> the
> > > > > > response
> > > > > > > > time
> > > > > > > > > > > should have been improved but results are opposites.
> > > > > > > > > > > Please someone help why response time is increased for
> > > > > collapsed
> > > > > > > > > queries.
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > > Parshant Kumar
> > > >

Re: Idle timeout expired and Early Client Disconnect errors

[Joel Bernstein]

This ticket shows how it is done in the solrconfig.xml:

https://issues.apache.org/jira/browse/SOLR-9103



Joel Bernstein
http://joelsolr.blogspot.com/


On Mon, Mar 8, 2021 at 9:18 AM ufuk yılmaz 
wrote:

> How do you “register” something like a CloudSolrStream btw? Using Blob
> Store API?
>
> Sent from Mail for Windows 10
>
> From: Susmit
> Sent: 06 March 2021 23:03
> To: users@solr.apache.org
> Subject: Re: Idle timeout expired and Early Client Disconnect errors
>
> better to use solr 8.9 and configure http timeouts from solr.in.sh
> workaround is bigger - need to extend cloudsolrstream , register it and
> install custom solrclientcache with overridden setcontext method
>
> Sent from my iPhone
>
> > On Mar 6, 2021, at 9:25 AM, ufuk yılmaz 
> wrote:
> >
> > How? O_O
> >
> > Sent from Mail for Windows 10
> >
> > From: Susmit
> > Sent: 06 March 2021 18:35
> > To: solr-u...@lucene.apache.org
> > Subject: Re: Idle timeout expired and Early Client Disconnect errors
> >
> > i have used a workaround to increase the default (hard coded) timeout of
> 2 min in solrclientcache.
> > i can run 9+ hour long streaming queries with no issues.
> >
> > Sent from my iPhone
> >
> >> On Mar 2, 2021, at 5:32 PM, ufuk yılmaz 
> wrote:
> >>
> >> I divided the query to 1000 pieces and removed the parallel stream
> clause, it seems to be working without timeout so far, if it does I just
> can divide it to even smaller pieces I guess.
> >>
> >> I tried to send all 1000 pieces in a “list” expression to be executed
> linearly, it didn’t work but I was just curious if it could handle such a
> large query 😃
> >>
> >> Now I’m just generating expression strings from java code and sending
> them one by one. I tried to use SolrJ for this, but encountered a weird
> problem where even the simplest expression (echo) stops working after a few
> iterations in a loop. I’m guessing the underlying HttpClient is not closing
> connections timely, hitting the OS per-host connection limit. I asked a
> separate question about this. I was following the example on lucidworks:
> https://lucidworks.com/post/streaming-expressions-in-solrj/
> >>
> >> I just modified my code to use regular REST calls using okhttp3, it’s a
> shame that I couldn’t use SolrJ since it truly streams every result 1 by 1
> continuously. REST just returns a single large response at the very end of
> the stream.
> >>
> >> Thanks again for your help.
> >>
> >> Sent from Mail for Windows 10
> >>
> >> From: Joel Bernstein
> >> Sent: 02 March 2021 00:19
> >> To: solr-u...@lucene.apache.org
> >> Subject: Re: Idle timeout expired and Early Client Disconnect errors
> >>
> >> Also the parallel function builds hash partitioning filters that could
> lead
> >> to timeouts if they take too long to build. Try the query without the
> >> parallel function if you're still getting timeouts when making the query
> >> smaller.
> >>
> >>
> >>
> >> Joel Bernstein
> >> http://joelsolr.blogspot.com/
> >>
> >>
>  On Mon, Mar 1, 2021 at 4:03 PM Joel Bernstein 
> wrote:
> >>>
> >>> The settings in your version are 30 seconds and 15 seconds for socket
> and
> >>> connection timeouts.
> >>>
> >>> Typically timeouts occur because one or more shards in the query are
> idle
> >>> beyond the timeout threshold. This happens because lot's of data is
> being
> >>> read from other shards.
> >>>
> >>> Breaking the query into small parts would be a good strategy.
> >>>
> >>>
> >>>
> >>>
> >>> Joel Bernstein
> >>> http://joelsolr.blogspot.com/
> >>>
> >>>
> >>> On Mon, Mar 1, 2021 at 3:30 PM ufuk yılmaz  >
> >>> wrote:
> >>>
>  Hello Mr. Bernstein,
> 
>  I’m using version 8.4. So, if I understand correctly, I can’t increase
>  timeouts and they are bound to happen in such a large stream. Should
> I just
>  reduce the output of my search expressions?
> 
>  Maybe I can split my search results into ~100 parts and run the same
>  query 100 times in series. Each part would emit ~3M documents so they
>  should finish before timeout?
> 
>  Is this a reasonable solution?
> 
>  Btw how long is the default hard-coded timeout value? Because
> yesterday I
>  ran another query which took more than 1 hour without any timeouts and
>  finished successfully.
> 
>  Sent from Mail for Windows 10
> 
>  From: Joel Bernstein
>  Sent: 01 March 2021 23:03
>  To: solr-u...@lucene.apache.org
>  Subject: Re: Idle timeout expired and Early Client Disconnect errors
> 
>  Oh wait, I misread your email. The idle timeout issue is configurable
> in:
> 
>  https://issues.apache.org/jira/browse/SOLR-14672
> 
>  This unfortunately missed the 8.8 release and will be 8.9.
> 
> 
> 
>  This i
> 
> 
> 
>  Joel Bernstein
>  http://joelsolr.blogspot.com/
> 
> 
> > On Mon, Mar 1, 2021 at 2:56 PM Joel Bernstein 
> wrote:
> 
> > What version are you using?
> >
> > Solr 8.7 has changes th

Fwd: Call for Presentations for ApacheCon 2021 now open

[Anshum Gupta]

FYI

-- Forwarded message -
From: Rich Bowen 
Date: Mon, Mar 8, 2021 at 12:32 PM
Subject: Call for Presentations for ApacheCon 2021 now open
To: 


The ApacheCon Planners and the Apache Software Foundation are pleased to
announce that ApacheCon@Home will be held online, September 21-23, 2021.
Once again, we’ll be featuring content from dozens of our projects, as well
as content about our community, how Apache works, business models around
Apache software, the legal aspects of open source, and many other topics.

Last year’s virtual ApacheCon@Home event was a big success, with 5,745
registrants from more than 150 countries, spanning every time zone, with
the virtual format delivering content to attendees who would never have
attended an in-person ApacheCon (83% of post-event poll responders in 2020
indicated this was their first ApacheCon ever)!

Given the great participation and excitement for last year’s event, we are
announcing the Call for Presentations is now open to presenters from around
the world until May 1st. Talks can be focused on the topics above, as well
as any of our amazing projects. Submit your talks today!

https://www.apachecon.com/acah2021/cfp.html

We look forward to reviewing your contribution to one of the most popular
open source software events in the world!

-- 
Rich Bowen, VP Conferences
The Apache Software Foundation
https://apachecon.com/
@apachecon

-
To unsubscribe, e-mail: announce-unsubscr...@apachecon.com
For additional commands, e-mail: announce-h...@apachecon.com



-- 
Anshum Gupta

Call for Presentations for ApacheCon 2021 now open

[Rich Bowen]

[Note: You are receiving this because you are subscribed to a users@ 
list on one or more Apache Software Foundation projects.]


The ApacheCon Planners and the Apache Software Foundation are pleased to 
announce that ApacheCon@Home will be held online, September 21-23, 2021. 
Once again, we’ll be featuring content from dozens of our projects, as 
well as content about our community, how Apache works, business models 
around Apache software, the legal aspects of open source, and many other 
topics.


Last year’s virtual ApacheCon@Home event was a big success, with 5,745 
registrants from more than 150 countries, spanning every time zone, with 
the virtual format delivering content to attendees who would never have 
attended an in-person ApacheCon (83% of post-event poll responders in 
2020 indicated this was their first ApacheCon ever)!


Given the great participation and excitement for last year’s event, we 
are announcing the Call for Presentations is now open to presenters from 
around the world until May 1st. Talks can be focused on the topics 
above, as well as any of our amazing projects. Submit your talks today!


https://www.apachecon.com/acah2021/cfp.html

We look forward to reviewing your contribution to one of the most 
popular open source software events in the world!



Rich, for the ApacheCon Planners

--
Rich Bowen, VP Conferences
The Apache Software Foundation
https://apachecon.com/
@apachecon

Please unsubscribe me

[Robin Wei]

Please unsubscribe me
Thanks

OutOfMemoryError: when deleting/updating the data

[Hitesh Khamesra]

We are getting OutOfMemoryError, while concurrently deleting and updating
the data. Looks like we don't have a thread pool to merge segment/index. Is
it possible to control this behavior?

Thanks.
Hitesh
===
"o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Server
error writing document id 1sadfasfzwg0!1tskuj5zwg08sasfasdsg0uqp6o_2 to the
index
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:244)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:76)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:259)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:489)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$versionAdd$0(DistributedUpdateProcessor.java:339)
at org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:339)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:225)
at
org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:245)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at
mn.fs.solr.AnitaUpdateProcessorFactory$UpdateProcessor.processAdd(AnitaUpdateProcessorFactory.java:42)
at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:154)
at
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:121)
at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:84)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2602)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:800)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:579)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:505)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at
org.eclipse.

Re: Caffeine Cache Metrics Broken?

[Stephen Lewis Bianamara]

Okay, I have a hypothesis after some testing. It seems like what might be
happening is that the cumulative stats are computed per process and per
cache type.

So if you change cache types from a cache you've had for a long time to a
new one, the cumulative numbers may seem oddly low. In that case it would
be best to perform a reset of the process in your solr cluster for accurate
metrics.

Does that seem right to you?

Thanks,
Stephen

On Fri, Mar 5, 2021, 10:32 AM Stephen Lewis Bianamara <
stephen.bianam...@gmail.com> wrote:

> Thanks Shawn. Something seems different between the two because Caffeine
> Cache is having much higher volume per hour than our previous
> implementation was. So I guess it is then more likely that it is something
> actually expected due to a change in what is getting kept/warmed, so I'll
> look into this more and get back to you if that doesn't end up making sense
> based on what I observe.
>
> Thanks again,
> Stephen
>
> On Tue, Mar 2, 2021 at 6:35 PM Shawn Heisey  wrote:
>
>> On 3/2/2021 3:47 PM, Stephen Lewis Bianamara wrote:
>> > I'm investigating a weird behavior I've observed in the admin page for
>> > caffeine cache metrics. It looks to me like on the older caches, warm-up
>> > queries were not counted toward hit/miss ratios, which of course makes
>> > sense, but on Caffeine cache it looks like they are. I'm using solr 8.3.
>> >
>> > Obviously this makes measuring its true impact a little tough. Is this
>> by
>> > any chance a known issue and already fixed in later versions?
>>
>> The earlier cache implementations are entirely native to Solr -- all the
>> source code is include in the Solr codebase.
>>
>> Caffeine is a third-party cache implementation that has been integrated
>> into Solr.  Some of the metrics might come directly from Caffeine, not
>> Solr code.
>>
>> I would expect warming queries to be counted on any of the cache
>> implementations.  One of the reasons that the warming capability exists
>> is to pre-populate the caches before actual queries begin.  If warming
>> queries are somehow excluded, then the cache metrics would not be correct.
>>
>> I looked into the code and did not find anything that would keep warming
>> queries from affecting stats.  But it is always possible that I just
>> didn't know what to look for.
>>
>> In the master branch (Solr 9.0), CaffeineCache is currently the only
>> implementation available.
>>
>> Thanks,
>> Shawn
>>
>

Re: OutOfMemoryError: when deleting/updating the data

[Shawn Heisey]

On 3/8/2021 8:18 PM, Hitesh Khamesra wrote:

We are getting OutOfMemoryError, while concurrently deleting and updating
the data. Looks like we don't have a thread pool to merge segment/index. Is
it possible to control this behavior?


Here's the important part from that massive exception stacktrace:

Caused by: java.lang.OutOfMemoryError: unable to create native thread:

There are exactly two ways you can fix OOME.  One is to increase the 
resource that has been depleted, the other is to change things so that 
fewer resources are required.


In this case you're running into an OS limit on the number of processes 
that a user is allowed to start.  The default setting for this limit on 
all Linux distros I have seen is 1024.  It's pretty easy to have a Solr 
instance with significantly more than a thousand threads.  On most 
operating systems, threads are handled by the kernel as processes.  Some 
have separate entities for threads, but that's not very common.


So what you're going to have to do is increase the number of processes 
that the user which is running Solr is allowed to have.


On most Linux distros this is usually done with the following file:

/etc/security/limits.conf

If you're running with a different OS, you'll need to research how to 
increase the limit.


Thanks,
Shawn

Re: Caffeine Cache Metrics Broken?

[Shawn Heisey]

On 3/8/2021 11:06 PM, Stephen Lewis Bianamara wrote:

So if you change cache types from a cache you've had for a long time to a
new one, the cumulative numbers may seem oddly low. In that case it would
be best to perform a reset of the process in your solr cluster for accurate
metrics.


If you change cache types, you have to restart Solr or reload the core 
in order to actually make the change active.  Restart or reload will 
reset all counters to zero.


Thanks,
Shawn

zk upconfig does not recognize ZK_HOST style url

[Subhajit Das]

Hi There,

When I was trying to upload new configset to zookeeper using Solr control 
script, the -z parameter is not recognizing ZK_HOST style string.

Say, I use ,,/solr, then the config is uploaded to  
directly, instead of /solr znode.

Can anyone please help on this matter, if this is how it is supposed to work.

Note: /solr,/solr,/solr seems o work.