That is an interesting finding. The default behavior is quite fast compared to the sort approach.
Joel Bernstein http://joelsolr.blogspot.com/ On Wed, Mar 17, 2021 at 3:42 AM Florin Babes <babesflo...@gmail.com> wrote: > @Gajendra we used a sort="score desc" even if we only use the head of the > group. By removing this sort and leaving the default behavior (selecting > the head with the highest score) our response time dropped with 42% in our > synthetic load tests. > > > În joi, 11 mar. 2021 la 20:40 Gajendra Dadheech <gajju3...@gmail.com> a > scris: > > > @florin > > > > Great advice. Null key for unique documents is really helpful. Any other > > such tricks that you are using to improve collapse performance ? > > > > On Tue, Mar 9, 2021, 2:45 PM Parshant Kumar > > <parshant.ku...@indiamart.com.invalid> wrote: > > > > > Hi Joel, > > > > > > 1) What are the response times for both methods. Saying one is faster > is > > > not specific enough. > > > > > > Response time for the grouped method is 167 ms for 0.65 million > requests. > > > Response time for the collapsed method is 177 ms for 0.65 million > > requests. > > > > > > 2) What is the cardinality of the collapse field, saying it's high is > not > > > specific enough. What is the actual cardinality? > > > > > > Cardinality of the collapse field is around 6.2 Million > > > > > > [image: image.png] > > > 3) Is ngroups used in the grouping query > > > > > > Yes, ngroups is used in grouping query. > > > > > > Thanks > > > Parshant Kumar > > > > > > > > > > > > > > > On Tue, Mar 9, 2021 at 12:30 AM Joel Bernstein <joels...@gmail.com> > > wrote: > > > > > >> Collapse is designed to outperform grouping in the following scenario: > > >> > > >> There is high cardinality on the group field and group.ngroups is > > needed. > > >> If either of these conditions is not satisfied grouping will typically > > be > > >> faster. > > >> > > >> You will need to provide some more information about your setup to get > > an > > >> answer to the collapse performance question. > > >> > > >> 1) What are the response times for both methods. Saying one is faster > is > > >> not specific enough. > > >> 2) What is the cardinality of the collapse field, saying it's high is > > not > > >> specific enough. What is the actual cardinality? > > >> 3) Is ngroups used in the grouping query. > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> Joel Bernstein > > >> http://joelsolr.blogspot.com/ > > >> > > >> > > >> On Mon, Mar 8, 2021 at 11:30 AM Gajendra Dadheech < > gajju3...@gmail.com> > > >> wrote: > > >> > > >> > @prashant Florin means to put null for parentglusrid in documents > > where > > >> > this field-value is only present in one document [Group has only one > > >> > document]. and then use nullPolicy to include/expand. > > >> > > > >> > > > >> > > > >> > On Mon, Mar 8, 2021 at 6:55 PM Parshant Kumar > > >> > <parshant.ku...@indiamart.com.invalid> wrote: > > >> > > > >> > > client should set to null the field if it's unique. > > >> > > > > >> > > @florin @Gajendra can you please explain more .I am not clear how > to > > >> > > perform this. > > >> > > > > >> > > On Mon, Mar 8, 2021 at 6:09 PM Florin Babes < > babesflo...@gmail.com> > > >> > wrote: > > >> > > > > >> > > > @Gajendra Our response time dropped by 36% and our rps increased > > >> with > > >> > > 27%. > > >> > > > > > >> > > > You have to reindex the core and the client should set to null > the > > >> > field > > >> > > if > > >> > > > it's unique. > > >> > > > > > >> > > > În lun., 8 mar. 2021 la 13:18, Parshant Kumar > > >> > > > <parshant.ku...@indiamart.com.invalid> a scris: > > >> > > > > > >> > > > > How can we make group_field null? Using nullPolicy=expand ? > > >> > > > > > > >> > > > > On Mon, Mar 8, 2021 at 4:41 PM Florin Babes < > > >> babesflo...@gmail.com> > > >> > > > wrote: > > >> > > > > > > >> > > > > > We improved the performance of collapse by making the > > >> group_field > > >> > > null > > >> > > > > for > > >> > > > > > the documents that have an unique value for group_field. > This > > >> might > > >> > > > help/ > > >> > > > > > > > >> > > > > > > > >> > > > > > În lun., 8 mar. 2021 la 12:40, Parshant Kumar > > >> > > > > > <parshant.ku...@indiamart.com.invalid> a scris: > > >> > > > > > > > >> > > > > > > yes,group_field is having high cardinality. > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > Thanks > > >> > > > > > > Parshant Kumar > > >> > > > > > > > > >> > > > > > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes < > > >> > babesflo...@gmail.com > > >> > > > > > >> > > > > > wrote: > > >> > > > > > > > > >> > > > > > > > Your group_field has a high cardinality? > > >> > > > > > > > Thanks, > > >> > > > > > > > Florin Babes > > >> > > > > > > > > > >> > > > > > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar > > >> > > > > > > > <parshant.ku...@indiamart.com.invalid> a scris: > > >> > > > > > > > > > >> > > > > > > > > Hi florin, > > >> > > > > > > > > > > >> > > > > > > > > I am using below. > > >> > > > > > > > > > > >> > > > > > > > > 1) fq={!collapse field=parentglusrid} > > >> > > > > > > > > 2) expand.rows=4 > > >> > > > > > > > > 3) expand=true > > >> > > > > > > > > > > >> > > > > > > > > Size of index is around 100GB. > > >> > > > > > > > > Solr version is 6.5 > > >> > > > > > > > > > > >> > > > > > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes < > > >> > > > babesflo...@gmail.com > > >> > > > > > > > >> > > > > > > > wrote: > > >> > > > > > > > > > > >> > > > > > > > > > Hello, > > >> > > > > > > > > > First let's call the field you collapse on > group_field > > >> > > > > > > > > > If group_field has a high cardinality you should > make > > >> > > > group_field > > >> > > > > > > null > > >> > > > > > > > > for > > >> > > > > > > > > > those documents that have a unique group_field and > set > > >> > > > > > > > nullPolicy=expand. > > >> > > > > > > > > > By doing that solr will use less memory for it's > > >> internal > > >> > > maps > > >> > > > > (so > > >> > > > > > > > faster > > >> > > > > > > > > > gc) and the head selecting will be faster. > > >> > > > > > > > > > What is your head selecting strategy? Can you share > > >> your fq > > >> > > > which > > >> > > > > > you > > >> > > > > > > > use > > >> > > > > > > > > > for collapsing? > > >> > > > > > > > > > > > >> > > > > > > > > > Thanks, > > >> > > > > > > > > > Florin Babes > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar > > >> > > > > > > > > > <parshant.ku...@indiamart.com.invalid> a scris: > > >> > > > > > > > > > > > >> > > > > > > > > > > anyone please help > > >> > > > > > > > > > > > > >> > > > > > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar < > > >> > > > > > > > > > > parshant.ku...@indiamart.com> > > >> > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > >> > > > > > > > > > > > Hi all, > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > We have implemented collapse queries in place of > > >> > grouped > > >> > > > > > queries > > >> > > > > > > on > > >> > > > > > > > > our > > >> > > > > > > > > > > > production solr. As mentioned in solr > > documentation > > >> > > > collapse > > >> > > > > > > > queries > > >> > > > > > > > > > are > > >> > > > > > > > > > > > recommended in place of grouped queries in terms > > of > > >> > > > > > performance . > > >> > > > > > > > But > > >> > > > > > > > > > > after > > >> > > > > > > > > > > > switching to collapsed queries from grouped > > queries > > >> > > > response > > >> > > > > > time > > >> > > > > > > > of > > >> > > > > > > > > > > > queries have increased. This is unexpected > > >> behaviour, > > >> > the > > >> > > > > > > response > > >> > > > > > > > > time > > >> > > > > > > > > > > > should have been improved but results are > > opposites. > > >> > > > > > > > > > > > Please someone help why response time is > increased > > >> for > > >> > > > > > collapsed > > >> > > > > > > > > > queries. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Thanks > > >> > > > > > > > > > > > Parshant Kumar > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > -- > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > -- > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > -- > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > -- > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > -- > > >> > > > > >> > > > > >> > > > >> > > > > > > > > >
