So here are the response times:

Response time for the grouped method is 167 ms for 0.65 million requests.
Response time for the collapsed method is 177 ms for 0.65 million requests.

If group.ngroups is used with 6+ million cardinality, then the result set
size before grouping must have been small. With a large result set grouping
would have been quite slow in this scenario.

With small result sets, grouping is just fine also.




Joel Bernstein
http://joelsolr.blogspot.com/


On Thu, Mar 11, 2021 at 1:40 PM Gajendra Dadheech <gajju3...@gmail.com>
wrote:

> @florin
>
> Great advice. Null key for unique documents is really helpful. Any other
> such tricks that you are using to improve collapse performance ?
>
> On Tue, Mar 9, 2021, 2:45 PM Parshant Kumar
> <parshant.ku...@indiamart.com.invalid> wrote:
>
> > Hi Joel,
> >
> > 1) What are the response times for both methods. Saying one is faster is
> > not specific enough.
> >
> > Response time for the grouped method is 167 ms for 0.65 million requests.
> > Response time for the collapsed method is 177 ms for 0.65 million
> requests.
> >
> > 2) What is the cardinality of the collapse field, saying it's high is not
> > specific enough. What is the actual cardinality?
> >
> > Cardinality of the collapse field is around 6.2 Million
> >
> > [image: image.png]
> > 3) Is ngroups used in the grouping query
> >
> > Yes, ngroups is used in grouping query.
> >
> > Thanks
> > Parshant Kumar
> >
> >
> >
> >
> > On Tue, Mar 9, 2021 at 12:30 AM Joel Bernstein <joels...@gmail.com>
> wrote:
> >
> >> Collapse is designed to outperform grouping in the following scenario:
> >>
> >> There is high cardinality on the group field and group.ngroups is
> needed.
> >> If either of these conditions is not satisfied grouping will typically
> be
> >> faster.
> >>
> >> You will need to provide some more information about your setup to get
> an
> >> answer to the collapse performance question.
> >>
> >> 1) What are the response times for both methods. Saying one is faster is
> >> not specific enough.
> >> 2) What is the cardinality of the collapse field, saying it's high is
> not
> >> specific enough. What is the actual cardinality?
> >> 3) Is ngroups used in the grouping query.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> Joel Bernstein
> >> http://joelsolr.blogspot.com/
> >>
> >>
> >> On Mon, Mar 8, 2021 at 11:30 AM Gajendra Dadheech <gajju3...@gmail.com>
> >> wrote:
> >>
> >> > @prashant Florin means to put null for parentglusrid in documents
> where
> >> > this field-value is only present in one document [Group has only one
> >> > document]. and then use nullPolicy to include/expand.
> >> >
> >> >
> >> >
> >> > On Mon, Mar 8, 2021 at 6:55 PM Parshant Kumar
> >> > <parshant.ku...@indiamart.com.invalid> wrote:
> >> >
> >> > > client should set to null the field if it's unique.
> >> > >
> >> > > @florin @Gajendra can you please explain more .I am not clear how to
> >> > > perform this.
> >> > >
> >> > > On Mon, Mar 8, 2021 at 6:09 PM Florin Babes <babesflo...@gmail.com>
> >> > wrote:
> >> > >
> >> > > > @Gajendra Our response time dropped by 36% and our rps increased
> >> with
> >> > > 27%.
> >> > > >
> >> > > > You have to reindex the core and the client should set to null the
> >> > field
> >> > > if
> >> > > > it's unique.
> >> > > >
> >> > > > În lun., 8 mar. 2021 la 13:18, Parshant Kumar
> >> > > > <parshant.ku...@indiamart.com.invalid> a scris:
> >> > > >
> >> > > > > How can we make group_field null? Using nullPolicy=expand ?
> >> > > > >
> >> > > > > On Mon, Mar 8, 2021 at 4:41 PM Florin Babes <
> >> babesflo...@gmail.com>
> >> > > > wrote:
> >> > > > >
> >> > > > > > We improved the performance of collapse by making the
> >> group_field
> >> > > null
> >> > > > > for
> >> > > > > > the documents that have an unique value for group_field. This
> >> might
> >> > > > help/
> >> > > > > >
> >> > > > > >
> >> > > > > > În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> >> > > > > > <parshant.ku...@indiamart.com.invalid> a scris:
> >> > > > > >
> >> > > > > > > yes,group_field is having high cardinality.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Thanks
> >> > > > > > > Parshant Kumar
> >> > > > > > >
> >> > > > > > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <
> >> > babesflo...@gmail.com
> >> > > >
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Your group_field has a high cardinality?
> >> > > > > > > > Thanks,
> >> > > > > > > > Florin Babes
> >> > > > > > > >
> >> > > > > > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> >> > > > > > > > <parshant.ku...@indiamart.com.invalid> a scris:
> >> > > > > > > >
> >> > > > > > > > > Hi florin,
> >> > > > > > > > >
> >> > > > > > > > > I am using below.
> >> > > > > > > > >
> >> > > > > > > > > 1) fq={!collapse field=parentglusrid}
> >> > > > > > > > > 2) expand.rows=4
> >> > > > > > > > > 3) expand=true
> >> > > > > > > > >
> >> > > > > > > > > Size of index is around 100GB.
> >> > > > > > > > > Solr version is 6.5
> >> > > > > > > > >
> >> > > > > > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <
> >> > > > babesflo...@gmail.com
> >> > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hello,
> >> > > > > > > > > > First let's call the field you collapse on group_field
> >> > > > > > > > > > If group_field has a high cardinality you should make
> >> > > > group_field
> >> > > > > > > null
> >> > > > > > > > > for
> >> > > > > > > > > > those documents that have a unique group_field and set
> >> > > > > > > > nullPolicy=expand.
> >> > > > > > > > > > By doing that solr will use less memory for it's
> >> internal
> >> > > maps
> >> > > > > (so
> >> > > > > > > > faster
> >> > > > > > > > > > gc) and the head selecting will be faster.
> >> > > > > > > > > > What is your head selecting strategy? Can you share
> >> your fq
> >> > > > which
> >> > > > > > you
> >> > > > > > > > use
> >> > > > > > > > > > for collapsing?
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > > Florin Babes
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> >> > > > > > > > > > <parshant.ku...@indiamart.com.invalid> a scris:
> >> > > > > > > > > >
> >> > > > > > > > > > > anyone please help
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> >> > > > > > > > > > > parshant.ku...@indiamart.com>
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Hi all,
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > We have implemented collapse queries in place of
> >> > grouped
> >> > > > > > queries
> >> > > > > > > on
> >> > > > > > > > > our
> >> > > > > > > > > > > > production solr. As mentioned in solr
> documentation
> >> > > > collapse
> >> > > > > > > > queries
> >> > > > > > > > > > are
> >> > > > > > > > > > > > recommended in place of grouped queries in terms
> of
> >> > > > > > performance .
> >> > > > > > > > But
> >> > > > > > > > > > > after
> >> > > > > > > > > > > > switching to collapsed queries from grouped
> queries
> >> > > > response
> >> > > > > > time
> >> > > > > > > > of
> >> > > > > > > > > > > > queries have increased. This is unexpected
> >> behaviour,
> >> > the
> >> > > > > > > response
> >> > > > > > > > > time
> >> > > > > > > > > > > > should have been improved but results are
> opposites.
> >> > > > > > > > > > > > Please someone help why response time is increased
> >> for
> >> > > > > > collapsed
> >> > > > > > > > > > queries.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Thanks
> >> > > > > > > > > > > > Parshant Kumar
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > --
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > --
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > > > --
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> > > --
> >> > >
> >> > >
> >> >
> >>
> >
> >
>

Reply via email to