Hi,
*Please find details below:*In our index, we have data of suppliers along
with their products which we display on front-end, wrt search requests.
*Example: For a supplier with id: 678, we have 2 products in our index*
*product-id(unique)*
*document1:*
{
product-id: 123
product-price: 2000rs
product-name: Jute bags
*supplier-id: 678company-name: BagFactoryLimited*
}
*document2:*
{
product-id: 863
product-price: 4500rs
product-name: trolley bags
*supplier-id: 678company-name: BagFactoryLimited*
}
As you can see from above, each document in our index contains
product details i.e product-id, product-price, product-name
and also supplier details i.e supplier-id, company-name
*Problem1: (while indexing)*
Here, whenever there is a change in supplier specific details/field, we are
re-indexing all the products of the supplier although the supplier data
will be the same in all of his products.
*FYI*
We re-index ~5Cr documents per day
*We would like to know, if there is any better way to optimize this which
helps to avoid indexing of redundant data*
*Problem2: (while querying)*
Now, when the data in our current index is queried, we display the single
most relevant product of a supplier. [even if the query matches 1 or more
documents in our index]
For this we are using a collapse query on supplier-id field (as we dont
know relationship between documents) [which is resource intensive]
*Ex: *
fq={!collapse field=supplier-id}
*FYI*
We serve ~25 Lakh Queries per day
*We would like to know if there is any better way to organize index, so
that we can avoid such resource intensive queries, thereby optimizing
search response*
*Our Solr Infra Stats: FYI*
*Version:* v9.6.1
*No. of nodes:* 8
*No. of shards:* 62
*Heap per node: *12G
*RAM per node: *50G
*No. of cpu cores per node: *16
*Count of docs:* ~20Cr
*Size of Index: *~250G
*Routing used:* implicit
Please let us know, if you need any other details
*Thanks & Regards,*
*Uday Kumar*
On Wed, Mar 5, 2025 at 12:08 PM Alessandro Benedetti <a.benede...@sease.io>
wrote:
> Hi Uday,
> Your email is a perfect example of
> https://en.m.wikipedia.org/wiki/XY_problem.
>
> Both for indexing and query time you need to explain your problems and use
> cases rather than your attempted solutions.
>
>
> Then we'll be able to give some recommendations.
>
>
> On Wed, 5 Mar 2025, 06:39 Uday Kumar, <uday.p...@indiamart.com.invalid>
> wrote:
>
> > Hi,
> > I would like to give some extra context here, so that it would help in
> > getting better suggestions
> >
> >
> > *Our goal:To improve our search system either by optimizing indexing or
> by
> > improving solr response times*
> >
> > *Current approach while indexing at our end:*
> > Even with change in a single field of document, we send the entire
> document
> > for indexing. (~2cr docs are being reindexed on a daily basis)
> > Solr version: V9.6.1
> >
> > *To Optimize Indexing:*
> > 1. POC on external file field: [which stores frequently changed fields in
> > external file and loads after each commit, instead of indexing into solr
> > for each change]
> >
> >
> https://solr.apache.org/guide/solr/latest/indexing-guide/external-files-processes.html
> > Observation:
> > a. works only with numeric fields
> > b. Also the community suggested not to go with this, as its old feature.
> > so, I dropped this.
> >
> > 2. POC on Inplace update: (Which helps in indexing fields which contains
> > changes, but not entire document)
> >
> >
> https://solr.apache.org/guide/solr/latest/indexing-guide/partial-document-updates.html#in-place-updates
> > Observation:
> > a. Works with only single values fields
> > b. Looks promising wrt indexing optimization but not suitable wrt our
> > schema (as we have more multivalued fields). so, dropped
> >
> >
> > Then we moved for alternatives which is expected to help in optimizing
> > response times
> >
> > *To improve Solr Response time:*Nested Documents POC:
> >
> >
> https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-nested-documents.html
> > *wrt this statement:*
> > "In terms of performance,* indexing the relationships between documents
> > usually yields much faster queries* than an equivalent "query time join",
> > since the relationships are already stored in the index and do not need
> to
> > be computed"
> >
> > But here we found, complete block will be reindexed even with change in
> > single child document
> > So, we would like to know more about this feature,
> > 1. If this complete block reindexing is heavy when compared with
> > traditional indexing? [As we have more documents for reindexing per
> single
> > day i.e ~2cr]
> > 2. What we can expect with this nested document feature in terms of
> > performance (wrt tradeoff in indexing/querying)
> > 3. If not, do we have any other alternative which we can work upon
> >
> > *Thanks & Regards,*
> > *Uday Kumar*
> >
> >
> > On Mon, Mar 3, 2025 at 7:17 PM Uday Kumar <uday.p...@indiamart.com>
> wrote:
> >
> > > Also in place updates happen on very specific conditions, have you
> > checked
> > > you satisfy them before even attempting to see some sort of impact on
> > your
> > > use case?
> > > Yes we considered those specifications, here, we didnt mean to say
> > > it's not impactful in itself. but with our project & schema
> > >
> > > *Thanks & Regards,*
> > > *Uday Kumar*
> > > *Product Search Tech*
> > >
> > >
> > > On Fri, Feb 28, 2025 at 6:06 PM Alessandro Benedetti <
> > > benedetti.ale...@gmail.com> wrote:
> > >
> > >> What is your problem? Rather than asking about a solution you
> attempted
> > is
> > >> usually better to start from the problem.
> > >>
> > >> You talk about grouping, have you considered field collapsing?
> > >>
> > >> According to my experience going with nested documents rarely justify
> > the
> > >> performance and functional overhead both at indexing and query time.
> > >>
> > >> But sometimes you need them.
> > >>
> > >> Also in place updates happen on very specific conditions, have you
> > checked
> > >> you satisfy them before even attempting to see some sort of impact on
> > your
> > >> use case?
> > >>
> > >> Cheers
> > >>
> > >> On Fri, 28 Feb 2025, 08:30 Uday Kumar, <uday.p...@indiamart.com
> > .invalid>
> > >> wrote:
> > >>
> > >> > Does this mean it will not be impactful in performance to use Nested
> > >> > Indexing in production with such an indexing rate?
> > >> >
> > >> > We have tried POC on inplace updates and found its not impactful
> > either
> > >> wrt
> > >> > our project, so we would not be using this in combination too
> > >> >
> > >> > *Thanks & Regards,*
> > >> > *Uday Kumar*
> > >> > *Product Search Tech*
> > >> >
> > >> >
> > >> > On Thu, Feb 27, 2025 at 12:31 PM Mikhail Khludnev <m...@apache.org>
> > >> wrote:
> > >> >
> > >> > > Changing one child rewrites the whole block period.
> > >> > > However in-place updating child docValues is promising in theory,
> > >> > although
> > >> > > I don't know how it works in practice.
> > >> > >
> > >> > > On Thu, Feb 27, 2025 at 8:05 AM Uday Kumar <
> uday.p...@indiamart.com
> > >> > > .invalid>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi all,
> > >> > > > We are doing a POC on indexing nested documents in expectation
> of
> > >> > > reducing
> > >> > > > grouping overhead while querying time.
> > >> > > >
> > >> > > > On Prod Indexing, we are using the traditional approach of
> > >> reindexing
> > >> > the
> > >> > > > entire document if there is any change in any of the fields. [we
> > >> > reindex
> > >> > > > ~2cr documents per day, FYI]
> > >> > > > Solr Version: v9.6.1
> > >> > > >
> > >> > > > But I have come across a caution in solr documentation: *DOC
> > >> > > > <
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-nested-documents.html#:~:text=By%20way%20of%20examples%3A%20nested,%2F%20colors)%20and%20supporting%20documentation%20(
> > >> > > > >*,
> > >> > > > where it says: *Solr must internally reindex an entire nested
> > >> document
> > >> > > tree
> > >> > > > if there are updates to it.*
> > >> > > > Which means If a root or parent has 1000 child documents, even
> > with
> > >> a
> > >> > > > change in single document in any one of the fields, entire
> nested
> > >> > childs
> > >> > > > are reindexed, which is not good enough.
> > >> > > >
> > >> > > > This made us rethink of performance gains that we will have, if
> > >> nested
> > >> > > > documents are used in production.
> > >> > > >
> > >> > > > If that's the case, pls let us know if there are any other
> > solutions
> > >> > > which
> > >> > > > would help us in performance gains.
> > >> > > >
> > >> > > > *Note:*
> > >> > > > We have already done POC on external file fields and In-Place
> > >> updates
> > >> > > where
> > >> > > > we found they are not impactful for our project.
> > >> > > >
> > >> > > > *Thanks & Regards,*
> > >> > > > *Uday Kumar*
> > >> > > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Sincerely yours
> > >> > > Mikhail Khludnev
> > >> > >
> > >> >
> > >>
> > >
> >
>
