childFilter Solr 6.6

[Roopa Rao]

Hi -
I am on version Solr 6.6 and using parent and child construct

Query as follows:
 [child parentFilter='field1:value1' childFilter='((field1:(value1)) AND
({!terms f=field2}value2))' limit=1]

Here I want to sort this children by field3 and return only the top 1
element (hence limit=1) . Is there a way to do this?

I notice that when the children are returned, it is being returned in the
order it was indexed. Is this guaranteed?
If it is going to return the order in which is indexed that will work in my
case too, as I index the field3 data in the order which it should be shown.

Thanks,
Roopa

LTR on child documents

[Roopa Rao]

Hi,

Is there a way to get the feature score on fields on the childDocuments in
LTR.
Note that there could be multiple childDocuments.

If there are example features that can be provided which uses query on the
child documents will be helpful

Thank you,
Roopa

Re: LTR on child documents

[Roopa Rao]

Hi Alessandro,

Thanks for the response.

I am using version Solr 6.6

Basically, I have parent documents and multiple child documents for each
parent.
I am trying to create features and get feature scores on parent attributes
(which is straightforward) and child attributes (which is what I am looking
for examples).
And then rerank documents based on an LTR model which will use these
features (on both parent & child).

Enclosing the parent-child structure and the query construction (where you
can see I am using a child transformer) and also a sample feature, current
sample features score output.



abc-1
feature-1281835650
parent-docs
Parent title1 text1 
Parent summary1 text2


child-doc-12345
feature-1281835650
child-docs
9735
A sample child doc desc
2021-09-01T00:00:00Z

productA
productB
productC



child-doc-56788
feature-1281835650
child-docs
3426
A sample child doc desc -
2
2021-09-02T00:00:00Z

productD
productE
productF




Query construction:
Get documents based on search of parent document attributes - title,
summary.
Get the corresponding child document in the response, filtered by products
eligible to, restrict to 1 (for display purposes).
Rerank based on a model and get feature scores on both attributes on parent
and attributes on child.

https://localhost:8983/testhandler?q=trying to
test&rows=100&start=0&q.op=AND&timeAllowed=2&fl=id,score,title_s,summary_s,[features
store=testFeatureStore],[explain]&fl=_childDocuments_&fl=[child
parentFilter='type:parent-docs' childFilter='((type:(child-docs)) AND
({!terms f=child_doc_product_s_mult}productA,productD))'
limit=1]&sort=score
desc&fq=(type:(parent-docs))&defType=edismax&df=title_qf_default&q.alt=*:*&pf=title_qf_default^2.0
title_qf_synonym^2.0&qf=title_qf_default^2.0
summary_qf_default^1.0&sow=false&lowercaseOperators=false&tie=0.0&rq={!ltr
model=testModel reRankDocs=100 efi.uq=$q}

Here trying to create features for title_s match with uq (user query) -
which works fine since it is on the parent.
Sample feature:
{
"name": "feature_title_match",
"class": "org.apache.solr.ltr.feature.SolrFeature",
"params": {
"q": "{!dismax qf=title_qf_default}${uq:none}"
},
"store": "testFeatureStore"
}

Similarly I need to create features for child_doc_resource_desc_s match
with uq (user query) and a few more features on child_doc attributes.

Right now I am getting scores on the parent attributes as expected.

*However writing features on the child document attributes and getting
feature scores for them is what I am looking for:*
Current Sample output:

originalScore=61.886406,feature_title_match=8.004536,feature_summary_match=3.340,feature_title_synonym_expansion=15.471601

Similarly want to build features: feature_child_doc_resource_desc which
would give a feature score of user query matched with
child_doc_resource_desc_s attribute of the child doc:

So an output like this may be? or feature scores on child comes under each
child document not sure
originalScore=61.886406,feature_title_match=8.004536,feature_summary_match=3.340,feature_title_synonym_expansion=15.471601,feature_child_doc_resource_desc=value1


Thank you!
Roopa

On Tue, Jun 22, 2021 at 1:04 PM Alessandro Benedetti 
wrote:

> Hi Roopa,
> can you elaborate a bit better?
> Child documents are documents nonetheless, so you can query them, rank them
> and re0rank them.
> So, what are you trying to do?
> Are you using any block join related query parsers?
> Do you want to combine it with learning to rank?
> How?
> Let us know and we can try to help!
>
> --
> Alessandro Benedetti
> Apache Lucene/Solr Committer
> Director, R&D Software Engineer, Search Consultant
>
> www.sease.io
>
>
> On Mon, 21 Jun 2021 at 19:12, Roopa Rao  wrote:
>
> > Hi,
> >
> > Is there a way to get the feature score on fields on the childDocuments
> in
> > LTR.
> > Note that there could be multiple childDocuments.
> >
> > If there are example features that can be provided which uses query on
> the
> > child documents will be helpful
> >
> > Thank you,
> > Roopa
> >
>

Re: LTR on child documents

[Roopa Rao]

Hi Alessandro,

Thank you for the details in your response and your interest in this.

Do you know if it might be coming in upcoming Solr releases?

Thanks,
Roopa

On Wed, Jun 23, 2021 at 6:49 AM Alessandro Benedetti 
wrote:

> Hi Roopa,
> that's clearer now, thanks.
>
> I don't think there's any support for feature extraction from children when
> calculating the feature vector of a parent.
> It's an interesting feature though.
> I would love to design and implement such functionality.
> If you want to do that on your own I recommend working from the current
> supported features implementations
> (/solr/contrib/ltr/src/java/org/apache/solr/ltr/feature)
>
> Cheers
> --
> Alessandro Benedetti
> Apache Lucene/Solr Committer
> Director, R&D Software Engineer, Search Consultant
>
> www.sease.io
>
>
> On Tue, 22 Jun 2021 at 19:47, Roopa Rao  wrote:
>
> > Hi Alessandro,
> >
> > Thanks for the response.
> >
> > I am using version Solr 6.6
> >
> > Basically, I have parent documents and multiple child documents for each
> > parent.
> > I am trying to create features and get feature scores on parent
> attributes
> > (which is straightforward) and child attributes (which is what I am
> looking
> > for examples).
> > And then rerank documents based on an LTR model which will use these
> > features (on both parent & child).
> >
> > Enclosing the parent-child structure and the query construction (where
> you
> > can see I am using a child transformer) and also a sample feature,
> current
> > sample features score output.
> >
> >
> > 
> > abc-1
> > feature-1281835650
> > parent-docs
> > Parent title1 text1 
> > Parent summary1 text2
> >
> > 
> > child-doc-12345
> > feature-1281835650
> > child-docs
> > 9735
> > A sample child doc
> desc
> > 2021-09-01T00:00:00Z
> > 
> > productA
> > productB
> > productC
> > 
> > 
> > 
> > child-doc-56788
> > feature-1281835650
> > child-docs
> > 3426
> > A sample child doc desc -
> > 2
> > 2021-09-02T00:00:00Z
> > 
> > productD
> > productE
> > productF
> > 
> > 
> > 
> >
> > Query construction:
> > Get documents based on search of parent document attributes - title,
> > summary.
> > Get the corresponding child document in the response, filtered by
> products
> > eligible to, restrict to 1 (for display purposes).
> > Rerank based on a model and get feature scores on both attributes on
> parent
> > and attributes on child.
> >
> > https://localhost:8983/testhandler?q=trying to
> >
> >
> test&rows=100&start=0&q.op=AND&timeAllowed=2&fl=id,score,title_s,summary_s,[features
> > store=testFeatureStore],[explain]&fl=_childDocuments_&fl=[child
> > parentFilter='type:parent-docs' childFilter='((type:(child-docs)) AND
> > ({!terms f=child_doc_product_s_mult}productA,productD))'
> > limit=1]&sort=score
> >
> >
> desc&fq=(type:(parent-docs))&defType=edismax&df=title_qf_default&q.alt=*:*&pf=title_qf_default^2.0
> > title_qf_synonym^2.0&qf=title_qf_default^2.0
> >
> summary_qf_default^1.0&sow=false&lowercaseOperators=false&tie=0.0&rq={!ltr
> > model=testModel reRankDocs=100 efi.uq=$q}
> >
> > Here trying to create features for title_s match with uq (user query) -
> > which works fine since it is on the parent.
> > Sample feature:
> > {
> > "name": "feature_title_match",
> > "class": "org.apache.solr.ltr.feature.SolrFeature",
> > "params": {
> > "q": "{!dismax qf=title_qf_default}${uq:none}"
> > },
> > "store": "testFeatureStore"
> > }
> >
> > Similarly I need to create features for child_doc_resource_desc_s match
> > with uq (user query) and a few more features on child_doc attributes.
> >
> > Right now I am getting scores on the parent attributes as expected.
> >
> > *However writing features on the child document attributes and getting
> > feature scores for them is what I am looking for:*
> > Current Sample output:
> >
> >  >
> 

MultipleAdditiveTreeModel

[Roopa Rao]

Hi All,

In LTR for MultipleAdditiveTreeModel what is the purpose of adding
NODE_SPLIT_SLACK
to the threshold?

Reference: org.apache.solr.ltr.model.MultipleAdditiveTreesModel

private static final float NODE_SPLIT_SLACK = 1E-6f;


public void setThreshold(float threshold) { this.threshold = threshold +
NODE_SPLIT_SLACK; }

We have a feature which can return 0.0 or 1.0

And model with this tree:

is_xyz_feature,threshold=0.9994,left=0.0010180053,right=-0.0057609854

However when Solr actually scores it it is taking it as follows
is_xyz_feature:1.0<= 1.01, Go Left

So all the time it goes to left which is incorrect.

Thanks,
Roopa

Re: MultipleAdditiveTreeModel

[Roopa Rao]

Okay, thank you for the input

Roopa

On Fri, Jul 16, 2021 at 5:55 AM Alessandro Benedetti 
wrote:

> Hi Roopa,
> I was not able to find why that slack was added.
> I am not sure why we would like to change the threshold.
> I would recommend creating a Jira issue and tag at least myself, Christine
> Poerschke and Diego Ceccarelli, so we can discuss and potentially open a
> pull request.
>
> Cheers
>
>
> --
> Alessandro Benedetti
> Apache Lucene/Solr Committer
> Director, R&D Software Engineer, Search Consultant
>
> www.sease.io
>
>
> On Thu, 15 Jul 2021 at 22:24, Roopa Rao  wrote:
>
> > Hi All,
> >
> > In LTR for MultipleAdditiveTreeModel what is the purpose of adding
> > NODE_SPLIT_SLACK
> > to the threshold?
> >
> > Reference: org.apache.solr.ltr.model.MultipleAdditiveTreesModel
> >
> > private static final float NODE_SPLIT_SLACK = 1E-6f;
> >
> >
> > public void setThreshold(float threshold) { this.threshold = threshold +
> > NODE_SPLIT_SLACK; }
> >
> > We have a feature which can return 0.0 or 1.0
> >
> > And model with this tree:
> >
> > is_xyz_feature,threshold=0.9994,left=0.0010180053,right=-0.0057609854
> >
> > However when Solr actually scores it it is taking it as follows
> > is_xyz_feature:1.0<= 1.01, Go Left
> >
> > So all the time it goes to left which is incorrect.
> >
> > Thanks,
> > Roopa
> >
>

Re: MultipleAdditiveTreeModel

[Roopa Rao]

Spyros, thank you for verifying this, we are planning to do something
similar.

Thanks,
Roopa

On Fri, Jul 16, 2021 at 12:09 PM Spyros Kapnissis  wrote:

> Hello,
>
> Just to verify this, we had come across the exact same issue when
> converting an XGBoost model to MUltipleAdditiveTrees. This was an issue
> specifically with the categorical features that take on integer values. We
> ended up subtracting 0.5 from the threshold value on any such split point
> on the converted model, so that it would output the same score as the input
> model.
>
> On Fri, Jul 16, 2021, 18:19 Roopa Rao  wrote:
>
> > Okay, thank you for the input
> >
> > Roopa
> >
> > On Fri, Jul 16, 2021 at 5:55 AM Alessandro Benedetti <
> a.benede...@sease.io
> > >
> > wrote:
> >
> > > Hi Roopa,
> > > I was not able to find why that slack was added.
> > > I am not sure why we would like to change the threshold.
> > > I would recommend creating a Jira issue and tag at least myself,
> > Christine
> > > Poerschke and Diego Ceccarelli, so we can discuss and potentially open
> a
> > > pull request.
> > >
> > > Cheers
> > >
> > >
> > > --
> > > Alessandro Benedetti
> > > Apache Lucene/Solr Committer
> > > Director, R&D Software Engineer, Search Consultant
> > >
> > > www.sease.io
> > >
> > >
> > > On Thu, 15 Jul 2021 at 22:24, Roopa Rao  wrote:
> > >
> > > > Hi All,
> > > >
> > > > In LTR for MultipleAdditiveTreeModel what is the purpose of adding
> > > > NODE_SPLIT_SLACK
> > > > to the threshold?
> > > >
> > > > Reference: org.apache.solr.ltr.model.MultipleAdditiveTreesModel
> > > >
> > > > private static final float NODE_SPLIT_SLACK = 1E-6f;
> > > >
> > > >
> > > > public void setThreshold(float threshold) { this.threshold =
> threshold
> > +
> > > > NODE_SPLIT_SLACK; }
> > > >
> > > > We have a feature which can return 0.0 or 1.0
> > > >
> > > > And model with this tree:
> > > >
> > > >
> > is_xyz_feature,threshold=0.9994,left=0.0010180053,right=-0.0057609854
> > > >
> > > > However when Solr actually scores it it is taking it as follows
> > > > is_xyz_feature:1.0<= 1.01, Go Left
> > > >
> > > > So all the time it goes to left which is incorrect.
> > > >
> > > > Thanks,
> > > > Roopa
> > > >
> > >
> >
>

Re: MultipleAdditiveTreeModel

[Roopa Rao]

Hi Alessandro,
I haven't created JIRA for this, we solved this the similar way that Spyros
described, by changing the threshold in the model.
Ya it would be good to understand why there is the SLACK added.

Thanks,
Roopa

On Mon, Jul 26, 2021 at 10:52 AM Alessandro Benedetti 
wrote:

> I didn't get any additional notification (or maybe I missed it).
> Has the Jira been created yet?
> Boolean features are quite common around Learning To Rank use cases.
> I do believe this contribution can be useful.
> If you don't have time to create the Jira or contribute the pull request,
> no worries, just let us know and we (committers) will organize to do it.
> Thanks for your help. without the effort of our users, Apache Solr wouldn't
> be the same.
> Cheers
> --
> Alessandro Benedetti
> Apache Lucene/Solr Committer
> Director, R&D Software Engineer, Search Consultant
>
> www.sease.io
>
>
> On Fri, 16 Jul 2021 at 20:29, Roopa Rao  wrote:
>
> > Spyros, thank you for verifying this, we are planning to do something
> > similar.
> >
> > Thanks,
> > Roopa
> >
> > On Fri, Jul 16, 2021 at 12:09 PM Spyros Kapnissis 
> > wrote:
> >
> > > Hello,
> > >
> > > Just to verify this, we had come across the exact same issue when
> > > converting an XGBoost model to MUltipleAdditiveTrees. This was an issue
> > > specifically with the categorical features that take on integer values.
> > We
> > > ended up subtracting 0.5 from the threshold value on any such split
> point
> > > on the converted model, so that it would output the same score as the
> > input
> > > model.
> > >
> > > On Fri, Jul 16, 2021, 18:19 Roopa Rao  wrote:
> > >
> > > > Okay, thank you for the input
> > > >
> > > > Roopa
> > > >
> > > > On Fri, Jul 16, 2021 at 5:55 AM Alessandro Benedetti <
> > > a.benede...@sease.io
> > > > >
> > > > wrote:
> > > >
> > > > > Hi Roopa,
> > > > > I was not able to find why that slack was added.
> > > > > I am not sure why we would like to change the threshold.
> > > > > I would recommend creating a Jira issue and tag at least myself,
> > > > Christine
> > > > > Poerschke and Diego Ceccarelli, so we can discuss and potentially
> > open
> > > a
> > > > > pull request.
> > > > >
> > > > > Cheers
> > > > >
> > > > >
> > > > > --
> > > > > Alessandro Benedetti
> > > > > Apache Lucene/Solr Committer
> > > > > Director, R&D Software Engineer, Search Consultant
> > > > >
> > > > > www.sease.io
> > > > >
> > > > >
> > > > > On Thu, 15 Jul 2021 at 22:24, Roopa Rao  wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > In LTR for MultipleAdditiveTreeModel what is the purpose of
> adding
> > > > > > NODE_SPLIT_SLACK
> > > > > > to the threshold?
> > > > > >
> > > > > > Reference: org.apache.solr.ltr.model.MultipleAdditiveTreesModel
> > > > > >
> > > > > > private static final float NODE_SPLIT_SLACK = 1E-6f;
> > > > > >
> > > > > >
> > > > > > public void setThreshold(float threshold) { this.threshold =
> > > threshold
> > > > +
> > > > > > NODE_SPLIT_SLACK; }
> > > > > >
> > > > > > We have a feature which can return 0.0 or 1.0
> > > > > >
> > > > > > And model with this tree:
> > > > > >
> > > > > >
> > > >
> > is_xyz_feature,threshold=0.9994,left=0.0010180053,right=-0.0057609854
> > > > > >
> > > > > > However when Solr actually scores it it is taking it as follows
> > > > > > is_xyz_feature:1.0<= 1.01, Go Left
> > > > > >
> > > > > > So all the time it goes to left which is incorrect.
> > > > > >
> > > > > > Thanks,
> > > > > > Roopa
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: MultipleAdditiveTreeModel

[Roopa Rao]

Thank you, Spyros.

Roopa

On Wed, Jul 28, 2021 at 3:00 PM Spyros Kapnissis  wrote:

> Hi Alessandro, Roopa, I created the ticket here:
> https://issues.apache.org/jira/browse/SOLR-15569 . I don't think I have
> permission to add people though, so please tag whomever you feel is
> necessary.
> Pls let me know if you need any more info, thanks!
>
> On Tue, Jul 27, 2021 at 1:00 PM Alessandro Benedetti  >
> wrote:
>
> > Hi Spyros, Roopa,
> > if you can create the Jira ticket with all the details you gathered, that
> > would be much appreciated.
> > If you tag me, Christine Poerschke, and Diego Ceccarelli at least, we'll
> > take over from there!
> > Thanks!
> > --
> > Alessandro Benedetti
> > Apache Lucene/Solr Committer
> > Director, R&D Software Engineer, Search Consultant
> >
> > www.sease.io
> >
> >
> > On Mon, 26 Jul 2021 at 21:29, Spyros Kapnissis  wrote:
> >
> > > Hi Alessandro, Roopa, I also agree that this issue should be further
> > > investigated and fixed. Please let me know if you need any help opening
> > the
> > > Jira ticket and provide more details.
> > >
> > > On Mon, Jul 26, 2021, 21:04 Roopa Rao  wrote:
> > >
> > > > Hi Alessandro,
> > > > I haven't created JIRA for this, we solved this the similar way that
> > > Spyros
> > > > described, by changing the threshold in the model.
> > > > Ya it would be good to understand why there is the SLACK added.
> > > >
> > > > Thanks,
> > > > Roopa
> > > >
> > > > On Mon, Jul 26, 2021 at 10:52 AM Alessandro Benedetti <
> > > > a.benede...@sease.io>
> > > > wrote:
> > > >
> > > > > I didn't get any additional notification (or maybe I missed it).
> > > > > Has the Jira been created yet?
> > > > > Boolean features are quite common around Learning To Rank use
> cases.
> > > > > I do believe this contribution can be useful.
> > > > > If you don't have time to create the Jira or contribute the pull
> > > request,
> > > > > no worries, just let us know and we (committers) will organize to
> do
> > > it.
> > > > > Thanks for your help. without the effort of our users, Apache Solr
> > > > wouldn't
> > > > > be the same.
> > > > > Cheers
> > > > > --
> > > > > Alessandro Benedetti
> > > > > Apache Lucene/Solr Committer
> > > > > Director, R&D Software Engineer, Search Consultant
> > > > >
> > > > > www.sease.io
> > > > >
> > > > >
> > > > > On Fri, 16 Jul 2021 at 20:29, Roopa Rao  wrote:
> > > > >
> > > > > > Spyros, thank you for verifying this, we are planning to do
> > something
> > > > > > similar.
> > > > > >
> > > > > > Thanks,
> > > > > > Roopa
> > > > > >
> > > > > > On Fri, Jul 16, 2021 at 12:09 PM Spyros Kapnissis <
> > ska...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > Just to verify this, we had come across the exact same issue
> when
> > > > > > > converting an XGBoost model to MUltipleAdditiveTrees. This was
> an
> > > > issue
> > > > > > > specifically with the categorical features that take on integer
> > > > values.
> > > > > > We
> > > > > > > ended up subtracting 0.5 from the threshold value on any such
> > split
> > > > > point
> > > > > > > on the converted model, so that it would output the same score
> as
> > > the
> > > > > > input
> > > > > > > model.
> > > > > > >
> > > > > > > On Fri, Jul 16, 2021, 18:19 Roopa Rao 
> wrote:
> > > > > > >
> > > > > > > > Okay, thank you for the input
> > > > > > > >
> > > > > > > > Roopa
> > > > > > > >
> > > > > > > > On Fri, Jul 16, 2021 at 5:55 AM Alessandro Benedetti <
> > > > > > > a.benede...@sease.io
> > > &g

Re: [TOKYO Lucene/Solr Meetup] Lecture by Alessandro Benedetti and Mingchun Zhao

[Roopa Rao]

Hi @Alessandro Benedetti 

Which version of Solr will this be targeted to be released?

Thanks


On Tue, Oct 15, 2024 at 10:20 AM Alessandro Benedetti 
wrote:

> Hi,
> I was talking with Kaya (who reads in cc) and it seems it's not possible to
> register to attend physically?
> For Guillaume, my part is on youtube already:
> https://www.youtube.com/watch?v=8x2cbT5CCEM
> Cheers
> --
> *Alessandro Benedetti*
> Director @ Sease Ltd.
> *Apache Lucene/Solr Committer*
> *Apache Solr PMC Member*
>
> e-mail: a.benede...@sease.io
>
>
> *Sease* - Information Retrieval Applied
> Consulting | Training | Open Source
>
> Website: Sease.io 
> LinkedIn  | Twitter
>  | Youtube
>  | Github
> 
>
>
> On Tue, 15 Oct 2024 at 18:35, Moriyasu Kannami <
> moriyasu.kann...@rondhuit.com> wrote:
>
> > Hi Guillaume!
> >
> > Thank you for showing interest in this event.
> >
> > > Will this event be available as a replay?
> >
> > No, this event is not available as a reply, sorry.
> >
> > Regards,
> > moriyasu
> >
> > 2024年10月15日(火) 18:21 Guillaume :
> >
> > > Hello,
> > > Great event and great topics !
> > > Will this event be available as a replay?
> > > Not sure I can make myself available for the Google Meet in real time.
> > >
> > > Thanks !
> > >
> > > Le mar. 15 oct. 2024 à 03:32, Moriyasu Kannami <
> > > moriyasu.kann...@rondhuit.com> a écrit :
> > >
> > > > Hi, Everyone!
> > > >
> > > > We apologize for the last-minute notice.
> > > > The Solr community in Tokyo(#SolrJP) will hold a meet-up on 16th
> > October
> > > > 2024 at 6:00 PM JST in ToranomonMinato-Ku Tokyo.
> > > > Online streaming via Google Meet is also available.
> > > >
> > > > The topics are the following two:
> > > >
> > > > 1. Solr 9.7 Load Test Report: Evaluation of Semantic Search
> Performance
> > > > Improvement through SIMD Optimization (JA) (25min)
> > > >   Speaker : Mingchun Zhao
> > > >   Apache ManifoldCF Committer & PMC member
> > > >
> > > > 2. Hybrid Search with Apache Solr Reciprocal Rank Fusion (EN) (45min)
> > > >   Speaker : Alessandro Benedetti
> > > >   Apache Lucene/Solr Committer & PMC Member
> > > >
> > > > If you wish to participate, please register using the following URL.
> > > > The registration for online participation closes at 9:00 AM on
> October
> > 16
> > > > JST.
> > > > Registration for in-person participation has closed.
> > > >
> > > > The event page(in Japanese):
> https://solr.doorkeeper.jp/events/178455
> > > >
> > > > You can also check the details of the lecture content at the URL
> above.
> > > > We look forward to your participation.
> > > >
> > > > Thanks!
> > > >
> > > > moriyasu
> > > >
> > >
> >
>