Thank you very much Alessandro. I will look into the code.
Danilo Tomasoni Fondazione The Microsoft Research - University of Trento Centre for Computational and Systems Biology (COSBI) Piazza Manifattura 1, 38068 Rovereto (TN), Italy tomas...@cosbi.eu<https://webmail.cosbi.eu/owa/redir.aspx?C=VNXi3_8-qSZTBi-FPvMwmwSB3IhCOjY8nuCBIfcNIs_5SgD-zNPWCA..&URL=mailto%3acalabro%40cosbi.eu> http://www.cosbi.eu<https://webmail.cosbi.eu/owa/redir.aspx?C=CkilyF54_imtLHzZqF1gCGvmYXjsnf4bzGynd8OXm__5SgD-zNPWCA..&URL=http%3a%2f%2fwww.cosbi.eu%2f> As for the European General Data Protection Regulation 2016/679 on the protection of natural persons with regard to the processing of personal data, we inform you that all the data we possess are object of treatment in the respect of the normative provided for by the cited GDPR. It is your right to be informed on which of your data are used and how; you may ask for their correction, cancellation or you may oppose to their use by written request sent by recorded delivery to The Microsoft Research – University of Trento Centre for Computational and Systems Biology Scarl, Piazza Manifattura 1, 38068 Rovereto (TN), Italy. P Please don't print this e-mail unless you really need to ________________________________ Da: Alessandro Benedetti <a.benede...@sease.io> Inviato: lunedì 27 giugno 2022 16:17 A: users@solr.apache.org <users@solr.apache.org> Oggetto: [suspected SPAM] Re: Semantic Knowledge Graph theoric question [CAUTION: EXTERNAL SENDER] [Please check correspondence between Sender Display Name and Sender Email Address before clicking on any link or opening attachments] Hi Davide, I assume that "abstracts_gene_pubtator_annotation_ids" just contains un-tokenized id, so I don't think there is a matter of tokenization, shingles etc. What we want is a single ID to be a single term in the index. If you want to debug the relatedness calculation, take a look here : org.apache.solr.search.facet.RelatednessAgg#computeRelatedness The formula you mentioned is ok, but I would recommend remote debugging Solr and putting some breakpoints there to investigate if something doesn't look right. Let me know! -------------------------- *Alessandro Benedetti* CEO @ Sease Ltd. *Apache Lucene/Solr Committer* *Apache Solr PMC Member* e-mail: a.benede...@sease.io *Sease* - Information Retrieval Applied Consulting | Training | Open Source Website: Sease.io <http://sease.io/> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter <https://twitter.com/seaseltd> | Youtube <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github <https://github.com/seaseltd> On Wed, 22 Jun 2022 at 08:37, Danilo Tomasoni <tomas...@cosbi.eu> wrote: > Hello Dave, first of all thank you for your answer. > > I need to clarify that I've used separate (and quite good) NER algorithms > offline and the results were imported to solr. > > Unfortunately the approach that you suggest using the morelikethis > functionality is not suitable for my needs since I need to discover > statistically significative relations between NER entities, while MLT will > give me NER entities "similar" to the ones I'm looking for, as far as I > understand. > > Anyone knows why the relatedness is high even if the foreground (and even > background) popularity is 0? > > Danilo Tomasoni > > Fondazione The Microsoft Research - University of Trento Centre for > Computational and Systems Biology (COSBI) > Piazza Manifattura 1, 38068 Rovereto (TN), Italy > tomas...@cosbi.eu< > https://webmail.cosbi.eu/owa/redir.aspx?C=VNXi3_8-qSZTBi-FPvMwmwSB3IhCOjY8nuCBIfcNIs_5SgD-zNPWCA..&URL=mailto%3acalabro%40cosbi.eu > > > http://www.cosbi.eu< > https://webmail.cosbi.eu/owa/redir.aspx?C=CkilyF54_imtLHzZqF1gCGvmYXjsnf4bzGynd8OXm__5SgD-zNPWCA..&URL=http%3a%2f%2fwww.cosbi.eu%2f > > > > As for the European General Data Protection Regulation 2016/679 on the > protection of natural persons with regard to the processing of personal > data, we inform you that all the data we possess are object of treatment in > the respect of the normative provided for by the cited GDPR. > It is your right to be informed on which of your data are used and how; > you may ask for their correction, cancellation or you may oppose to their > use by written request sent by recorded delivery to The Microsoft Research > – University of Trento Centre for Computational and Systems Biology Scarl, > Piazza Manifattura 1, 38068 Rovereto (TN), Italy. > P Please don't print this e-mail unless you really need to > ________________________________ > Da: Dave <hastings.recurs...@gmail.com> > Inviato: martedì 21 giugno 2022 19:51 > A: users@solr.apache.org <users@solr.apache.org> > Oggetto: Re: Semantic Knowledge Graph theoric question > > [CAUTION: EXTERNAL SENDER] > [Please check correspondence between Sender Display Name and Sender Email > Address before clicking on any link or opening attachments] > > > Two hints. The ner from solr isn’t very good, and the relatedness function > is iffy at best. > > I would take a different approach. Get the ner data as you have it now and > use shingles to make a better formed complete index using stop words then > use the mlt mech to see if it’s better. If it is, great if not it’s just > an idea. > > > > On Jun 21, 2022, at 12:02 PM, Danilo Tomasoni <tomas...@cosbi.eu> wrote: > > > > Hello all, > > I'm experimenting with the SKG features available through json.facet API > in solr 8.11 to discover semantic relations between medical text > pre-annotated with NER algorithms. > > I store the NER annotations, annotation id, span ecc in separate solr > fields, to keep text clean. > > > > The first results looks promising but I found a behaviour that surprises > me. > > To give a bit of context I'm looking for covid-related papers with a > standard query (q parameter) > > Then I set my foreground query to be a set of keywords in OR related to > the mithochondria, and the background query is set to *. > > > > Then the json.facet parameters are like > > > > "json.facet": { > > "gene":{ > > "type": "terms", > > "field": "abstracts_gene_pubtator_annotation_ids", > > "sort": { "r1": "desc" }, > > "limit": 3, > > "facet": { > > "r1" : "relatedness($fore,$back)" > > } > > } > > } > > This should give gene stored in abstracts_gene_pubtator_annotation_ids > that are more likely to occur in mitochondrial papers. > > Running a test query gives me this surprising result > > > > ... > > "gene": { > > "buckets": [ > > { > > "val": "3091", > > "count": 1, > > "rtitles1": { > > "relatedness": 0.55649, > > "foreground_popularity": 0, > > "background_popularity": 0.00018 > > } > > }, > > ... > > or for a similar query even bigger relatedness values > > ... > > "buckets": [ > > { > > "val": "MESH:D028361", > > "count": 1, > > "rabstract_conclusions0": { > > "relatedness": 0.91506, > > "foreground_popularity": 5e-05, > > "background_popularity": 5e-05 > > }, > > > > ... > > > > But If I recall the z-score formula > > > > countFG("3091") - totalFG * probBG > > ------------------------------------------------ > > sqrt( totalFG * (1-probBG)*probBG ) > > > > and set countFG("3091") to 1 this means that the relatedness should be > negative (or at most 0) if totalFG * probBG >=1, while here I find a quite > positive relatedness. > > Maybe this can be controlled with min_popularity, but I don't understand > how to use it in conjunction with type=terms and > field=abstracts_gene_pubtator_annotation_ids > > > > Can you please tell me the correct syntax, and if my reasoning is > correct? > > Thank you > > Danilo > > > > Danilo Tomasoni > > > > Fondazione The Microsoft Research - University of Trento Centre for > Computational and Systems Biology (COSBI) > > Piazza Manifattura 1, 38068 Rovereto (TN), Italy > > tomas...@cosbi.eu< > https://webmail.cosbi.eu/owa/redir.aspx?C=VNXi3_8-qSZTBi-FPvMwmwSB3IhCOjY8nuCBIfcNIs_5SgD-zNPWCA..&URL=mailto%3acalabro%40cosbi.eu > > > > http://www.cosbi.eu< > https://webmail.cosbi.eu/owa/redir.aspx?C=CkilyF54_imtLHzZqF1gCGvmYXjsnf4bzGynd8OXm__5SgD-zNPWCA..&URL=http%3a%2f%2fwww.cosbi.eu%2f > > > > > > As for the European General Data Protection Regulation 2016/679 on the > protection of natural persons with regard to the processing of personal > data, we inform you that all the data we possess are object of treatment in > the respect of the normative provided for by the cited GDPR. > > It is your right to be informed on which of your data are used and how; > you may ask for their correction, cancellation or you may oppose to their > use by written request sent by recorded delivery to The Microsoft Research > – University of Trento Centre for Computational and Systems Biology Scarl, > Piazza Manifattura 1, 38068 Rovereto (TN), Italy. > > P Please don't print this e-mail unless you really need to >
