Re: Deduplication of search result with custom with custom sort

2020-10-13 Thread Dmitry Emets
equent and > you are not strict on 1000, you might retrieve more let's say 2000 without > grouping and then do the deduping after.. > > Cheers, > Diego > > > From: java-user@lucene.apache.org At: 10/12/20 13:02:46To: > java-user@lucene.apache.org > Subject: Re: Dedu

Re: Deduplication of search result with custom with custom sort

2020-10-12 Thread Diego Ceccarelli (BLOOMBERG/ LONDON)
let's say 2000 without grouping and then do the deduping after.. Cheers, Diego From: java-user@lucene.apache.org At: 10/12/20 13:02:46To: java-user@lucene.apache.org Subject: Re: Deduplication of search result with custom with custom sort Thank you very much for helping! There isn't muc

Re: Deduplication of search result with custom with custom sort

2020-10-12 Thread Dmitry Emets
about the use case? > > There might be another way to achieve the same result. > > > > What are these documents? > > Why do you need 1000 docs per user? > > > > > > From: java-user@lucene.apache.org At: 10/09/20 14:25:02To: > > java-user@lucene.apache.org &

Re: Deduplication of search result with custom with custom sort

2020-10-09 Thread Jigar Shah
result. > > What are these documents? > Why do you need 1000 docs per user? > > > From: java-user@lucene.apache.org At: 10/09/20 14:25:02To: > java-user@lucene.apache.org > Subject: Re: Deduplication of search result with custom with custom sort > > 6_500_000 is the to

Re: Deduplication of search result with custom with custom sort

2020-10-09 Thread Diego Ceccarelli (BLOOMBERG/ LONDON)
: Deduplication of search result with custom with custom sort 6_500_000 is the total count of groups in the entire collection. I only return the top 1000 to users. I use Lucene where I have documents that can have the same docvalue, and I want to deduplicate this documents by this docvalue during search. Also

Re: Deduplication of search result with custom with custom sort

2020-10-09 Thread Dmitry Emets
t; >> dceccarel...@bloomberg.net> wrote: > >>> > >>> Is the field that you are using to dedupe stored as a docvalue? > >>> > >>> From: java-user@lucene.apache.org At: 10/09/20 12:18:04To: > >> java-user@lucene.apache.org > >>&

Re: Deduplication of search result with custom with custom sort

2020-10-09 Thread Erick Erickson
g.net> wrote: >>> >>> Is the field that you are using to dedupe stored as a docvalue? >>> >>> From: java-user@lucene.apache.org At: 10/09/20 12:18:04To: >> java-user@lucene.apache.org >>> Subject: Deduplication of search result with custom with cust

Re: Deduplication of search result with custom with custom sort

2020-10-09 Thread Dmitry Emets
:04To: > java-user@lucene.apache.org > > Subject: Deduplication of search result with custom with custom sort > > > > Hi, > > I need to deduplicate search results by specific field and I have no idea > > how to implement this properly. > > I have tried grouping

Re: Deduplication of search result with custom with custom sort

2020-10-09 Thread Erick Erickson
wrote: > > Is the field that you are using to dedupe stored as a docvalue? > > From: java-user@lucene.apache.org At: 10/09/20 12:18:04To: > java-user@lucene.apache.org > Subject: Deduplication of search result with custom with custom sort > > Hi, > I need to dedup

Re: Deduplication of search result with custom with custom sort

2020-10-09 Thread Diego Ceccarelli (BLOOMBERG/ LONDON)
How many documents in the collection, how many groups, and how long is it taking to do the grouping vs no grouping? Also, if you remove the custom sort is it still slow? From: java-user@lucene.apache.org At: 10/09/20 12:27:25To: Diego Ceccarelli (BLOOMBERG/ LONDON ) , java-user

Re: Deduplication of search result with custom with custom sort

2020-10-09 Thread Dmitry Emets
ect: Deduplication of search result with custom with custom sort > > Hi, > I need to deduplicate search results by specific field and I have no idea > how to implement this properly. > I have tried grouping with setGroupDocsLimit(1) and it gives me expected > results, but has not

Re:Deduplication of search result with custom with custom sort

2020-10-09 Thread Diego Ceccarelli (BLOOMBERG/ LONDON)
Is the field that you are using to dedupe stored as a docvalue? From: java-user@lucene.apache.org At: 10/09/20 12:18:04To: java-user@lucene.apache.org Subject: Deduplication of search result with custom with custom sort Hi, I need to deduplicate search results by specific field and I have no

Deduplication of search result with custom with custom sort

2020-10-09 Thread Dmitry Emets
Hi, I need to deduplicate search results by specific field and I have no idea how to implement this properly. I have tried grouping with setGroupDocsLimit(1) and it gives me expected results, but has not very good performance. I think that I need something like DiversifiedTopDocsCollector, but suit

Re: DisjunctionMaxQuery and custom Sort

2010-03-05 Thread gabriele renzi
On Fri, Mar 5, 2010 at 10:42 AM, Ian Lea wrote: > From the javadocs for the search with sort method it appears you are calling: > > NOTE: this does not compute scores by default; use > IndexSearcher.setDefaultFieldSortScoring(boolean, boolean) to enable > scoring. > > > Sounds like you need to cal

Re: DisjunctionMaxQuery and custom Sort

2010-03-05 Thread Ian Lea
>From the javadocs for the search with sort method it appears you are calling: NOTE: this does not compute scores by default; use IndexSearcher.setDefaultFieldSortScoring(boolean, boolean) to enable scoring. Sounds like you need to call that with at least the first arg set to true. -- Ian. O

DisjunctionMaxQuery and custom Sort

2010-03-05 Thread gabriele renzi
Hi everyone, In our search app we'd perform searches with a DisjunctionMaxQuery with code like searcher.search(query, hits) Now we'd like to add a custom sub-sort so that when the search returns two documents with the same score they will be ordered by a custom field (this is not a boost for t

Re: Memory Leak when using Custom Sort (i.e., DistanceSortSource) of LocalLucene with Lucene

2008-06-10 Thread Otis Gospodnetic
[EMAIL PROTECTED]> > To: java-user@lucene.apache.org > Sent: Monday, June 9, 2008 5:10:31 PM > Subject: Memory Leak when using Custom Sort (i.e., DistanceSortSource) of > LocalLucene with Lucene > > Hi, > > > > We had the memory leak issue when using DistanceSortSou

Memory Leak when using Custom Sort (i.e., DistanceSortSource) of LocalLucene with Lucene

2008-06-10 Thread Ethan Tao
interface SortComparatorSourceUncacheable extends Serializable { } 2. Have your custom sort class to implement the interface public class LocalSortSource extends DistanceSortSource implements SortComparatorSourceUncacheable { ... } 3. Modify Lucene's FieldSorterHitQueue.java to bypass cac

custom sort and out of memory

2008-02-17 Thread Brian Doyle
ds we would run out of memory about once every 1.5 hours. We modified our custom sort comparators to implement the equals and hashcode methods and used a WeakHashMap to cache the doc ids to and their lat/lon values. We ran some tests and it started to reuse those comparators and now it will go for ma

Re: custom sort

2006-08-11 Thread Chris Hostetter
" <[EMAIL PROTECTED]> : To: : Sent: Thursday, August 10, 2006 1:41 AM : Subject: Re: custom sort : : : : what you want is not a customized sort as much as a customized Score .. : scores can be customized by modifying your Similarity, class -- LIA has : some good info on this, but the b

Re: custom sort

2006-08-10 Thread Enrique Lamas
e document. I subclass DefaultSimilarity and set it to IndexSearcher, but I don't know what reimplementation of the methods is what I need. Thanks - Original Message - From: "Chris Hostetter" <[EMAIL PROTECTED]> To: Sent: Thursday, August 10, 2006 1:41 AM Subject: Re: c

Re: custom sort

2006-08-09 Thread Chris Hostetter
search.explain output for your searches to understand how your changes affect things. : Date: Wed, 9 Aug 2006 13:46:30 +0200 : From: Enrique Lamas <[EMAIL PROTECTED]> : Reply-To: java-user@lucene.apache.org : To: java-user@lucene.apache.org : Subject: custom sort : : Hi, : I want to execute a

custom sort

2006-08-09 Thread Enrique Lamas
Hi, I want to execute a query searching a few terms QueryParser queryParser = new MultiFieldQueryParser(new String[] {"tags", "title"}, ProcessConstants.analyzer); Query query = queryParser.parse("rocio ortega"); and I want to obtain the results sorted by the number of founded terms, but not c

Re: question about custom sort method

2006-07-19 Thread Aleksey Serba
Erik, You can reproduce OutOfMemory easily. I've attach test files - this is altered DistanceSortingTest example from LIA book. Also you can profile it and see caching of distances arrays. I'll try to investigate the problem, make patch to trunk version (probably non caching option) and get back

Re: question about custom sort method

2006-07-17 Thread Yonik Seeley
On 7/17/06, Aleksey Serba <[EMAIL PROTECTED]> wrote: Yonik, thank you for your suggestion, we use solr codebase already :) To tell the truth, first time i thought this is solr caching problem ( i've modify SolrIndexSearcher to get lucene Searcher and search directly without any solr caching ) I

Re: question about custom sort method

2006-07-17 Thread Aleksey Serba
Erik, I think Brian have the problem with continuous caching the same sorting values, i.e. he has a few points to calc distance from. In such case you can implement equals and hashCode methods based on point value and you'll have one cached comparator per different center point value. On 7/17/0

Re: question about custom sort method

2006-07-17 Thread Erik Hatcher
There is a known issue with the DistanceComparatorSource in the "Lucene in Action" source code: Maybe this advice will help fix the issue you're having? Erik On Jul 17, 2006, at 9:54 AM, Aleksey Serba wrote:

Re: question about custom sort method

2006-07-17 Thread Aleksey Serba
Hi! Peter, I have exactly the same situation described below. - I have DistanceComparatorSource to sort results by distance from specified spatial coordinates point. Point is different for each query. - I do not close Searcher after each query. - I get "java.lang.OutOfMemoryError: Java heap space

Re: question about custom sort method

2006-05-17 Thread Yonik Seeley
FunctionQuery could also be used for this. http://incubator.apache.org/solr/docs/api/org/apache/solr/search/function/FunctionQuery.html -Yonik http://incubator.apache.org/solr Solr, the open-source Lucene search server On 5/17/06, Peter Keegan <[EMAIL PROTECTED]> wrote: Suppose I have a custom

Re: question about custom sort method

2006-05-17 Thread Peter Keegan
Suppose I have a custom sorting 'DocScoreComparator' for computing distances on each search hit from a specified coordinate (similar to the DistanceComparatorSource example in LIA). Assume that the 'specified coordinate' is different for each query. This means a new custom comparator must be creat

Re: question about custom sort method

2006-05-12 Thread Erik Hatcher
Yes, it does compute these distances for all the terms for the field specified, but only once (per IndexReader). This is where the techniques Solr employs comes in real handy... warming up caches by running searches and sorts before putting a index into service. Erik On May 12, 2

question about custom sort method

2006-05-12 Thread Urvashi Gadi
I am looking at DistanceComparatorSource class (for csutom sorting) and looks like it calculates the distance for each record in the index and not just the records returned from search, making the system very slow. Is my observation correct? Are there ways to optimize this process? Thanks, Urv

Re: using custom sort method

2006-04-19 Thread Urvashi Gadi
requirement is to combine 2 or more fields using some critera (for example weighted average) and sort the search results based on the combined fields. I am looking at DistanceComparatorSource class to implement custom sort but it takes only one field for calculation and then sorts the result.

Re: using custom sort method

2006-04-18 Thread Chris Hostetter
: I have asked the exact same question a few weeks ago. I just follow the : customized distance example and loop the results again to get another : field and compute the scores. It will be painful if you need more than 3 : fields. So far I didn't find any other way to do it. Hope we can see It's

Re: using custom sort method

2006-04-18 Thread Yang Sun
he combined fields. I am looking at DistanceComparatorSource class to implement custom sort but it takes only one field for calculation and then sorts the result. Is there a way to use more than one field? I looked in sorts in succession by the criteria in each SortField class but this does

Re: using custom sort method

2006-04-18 Thread Urvashi Gadi
rvashi Gadi wrote: Hello All, My requirement is to combine 2 or more fields using some critera (for example weighted average) and sort the search results based on the combined fields. I am looking at DistanceComparatorSource class to implement custom sort but it takes only one fie

Re: using custom sort method

2006-04-18 Thread Erik Hatcher
example weighted average) and sort the search results based on the combined fields. I am looking at DistanceComparatorSource class to implement custom sort but it takes only one field for calculation and then sorts the result. Is there a way to use more than one field? I looked in sorts in

using custom sort method

2006-04-18 Thread Urvashi Gadi
Hello All, My requirement is to combine 2 or more fields using some critera (for example weighted average) and sort the search results based on the combined fields. I am looking at DistanceComparatorSource class to implement custom sort but it takes only one field for calculation and then

Re: Custom sort/basic question

2005-11-22 Thread Erik Hatcher
On 22 Nov 2005, at 11:29, Gus Kormeier wrote: Hey John, My understanding is that if you add a field with the same name as a previous field added, you will be overwriting the value stored in the document. So if you add: doc.add(Field.Text("sequence", "1")); doc.add(Field.Text("sequence",

RE: Custom sort/basic question

2005-11-22 Thread Gus Kormeier
ot;)); doc.add(Field.Text("sequence", "3")); Afterwards, the field sequence would hold a value of 3. I'm guessing that by now you have already tested that. -Gus -Original Message- From: John Powers [mailto:[EMAIL PROTECTED] Sent: Monday, November 21, 2005 3:01 PM To

Custom sort/basic question

2005-11-21 Thread John Powers
If I add keywords to a document at the same time, will they stay in that order? Create New doc A doc.add(Field.Text("category", "toys")); doc.add(Field.Text("sequence", "235")); doc.add(Field.Text("category", "bears")); doc.add(Field.Text("sequence", "63")); doc.add(Field.Text("category", "truc

Re: Custom sort with multiple fields?

2005-10-10 Thread Yonik Seeley
rformance would depend somewhat on what function you are calculating. -Yonik Now hiring -- http://tinyurl.com/7m67g On 10/9/05, Jeff Rodenburg <[EMAIL PROTECTED]> wrote: > > In following the LIA custom sort example, the calculated sort value is > based > on a field that contains all

Custom sort with multiple fields?

2005-10-09 Thread Jeff Rodenburg
In following the LIA custom sort example, the calculated sort value is based on a field that contains all necessary values, i.e. "x,y" which is split into two values for use in a distance algorithm. Suppose I want a custom sort basis that performs a similar calculation, but is based on

Re: custom sort

2005-09-08 Thread raymondcreel (sent by Nabble.com)
Hi thanks for the reply. Yes that sounds like it would work with the two searches. Perhaps a custom sort might be less overhead since it would just be one search, but I think your solution will work for my purposes. Thanks much. raymond -- Sent from the Lucene - Java Users forum at

Re: custom sort

2005-08-31 Thread Chris Hostetter
round : with the custom sorting and report back if I figure out an elegant way : to do it. you could definitely do this with a Custom Sort ... but a simpler way to go would be to do two searches. if the users's basic search criteria is soemthing like "foo:bar +yak:wak" and the criteria

Re: custom sort

2005-08-31 Thread Jason Haruska
gt; the collection. This is slow and ugly. I was hoping there might be a slicker > way to do it as part of the actual sort. I will play around with the custom > sorting and report back if I figure out an elegant way to do it. > > Thanks for all your replies. > Raymond > -- > Sent

Re: custom sort

2005-08-31 Thread raymondcreel (sent by Nabble.com)
ttp://www.nabble.com/custom-sort-t262833.html#a750675

Re: custom sort

2005-08-30 Thread Chris Hostetter
: You can just assign the field B some weight when creating the index? that implies that the field "A" being sorted on is SCORE ... which isn't allways the case. : Is it possible to write a custom sort for a query such that the first : N documents that match a certain addition

RE: custom sort

2005-08-30 Thread Mordo, Aviran (EXP N-NANNATEK)
When using sort there is no meaning for weight. Aviran http://www.aviransplace.com -Original Message- From: Chris Lu [mailto:[EMAIL PROTECTED] Sent: Tuesday, August 30, 2005 12:35 AM To: java-user@lucene.apache.org; raymondcreel Subject: Re: custom sort You can just assign the field

Re: custom sort

2005-08-29 Thread Chris Lu
You can just assign the field B some weight when creating the index? -- Chris Lu Lucene Search RAD on Any Database http://www.dbsight.net On 8/29/05, raymondcreel (sent by Nabble.com) <[EMAIL PROTECTED]> wrote: > > Is it possible to write a custom sort for a query s

custom sort

2005-08-29 Thread raymondcreel (sent by Nabble.com)
Is it possible to write a custom sort for a query such that the first N documents that match a certain additional criteria get pushed to the top of the sort? For instance say you sort your query based on field A, but you want to tweak the results such that the first 10 documents in the result