Re: delete entries from posting list Lucene 4.0

2012-03-29 Thread Andrzej Bialecki
On 27/03/2012 20:25, Zeynep P. wrote: While using the pruning package, I realised that ridf is calculated in RIDFTermPruningPolicy as follows: Math.log(1 - Math.pow(Math.E, termPositions.freq() / maxDoc)) - df However, according to the original paper (Blanco et al.) for residual idf, it should b

JoinUtil.createJoinQuery in 3.6 ?

2012-03-29 Thread kiwi clive
Hi Guys, Will this be available in Lucene 3.6 or is it only going into version 4.0 ? Clive

createJoinQuery use

2012-03-29 Thread kiwi clive
Hi Chaps, JoinUtil.createJoinQuery() specifies a Query for the from side of the join. Is it possible to query over both sides of the join (while still providing the two join fields) ? If not, what is the recommended best practice to do this? Thanks, and apologies for the dumb questions C

Re: JoinUtil.createJoinQuery in 3.6 ?

2012-03-29 Thread Michael McCandless
It'll be in both 3.6 and 4.0. Mike McCandless http://blog.mikemccandless.com On Thu, Mar 29, 2012 at 7:55 AM, kiwi clive wrote: > Hi Guys, > Will this be available in Lucene 3.6 or is it only going into version 4.0 ? > > Clive ---

conditional High Freq Terms in Lucene index

2012-03-29 Thread starz10de
HI, I am using HighFreqTerms class to compute the high frequent terms in the Lucene index and it works well. However, I am interested to compute the high frequent terms under some condition. I would like to compute the high frequent terms not for all documents in the index instead only for documen

Re: Can I add new field values to a existing lucene index ?

2012-03-29 Thread Michael McCandless
On Wed, Mar 28, 2012 at 2:30 PM, Tim Eck wrote: > Excuse my ignorance of lucene internals, but is the problem any easier if > the requirement is just to allow the addition/removal of stored only fields > (as opposed to indexed)? It would substantially simplify the problem... but even this simplif

Re: JoinUtil.createJoinQuery in 3.6 ?

2012-03-29 Thread Martijn v Groningen
Some extra notes: 1) he implementation in 3.6 is about 3 times slower. I noticed this during some tests that I ran on my test data set. The 3.6 implementation is definitely usable. Also on larger indices. I believe the average query time was around 350 ms on an index containing 10M documents. 2) Th

Re: createJoinQuery use

2012-03-29 Thread Martijn v Groningen
It is only possible to join on one side at the same time. You mean something like this: Query fromQuery = JoinUtil.createJoinQuery("from", "to", actualQuery, indexSearcher); Query toQuery = JoinUtil.createJoinQuery("to", "from", actualQuery, indexSearcher); And then use a boolean query to combine

PyLucene Error Message

2012-03-29 Thread David Mosca
Hello, I am using Lucene version 3.4 through the Python extension (pylucene) in a multi-threaded script. When I launch the script I sometimes get a fatal error message (log attached) and sometimes not (I always launch the same script, exactly in the same way). When I get the error message I kee

Re: PyLucene Error Message

2012-03-29 Thread Greg Bowyer
I dont see any attached log, can you attach the log please. -- Greg On 29/03/2012 07:35, David Mosca wrote: Hello, I am using Lucene version 3.4 through the Python extension (pylucene) in a multi-threaded script. When I launch the script I sometimes get a fatal error message (log attached) a

RE: PyLucene Error Message

2012-03-29 Thread David Mosca
I have re-attached the log. Thanks, David -Original Message- From: Greg Bowyer [mailto:gbow...@fastmail.co.uk] Sent: 29 March 2012 16:55 To: java-user@lucene.apache.org Subject: Re: PyLucene Error Message I dont see any attached log, can you attach the log please. -- Greg On 29/03/20

Re: PyLucene Error Message

2012-03-29 Thread Greg Bowyer
Wow a VM / hotspot crash, are you using threads ? If you are make sure you have correctly attached the vm env to each new thread *after* you have initialised it on the main thread This is done with the following lucene.getVMEnv().attachCurrentThread() On 29/03/2012 09:10, David Mosca wrote:

RE: PyLucene Error Message

2012-03-29 Thread David Mosca
Yes I am using threads (threading module). I initialise the JVM like this: jvm = lucene.initVM(initialheap='8G', maxheap='8G') Then I use it as follows: for i in range(8): t = ThreadSearch(queue, jvm) t.setDaemon(True) t.start() with ThreadSearch built like this: class

Re: PyLucene Error Message

2012-03-29 Thread Greg Bowyer
Its a bit crap, but can you stick a time.sleep(0.5) just after the attach to see if that cures it On 29/03/2012 09:56, David Mosca wrote: Yes I am using threads (threading module). I initialise the JVM like this:jvm = lucene.initVM(initialheap='8G', maxheap='8G') Then I use it as follows:

RE: PyLucene Error Message

2012-03-29 Thread David Mosca
Unfortunately not. Interestingly enough I had already tried a sleep(1) when populating the queue, with no results. From: Greg Bowyer [mailto:gbow...@fastmail.co.uk] Sent: 29 March 2012 18:09 To: David Mosca Cc: java-user@lucene.apache.org Subject: Re: PyLucene Error Message Its a bit crap, but

Re: PyLucene Error Message

2012-03-29 Thread Greg Bowyer
There seems to be a bug in pylucene where it races if you do not read the value of the attach try adding a wait in your thread like so class ThreadSearch(threading.Thread): def __init__(self, queue, jvm): threading.Thread.__init__(self) self.queue = queue self.jvm =

Re: PyLucene Error Message

2012-03-29 Thread Greg Bowyer
I can seem to reproduce this, it seems like there might be a race condition here, its quite hard to reproduce I have filed a defect with pylucene (https://issues.apache.org/jira/browse/PYLUCENE-17). On 29/03/12 12:36, Greg Bowyer wrote: There seems to be a bug in pylucene where it races if y

Re: JoinUtil.createJoinQuery in 3.6 ?

2012-03-29 Thread kiwi clive
Thank you for the feedback, that is very useful :-) From: Martijn v Groningen To: java-user@lucene.apache.org Sent: Thursday, March 29, 2012 3:14 PM Subject: Re: JoinUtil.createJoinQuery in 3.6 ? Some extra notes: 1) he implementation in 3.6 is about 3 times

Can anyone guide me for this...!!!

2012-03-29 Thread neeraj shah
im using lucene2.9 and i need display search result like filename,filepath,textContent.. this text content highlight the matched string. When click on the search result ,it will go to the exact file and and the exact line. i tried to do it but my index size is very huge, i think im doing something