Lucene index corruption on HDFS

I am building my code using Lucene 4.7.1 and Hadoop 2.4.0 . Here is what I am trying to do Create Index 1. Build index in RAMDirectory based on data stored on HDFS . 2. Once built , copy the index onto HDFS. Search Index 1. Bring in the index stored on HDFS into RAMDirector

Re:答复：mmap confusion in lucene

Hi, Zhiiang It seems that the jvm is smart enough to ignore the unused code. Try the following code: RandomAccessFile raf = new RandomAccessFile(new File("/root/xx.txt"), "r"); FileChannel rafc = raf.getChannel(); ByteBuffer buff = rafc.map(Fi

答复：mmap confusion in lucene

Hi Uwe,     Thank you for always help. For my first testing I am clear of it, it is becuase the OS cache the whole file because of copying data to java heap and it does not free the page, then I see 800M used by cache in the end. But for my last two testings, the OS has freed all the previou

RE: Finding words not followed by other words

Thanks, Koji! This solves it. I had forgotten all about SpanNotQuery. -Michael -Original Message- From: Koji Sekiguchi [mailto:k...@r.email.ne.jp] Sent: Saturday, July 12, 2014 10:45 AM To: java-user@lucene.apache.org Subject: Re: Finding words not followed by other words Hi Michael, I

Re: DrillSideways accepting FacetCollector parameter

This is right, except, if "wordcount" is one of the drill-sideways dims, you should use the drill-sideways collector instead of the drill-down one? Ie, once the user actually drills down on wordcount (if you expose that in your UI), you should no longer use the drill-down collector when faceting t

Re: A question about FacetField constructor, Solved

On Sun, Jun 22, 2014 at 1:15 PM, Shai Erera wrote: Can you post an example which demonstrates the problem? It's also > interesting how you count the facets, eg do you use a TaxonomyFacets object > or something else? > > Have you looked at the facet demo code? It contains examples for using > hier

Re: Warm up IndexReader

Maybe just use oal.index.SimpleMergedSegmentWarmer as a start and iterate from there? Mike McCandless http://blog.mikemccandless.com On Mon, Jul 14, 2014 at 12:02 PM, Ian Lea wrote: > There's no magic to it - just build a query or six and fire them at > your newly opened reader. If you want t

Re: Warm up IndexReader

There's no magic to it - just build a query or six and fire them at your newly opened reader. If you want to put the effort in you could track recent queries and use them, or make sure you warm up searches on particular fields. Likewise, if you use Lucene's sorting and/or filters, it might be wor

Re: deleteDocument with NRT

Solr has the notion of "soft commit" and "hard commit". A soft commit means Solr will reopen a new searcher. A hard commit means a flush to disk. All the update/delete logics are in Lucene, Solr doesn't maintain deleted doc IDs. It does maintain its own caches though. On Jul 14, 2014, at 03:09 AM

RE: mmap confusion in lucene

This is very easy to explain: In the first part you copy the whole memory mapped stuff into a on-heap byte array. You allocate this byte array in total and you then do a copy (actually this is a standard libc copy call) of the whole file. To do this copy, the underlying OS will need to swap in

Re: deleteDocument with NRT

How Solr handles this scenario... Is it reopening reader after every delete OR it maintains the list of delete documents in cache? Regards Ganesh On 7/11/2014 4:00 AM, Tri Cao wrote: You need to reopen your searcher after deleting. From Java doc for SearcherManager: In addition you should

Re: mmap confusion in lucene

Your code isn't doing what you think it is doing. You need to ensure things aren't eliminated by the compiler. On Mon, Jul 14, 2014 at 5:57 AM, wangzhijiang999 wrote: > Hi everybody, I found a problem confused me when I tested the mmap > feature in lucene. I tested to read a file size of

mmap confusion in lucene

Hi everybody, I found a problem confused me when I tested the mmap feature in lucene. I tested to read a file size of 800M by mmap method like below:   RandomAccessFile raf = new RandomAccessFile(new File(path), "r"); FileChannel rafc = raf.getChannel();ByteBuffer buff = rafc.map(FileCha

Re: DrillSideways accepting FacetCollector parameter

Ok, i have overriden buildFacetsResult(...) method like below, Its just adding additional numericFacets to MultiFacets collection. protected Facets buildFacetsResult(final FacetsCollector drillDowns, final FacetsCollector[] drillSideways, final String[] drillSidewaysDims) { Facets dril