bq. We switched to ByteBuffersDirectory with 7.5, but
I actually didn't see much performance improvements or savings in memory.
Once the indexes are built I don't think there will be much of a
difference. The core problem with RAMDirectory was related to
synchronizations during merges/ file manipu
I doubt using Redis as directory-storage will be very good. I'd expect
it to have much more latency for reads and writes compared to any of
lucene's own directories. And Lucene probably won't like it if another
Lucene-instance changes that database.
It may be interesting as a result-level cache th
Hi Joe,
Elasticsearch used to have an option to run with RAMDirectory but it
was removed long ago:
https://github.com/elastic/elasticsearch/pull/8536.
The recommended approach is to use MMapDirectory and to give
sufficient memory to the filesystem cache so that the most important
data-structures
Thanks Dawid, glad I asked!
Am Dienstag, den 25.09.2018, 10:46 +0200 schrieb Dawid Weiss:
> Use MMapDirectory on a temporary location, Matthias. If you really
> need in-memory indexes, a new Directory implementation is coming
> (RAMDirectory will be deprecated, then removed), but the difference
>
Use MMapDirectory on a temporary location, Matthias. If you really
need in-memory indexes, a new Directory implementation is coming
(RAMDirectory will be deprecated, then removed), but the difference
compared to MMapDirectory is typically not worth the hassle. See this
issue for more discussion.
h
On Tue, May 21, 2013 at 3:12 PM, Konstantyn Smirnov wrote:
> I want to refresh the topic a bit.
>
> Using the Lucene 4.3.0, I could'n find a method like expungeDeletes() in
> the
> IW anymore.
http://lucene.apache.org/core/4_3_0/core/org/apache/lucene/index/IndexWriter.html#forceMergeDeletes()
I want to refresh the topic a bit.
Using the Lucene 4.3.0, I could'n find a method like expungeDeletes() in the
IW anymore. I rely on lucence's MergePolicies to do the optimization, but I
need to keep the metadata up-to-date, docFreqs and termFreqs to name a few.
The only way to accomplish that w
Sent: Wednesday, July 11, 2012 10:07 AM
To: java-user@lucene.apache.org
Subject: Re: RAMDirectory and expungeDeletes()/optimize()
What I meant was your original email says "My code looks like",
followed by blank lines, and then "Doesn't it conflict with the
JavaDoc saying:", follow
What I meant was your original email says "My code looks like",
followed by blank lines, and then "Doesn't it conflict with the
JavaDoc saying:", followed by blank lines. Ie we can't see your code.
However, when I look at your email here at
http://lucene.472066.n3.nabble.com/RAMDirectory-and-expun
JavaDoc comes from here
http://lucene.apache.org/core/3_6_0/api/all/org/apache/lucene/index/IndexWriter.html#expungeDeletes()
other blanks are here because it's groovy :) Or what did you mean exactly?
--
View this message in context:
http://lucene.472066.n3.nabble.com/RAMDirectory-and-expungeDel
There are blanks at the important places (your code, and which
JavaDoc) in your email!
Mike McCandless
http://blog.mikemccandless.com
On Wed, Jul 11, 2012 at 6:18 AM, Konstantyn Smirnov wrote:
> Hi all
>
> in my app (Lucene 3.5.0 powered) I index the documents (not too many, say up
> to 100k) u
t; >> This is managed by your operating system. In general OS kernels like
>> Linux
>> >> or Windows use all free memory to cache disk accesses.
>> >>
>> >> -
>> >> Uwe Schindler
>> >> H.-H.-Meier-Allee 63, D-28213 Bremen
&
like
> Linux
> >> or Windows use all free memory to cache disk accesses.
> >>
> >> -
> >> Uwe Schindler
> >> H.-H.-Meier-Allee 63, D-28213 Bremen
> >> http://www.thetaphi.de
> >> eMail: u...@thetaphi.de
> >>
> >
;> H.-H.-Meier-Allee 63, D-28213 Bremen
> >> http://www.thetaphi.de
> >> eMail: u...@thetaphi.de
> >>
> >>
> >> > -Original Message-
> >> > From: Cheng [mailto:zhoucheng2...@gmail.com]
> >> > Sent: Monda
ee memory to cache disk accesses.
>>
>> -
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: u...@thetaphi.de
>>
>>
>> > -Original Message-
>> > From: Cheng [mailto:zhouc
hetaphi.de
> eMail: u...@thetaphi.de
>
>
> > -Original Message-
> > From: Cheng [mailto:zhoucheng2...@gmail.com]
> > Sent: Monday, June 04, 2012 6:10 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: RAMDirectory unexpectedly slows
> >
Ramdirectory is no longer an interesting technique for this. It makes
garbage collection do a lot of work. With memory-mapped directory the
data is cached by the OS instead of Java, and OS is very good at this.
TieredMergePolicy is much smarter about time spent merging segments.
Lucene In Action
cheng2...@gmail.com]
> Sent: Monday, June 04, 2012 6:10 PM
> To: java-user@lucene.apache.org
> Subject: Re: RAMDirectory unexpectedly slows
>
> Can I control the size of ram given to either MMapDirectory or
> ByteBufferDirectory?
>
> On Mon, Jun 4, 2012 at 11:42 PM, U
Can I control the size of ram given to either MMapDirectory or
ByteBufferDirectory?
On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler wrote:
> Hi,
>
> If you are using MMapDirectory or this ByteBufferDirectory (which is
> similar to the first) the used RAM is outside JVM heap, it is in the FS
> cac
Hi,
If you are using MMapDirectory or this ByteBufferDirectory (which is similar to
the first) the used RAM is outside JVM heap, it is in the FS cache of the OS
kernel. Giving too much memory to the JVM penalizes the OS cache, so give only
as much as the App needs. Lucene and the OS kernel will
> What about the ByteBufferDirectory? Can this specific directory utilize the
> 2GB memory I grant to the app?
BBD places the byte objects outside of the heap, so increasing the
heap size is only going to rob the system IO cache. With Lucene the
heap is only used for field caches and the terms di
Please shed more insight into the difference between JVM heap size and the
memory size used by Lucene.
What I am getting at is that no matter however much ram I give my apps,
Lucene can't utilize it. Is that right?
What about the ByteBufferDirectory? Can this specific directory utilize the
2GB me
If you want the index to be stored completely in RAM, there is the
ByteBuffer directory [1]. Though I do not see the point in putting an
index in RAM, it will be cached in RAM regardless in the OS system IO
cache.
1.
https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/ap
My indexes are 500MB+. So it seems like that RAMDirectory is not good for
that big a size.
My challenge, on the other side, is that I need to update the indexes very
frequently. So, do you think MMapDirectory is the solution?
Thanks.
On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky wrote:
> Fro
From the javadoc for RAMDirectory:
"Warning: This class is not intended to work with huge indexes. Everything
beyond several hundred megabytes will waste resources (GC cycles), because
it uses an internal buffer size of 1024 bytes, producing millions of
byte[1024] arrays. This class is optimi
dex stuff is more what you want?
>
> Lance
>
> On Tue, Jun 7, 2011 at 2:52 AM, zhoucheng2008 wrote:
>> Makes sense. Thanks
>>
>> -Original Message-
>> From: Toke Eskildsen [mailto:t...@statsbiblioteket.dk]
>> Sent: Tuesday, June 07, 2011 4:28 PM
&g
> To: java-user@lucene.apache.org
> Subject: Re: RAMDirectory doesn't win over FSDirectory all the time, why?
>
> On Mon, 2011-06-06 at 15:29 +0200, zhoucheng2008 wrote:
>> I read the lucene in action book and just tested the
>> FSversusRAMDirectoryTest.java with the follo
Makes sense. Thanks
-Original Message-
From: Toke Eskildsen [mailto:t...@statsbiblioteket.dk]
Sent: Tuesday, June 07, 2011 4:28 PM
To: java-user@lucene.apache.org
Subject: Re: RAMDirectory doesn't win over FSDirectory all the time, why?
On Mon, 2011-06-06 at 15:29 +0200, zhouchen
On Mon, 2011-06-06 at 15:29 +0200, zhoucheng2008 wrote:
> I read the lucene in action book and just tested the
> FSversusRAMDirectoryTest.java with the following uncommented:
> [...]Here is the output:
>
> RAMDirectory Time: 805 ms
>
> FSDirectory Time : 728 ms
This is the code, right?
http://ja
-
From: Uwe Schindler [mailto:u...@thetaphi.de]
Sent: Tuesday, June 07, 2011 12:04 AM
To: java-user@lucene.apache.org
Subject: RE: RAMDirectory doesn't win over FSDirectory all the time, why?
Hi,
It depends on the Lucene version, so if the test uses latest Lucene on a
64bit OS, it ma
.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de
> -Original Message-
> From: Michael McCandless [mailto:luc...@mikemccandless.com]
> Sent: Monday, June 06, 2011 5:58 PM
> To: java-user@lucene.apache.org
> Subject: Re: RAMDirectory doesn&#
This test is very old (from the 1st edition of the book but removed
from the 2nd).
Modern OS's cache newly written files in RAM, and this test doesn't
write very large files (I think?), so the test is really testing an
OS's IO cache vs Lucene's RAM Dir.
That said, I'm not sure why RAMDir would be
Hi
> currently I'm working on a students project including a lucene index. It's a
> kind of search engine working on extracted data from the web.
> It's working pretty good, but now I'm curious about two things:
> - when is it usefull to use a RAMDirectory? Is there a number of documents
> in an i
Thanks a lot!
of course performance :)
I used 2.41 and now with 2.9.1 it works:)
Michael McCandless pisze:
Are you sure you even need to do this (manage separate RAMDir)? Ie
what's driving this (performance)?
IndexWriter already has its own RAM buffer for newly added docs / deletions.
You ca
Are you sure you even need to do this (manage separate RAMDir)? Ie
what's driving this (performance)?
IndexWriter already has its own RAM buffer for newly added docs / deletions.
You can use the new near-realtime API (IndexWriter.getReader) in 2.9
to reopen your IndexReader, for searching the fu
Hi Rafal,
If what I understand about your implementation is correct, you could try a
parallelmultisearcher
http://lucene.apache.org/java/2_9_1/api/core/org/apache/lucene/search/ParallelMultiSearcher.html
--
Anshum Gupta
Naukri Labs!
http://ai-cafe.blogspot.com
The facts expressed here belong to
Created a RAMDirectory like directory class that uses
ByteArrayRandomAccessIO from http://reader.imagero.com/uio/ to allow
concurrent random file access.
On Thu, Jun 19, 2008 at 3:33 PM, Jason Rutherglen <
[EMAIL PROTECTED]> wrote:
> Looks like it cannot be used for a log system that needs concur
Looks like it cannot be used for a log system that needs concurrent read
write access to a file. Back to RandomAccessFile which will have buffering
issues, any experience with http://reader.imagero.com/uio/
On Thu, Jun 19, 2008 at 3:20 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote:
> createOutput(
createOutput() creates a new file, overwriting the old one.
If you open the IndexInput before you call createOutput() for the 2nd
time, you should see the file.
And you definitely shouldn't have more than one IndexOutput open on
the same file (but that's not your problem here).
-Yonik
On Thu, Ju
Here's code that reproduces it.
public void testMain() throws IOException {
RAMDirectory ramDirectory = new RAMDirectory();
IndexOutput output = ramDirectory.createOutput("test");
byte[] bytes = "hello world".getBytes("UTF-8");
output.writeBytes(bytes, bytes.length);
output.flu
Yes. Also close. But then reopen the IndexOutput again later, then open
the IndexInput. I'm not sure if this is the recomended usage of these
APIs. It seems everywhere else in the Lucene code base only one is open at
a time.
On Thu, Jun 19, 2008 at 12:50 PM, Yonik Seeley <[EMAIL PROTECTED]> wr
Did you try calling flush() on the IndexOutput before opening the IndexInput?
-Yonik
On Thu, Jun 19, 2008 at 12:13 PM, Jason Rutherglen
<[EMAIL PROTECTED]> wrote:
> Seeing strange behavior with RAMDirectory. Is a file designed to supported
> IndexOutput being open concurrently with IndexInput?
There is a constructor in the RAMDirectory that already does that.
http://lucene.apache.org/java/2_2_0/api/org/apache/lucene/store/RAMDirectory.html
I don't think that worth modify the internal Lucene's code to achieve
a extra bit of performance...
What would you do on next version? Modify it agai
RAMDirectory has a constructor that takes in another Directory and
loads it into memory. No Serialization necessary. Just index to a
FSDirectory using Lucene's normal indexing methods (it takes care of
buffering them internally) and then load the FSDirectory into a
RAMDirectory. Have a l
You can serialize this object RAMDirectory em disk. When start the
application , it read the file .ser and load the object in memory.
The time of load of file .ser is much fast.
You need change any classes of Lucece: Add the "implements Serialzable" in
any classes.
On Nov 27, 2007 4:28 AM, Chhab
> one can improve search performance by using a RAMDirectory created
from an underlying FSDirectory using one of the parameterised
constructors. Is this correct?
Absolutly
> Will a FSDirectory not automatically load the index into memory
provided enough RAM is available?
Not all index files are
Unless you have hard evidence that FSDirectories are too slow,
you shouldn't IMO. The only place I've found RAMDirectories
useful (so far) is in unit tests where I'm creating a very small
in-memory index, running a few tests, and throwing it away.
My first question for this is "why are you using
I guess it depends upon your goal. If you're asking what the difference
between writing to a RAMDirectory *then* flushing to an FSDIrectory,
I don't believer there's much, if any. As I remember (and my memory
isn't always...er...accurate), there's been discussion on this thread
by those who know t
lto:[EMAIL PROTECTED]
Sent: 28 February 2007 16:29
To: java-user@lucene.apache.org
Subject: Re: RamDirectory vs IndexWriter
Le Mercredi 28 Février 2007 16:19, WATHELET Thomas a écrit :
> I don't really understand the difference between using the ramDirectory
> and using IndexWriter.
&
Le Mercredi 28 Février 2007 16:19, WATHELET Thomas a écrit :
> I don't really understand the difference between using the ramDirectory
> and using IndexWriter.
>
> What's the difference between using ramDirectory instead of using
> IndexWriter with those properties set to:
> setMergeFactor(1000);se
: I'm using RAMDirectory
: If the number of documents to index is less than the maxBufferedDocs
: properties nothing is write into my index.
: ex: RAMDirectory ramDir =new RAMDirectory ();
: this.indexWriter.addIndexes(new Directory[] { ramDir });
: ramWriter.close();
:
I'm using now lucene-core-2.0.0.jar(26-05-2006) and I still have the
trouble...
-Original Message-
From: Otis Gospodnetic [mailto:[EMAIL PROTECTED]
Sent: 30 January 2007 17:18
To: java-user@lucene.apache.org
Subject: Re: RAMDirectory 2
Sounds vaguely familiar... and I think thi
Sounds vaguely familiar... and I think this was fixed a while back. Running
HEAD or at least 2.0.0?
Otis
- Original Message
From: WATHELET Thomas <[EMAIL PROTECTED]>
To: java-user@lucene.apache.org
Sent: Tuesday, January 30, 2007 11:22:52 PM
Subject: RAMDirectory 2
P.S. At one point
Ok. I reverted back to the version without a public clear() method.
Wolfgang.
On Nov 27, 2006, at 12:17 PM, jm wrote:
yes that would be ok for my, as long as I can reuse my child analyzer.
On 11/27/06, Wolfgang Hoschek <[EMAIL PROTECTED]> wrote:
On Nov 27, 2006, at 9:57 AM, jm wrote:
> On 1
yes that would be ok for my, as long as I can reuse my child analyzer.
On 11/27/06, Wolfgang Hoschek <[EMAIL PROTECTED]> wrote:
On Nov 27, 2006, at 9:57 AM, jm wrote:
> On 11/27/06, Wolfgang Hoschek <[EMAIL PROTECTED]> wrote:
>>
>> On Nov 26, 2006, at 8:57 AM, jm wrote:
>>
>> > I tested this.
On Nov 27, 2006, at 9:57 AM, jm wrote:
On 11/27/06, Wolfgang Hoschek <[EMAIL PROTECTED]> wrote:
On Nov 26, 2006, at 8:57 AM, jm wrote:
> I tested this. I use a single static analyzer for all my documents,
> and the caching analyzer was not working properly. I had to add a
> method to clear t
On 11/27/06, Wolfgang Hoschek <[EMAIL PROTECTED]> wrote:
On Nov 26, 2006, at 8:57 AM, jm wrote:
> I tested this. I use a single static analyzer for all my documents,
> and the caching analyzer was not working properly. I had to add a
> method to clear the cache each time a new document was to b
On Nov 26, 2006, at 8:57 AM, jm wrote:
I tested this. I use a single static analyzer for all my documents,
and the caching analyzer was not working properly. I had to add a
method to clear the cache each time a new document was to be indexed,
and then it worked as expected. I have never looked
I tested this. I use a single static analyzer for all my documents,
and the caching analyzer was not working properly. I had to add a
method to clear the cache each time a new document was to be indexed,
and then it worked as expected. I have never looked into lucenes inner
working so I am not sur
thanks. I'll try to get this working and see wether there is a perf
difference during the weekend.
On 11/23/06, Wolfgang Hoschek <[EMAIL PROTECTED]> wrote:
Out of interest, I've checked an implementation of something like
this into AnalyzerUtil SVN trunk:
/**
* Returns an analyzer wrappe
Out of interest, I've checked an implementation of something like
this into AnalyzerUtil SVN trunk:
/**
* Returns an analyzer wrapper that caches all tokens generated by
the underlying child analyzer's
* token stream, and delivers those cached tokens on subsequent
calls to
* tok
I've never tried it, but I guess you could write an Analyzer and
TokenFilter that no only feeds into IndexWriter on
IndexWriter.addDocument(), but as a sneaky side effect also
simultaneously saves its tokens into a list so that you could later
turn that list into another TokenStream to be a
checking one last thing, just in case...
as I mentioned, I have previously indexed the same document in another
index (for another purpose), as I am going to use the same analyzer,
would it be possible to avoid analyzing the doc again?
I see IndexWriter.addDocument() returns void, so it does not
Ok, thanks, I'll give MemoryIndex a go, and if that is not good enoguh
I will explore the other options then.
On 11/21/06, Wolfgang Hoschek <[EMAIL PROTECTED]> wrote:
On Nov 21, 2006, at 7:43 AM, jm wrote:
> Hi,
>
> I have to decide between using a RAMDirectory and MemoryIndex, but
> not sure
On Nov 21, 2006, at 7:43 AM, jm wrote:
Hi,
I have to decide between using a RAMDirectory and MemoryIndex, but
not sure what approach will work better...
I have to run many items (tens of thousands) against some queries (100
at most), but I have to do it one item at a time. And I already have
21 nov 2006 kl. 16.43 skrev jm:
Any thoughts?
You can also try InstantiatedIndex, similair in speed and design with
a MemoryIndex, but can handle multiple documents, IndexReader,
IndexWriter, IndexModifier et.c. just like any Directory
implementation. It requires a minor patch to the Lu
On Nov 21, 2006, at 12:38 PM, jm wrote:
Ok, thanks, I'll give MemoryIndex a go, and if that is not good enoguh
I will explore the other options then.
To get started you can use something like this:
for each document D:
MemoryIndex index = createMemoryIndex(D, ...)
for each query Q:
on this design would be
appreciated.
Kind regards
Peter Gelderbloem
-Original Message-
From: Peter Gelderbloem [mailto:[EMAIL PROTECTED]
Sent: 11 November 2005 15:07
To: java-user@lucene.apache.org
Subject: Re: RAMDirectory and Hibernate
Martijn,
Sorry for the late reply.
I've be
ECTED]
Sent: 29 October 2005 17:27
To: Peter Gelderbloem
Cc: java-user@lucene.apache.org
Subject: Re: RAMDirectory and Hibernate
Peter,
Have you had any success yet on this particular approach?
regards,
Martijn
Peter Gelderbloem wrote:
>Hi,
>I want to map RAMDirectory to a database via Hibe
Cause the RAMDirectory is not serializable, it's hard to send a index
to a remote computer. I think it's kind of tricky, but it would work.
1. Create a fresh new IndexWriter(let's name it toTransfer) with
temporary FSDirectory, /usr/tmp/some/directory for example.
2. Invoke the toTransfer.addIndex
Hello Otis,
Thanks alot !.
I will try.
Thanks again,
Youngho
- Original Message -
From: "Otis Gospodnetic" <[EMAIL PROTECTED]>
To: ; "Youngho Cho" <[EMAIL PROTECTED]>
Sent: Saturday, September 10, 2005 10:59 AM
Subject: Re: RAMDirectory add inde
Save it to disk using FSDirectory, zip the index directory, and copy it
to the remote host over the socket. There is no built-in Lucene-way of
doing this. You can also simply scp the index directory or use
whatever other file transfer protocol suits you.
Otis
--- Youngho Cho <[EMAIL PROTECTED]>
Hi Otis,
Thanks for your reply.
Youngho.
- Original Message -
From: "Otis Gospodnetic" <[EMAIL PROTECTED]>
To: ; "Youngho Cho" <[EMAIL PROTECTED]>
Sent: Saturday, September 10, 2005 12:33 AM
Subject: Re: RAMDirectory and Serializable
> Hi Yo
Hi Youngho,
--- Youngho Cho <[EMAIL PROTECTED]> wrote:
> I would like to add RAMDirectory to another machines' FSDirectory
> Can I treat a RAMDirectory as Serializable ?
>
> I found RAMDirectory and Serializable discussion a long time ago in
> the dev list. But I cound find the answer.
I recall
74 matches
Mail list logo
