Thanks!
On Wed, 21 May 2025 at 23:35, Adrien Grand wrote:
> Hello Ashwini,
>
> MMapDirectory will often perform a bit faster. While NIOFSDirectory needs
> to first copy data from the buffer cache to heap arrays, MMapDirectory can
> read directly into the buffer cache.
>
&g
Hello Ashwini,
MMapDirectory will often perform a bit faster. While NIOFSDirectory needs
to first copy data from the buffer cache to heap arrays, MMapDirectory can
read directly into the buffer cache.
Lucene's benchmark suite allows comparing these two directories. I haven't
done s
As per Lucene documentation there is not much difference between these 2
implementations and users should use the Open() method and Lucene choses
the best implementation based on the system.
https://lucene.apache.org/core/9_7_0/core/org/apache/lucene/store/FSDirectory.html
Is there any perf impa
nfusings.
- 原邮件 -
发件人： Uwe Schindler
收件人： java-user@lucene.apache.org
抄送：
发送日期： 2013年8月7日, 星期三, 1:17 下午
主题: RE: 回复：RE: why lucene not use DirectByteBuffer in NIOFSDirectory
Hi,
1)
ByteBuffer bb=ByteBuffer.wrap(new byte[len]); //bb is HeapByteBuffer
channel.read
ijiang...@aliyun.com [mailto:wangzhijiang...@aliyun.com]
Sent: Wednesday, August 07, 2013 11:36 AM
To: java-user
Subject: 回复：RE: why lucene not use DirectByteBuffer in NIOFSDirectory
Hi Uwe:
Thank you for your detail explaination and I learnt a lot from your
message.
First, the direct
---发件人：Uwe
Schindler发送日期：2013年7月31日
18:18收件人：java-user@lucene.apache.org;wangzhijiang...@yahoo.com.cn;主 题：RE: why
lucene not use DirectByteBuffer in NIOFSDirectory Hi,There is a
misunderstanding: Just by allocating a direct buffer, there is still no
difference to a heap buffer in the workflo
es as used by Lucene.
- NIOFSDirectory with direct buffers: Needs to copy data from FS cache to
direct buffer memory (outside heap). Access times slower to direct buffers than
to heap buffers -> 2 times bad
- NIOFSDirectory with heap buffers: Needs to copy data from FS cache to heap.
Access t
I read this article "Use Lucene's MMapDirectory on 64bit platforms, please!"
and it said the MMapDirectory is better than other Directory because it will
void copy data between file system cache and java heap.
I checked the source code of NIOFSDirectory, and in new Buffer m
"NIOFSDirectory.open(File)"
only opens a NIOFSDirectory and has the same semantics as "new
NIOFSDirectory(File)", which is what I would expect.
Obviously there is the Java weirdness that using FSDirectory nioFSDirectory =
new NIOFSDirectory();, then nioFSDirectory.open() will call FS
Hi,
> Thanks for that, I had not realised that the open method was simply
> inherited from the FSDirectory, I should have read the docs.
>
> However, it makes no sense to me that a call to NIOFSDirectory.open() can
> end up opening a MMapDirectory, and in general that a call to one subclass
> can
Thanks for that, I had not realised that the open method was simply inherited
from the FSDirectory, I should have read the docs.
However, it makes no sense to me that a call to NIOFSDirectory.open() can end
up opening a MMapDirectory, and in general that a call to one subclass can end
up openin
If you want a specific directory implementation, instantiate it directly (new
NIOFSDirectory)? If you use FSDirectory.open() it will use the autodetected one
depending on OS and bitness.
About configuring your server from MMAP in a correct way, read
http://blog.thetaphi.de/2012/07/use-lucenes
Hi all,
I run my code on a cluster where I have to preset resource limits and therefore
the processes have limited virtual memory causing OOME when using MMapDirectory
on large indexes.
This means I explicitly use NIOFSDirectory (i.e. Directory indexDirectory =
NIOFSDirectory.open(indexFile
for IndexWriter to work. This was
>>> essentially caused by the embedding application making heavy use of JVM's
>>> direct memory buffers and not leaving enough headroom for NIOFSDirectory to
>>> operate. So what are the approximate guidelines, if any, in terms of JVM
&g
of JVM's
>> direct memory buffers and not leaving enough headroom for NIOFSDirectory to
>> operate. So what are the approximate guidelines, if any, in terms of JVM
>> configuration for this choice of Directory to operate safely? Basically,
>> what I am looking for is a rou
rect memory available for IndexWriter to work. This was essentially
> caused by the embedding application making heavy use of JVM's direct memory
> buffers and not leaving enough headroom for NIOFSDirectory to operate. So
> what are the approximate guidelines, if any, in terms of JVM configura
Hello,
I have recently run into the situation when there was not a sufficient
amount of direct memory available for IndexWriter to work. This was
essentially caused by the embedding application making heavy use of JVM's
direct memory buffers and not leaving enough headroom for NIOFSDirecto
normally(without
> interrupt). I also tried to put some Thread.sleep() on end of run(), but
> still it didn't help.
> Finally i solved the problem by making mythread reusable. This is not
> perfect solution for me, but for now it's w
RLClassLoader
Conclusion is that you can't access NIOFSDirectory in thread which will
end normally or by interrupt. It happened in 3.3.0.
Thanks
--
Regards
-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For a
hetaphi.de
> -Original Message-
> From: dcli...@gmail.com [mailto:dcli...@gmail.com] On Behalf Of Dan
> Sent: Sunday, December 06, 2009 9:30 PM
> To: java-user@lucene.apache.org
> Subject: NIOFSDirectory AssertionError in sun.nio.ch.NativeThreadSet
>
> I've been run
I've been running some tests with Lucene 2.9.1 on a Linux box with a
Sun JVM and getting a sun.nio.ch.NateiveThreadSet Assertion error (see
below for stacktrace).
Does anyone know what this error means? Any suggestions for a workaround?
We used the following to open the index.
FSDirectory
> Use this constructor to create an instance of NIODirectory:
>>
>> /** Create a new NIOFSDirectory for the named location.
>> *
>> * @param path the path of the directory
>> * @param lockFactory the lock factory to use, or null for the default.
>> * @t
Understood. Thanks! :-)
-glen
2008/12/4 John Wang <[EMAIL PROTECTED]>:
> NIOFSDirectory.getDirectory simple calls the static method on the parent
> class: FSDirectory.getDirectory.
> Which returns an instance of FSDirectory.
>
> IMO: NIOFSDirectory solves concurrent read pr
NIOFSDirectory.getDirectory simple calls the static method on the parent
class: FSDirectory.getDirectory.
Which returns an instance of FSDirectory.
IMO: NIOFSDirectory solves concurrent read problems, generally you don't
want concurrent writes.
-John
On Thu, Dec 4, 2008 at 2:44 PM, Glen N
Am I missing something here?
Why not use:
IndexWriter writer = new IndexWriter(NIOFSDirectory.getDirectory(new
File(filename), analyzer, true);
Another question: is NIOFSDirectory to be used with IndexWriter? If
no, could someone explain?
thanks,
-glen
2008
Thanks!
-John
On Thu, Dec 4, 2008 at 2:16 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote:
> Details in the bug:
> https://issues.apache.org/jira/browse/LUCENE-1451
>
> Use this constructor to create an instance of NIODirectory:
>
> /** Create a new NIOFSDirector
gt; a map FSDirectory keeps statically. Should subclasses of FSDirectory be
> modifying the map?
> This is not a question about how to subclass or customize FSDirectory.
> This
> is more on how to use NIOFSDirectory class. I am hoping for a simply
> answer,
> is what I am doing (setting t
Details in the bug:
https://issues.apache.org/jira/browse/LUCENE-1451
Use this constructor to create an instance of NIODirectory:
/** Create a new NIOFSDirectory for the named location.
*
* @param path the path of the directory
* @param lockFactory the lock factory to use, or null for
That does not help. The File/path is not stored with the instance. It is in
a map FSDirectory keeps statically. Should subclasses of FSDirectory be
modifying the map?
This is not a question about how to subclass or customize FSDirectory. This
is more on how to use NIOFSDirectory class. I am hoping
>>> we are running, performance improved by a factor of 5 (to be conservative).
>>>
>>> Great job, this is awesome, a simple change and made a huge difference.
>>>
>>>To get NIOFSDirectory installed, I didn't find any documentati
version 2.4, sorry for not clarifying.
Yonik, pardon my ignorance. I still don't get it. When instantiating
NIOFSDIrectory, how would I specify the path? I see only the empty
constructor.
With FSDirectory, you use the factory: getDirectory(File)
-John
On Thu, Dec 4, 2008 at 1:26 PM,
contention on FSDIrectory is gone, and for the set of queries
>> we are running, performance improved by a factor of 5 (to be conservative).
>>
>>Great job, this is awesome, a simple change and made a huge difference.
>>
>>To get NIOFSDirectory installed, I di
vative).
>
>Great job, this is awesome, a simple change and made a huge difference.
>
>To get NIOFSDirectory installed, I didn't find any documentation
> (doesn't mean there aren't any), after reading the code, I resorted to:
>
> static
> {
NIOFSDirectory installed, I didn't find any documentation
(doesn't mean there aren't any), after reading the code, I resorted to:
static
{
System.setProperty("org.apache.lucene.FSDirectory.class",NIOFSDirectory.class.getName());
}
I am sure this is not the intended
34 matches
Mail list logo
