I think so.
But ideally I would like:
- to add the property to set the mergerscheduler and expose that
- automatically force the setting if the IW is used by Infinispan
(not sure how easy that is).
On 14 août 09, at 13:35, Łukasz Moreń wrote:
Yes, using SerialMergeScheduler helped. Thanks!
Yes, using SerialMergeScheduler helped. Thanks! Restriction on merge
scheduler should be set in IndexWriterSetting enum, similarly to i.e.
max_buffered_delete_terms or max_buffered_docs properties?
2009/8/14 Emmanuel Bernard
> OK so let's try something.Lukasz, can you try and use
> the SerialMer
Yes let's keep it simple for know, even if I think Manik's suggestion
should work: if the IndexWriter will do reads it can use a
"Future-like" pattern to make the other thread return values
maintaining order of transformations applied to the index).
The IndexWriter must be able to maintain a cohere
OK so let's try something.
Lukasz, can you try and use the SerialMergeScheduler policy on the
index writer and see what is going on.
It will make indexing slower for Infinispan but it seems we can't do
much better in the short time.
If that works then we will put some restriction in place wh
On 14 août 09, at 06:32, Manik Surtani wrote:
On 14 Aug 2009, at 11:23, Łukasz Moreń wrote:
Expensive is replication to all Infinispan nodes. IndexWriter
creates segment files, merge it to one compound segment, delete
already useless descriptor files - many files that must be
replicated
On 14 Aug 2009, at 11:23, Łukasz Moreń wrote:
Expensive is replication to all Infinispan nodes. IndexWriter
creates segment files, merge it to one compound segment, delete
already useless descriptor files - many files that must be
replicated. Some of them even don't need to be replicated b
Expensive is replication to all Infinispan nodes. IndexWriter creates
segment files, merge it to one compound segment, delete already useless
descriptor files - many files that must be replicated. Some of them even
don't need to be replicated because they are inserted into directory at the
begin of
On 14 Aug 2009, at 10:17, Łukasz Moreń wrote:
Yes, but i.e. FSDirectory flushes changes if any file descriptor is
created/updated - can be many in one IndexWriter life.
In infinispan case implementation, I want to commit changes only
when IndexWriter is closing - batch all modifications.
If
Yes, but i.e. FSDirectory flushes changes if any file descriptor is
created/updated - can be many in one IndexWriter life.
In infinispan case implementation, I want to commit changes only when
IndexWriter is closing - batch all modifications.
If I switch to transaction per descriptor modification -
On 14 Aug 2009, at 03:00, Emmanuel Bernard wrote:
> Yes but Lukasz do see data disappearing (see initial email). So there
> is a problem somewhere?
Regarding the data "disappearing", this seems to be an Infinispan JTA
bug when dealing with certain TMs. Lukasz has reported this [1] and
we ha
On 13 Aug 2009, at 19:11, Emmanuel Bernard wrote:
>
> Infini guys, do you support transactional operation spanning several
> concurrent threads?
The *same* tx on different threads? That's disallowed by the JTA spec.
Cheers
--
Manik Surtani
ma...@jboss.org
Lead, Infinispan
Lead, JBoss Cache
htt
Yes but Lukasz do see data disappearing (see initial email). So there
is a problem somewhere?
Lukasz, can you maybe share the code and test with us so someone can
have a closer look with you?
Emmanuel
On 13 août 09, at 18:56, Sanne Grinovero wrote:
> I am not an expert on this part of Lucene
I am not an expert on this part of Lucene, but it looks like to me
that the IndexWriter is the "driver/coordinator", and it's decisions
are affected by a pluggable MergeScheduler; they do stuff on the
internal buffers of the IndexWriter (dequeue the pending segments to
be written to the index), but
Yes, right, MergeSchedulers.
2009/8/14 Sanne Grinovero
> what are these "other" threads? Are you speaking about the MergeSchedulers?
>
> 2009/8/13 Łukasz Moreń :
> > IndexWriter processes index update and delegates some job to other
> > threads and waits when they finish. These "other" threads w
what are these "other" threads? Are you speaking about the MergeSchedulers?
2009/8/13 Łukasz Moreń :
> IndexWriter processes index update and delegates some job to other
> threads and waits when they finish. These "other" threads works on
> data modified
> in IndexWriter transaction. So I think if
IndexWriter processes index update and delegates some job to other
threads and waits when they finish. These "other" threads works on
data modified
in IndexWriter transaction. So I think if I use transaction per
thread, "others" would not see data modified by IndexWriter until
commit.
2009/8/13, E
Ah I thought it was using multiple threads because of your mass
indexing. I did not know some threads were span specifically for the
Infinispan directory.
On 13 août 09, at 17:34, Sanne Grinovero wrote:
> Hi Łukasz,
> what is your usage of these threads? did you consider using one
> transacti
It's a bit complex but the Lucene IndexWriter works better if you can
inject what has to be indexed in a multi threaded way because
analyzing the data before writing it to the index can be heavy.
The IW does rely on what is called a Directory (implemented by
Infinispan in this case).
On 13 a
Hi Łukasz,
what is your usage of these threads? did you consider using one
transaction per thread?
Sanne
2009/8/13 Łukasz Moreń :
> Newly created threads were not associated with any transaction, so I
> suppose it was a problem. Sharing transaction between threads seems to
> be a good solution.
>
Newly created threads were not associated with any transaction, so I
suppose it was a problem. Sharing transaction between threads seems to
be a good solution.
Thanks for help!
2009/8/13, Jason T. Greene :
> Correct. Also there could be read races as well, so if you are going to
> share a tx betwe
Modifying a transaction means applying muations (like SQL INSERT /
UPDATE / DELETE) to the transactional resource?
On 13 août 09, at 15:07, Jason T. Greene wrote:
> When using transactions, the context is bound to the transaction,
> and you can move a transaction between threads. However, you
21 matches
Mail list logo
