2008/12/16 Hardy Ferentschik <hibern...@ferentschik.de>: > Hi Sanne, > > On Mon, 15 Dec 2008 20:15:52 +0100, Sanne Grinovero > <sanne.grinov...@gmail.com> wrote: > >> some more steps towards mass indexing > > What exactly is actually your definition of "mass indexing"? The use of > FullTextSession.index() > to (re-)build your index?
I have split the problem it two: first part is to just speedup the whole process changing the backend only, so people will get the new benefits using the recommended procedure described in the book and in the reference docs: so yes I mean using "FullTextSession.index()". The second part is to add a new API which wraps the complexity of the object loading, and uses FullTextSession.index() under the covers (I hope to). It is not really clear at the moment which arguments this new API will need, as people may want to finetune the process in several ways, but it will look like the same as the recommended approach, but using several threads. > >> do you agree I'll drop the capability to use an IndexReader to make >> changes to the index? >> This implies I'll simplify the backend by removing all methods working >> on an IndexReader (they are not needed anymore), >> and is required to reuse the IndexWriter as next improvement. > > I assume you are talking about removing performWork(LuceneWork work, > IndexReader reader) > from the LuceneWorkDelegate interface. I think that makes sense. I think we > still agree > that it is a good idea to only use the IndexWriter to apply changes. Nice I'll start with this as it really helps minimizing the complexity of the backend. > >> Also we'll need to change the FSMasterDirectoryProvider to use >> Lucene's snapshotting feature instead of the DirectoryProvider lock (is >> HSEARCH-152). >> Another solution would be to replace the lock-holding with a signal >> like "please release it to me for a moment" >> or even delegate the task to the thread owning the IndexWriter. > > Not sure about this one. I haven't looked into this snapshot feature yet. This feature makes it possible to make consistent copies of the index even when an IndexWriter is being used to make changes. I would need this to make sure the index can still be hot copied even when we hold a lock for longer times (avoid closing the IndexWriter). > >> BTW this will mean the DirectoryProvider won't be used by anybody. > Really? Sorry, of course not, I meant to say the "DirectoryProvider-specific LOCK"; It is currently used only to achieve mutual exclusion between the IndexWriter activity and the FSMasterDirectoryProvider's copy task, so if we implement the snapshotting it won't be useful anymore. > > --Hardy > Sanne _______________________________________________ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
