Or if you don't want to use Solr, rsync is very good for this. Only takes changes, takes care of deletions, very robust. That is what Solr uses.
See http://wiki.apache.org/solr/CollectionDistribution for details or ideas. -- Ian. On Sun, Jun 21, 2009 at 4:32 AM, Otis Gospodnetic < otis_gospodne...@yahoo.com> wrote: > > Hi Amin, > > Have a look at Solr, it may be what you are after: > http://lucene.apache.org/solr/ > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > ----- Original Message ---- > > From: Amin Mohammed-Coleman <ami...@gmail.com> > > To: java-user@lucene.apache.org > > Sent: Saturday, June 20, 2009 1:32:18 PM > > Subject: Lucene directory copy - master copy to local index > > > > Hi > > I am prototyping the following situation: > > > > 1) Multiple nodes in a cluster > > 2) Each node has a local index > > 3) Search requests are maded against the local index > > 4) Index updates are sent to a JMS where a master process adds document > to > > index > > 5) Each node is configured to check whether the local index is out of > date > > and needs to copy from master index. > > > > I have created a class that will be configured (quartz job) to copy > master > > directory files to the local directory. I was wondering if I could get > > advice on whether this is the best way to copy master directory changes > to > > the local. > > > > public class LocalIndexProvider { > > > > private Directory masterDirectory; > > > > private Directory localDirectory; > > > > private final Logger LOGGER = Logger.getLogger(getClass()); > > > > public LocalIndexProvider(Directory masterDirectory, Directory > > localDirectory) { > > > > this.masterDirectory = masterDirectory; > > > > this.localDirectory = localDirectory; > > > > } > > > > public void updateLocalIndex() throws Exception { > > > > StopWatch stopWatch = new StopWatch("copy-time"); > > > > stopWatch.start(); > > > > String[] masterFiles = masterDirectory.list(); > > > > String[] localFiles = localDirectory.list(); > > > > Lock lock = localDirectory.getLockFactory().makeLock("test"); > > > > lock.obtain(); > > > > try { > > > > if (localFiles.length != masterFiles.length) { > > > > Directory.copy(masterDirectory, localDirectory, false); > > > > } > > > > }finally { > > > > lock.release(); > > > > } > > > > stopWatch.stop(); > > > > LOGGER.debug("total time taken :" + stopWatch.getTotalTimeMillis()); > > > > } > > > > } > > > > > > > > The above is just a proof of concept and I am open to any comments about > it. > > It may be inefficient or it may be. I have looked at the source code > from > > hibernate search and it seems as though they copy each file from the > > directory (master to local). I'm not sure whether this is something I > need > > to do or the above (with more refinement) would be enough. > > > > Any help would be highly appreciated. > > > > Cheers > > Amin > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
