Hi Thanks for your replies. Is there any documentation that I can look at for using rsync? I am thinking of creating my own solution (early days) I might come back to Solr.
Cheers Amin On Mon, Jun 22, 2009 at 10:24 AM, Michael McCandless < luc...@mikemccandless.com> wrote: > Solr (as of 1.4) is moving to Java only implementation for replication > (http://wiki.apache.org/solr/SolrReplication). > > If you roll your own, make sure to only run rsync at "proper" times > according to the master, eg, it's probably simplest to close the > writer on the master before running rsync. If you don't want to pause > indexing when copying, then you should use SnapshotDeletionPolicy to > get the list of files that need copying and pass that list to rsync. > If you simply rsync whenever you want to, and a writer is open on the > index, it's easy to get a corrupt copy. > > Mike > > On Mon, Jun 22, 2009 at 5:09 AM, Ian Lea<ian....@gmail.com> wrote: > > Or if you don't want to use Solr, rsync is very good for this. Only > takes > > changes, takes care of deletions, very robust. That is what Solr uses. > > > > See http://wiki.apache.org/solr/CollectionDistribution for details or > ideas. > > > > > > -- > > Ian. > > > > On Sun, Jun 21, 2009 at 4:32 AM, Otis Gospodnetic < > > otis_gospodne...@yahoo.com> wrote: > > > >> > >> Hi Amin, > >> > >> Have a look at Solr, it may be what you are after: > >> http://lucene.apache.org/solr/ > >> > >> Otis > >> -- > >> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > >> > >> > >> > >> ----- Original Message ---- > >> > From: Amin Mohammed-Coleman <ami...@gmail.com> > >> > To: java-user@lucene.apache.org > >> > Sent: Saturday, June 20, 2009 1:32:18 PM > >> > Subject: Lucene directory copy - master copy to local index > >> > > >> > Hi > >> > I am prototyping the following situation: > >> > > >> > 1) Multiple nodes in a cluster > >> > 2) Each node has a local index > >> > 3) Search requests are maded against the local index > >> > 4) Index updates are sent to a JMS where a master process adds > document > >> to > >> > index > >> > 5) Each node is configured to check whether the local index is out of > >> date > >> > and needs to copy from master index. > >> > > >> > I have created a class that will be configured (quartz job) to copy > >> master > >> > directory files to the local directory. I was wondering if I could > get > >> > advice on whether this is the best way to copy master directory > changes > >> to > >> > the local. > >> > > >> > public class LocalIndexProvider { > >> > > >> > private Directory masterDirectory; > >> > > >> > private Directory localDirectory; > >> > > >> > private final Logger LOGGER = Logger.getLogger(getClass()); > >> > > >> > public LocalIndexProvider(Directory masterDirectory, Directory > >> > localDirectory) { > >> > > >> > this.masterDirectory = masterDirectory; > >> > > >> > this.localDirectory = localDirectory; > >> > > >> > } > >> > > >> > public void updateLocalIndex() throws Exception { > >> > > >> > StopWatch stopWatch = new StopWatch("copy-time"); > >> > > >> > stopWatch.start(); > >> > > >> > String[] masterFiles = masterDirectory.list(); > >> > > >> > String[] localFiles = localDirectory.list(); > >> > > >> > Lock lock = localDirectory.getLockFactory().makeLock("test"); > >> > > >> > lock.obtain(); > >> > > >> > try { > >> > > >> > if (localFiles.length != masterFiles.length) { > >> > > >> > Directory.copy(masterDirectory, localDirectory, false); > >> > > >> > } > >> > > >> > }finally { > >> > > >> > lock.release(); > >> > > >> > } > >> > > >> > stopWatch.stop(); > >> > > >> > LOGGER.debug("total time taken :" + stopWatch.getTotalTimeMillis()); > >> > > >> > } > >> > > >> > } > >> > > >> > > >> > > >> > The above is just a proof of concept and I am open to any comments > about > >> it. > >> > It may be inefficient or it may be. I have looked at the source code > >> from > >> > hibernate search and it seems as though they copy each file from the > >> > directory (master to local). I'm not sure whether this is something I > >> need > >> > to do or the above (with more refinement) would be enough. > >> > > >> > Any help would be highly appreciated. > >> > > >> > Cheers > >> > Amin > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: java-user-h...@lucene.apache.org > >> > >> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
