Re: Repair taking a long, long time

2011-07-20 Thread aaron morton
The first thing to do is understand what the server is doing. As Edward said, there are two phases to the repair first the differences are calculated and then they are shared between the neighbours. Lets an a third step, once the neighbour gets the data it has to rebuild the indexes and bloom

Re: Repair taking a long, long time

2011-07-20 Thread David Boxenhorn
As I indicated below (but didn't say specifically) another option is to set read repair chance to 1.0 for all your CFs and loop over all your data, since read triggers a read repair. On Wed, Jul 20, 2011 at 4:58 PM, Maxim Potekhin wrote: > ** > I can re-load all data that I have in the cluster,

Re: Repair taking a long, long time

2011-07-20 Thread Boris Yen
We also got the same problem when using 0.8.0. As far as I know, there are a few issues relative to 'repair' has been marked as resolved at 0.8.1. Hope this could really solve our problem. On Wed, Jul 20, 2011 at 8:47 PM, David Boxenhorn wrote: > I have this problem too, and I don't understand w

Re: Repair taking a long, long time

2011-07-20 Thread Maxim Potekhin
I can re-load all data that I have in the cluster, from a flat-file cache I have on NFS, many times faster than the nodetool repair takes. And that's not even accurate because as other noted nodetool repair eats up disk space for breakfast and takes more than 24hrs on 200GB data load, at which po

Re: Repair taking a long, long time

2011-07-20 Thread David Boxenhorn
I have this problem too, and I don't understand why. I can repair my nodes very quickly by looping though all my data (when you read your data it does read-repair), but nodetool repair takes forever. I understand that nodetool repair builds merkle trees, etc. etc., so it's a different algorithm, b

Re: Repair taking a long, long time

2011-07-19 Thread Maxim Potekhin
Thanks Edward. I'm told by our IT that the switch connecting the nodes is pretty fast. Seriously, in my house I copy complete DVD images from my bedroom to the living room downstairs via WiFi, and a dozen of GB does not seem like a problem, on dirt cheap hardware (Patriot Box Office). I also hav

Re: Repair taking a long, long time

2011-07-19 Thread Edward Capriolo
On Tue, Jul 19, 2011 at 11:36 AM, Maxim Potekhin wrote: > Thanks for pointer. As a comment to your disk speed comment -- > I had already installed SSDs on these nodes. > > Maxim > > > On 7/19/2011 9:41 AM, Edward Capriolo wrote: > > > > On Tue, Jul 19, 2011 at 7:24 AM, Maxim Potekhin wrote: > >

Re: Repair taking a long, long time

2011-07-19 Thread Maxim Potekhin
Thanks for pointer. As a comment to your disk speed comment -- I had already installed SSDs on these nodes. Maxim On 7/19/2011 9:41 AM, Edward Capriolo wrote: On Tue, Jul 19, 2011 at 7:24 AM, Maxim Potekhin > wrote: We have something of the order of 200GB load

Re: Repair taking a long, long time

2011-07-19 Thread Edward Capriolo
On Tue, Jul 19, 2011 at 7:24 AM, Maxim Potekhin wrote: > We have something of the order of 200GB load on each of 3 machines in a > balanced cluster under 0.8.1. > I started repair about 24hrs ago and did some moderate amount of inserts > since then (a small fraction of > data load). The repair st

Repair taking a long, long time

2011-07-19 Thread Maxim Potekhin
We have something of the order of 200GB load on each of 3 machines in a balanced cluster under 0.8.1. I started repair about 24hrs ago and did some moderate amount of inserts since then (a small fraction of data load). The repair still appears to be running. What could go wrong? Thanks, Maxim