Running java-driver in parallel cassandra-driver-core 2.1.5, multithreading wokrs extremely slow.

[Serega Sheypak]

Hi, I have weird driver behaviour. Can you help me please to find the
problem?
Problem: I try to insert data using 10 threads.
I see that 10 thread starts, they start to insert some data and then they
hung. It takes enormous amount of time to insert (seconds for 1K inserts).
It runs 1K per second if I use single thread to insert.

Here is my code:
https://gist.github.com/seregasheypak/6ddf5b24cd1c195f5355

Re: Running java-driver in parallel cassandra-driver-core 2.1.5, multithreading wokrs extremely slow.

[Vova Shelgunov]

Did you tried to use BatchStatement?
On Jul 2, 2015 11:00 AM, "Serega Sheypak"  wrote:

> Hi, I have weird driver behaviour. Can you help me please to find the
> problem?
> Problem: I try to insert data using 10 threads.
> I see that 10 thread starts, they start to insert some data and then they
> hung. It takes enormous amount of time to insert (seconds for 1K inserts).
> It runs 1K per second if I use single thread to insert.
>
> Here is my code:
> https://gist.github.com/seregasheypak/6ddf5b24cd1c195f5355
>

Re: Running java-driver in parallel cassandra-driver-core 2.1.5, multithreading wokrs extremely slow.

[Serega Sheypak]

What is the reason to do that? I understand BatchStatement as a kind of
atomic insert hack.
How it can help me to solve concurrency problem? 1 thread with sync insert
gives me 1K ops/sec. 10 threads give me 20 ops/sec :)

Here are metrics for single thread async insert:

-- Timers
--

insertTimer

 count = 4641205

 mean rate = 14971.58 calls/second

 1-minute rate = 18410.90 calls/second

 5-minute rate = 10555.73 calls/second

15-minute rate = 4511.58 calls/second

   min = 0.00 milliseconds

   max = 0.12 milliseconds

  mean = 0.01 milliseconds

stddev = 0.01 milliseconds

median = 0.01 milliseconds

  75% <= 0.01 milliseconds

  95% <= 0.01 milliseconds

  98% <= 0.02 milliseconds

  99% <= 0.02 milliseconds

99.9% <= 0.12 milliseconds


what should I do to reach better performance when i use several threads?

2015-07-02 10:34 GMT+02:00 Vova Shelgunov :

> Did you tried to use BatchStatement?
> On Jul 2, 2015 11:00 AM, "Serega Sheypak" 
> wrote:
>
>> Hi, I have weird driver behaviour. Can you help me please to find the
>> problem?
>> Problem: I try to insert data using 10 threads.
>> I see that 10 thread starts, they start to insert some data and then they
>> hung. It takes enormous amount of time to insert (seconds for 1K inserts).
>> It runs 1K per second if I use single thread to insert.
>>
>> Here is my code:
>> https://gist.github.com/seregasheypak/6ddf5b24cd1c195f5355
>>
>

Re: Error while adding a new node.

[Neha Trivedi]

any help?

On Thu, Jul 2, 2015 at 6:18 AM, Neha Trivedi  wrote:

> also:
> root@cas03:~# sudo service cassandra start
> root@cas03:~# lsof -n | grep java | wc -l
> 5315
> root@cas03:~# lsof -n | grep java | wc -l
> 977317
> root@cas03:~# lsof -n | grep java | wc -l
> 880240
> root@cas03:~# lsof -n | grep java | wc -l
> 882402
>
>
> On Wed, Jul 1, 2015 at 6:31 PM, Neha Trivedi 
> wrote:
>
>> One of the column family has SStable count as under :
>> SSTable count: 98506
>>
>> Can it be because of 2.1.3 version of cassandra..
>> I found this : https://issues.apache.org/jira/browse/CASSANDRA-8964
>>
>> regards
>> Neha
>>
>>
>> On Wed, Jul 1, 2015 at 5:40 PM, Jason Wee  wrote:
>>
>>> nodetool cfstats?
>>>
>>> On Wed, Jul 1, 2015 at 8:08 PM, Neha Trivedi 
>>> wrote:
>>>
 Hey..
 nodetool compactionstats
 pending tasks: 0

 no pending tasks.

 Dont have opscenter. how do I monitor sstables?


 On Wed, Jul 1, 2015 at 4:28 PM, Alain RODRIGUEZ 
 wrote:

> You also might want to check if you have compactions pending
> (Opscenter / nodetool compactionstats).
>
> Also you can monitor the number of sstables.
>
> C*heers
>
> Alain
>
> 2015-07-01 11:53 GMT+02:00 Neha Trivedi :
>
>> Thanks I will checkout.
>> I increased the ulimit to 10, but I am getting the same error,
>> but after a while.
>> regards
>> Neha
>>
>>
>> On Wed, Jul 1, 2015 at 2:22 PM, Alain RODRIGUEZ 
>> wrote:
>>
>>> Just check the process owner to be sure (top, htop, ps, ...)
>>>
>>>
>>> http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html#reference_ds_sxl_gf3_2k__user-resource-limits
>>>
>>> C*heers,
>>>
>>> Alain
>>>
>>> 2015-07-01 7:33 GMT+02:00 Neha Trivedi :
>>>
 Arun,
 I am logging on to Server as root and running (sudo service
 cassandra start)

 regards
 Neha

 On Wed, Jul 1, 2015 at 11:00 AM, Neha Trivedi <
 nehajtriv...@gmail.com> wrote:

> Thanks Arun ! I will try and get back !
>
> On Wed, Jul 1, 2015 at 10:32 AM, Arun  wrote:
>
>> Looks like you have too many open files issue. Increase the
>> ulimit for the user.
>>
>>  If you are starting the cassandra daemon using user cassandra,
>> increase the ulimit for that user.
>>
>>
>> > On Jun 30, 2015, at 21:16, Neha Trivedi 
>> wrote:
>> >
>> > Hello,
>> > I have a 4 node cluster with SimpleSnitch.
>> > Cassandra :  Cassandra 2.1.3
>> >
>> > I am trying to add a new node (cassandra 2.1.7) and I get the
>> following error.
>> >
>> > ERROR [STREAM-IN-] 2015-06-30 05:13:48,516
>> JVMStabilityInspector.java:94 - JVM state determined to be unstable.
>> Exiting forcefully due to:
>> > java.io.FileNotFoundException:
>> /var/lib/cassandra/data/-Index.db (Too many open files)
>> >
>> > I increased the MAX_HEAP_SIZE then I get :
>> > ERROR [CompactionExecutor:9] 2015-06-30 23:31:44,792
>> CassandraDaemon.java:223 - Exception in thread
>> Thread[CompactionExecutor:9,1,main]
>> > java.lang.RuntimeException: java.io.FileNotFoundException:
>> /var/lib/cassandra/data/-Data.db (Too many open files)
>> > at
>> org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52)
>> ~[apache-cassandra-2.1.7.jar:2.1.7]
>> >
>> > Is it because of the different version of Cassandra (2.1.3 and
>> 2.17) ?
>> >
>> > regards
>> > N
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>>
>
>

>>>
>>
>

>>>
>>
>

Re: Error while adding a new node.

[Carlos Rolo]

Indeed you should upgrade to 2.1.7.

And then report if you are still facing problems. Versions up to 2.1.5 (in
the 2.1.x series) are not considered stable.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com

On Thu, Jul 2, 2015 at 11:40 AM, Neha Trivedi 
wrote:

> any help?
>
> On Thu, Jul 2, 2015 at 6:18 AM, Neha Trivedi 
> wrote:
>
>> also:
>> root@cas03:~# sudo service cassandra start
>> root@cas03:~# lsof -n | grep java | wc -l
>> 5315
>> root@cas03:~# lsof -n | grep java | wc -l
>> 977317
>> root@cas03:~# lsof -n | grep java | wc -l
>> 880240
>> root@cas03:~# lsof -n | grep java | wc -l
>> 882402
>>
>>
>> On Wed, Jul 1, 2015 at 6:31 PM, Neha Trivedi 
>> wrote:
>>
>>> One of the column family has SStable count as under :
>>> SSTable count: 98506
>>>
>>> Can it be because of 2.1.3 version of cassandra..
>>> I found this : https://issues.apache.org/jira/browse/CASSANDRA-8964
>>>
>>> regards
>>> Neha
>>>
>>>
>>> On Wed, Jul 1, 2015 at 5:40 PM, Jason Wee  wrote:
>>>
 nodetool cfstats?

 On Wed, Jul 1, 2015 at 8:08 PM, Neha Trivedi 
 wrote:

> Hey..
> nodetool compactionstats
> pending tasks: 0
>
> no pending tasks.
>
> Dont have opscenter. how do I monitor sstables?
>
>
> On Wed, Jul 1, 2015 at 4:28 PM, Alain RODRIGUEZ 
> wrote:
>
>> You also might want to check if you have compactions pending
>> (Opscenter / nodetool compactionstats).
>>
>> Also you can monitor the number of sstables.
>>
>> C*heers
>>
>> Alain
>>
>> 2015-07-01 11:53 GMT+02:00 Neha Trivedi :
>>
>>> Thanks I will checkout.
>>> I increased the ulimit to 10, but I am getting the same error,
>>> but after a while.
>>> regards
>>> Neha
>>>
>>>
>>> On Wed, Jul 1, 2015 at 2:22 PM, Alain RODRIGUEZ 
>>> wrote:
>>>
 Just check the process owner to be sure (top, htop, ps, ...)


 http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html#reference_ds_sxl_gf3_2k__user-resource-limits

 C*heers,

 Alain

 2015-07-01 7:33 GMT+02:00 Neha Trivedi :

> Arun,
> I am logging on to Server as root and running (sudo service
> cassandra start)
>
> regards
> Neha
>
> On Wed, Jul 1, 2015 at 11:00 AM, Neha Trivedi <
> nehajtriv...@gmail.com> wrote:
>
>> Thanks Arun ! I will try and get back !
>>
>> On Wed, Jul 1, 2015 at 10:32 AM, Arun 
>> wrote:
>>
>>> Looks like you have too many open files issue. Increase the
>>> ulimit for the user.
>>>
>>>  If you are starting the cassandra daemon using user cassandra,
>>> increase the ulimit for that user.
>>>
>>>
>>> > On Jun 30, 2015, at 21:16, Neha Trivedi <
>>> nehajtriv...@gmail.com> wrote:
>>> >
>>> > Hello,
>>> > I have a 4 node cluster with SimpleSnitch.
>>> > Cassandra :  Cassandra 2.1.3
>>> >
>>> > I am trying to add a new node (cassandra 2.1.7) and I get the
>>> following error.
>>> >
>>> > ERROR [STREAM-IN-] 2015-06-30 05:13:48,516
>>> JVMStabilityInspector.java:94 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> > java.io.FileNotFoundException:
>>> /var/lib/cassandra/data/-Index.db (Too many open files)
>>> >
>>> > I increased the MAX_HEAP_SIZE then I get :
>>> > ERROR [CompactionExecutor:9] 2015-06-30 23:31:44,792
>>> CassandraDaemon.java:223 - Exception in thread
>>> Thread[CompactionExecutor:9,1,main]
>>> > java.lang.RuntimeException: java.io.FileNotFoundException:
>>> /var/lib/cassandra/data/-Data.db (Too many open files)
>>> > at
>>> org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52)
>>> ~[apache-cassandra-2.1.7.jar:2.1.7]
>>> >
>>> > Is it because of the different version of Cassandra (2.1.3 and
>>> 2.17) ?
>>> >
>>> > regards
>>> > N
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>>
>>
>>
>

>>>
>>
>

>>>
>>
>

-- 


--

Re: Running java-driver in parallel cassandra-driver-core 2.1.5, multithreading wokrs extremely slow.

[Jonathan Haddad]

If you post the code you're using to test, it would be helpful.  You should
also use cassandra-stress to see if you get similar results.

On Thu, Jul 2, 2015 at 1:39 AM Serega Sheypak 
wrote:

> What is the reason to do that? I understand BatchStatement as a kind of
> atomic insert hack.
> How it can help me to solve concurrency problem? 1 thread with sync insert
> gives me 1K ops/sec. 10 threads give me 20 ops/sec :)
>
> Here are metrics for single thread async insert:
>
> -- Timers
> --
>
> insertTimer
>
>  count = 4641205
>
>  mean rate = 14971.58 calls/second
>
>  1-minute rate = 18410.90 calls/second
>
>  5-minute rate = 10555.73 calls/second
>
> 15-minute rate = 4511.58 calls/second
>
>min = 0.00 milliseconds
>
>max = 0.12 milliseconds
>
>   mean = 0.01 milliseconds
>
> stddev = 0.01 milliseconds
>
> median = 0.01 milliseconds
>
>   75% <= 0.01 milliseconds
>
>   95% <= 0.01 milliseconds
>
>   98% <= 0.02 milliseconds
>
>   99% <= 0.02 milliseconds
>
> 99.9% <= 0.12 milliseconds
>
>
> what should I do to reach better performance when i use several threads?
>
> 2015-07-02 10:34 GMT+02:00 Vova Shelgunov :
>
>> Did you tried to use BatchStatement?
>> On Jul 2, 2015 11:00 AM, "Serega Sheypak" 
>> wrote:
>>
>>> Hi, I have weird driver behaviour. Can you help me please to find the
>>> problem?
>>> Problem: I try to insert data using 10 threads.
>>> I see that 10 thread starts, they start to insert some data and then
>>> they hung. It takes enormous amount of time to insert (seconds for 1K
>>> inserts). It runs 1K per second if I use single thread to insert.
>>>
>>> Here is my code:
>>> https://gist.github.com/seregasheypak/6ddf5b24cd1c195f5355
>>>
>>
>

Re: [MASSMAIL]Re: Error while adding a new node.

[Marcos Ortiz]

The recommended version to use is 2.1.5 because, like you Carlos said, 
2.1.6 and 2.1.7 are very new to consider them like

stable.

On 02/07/15 08:55, Carlos Rolo wrote:

Indeed you should upgrade to 2.1.7.

And then report if you are still facing problems. Versions up to 2.1.5 
(in the 2.1.x series) are not considered stable.


Regards,

Carlos Juzarte Rolo
Cassandra Consultant
Pythian - Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: 
_linkedin.com/in/carlosjuzarterolo 
_

Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com 

On Thu, Jul 2, 2015 at 11:40 AM, Neha Trivedi > wrote:


any help?

On Thu, Jul 2, 2015 at 6:18 AM, Neha Trivedi
mailto:nehajtriv...@gmail.com>> wrote:

also:
root@cas03:~# sudo service cassandra start
root@cas03:~# lsof -n | grep java | wc -l
5315
root@cas03:~# lsof -n | grep java | wc -l
977317
root@cas03:~# lsof -n | grep java | wc -l
880240
root@cas03:~# lsof -n | grep java | wc -l
882402


On Wed, Jul 1, 2015 at 6:31 PM, Neha Trivedi
mailto:nehajtriv...@gmail.com>> wrote:

One of the column family has SStable count as under :
SSTable count: 98506

Can it be because of 2.1.3 version of cassandra..
I found this :
https://issues.apache.org/jira/browse/CASSANDRA-8964

regards
Neha


On Wed, Jul 1, 2015 at 5:40 PM, Jason Wee
mailto:peich...@gmail.com>> wrote:

nodetool cfstats?

On Wed, Jul 1, 2015 at 8:08 PM, Neha Trivedi
mailto:nehajtriv...@gmail.com>> wrote:

Hey..
nodetool compactionstats
pending tasks: 0

no pending tasks.

Dont have opscenter. how do I monitor sstables?


On Wed, Jul 1, 2015 at 4:28 PM, Alain RODRIGUEZ
mailto:arodr...@gmail.com>>
wrote:

You also might want to check if you have
compactions pending (Opscenter / nodetool
compactionstats).

Also you can monitor the number of sstables.

C*heers

Alain

2015-07-01 11:53 GMT+02:00 Neha Trivedi
mailto:nehajtriv...@gmail.com>>:

Thanks I will checkout.
I increased the ulimit to 10, but I am
getting the same error, but after a while.
regards
Neha


On Wed, Jul 1, 2015 at 2:22 PM, Alain
RODRIGUEZ mailto:arodr...@gmail.com>> wrote:

Just check the process owner to be
sure (top, htop, ps, ...)


http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html#reference_ds_sxl_gf3_2k__user-resource-limits

C*heers,

Alain

2015-07-01 7:33 GMT+02:00 Neha Trivedi
mailto:nehajtriv...@gmail.com>>:

Arun,
I am logging on to Server as root
and running (sudo service
cassandra start)

regards
Neha

On Wed, Jul 1, 2015 at 11:00 AM,
Neha Trivedi
mailto:nehajtriv...@gmail.com>>
wrote:

Thanks Arun ! I will try and
get back !

On Wed, Jul 1, 2015 at 10:32
AM, Arun mailto:arunsi...@gmail.com>>
wrote:

Looks like you have too
many open files issue.
Increase the ulimit for
the user.

 If you are starting the
cassandra daemon using
user cassandra, increase
the ulimit for that user.


> On Jun 30, 2015, at
  

Re: [MASSMAIL]Re: Error while adding a new node.

[Carlos Rolo]

Marco you should also avoid 2.1.5 and 2.1.6 because of
https://issues.apache.org/jira/browse/CASSANDRA-9549

I know (And often don't recommend last versions, I'm still recommending
2.0.x series unless someone is already in 2.1.x) but given the above bug,
2.1.7 is the best option.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com

On Thu, Jul 2, 2015 at 3:20 PM, Marcos Ortiz  wrote:

>  The recommended version to use is 2.1.5 because, like you Carlos said,
> 2.1.6 and 2.1.7 are very new to consider them like
> stable.
>
> On 02/07/15 08:55, Carlos Rolo wrote:
>
>  Indeed you should upgrade to 2.1.7.
>
>  And then report if you are still facing problems. Versions up to 2.1.5
> (in the 2.1.x series) are not considered stable.
>
>Regards,
>
>  Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
>  rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
> *
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> www.pythian.com
>
> On Thu, Jul 2, 2015 at 11:40 AM, Neha Trivedi 
> wrote:
>
>> any help?
>>
>> On Thu, Jul 2, 2015 at 6:18 AM, Neha Trivedi 
>> wrote:
>>
>>> also:
>>> root@cas03:~# sudo service cassandra start
>>> root@cas03:~# lsof -n | grep java | wc -l
>>> 5315
>>> root@cas03:~# lsof -n | grep java | wc -l
>>> 977317
>>> root@cas03:~# lsof -n | grep java | wc -l
>>> 880240
>>> root@cas03:~# lsof -n | grep java | wc -l
>>> 882402
>>>
>>>
>>> On Wed, Jul 1, 2015 at 6:31 PM, Neha Trivedi 
>>> wrote:
>>>
   One of the column family has SStable count as under :
 SSTable count: 98506

  Can it be because of 2.1.3 version of cassandra..
  I found this : https://issues.apache.org/jira/browse/CASSANDRA-8964

  regards
  Neha


 On Wed, Jul 1, 2015 at 5:40 PM, Jason Wee  wrote:

> nodetool cfstats?
>
> On Wed, Jul 1, 2015 at 8:08 PM, Neha Trivedi 
> wrote:
>
>>  Hey..
>> nodetool compactionstats
>> pending tasks: 0
>>
>>  no pending tasks.
>>
>>  Dont have opscenter. how do I monitor sstables?
>>
>>
>> On Wed, Jul 1, 2015 at 4:28 PM, Alain RODRIGUEZ 
>> wrote:
>>
>>> You also might want to check if you have compactions pending
>>> (Opscenter / nodetool compactionstats).
>>>
>>>  Also you can monitor the number of sstables.
>>>
>>>  C*heers
>>>
>>>  Alain
>>>
>>> 2015-07-01 11:53 GMT+02:00 Neha Trivedi :
>>>
   Thanks I will checkout.
  I increased the ulimit to 10, but I am getting the same error,
 but after a while.
  regards
  Neha


 On Wed, Jul 1, 2015 at 2:22 PM, Alain RODRIGUEZ >>> > wrote:

>  Just check the process owner to be sure (top, htop, ps, ...)
>
>
> http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html#reference_ds_sxl_gf3_2k__user-resource-limits
>
>  C*heers,
>
>  Alain
>
> 2015-07-01 7:33 GMT+02:00 Neha Trivedi :
>
>>   Arun,
>>  I am logging on to Server as root and running (sudo service
>> cassandra start)
>>
>>  regards
>>  Neha
>>
>> On Wed, Jul 1, 2015 at 11:00 AM, Neha Trivedi <
>> nehajtriv...@gmail.com> wrote:
>>
>>> Thanks Arun ! I will try and get back !
>>>
>>> On Wed, Jul 1, 2015 at 10:32 AM, Arun 
>>> wrote:
>>>
 Looks like you have too many open files issue. Increase the
 ulimit for the user.

  If you are starting the cassandra daemon using user cassandra,
 increase the ulimit for that user.


 > On Jun 30, 2015, at 21:16, Neha Trivedi <
 nehajtriv...@gmail.com> wrote:
 >
 > Hello,
 > I have a 4 node cluster with SimpleSnitch.
 > Cassandra :  Cassandra 2.1.3
 >
 > I am trying to add a new node (cassandra 2.1.7) and I get the
 following error.
 >
 > ERROR [STREAM-IN-] 2015-06-30 05:13:48,516
 JVMStabilityInspector.java:94 - JVM state determined to be 
 unstable.
 Exiting forcefully due to:
 > java.io.FileNotFoundException:
 /var/lib/cassandra/data/-Index.db (Too many open files)
 >
 > I increased the MAX_HEAP_SIZE then I get :
 > ERROR [CompactionExecutor:9] 2015-06-30 23:31:44,792
 CassandraDaemon.java:223 - Exception in thread
 Thread[CompactionExecutor:9,1,mai

Re: Experiencing Timeouts on one node

[Jason Wee]

you should check the network connectivity for this node and also its system
average load. is that typo or literary what it is, cassandra 1.2.15.*1* and
java 6 update *85* ?



On Thu, Jul 2, 2015 at 12:59 AM, Shashi Yachavaram 
wrote:

> We have a 28 node cluster, out of which only one node is experiencing
> timeouts.
> We thought it was the raid, but there are two other nodes on the same raid
> without
> any problem. Also The problem goes away if we reboot the node, and then
> reappears
> after seven  days. The following hinted hand-off timeouts are seen on the
> node
> experiencing the timeouts. Also we did not notice any gossip errors.
>
> I was wondering if anyone has seen this issue and how they resolved it.
>
> Cassandra Version: 1.2.15.1
> OS: Linux cm 2.6.32-504.8.1.el6.x86_64 #1 SMP Fri Dec 19 12:09:25 EST 2014
> x86_64 x86_64 x86_64 GNU/Linux
> java version "1.6.0_85"
>
>
> 
> INFO [HintedHandoff:2] 2015-06-17 22:52:08,130 HintedHandOffManager.java
> (line 296) Started hinted handoff for host:
> 4fe86051-6bca-4c28-b09c-1b0f073c1588 with IP: /192.168.1.122
>  INFO [HintedHandoff:1] 2015-06-17 22:52:08,131 HintedHandOffManager.java
> (line 296) Started hinted handoff for host:
> bbf0878b-b405-4518-b649-f6cf7c9a6550 with IP: /192.168.1.119
>  INFO [HintedHandoff:2] 2015-06-17 22:52:17,634 HintedHandOffManager.java
> (line 422) Timed out replaying hints to /192.168.1.122; aborting (0
> delivered)
>  INFO [HintedHandoff:2] 2015-06-17 22:52:17,635 HintedHandOffManager.java
> (line 296) Started hinted handoff for host:
> f7b7ab10-4d42-4f0c-af92-2934a075bee3 with IP: /192.168.1.108
>  INFO [HintedHandoff:1] 2015-06-17 22:52:17,643 HintedHandOffManager.java
> (line 422) Timed out replaying hints to /192.168.1.119; aborting (0
> delivered)
>  INFO [HintedHandoff:1] 2015-06-17 22:52:17,643 HintedHandOffManager.java
> (line 296) Started hinted handoff for host:
> ddb79f35-3e2b-4be8-84d8-7942086e2b73 with IP: /192.168.1.104
>  INFO [HintedHandoff:2] 2015-06-17 22:52:27,143 HintedHandOffManager.java
> (line 422) Timed out replaying hints to /192.168.1.108; aborting (0
> delivered)
>  INFO [HintedHandoff:2] 2015-06-17 22:52:27,144 HintedHandOffManager.java
> (line 296) Started hinted handoff for host:
> 6a2fa431-4a51-44cb-af19-1991c960e075 with IP: /192.168.1.117
>  INFO [HintedHandoff:1] 2015-06-17 22:52:27,153 HintedHandOffManager.java
> (line 422) Timed out replaying hints to /192.168.1.104; aborting (0
> delivered)
>  INFO [HintedHandoff:1] 2015-06-17 22:52:27,154 HintedHandOffManager.java
> (line 296) Started hinted handoff for host:
> cf03174a-533c-44d6-a679-e70090ad2bc5 with IP: /192.168.1.107
>
> 
>
> Thanks
> -shashi..
>

Re: [MASSMAIL]Re: Error while adding a new node.

[Neha Trivedi]

thanks for the reply.!!
I will update it to 2.1.7 and checkout.

On Thu, Jul 2, 2015 at 6:59 PM, Carlos Rolo  wrote:

> Marco you should also avoid 2.1.5 and 2.1.6 because of
> https://issues.apache.org/jira/browse/CASSANDRA-9549
>
> I know (And often don't recommend last versions, I'm still recommending
> 2.0.x series unless someone is already in 2.1.x) but given the above bug,
> 2.1.7 is the best option.
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
> *
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> www.pythian.com
>
> On Thu, Jul 2, 2015 at 3:20 PM, Marcos Ortiz  wrote:
>
>>  The recommended version to use is 2.1.5 because, like you Carlos said,
>> 2.1.6 and 2.1.7 are very new to consider them like
>> stable.
>>
>> On 02/07/15 08:55, Carlos Rolo wrote:
>>
>>  Indeed you should upgrade to 2.1.7.
>>
>>  And then report if you are still facing problems. Versions up to 2.1.5
>> (in the 2.1.x series) are not considered stable.
>>
>>Regards,
>>
>>  Carlos Juzarte Rolo
>> Cassandra Consultant
>>
>> Pythian - Love your data
>>
>>  rolo@pythian | Twitter: cjrolo | Linkedin: 
>> *linkedin.com/in/carlosjuzarterolo
>> *
>> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
>> www.pythian.com
>>
>> On Thu, Jul 2, 2015 at 11:40 AM, Neha Trivedi 
>> wrote:
>>
>>> any help?
>>>
>>> On Thu, Jul 2, 2015 at 6:18 AM, Neha Trivedi 
>>> wrote:
>>>
 also:
 root@cas03:~# sudo service cassandra start
 root@cas03:~# lsof -n | grep java | wc -l
 5315
 root@cas03:~# lsof -n | grep java | wc -l
 977317
 root@cas03:~# lsof -n | grep java | wc -l
 880240
 root@cas03:~# lsof -n | grep java | wc -l
 882402


 On Wed, Jul 1, 2015 at 6:31 PM, Neha Trivedi 
 wrote:

>   One of the column family has SStable count as under :
> SSTable count: 98506
>
>  Can it be because of 2.1.3 version of cassandra..
>  I found this : https://issues.apache.org/jira/browse/CASSANDRA-8964
>
>  regards
>  Neha
>
>
> On Wed, Jul 1, 2015 at 5:40 PM, Jason Wee  wrote:
>
>> nodetool cfstats?
>>
>> On Wed, Jul 1, 2015 at 8:08 PM, Neha Trivedi 
>> wrote:
>>
>>>  Hey..
>>> nodetool compactionstats
>>> pending tasks: 0
>>>
>>>  no pending tasks.
>>>
>>>  Dont have opscenter. how do I monitor sstables?
>>>
>>>
>>> On Wed, Jul 1, 2015 at 4:28 PM, Alain RODRIGUEZ 
>>> wrote:
>>>
 You also might want to check if you have compactions pending
 (Opscenter / nodetool compactionstats).

  Also you can monitor the number of sstables.

  C*heers

  Alain

 2015-07-01 11:53 GMT+02:00 Neha Trivedi :

>   Thanks I will checkout.
>  I increased the ulimit to 10, but I am getting the same
> error, but after a while.
>  regards
>  Neha
>
>
> On Wed, Jul 1, 2015 at 2:22 PM, Alain RODRIGUEZ <
> arodr...@gmail.com> wrote:
>
>>  Just check the process owner to be sure (top, htop, ps, ...)
>>
>>
>> http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html#reference_ds_sxl_gf3_2k__user-resource-limits
>>
>>  C*heers,
>>
>>  Alain
>>
>> 2015-07-01 7:33 GMT+02:00 Neha Trivedi :
>>
>>>   Arun,
>>>  I am logging on to Server as root and running (sudo service
>>> cassandra start)
>>>
>>>  regards
>>>  Neha
>>>
>>> On Wed, Jul 1, 2015 at 11:00 AM, Neha Trivedi <
>>> nehajtriv...@gmail.com> wrote:
>>>
 Thanks Arun ! I will try and get back !

 On Wed, Jul 1, 2015 at 10:32 AM, Arun 
 wrote:

> Looks like you have too many open files issue. Increase the
> ulimit for the user.
>
>  If you are starting the cassandra daemon using user
> cassandra, increase the ulimit for that user.
>
>
> > On Jun 30, 2015, at 21:16, Neha Trivedi <
> nehajtriv...@gmail.com> wrote:
> >
> > Hello,
> > I have a 4 node cluster with SimpleSnitch.
> > Cassandra :  Cassandra 2.1.3
> >
> > I am trying to add a new node (cassandra 2.1.7) and I get
> the following error.
> >
> > ERROR [STREAM-IN-] 2015-06-30 05:13:48,516
> JVMStabilityInspector.java:94 - JVM state determined to be 
> unstable.
> Exiting forcefully due to:
> > java.io.FileNotFoundExcep

Re: Experiencing Timeouts on one node

[Shashi Yachavaram]

Jason,

The load was evenly distributed. And regarding network connectivity, our
applications were successfully able to connect to the node, but the read
and write operations were timing out. Also we were able to ssh to this
node.

I just pasted  "/bin/nodetool -h node version" and "java -version".

Thanks
shashi

On Thu, Jul 2, 2015 at 8:42 AM, Jason Wee  wrote:

> you should check the network connectivity for this node and also its
> system average load. is that typo or literary what it is, cassandra
> 1.2.15.*1* and java 6 update *85* ?
>
>
>
> On Thu, Jul 2, 2015 at 12:59 AM, Shashi Yachavaram 
> wrote:
>
>> We have a 28 node cluster, out of which only one node is experiencing
>> timeouts.
>> We thought it was the raid, but there are two other nodes on the same
>> raid without
>> any problem. Also The problem goes away if we reboot the node, and then
>> reappears
>> after seven  days. The following hinted hand-off timeouts are seen on the
>> node
>> experiencing the timeouts. Also we did not notice any gossip errors.
>>
>> I was wondering if anyone has seen this issue and how they resolved it.
>>
>> Cassandra Version: 1.2.15.1
>> OS: Linux cm 2.6.32-504.8.1.el6.x86_64 #1 SMP Fri Dec 19 12:09:25 EST
>> 2014 x86_64 x86_64 x86_64 GNU/Linux
>> java version "1.6.0_85"
>>
>>
>> 
>> INFO [HintedHandoff:2] 2015-06-17 22:52:08,130 HintedHandOffManager.java
>> (line 296) Started hinted handoff for host:
>> 4fe86051-6bca-4c28-b09c-1b0f073c1588 with IP: /192.168.1.122
>>  INFO [HintedHandoff:1] 2015-06-17 22:52:08,131 HintedHandOffManager.java
>> (line 296) Started hinted handoff for host:
>> bbf0878b-b405-4518-b649-f6cf7c9a6550 with IP: /192.168.1.119
>>  INFO [HintedHandoff:2] 2015-06-17 22:52:17,634 HintedHandOffManager.java
>> (line 422) Timed out replaying hints to /192.168.1.122; aborting (0
>> delivered)
>>  INFO [HintedHandoff:2] 2015-06-17 22:52:17,635 HintedHandOffManager.java
>> (line 296) Started hinted handoff for host:
>> f7b7ab10-4d42-4f0c-af92-2934a075bee3 with IP: /192.168.1.108
>>  INFO [HintedHandoff:1] 2015-06-17 22:52:17,643 HintedHandOffManager.java
>> (line 422) Timed out replaying hints to /192.168.1.119; aborting (0
>> delivered)
>>  INFO [HintedHandoff:1] 2015-06-17 22:52:17,643 HintedHandOffManager.java
>> (line 296) Started hinted handoff for host:
>> ddb79f35-3e2b-4be8-84d8-7942086e2b73 with IP: /192.168.1.104
>>  INFO [HintedHandoff:2] 2015-06-17 22:52:27,143 HintedHandOffManager.java
>> (line 422) Timed out replaying hints to /192.168.1.108; aborting (0
>> delivered)
>>  INFO [HintedHandoff:2] 2015-06-17 22:52:27,144 HintedHandOffManager.java
>> (line 296) Started hinted handoff for host:
>> 6a2fa431-4a51-44cb-af19-1991c960e075 with IP: /192.168.1.117
>>  INFO [HintedHandoff:1] 2015-06-17 22:52:27,153 HintedHandOffManager.java
>> (line 422) Timed out replaying hints to /192.168.1.104; aborting (0
>> delivered)
>>  INFO [HintedHandoff:1] 2015-06-17 22:52:27,154 HintedHandOffManager.java
>> (line 296) Started hinted handoff for host:
>> cf03174a-533c-44d6-a679-e70090ad2bc5 with IP: /192.168.1.107
>>
>> 
>>
>> Thanks
>> -shashi..
>>
>
>

Re: Experiencing Timeouts on one node

[Alain RODRIGUEZ]

Hi,

I am not sure about what is happening (I have never seen this error
before). Yet from
https://github.com/apache/cassandra/blob/cassandra-1.2/CHANGES.txt  it
looks like some bugs were fixed in late revision of 1.2.x.

I would advice you upgrading to last 1.2.19 (It is an old and stable
version, I see no reason not doing it).

"The problem goes away if we reboot the node, and then reappears after
seven days"
--> Have you TTLs on any table ? (Set to 7 days ?)
--> Do you see any GC warn / heap pressures ?

C*heers,

Alain



2015-07-02 16:20 GMT+02:00 Shashi Yachavaram :

> Jason,
>
> The load was evenly distributed. And regarding network connectivity, our
> applications were successfully able to connect to the node, but the read
> and write operations were timing out. Also we were able to ssh to this
> node.
>
> I just pasted  "/bin/nodetool -h node version" and "java -version".
>
> Thanks
> shashi
>
> On Thu, Jul 2, 2015 at 8:42 AM, Jason Wee  wrote:
>
>> you should check the network connectivity for this node and also its
>> system average load. is that typo or literary what it is, cassandra
>> 1.2.15.*1* and java 6 update *85* ?
>>
>>
>>
>> On Thu, Jul 2, 2015 at 12:59 AM, Shashi Yachavaram 
>> wrote:
>>
>>> We have a 28 node cluster, out of which only one node is experiencing
>>> timeouts.
>>> We thought it was the raid, but there are two other nodes on the same
>>> raid without
>>> any problem. Also The problem goes away if we reboot the node, and then
>>> reappears
>>> after seven  days. The following hinted hand-off timeouts are seen on
>>> the node
>>> experiencing the timeouts. Also we did not notice any gossip errors.
>>>
>>> I was wondering if anyone has seen this issue and how they resolved it.
>>>
>>> Cassandra Version: 1.2.15.1
>>> OS: Linux cm 2.6.32-504.8.1.el6.x86_64 #1 SMP Fri Dec 19 12:09:25 EST
>>> 2014 x86_64 x86_64 x86_64 GNU/Linux
>>> java version "1.6.0_85"
>>>
>>>
>>> 
>>> INFO [HintedHandoff:2] 2015-06-17 22:52:08,130 HintedHandOffManager.java
>>> (line 296) Started hinted handoff for host:
>>> 4fe86051-6bca-4c28-b09c-1b0f073c1588 with IP: /192.168.1.122
>>>  INFO [HintedHandoff:1] 2015-06-17 22:52:08,131
>>> HintedHandOffManager.java (line 296) Started hinted handoff for host:
>>> bbf0878b-b405-4518-b649-f6cf7c9a6550 with IP: /192.168.1.119
>>>  INFO [HintedHandoff:2] 2015-06-17 22:52:17,634
>>> HintedHandOffManager.java (line 422) Timed out replaying hints to /
>>> 192.168.1.122; aborting (0 delivered)
>>>  INFO [HintedHandoff:2] 2015-06-17 22:52:17,635
>>> HintedHandOffManager.java (line 296) Started hinted handoff for host:
>>> f7b7ab10-4d42-4f0c-af92-2934a075bee3 with IP: /192.168.1.108
>>>  INFO [HintedHandoff:1] 2015-06-17 22:52:17,643
>>> HintedHandOffManager.java (line 422) Timed out replaying hints to /
>>> 192.168.1.119; aborting (0 delivered)
>>>  INFO [HintedHandoff:1] 2015-06-17 22:52:17,643
>>> HintedHandOffManager.java (line 296) Started hinted handoff for host:
>>> ddb79f35-3e2b-4be8-84d8-7942086e2b73 with IP: /192.168.1.104
>>>  INFO [HintedHandoff:2] 2015-06-17 22:52:27,143
>>> HintedHandOffManager.java (line 422) Timed out replaying hints to /
>>> 192.168.1.108; aborting (0 delivered)
>>>  INFO [HintedHandoff:2] 2015-06-17 22:52:27,144
>>> HintedHandOffManager.java (line 296) Started hinted handoff for host:
>>> 6a2fa431-4a51-44cb-af19-1991c960e075 with IP: /192.168.1.117
>>>  INFO [HintedHandoff:1] 2015-06-17 22:52:27,153
>>> HintedHandOffManager.java (line 422) Timed out replaying hints to /
>>> 192.168.1.104; aborting (0 delivered)
>>>  INFO [HintedHandoff:1] 2015-06-17 22:52:27,154
>>> HintedHandOffManager.java (line 296) Started hinted handoff for host:
>>> cf03174a-533c-44d6-a679-e70090ad2bc5 with IP: /192.168.1.107
>>>
>>> 
>>>
>>> Thanks
>>> -shashi..
>>>
>>
>>
>

joining a node caused loads on some existing nodes to skyrocket

[KZ Win]

We had six node clusters and when we attempted to join a node to this, cpu
load on two gradually climbed to abnormally high number.   Stopping the
join and shutting down cassandra on two high-load nodes restored the
cluster health (we have RF=3)

Anyone have any insight on this cassandra behavior?  We have done node join
many times before; most recent was just 4 days before.  The

The following unusual messages in the relevant time period for two nodes.
We are using cassandra 2.0.10


Jun 30 16:47:30 cass-22.pelotime.com cassandra-serverERROR [GossipStage:1]
CassandraDaemon.java (line 199) Exception in thread
Thread[GossipStage:1,5,main]

Jun 30 16:47:30 cass-22.pelotime.com java.lang.NullPointerException

Jun 30 16:47:30 cass-22.pelotime.com at
org.apache.cassandra.gms.Gossiper.convict(Gossiper.java:301)

Jun 30 16:47:30 cass-22.pelotime.com at
org.apache.cassandra.gms.FailureDetector.forceConviction(FailureDetector.java:251)

Jun 30 16:47:30 cass-22.pelotime.com at
org.apache.cassandra.gms.GossipShutdownVerbHandler.doVerb(GossipShutdownVerbHandler.java:37)

Jun 30 16:47:30 cass-22.pelotime.com at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)

Jun 30 16:47:30 cass-22.pelotime.com at
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

Jun 30 16:47:30 cass-22.pelotime.com at
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

Jun 30 16:47:30 cass-22.pelotime.com at java.lang.Thread.run(Unknown
Source)

Jun 30 16:47:30 cass-22.pelotime.com cassandra-server INFO [GossipStage:2]
Gossiper.java (line 910) Node /10.0.251.77 is now part of the cluster

Jun 30 16:47:35 cass-22.pelotime.com cassandra-server INFO [HANDSHAKE-/
10.0.251.77] OutboundTcpConnection.java (line 386) Handshaking version with
/10.0.251.77

Jun 30 16:47:35 cass-22.pelotime.com cassandra-server INFO
[RequestResponseStage:138] Gossiper.java (line 876) InetAddress /10.0.251.77
is now UP

Jun 30 16:47:38 cass-22.pelotime.com cassandra-server INFO [GossipStage:2]
Gossiper.java (line 890) InetAddress /10.0.251.77 is now DOWN

Jun 30 16:48:02 cass-22.pelotime.com cassandra-server INFO [HANDSHAKE-/
10.0.251.77] OutboundTcpConnection.java (line 386) Handshaking version with
/10.0.251.77

Jun 30 16:48:05 cass-22.pelotime.com cassandra-server INFO [GossipTasks:1]
Gossiper.java (line 658) FatClient /10.0.251.77 has been silent for
3ms, removing from gossip

Jun 30 16:48:05 cass-22.pelotime.com cassandra-server INFO [HANDSHAKE-/
10.0.251.77] OutboundTcpConnection.java (line 386) Handshaking ve



Jun 30 16:48:59 cass-24.pelotime.com cassandra-server INFO [HANDSHAKE-/
10.0.251.77] OutboundTcpConnection.java (line 386) Handshaking version with
/10.0.251.77

Jun 30 16:48:59 cass-24.pelotime.com cassandra-server INFO
[RequestResponseStage:26] Gossiper.java (line 876) InetAddress /10.0.251.77
is now UP

Jun 30 16:48:59 cass-24.pelotime.com cassandra-server INFO [HANDSHAKE-/
10.0.251.77] OutboundTcpConnection.java (line 386) Handshaking version with
/10.0.251.77

Jun 30 16:50:52 cass-24.pelotime.com cassandra-serverERROR [STREAM-OUT-/
10.0.251.77] StreamSession.java (line 454) [Stream
#5f2251e0-1f69-11e5-94c0-d9033a25abe9] Streaming error occurred

Jun 30 16:50:52 cass-24.pelotime.com java.io.IOException: Broken pipe

Jun 30 16:50:52 cass-24.pelotime.com at
sun.nio.ch.FileChannelImpl.transferTo0(Native Method)

Jun 30 16:50:52 cass-24.pelotime.com at
sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)

Jun 30 16:50:52 cass-24.pelotime.com at
sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)

Jun 30 16:50:52 cass-24.pelotime.com at
org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:74)

Jun 30 16:50:52 cass-24.pelotime.com at
org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:59)

Jun 30 16:50:52 cass-24.pelotime.com at
org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42)

Jun 30 16:50:52 cass-24.pelotime.com at
org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)

Jun 30 16:50:52 cass-24.pelotime.com at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339)

Jun 30 16:50:52 cass-24.pelotime.com at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:319)

Jun 30 16:50:52 cass-24.pelotime.com at java.lang.Thread.run(Unknown
Source)

Jun 30 16:50:52 cass-24.pelotime.com cassandra-serverERROR [STREAM-OUT-/
10.0.251.77] StreamSession.java (line 454) [Stream
#5f2251e0-1f69-11e5-94c0-d9033a25abe9] Streaming error occurred

Jun 30 16:50:52 cass-24.pelotime.com java.io.IOException: Broken pipe

Jun 30 16:50:52 cass-24.pelotime.com at
sun.nio.ch.FileDispatcherImpl.write0(Native Method)

Jun 30 16:50:52 cass-24.pelotime.com at
sun.nio.ch.SocketDi

What are problems with schema disagreement

[John Wong]

Hi.

Here is a schema disagreement we encountered.
Schema versions:
b6467059-5897-3cc1-9ee2-73f31841b0b0: [10.0.1.100, 10.0.1.109]
c8971b2d-0949-3584-aa87-0050a4149bbd: [10.0.1.55, 10.0.1.16,
10.0.1.77]
c733920b-2a31-30f0-bca1-45a8c9130a2c: [10.0.1.221]

We deployed an application which would send a schema update (DDL=auto). We
found this prod cluster had 3 schema difference. Other existing
applications were fine, so some people were curious what if we left this
problem alone until off hours.

Is there any concerns with not resolve schema disagreement right away? FWIW
we went ahead and restarted 221 first, and continue with the rest of the
minors.

Thanks.

John

Re: What are problems with schema disagreement

[graham sanderson]

What version of C* are you running? Some versions of 2.0.x might occasionally 
fail to propagate schema changes in a timely fashion (though they would fix 
themselves eventually - in the order of a few minutes)

> On Jul 2, 2015, at 9:37 PM, John Wong  wrote:
> 
> Hi.
> 
> Here is a schema disagreement we encountered.
> Schema versions:
> b6467059-5897-3cc1-9ee2-73f31841b0b0: [10.0.1.100, 10.0.1.109]
> c8971b2d-0949-3584-aa87-0050a4149bbd: [10.0.1.55, 10.0.1.16, 
> 10.0.1.77]
> c733920b-2a31-30f0-bca1-45a8c9130a2c: [10.0.1.221]
> 
> We deployed an application which would send a schema update (DDL=auto). We 
> found this prod cluster had 3 schema difference. Other existing applications 
> were fine, so some people were curious what if we left this problem alone 
> until off hours.
> 
> Is there any concerns with not resolve schema disagreement right away? FWIW 
> we went ahead and restarted 221 first, and continue with the rest of the 
> minors.
> 
> Thanks.
> 
> John
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: What are problems with schema disagreement

[John Wong]

On Thu, Jul 2, 2015 at 11:01 PM, graham sanderson  wrote:

> What version of C* are you running? Some versions of 2.0.x might
> occasionally fail to propagate schema changes in a timely fashion (though
> they would fix themselves eventually - in the order of a few minutes)
>
>
Hi Graham. Thanks. We are still running on 1.2.16, but we do plan to
upgrade in the near future. The load on the cluster at the time was very
very low. All nodes were responsive, except nothing was show up in the logs
after certain time, which led me to believe something happened internal,
although that was a poor wild guess.

But is it safe to be okay with schema disagreement? I worry about data
consistency if I let it sit too long.

Thanks.

John

> > On Jul 2, 2015, at 9:37 PM, John Wong  wrote:
> >
> > Hi.
> >
> > Here is a schema disagreement we encountered.
> > Schema versions:
> > b6467059-5897-3cc1-9ee2-73f31841b0b0: [10.0.1.100, 10.0.1.109]
> > c8971b2d-0949-3584-aa87-0050a4149bbd: [10.0.1.55, 10.0.1.16,
> 10.0.1.77]
> > c733920b-2a31-30f0-bca1-45a8c9130a2c: [10.0.1.221]
> >
> > We deployed an application which would send a schema update (DDL=auto).
> We found this prod cluster had 3 schema difference. Other existing
> applications were fine, so some people were curious what if we left this
> problem alone until off hours.
> >
> > Is there any concerns with not resolve schema disagreement right away?
> FWIW we went ahead and restarted 221 first, and continue with the rest of
> the minors.
> >
> > Thanks.
> >
> > John
> >
>
>