Re: Increasing size of "Batch of prepared statements"

[shahab]

Thanks Jens for the comments.

As I am trying "cassandra stress tool", does it mean that the tool is
executing batch of "Insert" statements (probably hundreds, or thousands)
 to the cassandra (for the sake of stressing Cassnadra ?

best,
/Shahab

On Wed, Oct 22, 2014 at 8:14 PM, Jens Rantil  wrote:

> Shabab,
>
> Apologize for the late answer.
>
> On Mon, Oct 6, 2014 at 2:38 PM, shahab  wrote:
>
>> But do you mean that inserting columns with large size (let's say a text
>> with 20-30 K) is potentially problematic in Cassandra?
>
>
> AFAIK, the size _warning_ you are getting relates to the size of the batch
> of prepared statements (INSERT INTO mykeyspace.mytable VALUES (?,?,?,?)).
> That is, it has nothing to do with the actual content of your row. 20-30 K
> shouldn't be a problem. But it's considered good practise to split larger
> files (maybe > 5 MB into chunks) since it makes operations easier to your
> cluster more likely to spread more evenly across cluster.
>
>
>> What shall i do if I want columns with large size?
>
>
> Just don't insert to many rows in a single batch and you should be fine.
> Like Shane's JIRA ticket said, the warning is to let you know you are not
> following best practice when adding too many rows in a single batch. It can
> create bottlenecks in a single Cassandra node.
>
> Cheers,
> Jens
>
> --
> Jens Rantil
> Backend engineer
> Tink AB
>
> Email: jens.ran...@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
> Facebook  Linkedin
> 
>  Twitter 
>

Multi Datacenter / MultiRegion on AWS Best practice ?

[Alain RODRIGUEZ]

Hi,

We are currently wondering about the best way to configure network
architecture to have a Cassandra cluster multi DC.

Reading previous messages on this mailing list, I see 2 main ways to do
this:

1 - 2 private VPC, joined by a VPN tunnel linking 2 regions. C* using
EC2Snitch (or PropertyFileSnitch) and private IPs.
2 - 2 public VPC. C* using EC2MultiRegionSnitch (and so public IPs for
seeds and broadcast, private for listen address).

On solution one we are not confident on VPN tunnel about stability and
performances, the rest should work just fine.

On solution 2, we would need to open IPs one by one on 3 ports (7000, 9042,
9160) at least. 100 entries in a security group would allow us to have a
maximum of ~30 nodes. An other issuer is that a ring describe (using
astyanax let's say) would also give to clients public IPs, our clients
which are also inside the VPC, would have to go to the internet before
coming back to VPC, creating unnecessary latencies.

What are your advices regarding best practices for a multiDC (cross region)
inside AWS cloud ?

And by the way, how to configure Astyanax when using EC2MultiRegionSnitch
(and public IP for broadcasting) to use private IPs instead of public ones ?

Alain

Operating on large cluster

[Alain RODRIGUEZ]

Hi,

I was wondering about how do you guys handle a large cluster (50+ machines).

I mean there is sometime you need to change configuration (cassandra.yaml)
or send a command to one, some or all nodes (cleanup, upgradesstables,
setstramthoughput or whatever).

So far we have been using things like custom scripts for repairs or any
routine maintenance and cssh for specific and one shot actions on the
cluster. But I guess this doesn't really scale, I guess we coul use pssh
instead. For configuration changes we use Capistrano that might scale
properly.

So I would like to known, what are the methods that operators use on large
cluster out there ? Have some of you built some open sourced "cluster
management" interfaces or scripts that could make things easier while
operating on large Cassandra clusters ?

Alain

Re: Operating on large cluster

[Jens Rantil]

Hi,


While I am nowhere close to 50+ machines I've been using Saltstack for both 
configuration management as well as remote execution. I has worked great for me 
and supposedly scales to 1000+ machines.




Cheers,

Jens


—
Sent from Mailbox

On Thu, Oct 23, 2014 at 11:18 AM, Alain RODRIGUEZ 
wrote:

> Hi,
> I was wondering about how do you guys handle a large cluster (50+ machines).
> I mean there is sometime you need to change configuration (cassandra.yaml)
> or send a command to one, some or all nodes (cleanup, upgradesstables,
> setstramthoughput or whatever).
> So far we have been using things like custom scripts for repairs or any
> routine maintenance and cssh for specific and one shot actions on the
> cluster. But I guess this doesn't really scale, I guess we coul use pssh
> instead. For configuration changes we use Capistrano that might scale
> properly.
> So I would like to known, what are the methods that operators use on large
> cluster out there ? Have some of you built some open sourced "cluster
> management" interfaces or scripts that could make things easier while
> operating on large Cassandra clusters ?
> Alain

Re: Increasing size of "Batch of prepared statements"

[Jens Rantil]

Hi again Shabab,




Yes, it seems that way. I have no experience with the “cassandra stress tool”, 
but wouldn’t be surprised if the batch size could be tweaked.




Cheers,

Jens


———
Jens Rantil
Backend engineer
Tink AB

Email: jens.ran...@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook Linkedin Twitter

On Thu, Oct 23, 2014 at 10:00 AM, shahab  wrote:

> Thanks Jens for the comments.
> As I am trying "cassandra stress tool", does it mean that the tool is
> executing batch of "Insert" statements (probably hundreds, or thousands)
>  to the cassandra (for the sake of stressing Cassnadra ?
> best,
> /Shahab
> On Wed, Oct 22, 2014 at 8:14 PM, Jens Rantil  wrote:
>> Shabab,
>>
>> Apologize for the late answer.
>>
>> On Mon, Oct 6, 2014 at 2:38 PM, shahab  wrote:
>>
>>> But do you mean that inserting columns with large size (let's say a text
>>> with 20-30 K) is potentially problematic in Cassandra?
>>
>>
>> AFAIK, the size _warning_ you are getting relates to the size of the batch
>> of prepared statements (INSERT INTO mykeyspace.mytable VALUES (?,?,?,?)).
>> That is, it has nothing to do with the actual content of your row. 20-30 K
>> shouldn't be a problem. But it's considered good practise to split larger
>> files (maybe > 5 MB into chunks) since it makes operations easier to your
>> cluster more likely to spread more evenly across cluster.
>>
>>
>>> What shall i do if I want columns with large size?
>>
>>
>> Just don't insert to many rows in a single batch and you should be fine.
>> Like Shane's JIRA ticket said, the warning is to let you know you are not
>> following best practice when adding too many rows in a single batch. It can
>> create bottlenecks in a single Cassandra node.
>>
>> Cheers,
>> Jens
>>
>> --
>> Jens Rantil
>> Backend engineer
>> Tink AB
>>
>> Email: jens.ran...@tink.se
>> Phone: +46 708 84 18 32
>> Web: www.tink.se
>>
>> Facebook  Linkedin
>> 
>>  Twitter 
>>

Re: Increasing size of "Batch of prepared statements"

[shahab]

OK, Thanks again Jens.

best,
/Shahab

On Thu, Oct 23, 2014 at 1:22 PM, Jens Rantil  wrote:

> Hi again Shabab,
>
> Yes, it seems that way. I have no experience with the “cassandra stress
> tool”, but wouldn’t be surprised if the batch size could be tweaked.
>
> Cheers,
> Jens
>
> ——— Jens Rantil Backend engineer Tink AB Email: jens.ran...@tink.se
> Phone: +46 708 84 18 32 Web: www.tink.se Facebook Linkedin Twitter
>
>
> On Thu, Oct 23, 2014 at 10:00 AM, shahab  wrote:
>
>> Thanks Jens for the comments.
>>
>> As I am trying "cassandra stress tool", does it mean that the tool is
>> executing batch of "Insert" statements (probably hundreds, or thousands)
>>  to the cassandra (for the sake of stressing Cassnadra ?
>>
>> best,
>> /Shahab
>>
>> On Wed, Oct 22, 2014 at 8:14 PM, Jens Rantil  wrote:
>>
>>>  Shabab,
>>>
>>> Apologize for the late answer.
>>>
>>> On Mon, Oct 6, 2014 at 2:38 PM, shahab  wrote:
>>>
 But do you mean that inserting columns with large size (let's say a
 text with 20-30 K) is potentially problematic in Cassandra?
>>>
>>>
>>> AFAIK, the size _warning_ you are getting relates to the size of the
>>> batch of prepared statements (INSERT INTO mykeyspace.mytable VALUES
>>> (?,?,?,?)). That is, it has nothing to do with the actual content of
>>> your row. 20-30 K shouldn't be a problem. But it's considered good practise
>>> to split larger files (maybe > 5 MB into chunks) since it makes operations
>>> easier to your cluster more likely to spread more evenly across cluster.
>>>
>>>
 What shall i do if I want columns with large size?
>>>
>>>
>>> Just don't insert to many rows in a single batch and you should be fine.
>>> Like Shane's JIRA ticket said, the warning is to let you know you are not
>>> following best practice when adding too many rows in a single batch. It can
>>> create bottlenecks in a single Cassandra node.
>>>
>>> Cheers,
>>> Jens
>>>
>>> --
>>> Jens Rantil
>>> Backend engineer
>>> Tink AB
>>>
>>> Email: jens.ran...@tink.se
>>> Phone: +46 708 84 18 32
>>> Web: www.tink.se
>>>
>>>  Facebook  Linkedin
>>> 
>>>  Twitter 
>>>
>>
>>
>

Cassandra Node Commissioning

[Aravindan T]

Hi

 

We are facing several problems during commissioning of new nodes to the 
existing cluster. The existing cluster(5 nodes) is holding data of 13 TB and 
daily 0.1 TB of data will be loaded.Ten days back,we started adding 5 nodes. In 
the middle of the commissioning process, the bootstrap process is getting 
failed many times, displaying an error STREAM FAILED in the new node and BROKEN 
PIPE in the old node. Whenever we face this problem, we are restarting the new 
node. Heare are the few questions about the node joining process.

 

1) Whenever we are restarting the node to join, will the bootstrap process 
resume or restart from the begining? If so, should we wipe off the data 
directories and do a fresh restart.

2) How much time it might take to complete the node joining process.( Network 
bandwidth :1Gbps) ?

3) Can we add a node directly by setting auto_bootstrap to false and run a 
nodetool repair in that node?

4) How to monitor the percentage of load balancing in each node?

5) Can we increase the streaming speed by using nodetool setstreamthroughput 
property? What is the difference between 
stream_throughput_outbound_megabits_per_sec property in cassandra.yaml file and 
nodetool setsreamthroughput property?

6) Can we scp some data from old nodes to new nodes and do a restart?


Regards,
Aravindan Thangavelu

=-=-=
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you

Re: Empty cqlsh cells vs. null

[DuyHai Doan]

Hello Jens

What do you mean by "cqlsh explicitely writes 'null' in those cells" ?  Are
you seing textual value "null" written in the cells ?


 Null in CQL can have 2 meanings:

1. the column did not exist (or more precisely, has never been created)
2. the column did exist sometimes in the past (has been created) but then
has been deleted (tombstones)



On Thu, Oct 23, 2014 at 8:37 AM, Jens Rantil  wrote:

>  Hi,
>
> Not sure this is a Datastax specific question to be asked elsewhere. In
> that case, let me know.
>
> Anyway, I have populated a Cassandra table from DSE Hive. When I fire up
> cqlsh and execute a SELECT against the table I have columns of INT type
> that are empty. At first I thought these were null, but it turns out that
> cqlsh explicitly writes "null" in those cells. What can I make of this? A
> bug in Hive serialization to Cassandra?
>
> Cheers,
> Jens
>
> —
> Sent from Mailbox 
>

Re: Empty cqlsh cells vs. null

[Adam Holmberg]

'null' is how cqlsh displays empty cells:
https://github.com/apache/cassandra/blob/trunk/pylib/cqlshlib/formatting.py#L47-L58

On Thu, Oct 23, 2014 at 9:36 AM, DuyHai Doan  wrote:

> Hello Jens
>
> What do you mean by "cqlsh explicitely writes 'null' in those cells" ?
> Are you seing textual value "null" written in the cells ?
>
>
>  Null in CQL can have 2 meanings:
>
> 1. the column did not exist (or more precisely, has never been created)
> 2. the column did exist sometimes in the past (has been created) but then
> has been deleted (tombstones)
>
>
>
> On Thu, Oct 23, 2014 at 8:37 AM, Jens Rantil  wrote:
>
>>  Hi,
>>
>> Not sure this is a Datastax specific question to be asked elsewhere. In
>> that case, let me know.
>>
>> Anyway, I have populated a Cassandra table from DSE Hive. When I fire up
>> cqlsh and execute a SELECT against the table I have columns of INT type
>> that are empty. At first I thought these were null, but it turns out that
>> cqlsh explicitly writes "null" in those cells. What can I make of this? A
>> bug in Hive serialization to Cassandra?
>>
>> Cheers,
>> Jens
>>
>> —
>> Sent from Mailbox 
>>
>
>

Re: frequently update/read table and level compaction

[DuyHai Doan]

#1
 There are 2 levels of JMX metrics: the one for each table and the one
related to the StorageProxy. Depending on each "readcount" you're looking
at, the meaning can be different, watch this video for more explanation:
https://www.youtube.com/watch?v=w6aD4vAY_a8&index=14&list=PLqcm6qE9lgKJkxYZUOIykswDndrOItnn2

#2 Same answer as above, it depends on each level you're looking at

#3
It means that on a given level, each partition fit entirely in 1 SSTable.
Because of that, you cannot have a partition whose head is in 1 SSTable and
a tail in another SSTable. Consequently where you're fetching a range of
"CQL rows" from a partition, Cassandra only needs to touch 1 physical
SSTable on disk and read sequentially from the beginning of the partition.
With STCS the partition may span on several SSTables so it will cost time
to touch many files.

#4
It depends really on your data model. How did you perform you "changes", do
you just issue "UPDATE" statements frequently or do you use timeseries
pattern ? The first scenario may cause performance issue because before
compaction arrives, you will force C* to load a bunch of "old" values in
memory and filter all of them to return the last one. Have a look at those
slides:
http://www.slideshare.net/doanduyhai/cassandra-best-practices-and-worst-anti-patterns-meetup-in-germany
(slides 23-28)


#5
Again, no universal answer, all depend on your insertion rate/update
frequency/data access pattern. Beware of the fact that leveled compaction
eats a lot of your disk I/O. You'd better have enough I/O bandwidth
otherwise the result could be worse than SizeTiered.

 In theory, if your I/O can keep up with Leveled compaction, the read
performance will be steady

 Hope that it helps


On Tue, Oct 21, 2014 at 5:24 AM, Jimmy Lin  wrote:

> Hi,
> I have a column family/ table that has frequent update on one of the
> column, and one column that has infrequent update. Rest of the columns
> never changed. Our application also read frequently on this table.
>
> We have seen some read latency issue on this table and plan to switch to
> use level compaction on this table. Few questions:
>
> #1
> In Cassandra Server JMX, there is "readcount" attribute, what is consider
> a read? accessing a row will consider a read count?
>
> #2
> For JMX "read latency", does it include consistency level (fetching data
> from other nodes) or coordinator related work load?
>
> #3
> From the doc, level compaction stated it will guarantee that all sstables
> in same level are 'non-overlapping", what does it really mean? (trying to
> visualize how this can reduce read latency)
>
> #4
> If I change my select CQL query not to include the frequently changed
> column, will that improve read latency?
>
> #5
> How significant of the improvement of change compaction from sized to
> level? is  it day and night difference? and it is true that while sized
> compaction latency will get worse, level compaction can give very
> consistent read latency for long long time?
>
>
> Thanks
>
>

Re: Increasing size of "Batch of prepared statements"

[Tyler Hobbs]

CASSANDRA-8091 (Stress tool creates too large batches) is relevant:
https://issues.apache.org/jira/browse/CASSANDRA-8091

On Thu, Oct 23, 2014 at 6:28 AM, shahab  wrote:

> OK, Thanks again Jens.
>
> best,
> /Shahab
>
> On Thu, Oct 23, 2014 at 1:22 PM, Jens Rantil  wrote:
>
>> Hi again Shabab,
>>
>> Yes, it seems that way. I have no experience with the “cassandra stress
>> tool”, but wouldn’t be surprised if the batch size could be tweaked.
>>
>> Cheers,
>> Jens
>>
>> ——— Jens Rantil Backend engineer Tink AB Email: jens.ran...@tink.se
>> Phone: +46 708 84 18 32 Web: www.tink.se Facebook Linkedin Twitter
>>
>>
>> On Thu, Oct 23, 2014 at 10:00 AM, shahab  wrote:
>>
>>> Thanks Jens for the comments.
>>>
>>> As I am trying "cassandra stress tool", does it mean that the tool is
>>> executing batch of "Insert" statements (probably hundreds, or thousands)
>>>  to the cassandra (for the sake of stressing Cassnadra ?
>>>
>>> best,
>>> /Shahab
>>>
>>> On Wed, Oct 22, 2014 at 8:14 PM, Jens Rantil 
>>> wrote:
>>>
  Shabab,

 Apologize for the late answer.

 On Mon, Oct 6, 2014 at 2:38 PM, shahab  wrote:

> But do you mean that inserting columns with large size (let's say a
> text with 20-30 K) is potentially problematic in Cassandra?


 AFAIK, the size _warning_ you are getting relates to the size of the
 batch of prepared statements (INSERT INTO mykeyspace.mytable VALUES
 (?,?,?,?)). That is, it has nothing to do with the actual content of
 your row. 20-30 K shouldn't be a problem. But it's considered good practise
 to split larger files (maybe > 5 MB into chunks) since it makes operations
 easier to your cluster more likely to spread more evenly across cluster.


> What shall i do if I want columns with large size?


 Just don't insert to many rows in a single batch and you should be
 fine. Like Shane's JIRA ticket said, the warning is to let you know you are
 not following best practice when adding too many rows in a single batch. It
 can create bottlenecks in a single Cassandra node.

 Cheers,
 Jens

 --
 Jens Rantil
 Backend engineer
 Tink AB

 Email: jens.ran...@tink.se
 Phone: +46 708 84 18 32
 Web: www.tink.se

  Facebook  Linkedin
 
  Twitter 

>>>
>>>
>>
>


-- 
Tyler Hobbs
DataStax 

Re: Operating on large cluster

[Ranjib Dey]

We use chef for configuration management and blender for on demand jobs

https://github.com/opscode/chef
https://github.com/PagerDuty/blender
 On Oct 23, 2014 2:18 AM, "Alain RODRIGUEZ"  wrote:

> Hi,
>
> I was wondering about how do you guys handle a large cluster (50+
> machines).
>
> I mean there is sometime you need to change configuration (cassandra.yaml)
> or send a command to one, some or all nodes (cleanup, upgradesstables,
> setstramthoughput or whatever).
>
> So far we have been using things like custom scripts for repairs or any
> routine maintenance and cssh for specific and one shot actions on the
> cluster. But I guess this doesn't really scale, I guess we coul use pssh
> instead. For configuration changes we use Capistrano that might scale
> properly.
>
> So I would like to known, what are the methods that operators use on large
> cluster out there ? Have some of you built some open sourced "cluster
> management" interfaces or scripts that could make things easier while
> operating on large Cassandra clusters ?
>
> Alain
>

Re: Increasing size of "Batch of prepared statements"

[shahab]

Thanks Tyler for sharing this. It is exactly what I was looking for to know.

best,
/Shahab

On Thu, Oct 23, 2014 at 5:37 PM, Tyler Hobbs  wrote:

> CASSANDRA-8091 (Stress tool creates too large batches) is relevant:
> https://issues.apache.org/jira/browse/CASSANDRA-8091
>
> On Thu, Oct 23, 2014 at 6:28 AM, shahab  wrote:
>
>> OK, Thanks again Jens.
>>
>> best,
>> /Shahab
>>
>> On Thu, Oct 23, 2014 at 1:22 PM, Jens Rantil  wrote:
>>
>>> Hi again Shabab,
>>>
>>> Yes, it seems that way. I have no experience with the “cassandra stress
>>> tool”, but wouldn’t be surprised if the batch size could be tweaked.
>>>
>>> Cheers,
>>> Jens
>>>
>>> ——— Jens Rantil Backend engineer Tink AB Email: jens.ran...@tink.se
>>> Phone: +46 708 84 18 32 Web: www.tink.se Facebook Linkedin Twitter
>>>
>>>
>>> On Thu, Oct 23, 2014 at 10:00 AM, shahab 
>>> wrote:
>>>
 Thanks Jens for the comments.

 As I am trying "cassandra stress tool", does it mean that the tool is
 executing batch of "Insert" statements (probably hundreds, or thousands)
  to the cassandra (for the sake of stressing Cassnadra ?

 best,
 /Shahab

 On Wed, Oct 22, 2014 at 8:14 PM, Jens Rantil 
 wrote:

>  Shabab,
>
> Apologize for the late answer.
>
> On Mon, Oct 6, 2014 at 2:38 PM, shahab 
> wrote:
>
>> But do you mean that inserting columns with large size (let's say a
>> text with 20-30 K) is potentially problematic in Cassandra?
>
>
> AFAIK, the size _warning_ you are getting relates to the size of the
> batch of prepared statements (INSERT INTO mykeyspace.mytable VALUES
> (?,?,?,?)). That is, it has nothing to do with the actual content of
> your row. 20-30 K shouldn't be a problem. But it's considered good 
> practise
> to split larger files (maybe > 5 MB into chunks) since it makes operations
> easier to your cluster more likely to spread more evenly across cluster.
>
>
>> What shall i do if I want columns with large size?
>
>
> Just don't insert to many rows in a single batch and you should be
> fine. Like Shane's JIRA ticket said, the warning is to let you know you 
> are
> not following best practice when adding too many rows in a single batch. 
> It
> can create bottlenecks in a single Cassandra node.
>
> Cheers,
> Jens
>
> --
> Jens Rantil
> Backend engineer
> Tink AB
>
> Email: jens.ran...@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
>  Facebook  Linkedin
> 
>  Twitter 
>


>>>
>>
>
>
> --
> Tyler Hobbs
> DataStax 
>

Re: are repairs in 2.0 more expensive than in 1.2

[Sean Bridges]

We switched to to parallel repairs, and now our repairs in 2.0 are behaving
like the repairs in 1.2.

The change from parallel to sequential is very dramatic.  For a small
cluster with 3 nodes, using cassandra 2.0.10,  a parallel repair takes 2
hours, and io throughput peaks at 6 mb/s.  Sequential repair takes 40
hours, with average io around 27 mb/s.  Should I file a jira?

Sean

On Wed, Oct 15, 2014 at 9:23 PM, Sean Bridges 
wrote:

> Thanks Robert.  Does the switch to sequential from parallel explain why IO
> increases, we see significantly higher IO with 2.10.
>
> The nodetool docs [1] hint at the reason for defaulting to sequential,
>
> "This allows the dynamic snitch to maintain performance for your
> application via the other replicas, because at least one replica in the
> snapshot is not undergoing repair."
>
> Sean
>
> [1]
> http://www.datastax.com/documentation/cassandra/2.0/cassandra/tools/toolsRepair.html
>
>
> On Wed, Oct 15, 2014 at 5:36 PM, Robert Coli  wrote:
>
>> On Wed, Oct 15, 2014 at 4:54 PM, Sean Bridges 
>> wrote:
>>
>>> We upgraded a cassandra cluster from 1.2.18 to 2.0.10, and it looks like
>>> repair is significantly more expensive now.  Is this expected?
>>>
>>
>> It depends on what you mean by "expected." Operators usually don't expect
>> defaults with such dramatic impacts to change without them understanding
>> why, but there is a reason for it.
>>
>> In 2.0 the default for repair was changed to be non-parallel. To get the
>> old behavior, you need to supply -par as an argument.
>>
>> The context-free note you missed the significance of in NEWS.txt for
>> version 2.0.2 says :
>>
>> - Nodetool defaults to Sequential mode for repair operations
>>
>> What this doesn't say is how almost certainly unreasonable this is as a
>> default, because this means that repair is predictably slower in direct
>> relationship to your replication factor, and the default for
>> gc_grace_seconds (the time box in which one must complete a repair) did not
>> change at the same time. The ticket where the change happens [1] does not
>> specify a rationale, so your guess is as good as mine as to the reasoning
>> which not only felt the change was necessary but reasonable.
>>
>> Leaving aside the problem you've encountered ("upgraders notice that
>> their repairs (which already took forever) are suddenly WAY SLOWER") this
>> default is also quite pathological for anyone operating with a RF over 3,
>> which are valid, if very uncommon, configurations.
>>
>> In summary, if, as an operator, you disagree that making repair slower by
>> default as a factor of replication factor is reasonable, I suggest filing a
>> JIRA and letting the project know. At least in that case there is a chance
>> they might explain the rationale for so blithely making a change that has
>> inevitable impact on operators... ?
>>
>> =Rob
>> [1] https://issues.apache.org/jira/browse/CASSANDRA-5950
>> http://twitter.com/rcolidba
>>
>
>

Re: are repairs in 2.0 more expensive than in 1.2

[Robert Coli]

On Thu, Oct 23, 2014 at 9:33 AM, Sean Bridges 
wrote:

> The change from parallel to sequential is very dramatic.  For a small
> cluster with 3 nodes, using cassandra 2.0.10,  a parallel repair takes 2
> hours, and io throughput peaks at 6 mb/s.  Sequential repair takes 40
> hours, with average io around 27 mb/s.  Should I file a jira?
>

As you are an actual user actually encountering the problem I had only
conjectured about, you are the person best suited to file such a ticket on
the reasonableness of the -par default. :D

=Rob
http://twitter.com/rcolidba

Re: Operating on large cluster

[Michael Shuler]

On 10/23/2014 04:18 AM, Alain RODRIGUEZ wrote:

I was wondering about how do you guys handle a large cluster (50+ machines).


Configuration management tools are awesome, until they aren't. Having 
used or played with all the popular ones, and having been bitten by 
failures of those tools on large clusters, my long-time preference has 
been using a VCS to check configs and scripts in/out and parallel ssh 
(whichever one you like). Simple is good. If you don't deeply understand 
the config management system you have chosen, the unexpected may(will?) 
eventually happen. To all the servers at once.


Even when you are careful, we are human. No tool can prevent *all* 
mistakes. Test everything in a staging environment, first!


--
Kind regards,
Michael

PS. even staging doesn't prevent fallibility.. :)
https://twitter.com/mshuler/status/520667739615395840

Re: Operating on large cluster

[Eric Plowe]

I am a big fan of perl-ssh-tools (https://github.com/tobert/perl-ssh-tools)
to let me manage my nodes and SVN to store configs.

~Eric Plowe


On Thu, Oct 23, 2014 at 3:07 PM, Michael Shuler 
wrote:

> On 10/23/2014 04:18 AM, Alain RODRIGUEZ wrote:
>
>> I was wondering about how do you guys handle a large cluster (50+
>> machines).
>>
>
> Configuration management tools are awesome, until they aren't. Having used
> or played with all the popular ones, and having been bitten by failures of
> those tools on large clusters, my long-time preference has been using a VCS
> to check configs and scripts in/out and parallel ssh (whichever one you
> like). Simple is good. If you don't deeply understand the config management
> system you have chosen, the unexpected may(will?) eventually happen. To all
> the servers at once.
>
> Even when you are careful, we are human. No tool can prevent *all*
> mistakes. Test everything in a staging environment, first!
>
> --
> Kind regards,
> Michael
>
> PS. even staging doesn't prevent fallibility.. :)
> https://twitter.com/mshuler/status/520667739615395840
>

Re: Operating on large cluster

[Roni Balthazar]

Hi,

We use Puppet to manage our Cassandra configuration. (http://puppetlabs.com)

You can use Cluster SSH to send commands to the server as well.

Another good choice is Saltstack.

Regards,

Roni

On Thu, Oct 23, 2014 at 5:18 AM, Alain RODRIGUEZ  wrote:

> Hi,
>
> I was wondering about how do you guys handle a large cluster (50+
> machines).
>
> I mean there is sometime you need to change configuration (cassandra.yaml)
> or send a command to one, some or all nodes (cleanup, upgradesstables,
> setstramthoughput or whatever).
>
> So far we have been using things like custom scripts for repairs or any
> routine maintenance and cssh for specific and one shot actions on the
> cluster. But I guess this doesn't really scale, I guess we coul use pssh
> instead. For configuration changes we use Capistrano that might scale
> properly.
>
> So I would like to known, what are the methods that operators use on large
> cluster out there ? Have some of you built some open sourced "cluster
> management" interfaces or scripts that could make things easier while
> operating on large Cassandra clusters ?
>
> Alain
>

Re: are repairs in 2.0 more expensive than in 1.2

[Janne Jalkanen]

On 23 Oct 2014, at 21:29 , Robert Coli  wrote:

> On Thu, Oct 23, 2014 at 9:33 AM, Sean Bridges  wrote:
> The change from parallel to sequential is very dramatic.  For a small cluster 
> with 3 nodes, using cassandra 2.0.10,  a parallel repair takes 2 hours, and 
> io throughput peaks at 6 mb/s.  Sequential repair takes 40 hours, with 
> average io around 27 mb/s.  Should I file a jira?
> 
> As you are an actual user actually encountering the problem I had only 
> conjectured about, you are the person best suited to file such a ticket on 
> the reasonableness of the -par default. :D

Hm?  I’ve been banging my head against the exact same problem (cluster size 
five nodes, RF=3, ~40GB/node) - paraller repair takes about 6 hrs whereas 
serial takes some 48 hours or so. In addition, the compaction impact is roughly 
the same - that is, there’s the same number of compactions triggered per 
minute, but serial runs eight times more of them. There does not seem to be a 
difference between the node response latency during parallel or serial repair.

NB: We do increase our compaction throughput during calmer times, and lower it 
through busy times, and the serial compaction takes enough time to hit the busy 
period - that might also have an impact to the overall performance.

If I had known that this had so far been a theoretical problem, I would’ve 
spoken up earlier. Perhaps serial repair is not the best default.

/Janne

Re: are repairs in 2.0 more expensive than in 1.2

[Robert Coli]

On Thu, Oct 23, 2014 at 2:04 PM, Janne Jalkanen 
wrote:

>
> If I had known that this had so far been a theoretical problem, I would’ve
> spoken up earlier. Perhaps serial repair is not the best default.
>

Unfortunately you must not hang out in #cassandra on freenode, where I've
been ranting^Wcomplaining^Wtalking about this default since it was merged.
I did not file a JIRA ticket because my presumption was that any report
that did not describe an actual problem experienced in an actual production
deploy would be minimized or ignored. I was also not completely convinced
that my conjecture about just how severely bad the behavior was was
correct, which is a reasonable concern for a skeptic or critic to have... :D

Your perspective as a production operator having a negative experience of
this default in production is welcomed in JIRA.

If either you or Sean Bridges files a JIRA, please post its URL to the
thread for future reference. :D

=Rob
http://twitter.com/rcolidba

Re: Operating on large cluster

[Otis Gospodnetic]

Hi Alain,

We use Puppet and introducing Ansible at Sematext.  Not for Cassandra, but
for other similar tech.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Thu, Oct 23, 2014 at 5:18 AM, Alain RODRIGUEZ  wrote:

> Hi,
>
> I was wondering about how do you guys handle a large cluster (50+
> machines).
>
> I mean there is sometime you need to change configuration (cassandra.yaml)
> or send a command to one, some or all nodes (cleanup, upgradesstables,
> setstramthoughput or whatever).
>
> So far we have been using things like custom scripts for repairs or any
> routine maintenance and cssh for specific and one shot actions on the
> cluster. But I guess this doesn't really scale, I guess we coul use pssh
> instead. For configuration changes we use Capistrano that might scale
> properly.
>
> So I would like to known, what are the methods that operators use on large
> cluster out there ? Have some of you built some open sourced "cluster
> management" interfaces or scripts that could make things easier while
> operating on large Cassandra clusters ?
>
> Alain
>