How are writes handled while adding nodes to cluster?

[Erik Forsberg]

Hi!

How are writes handled while I'm adding a node to a cluster, i.e. while
the new node is in JOINING state?

Are they queued up as hinted handoffs, or are they being written to the
joining node?

In the former case I guess I have to make sure my max_hint_window_in_ms
is long enough for the node to become NORMAL or hints will get dropped
and I must do repair. Am I right?

Thanks,
\EF

java.lang.IllegalArgumentException: Mutation of X bytes is too large for the maxiumum size of Y

[George Sigletos]

Hello,

I have been frequently receiving those warnings:

java.lang.IllegalArgumentException: Mutation of 35141120 bytes is too large
for the maxiumum size of 33554432
 at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
~[apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:383)
~[apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:363)
~[apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-2.1.9.jar:2.1.9]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
~[na:1.7.0_75]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.1.9.jar:2.1.9]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_75]

Sometimes I can trigger them myself by trying to add the contents of  a
text document that is less than 1 MB.

Initially I did increase the "commitlog_segment_size_in_mb" from 32 to 64.
Thinking to further increase it to 96.

But would that be a solution to the problem? What could be possibly causing
this?

Thank you,
George

Re: Cassandra Configuration VS Static IPs.

[Gene]

Rather than using public IPs on the Cassandra nodes at all I would set up a
VPN.

1. Make sure all Cassandra nodes are on the same private network with
static IPs
2. Set up a new micro instance with a static public IP, make sure it has a
static IP on the private network as well
3. Install OpenVPN on the new micro instance, configure it to tunnel
traffic to that private network
4. Configure your cassandra nodes to permit access from this instance
5. Configure your workstation to make a VPN connection and use the static,
private IPs of the cassandra nodes for your client connectivity

-Gene


On Sun, Oct 4, 2015 at 5:31 PM, Renato Perini 
wrote:

> Jonathan, I have some difficulties in understanding what you're talking
> about. A client is a program connecting to a cassandra instance. All it
> needs to know is an IP, a keyspace and a table to operate. My client is
> nothing more than a simple textual version of a program like datastax
> devcenter. No "same dc" concepts are involved for using it.
> As for AWS, I'm not changing anything. The instances, as I said multiple
> times, don't have an elastic ip, so the public IP is dynamic. This means it
> changes automatically at every reboot.
>
>
> Il 05/10/2015 02:22, Jonathan Haddad ha scritto:
>
> If your client is in the same DC, then you shouldn't use *public* ip
> addresses.  If you're using a recent version of Cassandra you can just set
> the listen_interface and rpc_interface to whatever network interface you've
> got.
>
> If you're really changing IPs when you reboot machines (I have no idea why
> you'd do this, AWS definitely doesn't work this way) then I think you're
> going to hit a whole set of other issues.
>
>
> On Sun, Oct 4, 2015 at 7:10 PM Renato Perini < 
> renato.per...@gmail.com> wrote:
>
>> Yes, the client uses the same datacenter (us-west-2).
>> Maybe I haven't explained well the situation. I'm not asking to connect
>> to nodes *without* using a static IP address, but allowing Cassandra to
>> determine the current public address at the time of connection.
>> Spark, for example, uses shell scripts for configuration, so the public
>> IP (in AWS) can be assigned using the command `curl
>> http://169.254.169.254/latest/meta-data/public-ipv4`, whatever it is at
>> the time of boot.
>> Cassandra uses a yaml file for the main configuration, so this is
>> impossibile to achieve. Basically I would like to make the client connect
>> correctly on all nodes using their public IPs without being required to
>> know them (the client would discover them dynamically while connecting).
>>
>>
>>
>> Il 05/10/2015 00:55, Jonathan Haddad ha scritto:
>>
>> So you're not running the client in the same DC as your Cassandra
>> cluster.  In that case you'll need to be able to connect to the public
>> address of all the nodes.  Technically you could have a whitelist and only
>> connect to 1, I wouldn't recommend it.
>>
>> This is no different than any other database in that you would need a
>> public address to be able to connect to the servers from a machine not in
>> your datacenter.  How else would you connect to them if you don't provide
>> access?
>>
>> On Sun, Oct 4, 2015 at 6:35 PM Renato Perini 
>> wrote:
>>
>>> Seems to be not the case when connecting to my (single) data center
>>> using the java connector with a small client I have developed for testing.
>>> For the broadcast_rpc_address I have configured the local IP of the
>>> nodes. The cluster works fine and nodes communicates fairly well using
>>> their local IPs. When I connect to a node (let's say node 1) from the
>>> outside using the java driver and the node's public IP, the cluster
>>> discovery uses internal IPs for contacting other nodes, leading to
>>> (obviously) errors.
>>>
>>> As for AWS, Elastic IPs are free as long as they're associated to an
>>> instance and the machines are up 24h/7. I have to shut down the machines
>>> during the night for various reasons, so unfortunately they're not totally
>>> free for my use case.
>>>
>>>
>>>
>>> Il 05/10/2015 00:04, Jonathan Haddad ha scritto:
>>>
>>> Public IP?  No, not required unless you're running multiple DCs.
>>>
>>> Where are you running a DC that IPs aren't cheap?  If you're in AWS
>>> they're basically free (or at least the cheapest section of your bill by
>>> far)
>>>
>>>
>>>
>>> On Sun, Oct 4, 2015 at 5:59 PM Renato Perini 
>>> wrote:
>>>
 Is cassandra really supposed to have a static public IP for each and
 single node in the cluster?
 This seems to be expensive (static IPs are nor free neither cheap),
 still the broadcast_rpc_address expects a static IP for client
 communications (load balancing, contact points, etc.)
 Is there some mechanism to determine a public IP at runtime?

 Basically, I have nodes (machines) with dynamic public IPs and I cannot
 embed them in the cassandra.yaml file because of their dynamic nature
 (they change at each reboot).
 Any solution to this?

 Thanks.
>>>

Re: java.lang.IllegalArgumentException: Mutation of X bytes is too large for the maxiumum size of Y

[Kiran mk]

Do you see more dropped mutation messages in nodetool tpstats output.
On Oct 6, 2015 7:51 PM, "George Sigletos"  wrote:

> Hello,
>
> I have been frequently receiving those warnings:
>
> java.lang.IllegalArgumentException: Mutation of 35141120 bytes is too
> large for the maxiumum size of 33554432
>  at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:383)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:363)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at
> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> ~[na:1.7.0_75]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-2.1.9.jar:2.1.9]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_75]
>
> Sometimes I can trigger them myself by trying to add the contents of  a
> text document that is less than 1 MB.
>
> Initially I did increase the "commitlog_segment_size_in_mb" from 32 to 64.
> Thinking to further increase it to 96.
>
> But would that be a solution to the problem? What could be possibly
> causing this?
>
> Thank you,
> George
>

Re: java.lang.IllegalArgumentException: Mutation of X bytes is too large for the maxiumum size of Y

[George Sigletos]

I see no dropped mutation in any of the nodes:

Message type   Dropped
RANGE_SLICE  0
READ_REPAIR  0
PAGED_RANGE  0
BINARY   0
READ 0
MUTATION 0
_TRACE   0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0

On Tue, Oct 6, 2015 at 5:35 PM, Kiran mk  wrote:

> Do you see more dropped mutation messages in nodetool tpstats output.
> On Oct 6, 2015 7:51 PM, "George Sigletos"  wrote:
>
>> Hello,
>>
>> I have been frequently receiving those warnings:
>>
>> java.lang.IllegalArgumentException: Mutation of 35141120 bytes is too
>> large for the maxiumum size of 33554432
>>  at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:383)
>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:363)
>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>> at
>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>> at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>> Source) ~[na:1.7.0_75]
>> at
>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>> [apache-cassandra-2.1.9.jar:2.1.9]
>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_75]
>>
>> Sometimes I can trigger them myself by trying to add the contents of  a
>> text document that is less than 1 MB.
>>
>> Initially I did increase the "commitlog_segment_size_in_mb" from 32 to
>> 64. Thinking to further increase it to 96.
>>
>> But would that be a solution to the problem? What could be possibly
>> causing this?
>>
>> Thank you,
>> George
>>
>

Re: Cassandra Configuration VS Static IPs.

[Jonathan Haddad]

Personally I wouldn't bother with a VPN.  You can limit access to a group
of servers easily via security groups and not have a single point of
failure.  I wrote a utility to manage this.  If you're bringing up a new
VPC + Cassandra servers every day, you can trivially apply a set of rules
to your VPC:
http://rustyrazorblade.com/2014/06/an-introduction-to-roadhouse/




On Tue, Oct 6, 2015 at 10:30 AM Gene  wrote:

> Rather than using public IPs on the Cassandra nodes at all I would set up
> a VPN.
>
> 1. Make sure all Cassandra nodes are on the same private network with
> static IPs
> 2. Set up a new micro instance with a static public IP, make sure it has a
> static IP on the private network as well
> 3. Install OpenVPN on the new micro instance, configure it to tunnel
> traffic to that private network
> 4. Configure your cassandra nodes to permit access from this instance
> 5. Configure your workstation to make a VPN connection and use the static,
> private IPs of the cassandra nodes for your client connectivity
>
> -Gene
>
>
> On Sun, Oct 4, 2015 at 5:31 PM, Renato Perini 
> wrote:
>
>> Jonathan, I have some difficulties in understanding what you're talking
>> about. A client is a program connecting to a cassandra instance. All it
>> needs to know is an IP, a keyspace and a table to operate. My client is
>> nothing more than a simple textual version of a program like datastax
>> devcenter. No "same dc" concepts are involved for using it.
>> As for AWS, I'm not changing anything. The instances, as I said multiple
>> times, don't have an elastic ip, so the public IP is dynamic. This means it
>> changes automatically at every reboot.
>>
>>
>> Il 05/10/2015 02:22, Jonathan Haddad ha scritto:
>>
>> If your client is in the same DC, then you shouldn't use *public* ip
>> addresses.  If you're using a recent version of Cassandra you can just set
>> the listen_interface and rpc_interface to whatever network interface you've
>> got.
>>
>> If you're really changing IPs when you reboot machines (I have no idea
>> why you'd do this, AWS definitely doesn't work this way) then I think
>> you're going to hit a whole set of other issues.
>>
>>
>> On Sun, Oct 4, 2015 at 7:10 PM Renato Perini < 
>> renato.per...@gmail.com> wrote:
>>
>>> Yes, the client uses the same datacenter (us-west-2).
>>> Maybe I haven't explained well the situation. I'm not asking to connect
>>> to nodes *without* using a static IP address, but allowing Cassandra to
>>> determine the current public address at the time of connection.
>>> Spark, for example, uses shell scripts for configuration, so the public
>>> IP (in AWS) can be assigned using the command `curl
>>> http://169.254.169.254/latest/meta-data/public-ipv4`, whatever it is at
>>> the time of boot.
>>> Cassandra uses a yaml file for the main configuration, so this is
>>> impossibile to achieve. Basically I would like to make the client connect
>>> correctly on all nodes using their public IPs without being required to
>>> know them (the client would discover them dynamically while connecting).
>>>
>>>
>>>
>>> Il 05/10/2015 00:55, Jonathan Haddad ha scritto:
>>>
>>> So you're not running the client in the same DC as your Cassandra
>>> cluster.  In that case you'll need to be able to connect to the public
>>> address of all the nodes.  Technically you could have a whitelist and only
>>> connect to 1, I wouldn't recommend it.
>>>
>>> This is no different than any other database in that you would need a
>>> public address to be able to connect to the servers from a machine not in
>>> your datacenter.  How else would you connect to them if you don't provide
>>> access?
>>>
>>> On Sun, Oct 4, 2015 at 6:35 PM Renato Perini 
>>> wrote:
>>>
 Seems to be not the case when connecting to my (single) data center
 using the java connector with a small client I have developed for testing.
 For the broadcast_rpc_address I have configured the local IP of the
 nodes. The cluster works fine and nodes communicates fairly well using
 their local IPs. When I connect to a node (let's say node 1) from the
 outside using the java driver and the node's public IP, the cluster
 discovery uses internal IPs for contacting other nodes, leading to
 (obviously) errors.

 As for AWS, Elastic IPs are free as long as they're associated to an
 instance and the machines are up 24h/7. I have to shut down the machines
 during the night for various reasons, so unfortunately they're not totally
 free for my use case.



 Il 05/10/2015 00:04, Jonathan Haddad ha scritto:

 Public IP?  No, not required unless you're running multiple DCs.

 Where are you running a DC that IPs aren't cheap?  If you're in AWS
 they're basically free (or at least the cheapest section of your bill by
 far)



 On Sun, Oct 4, 2015 at 5:59 PM Renato Perini 
 wrote:

> Is cassandra really supposed to have a static public IP for each and
>>>

Re: Repair corrupt SSTable from power outage?

[John Anderson]

I'm running 2.1.9

On Fri, Oct 2, 2015 at 8:40 AM, Tyler Hobbs  wrote:

> What version of Cassandra are you running?  It sounds like the disk
> failure policy is incorrectly being applied to scrub, which kind of defeats
> the purpose of it.  I'd recommend opening a ticket (
> https://issues.apache.org/jira/browse/CASSANDRA) with the information you
> posted here.
>
> On Fri, Oct 2, 2015 at 3:01 AM, George Sigletos 
> wrote:
>
>> I'm also facing problems regarding corrupt sstables and also couldn't run
>> sstablescrub successfully.
>>
>> I restarted my nodes with disk failure policy "best_effort", then I run
>> the "nodetool scrub "
>>
>> Once done I removed the corrupt tables manually and started repair
>>
>>
>> On Thu, Oct 1, 2015 at 7:27 PM, John Anderson  wrote:
>>
>>> I have a 25 node cluster and we lost power on one of the racks last
>>> night and now 6 of our nodes will not start up and we are getting the
>>> following error:
>>>
>>> INFO  [main] 2015-10-01 10:19:22,111 CassandraDaemon.java:122 - JMX is
>>> enabled to receive remote connections on port: 7199
>>> INFO  [main] 2015-10-01 10:19:22,124 CacheService.java:111 -
>>> Initializing key cache with capacity of 100 MBs.
>>> INFO  [main] 2015-10-01 10:19:22,129 CacheService.java:133 -
>>> Initializing row cache with capacity of 0 MBs
>>> INFO  [main] 2015-10-01 10:19:22,133 CacheService.java:150 -
>>> Initializing counter cache with capacity of 50 MBs
>>> INFO  [main] 2015-10-01 10:19:22,135 CacheService.java:161 - Scheduling
>>> counter cache save to every 7200 seconds (going to save all keys).
>>> INFO  [main] 2015-10-01 10:19:22,211 ColumnFamilyStore.java:363 -
>>> Initializing system.sstable_activity
>>> INFO  [SSTableBatchOpen:1] 2015-10-01 10:19:22,639
>>> SSTableReader.java:478 - Opening
>>> /mnt/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-45
>>> (805 bytes)
>>> ERROR [SSTableBatchOpen:1] 2015-10-01 10:19:22,657 FileUtils.java:447 -
>>> Exiting forcefully due to file system exception on startup, disk failure
>>> policy "stop"
>>> org.apache.cassandra.io.sstable.CorruptSSTableException:
>>> java.io.EOFException
>>> at
>>> org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:131)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>> at
>>> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>> at
>>> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>> at
>>> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>> at
>>> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>> at
>>> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>> at
>>> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>> at
>>> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>> at
>>> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>> at
>>> org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> [na:1.8.0_60]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> [na:1.8.0_60]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> [na:1.8.0_60]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> [na:1.8.0_60]
>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>> Caused by: java.io.EOFException: null
>>> at
>>> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
>>> ~[na:1.8.0_60]
>>> at java.io.DataInputStream.readUTF(DataInputStream.java:589)
>>> ~[na:1.8.0_60]
>>> at java.io.DataInputStream.readUTF(DataInputStream.java:564)
>>> ~[na:1.8.0_60]
>>> at
>>> org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:106)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>> ... 14 common frames omitted
>>>
>>>
>>> I found some people recommending scrubbing the sstable so I attempted
>>> that and got the following error:
>>>
>>> bin/sstablescrub system sstable_activity -v
>>>
>>>
>>> ERROR 17:26:03 Exiting forcefully due to file system exception on
>>> startup, disk failure policy "stop"
>>> org.apache.cassandra.io.sstable.CorruptSSTableException:
>>> java.io.EOFE

Re: How are writes handled while adding nodes to cluster?

[Tyler Hobbs]

When a node is joining, writes are sent to both the current replicas *and*
the joining replica.  However, the joining replica does not count towards
the consistency level.  So, for example, if you write at
ConsistencyLevel.TWO, and only one existing replica and the joining replica
respond, the write will be considered a failure.

On Tue, Oct 6, 2015 at 4:43 AM, Erik Forsberg  wrote:

> Hi!
>
> How are writes handled while I'm adding a node to a cluster, i.e. while
> the new node is in JOINING state?
>
> Are they queued up as hinted handoffs, or are they being written to the
> joining node?
>
> In the former case I guess I have to make sure my max_hint_window_in_ms
> is long enough for the node to become NORMAL or hints will get dropped
> and I must do repair. Am I right?
>
> Thanks,
> \EF
>



-- 
Tyler Hobbs
DataStax 

Re: How are writes handled while adding nodes to cluster?

[Robert Coli]

On Tue, Oct 6, 2015 at 11:10 AM, Tyler Hobbs  wrote:

> When a node is joining, writes are sent to both the current replicas *and*
> the joining replica.  However, the joining replica does not count towards
> the consistency level.  So, for example, if you write at
> ConsistencyLevel.TWO, and only one existing replica and the joining replica
> respond, the write will be considered a failure.
>

The word I usually use to summarize the nature of the writes being sent to
the joining replica is :

"Extra"

In that they are otherwise just like replication, but no one but the
joining node cares if they are delivered...

=Rob

Maximum node decommission // bootstrap at once.

[Kevin Burton]

We're in the middle of migrating datacenters.

We're migrating from 13 nodes to 30 nodes in the new datacenter.

The plan was to bootstrap the 30 nodes first, wait until they have joined.
 then we're going to decommission the old ones.

How many nodes can we bootstrap at once?  How many can we decommission?

I remember reading docs for this but hell if I can find it now :-P

I know what the answer is theoretically.  I just want to make sure we do
everything properly.

Kevin

-- 

We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile

Re: Maximum node decommission // bootstrap at once.

[Robert Coli]

On Tue, Oct 6, 2015 at 12:32 PM, Kevin Burton  wrote:

> How many nodes can we bootstrap at once?  How many can we decommission?
>

short answer : 1 node can join or part at simultaneously

longer answer : https://issues.apache.org/jira/browse/CASSANDRA-2434 /
https://issues.apache.org/jira/browse/CASSANDRA-7069 /
-Dconsistent.rangemovement

Have you considered using replace_address to replace your existing 13
nodes, at which point you just have to join 17 more?

=Rob

Re: Maximum node decommission // bootstrap at once.

[Kevin Burton]

I'm not sure which is faster/easier.  Just joining one box at a time and
then decommissioning or using replace_address.

this stuff is always something you do rarely and then more complex than it
needs to be.

This complicates long term migration too.  Having to have gigabit is
somewhat of a problem in that you might now actually have it where you're
going.

We're migrating from Washington, DC to Germany so we have to change TCP
send/receive buffers to get decent bandwidth.

But I think we can do this at 1Gb per so per box.


On Tue, Oct 6, 2015 at 12:48 PM, Robert Coli  wrote:

> On Tue, Oct 6, 2015 at 12:32 PM, Kevin Burton  wrote:
>
>> How many nodes can we bootstrap at once?  How many can we decommission?
>>
>
> short answer : 1 node can join or part at simultaneously
>
> longer answer : https://issues.apache.org/jira/browse/CASSANDRA-2434 /
> https://issues.apache.org/jira/browse/CASSANDRA-7069 /
> -Dconsistent.rangemovement
>
> Have you considered using replace_address to replace your existing 13
> nodes, at which point you just have to join 17 more?
>
> =Rob
>
>



-- 

We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile

Re: Maximum node decommission // bootstrap at once.

[Bryan Cheng]

Honestly, we've had more luck bootstrapping in our old DC (defining
topology properties as the new DC) and using rsync to migrate the data
files to new machines in the new datacenter. We had 10gig within the
datacenter but significantly less than this cross-DC, which lead to a lot
of broken streaming pipes and wasted effort. This might make sense
depending on your link quality and the resources/time you have available to
do TCP tuning,

On Tue, Oct 6, 2015 at 1:29 PM, Kevin Burton  wrote:

> I'm not sure which is faster/easier.  Just joining one box at a time and
> then decommissioning or using replace_address.
>
> this stuff is always something you do rarely and then more complex than it
> needs to be.
>
> This complicates long term migration too.  Having to have gigabit is
> somewhat of a problem in that you might now actually have it where you're
> going.
>
> We're migrating from Washington, DC to Germany so we have to change TCP
> send/receive buffers to get decent bandwidth.
>
> But I think we can do this at 1Gb per so per box.
>
>
> On Tue, Oct 6, 2015 at 12:48 PM, Robert Coli  wrote:
>
>> On Tue, Oct 6, 2015 at 12:32 PM, Kevin Burton  wrote:
>>
>>> How many nodes can we bootstrap at once?  How many can we decommission?
>>>
>>
>> short answer : 1 node can join or part at simultaneously
>>
>> longer answer : https://issues.apache.org/jira/browse/CASSANDRA-2434 /
>> https://issues.apache.org/jira/browse/CASSANDRA-7069 /
>> -Dconsistent.rangemovement
>>
>> Have you considered using replace_address to replace your existing 13
>> nodes, at which point you just have to join 17 more?
>>
>> =Rob
>>
>>
>
>
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> 
>
>

Re: Maximum node decommission // bootstrap at once.

[Kevin Burton]

OH. interesting.  Yeah. That's another strategy.  We've already done a
bunch of TCP tuning... we get about 1Gbit with large TCP windows.  So I
think we have that part done.

It's sad that CS can't resume...

Plan be we will just rsync the data.. Does it pretty much work just by
putting the data in a directory or do you have to do anything special?

On Tue, Oct 6, 2015 at 1:34 PM, Bryan Cheng  wrote:

> Honestly, we've had more luck bootstrapping in our old DC (defining
> topology properties as the new DC) and using rsync to migrate the data
> files to new machines in the new datacenter. We had 10gig within the
> datacenter but significantly less than this cross-DC, which lead to a lot
> of broken streaming pipes and wasted effort. This might make sense
> depending on your link quality and the resources/time you have available to
> do TCP tuning,
>
> On Tue, Oct 6, 2015 at 1:29 PM, Kevin Burton  wrote:
>
>> I'm not sure which is faster/easier.  Just joining one box at a time and
>> then decommissioning or using replace_address.
>>
>> this stuff is always something you do rarely and then more complex than
>> it needs to be.
>>
>> This complicates long term migration too.  Having to have gigabit is
>> somewhat of a problem in that you might now actually have it where you're
>> going.
>>
>> We're migrating from Washington, DC to Germany so we have to change TCP
>> send/receive buffers to get decent bandwidth.
>>
>> But I think we can do this at 1Gb per so per box.
>>
>>
>> On Tue, Oct 6, 2015 at 12:48 PM, Robert Coli 
>> wrote:
>>
>>> On Tue, Oct 6, 2015 at 12:32 PM, Kevin Burton 
>>> wrote:
>>>
 How many nodes can we bootstrap at once?  How many can we decommission?

>>>
>>> short answer : 1 node can join or part at simultaneously
>>>
>>> longer answer : https://issues.apache.org/jira/browse/CASSANDRA-2434 /
>>> https://issues.apache.org/jira/browse/CASSANDRA-7069 /
>>> -Dconsistent.rangemovement
>>>
>>> Have you considered using replace_address to replace your existing 13
>>> nodes, at which point you just have to join 17 more?
>>>
>>> =Rob
>>>
>>>
>>
>>
>>
>> --
>>
>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>> Engineers!
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> 
>>
>>
>


-- 

We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile

Re: Maximum node decommission // bootstrap at once.

[Robert Coli]

On Tue, Oct 6, 2015 at 2:14 PM, Kevin Burton  wrote:

> Plan be we will just rsync the data.. Does it pretty much work just by
> putting the data in a directory or do you have to do anything special?
>

http://www.pythian.com/blog/bulk-loading-options-for-cassandra/

Be careful, with vnodes the rsync approach gets meaningfully harder.

=Rob

Re: Maximum node decommission // bootstrap at once.

[Bryan Cheng]

Robert, I might be misinterpreting you but I *think* your link is talking
about bootstrapping a new node by bulk loading replica data from your
existing cluster?

I was referring to using Cassandra's bootstrap to get the node to join and
run (as a member of DC2 but with physical residence in DC1), and then
transfer the /data directory to a new machine to assume the identity of the
old. I *believe* that from the cluster point of view this is just the node
being down for an extended period of time (so the usual caveats apply)

On Tue, Oct 6, 2015 at 2:20 PM, Robert Coli  wrote:

> On Tue, Oct 6, 2015 at 2:14 PM, Kevin Burton  wrote:
>
>> Plan be we will just rsync the data.. Does it pretty much work just by
>> putting the data in a directory or do you have to do anything special?
>>
>
> http://www.pythian.com/blog/bulk-loading-options-for-cassandra/
>
> Be careful, with vnodes the rsync approach gets meaningfully harder.
>
> =Rob
>

Re: Maximum node decommission // bootstrap at once.

[Robert Coli]

On Tue, Oct 6, 2015 at 3:41 PM, Bryan Cheng  wrote:

> Robert, I might be misinterpreting you but I *think* your link is talking
> about bootstrapping a new node by bulk loading replica data from your
> existing cluster?
>

Sure... more generically it's about treating the set of data from a source
cluster as if it might be the data of a target cluster.


> I was referring to using Cassandra's bootstrap to get the node to join and
> run (as a member of DC2 but with physical residence in DC1), and then
> transfer the /data directory to a new machine to assume the identity of the
> old. I *believe* that from the cluster point of view this is just the node
> being down for an extended period of time (so the usual caveats apply)
>

Sure, as long as you specify initial_token in the target DC, this is
approximately the same operation as doing it between two different clusters.

In the same cluster : new node takes over generation number of old number
via auto_bootstrap:false or etc.
In different clusters : node bootstraps/auto_bootstrap:falses to same
ownership.

=Rob

Cassandra class Client.java usage from Eclipse console

[Xin Ren]

Cassandra version: 3.0

native protocol version: 4

Hi, I'm trying to start Cassandra server and client *within Eclipse*, and
just wondering the usage of this class org.apache.cassandra.transport.Client in
Cassandra source code.

I git cloned the source code and opened the project in Eclipse Luna, and
started the class org.apache.cassandra.service.CassandraDaemon in Eclipse.

If I use command line terminal to start cqlsh by ./bin/cqlsh, then
everything would work fine.

But, if I start the Client class (org.apache.cassandra.transport.Client) *in
Eclipse* with arguments 127.0.0.1 9042 4, my in.readLine() from Eclipse
console is failed. (host and port information is right here, I've checked
the server info)

I've tried in Eclipse command line:

   - startup - working
   - desc tables - not working, "line 1:0 no viable alternative at input
   'desc' ([desc]...)"

In source code there are if-else statements to look for *STARTUP, QUERY,
PREPARE, EXECUTE, OPTIONS,* etc. I'm sure I'm not giving the correct
parameter for this class, and I tried to debug method *parseLine*() to find
the usage but failed.

I tried to google out some documentation on it but little was found. Anyone
knows the usage of this class from Eclipse console? Thank you very much.

Source code of Client class:
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/transport/Client.java