To unsubscribe, please send an email to
user-unsubscr...@cassandra.apache.org
On 20/07/2022 18:34, emmanuel warreng wrote:
Unsubscribe
On Thu, Jul 7, 2022, 16:49 Marc Hoppins wrote:
Hi all,
Cluster of 2 DC and 24 nodes
DC1 (RF3) = 12 nodes, 16 tokens each
DC2 (RF3) = 12 no
Unsubscribe
On Thu, Jul 7, 2022, 16:49 Marc Hoppins wrote:
> Hi all,
>
> Cluster of 2 DC and 24 nodes
>
> DC1 (RF3) = 12 nodes, 16 tokens each
> DC2 (RF3) = 12 nodes, 16 tokens each
>
> Adding 12 more nodes to DC1: I installed Cassandra (version is the same
> across all nodes) but, after the fir
> The reply to suggest that folk head off a pay for a course when there are
> ‘pre-sales’ questions is not a practical response as any business is
> unlikely to be spending speculative money.
>
>
>
> *From:* Jeff Jirsa
> *Sent:* Tuesday, July 12, 2022 4:43 PM
> *To:* cassa
ask.
The reply to suggest that folk head off a pay for a course when there
are ‘pre-sales’ questions is not a practical response as any business
is unlikely to be spending speculative money.
*From:*Jeff Jirsa
*Sent:* Tuesday, July 12, 2022 4:43 PM
*To:* cassandra
*Cc:* Bowen Song
*Subjec
’ questions is not a practical response as any business is unlikely
to be spending speculative money.
From: Jeff Jirsa
Sent: Tuesday, July 12, 2022 4:43 PM
To: cassandra
Cc: Bowen Song
Subject: Re: Adding nodes
EXTERNAL
On Tue, Jul 12, 2022 at 7:27 AM Marc Hoppins
mailto:marc.hopp...@eset.com
On Tue, Jul 12, 2022 at 7:27 AM Marc Hoppins wrote:
>
> I was asking the questions but no one cared to answer.
>
This is probably a combination of "it is really hard to answer a question
with insufficient data" and your tone. Nobody here gets paid to help you
solve your company's problems except
motely
>
>
>
> *From:* Bowen Song via user
> *Sent:* Tuesday, July 12, 2022 12:29 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Adding nodes
>
>
>
> EXTERNAL
>
> For RF=2 in your DC1, you will not be able to achieve both strong
> consistency and
, the whole DC is useless.
Sean R. Durity
INTERNAL USE
From: Marc Hoppins
Sent: Tuesday, July 12, 2022 8:49 AM
To: user@cassandra.apache.org; Bowen Song
Subject: [EXTERNAL] RE: Adding nodes
The data guys want 2 copies of data in DC1 and that data to be replicated
offsite to DC1 for 1 copy (DR
, data or whatever, it is a
serious limitation as not every business/organisation is going to have
multiple racks available.
*From:*Bowen Song via user
*Sent:* Monday, July 11, 2022 8:57 PM
*To:* user@cassandra.apache.org
*Subject:* Re: Adding nodes
EXTERNAL
I've noticed the joining node
/organisation is
going to have multiple racks available.
From: Bowen Song via user
Sent: Monday, July 11, 2022 8:57 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
I've noticed the joining node has a different rack than the rest of the nodes,
is this intended? Will you add al
:*Joe Obernberger
*Sent:* Monday, July 11, 2022 1:29 PM
*To:* user@cassandra.apache.org
*Subject:* Re: Adding nodes
EXTERNAL
I too came from HBase and discovered adding several nodes at a time
doesn't work. Are you absolutely sure that the clocks are in sync
across the nodes? This has bitten
uld be run after adding all
nodes to reduce unnecessary strain on the cluster.
From: Marc Hoppins
Sent: Monday, July 11, 2022 2:15 PM
To: user@cassandra.apache.org
Subject: RE: Adding nodes
All clocks are fine.
Why would time synch would affect whether or not a node appears in the nodetool
statu
Obernberger
Sent: Monday, July 11, 2022 1:29 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
I too came from HBase and discovered adding several nodes at a time doesn't
work. Are you absolutely sure that the clocks are in sync across the nodes?
This has bitten me se
er
*Sent:* Monday, July 11, 2022 11:56 AM
*To:* user@cassandra.apache.org
*Subject:* Re: Adding nodes
EXTERNAL
Checking on multiple nodes won't help if the joining node suffers
from any of the issues I described, as it will likely be flipping up
and down frequently, and the existing nodes
s=0.00, real=0.03 secs]
*From:*Bowen Song via user
*Sent:* Monday, July 11, 2022 11:56 AM
*To:* user@cassandra.apache.org
*Subject:* Re: Adding nodes
EXTERNAL
Checking on multiple nodes won't help if the joining node suffers from
any of the issues I described, as it will likely be flipping u
NEW NODE manually. No seconds. No pauses. No
timeouts. Just waiting.
From: Bowen Song via user
Sent: Monday, July 11, 2022 12:13 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
How long doe it take to add a new node? I'm 100% sure neither 90s nor 120s is
the answer
t new
node in the ticket. Therefore, if I know when my first node is
finished, I will have an idea of how long before I check for the when
subsequent nodes can be joined.
*From:*Bowen Song via user
*Sent:* Monday, July 11, 2022 11:25 AM
*To:* user@cassandra.apache.org
*Subject:* Re: Adding nodes
EXTER
ser=0.22 sys=0.00, real=0.03 secs]
From: Bowen Song via user
Sent: Monday, July 11, 2022 11:56 AM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
Checking on multiple nodes won't help if the joining node suffers from any of
the issues I described, as it will likely be fli
e – which is already in the cluster, has been so for
several weeks, and is also functioning without error.
*From:*Bowen Song via user
*Sent:* Monday, July 11, 2022 11:40 AM
*To:* user@cassandra.apache.org
*Subject:* Re: Adding nodes
EXTERNAL
A node in joining state can disappearing from th
Sent: Monday, July 11, 2022 11:40 AM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
A node in joining state can disappearing from the cluster from other nodes'
perspective if the joining node stops sending/receiving gossip messages to
other nodes. This can happen whe
uent nodes can be joined.
From: Bowen Song via user
Sent: Monday, July 11, 2022 11:25 AM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
Sleeping/pausing for a fixed amount of time between operations at best is a
hack to workaround an unknown issue, but it's almost alwa
new node shows up but from the
original seed node, it does not appear in the nodetool status. Can
anyone shed any light on this phenomena?
*From:*Marc Hoppins
*Sent:* Monday, July 11, 2022 10:02 AM
*To:* user@cassandra.apache.org
*Cc:* Bowen Song
*Subject:* RE: Adding nodes
Well then…
I left
*Sent:* Friday, July 8, 2022 5:46 PM
*To:* cassandra
*Cc:* Bowen Song
*Subject:* Re: Adding nodes
EXTERNAL
Having a node UJ but not sending/receiving other streams is an invalid
state (unless 4.0 moved the streaming data out of netstats? I'm not
100% sure, but I'm 99% sure it s
up but from the original seed node, it
does not appear in the nodetool status. Can anyone shed any light on this
phenomena?
From: Marc Hoppins
Sent: Monday, July 11, 2022 10:02 AM
To: user@cassandra.apache.org
Cc: Bowen Song
Subject: RE: Adding nodes
Well then…
I left this on Friday (still
which is unlikely to cause problems? After all, in the modern world of
big (how big is big?) data, 600G per node is far less than the real BIG
big-data.
Marc
From: Jeff Jirsa
Sent: Friday, July 8, 2022 5:46 PM
To: cassandra
Cc: Bowen Song
Subject: Re: Adding nodes
EXTERNAL
Having a node UJ
878 2
> None of this addresses the issue of not being able to add more nodes.
>
> -Original Message-
> From: Bowen Song via user
> Sent: Friday, July 8, 2022 11:47 AM
> To: user@cassandra.apache.org
> Subject: Re: Adding nodes
>
> EXTERNAL
>
>
> I would ass
uly 8, 2022 11:47 AM
To:user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
I would assume that's 85 GB (i.e. gigabytes) then. Which is approximately 79
GiB (i.e. gibibytes). This still sounds awfully slow - less than 1MB/s over a
full day (24 hours).
You said CPU and network ar
2
None of this addresses the issue of not being able to add more nodes.
-Original Message-
From: Bowen Song via user
Sent: Friday, July 8, 2022 11:47 AM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
I would assume that's 85 GB (i.e. gigabytes) then
e is added within 2minutes. It is
hardly practical in production.
-Original Message-
From: Bowen Song via user
Sent: Thursday, July 7, 2022 8:43 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
86Gb (that's gigabits, which is 10.75GB, gigabytes) took an e
, 2022 8:43 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
86Gb (that's gigabits, which is 10.75GB, gigabytes) took an entire day seems
obviously too long. I would check the network bandwidth, disk IO and CPU usage
and find out what is the bottleneck.
On 07/07/2022
86Gb (that's gigabits, which is 10.75GB, gigabytes) took an entire day
seems obviously too long. I would check the network bandwidth, disk IO
and CPU usage and find out what is the bottleneck.
On 07/07/2022 15:48, Marc Hoppins wrote:
Hi all,
Cluster of 2 DC and 24 nodes
DC1 (RF3) = 12 nodes,
What version are you using?
When you run `nodetool netstats` on the joining node, what is the output?
How much data is there per node (presumably more than 86G)?
On Thu, Jul 7, 2022 at 7:49 AM Marc Hoppins wrote:
> Hi all,
>
> Cluster of 2 DC and 24 nodes
>
> DC1 (RF3) = 12 nodes, 16 tokens
Increasing RF will result in nodes that previously didn't have a replica of
the data now being responsible for it. This means that a repair is required
after increasing the RF.
Until the repair completes you will suffer from inconsistencies in data.
For example, in a 3 node cluster with RF 2, node
I think the token ranges that are clean/completed and potentially streamed down
to additional node , won’t be cleaned again so potentially you’ll need to run
cleanup once again.
Can you can stop cleanup, add additional node and start cleanup over again so
to get nodes clean in single shot!
F
On Sun, Jun 28, 2015 at 10:46 AM, Anuj Wadehra
wrote:
> Thanks Jake!! But I think most people have 2.0.x in Production right now
> as 2.1.6 is very recently declared Production Ready. I think the bug is too
> important to be left open in 2.0.x as it leads to data loss. Should I open
> JIRA?
>
So
Thanks Jake!! But I think most people have 2.0.x in Production right now as
2.1.6 is very recently declared Production Ready. I think the bug is too
important to be left open in 2.0.x as it leads to data loss. Should I open JIRA?
ThanksAnuj Wadehra
On Thursday, 25 June 2015 2:47 AM, Jake
This is no longer an issue in 2.1.
https://issues.apache.org/jira/browse/CASSANDRA-2434
We now make sure the replica we bootstrap from is the one that will no
longer own that range
On Wed, Jun 24, 2015 at 4:58 PM, Alain RODRIGUEZ wrote:
> It looks to me that can indeed happen theoretically (I m
It looks to me that can indeed happen theoretically (I might be wrong).
However,
- Hinted Handoff tends to remove this issue, if this is big worry, you
might want to make sure HH are enabled and well tuned
- Read Repairs (synchronous or not) might have mitigate things also, if you
read fresh data
The documentation is referring to Consistent Range Movements.
There is a change in 2.1 that won't allow you to bootstrap multiple nodes
at the same time unless you explicitly turn off consistent range movements.
Check out the jira:
https://issues.apache.org/jira/browse/CASSANDRA-2434
All the bes
OK.
Thanks.
I'll monitor the resources status (network, memory, cpu, io) as I go
and try to bootsrap them at chunks which seems not to have a bad
impact.
Will do regarding the cleanup.
Thanks!
On Mon, Apr 20, 2015 at 4:08 PM, Carlos Rolo wrote:
> Independent of the snitch, data needs to travel t
Hi Colin,
To remove your address from the list, send a message to:
Cheers,
Matt
*From:* Colin Clark [mailto:co...@clark.ws]
*Sent:* 20 April 2015 14:10
*To:* user@cassandra.apache.org
*Subject:* Re: Adding nodes to existing cluster
unsubscribe
On Apr 20, 2015, at 8:08 AM
unsubscribe
> On Apr 20, 2015, at 8:08 AM, Carlos Rolo wrote:
>
> Independent of the snitch, data needs to travel to the new nodes (plus all
> the keyspace information that goes via gossip). So I won't bootstrap them all
> at once, even if it is only for network traffic generated.
>
> Don't
Independent of the snitch, data needs to travel to the new nodes (plus all
the keyspace information that goes via gossip). So I won't bootstrap them
all at once, even if it is only for network traffic generated.
Don't forget to run cleanup on the old nodes once all nodes are in place to
reclaim di
Thanks for the response.
Sure we'll monitor as we're adding nodes.
We're now using 6 nodes on each DC. (We have 2 DCs)
Each node contains ~800GB
Do you know how rack configurations are relevant here?
Do you see any reason to bootstrap them one by one if we're not using
rack awareness?
On Mon, Ap
Start one node at a time. Wait 2 minutes before starting each node.
How much data and nodes you have already? Depending on that, the streaming
of data can stress on the resources you have.
I would recommend to start one and monitor, if things are ok, add another
one. And so on.
Regards,
Carlos
On Mon, Nov 18, 2013 at 10:28 AM, Carlos Alvarez wrote:
> Here
> http://www.datastax.com/documentation/cassandra/1.2/webhelp/cassandra/operations/ops_add_node_to_cluster_t.html
> says
> that it is needed to wait 2 minutes between adding nodes.
>
> I was trying to figure out why, and how to check
Thank you very much. I'll try to find the code.
Carlos.
On Thu, Nov 21, 2013 at 4:35 AM, Aaron Morton wrote:
> Of the top of my head there are some delays in the code do with schema
> synchronisation and bootstrapping. I’m not sure if it adds up to two
> minutes but it sounds ok. But the two m
Of the top of my head there are some delays in the code do with schema
synchronisation and bootstrapping. I’m not sure if it adds up to two minutes
but it sounds ok. But the two minute window would allow a new node to startup,
join, sync the schema and start the bootstrap process.
> I was try
Opened a ticket:
https://issues.apache.org/jira/browse/CASSANDRA-5525
On Mon, Apr 29, 2013 at 2:24 AM, aaron morton wrote:
> is this understanding correct "we had a 12 node cluster with 256 vnodes on
> each node (upgraded from 1.1), we added two additional nodes that streamed
> so much data (60
They were all restarted a couple times after adding 'num_tokens: 256' to
cassandra.yaml.
Yes and nodetool ring became 'unusable' due to all the new tokens.
On Mon, Apr 29, 2013 at 10:24 AM, Sam Overton wrote:
> Did you update num_tokens on the existing hosts and restart them, before
> you trie
Did you update num_tokens on the existing hosts and restart them, before
you tried bootstrapping in the new node? If the new node tried to stream
all the data in the cluster then this would be consistent with you having
missed that step.
You should see "Calculating new tokens" in the logs of the e
is this understanding correct "we had a 12 node cluster with 256 vnodes on each
node (upgraded from 1.1), we added two additional nodes that streamed so much
data (600+Gb when other nodes had 150-200GB) during the joining phase that they
filled their local disks and had to be killed" ?
Can you
On Sun, Apr 28, 2013 at 2:19 PM, aaron morton wrote:
> We're going to try running a shuffle before adding a new node again...
>> maybe that will help
>>
> I don't think hurt but I doubt it will help.
>
We had to bail on shuffle since we need to add capacity ASAP and not in 20
days.
>
>It
> We're going to try running a shuffle before adding a new node again... maybe
> that will help
I don't think hurt but I doubt it will help.
>> It seems when new nodes join, they are streamed *all* sstables in the
>> cluster.
>
How many nodes did you join, what was the num_tokens ?
Did yo
I believe that "nodetool rebuild" is used to add a new datacenter, not just
a new host to an existing cluster. Is that what you ran to add the node?
-Bryan
On Fri, Apr 26, 2013 at 1:27 PM, John Watson wrote:
> Small relief we're not the only ones that had this issue.
>
> We're going to try r
Small relief we're not the only ones that had this issue.
We're going to try running a shuffle before adding a new node again...
maybe that will help
- John
On Fri, Apr 26, 2013 at 5:07 AM, Francisco Nogueira Calmon Sobral <
fsob...@igcorp.com.br> wrote:
> I am using the same version and obser
I am using the same version and observed something similar.
I've added a new node, but the instructions from Datastax did not work for me.
Then I ran "nodetool rebuild" on the new node. After finished this command, it
contained two times the load of the other nodes. Even when I ran "nodetool
cl
After clearing the system files I was able to successfully join the node.
thanks.
From: aaron morton [mailto:aa...@thelastpickle.com]
Sent: Thursday, March 15, 2012 2:13 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes to cluster (Cassandra 1.0.8)
There have been two similar reports
>
> 2012/3/14 Rishabh Agrawal :
>> I am using storage port 7000 (deafult) across all nodes.
>>
>> -Original Message-
>> From: Maki Watanabe [mailto:watanabe.m...@gmail.com]
>> Sent: Wednesday, March 14, 2012 3:42 PM
>> To: user@cassandra.apache.org
&g
12 3:42 PM
> To: user@cassandra.apache.org
> Subject: Re: Adding nodes to cluster (Cassandra 1.0.8)
>
> Do you use same storage_port across 3 nodes?
> Can you access to the storage_port of the seed node from the last (failed)
> node?
>
> 2012/3/14 Rishabh Agrawal :
>&
gt; To: user@cassandra.apache.org
> Subject: Re: Adding nodes to cluster (Cassandra 1.0.8)
>
> Do you use same storage_port across 3 nodes?
> Can you access to the storage_port of the seed node from the last (failed)
> node?
>
> 2012/3/14 Rishabh Agrawal :
> > I was a
I am using storage port 7000 (deafult) across all nodes.
-Original Message-
From: Maki Watanabe [mailto:watanabe.m...@gmail.com]
Sent: Wednesday, March 14, 2012 3:42 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes to cluster (Cassandra 1.0.8)
Do you use same storage_port
Do you use same storage_port across 3 nodes?
Can you access to the storage_port of the seed node from the last (failed) node?
2012/3/14 Rishabh Agrawal :
> I was able to successfully join a node to already existing one-node cluster
> (without giving any intital_token), but when I add another machi
-
From: Henry Luo
Sent: Tuesday, November 02, 2010 12:10 PM
To: user@cassandra.apache.org
Subject: RE: Adding nodes in new data center
As suggested, we did do removetoken, set autobootstrap true, initial token and
start. Now the new nodes are in the Boostrapping state for a while, but nothing
]
Sent: Monday, November 01, 2010 6:08 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes in new data center
On Mon, Nov 1, 2010 at 6:01 PM, Henry Luo wrote:
> We have a cluster running in one data center, and are adding some in a new
> data center. There are some data already
On Mon, Nov 1, 2010 at 6:01 PM, Henry Luo wrote:
> We have a cluster running in one data center, and are adding some in a new
> data center. There are some data already in the current cluster.
>
> We did something wrong at first by not having AutoBootstrap on, when we saw
> no data being migrated.
You need to specify your initial tokens. LoadBalance really doesn't do a
good job of balancing the load. Take a look at "Load Balancing" in
http://wiki.apache.org/cassandra/Operations There is a little python script
in there to help you pick tokens for a given cluster size.
If you don't want to
67 matches
Mail list logo
