Hi all,
Why is RACKDC preferred for production than TOPOLOGY?
Surely one common file is far simpler to distribute than deal with the
mucky-muck of various configs for each host if they are in one rack or another
and/or one datacentre or another? It is also fairly self-documenting of the
setup
o update the
topology file on all hosts.
On Thu, 2 Jun 2022 at 09:13 Marc Hoppins
mailto:marc.hopp...@eset.com>> wrote:
Hi all,
Why is RACKDC preferred for production than TOPOLOGY?
Surely one common file is far simpler to distribute than deal with the
mucky-muck of various configs for e
Hi all,
Am new to Cassandra. Just finished installing on 22 nodes across 2 datacentres.
If I run nodetool describecluster I get
Stats for all nodes:
Live: 22
Joining: 0
Moving: 0
Leaving: 0
Unreachable: 0
Data Centers:
BA #Nodes: 9 #Down: 0
g caused by duplicated node IDs (the Host ID column in
`nodetool status`). Did you by any chance copied the Cassandra data directory
between nodes? (e.g. spinning up a new node from a VM snapshot that contains a
non-empty data directory)
On 03/06/2022 12:38, Marc Hoppins wrote:
> Hi all,
&g
Hi all,
Say we have 2 datacentres with 12 nodes in each. All hardware is the same.
4-core, 2 x HDD (eg, 4TiB)
num_tokens = 16 as a start point
If a plan is to gradually increase the nodes per DC, and new hardware will have
more of everything, especially storage, I assume I increase the num_tok
existing 16-token cluster.
On Wed, Jun 15, 2022 at 1:33 AM Marc Hoppins
mailto:marc.hopp...@eset.com>> wrote:
Hi all,
Say we have 2 datacentres with 12 nodes in each. All hardware is the same.
4-core, 2 x HDD (eg, 4TiB)
num_tokens = 16 as a start point
If a plan is to gradually increase t
Hi guys,
Documentation (for almost everything) uses IP addresses for seeds, is it
possible to use the FQDN instead for the seeds (cass.yaml)? It is far easier
to read/use names.
Thanks
M
mes, since we don't do that for
production.
Sean R. Durity
INTERNAL USE
-Original Message-
From: Marc Hoppins mailto:marc.hopp...@eset.com>>
Sent: Thursday, June 23, 2022 3:33 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: [EXTERNAL]
Hi all,
Cluster of 2 DC and 24 nodes
DC1 (RF3) = 12 nodes, 16 tokens each
DC2 (RF3) = 12 nodes, 16 tokens each
Adding 12 more nodes to DC1: I installed Cassandra (version is the same across
all nodes) but, after the first node added, I couldn't seem to add any further
nodes.
I check nodetool
15:48, Marc Hoppins wrote:
> Hi all,
>
> Cluster of 2 DC and 24 nodes
>
> DC1 (RF3) = 12 nodes, 16 tokens each
> DC2 (RF3) = 12 nodes, 16 tokens each
>
> Adding 12 more nodes to DC1: I installed Cassandra (version is the same
> across all nodes) but, after the first node a
core while all other cores are idle.
On 08/07/2022 07:09, Marc Hoppins wrote:
> Thank you for pointing that out.
>
> 85 gigabytes/gibibytes/GIGABYTES/GIBIBYTES/whatever name you care to
> give it
>
> CPU and bandwidth are not the problem.
>
> Version 4.0.3 but, as I stated
cluster sees it disappear from the ring.
Then, start it fresh and let it bootstrap again. (you could alternatively try
the resumable bootstrap option, but I never use it).
On Fri, Jul 8, 2022 at 2:56 AM Marc Hoppins
mailto:marc.hopp...@eset.com>> wrote:
Ifconfig shows RX of 1.1T. Th
up but from the original seed node, it
does not appear in the nodetool status. Can anyone shed any light on this
phenomena?
From: Marc Hoppins
Sent: Monday, July 11, 2022 10:02 AM
To: user@cassandra.apache.org
Cc: Bowen Song
Subject: RE: Adding nodes
Well then…
I left this on Friday (still
ans it will interfere with client
read/write operations.
A few hundred GB to a few TB per node is pretty common in Cassandra clusters.
Big data is not about how much data on EACH node, it's about how much data in
TOTAL.
On 11/07/2022 09:01, Marc Hoppins wrote:
Well then…
I left this on Friday (st
ind
out why is it happening, and make appropriate changes if needed.
On 11/07/2022 09:23, Marc Hoppins wrote:
Further oddities…
I was sitting here watching our new new node being added (nodetool status being
run from one of the seed nodes) and all was going well. Then I noticed that
our new
unchanged for hours and the outputs aren't the
same between nodes, it could be an indicator of something else that had gone
wrong.
Does the strange behaviour goes away after the joining node completes the
streaming and fully joins the cluster?
On 11/07/2022 10:46, Marc Hoppins wrote:
I am beg
n 11/07/2022 10:41, Marc Hoppins wrote:
“Where did you come up with the 90 seconds number?” The database folk came up
with THAT number. For myself, I timed adding a new node at 120 seconds for the
initial setup with no data in the cluster.
“What exactly are you waiting for by doing that?” I wa
s you provided. I can see 1 full GC in the 3 hours since the node
started. It's not necessarily a problem (if it only occasionally happens during
the initial bootstraping process), but it should justify an investigation if
this is the first time you've seen it.
On 11/07/2022 11:09, Mar
uld be run after adding all
nodes to reduce unnecessary strain on the cluster.
From: Marc Hoppins
Sent: Monday, July 11, 2022 2:15 PM
To: user@cassandra.apache.org
Subject: RE: Adding nodes
All clocks are fine.
Why would time synch would affect whether or not a node appears in the nodetool
statu
l new nodes to this rack and have RF=2 in that
DC?
In principal, you should have equal number of servers (vnodes) in each rack,
and have the rack number = RF or 1.
On 11/07/2022 13:15, Marc Hoppins wrote:
All clocks are fine.
Why would time synch would affect whether or not a node appears i
’ questions is not a practical response as any business is unlikely
to be spending speculative money.
From: Jeff Jirsa
Sent: Tuesday, July 12, 2022 4:43 PM
To: cassandra
Cc: Bowen Song
Subject: Re: Adding nodes
EXTERNAL
On Tue, Jul 12, 2022 at 7:27 AM Marc Hoppins
mailto:marc.hopp...@eset.com
Hulloa all,
Service on two nodes stopped yesterday and I can find nothing to indicate why.
I have checked Cassandra system.logs, gc.logs and debug.logs as well as OS logs
and all I can see is the following - which is far from helpful:
DAEMON.LOG
Aug 3 11:39:12 cassandra19 systemd[1]: cassandr
t? Some
system logs, such as OOM killer and MCE error logs, don't go into the
DAEMON.LOG file.
On 04/08/2022 11:00, Marc Hoppins wrote:
Hulloa all,
Service on two nodes stopped yesterday and I can find nothing to indicate why.
I have checked Cassandra system.logs, gc.logs and debug.logs as
Hi all,
I added a node but forgot to specify the correct rack so I stopped the join and
removed it. When I tried adding it again it was taking a LONG time to join. I
tried draining before stopping the service but that failed. I killed the
process and cleared the directories but the cluster st
INFO [StorageServiceShutdownHook] 2022-08-24 09:30:15,847
MessagingService.java:441 - Waiting for messaging service to quiesce
-Original Message-
From: Marc Hoppins
Sent: Wednesday, August 24, 2022 9:06 AM
To: user@cassandra.apache.org
Subject: Erroneous node. - node is not a member of the
H
Update:
I shut the server down and the node finally disappeared from the status.
I then restarted the server on the similarly named node (dc1-cass14) and it
came up...however, it is UJ. Was this due to the amount of time spent
unavailable?
M
-Original Message-
From: Marc Hoppins
CassandraRoleManager.java:370 -
Setup task failed with error, rescheduling
Am I going to have to remove this node and try again?
-Original Message-
From: Marc Hoppins
Sent: Wednesday, August 24, 2022 11:59 AM
To: user@cassandra.apache.org
Subject: RE: Erroneous node. - node is not a member of the
Hulloa all,
Can anyone shed light on the order which nodes will deliver data to a new node?
Or point me toward a suitable chart/document?
Does the new node accept data from each node in turn or simultaneously from
multiple nodes?
Thanks
Marc
@cassandra.apache.org
Subject: Re: Bootstrap data streaming order
EXTERNAL
The data is requested asynchronously from peers. There is some logic to select
the peers however there isn’t a set order for data delivery. Why do you ask?
>
> On Sep 8, 2022, at 11:35 PM, Marc Hoppins wrote:
>
> Hulloa
of the nodes matter?
On Fri, Sep 9, 2022 at 10:27 AM Marc Hoppins
mailto:marc.hopp...@eset.com>> wrote:
Curiosity as to which data/node starts first, what determines the delivery
sequence, how many nodes send data at once and what determines that limit?
The usual kind of que
Hi all,
Restarting the service on a node. Checking status from a remote node, I see:
(prod) marc.hoppins.ipa@ba-cassandra01:~ $ /opt/cassandra/bin/nodetool status
-r|grep 03
UN ba-cassandra09 779.03 GiB 16 ?
1fc8061d-2dd4-4b2c-97fa-e492063da495 SSW09
UN ba-cassandra20 796.94
"nodetool info" output, and make sure "Gossip active" and "Native Transport
active" (unless you have disabled it) are "true".
On 23/09/2022 08:17, Marc Hoppins wrote:
> Hi all,
>
> Restarting the service on a node. Checking status from a remote
Hulloa all,
I started a decommission. Node load was 1.08TiB. After 6 or so hours the load
is at 1.12TiB. Shouldn't it be DECREASING?
Hi all,
Looking at upgrading our install from 4.0.3 to 4.0.6.
We have replication from one datacentre to a backup site. Other than modifying
the replication config from dc1 to dc2, is there a simple method or command to
stop replication for a period?
The idea being that, should something go a
On every node?
From: Erick Ramirez
Sent: Wednesday, October 12, 2022 3:20 PM
To: user@cassandra.apache.org
Subject: Re: Upgrade
EXTERNAL
It's just a minor patch upgrade so all you're really upgrading is the binaries.
In any case, switching off replication is not the recommended approach. The
r
Groovy. Thanks.
From: Erick Ramirez
Sent: Wednesday, October 12, 2022 4:08 PM
To: user@cassandra.apache.org
Subject: Re: Upgrade
EXTERNAL
That's correct. Cheers!
Hi all,
What (if any) problems could we expect from an upgrade?
Ie., If we have 12 nodes and I upgrade them one-at-a-time, some will be on the
new version and others on the old.
Assuming that daily operations continue during this process, could problems
occur with streaming replica from one n
Hi, all,
The config has data limits described as KB, MB, etc. Are these KB MB or KiB
MiB? (curses to the lazy modern age for forcing a change) Nodetool status
reports TiB. I assume these are all really base2 numbers but am just seeking to
clarify.
Eg.,
# Default value ("auto") is 1/256th of
HI all,
This is a test setup so has quickly been configured, and with a nominal amount
of test data. Initially, I was getting "'Malformed IPv6 address at index 7"
errors so I appended "-Dcom.sun.jndi.rmiURLParsing=legacy" which removed that.
However, snapshots are not being performed. Is it
Ah.
Version is 4.0.3
nodetool snapshot
From research, ‘nodetool snapshot’ will snapshot all keyspaces by default. So,
as I want to update to a new version, I assume this is what I want. Nodetool
snapshot produces the output as previously posted. None of it makes any sense
to me, especially
Hi all,
On a test setup I a looking to do an upgrade from 4.0.3 to 4.0.6.
Would one typically snapshot before DRAIN or after?
If DRAIN after snapshot, I would have to restart the service to snapshot and
would this not then be accepting new operations/data?
If DRAIN before snapshot, would there
Hi, all,
We had a failed HDD on one node. The node was shut down pending repair. There
are now 4 other nodes with Cassandra not running and unable to startup due to
the following kinds of error. Is this kind of thing due to the original
stopped node?
ERROR [main] 2022-12-12 14:58:10,838 LogR
HI all,
Is there a config setting to only log INFO line itself and omit the remaining
java/netty items? These are repeated every 30 seconds which creates
un-necessary spam in the system log. Despite having logback configured at INFO
level, these extra items keep appearing.
INFO [Messaging-E
Hi all,
If SSSD stops responding to requests/listening, is this going to cause the
Cassandra service to shut down? I didn't see anything to indicate such
behaviour in the config, only for disk issues.
I had two hosts where SSSD was not accepting logins and, after restarting that
service and l
Update: It may be that the load on these hosts is causing problems for SSSD not
the other way around. In any case, it seems that both services are off at the
same time.
From: Marc Hoppins
Sent: Wednesday, December 14, 2022 10:59 AM
To: user@cassandra.apache.org
Subject: SSSD and Cassandra
HI all,
I was pondering this very situation.
We have a node with a crapped-out disk (not the first time). Removenode vs
repairnode: in regard time, there is going to be little difference twixt
replacing a dead node and removing then re-installing a node. There is going
to be a bunch of reads/
Hulloa all,
I read a thing re. adding new nodes where the recommendation was to run cleanup
on the nodes after adding a new node to remove redundant token ranges.
I timed this way back when we only had ~20G of data per node and it took
approx. 5 mins per node. After adding a node on Tuesday, I
lean period (low traffic). Also you can
use suboptions of keyspace and table names to plan it such a way that I/O
pressure is not much.
Regards
Manish
On Thu, Feb 16, 2023 at 3:12 PM Marc Hoppins
mailto:marc.hopp...@eset.com>> wrote:
Hulloa all,
I read a thing re. adding new nodes where
compaction_throughput_per_mb to zero
effectively disables throttling, which means cleanup and compaction will run as
fast as the instance will allow. For normal use, I'd recommend capping that at
8 or 16.
Aaron
On Thu, Feb 16, 2023 at 9:43 AM Marc Hoppins
mailto:marc.hopp...@eset.com>
49 matches
Mail list logo
