Re: Procedures for moving part of a C* cluster to a different datacenter

prepared to fail at that point and experiment with different settings, but the good thing about this process is the fact that you can rollback at any stage without affecting the original cluster. Paul Chandler > On 3 Apr 2019, at 10:46, Stefan Miklosovic > wrote: > > On Wed, 3

Re: New user on Ubuntu 18.04 laptop, nodetest status throws NullPointerException

commands in the directory ~/.cassandra, and from the stack trace you supply it looks like it is failing to create that directory. So I would check the file system permissions there. Thanks Paul Chandler > On 3 Apr 2019, at 15:15, David Taylor wrote: > > I am running a System87 Oryx Pro la

Re: New user on Ubuntu 18.04 laptop, nodetest status throws NullPointerException

nodetest. > > Running nodetool status (or nodetool --help) results in the same stack trace > as before. > > On Wed, Apr 3, 2019 at 11:34 AM Paul Chandler <mailto:p...@redshots.com>> wrote: > David, > > When you start cassandra all the logs go to system.log normally i

Re: All time blocked in nodetool tpstats

being dropped then increase concurrent_reads, I normally change it to 96 to start with, but it will depend on size of your nodes. Otherwise it might be badly designed queries, have you investigated which queries are producing the client timeouts? Regards Paul Chandler > On 9 Apr 2019, at

Re: All time blocked in nodetool tpstats

the threads in steps, I normally go in steps of 32., but that is based on the size of machines I normally work with. But as Anthony said, if it is a high read system, then it could easily be tombstones or garbage collection. Thanks Paul Chandler > On 11 Apr 2019, at 03:57, Abdul Patel wr

Re: cass-2.2 trigger - how to get clustering columns and value?

feed into the new cluster, this this will work, however it you want it for anything else then it doesn’t help at all. I can supply more details later if this method is of interest. Thanks Paul Chandler > On 10 Apr 2019, at 22:52, Carl Mueller > wrote: > > We have a multitenant

Re: multiple snitches in the same cluster

not an issue. Thanks Paul Chandler www.redshots.com PS Myself and Gilberto Müeller are presenting at Datastax Accelerate on this very subject, how we migrated 91 clusters to Google, including what problems we had along the way. It would be worth you attending that session if you are there

Re: multiple snitches in the same cluster

GossipingPropertyFileSnitch, although that comes with the one caveat that I have never tried it on Amazon, only GCP. Thanks Paul Chandler www.redshots.com > On 16 Apr 2019, at 15:02, Shravan R wrote: > > Thanks Paul. Glad to know that you are speaking on the very subject soon. > > Even though you

Re: Using Cassandra as an object store

invoices. With a small amount of reads per object, then you can specify smaller CPUs and memory machines with a large amount of storage. If there are a large amount of reads, them you need to think much more carefully about memory and CPU, as per the Walmart article you referenced. Thanks Paul

Re: TWCS sstables not dropping even though all data is expired

Hi Mike, It sounds like that record may have been deleted, if that is the case then it would still be shown in this sstable, but the deleted tombstone record would be in a later sstable. You can use nodetool getsstables to work out which sstables contain the data. I recommend reading The Last

Re: TWCS sstables not dropping even though all data is expired

hough I > have a few others that share a very similar schema, and only some nodes) > seems like it will help me prevent it. > > > On Thu, May 2, 2019 at 1:00 PM Paul Chandler <mailto:p...@redshots.com>> wrote: > Hi Mike, > > It sounds like that record may have been

Re: Corrupted sstables

Roy, I have seen this exception before when a column had been dropped then re added with the same name but a different type. In particular we dropped a column and re created it as static, then had this exception from the old sstables created prior to the ddl change. Not sure if this applies in

Re: Corrupted sstables

n some machines the > sstable even does not exist on the filesystem. > On one machine I was able to dump the sstable to dump file without any issue > . Any idea how to tackle this issue ? > > > On Tue, May 7, 2019 at 12:32 AM Paul Chandler <mailto:p...@redshots.com&g

Re: Counter table in Cassandra

ers/ <http://www.redshots.com/cassandra-counting-without-using-counters/> I hope these links help. Regards Paul Chandler > On 29 May 2019, at 10:18, Attila Wind wrote: > > Hi Garvit, > > I can not answer your main question but when I read your lines one thing was >

Re: Collecting Latency Metrics

There are various attributes under org.apache.cassandra.metrics.ClientRequest.Latency.Read these measure the latency in milliseconds Thanks Paul www.redshots.com > On 29 May 2019, at 15:31, shalom sagges wrote: > > Hi All, > > I'm creating a dashboard that should collect read/write latency

Re: TWCS sstables not dropping even though all data is expired

;row", "position" : 69, "clustering" : [ 6 ], "liveness_info" : { "tstamp" : "2019-06-03T14:56:54.926536Z", "ttl" : 300, "expires_at" : "2019-06-03T15:01:54Z", "expired" : true }, "cells" : [

Re: Understanding output of read/write histogram using opscenter API

Hi Rahul, Opscenter is a Datastax product, have you raised a support request with them ( https://support.datastax.com <https://support.datastax.com/> ), they should be able to answer this sort of question. Regards Paul Chandler > On 4 Jun 2019, at 12:24, Bhardwaj, Rahul wrote: &g

Re: Expanding from 1 to 2 datacenters

moving 90+ clusters: http://www.redshots.com/accelerate/ <http://www.redshots.com/accelerate/> Happy to answer any more questions. Regards Paul Chandler www.redshots.com > On 26 Jun 2019, at 16:19, Voytek Jarnot wrote: > > I started an higher-level thread years ago about movi

Re: Cheat Sheet for Unix based OS, Performance troubleshooting

I have always found Amy's Cassandra 2.1 tuning guide great for the Linux performance tuning: https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html Sent from my iPhone > On 26 Jul 2019, at 23:49, Krish Donald wrote: > > Any one has Cheat Sheet for Unix based OS, Performance troubl

Re: Differing snitches in different datacenters

Hi Voytek, I looked into this a little while ago, and couldn’t really find a definitive answer. We ended up keeping the GossipingPropertyFileSnitch in our GCP Datacenter, the only downside that I could see is that you have to manually specify the rack and DC. But doing it that way does allow yo

Re: Understanding TRACE logging

Hi Shalom, When tracking down specific queries I have used ngrep and fed the results into Wireshark, this will allow you to find out everything about the requests coming into the node from the client, as long as the connection is not encrypted. I wrote this up here a few months ago: http://www

Re: TWCS and gc_grace_seconds

Hi Adarsh, You will have problems if you manually delete data when using TWCS. To fully understand why, I recommend reading this The Last Pickle post: https://thelastpickle.com/blog/2016/12/08/TWCS-part1.html And this post I wrote that dives deeper into the problems with deletes: http://www.red

Re: Ram & Space...

We had what sounds like a similar problem with a DSE cluster a little while ago, It was not being used, and had no tables in it. The memory kept rising until it was killed by the oom-killer. We spent along time trying to get to the bottom of the problem, but it suddenly stopped when the develop

Re: Cassandra going OOM due to tombstones (heapdump screenshots provided)

Hi Behroz, It looks like the number of tables is the problem, with 5,000 - 10,000 tables, that is way above the recommendations. Take a look here: https://docs.datastax.com/en/dse-planning/doc/planning/planningAntiPatterns.html#planningAntiPatterns__AntiPatTooManyTables

Corruption of frozen UDT during upgrade

Hi all, I have looked at the release notes for the up coming release 3.11.6 and seen the part about corruption of frozen UDT types during upgrade from 3.0. We have a number of cluster using UDT and have been upgrading to 3.11.4 and haven’t noticed any problems. In the ticket ( CASSANDRA-15035

Re: Corruption of frozen UDT during upgrade

e headers are fine, the scrub will be a no-op. Otherwise, it will report > that new metadata files are being written. For more details, see > https://support.datastax.com/hc/en-us/articles/360025955351 > <https://support.datastax.com/hc/en-us/articles/360025955351>. Cheers! > >

Re: Corruption of frozen UDT during upgrade

Thanks Erick, looks like I have a bit of detective work to do on Monday, to work out which of my list of clusters started out as 2.* or DSE 4.* and whether they had UDT’s at that time. > On 15 Feb 2020, at 00:50, Erick Ramirez wrote: > > I am still having problems reproducing this, so I am wo

Re: disable debug message on read repair

3.0/cassandra/configuration/configLoggingLevels.html> Thanks Paul Chandler www.redshots.com > On 10 Mar 2020, at 08:56, Gil Ganz wrote: > > That's one option, I wish I there was a way to disable just that and not the > entire debug log level, there are some things there I would li

Stopping a Nodetool move

Hi all, Is there a way to stop a nodetool move that is currently in progress? It is not moving the data between the nodes as expected and I would like to stop it before it completes. Thank you Paul - To unsubscribe, e-mai

Tool for schema upgrades

Hi all, Can anyone recommend a tool to perform schema DDL upgrades, that follows best practice to ensure you don’t get schema mismatches if running multiple upgrade statements in one migration ? Thanks Paul - To unsubscribe,

Re: Tool for schema upgrades

ps://github.com/Cobliteam/cassandra-migrate> >- [ ] https://github.com/patka/cassandra-migration > <https://github.com/patka/cassandra-migration> >- [ ] https://github.com/comeara/pillar <https://github.com/comeara/pillar> > > On Thu, Oct 8, 2020 at 5:45 PM

Re: username/password error when using nodetool flush

Hi Manu, nodetool uses the JMX user and password, I think the normal default for that is for it not being required, but not sure if that is the case for the setup you are using. So just try nodetool flush and see if that works. Regards Paul Sent from my iPhone > On 1 Jan 2021, at 20:41, Ma

Re: username/password error when using nodetool flush

:14, Manu Chadha wrote: > >  Just nodetool doesn't work unfortunately > > Sent from my iPhone > >>> On 1 Jan 2021, at 21:28, Paul Chandler wrote: >>> >>  Hi Manu, >> >> nodetool uses the JMX user and password, I think the normal defa

Cassandra 4.0 and changing DC setting

All, We have a use case where we need to change the datacenter name for a cassandra cluster, we have a script to do this that involves a short downtime. This does the following 1) Change replication factor for the system key spaces to be { ‘OLD_DC’ : ‘3’, ’NEW_DC”: ‘3’ } 2) Change the dc va

Re: Cassandra 4.0 and changing DC setting

ood idea by querying, doing updates and observing > the effect. > > raft.so - Cassandra consulting, support, managed services > > On Sat., 20 Feb. 2021, 02:29 Paul Chandler, <mailto:p...@redshots.com>> wrote: > All, > > We have a use case where we need to change the

Re: Cassandra 4.0 and changing DC setting

can be changed as normal afterwards. Thanks for you help. Paul > On 21 Feb 2021, at 22:30, Kane Wilson wrote: > > Make sure you test it on a practice cluster. Messing with the system tables > is risky business! > > raft.so <https://raft.so/> - Cassandra consulting

Re: No node was available to execute query error

Hi Joe This could also be caused by the replication factor of the keyspace, if you have NetworkTopologyStrategy and it doesn’t list a replication factor for the datacenter datacenter1 then you will get this error message too. Paul > On 12 Mar 2021, at 13:07, Erick Ramirez wrote: > > Does it

Cassandra 4.0 and python

qlsh#L65> All our clusters are currently on Ubuntu 16.04 which does not come with python 3.6, so this is going to be a major pain to upgrade them to 4.0. Does the apt packaging really need to specify 3.6 ? Thanks Paul Chandler

Re: Cassandra 4.0 and python

gt; > Not sure if apt has some way to force install/ignore dependencies, however if > you do that it may work, otherwise your only workaround would be to install > from the tarball. > > raft.so <https://raft.so/> - Cassandra consulting, support, and managed > services >

Re: TTL and disk space releasing

Hi Michael, I have had similar problems in the past, and found this Last Pickle post very useful: https://thelastpickle.com/blog/2016/12/08/TWCS-part1.html This should help you pinpoint what is stopping the SSTables being deleted. Assuming you are never manually deleting records from the table

Hint file getting stuck

Hi all We keep having a problem with hint files on one of our Cassandra nodes (v 3.11.6 ), there keeps being the following error messages repeated for same file. INFO [HintsDispatcher:25] 2021-11-02 08:55:29,830 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 72a18469-b7d2-499

Re: Anyone connecting the Cassandra on a server

I wrote a blog post describing how to do this a few years ago: http://www.redshots.com/who-is-connecting-to-a-cassandra-cluster/ Sent from my iPhone > On 19 Nov 2021, at 18:13, Saha, Sushanta K > wrote: > >  > I need to shutdown an old Apache Cassandra server for good. Running 3.0.x. > Any

Cassandra 4.0 hanging on restart

of the nodes is hanging again. Does anyone have an ideas what is causing the problems ? Thanks Paul Chandler

Re: Cassandra 4.0 hanging on restart

ed to replaying commit logs if this is the case. > > However, if you don't find any logs related to replaying commit logs, the > cause may be completely different. > > > On 19/01/2022 11:54, Paul Chandler wrote: >> Hi all, >> >> We have upgraded a couple

Re: Cassandra 4.0 hanging on restart

a/data >> find . -name '*Data*' | while read datf; do echo $datf ; sudo -u >> cassandra sstablemetadata $datf; done >> ~/sstablemetadata.txt >> cqlsh -e "paging off; select * from system.repairs" >> ~/repairs.out >> >> $ egrep 'Rep

Re: Cassandra 4.0 hanging on restart

ord was not deleted during the cleaned up for a number > of reasons. > > On 24/01/2022 19:45, Paul Chandler wrote: >> Hi Bowen, >> >> Yes, there does seem to be a lot of rows, on one of the upgraded clusters >> there 75,000 rows. >> >> I have been exp

Re: Unsolicited emails from IRONMAN Monttremblant

Could it be that email address is in the user group? If so your email could have triggered that automatic response, however I have not received anything after my recent emails There was an email on the dev list on 14/4/2020 that said the following, so they do seem to have had an interest in C

Re: Cassandra 4.0 hanging on restart

may need to reset the "repairedAt" value in all SSTables using > the "sstablerepairedset" tool if you decide to move on to use subrange full > repairs. > > > > On 25/01/2022 12:39, Paul Chandler wrote: >> Hi Bowen, >> >> Yes there are a

Re: Cassandra 4.0 hanging on restart

ge clusters. > > On 26/01/2022 10:19, Paul Chandler wrote: >> I changed the the range repair to be full repair, reset the repairedAt for >> all SSTables and deleted the old data out of the system.repairs table. >> >> This then did not create any new rows in the sy

Re: Cassandra 4.0 hanging on restart

Thanks Erick and Bowen I do find all the different parameters for repairs confusing, and even reading up on it now, I see Datastax warns against incremental repairs with -pr, but then the code here seems to negate the need for this warning. Anyway running it like this, produces data in the syst

Re: sstables changing in snapshots

Hi all, Was there any further progress made on this? Did a Jira get created? I have been debugging our backup scripts and seem to have found the same problem. As far as I can work out so far, it seems that this happens when a new snapshot is created and the old snapshot is being tarred. I ge

Re: sstables changing in snapshots

rg/jira/projects/CASSANDRA/issues/> > > It would be helpful to provide > 1. The version of the cassandra > 2. The options used for snapshotting > > - Yifan > > On Tue, Mar 22, 2022 at 9:41 AM Paul Chandler <mailto:p...@redshots.com>> wrote: > Hi all, > >

Re: sstables changing in snapshots

: > I do not think there is a ticket already. Feel free to create one. > https://issues.apache.org/jira/projects/CASSANDRA/issues/ > <https://issues.apache.org/jira/projects/CASSANDRA/issues/> > > It would be helpful to provide > 1. The version of the cassandra > 2. The

Re: Cassandra migration process

I would recommend creating a second Cassandra Datacenter for the cluster, rather than single nodes in the same DC, this is likely to cause latency issues, due to quorum queries being across datacenters. We did this several times, moving from Rackspace to GCP, this is all documented in 3 blog po

Re: Wrong Consistency level seems to be used

see if that ticket applies to your experience. Thanks Paul Chandler > On 21 Jul 2022, at 15:12, pwozniak wrote: > > Yes, I did it. Nothing like this in my code. Consistency level is set only in > one place (shown below). > > > > On 7/21/22 4:08 PM, manish khandelw

Re: Best compaction strategy for rarely used data

Hi Lapo Take a look at TWCS, I think that could help your use case: https://thelastpickle.com/blog/2016/12/08/TWCS-part1.html Regards Paul Chandler Sent from my iPhone > On 29 Dec 2022, at 08:55, Lapo Luchini wrote: > > Hi, I have a table which gets (a lot of) data that is wri

Mixed Cluster 4.0 and 4.1

any other issues to look out for? Thanks Paul Chandler

Re: Mixed Cluster 4.0 and 4.1

quired to complete a rolling upgrade, it's the number of DCs and racks > matter. > > Cheers, > Bowen > > On 24/04/2024 16:16, Paul Chandler wrote: >> Hi all, >> >> We have some large clusters ( 1000+ nodes ), these are across multiple >> datacenters. >

Re: Cassandra 5 upgrade and Schema Agreement failures

nodes as 4.1.3, then just upgrade > the software, correct? > > Thank you! > > Luciano > > > On Thu, Jan 30, 2025 at 6:57 AM Paul Chandler wrote: >> >> Hi Luciano, >> >> The problem occurs due to Cassandra 5 making changes to the system tables,

Re: Cassandra 5 Upgrade - Storage Compatibility Modes

this as a fundamental workflow that every C* operator should have > available, and you should be able to do them without there being any concern. > > Jon > > > On 2024/12/17 16:01:06 Paul Chandler wrote: >> All, >> >> We are getting a lot of push back on the 3 s

Cassandra 5 Upgrade - Storage Compatibility Modes

nks Paul Chandler

Re: Cassandra 5 Upgrade - Storage Compatibility Modes

> your immediate problem by avoiding it, I'd strive to solve the underlying > problem that your org is running Cassandra with unnecessary limitations > practices that make your life harder. > > Jon > > > On Tue, Dec 17, 2024 at 12:37 PM Paul Chandler <mailto:p.

Re: Cassandra 5 Upgrade - Storage Compatibility Modes

OK, it seems like I didn’t explain it too well, but yes it is the rolling restart 3 times as part of the upgrade that is causing the push back, my message was a bit vague on the use cases because there are confidentiality agreements in place so I can’t share too much. We have had problems in th

Re: Cassandra 5 upgrade and Schema Agreement failures

Hi Luciano, The problem occurs due to Cassandra 5 making changes to the system tables, so the cluster will be in schema mismatch during the upgrade process, until all the nodes are on 5.0 Normally this would not be a problem, as the system tables are not replicated anyway, but, as you are find

Re: SSTable Migration to oa-* Format After Upgrade to 5

iner > (54) 996309845 > >> On Sun, Mar 16, 2025 at 2:58 AM Paul Chandler wrote: >> >> Hi Luciano, >> >> It sounds like you could have the storage_compatibility_mode set to the >> default CASSANDRA_4 value, check this and change it to UPGRADING o

Re: SSTable Migration to oa-* Format After Upgrade to 5

t; > Luciano Greiner > > >> On Sun, Mar 16, 2025 at 2:33 PM Paul Chandler wrote: >> >> Yes, that should sort it out. >> >> Regards >> >> Paul >> Sent from my iPhone >> >>>> On 16 Mar 2025, at 18:34, Luciano Greiner >

Re: SSTable Migration to oa-* Format After Upgrade to 5

Hi Luciano, It sounds like you could have the storage_compatibility_mode set to the default CASSANDRA_4 value, check this and change it to UPGRADING or NONE. Full details can be found in the Cassandra.yaml https://cassandra.apache.org/doc/latest/cassandra/managing/configuration/cass_yaml_file.h