Re: Cassandra 4.0 upgrade - Upgradesstables

Though it is not required to run upgradesstables, but upgradesstables -a will re-write the file to kick out tombstones, in sizeTieredcompaction, the largest files may stay a long time to wait for the next compaction to kick out tombstones. So it really depends, to run it or not, usually upgrades

Re: Cassandra 4.0 upgrade - Upgradesstables

Thank you On Tue, Aug 16, 2022 at 11:48 AM C. Scott Andreas wrote: > No downside at all for 3.x -> 4.x (however, Cassandra 3.x reading 2.1 > SSTables incurred a performance hit). > > Many users of Cassandra don't run upgradesstables after 3.x -> 4.x > upgrades at all. It's not necessary to run u

Re: Cassandra 4.0 upgrade - Upgradesstables

No downside at all for 3.x -> 4.x (however, Cassandra 3.x reading 2.1 SSTables incurred a performance hit).Many users of Cassandra don't run upgradesstables after 3.x -> 4.x upgrades at all. It's not necessary to run until a hypothetical future time if/when support for reading Cassandra 3.x SST

Re: Cassandra 4.0 upgrade - Upgradesstables

Thank you Erick, > it is going to be single-threaded by default so it will take a while to get through all the sstables on dense nodes Is there any downside if the upgradesstables take longer (example 1-2 days), other than I/O? Also when is the upgradesstable get triggered? after every node is up

Re: Cassandra 4.0 upgrade - Upgradesstables

As convenient as it is, there are a few caveats and it isn't a silver bullet. The automatic feature will only kick in if there are no other compactions scheduled. Also, it is going to be single-threaded by default so it will take a while to get through all the sstables on dense nodes. In contrast,

Re: Cassandra 4.0 randomly freezes on heavy load

OK, I think I figured out what was wrong in my setup. Appears that during the upgrade course I totally missed Garbage Collector settings in jvm11-server.options. After installation from RPM it defines the default CMS garbage collector, which is totally wrong for my hi-load environment. I returned b

Re: Cassandra 4.0 randomly freezes on heavy load

> I can reproduce this with a huge load using dsbulk, but still can't determine the cause of the problem. Can you get a thread dump (jstack ) when the system freezes? This might be helpful to determine the cause of the freeze. Also, can you reproduce this in a simpler environment (ccm + dsbulk)?

Re: Cassandra 4.0 randomly freezes on heavy load

Okay, that ruled it out. Anything interesting in the GC logs? Was Cassandra stuck at a GC safepoint? You may need to enable the detailed GC logs to see these. On 25/02/2022 10:02, Azamat Hackimov wrote: Hello! No, I have a directly attached NVMe disk, and there are no IO or network issues. п

Re: Cassandra 4.0 randomly freezes on heavy load

Hello! No, I have a directly attached NVMe disk, and there are no IO or network issues. пт, 25 февр. 2022 г. в 12:50, Bowen Song : > > Do you have any network based mountpoint, such as NFS or samba? I have > seen similar behaviour on other Java based applications at GC safepoint > when the networ

Re: Cassandra 4.0 randomly freezes on heavy load

Do you have any network based mountpoint, such as NFS or samba? I have seen similar behaviour on other Java based applications at GC safepoint when the network based filesystem loses their connection and reconnects. On 25/02/2022 06:09, Azamat Hackimov wrote: Hello! I recently migrated Cassan

Re: Cassandra 4.0 upgrade from Cassandra 3x

You can see upgrading instructions here https://github.com/apache/cassandra/blob/cassandra-4.0.2/NEWS.txt. On Fri, Feb 11, 2022 at 2:52 AM Abdul Patel wrote: > Hi > apart from standard upgrade process any thing specific needs ti be > handled separately for this upgrade process? > > Any changes

Re: Cassandra 4.0 upgrade from Cassandra 3x

Make sure you go through all the instructions in https://github.com/apache/cassandra/blob/trunk/NEWS.txt. It's also highly recommended that you upgrade to the latest 3.0.x or 3.11.x version before upgrading to 4.0. Generally there are no changes required on the client side apart from setting the p

Re: Cassandra 4.0 hanging on restart

Hi Paul, It's only problematic if you are trying to do *a lots of* subrange incremental repairs. The whole point of incremental repair is each repair is incremental and it will only touch the recently changed data, therefore you shouldn't need to split each node into too many subranges to re

Re: Cassandra 4.0 hanging on restart

Thanks Erick and Bowen I do find all the different parameters for repairs confusing, and even reading up on it now, I see Datastax warns against incremental repairs with -pr, but then the code here seems to negate the need for this warning. Anyway running it like this, produces data in the syst

Re: Cassandra 4.0 hanging on restart

Hi Erick, From the source code: https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/service/StorageService.java#L4042 The -pr option has no effect if -st and -et are specified. Therefore, the command results in an incremental repair

Re: Cassandra 4.0 hanging on restart

I just came across this thread and noted that you're running repairs with -pr which are not incremental repairs. Was that a typo? Cheers!

Re: Cassandra 4.0 hanging on restart

Yes, I understand that exposing JMX securely outside localhost involves a fair amount of work. The sidecar option will use more RAM per Cassandra node, which may or may not be an issue depending on the individual circumstance. It is not a quick win if neither of those two options are optimal fo

Re: Cassandra 4.0 hanging on restart

We don’t expose the JMX port outside localhost, so last time we looked it was not possible, I see now there is the sidecar option, but that sounds like there are number of caveats, particularly around resources, that may cause some issues with our setup. So at the moment reaper does not seem lik

Re: Cassandra 4.0 hanging on restart

I'm glad that it fixed the problem. Now, may I interest you with Cassandra Reaper ? In my experience it has managed the load fairly well on large clusters. On 26/01/2022 10:19, Paul Chandler wrote: I changed the the range repair to be full repair, reset the repaired

Re: Cassandra 4.0 hanging on restart

I changed the the range repair to be full repair, reset the repairedAt for all SSTables and deleted the old data out of the system.repairs table. This then did not create any new rows in the system.repairs table, and the node was able to restart without any problem, so this seems to be a soluti

Re: Cassandra 4.0 hanging on restart

That would indicate the "isSuperseded(session)" call returned false. After looking at the source code, it seems the subrange incremental repair is likely causing this. Would you mind to try either subrange full repair or full range incremental repair? You may need to reset the "repairedAt" val

Re: Cassandra 4.0 hanging on restart

Hi Bowen, Yes there are a large number of "Skipping delete of FINALIZED LocalSession” messages. We have a script that repairs ranges, stepping through the complete range in 5 days, this should create 1600 ranges over the 5 days, this runs commands like this: nodetool -h localhost -p 7199 repa

Re: Cassandra 4.0 hanging on restart

From the source code I've read, by default Cassandra will run a clean up for the system.repairs table every 10 minutes, any row related to a repair that has completed over 1 day ago will be automatically removed. I highly doubt that you have ran 75,000 repairs in the 24 hours prior to shutting

Re: Cassandra 4.0 hanging on restart

Hi Bowen, Yes, there does seem to be a lot of rows, on one of the upgraded clusters there 75,000 rows. I have been experimenting on a test cluster, this has about a 5 minute pause, and around 15,000 rows. If I clear the system.repairs table ( by deleting the sstables ) then this does not pau

Re: Cassandra 4.0 hanging on restart

Hmm, interesting... Try "select * from system.repairs;" in cqlsh on a slow starting node, do you get a lots of rows? This is the most obvious loop run (indirectly) by the ActiveRepairService.start(). On 24/01/2022 13:30, Romain Anselin wrote: Hi everyone, We generated a JFR profile of the st

Re: Cassandra 4.0 hanging on restart

Hi everyone, We generated a JFR profile of the startup phase of Cassandra with Paul, and it would appear that the time is spent in the ActiveRepairSession within the main thread (11mn of execution of the "main" thread in his environment, vs 15s in mine), which has been introduced in CASSANDRA-

Re: Cassandra 4.0 hanging on restart

Hi Bowen, Thanks for the reply, these have been our normal shutdowns, so we do a nodetool drain before restarting the service, so I would have thought there should not be any commtlogs However there is these messages for one commit log, But looks like it has finished quickly and correctly: IN

Re: Cassandra 4.0 hanging on restart

Nothing obvious from the logs you posted. Generally speaking, replaying commit log is often the culprit when a node takes a long time to start. I have seen many nodes with large memtable and commit log size limit spending over half an hour replaying the commit log. I usually do a "nodetool flu

Re: cassandra 4.0 java 11 support

apache.org Subject: Re: cassandra 4.0 java 11 support There's been some discussion around removing the "experimental" tag for C* 4.0 + Java 11 so by all means, we encourage everyone to try it and report back to the community if you run into issues. Java 11 support was added 2 year

Re: cassandra 4.0 java 11 support

There's been some discussion around removing the "experimental" tag for C* 4.0 + Java 11 so by all means, we encourage everyone to try it and report back to the community if you run into issues. Java 11 support was added 2 years ago so I think most of the issues have been ironed out. Now that 4.0 i

Re: cassandra 4.0 java 11 support

Experimental means anything can happen - dragons, unicorns, ... On 27/07/2021 21:32, CPC wrote: Hi , At cassandra site https://cassandra.apache.org/doc/latest/cassandra/new/java11.html , it says java 11 support is experiment

Re: Cassandra 4.0 GA

Thank you for the information On Wed, May 26, 2021 at 5:40 PM Kane Wilson wrote: > On Tue, May 25, 2021 at 6:36 AM Jai Bheemsen Rao Dhanwada < > jaibheem...@gmail.com> wrote: > >> Hello All, >> >> I see that Cassandra 4.0 RC1 is released in April, is there going to be >> an official 4.0 GA relea

Re: Cassandra 4.0 GA

On Tue, May 25, 2021 at 6:36 AM Jai Bheemsen Rao Dhanwada < jaibheem...@gmail.com> wrote: > Hello All, > > I see that Cassandra 4.0 RC1 is released in April, is there going to be an > official 4.0 GA release or is RC1 considered as an official GA release with > Production use? If not is there a te

Re: Cassandra 4.0 and python

Thanks Kane, If anyone else is interested in this, I created a Jira ticket : https://issues.apache.org/jira/browse/CASSANDRA-16641 but the response is that 3.6 is the minimum official supported version, although 2.7 still should work. Thi

Re: Cassandra 4.0 and python

No, I suspect the deb package dependencies haven't been updated correctly, as 2.7 should definitely still work. Could you raise a JIRA for this issue? Not sure if apt has some way to force install/ignore dependencies, however if you do that it may work, otherwise your only workaround would be to i

Re: Cassandra 4.0 and changing DC setting

Great thanks for testing this Paul. You're likely the first one to actually try it on 4.0. Albeit I still think we should have a flag to ALTER to do this for you... not ideal to have users messing with system tables. raft.so - Cassandra consulting, support, and managed services On Mon, Feb 22, 2

Re: Cassandra 4.0 and changing DC setting

Yes, I am only running this on test clusters, I don’t run anything like this without lots of tests first. Anyway this worked well, so thanks for the info. For anyone else who needs this, the cql statement to do this was: insert into system_schema.keyspaces ( keyspace_name , durable_writes, rep

Re: Cassandra 4.0 and changing DC setting

Make sure you test it on a practice cluster. Messing with the system tables is risky business! raft.so - Cassandra consulting, support, and managed services On Sun, Feb 21, 2021 at 11:12 PM Paul Chandler wrote: > Hi Kane, > > That sounds a good idea, I will give it a try on Monday. > > Thanks

Re: Cassandra 4.0 and changing DC setting

Hi Kane, That sounds a good idea, I will give it a try on Monday. Thanks Paul > On 21 Feb 2021, at 11:33, Kane Wilson wrote: > > There has been proposals to add a force/unsafe flag to alter DC but it hasn't > been actioned and at this rate seems unlikely to make it into 4.0. There is > how

Re: Cassandra 4.0 and changing DC setting

There has been proposals to add a force/unsafe flag to alter DC but it hasn't been actioned and at this rate seems unlikely to make it into 4.0. There is however a workaround, albeit not very user friendly. You should be able to modify the system_schema tables directly to do your DC updates. I am y

Re: Cassandra 4.0-beta1 available on FreeBSD

Angelo Polo wrote: Cassandra 4.0-beta1 is now available on FreeBSD. By the way I'm runinning a 6 nodes production cluster using 3.11.6 on FreeBSD 12.1/amd64 and I'm very happy about it (thanks Angelo for maintaining the FreeBSD Port!). I hope your 4.0-beta2 patch will be accepted soon (and

RE: Cassandra 4.0

Are there binary builds available for testing or is it source only? -- Jacques-Henri Berthemet -Original Message- From: Nate McCall Sent: Wednesday, October 24, 2018 10:02 PM To: Cassandra Users Subject: Re: Cassandra 4.0 When it's ready :) In all seriousness, the past two

Re: Cassandra 4.0

When it's ready :) In all seriousness, the past two blog posts include some discussion on our motivations and current goals with regard to 4.0: http://cassandra.apache.org/blog/ On Wed, Oct 24, 2018 at 4:49 AM Abdul Patel wrote: > > Hi all, > > Any idea when 4.0 is planned to release? --

Re: Cassandra 4.0

On 01/17/2017 05:43 AM, Artem Rokhin wrote: > Do you have any estimates for Cassandra 4.0 release? Follow the dev@ mailing list for info on upcoming release votes and other release discussion. http://mail-archives.apache.org/mod_mbox/cassandra-dev/ -- Kind regards, Michael