Hi Luciano,
The problem occurs due to Cassandra 5 making changes to the system tables, so
the cluster will be in schema mismatch during the upgrade process, until all
the nodes are on 5.0
Normally this would not be a problem, as the system tables are not replicated
anyway, but, as you are finding, it will stop you adding, or removing nodes
during an upgrade process.
Given this, it maybe best to split this operation, firstly to provision the new
nodes with 4.1.3, then upgrade the nodes to Cassandra 5.
Thanks
Paul
> On 30 Jan 2025, at 02:38, Luciano Greiner <luciano.grei...@gmail.com> wrote:
>
> Hello.
>
> I am upgrading a small Cassandra 4.1.3 cluster (2 sites, 3 nodes each)
> to Cassandra 5.
>
> Given we're using an old Centos OS in our nodes, I decided to get new
> nodes provisioned in the cluster (with version 5 on compatibility
> mode), and then decommission the old nodes when all is completed.
>
> There is an issue about schema agreement when I try to bootstrap the
> first Cassandra 5 node into the cluster:
>
> WARN [main] 2025-01-30 01:55:40,012
> DefaultSchemaUpdateHandler.java:131 - There are nodes in the cluster
> with a different schema version than us, from which we did not merge
> schemas: our version: (59adb24e-f3cd-3e02-97f0-5b395827453f),
> outstanding versions -> endpoints:
> {9bd8dc68-76f1-3b55-b60a-bb69ea16e53a=[/10.1.9.10:7000,
> /10.1.9.11:7000, /10.2.8.20:7000, /10.1.9.13:7000, /10.2.8.24:7000,
> /10.2.8.25:7000]}. Use -Dcassandra.skip_schema_check=true to ignore
> this, -Dcassandra.skip_schema_check_for_endpoints=<ep1[,epN]> to skip
> specific endpoints, or
> -Dcassandra.skip_schema_check_for_versions=<ver1[,verN]> to skip
> specific schema versions
> ERROR [main] 2025-01-30 01:55:40,012
> DefaultSchemaUpdateHandler.java:142 - Didn't receive schemas for all
> known versions within the PT30S. Use
> -Dcassandra.skip_schema_check=true to skip this check.
> ERROR [main] 2025-01-30 01:55:40,014 CassandraDaemon.java:887 -
> Exception encountered during startup
> java.lang.IllegalStateException: Could not achieve schema readiness in PT30S
>
> So I ran the describecluster. All 6 nodes are using the same schema
> version 9bd8dc68-76f1-3b55-b60a-bb69ea16e53a. Still, I've gone ahead
> and ran resetlocalschema and restarted each node, one at a time. No
> luck.
>
> Then I tried to skip the schema check, but then I get several table
> not found errors raised on the new node logs until it eventually shuts
> down.
>
> I am running out of ideas here.
>
> Does anybody have any thoughts?
>
> Thank you
>
> Luciano
