Jaydeep, Thank you for taking time to answer my questions and for the links to the design and overview docs, which are excellent and answer all of my remaining questions. Sorry I missed those links in the CEP page.
Great work and I will continue to follow your progress on this powerful new feature. Thanks! -Dave On Sat, Mar 8, 2025 at 9:36 AM Jaydeep Chovatia <chovatia.jayd...@gmail.com> wrote: > Hi David, > > Thanks for the kind words! > > >Is there a goal in this CEP to make automated repair work during rolling > upgrades, when multiple versions exist in the cluster? > We debated a lot on this over ASF Slack > (#cassandra-repair-scheduling-cep37). The summary is that, ideally, we want > to have a repair function during the mixed version, but the reality is that > currently, there is no test suite available inside Apache Cassandra to > verify the streaming behavior during the mixed version, so the confidence > is low. > We agreed on the following: 1) Keeping safety in mind, we should by > default disable the repair during mixed version 2) Add a comprehensive test > suite 3) Allow repair during mixed version. Currently, we are at #1 > > >Would automated repair be smart enough to automatically stop, if it sees > incompatible versions? > That's the plan, and we already have PR (CASSANDRA-20048 > <https://issues.apache.org/jira/browse/CASSANDRA-20048>) out from Chris > Lohfink. The thing we are debating is whether to stop only during major > version mismatch or also during the minor version, and we are leaning > towards only disabling for the major version mismatch. Regardless, this > should be available soon. > We are also extending this further as per feedback from David Capwell that > we should automatically stop repair if we detect a new DC or keyspace RF is > changed. That will be covered later as part of CASSANDRA-20414 > <https://issues.apache.org/jira/browse/CASSANDRA-20414> > > >If automated repair must be disabled for the entire cluster, will this be > a single nodetool command, or must automated repair be disabled on each > node individually? > Yes, it is a nodetool command and does not require any restarts! All the > *nodetool* command details are currently covered in the design doc > <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit?tab=t.0#heading=h.89fmsespiosd>, > and the same details will also be available in the Cassandra overview.adoc > <https://github.com/apache/cassandra/pull/3598/files?short_path=e901018#diff-e90101885c1188844bb4188d1301277bfdc4a9e1e705c4ab8a6cc5a4b44460c0> > . > > >Would it make sense for automated repair to upgrade sstables, if it finds > old formats? (Maybe this could be a feature that could be optionally > enabled?) > My opinion is that it should not be part of the repair. It is best suited > as part of the Cassandra upgrade framework; I guess Paulo M is looking at > it. > > >W.R.T. the repair logging tables in the system_distributed keyspace, will > these tables have a configurable TTL, or must they be periodically > truncated to limit their size? > The number of entries will equal the number of Cassandra nodes in a > cluster. There is no TTL because each row represents the repair status of > that particular node. The entries would be automatically added/removed as > nodes are added/removed from the Cassandra cluster. > > Jaydeep > > On Sat, Mar 8, 2025 at 7:46 AM Dave Herrington <he...@rhinosource.com> > wrote: > >> Jaydeep, >> >> Thank you for your excellent efforts on this mission-critical feature. >> The stated goals of CEP-37 are noble and stand to make valuable >> improvements for cluster operations. I look forward to testing these new >> capabilities. >> >> My apologies up-front if you’ve already answered these questions. I did >> read the CEP a number of times and the linked JIRAs, but these are my >> questions that I couldn’t answer myself. >> >> I’m interested to understand the goals of CEP-37 W.R.T. to rolling >> upgrades of large clusters, as I am responsible for maintaining the cluster >> operations runbooks for a number of customers. >> >> Operators have to navigate the upgrade gauntlet with automated repairs >> disabled and get all nodes upgraded within gc_grace_seconds and then do a >> full repair, before restarting automated repairs. >> >> I see that CASSANDRA-7530 >> https://issues.apache.org/jira/browse/CASSANDRA-7530 is related to this. >> >> Is there a goal in this CEP to make automated repair work during rolling >> upgrades, when multiple versions exist in the cluster? >> >> (I think this would imply that stopping automated repairs would no longer >> be a pre-upgrade step.) >> >> Would automated repair be smart enough to automatically stop, if it sees >> incompatible versions? >> >> Would automated repair continue between nodes with compatible versions, >> or would it stop for the entire cluster? >> >> If automated repair must be disabled for the entire cluster, will this be >> a single nodetool command, or must automated repair be disabled on each >> node individually? >> >> Would it make sense for automated repair to upgrade sstables, if it finds >> old formats? (Maybe this could be a feature that could be optionally >> enabled?) >> >> W.R.T. the repair logging tables in the system_distributed keyspace, will >> these tables have a configurable TTL, or must they be periodically >> truncated to limit their size? >> >> Thanks, >> -Dave >> >> David A. Herrington II >> President and Chief Engineer >> RhinoSource, Inc. >> >> *Data Lake Architecture, Cloud Computing and Advanced Analytics.* >> >> www.rhinosource.com >> >> >> On Fri, Mar 7, 2025 at 11:48 AM Jaydeep Chovatia < >> chovatia.jayd...@gmail.com> wrote: >> >>> Hello Everyone, >>> >>> I wanted to update you on CEP-37 >>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution> >>> (Jira: >>> CASSANDRA-19918 <https://issues.apache.org/jira/browse/CASSANDRA-19918>) >>> work. >>> Over the last year, some of us (Andy Tolbert, Chris Lohfink, Francisco >>> Guerrero, and Kristijonas Zalys) have been working closely on making >>> CEP-37 rock solid, with support from Josh McKenzie, Dinesh Joshi, and David >>> Capwell. >>> First and foremost, a huge thank you to everyone, including the >>> broader Apache Cassandra community, for their invaluable contributions in >>> making CEP-37 robust and solid! >>> >>> Here is the current status: >>> >>> *Feature stability* >>> >>> - *Voted feature:* All the features mentioned in CEP-37 have worked >>> as expected. >>> - *Post-voted feature:* A few new minor improvements >>> >>> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=272927365#CEP37ApacheCassandraUnifiedRepairSolution-Post-VoteUpdates> >>> have been added to post-voting, and they are also working as expected. >>> - Tested the functionality by multiple people over the period of >>> time. >>> - Some other facts: it has already been validated at scale >>> <https://www.youtube.com/watch?v=xFicEj6Nhq8>. Another big Cassandra >>> use case is in the process of validating/adopting it in their >>> environment. >>> >>> *Source Code* >>> >>> - It is an opt-in feature; nobody notices anything unless someone >>> opts in. >>> - By default, this feature is pretty isolated (in a separate >>> package) from the source code point of view (94% of the source code >>> lines are in the new files) >>> - A thorough documentation has been added: >>> - overview.doc >>> - metrics.doc >>> - cassandra.yaml doc >>> - NEWS.txt overview >>> - Five people (Andy Tolbert, Chris Lohfink, Francisco Guerrero, and >>> Kristijonas Zalys) have contributed. >>> - The source code has been reviewed multiple times by the same five >>> people. >>> >>> *Test Coverage* >>> >>> - A comprehensive test coverage has been added to cover all aspects. >>> - The entire test suite has been passing >>> >>> >>> We are in the final review phase and nearly ready to merge. If anyone >>> has any last-minute feedback, this is the final opportunity for review. >>> >>> Thank you! >>> Andy Tolbert, Chris Lohfink, Francisco Guerrero, Kristijonas Zalys, and >>> Jaydeep >>> >> -- -Dave David A. Herrington II President and Chief Engineer RhinoSource, Inc. *Data Lake Architecture, Cloud Computing and Advanced Analytics.* www.rhinosource.com
