This is all really exciting. Getting a built in, orchestrated repair is a massive achievement. Thank you for your work on this, it's incredibly valuable to the community!!
Jon On Sun, Mar 9, 2025 at 2:25 PM Jaydeep Chovatia <chovatia.jayd...@gmail.com> wrote: > No problem, Dave! Thank you. > > Jaydeep > > On Sun, Mar 9, 2025 at 10:46 AM Dave Herrington <he...@rhinosource.com> > wrote: > >> Jaydeep, >> >> Thank you for taking time to answer my questions and for the links to the >> design and overview docs, which are excellent and answer all of my >> remaining questions. Sorry I missed those links in the CEP page. >> >> Great work and I will continue to follow your progress on this powerful >> new feature. >> >> Thanks! >> -Dave >> >> On Sat, Mar 8, 2025 at 9:36 AM Jaydeep Chovatia < >> chovatia.jayd...@gmail.com> wrote: >> >>> Hi David, >>> >>> Thanks for the kind words! >>> >>> >Is there a goal in this CEP to make automated repair work during >>> rolling upgrades, when multiple versions exist in the cluster? >>> We debated a lot on this over ASF Slack >>> (#cassandra-repair-scheduling-cep37). The summary is that, ideally, we want >>> to have a repair function during the mixed version, but the reality is that >>> currently, there is no test suite available inside Apache Cassandra to >>> verify the streaming behavior during the mixed version, so the confidence >>> is low. >>> We agreed on the following: 1) Keeping safety in mind, we should by >>> default disable the repair during mixed version 2) Add a comprehensive test >>> suite 3) Allow repair during mixed version. Currently, we are at #1 >>> >>> >Would automated repair be smart enough to automatically stop, if it >>> sees incompatible versions? >>> That's the plan, and we already have PR (CASSANDRA-20048 >>> <https://issues.apache.org/jira/browse/CASSANDRA-20048>) out from Chris >>> Lohfink. The thing we are debating is whether to stop only during major >>> version mismatch or also during the minor version, and we are leaning >>> towards only disabling for the major version mismatch. Regardless, this >>> should be available soon. >>> We are also extending this further as per feedback from David >>> Capwell that we should automatically stop repair if we detect a new DC or >>> keyspace RF is changed. That will be covered later as part of >>> CASSANDRA-20414 <https://issues.apache.org/jira/browse/CASSANDRA-20414> >>> >>> >If automated repair must be disabled for the entire cluster, will this >>> be a single nodetool command, or must automated repair be disabled on each >>> node individually? >>> Yes, it is a nodetool command and does not require any restarts! All the >>> *nodetool* command details are currently covered in the design doc >>> <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit?tab=t.0#heading=h.89fmsespiosd>, >>> and the same details will also be available in the Cassandra >>> overview.adoc >>> <https://github.com/apache/cassandra/pull/3598/files?short_path=e901018#diff-e90101885c1188844bb4188d1301277bfdc4a9e1e705c4ab8a6cc5a4b44460c0> >>> . >>> >>> >Would it make sense for automated repair to upgrade sstables, if it >>> finds old formats? (Maybe this could be a feature that could be optionally >>> enabled?) >>> My opinion is that it should not be part of the repair. It is best >>> suited as part of the Cassandra upgrade framework; I guess Paulo M is >>> looking at it. >>> >>> >W.R.T. the repair logging tables in the system_distributed keyspace, >>> will these tables have a configurable TTL, or must they be periodically >>> truncated to limit their size? >>> The number of entries will equal the number of Cassandra nodes in a >>> cluster. There is no TTL because each row represents the repair status of >>> that particular node. The entries would be automatically added/removed as >>> nodes are added/removed from the Cassandra cluster. >>> >>> Jaydeep >>> >>> On Sat, Mar 8, 2025 at 7:46 AM Dave Herrington <he...@rhinosource.com> >>> wrote: >>> >>>> Jaydeep, >>>> >>>> Thank you for your excellent efforts on this mission-critical feature. >>>> The stated goals of CEP-37 are noble and stand to make valuable >>>> improvements for cluster operations. I look forward to testing these new >>>> capabilities. >>>> >>>> My apologies up-front if you’ve already answered these questions. I >>>> did read the CEP a number of times and the linked JIRAs, but these are my >>>> questions that I couldn’t answer myself. >>>> >>>> I’m interested to understand the goals of CEP-37 W.R.T. to rolling >>>> upgrades of large clusters, as I am responsible for maintaining the cluster >>>> operations runbooks for a number of customers. >>>> >>>> Operators have to navigate the upgrade gauntlet with automated repairs >>>> disabled and get all nodes upgraded within gc_grace_seconds and then do a >>>> full repair, before restarting automated repairs. >>>> >>>> I see that CASSANDRA-7530 >>>> https://issues.apache.org/jira/browse/CASSANDRA-7530 is related to >>>> this. >>>> >>>> Is there a goal in this CEP to make automated repair work during >>>> rolling upgrades, when multiple versions exist in the cluster? >>>> >>>> (I think this would imply that stopping automated repairs would no >>>> longer be a pre-upgrade step.) >>>> >>>> Would automated repair be smart enough to automatically stop, if it >>>> sees incompatible versions? >>>> >>>> Would automated repair continue between nodes with compatible versions, >>>> or would it stop for the entire cluster? >>>> >>>> If automated repair must be disabled for the entire cluster, will this >>>> be a single nodetool command, or must automated repair be disabled on each >>>> node individually? >>>> >>>> Would it make sense for automated repair to upgrade sstables, if it >>>> finds old formats? (Maybe this could be a feature that could be optionally >>>> enabled?) >>>> >>>> W.R.T. the repair logging tables in the system_distributed keyspace, >>>> will these tables have a configurable TTL, or must they be periodically >>>> truncated to limit their size? >>>> >>>> Thanks, >>>> -Dave >>>> >>>> David A. Herrington II >>>> President and Chief Engineer >>>> RhinoSource, Inc. >>>> >>>> *Data Lake Architecture, Cloud Computing and Advanced Analytics.* >>>> >>>> www.rhinosource.com >>>> >>>> >>>> On Fri, Mar 7, 2025 at 11:48 AM Jaydeep Chovatia < >>>> chovatia.jayd...@gmail.com> wrote: >>>> >>>>> Hello Everyone, >>>>> >>>>> I wanted to update you on CEP-37 >>>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution> >>>>> (Jira: >>>>> CASSANDRA-19918 >>>>> <https://issues.apache.org/jira/browse/CASSANDRA-19918>) work. >>>>> Over the last year, some of us (Andy Tolbert, Chris Lohfink, >>>>> Francisco Guerrero, and Kristijonas Zalys) have been working closely on >>>>> making CEP-37 rock solid, with support from Josh McKenzie, Dinesh Joshi, >>>>> and David Capwell. >>>>> First and foremost, a huge thank you to everyone, including the >>>>> broader Apache Cassandra community, for their invaluable contributions in >>>>> making CEP-37 robust and solid! >>>>> >>>>> Here is the current status: >>>>> >>>>> *Feature stability* >>>>> >>>>> - *Voted feature:* All the features mentioned in CEP-37 have >>>>> worked as expected. >>>>> - *Post-voted feature:* A few new minor improvements >>>>> >>>>> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=272927365#CEP37ApacheCassandraUnifiedRepairSolution-Post-VoteUpdates> >>>>> have been added to post-voting, and they are also working as expected. >>>>> - Tested the functionality by multiple people over the period of >>>>> time. >>>>> - Some other facts: it has already been validated at scale >>>>> <https://www.youtube.com/watch?v=xFicEj6Nhq8>. Another big >>>>> Cassandra use case is in the process of validating/adopting it in their >>>>> environment. >>>>> >>>>> *Source Code* >>>>> >>>>> - It is an opt-in feature; nobody notices anything unless someone >>>>> opts in. >>>>> - By default, this feature is pretty isolated (in a separate >>>>> package) from the source code point of view (94% of the source code >>>>> lines are in the new files) >>>>> - A thorough documentation has been added: >>>>> - overview.doc >>>>> - metrics.doc >>>>> - cassandra.yaml doc >>>>> - NEWS.txt overview >>>>> - Five people (Andy Tolbert, Chris Lohfink, Francisco Guerrero, >>>>> and Kristijonas Zalys) have contributed. >>>>> - The source code has been reviewed multiple times by the same >>>>> five people. >>>>> >>>>> *Test Coverage* >>>>> >>>>> - A comprehensive test coverage has been added to cover all >>>>> aspects. >>>>> - The entire test suite has been passing >>>>> >>>>> >>>>> We are in the final review phase and nearly ready to merge. If anyone >>>>> has any last-minute feedback, this is the final opportunity for review. >>>>> >>>>> Thank you! >>>>> Andy Tolbert, Chris Lohfink, Francisco Guerrero, Kristijonas Zalys, >>>>> and Jaydeep >>>>> >>>> >> >> -- >> -Dave >> >> David A. Herrington II >> President and Chief Engineer >> RhinoSource, Inc. >> >> *Data Lake Architecture, Cloud Computing and Advanced Analytics.* >> >> www.rhinosource.com >> >
