I stand corrected. C in TCM is "cluster" :D Anyway. Configuration is super reasonable to be put there.
On Fri, Dec 20, 2024 at 7:42 PM Štefan Miklošovič <smikloso...@apache.org> wrote: > I am super hesitant to base distributed guardrails or any configuration > for that matter on anything but TCM. Does not "C" in TCM stand for > "configuration" anyway? So rename it to TSM like "schema" then if it is > meant to be just for that. It seems to be quite ridiculous to code tables > with caches on top when we have way more effective tooling thanks to CEP-21 > to deal with that with clear advantages of getting rid of all of that old > mechanism we have in place. > > I have not seen any concrete examples of risks why using TCM should be > just for what it is currently for. Why not put the configuration meant to > be cluster-wide into that? > > What is it ... performance? What does even the term "additional > complexity" mean? Complex in what? Do you think that putting there 3 types > of transformations in case of guardrails which flip some booleans and > numbers would suddenly make TCM way more complex? Come on ... > > This has nothing to do with what Jordan is trying to introduce. I think we > all agree he knows what he is doing and if he evaluates that TCM is too > much for his use case (or it is not a good fit) that is perfectly fine. > > On Fri, Dec 20, 2024 at 7:22 PM Paulo Motta <pa...@apache.org> wrote: > >> > It should be possible to use distributed system tables just fine for >> capabilities, config and guardrails. >> >> I have been thinking about this recently and I agree we should be wary >> about introducing new TCM states and create additional complexity that can >> be serviced by existing data dissemination mechanisms (gossip/system >> tables). I would prefer that we take a more phased and incremental approach >> to introduce new TCM states. >> >> As a way to accomplish that, I have thought about introducing a new >> generic TCM state "In Maintenance", where schema or membership changes are >> "frozen/disallowed" while an external operation is taking place. This >> "external operation" could mean many things: >> - Upgrade >> - Downgrade >> - Migration >> - Capability Enablement/Disablement >> >> These could be sub-states of the "Maintenance" TCM state, that could be >> managed externally (via cache/gossip/system tables/sidecar). Once these >> sub-states are validated thouroughly and mature enough, we could "promote" >> them to top-level TCM states. >> >> In the end what really matters is that cluster and schema membership >> changes do not happen while a miscellaneous operation is taking place. >> >> Would this make sense as an initial way to integrate TCM with >> capabilities framework ? >> >> On Fri, Dec 20, 2024 at 4:53 AM Benedict <bened...@apache.org> wrote: >> >>> If you perform a read from a distributed table on startup you will find >>> the latest information. What catchup are you thinking of? I don’t think any >>> of the features we talked about need a log, only the latest information. >>> >>> We can (and should) probably introduce event listeners for distributed >>> tables, as this is also a really great feature, but I don’t think this >>> should be necessary here. >>> >>> Regarding disagreements: if you use LWTs then there are no consistency >>> issues to worry about. >>> >>> Again, I’m not opposed to using TCM, although I am a little worried TCM >>> is becoming our new hammer with everything a nail. It would be better IMO >>> to keep TCM scoped to essential functionality as it’s critical to >>> correctness. Perhaps we could extend its APIs to less critical services >>> without intertwining them with membership, schema and epoch handling. >>> >>> On 20 Dec 2024, at 09:43, Štefan Miklošovič <smikloso...@apache.org> >>> wrote: >>> >>> >>> I find TCM way more comfortable to work with. The capability of log >>> being replayed on restart and catching up with everything else >>> automatically is god-sent. If we had that on "good old distributed tables", >>> then is it not true that we would need to take extra care of that, e.g. we >>> would need to repair it etc ... It might be the source of the discrepancies >>> / disagreements etc. TCM is just "maintenance-free" and _just works_. >>> >>> I think I was also investigating distributed tables but was just pulled >>> towards TCM naturally because of its goodies. >>> >>> On Fri, Dec 20, 2024 at 10:08 AM Benedict <bened...@apache.org> wrote: >>> >>>> TCM is a perfectly valid basis for this, but TCM is only really >>>> *necessary* to solve meta config problems where we can’t rely on the rest >>>> of the database working. Particularly placement issues, which is why schema >>>> and membership need to live there. >>>> >>>> It should be possible to use distributed system tables just fine for >>>> capabilities, config and guardrails. >>>> >>>> That said, it’s possible config might be better represented as part of >>>> the schema (and we already store some relevant config there) in which case >>>> it would live in TCM automatically. Migrating existing configs to a >>>> distributed setup will be fun however we do it though. >>>> >>>> Capabilities also feel naturally related to other membership >>>> information, so TCM might be the most suitable place, particularly for >>>> handling downgrades after capabilities have been enabled (if we ever expect >>>> to support turning off capabilities and then downgrading - which today we >>>> mostly don’t). >>>> >>>> On 20 Dec 2024, at 08:42, Štefan Miklošovič <smikloso...@apache.org> >>>> wrote: >>>> >>>> >>>> Jordan, >>>> >>>> I also think that having it on TCM would be ideal and we should explore >>>> this path first before doing anything custom. >>>> >>>> Regarding my idea about the guardrails in TCM, when I prototyped that >>>> and wanted to make it happen, there was a little bit of a pushback (1) >>>> (even though super reasonable one) that TCM is just too young at the moment >>>> and it would be desirable to go through some stabilisation period. >>>> >>>> Another idea was that we should not make just guardrails happen but the >>>> whole config should be in TCM. From what I put together, Sam / Alex does >>>> not seem to be opposed to this idea, rather the opposite, but having CEP >>>> about that is way more involved than having just guardrails there. I >>>> consider guardrails to be kind of special and I do not think that having >>>> all configurations in TCM (which guardrails are part of) is the absolute >>>> must in order to deliver that. I may start with guardrails CEP and you may >>>> explore Capabilities CEP on TCM too, if that makes sense? >>>> >>>> I just wanted to raise the point about the time this would be >>>> delivered. If Capabilities are built on TCM and I wanted to do Guardrails >>>> on TCM too but was explained it is probably too soon, I guess you would >>>> experience something similar. >>>> >>>> Sam's comment is from May and maybe a lot has changed since in then and >>>> his comment is not applicable anymore. It would be great to know if we >>>> could build on top of the current trunk already or we will wait until >>>> 5.1/6.0 is delivered. >>>> >>>> (1) >>>> https://issues.apache.org/jira/browse/CASSANDRA-19593?focusedCommentId=17844326&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17844326 >>>> >>>> On Fri, Dec 20, 2024 at 2:17 AM Jordan West <jorda...@gmail.com> wrote: >>>> >>>>> Firstly, glad to see the support and enthusiasm here and in the recent >>>>> Slack discussion. I think there is enough for me to start drafting a CEP. >>>>> >>>>> Stefan, global configuration and capabilities do have some overlap but >>>>> not full overlap. For example, you may want to set globally that a cluster >>>>> enables feature X or control the threshold for a guardrail but you still >>>>> need to know if all nodes support feature X or have that guardrail, the >>>>> latter is what capabilities targets. I do think capabilities are a step >>>>> towards supporting global configuration and the work you described is >>>>> another step (that we could do after capabilities or in parallel with them >>>>> in mind). I am also supportive of exploring global configuration for the >>>>> reasons you mentioned. >>>>> >>>>> In terms of how capabilities get propagated across the cluster, I >>>>> hadn't put much thought into it yet past likely TCM since this will be a >>>>> new feature that lands after TCM. In Riak, we had gossip (but more mature >>>>> than C*s -- this was an area I contributed to a lot so very familiar) to >>>>> disseminate less critical information such as capabilities and a separate >>>>> layer that did TCM. Since we don't have this in C* I don't think we would >>>>> want to build a separate distribution channel for capabilities metadata >>>>> when we already have TCM in place. But I plan to explore this more as I >>>>> draft the CEP. >>>>> >>>>> Jordan >>>>> >>>>> On Thu, Dec 19, 2024 at 1:48 PM Štefan Miklošovič < >>>>> smikloso...@apache.org> wrote: >>>>> >>>>>> Hi Jordan, >>>>>> >>>>>> what would this look like from the implementation perspective? I was >>>>>> experimenting with transactional guardrails where an operator would >>>>>> control >>>>>> the content of a virtual table which would be backed by TCM so whatever >>>>>> guardrail we would change, this would be automatically and transparently >>>>>> propagated to every node in a cluster. The POC worked quite nicely. TCM >>>>>> is >>>>>> just a vehicle to commit a change which would spread around and all these >>>>>> settings would survive restarts. We would have the same configuration >>>>>> everywhere which is not currently the case because guardrails are >>>>>> configured per node and if not persisted to yaml, on restart their values >>>>>> would be forgotten. >>>>>> >>>>>> Guardrails are just an example, what is quite obvious is to expand >>>>>> this idea to the whole configuration in yaml. Of course, not all >>>>>> properties >>>>>> in yaml make sense to be the same cluster-wise (ip addresses etc ...), >>>>>> but >>>>>> the ones which do would be again set everywhere the same way. >>>>>> >>>>>> The approach I described above is that we make sure that the >>>>>> configuration is same everywhere, hence there can be no misunderstanding >>>>>> what features this or that node has, if we say that all nodes have to >>>>>> have >>>>>> a particular feature because we said so in TCM log so on restart / >>>>>> replay, >>>>>> a node with "catch up" with whatever features it is asked to turn on. >>>>>> >>>>>> Your approach seems to be that we distribute what all capabilities / >>>>>> features a cluster supports and that each individual node configures >>>>>> itself >>>>>> in some way or not to comply? >>>>>> >>>>>> Is there any intersection in these approaches? At first sight it >>>>>> seems somehow related. How is one different from another from your point >>>>>> of >>>>>> view? >>>>>> >>>>>> Regards >>>>>> >>>>>> (1) https://issues.apache.org/jira/browse/CASSANDRA-19593 >>>>>> >>>>>> On Thu, Dec 19, 2024 at 12:00 AM Jordan West <jw...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> In a recent discussion on the pains of upgrading one topic that came >>>>>>> up is a feature that Riak had called Capabilities [1]. A major pain with >>>>>>> upgrades is that each node independently decides when to start using >>>>>>> new or >>>>>>> modified functionality. Even when we put this behind a config (like >>>>>>> storage >>>>>>> compatibility mode) each node immediately enables the feature when the >>>>>>> config is changed and the node is restarted. This causes various types >>>>>>> of >>>>>>> upgrade pain such as failed streams and schema disagreement. A >>>>>>> recent example of this is CASSANRA-20118 [2]. In some cases operators >>>>>>> can >>>>>>> prevent this from happening through careful coordination (e.g. ensuring >>>>>>> upgrade sstables only runs after the whole cluster is upgraded) but >>>>>>> typically requires custom code in whatever control plane the operator is >>>>>>> using. A capabilities framework would distribute the state of what >>>>>>> features >>>>>>> each node has (and their status e.g. enabled or not) so that the cluster >>>>>>> can choose to opt in to new features once the whole cluster has them >>>>>>> available. From experience, having this in Riak made upgrades a >>>>>>> significantly less risky process and also paved a path towards >>>>>>> repeatable >>>>>>> downgrades. I think Cassandra would benefit from it as well. >>>>>>> >>>>>>> Further, other tools like analytics could benefit from having this >>>>>>> information since currently it's up to the operator to manually >>>>>>> determine >>>>>>> the state of the cluster in some cases. >>>>>>> >>>>>>> I am considering drafting a CEP proposal for this feature but wanted >>>>>>> to take the general temperature of the community and get some early >>>>>>> thoughts while working on the draft. >>>>>>> >>>>>>> Looking forward to hearing y'alls thoughts, >>>>>>> Jordan >>>>>>> >>>>>>> [1] >>>>>>> https://github.com/basho/riak_core/blob/25d9a6fa917eb8a2e95795d64eb88d7ad384ed88/src/riak_core_capability.erl#L23-L72 >>>>>>> >>>>>>> [2] https://issues.apache.org/jira/browse/CASSANDRA-20118 >>>>>>> >>>>>>
