Kind of a nit, but I advocate for us having a guardrails.yaml and updating this line: > > *cassandra.yaml:* allows configuring individual guardrails at startup, > being globally disabled by default. These guardrails will also be > dynamically configurable through JMX and/or virtual tables.
The cassandra.yaml file is pretty big already and configuration of guardrails seems like a good candidate to modularize here. Could also be done during implementation (or discussed then), so don't hold up voting on me. :) On Wed, Nov 10, 2021 at 3:06 PM David Capwell <dcapw...@apple.com.invalid> wrote: > I am cool with voting. I would love to see added something like “all guard > rails must work in a rolling fashion where some nodes do not have the > latest guard rails”, but also cool leaving this to JIRA. > > > On Nov 10, 2021, at 11:17 AM, Ekaterina Dimitrova <e.dimitr...@gmail.com> > wrote: > > > > I am personally ready to give you my vote :-) > > > > On Wed, 10 Nov 2021 at 13:09, Andrés de la Peña <adelap...@apache.org> > > wrote: > > > >> I think the updated CEP incorporates the feedback above, unless I'm > missing > >> something. Are we ready to start a vote? > >> > >> On Tue, 2 Nov 2021 at 15:54, Andrés de la Peña <adelap...@apache.org> > >> wrote: > >> > >>> Under "Migrating existing cassandra.yaml warn/fail thresholds”, I > >> recently > >>>> added a few things which are basically guardrails, so should be > >> included in > >>>> this set; they are configured by track_warnings > (coordinator_read_size, > >>>> local_read_size, and row_index_size). With track_warnings I setup the > >>>> plumbing to have read queries trigger warnings (or abort the query) to > >> the > >>>> client exists (under "Event logging" you mention "and also to the > client > >>>> connection when applicable”) and isn’t limited to the coordinator > >>>> participating in the query (previous limitation for tombstone > warnings). > >>>> One thing I found which was problematic for track_warnings was that > >>>> altering clients is annoying as java and python both ignore the error > >>>> message we send (see > >>>> > >> > https://github.com/datastax/java-driver/blob/3.11.0/driver-core/src/main/java/com/datastax/driver/core/Responses.java#L73-L131 > >> ). > >>>> We log client warnings (if enabled) but ignore any detailed error > >> message > >>>> received from the server; it would be good to talk about client > >>>> integrations and how users are informed of issues in more detail. > >>> > >>> > >>> I have updated the CEP to include those thresholds among the ones that > >>> could be migrated once we have the guardrails framework ready. I have > >> also > >>> mentioned the usage of internal messaging to be able to propagate the > >>> outcome of guardrails triggered on nodes that are not the coordinator, > >> and > >>> the need of making changes on drivers. > >>> > >>> What I meant by "and also to the client connection when applicable" is > >>> that some guardrails can be applied to things that are nor necessarily > >>> associated to a client connection, such as compaction. I have tried to > be > >>> more explicit about that. > >>> > >>> > >>> On Tue, 2 Nov 2021 at 12:53, Andrés de la Peña < > a.penya.gar...@gmail.com > >>> > >>> wrote: > >>> > >>>> Being able to configure guardrails dynamically makes a lot of sense to > >>>> me, I have updated the CEP to mention that. I think we don't need to > >> decide > >>>> yet whether it would be done through JMX and/or virtual tables. > >>>> > >>>> On Mon, 1 Nov 2021 at 20:35, C. Scott Andreas <sc...@paradoxica.net> > >>>> wrote: > >>>> > >>>>> Re: "I think you all know my feels on JMX." – > >>>>> > >>>>> Super fair - I'd meant to speak in terms of desired outcome ("the > >>>>> feature should be dynamically configurable at runtime") rather than > >>>>> implementation ("this should be via JMX"). 👍 > >>>>> > >>>>> On Nov 1, 2021, at 1:24 PM, David Capwell <dcapw...@apple.com.INVALID > > > >>>>> wrote: > >>>>> > >>>>> > >>>>> If anyone wants to bite off making > >>>>> > >> > https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java > >>>>> < > >>>>> > >> > https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java > >>> > >>>>> support mutability then we get vtable support. I am cool with JMX > >> and/or > >>>>> vtable, to me its just more important to allow dynamic setting of > these > >>>>> configs. > >>>>> > >>>>> On Nov 1, 2021, at 10:36 AM, bened...@apache.org wrote: > >>>>> > >>>>> having them only configured via yaml seems like a bad outcome > >>>>> > >>>>> > >>>>> +1 > >>>>> > >>>>> I would like to see us move towards configuration being driven > through > >>>>> virtual tables where possible, so that the whole cluster can be > managed > >>>>> from a single interface. Not sure if this is the right place to bite > >> this > >>>>> off, but perhaps? > >>>>> > >>>>> From: Jeff Jirsa <jji...@gmail.com> > >>>>> Date: Monday, 1 November 2021 at 16:47 > >>>>> To: Cassandra DEV <dev@cassandra.apache.org> > >>>>> Subject: Re: [DISCUSS] CEP-3: Guardrails > >>>>> Without bike-shedding too much, guardrails would be great, building > >> them > >>>>> into a more general purpose framework that limits various dangerous > >>>>> things > >>>>> would be fantastic. The CEP says that the guardrails should be > distinct > >>>>> from the capability restrictions ( > >>>>> https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't > >> see > >>>>> why > >>>>> that needs to be the case. A system-level guardrail and a > >> personal-level > >>>>> guardrail are both restrictions, they just have different scopes, so > >>>>> implement the restriction framework first, and allow the scopes to be > >>>>> expanded as needed? > >>>>> > >>>>> Naming wise, I don't know that I'd actually surface these as > >>>>> "guardrails", > >>>>> but more as general "limits", and having them only configured via > yaml > >>>>> seems like a bad outcome > >>>>> > >>>>> > >>>>> > >>>>> https://issues.apache.org/jira/browse/CASSANDRA-8303 > >>>>> > >>>>> > >>>>> > >>>>> On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña < > adelap...@apache.org > >>> > >>>>> wrote: > >>>>> > >>>>> Hi everyone, > >>>>> > >>>>> I'd like to start a discussion about Guardrails proposal: > >>>>> > >>>>> > >>>>> > >> > https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails > >>>>> > >>>>> Guardrails are an easy way to enforce system-wide soft and hard > limits > >> to > >>>>> prevent anti-patterns of bad usage and in the long run make it not > >>>>> possible > >>>>> to severely degrade the performance of a node/cluster through user > >>>>> actions > >>>>> such as having too many secondary indexes, too large partitions, > almost > >>>>> full disks, etc. > >>>>> > >>>>> Thanks, > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
