+1. Made changes to the design document linked against the CEP to reflect this feedback. Specifically, the following sections have been updated * Operations to blacklist * Blacklist information store
Thanks, Sumanth On Fri, Aug 27, 2021 at 7:57 AM Joshua McKenzie <jmcken...@apache.org> wrote: > I can see the case for all three: > * Deny both reads and writes to a partition (wide, heavily tombstones, too > many stables, etc) causing disruption to a replica set; don't want further > growth nor reads until operator intervention > * Deny reads but allow writing to rectify problems on a partition > (intervention window; see above) > * Deny writes to a partition but allow reads (prevent partitions growing > unbounded, or potentially evolving into a future feature creating a ceiling > on partition sizes that kicks in and demands application intervention to > reduce partition size at a guardrail limit) > > So yeah, at least to me at face value it seems like it'd be worth it not > only to allow denylisting both reads and writes, but to be able to choose > from the set of reads|writes|both on a per-partition basis. > > ~Josh > > > On Thu, Aug 26, 2021 at 2:16 PM Sumanth Pasupuleti < > sumanth.pasupuleti...@gmail.com> wrote: > > > Thank you, Josh for the elaborate explanation of a potential scenario > where > > denylisting writes would make sense. > > I, 100% agree that could benefit in a situation where we would want to > deny > > writes to a partition that we do not have much control on (which is true > in > > most situations) and such behavior can eventually lead to unavailability > of > > other partitions too, as you indicate. > > > > Do you think it makes sense to make it configurable per partition though? > > As in, maybe by default, we would want to deny both reads and writes to a > > partition, but for certain partitions, we may still want to allow writes > > just so we can issue a delete against that partition as an example. > > Ofcourse this would make the feature and the interface more heavy, and we > > need to think through if its worth it. I personally feel it could be > worth > > it, especially if we agree on the default behavior that makes the > interface > > simple in most cases. Thoughts? > > > > And yes, so good to see CEP process reaping benefits in multiple ways - > > especially around collaboration and documentation. > > > > > > On Thu, Aug 26, 2021 at 8:31 AM Joshua McKenzie <jmcken...@apache.org> > > wrote: > > > > > The design doc and CEP currently pass on blocklisting / denylisting > > writes > > > at this time. In the proposed new patch it states: > > > "Note: We do not want to blacklist writes since it is the reads that > > > primarily impact the performance when reading a bad partition, and we > may > > > want writes to be allowed to “fix” a bad partition. We could revisit > this > > > in the future" > > > > > > In situations where you have an air gap between database ops and > > > application access (ops <> application teams, or more autonomous > > > application access patterns, self-service, etc), you can easily get > into > > a > > > situation where you have either a pathological client hammering writes > > to a > > > specific partition causing impact to other clients or in the worst > case, > > > the replica set, or unbounded partition growth that again leads to > > > performance degradation or replica set unavailability. The tradeoff > there > > > becomes "do we interrupt the application's ability to write to this > > > partition now, or do we instead defer and risk losing access to *all* > > > partitions on this replica set and still interrupt their access > > eventually > > > anyway?" > > > > > > Given this, I strongly advocate for support of denylisting both reads > > *and* > > > writes on these grounds; operators need another tool in their toolbox > to > > > deal with situations where specific partition writing has wider > negative > > > impacts on the replicas. > > > > > > Acknowledging of course that there was extensive discussion on this > back > > in > > > 2018, and that would have been a *great* time to engage in the > > discussion. > > > =/ Good thing we have this new CEP process! :) > > > > > > Curious what you think about this perspective Sumanth. > > > > > > ~Josh > > > > > > > > > On Tue, Aug 17, 2021 at 2:04 PM Joshua McKenzie <jmcken...@apache.org> > > > wrote: > > > > > > > Certainly. I'll take on distilling a high level view of the feature > > from > > > > what Jon and I are working on to bring to this discussion. > > > > > > > > > > > > On Tue, Aug 17, 2021 at 1:40 PM Sumanth Pasupuleti < > > > > sumanth.pasupuleti...@gmail.com> wrote: > > > > > > > >> +1 to collaborating on the patch, Josh. My 2 cents would be to > > continue > > > to > > > >> pursue this CEP in the community through Discuss and Vote phases and > > > then > > > >> invest further on the patch (based on the Vote phase outcome), so we > > can > > > >> reflect any additional feedback we may gather from the community > > through > > > >> this CEP. > > > >> > > > >> Thanks, > > > >> Sumanth > > > >> > > > >> On Tue, Aug 17, 2021 at 10:13 AM Joshua McKenzie < > > jmcken...@apache.org> > > > >> wrote: > > > >> > > > >> > Sumanth, > > > >> > > > > >> > Jon Meredith and I are recently working on an OSS patch of one of > > > those > > > >> "ad > > > >> > hoc" implementations of this feature that's been running at scale > > for > > > >> > awhile like Jirsa mentioned; sorry for not catching the discussion > > on > > > >> > https://issues.apache.org/jira/browse/CASSANDRA-12106 and > engaging > > > >> > earlier! > > > >> > > > > >> > I believe I can have a tidied up patch on 4.0 of this by early > next > > > >> week - > > > >> > then we can look at the two implementations and take best of both. > > > What > > > >> do > > > >> > you think? > > > >> > > > > >> > ~Josh > > > >> > > > > >> > On Mon, Aug 16, 2021 at 2:14 PM Sumanth Pasupuleti < > > > >> > sumanth.pasupuleti...@gmail.com> wrote: > > > >> > > > > >> > > Starting a discussion thread for CEP-13 > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-13%3A+Denylisting+partitions > > > >> > > > > > >> > > This CEP proposes adding a new feature to Cassandra to be able > to > > > >> > denylist > > > >> > > partitions. > > > >> > > > > > >> > > Looking forward to any feedback/ thoughts. > > > >> > > > > > >> > > Thanks, > > > >> > > Sumanth > > > >> > > > > > >> > > > > >> > > > > > > > > > >
