I'm +1 on where it currently stands after the revisions. Consider resolving out comment threads on the design doc that are closed so we can see if there's any outstanding discussions from a high level?
~Josh On Mon, Aug 30, 2021 at 1:14 AM Sumanth Pasupuleti < sumanth.pasupuleti...@gmail.com> wrote: > +1. Made changes to the design document linked against the CEP to reflect > this feedback. Specifically, the following sections have been updated > * Operations to blacklist > * Blacklist information store > > Thanks, > Sumanth > > > On Fri, Aug 27, 2021 at 7:57 AM Joshua McKenzie <jmcken...@apache.org> > wrote: > > > I can see the case for all three: > > * Deny both reads and writes to a partition (wide, heavily tombstones, > too > > many stables, etc) causing disruption to a replica set; don't want > further > > growth nor reads until operator intervention > > * Deny reads but allow writing to rectify problems on a partition > > (intervention window; see above) > > * Deny writes to a partition but allow reads (prevent partitions growing > > unbounded, or potentially evolving into a future feature creating a > ceiling > > on partition sizes that kicks in and demands application intervention to > > reduce partition size at a guardrail limit) > > > > So yeah, at least to me at face value it seems like it'd be worth it not > > only to allow denylisting both reads and writes, but to be able to choose > > from the set of reads|writes|both on a per-partition basis. > > > > ~Josh > > > > > > On Thu, Aug 26, 2021 at 2:16 PM Sumanth Pasupuleti < > > sumanth.pasupuleti...@gmail.com> wrote: > > > > > Thank you, Josh for the elaborate explanation of a potential scenario > > where > > > denylisting writes would make sense. > > > I, 100% agree that could benefit in a situation where we would want to > > deny > > > writes to a partition that we do not have much control on (which is > true > > in > > > most situations) and such behavior can eventually lead to > unavailability > > of > > > other partitions too, as you indicate. > > > > > > Do you think it makes sense to make it configurable per partition > though? > > > As in, maybe by default, we would want to deny both reads and writes > to a > > > partition, but for certain partitions, we may still want to allow > writes > > > just so we can issue a delete against that partition as an example. > > > Ofcourse this would make the feature and the interface more heavy, and > we > > > need to think through if its worth it. I personally feel it could be > > worth > > > it, especially if we agree on the default behavior that makes the > > interface > > > simple in most cases. Thoughts? > > > > > > And yes, so good to see CEP process reaping benefits in multiple ways - > > > especially around collaboration and documentation. > > > > > > > > > On Thu, Aug 26, 2021 at 8:31 AM Joshua McKenzie <jmcken...@apache.org> > > > wrote: > > > > > > > The design doc and CEP currently pass on blocklisting / denylisting > > > writes > > > > at this time. In the proposed new patch it states: > > > > "Note: We do not want to blacklist writes since it is the reads that > > > > primarily impact the performance when reading a bad partition, and we > > may > > > > want writes to be allowed to “fix” a bad partition. We could revisit > > this > > > > in the future" > > > > > > > > In situations where you have an air gap between database ops and > > > > application access (ops <> application teams, or more autonomous > > > > application access patterns, self-service, etc), you can easily get > > into > > > a > > > > situation where you have either a pathological client hammering > writes > > > to a > > > > specific partition causing impact to other clients or in the worst > > case, > > > > the replica set, or unbounded partition growth that again leads to > > > > performance degradation or replica set unavailability. The tradeoff > > there > > > > becomes "do we interrupt the application's ability to write to this > > > > partition now, or do we instead defer and risk losing access to *all* > > > > partitions on this replica set and still interrupt their access > > > eventually > > > > anyway?" > > > > > > > > Given this, I strongly advocate for support of denylisting both reads > > > *and* > > > > writes on these grounds; operators need another tool in their toolbox > > to > > > > deal with situations where specific partition writing has wider > > negative > > > > impacts on the replicas. > > > > > > > > Acknowledging of course that there was extensive discussion on this > > back > > > in > > > > 2018, and that would have been a *great* time to engage in the > > > discussion. > > > > =/ Good thing we have this new CEP process! :) > > > > > > > > Curious what you think about this perspective Sumanth. > > > > > > > > ~Josh > > > > > > > > > > > > On Tue, Aug 17, 2021 at 2:04 PM Joshua McKenzie < > jmcken...@apache.org> > > > > wrote: > > > > > > > > > Certainly. I'll take on distilling a high level view of the feature > > > from > > > > > what Jon and I are working on to bring to this discussion. > > > > > > > > > > > > > > > On Tue, Aug 17, 2021 at 1:40 PM Sumanth Pasupuleti < > > > > > sumanth.pasupuleti...@gmail.com> wrote: > > > > > > > > > >> +1 to collaborating on the patch, Josh. My 2 cents would be to > > > continue > > > > to > > > > >> pursue this CEP in the community through Discuss and Vote phases > and > > > > then > > > > >> invest further on the patch (based on the Vote phase outcome), so > we > > > can > > > > >> reflect any additional feedback we may gather from the community > > > through > > > > >> this CEP. > > > > >> > > > > >> Thanks, > > > > >> Sumanth > > > > >> > > > > >> On Tue, Aug 17, 2021 at 10:13 AM Joshua McKenzie < > > > jmcken...@apache.org> > > > > >> wrote: > > > > >> > > > > >> > Sumanth, > > > > >> > > > > > >> > Jon Meredith and I are recently working on an OSS patch of one > of > > > > those > > > > >> "ad > > > > >> > hoc" implementations of this feature that's been running at > scale > > > for > > > > >> > awhile like Jirsa mentioned; sorry for not catching the > discussion > > > on > > > > >> > https://issues.apache.org/jira/browse/CASSANDRA-12106 and > > engaging > > > > >> > earlier! > > > > >> > > > > > >> > I believe I can have a tidied up patch on 4.0 of this by early > > next > > > > >> week - > > > > >> > then we can look at the two implementations and take best of > both. > > > > What > > > > >> do > > > > >> > you think? > > > > >> > > > > > >> > ~Josh > > > > >> > > > > > >> > On Mon, Aug 16, 2021 at 2:14 PM Sumanth Pasupuleti < > > > > >> > sumanth.pasupuleti...@gmail.com> wrote: > > > > >> > > > > > >> > > Starting a discussion thread for CEP-13 > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-13%3A+Denylisting+partitions > > > > >> > > > > > > >> > > This CEP proposes adding a new feature to Cassandra to be able > > to > > > > >> > denylist > > > > >> > > partitions. > > > > >> > > > > > > >> > > Looking forward to any feedback/ thoughts. > > > > >> > > > > > > >> > > Thanks, > > > > >> > > Sumanth > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >
