I can see the case for all three: * Deny both reads and writes to a partition (wide, heavily tombstones, too many stables, etc) causing disruption to a replica set; don't want further growth nor reads until operator intervention * Deny reads but allow writing to rectify problems on a partition (intervention window; see above) * Deny writes to a partition but allow reads (prevent partitions growing unbounded, or potentially evolving into a future feature creating a ceiling on partition sizes that kicks in and demands application intervention to reduce partition size at a guardrail limit)
So yeah, at least to me at face value it seems like it'd be worth it not only to allow denylisting both reads and writes, but to be able to choose from the set of reads|writes|both on a per-partition basis. ~Josh On Thu, Aug 26, 2021 at 2:16 PM Sumanth Pasupuleti < sumanth.pasupuleti...@gmail.com> wrote: > Thank you, Josh for the elaborate explanation of a potential scenario where > denylisting writes would make sense. > I, 100% agree that could benefit in a situation where we would want to deny > writes to a partition that we do not have much control on (which is true in > most situations) and such behavior can eventually lead to unavailability of > other partitions too, as you indicate. > > Do you think it makes sense to make it configurable per partition though? > As in, maybe by default, we would want to deny both reads and writes to a > partition, but for certain partitions, we may still want to allow writes > just so we can issue a delete against that partition as an example. > Ofcourse this would make the feature and the interface more heavy, and we > need to think through if its worth it. I personally feel it could be worth > it, especially if we agree on the default behavior that makes the interface > simple in most cases. Thoughts? > > And yes, so good to see CEP process reaping benefits in multiple ways - > especially around collaboration and documentation. > > > On Thu, Aug 26, 2021 at 8:31 AM Joshua McKenzie <jmcken...@apache.org> > wrote: > > > The design doc and CEP currently pass on blocklisting / denylisting > writes > > at this time. In the proposed new patch it states: > > "Note: We do not want to blacklist writes since it is the reads that > > primarily impact the performance when reading a bad partition, and we may > > want writes to be allowed to “fix” a bad partition. We could revisit this > > in the future" > > > > In situations where you have an air gap between database ops and > > application access (ops <> application teams, or more autonomous > > application access patterns, self-service, etc), you can easily get into > a > > situation where you have either a pathological client hammering writes > to a > > specific partition causing impact to other clients or in the worst case, > > the replica set, or unbounded partition growth that again leads to > > performance degradation or replica set unavailability. The tradeoff there > > becomes "do we interrupt the application's ability to write to this > > partition now, or do we instead defer and risk losing access to *all* > > partitions on this replica set and still interrupt their access > eventually > > anyway?" > > > > Given this, I strongly advocate for support of denylisting both reads > *and* > > writes on these grounds; operators need another tool in their toolbox to > > deal with situations where specific partition writing has wider negative > > impacts on the replicas. > > > > Acknowledging of course that there was extensive discussion on this back > in > > 2018, and that would have been a *great* time to engage in the > discussion. > > =/ Good thing we have this new CEP process! :) > > > > Curious what you think about this perspective Sumanth. > > > > ~Josh > > > > > > On Tue, Aug 17, 2021 at 2:04 PM Joshua McKenzie <jmcken...@apache.org> > > wrote: > > > > > Certainly. I'll take on distilling a high level view of the feature > from > > > what Jon and I are working on to bring to this discussion. > > > > > > > > > On Tue, Aug 17, 2021 at 1:40 PM Sumanth Pasupuleti < > > > sumanth.pasupuleti...@gmail.com> wrote: > > > > > >> +1 to collaborating on the patch, Josh. My 2 cents would be to > continue > > to > > >> pursue this CEP in the community through Discuss and Vote phases and > > then > > >> invest further on the patch (based on the Vote phase outcome), so we > can > > >> reflect any additional feedback we may gather from the community > through > > >> this CEP. > > >> > > >> Thanks, > > >> Sumanth > > >> > > >> On Tue, Aug 17, 2021 at 10:13 AM Joshua McKenzie < > jmcken...@apache.org> > > >> wrote: > > >> > > >> > Sumanth, > > >> > > > >> > Jon Meredith and I are recently working on an OSS patch of one of > > those > > >> "ad > > >> > hoc" implementations of this feature that's been running at scale > for > > >> > awhile like Jirsa mentioned; sorry for not catching the discussion > on > > >> > https://issues.apache.org/jira/browse/CASSANDRA-12106 and engaging > > >> > earlier! > > >> > > > >> > I believe I can have a tidied up patch on 4.0 of this by early next > > >> week - > > >> > then we can look at the two implementations and take best of both. > > What > > >> do > > >> > you think? > > >> > > > >> > ~Josh > > >> > > > >> > On Mon, Aug 16, 2021 at 2:14 PM Sumanth Pasupuleti < > > >> > sumanth.pasupuleti...@gmail.com> wrote: > > >> > > > >> > > Starting a discussion thread for CEP-13 > > >> > > > > >> > > > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-13%3A+Denylisting+partitions > > >> > > > > >> > > This CEP proposes adding a new feature to Cassandra to be able to > > >> > denylist > > >> > > partitions. > > >> > > > > >> > > Looking forward to any feedback/ thoughts. > > >> > > > > >> > > Thanks, > > >> > > Sumanth > > >> > > > > >> > > > >> > > > > > >
