As Henrik said we have been refactoring access to these different internal APIs as part of some larger work. For this CEP we pulled together a bunch of the smaller ones into one place, similar to the refactoring proposed in CEP-10, as we felt doing many small CEPs, one per module, would be less productive if there was support in the project in general for trying to standardize access to different sections of the code and start creating a more defined internal API. If there is consensus that it would be better to propose each change as its own CEP, or even just as single tickets without a CEP for these internal refactors, we can do that as well. The CEP process is evolving as we go through these, so just trying to figure out the best way forward.
The currently proposed changes in CEP-18 should all include improved test coverage of the modules in question. We have been developing them all with a requirement that all changes have at least %80 code coverage from sonar cloud jacoco reports. We have also found and fixed some bugs in the existing code during this development work. To me having some defined interfaces for interacting with different sections of the code is a huge boon for improving developer productivity going forward in the project. Every place where we can reduce the amount of code reaching inside another module to get at a random internal class is a positive, as it prevents unknown side effects when changing that module when the person developing the new feature did not realize other parts of the code were depending on some current internal behavior that was not clearing part of the modules interface. On the question of changing internal interfaces that I have seen in some other venues, I do not think creating such interfaces should prevent us from changing them as needed for future work. I think having the interfaces actually improves on our ability to do so without breaking other parts of the code. My suggestion would be that we try not to make such changes in patch releases if possible, but again I wouldn’t let that hold anything back. So do people feel we should re-propose these as multiple CEP’s or just tickets? Or do people prefer to have a discussion/vote on the idea of improving the modularity of the code base in general? -Jeremiah > On Oct 25, 2021, at 9:26 AM, bened...@apache.org wrote: > > Thanks Henrik for the additional context. > > I’m not personally a fan of modularity only for modularity’s sake. Everything > in software is a balancing act of competing priorities, and while > pluggability supports certain use cases it can slow down development or > prevent deeper integrations by preventing assumptions about how systems > operate. > > To be clear, I’m fully in favour of helping to enable your use cases, I just > think it is important to make a decision for each refactor based on the > merits and goals in question. If the justification is improved testing, then > testing should be a core goal of the CEP. If it’s enabling a feature to be > upstreamed later, I personally would prefer to tie the refactors to those > features – which I hope will all find broad support for inclusion; certainly > those I have heard of, I am eager to see arrive in Cassandra. > > If the goal is to support entirely external features, we have to decide what > kind of support we offer to these APIs, and this probably needs to be > discussed on a per-API basis with the justification for pluggability weighed > against any constraints this imposes on development. The most obvious example > here is membership and schema, which I think is a primarily to support an > external dependency but we expect this area of the codebase to be > significantly revised over the coming months. > > > From: Henrik Ingo <henrik.i...@datastax.com <mailto:henrik.i...@datastax.com>> > Date: Monday, 25 October 2021 at 14:52 > To: dev@cassandra.apache.org <mailto:dev@cassandra.apache.org> > <dev@cassandra.apache.org <mailto:dev@cassandra.apache.org>> > Subject: Re: [DISCUSS] CEP-18: Improving Modularity > Hi Benedict > > This CEP is a bundle of APIs arising out of our recent work to re-architect > Cassandra into a more cloud native architecture. What our product marketing > has chosen to call "Serverless" is a variant of Cassandra where we have > separated compute from storage (coordinator vs data node), used S3-like > storage, and made various improvements to better support multi-tenancy in a > single Cassandra (Serverless) cluster. This whitepaper [1] explains this > work in detail for those of you interested to learn more. (Apologies that > it requires registration and the first page may at times sound a bit > marketingy, but it's really the most detailed report we have published so > far.) > > [1] https://www.datastax.com/resources/whitepaper/astra-serverless > > The above work was implemented in a way where by default a user can > continue to run Cassandra in the familiar "classic" way. The APIs > introduced by CEP-18 on the other hand allow alternate or additional > functionality to be provided, which in our case we have used to create a > "serverless" way of deploying a Cassandra cluster. > > The logic behind proposing this bundle of APIs separately, is roughly for > these reasons: > > The APIs touch existing code and functionality, so to minimize risk to the > next Cassandra release, it would make sense to try to complete merging this > work as early as possible in the development cycle. For the same reason, > keeping the new implementations out of this CEP allows us to focus review - > both of the CEP, and the eventual pull requests - on the APIs themselves, > whereas the related implementations (or plug-ins) would add to the scope > quite significantly. On the other hand non-default plugin functionality can > be added later with much lower risk. > > Second, while it's completely fair to ask for context, why was this > particular refactoring or API done in the first place, the assumption for a > CEP like this one is that better defined interfaces, that are better > documented and come with better test coverage than existing code, should be > enough legs to stand on in itself. Also, in the best case a good API will > also enable other implementations than the one we had in mind when > developing the API, so we wouldn't want to tie the discussion too much into > the implementation that happened to be the first. (As an example of this > working out nicely, your own work in CASSANDRA-16926 was for you motivated > by enabling a new kind of testing, but it also just so happens it is the > same work that enables someone to implement remote file storage, which we > therefore could drop from this CEP-18.) > > Conversely also, it was our expectation when proposing this CEP that > "better modularity" at least on a high level should be a fairly > straightforward conversation, while the actual plugins that make up our > "serverless" new architecture may reasonably ignite much more debate, or at > least questions as to how they work. As we have a backlog of several fairly > substantial CEPs lined up, we are trying to be very mindful of the > bandwidth of the developers on this list. For example, last week Jacek also > proposed CEP-17 for discussion. So we are trying to focus the discussion on > what's in CEP-17 and CEP-18 for now. (In addition I remember at least 2 > CEPs that were discussed but not yet voted on. I don't know if this adds to > cognitive load for anyone else than myself.) > > henrik > > On Mon, Oct 25, 2021 at 12:39 PM bened...@apache.org <bened...@apache.org> > wrote: > >> Hi Jeremiah, >> >> My personal view is that work to modularise the codebase should be tied to >> specific use cases. If improved testing is the purpose of this work, I >> think it would help to include those improved tests that you plan to >> support as goals for the CEP. >> >> If on the other hand some of this work is primarily intended to enable >> certain features, I personally think it would be preferable to tie them to >> those features - perhaps with their own CEP? >> >> >> From: Jeremiah Jordan <jeremiah.jor...@gmail.com> >> Date: Friday, 22 October 2021 at 16:24 >> To: Cassandra DEV <dev@cassandra.apache.org> >> Subject: [DISCUSS] CEP-18: Improving Modularity >> Hi All, >> As has been seen with the work already started in CEP-10, increasing the >> modularity of our subsystems can improve their testability, and also the >> ability to try new implementations without breaking things. >> >> Our team has been working on doing this and CEP-18 has been created to >> propose adding more modularity to a few different subsystems. >> >> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-18%3A+Improving+Modularity >> >> CASSANDRA-17044 has already been created for Schema Storage changes related >> to this work and more JIRAs and PRs are to follow for the other subsystems >> proposed in the CEP. >> >> Thanks, >> -Jeremiah Jordan >> > > > -- > > Henrik Ingo > > +358 40 569 7354 <358405697354> > > [image: Visit us online.] <https://www.datastax.com/> [image: Visit us on > Twitter.] <https://twitter.com/DataStaxEng> [image: Visit us on YouTube.] > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_channel_UCqA6zOSMpQ55vvguq4Y0jAg&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=bmIfaie9O3fWJAu6lESvWj3HajV4VFwgwgVuKmxKZmE&s=16sY48_kvIb7sRQORknZrr3V8iLTfemFKbMVNZhdwgw&e=> > [image: Visit my LinkedIn profile.] > <https://urldefense.com/v3/__https://www.linkedin.com/in/heingo/__;!!PbtH5S7Ebw!MiGmtcfVF1M2qLDlD18xw2bDHMJqp1cPfnoa-7WDdoWmM26YYo2vM-znIXghiwXv$ > > <https://urldefense.com/v3/__https://www.linkedin.com/in/heingo/__;!!PbtH5S7Ebw!MiGmtcfVF1M2qLDlD18xw2bDHMJqp1cPfnoa-7WDdoWmM26YYo2vM-znIXghiwXv$> > >
