Folks, We already have multiple IEPs for Ignite 3.0, and as far as I know, there are contributors that would like to work on them (or probably already started). That said, we should make a decision as soon as possible.
At this point, it doesn't seem that there are any strong objections to the technical side of things. So I would suggest the following: 1. Proceed with Alexey's approach to the development process, as it seems to be the best (in my opinion - the only) way to address all the technical concerns and issues expressed in the thread. We'll start by creating a new repo and a new TC project. 2. Start a separate discussion around transparency. If there are any changes we need to make to our contributor guidelines, I am happy to talk them through, but I don't think it's reasonable to delay feature development because of this. In the short term, I will make sure that everything that happens within the new repo is as open to the community as possible. Nikolay, Maxim, are you OK with this route? -Val On Tue, Nov 10, 2020 at 4:55 PM Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > Maxim, > > 2.x and 3.x will have to coexist for some time - I don't see how we can > avoid this considering the set of proposed changes. That said, we > effectively will need to have two "masters" - one for each major version. > Master for 3.x can technically be a branch in the existing repo, but having > a separate repo seems cleaner, simply because it will not be a "branch" in > the traditional sense. > > Note that the new repo will still be under the Apache org, with the same > set of committers, managed by the community, etc. All the development > happening for 3.0 must follow the rules that we currently have (if > anything, it's an opportunity to improve those rules). > > As I said during the call on Friday, I strongly believe that if there is a > transparency issue, it will exist regardless of the approach we choose for > 3.0. If community members develop without IEPs or public discussions, this > will happen for both 2.x and 3.x unless we address this separately. I don't > see how this is related to Alexey's suggestion, which targets *technical* > issues with the product more than anything else. This a way to achieve > better modularity, introduce better coverage with unit tests, reduce > conflicts during development, etc. > > Coming back to transparency, let's identify the issues and fix them. It > probably makes sense to have a separate discussion on this topic. > > -Val > > On Tue, Nov 10, 2020 at 1:05 PM Maxim Muzafarov <mmu...@apache.org> wrote: > >> Sergey, >> >> >> Your summary makes sense to me. >> >> However, how we come up from *Development transparency* to *create a >> separate public repository dedicated for 3.0*? For me *development >> transparency* is about making changes in the master branch. These >> changes will definitely be seen by all the Ignite developers. >> >> A dedicated public repository is technically public and visible for >> everyone, but it allows development without IEPs, without public >> discussion (since all the code changes are not related to the master >> branch) it also allows a large number of assumptions and deviations >> (like code-style violations). It also not about *development >> transparency* since developers which are working on 3.0 is only a >> subset of all Ignite developers which may continue working on 2.x. For >> me, this would be a huge step backwards. >> >> Ignite veterans should remember how long the branch stabilization took >> for the 2.x version with the PDS. >> >> >> I think each breaking change should be passed through the master branch. >> >> On Tue, 10 Nov 2020 at 22:18, Alexei Scherbakov >> <alexey.scherbak...@gmail.com> wrote: >> > >> > Makes sense to me. >> > >> > вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov < >> sergey.chugu...@gmail.com>: >> > >> > > Igniters, >> > > >> > > I thought over Friday meeting ideas and concerns and summarized them >> in >> > > these three points: >> > > >> > > >> > > 1. *Components design unification approach.* New proposed >> components >> > > will be developed by different contributors, but they need to be >> unified >> > > and should integrate with each other easily. To ensure that I >> suggest >> > > calling an architecture group that will create design guidelines >> for all >> > > components and high-level overview of overall architecture. How >> code is >> > > split into components, what are component boundaries, how component >> > > lifecycle works and what are its interfaces - all these and other >> > > questions >> > > should be covered. >> > > >> > > 2. *Scope management.* Apache 3.0 should be implemented within a >> > > reasonable time, so we need some procedure to decide whether a >> > > particular >> > > feature should be dropped from the scope of 3.0 and postponed to >> 3.1 >> > > release. To do so I suggest to range all features by two >> parameters: >> > > criticality for 3.0 and amount of breaking changes. 3.0 scope >> should >> > > include features of high criticality AND features with a big >> amount of >> > > breaking changes. All other features can be made optional. >> > > >> > > 3. *Development transparency.* Development of all components >> should be >> > > made as transparent for everyone as possible. Any contributor >> should be >> > > able to look over any component at any stage of development. To >> achieve >> > > this I suggest to create a separate public repository dedicated >> for 3.0 >> > > development. It will make the code available for everyone but when >> > > development of 3.0 is done we won't loose any stars of our current >> > > repository as we merge dev repo into main one and drop dev. >> > > >> > > Do these ideas make sense to you? Are there any concerns not covered >> by >> > > these suggestions? >> > > >> > > On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova < >> romanova.ks....@gmail.com >> > > > >> > > wrote: >> > > >> > > > Here are the slides from Alexey Goncharuk. Let's think this over and >> > > > continue on Monday: >> > > > >> > > > >> > > >> https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf >> > > > >> > > > чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <a...@apache.org>: >> > > > >> > > > > Folks, >> > > > > >> > > > > Should we perform cleanup work before (r)evolutional changes? >> > > > > My huge proposal is to get rid of things which we don't need >> anyway >> > > > > - local caches, >> > > > > - strange tx modes, >> > > > > - code overcomplexity because of RollingUpgrade feature never >> attended >> > > at >> > > > > AI, >> > > > > - etc, >> > > > > before choosing the way. >> > > > > >> > > > > On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko < >> > > > > valentin.kuliche...@gmail.com> wrote: >> > > > > >> > > > > > Ksenia, thanks for scheduling this on such short notice! >> > > > > > >> > > > > > As for the original topic, I do support Alexey's idea. We're not >> > > going >> > > > to >> > > > > > rewrite anything from scratch, as most of the components are >> going to >> > > > be >> > > > > > moved as-is or with minimal modifications. However, the changes >> that >> > > > are >> > > > > > proposed imply serious rework of the core parts of the code, >> which >> > > are >> > > > > not >> > > > > > properly decoupled from each other and from other parts. This >> makes >> > > the >> > > > > > incremental approach borderline impossible. Developing in a new >> repo, >> > > > > > however, addresses this concern. As a bonus, we can also >> refactor the >> > > > > code, >> > > > > > introduce better decoupling, get rid of kernel context, and >> develop >> > > > unit >> > > > > > tests (finally!). >> > > > > > >> > > > > > Basically, this proposal only affects the *process*, not the >> set of >> > > > > changes >> > > > > > we had discussed before. Ignite 3.0 is our unique chance to make >> > > things >> > > > > > right. >> > > > > > >> > > > > > -Val >> > > > > > >> > > > > > On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova < >> > > > > romanova.ks....@gmail.com >> > > > > > > >> > > > > > wrote: >> > > > > > >> > > > > > > Pavel, all the interesting points will be anyway published >> here in >> > > > > > English >> > > > > > > (as the principal "if it's not on devlist it doesn't >> happened" is >> > > > still >> > > > > > > relevant). This is just a quick call for a group of >> developers. >> > > Later >> > > > > we >> > > > > > > can do a separate presentation of idea and discussion in >> English as >> > > > we >> > > > > > did >> > > > > > > for the Ignite 3.0 draft of changes. >> > > > > > > >> > > > > > > вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn < >> ptupit...@apache.org >> > > >: >> > > > > > > >> > > > > > > > Kseniya, >> > > > > > > > >> > > > > > > > Thanks for scheduling this call. >> > > > > > > > Do you think we can switch to English if non-Russian >> speaking >> > > > > community >> > > > > > > > members decide to join? >> > > > > > > > >> > > > > > > > On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova < >> > > > > > > romanova.ks....@gmail.com >> > > > > > > > > >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > > > Let's do this community discussion open. Here's the link >> on >> > > zoom >> > > > > call >> > > > > > > in >> > > > > > > > > Russian for Friday 6 PM: >> > > > > > > > > >> > > > > >> https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/ >> > > > > > > > > >> > > > > > > > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov < >> > > > nizhi...@apache.org >> > > > > >: >> > > > > > > > > >> > > > > > > > > > Time works for me. >> > > > > > > > > > >> > > > > > > > > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk < >> > > > > > > > alexey.goncha...@gmail.com >> > > > > > > > > > >> > > > > > > > > > написал(а): >> > > > > > > > > > > >> > > > > > > > > > > Nikolay, >> > > > > > > > > > > >> > > > > > > > > > > I am up for the call. I will try to explain my >> reasoning in >> > > > > > greater >> > > > > > > > > > detail >> > > > > > > > > > > and will be glad to hear the concerns. Will this >> Friday, >> > > Nov >> > > > > 6th, >> > > > > > > > work? >> > > > > > > > > > > >> > > > > > > > > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov < >> > > > > > nizhi...@apache.org >> > > > > > > >: >> > > > > > > > > > > >> > > > > > > > > > >> Igniters, should we have a call for this topic? >> > > > > > > > > > >> >> > > > > > > > > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn < >> > > > > ptupit...@apache.org >> > > > > > > >> > > > > > > > > > >> написал(а): >> > > > > > > > > > >>> >> > > > > > > > > > >>>> not intend to rewrite everything from scratch >> > > > > > > > > > >>> >> > > > > > > > > > >>>> Every single test from Ignite 2.x should be moved >> to >> > > > Ignite >> > > > > 3 >> > > > > > > > > > >>>> regardless of how we choose to proceed. >> > > > > > > > > > >>> >> > > > > > > > > > >>> Alexey, thank you for the explanation, this >> addresses all >> > > > of >> > > > > my >> > > > > > > > > > concerns. >> > > > > > > > > > >>> >> > > > > > > > > > >>> >> > > > > > > > > > >>> >> > > > > > > > > > >>> >> > > > > > > > > > >>> >> > > > > > > > > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov < >> > > > > > > > > > >> andrey.mashen...@gmail.com> >> > > > > > > > > > >>> wrote: >> > > > > > > > > > >>> >> > > > > > > > > > >>>> Hi, Igniters. >> > > > > > > > > > >>>> >> > > > > > > > > > >>>> * AFAIU, we need a new repo if we want to apply >> > > different >> > > > > > > > > restrictions >> > > > > > > > > > >> to >> > > > > > > > > > >>>> pull requests, >> > > > > > > > > > >>>> otherwise I see no difference for myself. >> > > > > > > > > > >>>> E.g. make static analysis (do we have?), compile, >> > > styles, >> > > > > and >> > > > > > > > > javadoc >> > > > > > > > > > >>>> checks mandatory. >> > > > > > > > > > >>>> >> > > > > > > > > > >>>> I think that relaxed requirements here will lead >> to bad >> > > > > > product >> > > > > > > > > > quality. >> > > > > > > > > > >>>> >> > > > > > > > > > >>>> * Agree with Pavel, we should 'keep' integrations >> tests >> > > > > > somehow. >> > > > > > > > > > >>>> During active development tests will be broken >> most of >> > > > time, >> > > > > > so, >> > > > > > > > > > >>>> I'd port them e.g. suite-by-suite once we will >> have a >> > > > stable >> > > > > > and >> > > > > > > > > > >> featured >> > > > > > > > > > >>>> environment to run them and of course make test's >> code >> > > > clear >> > > > > > and >> > > > > > > > > avoid >> > > > > > > > > > >>>> bad/non-relevant ones. >> > > > > > > > > > >>>> >> > > > > > > > > > >>>> * I like bottom-up approach. >> > > > > > > > > > >>>> With it we could make a better framework. I mean >> clear >> > > > > > component >> > > > > > > > > > >> lifecycle, >> > > > > > > > > > >>>> component wiring mechanics, general methods to >> approach >> > > > core >> > > > > > > > > > components >> > > > > > > > > > >>>> such as exchange/communication >> > > > > > > > > > >>>> to avoid code mess like we have in ExchangeFuture >> with >> > > all >> > > > > > these >> > > > > > > > > > custom >> > > > > > > > > > >>>> callbacks for each component, interfaces like >> > > > > > > > > > >>>> PartitionsExchangeAware, >> IgniteChangeGlobalStateSupport >> > > > and >> > > > > > > > > > >>>> a pack of >> > > > > > > > > > >> >> > > > > start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected >> > > > > > > > > > >>>> and so on in various unexpected places. >> > > > > > > > > > >>>> Hope, we will be able to port most of the good >> code to >> > > the >> > > > > new >> > > > > > > > > > framework >> > > > > > > > > > >>>> version. >> > > > > > > > > > >>>> >> > > > > > > > > > >>>> >> > > > > > > > > > >>>> >> > > > > > > > > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk < >> > > > > > > > > > >>>> alexey.goncha...@gmail.com> >> > > > > > > > > > >>>> wrote: >> > > > > > > > > > >>>> >> > > > > > > > > > >>>>> Nikolay, Pavel, >> > > > > > > > > > >>>>> >> > > > > > > > > > >>>>> Thanks for the feedback! First of all, I wanted to >> > > stress >> > > > > > that >> > > > > > > I >> > > > > > > > do >> > > > > > > > > > not >> > > > > > > > > > >>>>> intend to rewrite everything from scratch (I >> never used >> > > > > this >> > > > > > > > > phrase). >> > > > > > > > > > >>>> There >> > > > > > > > > > >>>>> are significant parts of code that would be moved >> with >> > > > > > minimal >> > > > > > > > > > >>>>> modifications. Second, I never said that we will >> get >> > > rid >> > > > of >> > > > > > the >> > > > > > > > old >> > > > > > > > > > >> tests >> > > > > > > > > > >>>>> codebase. Every single test from Ignite 2.x >> should be >> > > > moved >> > > > > > to >> > > > > > > > > > Ignite 3 >> > > > > > > > > > >>>>> regardless of how we choose to proceed. >> > > > > > > > > > >>>>> >> > > > > > > > > > >>>>> My point is that for some parts of the code a >> clean >> > > > > bottom-up >> > > > > > > > > > >>>>> implementation will be cheaper in many ways. Let >> me >> > > give >> > > > > you >> > > > > > a >> > > > > > > > few >> > > > > > > > > > >>>> concrete >> > > > > > > > > > >>>>> examples: >> > > > > > > > > > >>>>> >> > > > > > > > > > >>>>> - I think no one can object that we need a >> cleanly >> > > > > separated >> > > > > > > > > > >>>> persistence >> > > > > > > > > > >>>>> layer for Ignite. There is a very raw draft IEP >> for >> > > this >> > > > > > > > already. >> > > > > > > > > On >> > > > > > > > > > >>>> the >> > > > > > > > > > >>>>> other hand, I think we also can agree that we >> need a >> > > > > > > split-brain >> > > > > > > > > > >>>>> resistant >> > > > > > > > > > >>>>> replication protocol for caches. There is also >> an IEP >> > > > for >> > > > > > > this. >> > > > > > > > > > >>>> Neither >> > > > > > > > > > >>>>> of >> > > > > > > > > > >>>>> the changes is a good fit for 2.x because they >> are >> > > > likely >> > > > > to >> > > > > > > > > > >> introduce >> > > > > > > > > > >>>>> breaking changes in the persistence layer, >> > > configuration >> > > > > and >> > > > > > > > > > >> behavior. >> > > > > > > > > > >>>>> Additionally, these components are now tightly >> > > coupled, >> > > > so >> > > > > > > there >> > > > > > > > > is >> > > > > > > > > > >> no >> > > > > > > > > > >>>>> way >> > > > > > > > > > >>>>> these two changes can be implemented in parallel >> and >> > > > then >> > > > > > > merged >> > > > > > > > > > >>>>> together >> > > > > > > > > > >>>>> easily. So what we will end up with is having to >> > > > implement >> > > > > > > these >> > > > > > > > > > >>>> changes >> > > > > > > > > > >>>>> sequentially, fixing all existing tests twice, >> and >> > > > > > essentially >> > > > > > > > > > >>>> throwing >> > > > > > > > > > >>>>> away half of the work done because the other >> part of >> > > the >> > > > > > > change >> > > > > > > > is >> > > > > > > > > > >>>>> re-implemented >> > > > > > > > > > >>>>> - Similar example goes with getting rid of >> > > > > > > IgniteInternalFuture >> > > > > > > > > and >> > > > > > > > > > >>>>> replacing it with CompletableFuture, and any >> other >> > > > change >> > > > > > that >> > > > > > > > > > >> touches >> > > > > > > > > > >>>>> the >> > > > > > > > > > >>>>> asynchronous part of the code. >> > > > > > > > > > >>>>> >> > > > > > > > > > >>>>> Third, I do not see how this choice affects the >> UX of >> > > > > Ignite. >> > > > > > > The >> > > > > > > > > end >> > > > > > > > > > >>>> user >> > > > > > > > > > >>>>> experience must be fixed in the IEP regardless of >> the >> > > > > > > development >> > > > > > > > > > >> process >> > > > > > > > > > >>>>> and the fact that we have gaps in this area in >> Ignite >> > > 2.x >> > > > > > just >> > > > > > > > > > confirms >> > > > > > > > > > >>>>> that. >> > > > > > > > > > >>>>> >> > > > > > > > > > >>>>> Pavel, agree that a repo/branch is a >> technicality, I >> > > > guess >> > > > > if >> > > > > > > > > > >>>> reformulate, >> > > > > > > > > > >>>>> my point is that we might agree to have a single >> > > > > development >> > > > > > > > master >> > > > > > > > > > >>>> branch >> > > > > > > > > > >>>>> with 'disassembled' end-to-end functionality for >> some >> > > > > period >> > > > > > of >> > > > > > > > > time >> > > > > > > > > > to >> > > > > > > > > > >>>>> speed up development, and re-assemble the core >> features >> > > > > after >> > > > > > > > > having >> > > > > > > > > > >>>>> submodules tested independently. >> > > > > > > > > > >>>>> >> > > > > > > > > > >>>>> Nikolay, >> > > > > > > > > > >>>>>> We have many features that have to evolve. >> > > > > > > > > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper >> > > > support, >> > > > > > > etc. >> > > > > > > > > > >>>>> This is not very specific. In the end, resources >> are >> > > > > limited >> > > > > > > and >> > > > > > > > we >> > > > > > > > > > >> will >> > > > > > > > > > >>>>> not be able to drive both tracks simultaneously, >> > > > especially >> > > > > > > > after a >> > > > > > > > > > >>>> couple >> > > > > > > > > > >>>>> of features having been implemented for Ignite >> 3.0. If >> > > > > there >> > > > > > > are >> > > > > > > > > > indeed >> > > > > > > > > > >>>>> some major changes that we want to do in Ignite >> 2.x >> > > > instead >> > > > > > of >> > > > > > > > > > putting >> > > > > > > > > > >>>>> effort into 3.0 - let's discuss them. I am just >> not >> > > aware >> > > > > of >> > > > > > > any, >> > > > > > > > > > >> that's >> > > > > > > > > > >>>>> why I am eager to move forward with Ignite 3.0. >> > > > > > > > > > >>>>> >> > > > > > > > > > >>>>>> We have bugs and issues that can be fixed in 2.x >> > > without >> > > > > > > > breaking >> > > > > > > > > > >>>> backward >> > > > > > > > > > >>>>> compatibility. >> > > > > > > > > > >>>>>> We have many users who are happy with the 2.x >> with all >> > > > > it’s >> > > > > > > > > issues. >> > > > > > > > > > >>>>> These changes will be covered by end-to-end tests >> and >> > > > > > migrated >> > > > > > > to >> > > > > > > > > > >> Ignite >> > > > > > > > > > >>>>> 3.0, so I see no issues here. >> > > > > > > > > > >>>>> >> > > > > > > > > > >>>>> Finally, Anton & Nikolay >> > > > > > > > > > >>>>> I do not have an estimate for this simply because >> the >> > > > > > activity >> > > > > > > is >> > > > > > > > > > >>>>> community-driven and it depends on the number of >> people >> > > > > > willing >> > > > > > > > to >> > > > > > > > > > >>>>> contribute. With the current pace, I would hope >> to have >> > > > an >> > > > > RC >> > > > > > > of >> > > > > > > > > > Ignite >> > > > > > > > > > >>>> 3.0 >> > > > > > > > > > >>>>> to be ready by the end of 2021. My gut feeling is >> that >> > > by >> > > > > > > moving >> > > > > > > > > with >> > > > > > > > > > >>>>> incremental changes, we will not be able to >> implement >> > > > even >> > > > > > half >> > > > > > > > of >> > > > > > > > > > the >> > > > > > > > > > >>>>> wishlist by that time. >> > > > > > > > > > >>>>> I doubt that releasing several major releases with >> > > > breaking >> > > > > > > > changes >> > > > > > > > > > >> will >> > > > > > > > > > >>>>> make Ignite users happy either because each >> upgrade >> > > will >> > > > > cost >> > > > > > > > > Ignite >> > > > > > > > > > >>>> users >> > > > > > > > > > >>>>> money, so the fewer major versions we release, the >> > > > better. >> > > > > > Thus >> > > > > > > > my >> > > > > > > > > > wish >> > > > > > > > > > >>>> to >> > > > > > > > > > >>>>> include all breaking changes in one release. >> > > > > > > > > > >>>>> >> > > > > > > > > > >>>>> I'll be now quiet for a while, let's see what >> other >> > > > > community >> > > > > > > > > members >> > > > > > > > > > >>>>> think. >> > > > > > > > > > >>>>> >> > > > > > > > > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn < >> > > > > > > > ptupit...@apache.org >> > > > > > > > > >: >> > > > > > > > > > >>>>> >> > > > > > > > > > >>>>>> 1. Rewriting from scratch is never a good idea. >> > > > > > > > > > >>>>>> We don't want to follow the path of Netscape and >> lose >> > > > all >> > > > > > our >> > > > > > > > > users >> > > > > > > > > > >>>>>> by the time we have a working 3.0 [1] >> > > > > > > > > > >>>>>> >> > > > > > > > > > >>>>>> 2. Not sure about new repo - seems like some >> pain and >> > > no >> > > > > > gain, >> > > > > > > > > > what's >> > > > > > > > > > >>>> the >> > > > > > > > > > >>>>>> problem with a branch? >> > > > > > > > > > >>>>>> >> > > > > > > > > > >>>>>> 3. We should keep existing integration tests when >> > > > > possible. >> > > > > > > > > > >>>>>> We have accumulated a lot of edge case knowledge >> over >> > > > the >> > > > > > > years, >> > > > > > > > > > >>>>>> it is not a good idea to send all of that down >> the >> > > > drain. >> > > > > > > > > > >>>>>> Yes, integration tests are slow, but they are >> the most >> > > > > > > valuable. >> > > > > > > > > > >>>>>> I think we can move more stuff into nightly runs >> and >> > > > have >> > > > > a >> > > > > > > fast >> > > > > > > > > and >> > > > > > > > > > >>>>> modern >> > > > > > > > > > >>>>>> basic suite. >> > > > > > > > > > >>>>>> >> > > > > > > > > > >>>>>> >> > > > > > > > > > >>>>>> Alexey, you are much more familiar with the >> Ignite >> > > core >> > > > > > > codebase >> > > > > > > > > > than >> > > > > > > > > > >>>>> most >> > > > > > > > > > >>>>>> of us, >> > > > > > > > > > >>>>>> can you please explain in more detail which >> particular >> > > > > > > feature, >> > > > > > > > in >> > > > > > > > > > >> your >> > > > > > > > > > >>>>>> opinion, >> > > > > > > > > > >>>>>> mandates this "start from scratch" approach? >> > > > > > > > > > >>>>>> Is it really not possible at all to follow a less >> > > > radical >> > > > > > way? >> > > > > > > > > > >>>>>> >> > > > > > > > > > >>>>>> >> > > > > > > > > > >>>>>> [1] >> > > > > > > > > > >>>>>> >> > > > > > > > > > >>>>>> >> > > > > > > > > > >>>>> >> > > > > > > > > > >>>> >> > > > > > > > > > >> >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ >> > > > > > > > > > >>>>>> >> > > > > > > > > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov < >> > > > > > > > > nizhi...@apache.org >> > > > > > > > > > > >> > > > > > > > > > >>>>>> wrote: >> > > > > > > > > > >>>>>> >> > > > > > > > > > >>>>>>> Hello, Alexey. >> > > > > > > > > > >>>>>>> >> > > > > > > > > > >>>>>>> I think that «rewriting from scratch» approach >> has a >> > > > high >> > > > > > > risk >> > > > > > > > to >> > > > > > > > > > >>>> make >> > > > > > > > > > >>>>>> new >> > > > > > > > > > >>>>>>> features unusable. >> > > > > > > > > > >>>>>>> At the time Ignite2 was started no-one wants to >> do >> > > bad >> > > > UX >> > > > > > or >> > > > > > > > bad >> > > > > > > > > > >>>>>> features. >> > > > > > > > > > >>>>>>> Nevertheless, it happen. >> > > > > > > > > > >>>>>>> >> > > > > > > > > > >>>>>>> I think we can avoid it with the Ignite3 and >> > > successors >> > > > > if >> > > > > > we >> > > > > > > > > will >> > > > > > > > > > >>>> move >> > > > > > > > > > >>>>>>> step by step without keeping backward >> compatibility >> > > > > > > > > > >>>>>>> With the step by step approach, we can focus on >> each >> > > > > > > component >> > > > > > > > > > >>>>>> separately. >> > > > > > > > > > >>>>>>> >> > > > > > > > > > >>>>>>>> What new features are we planning to implement >> for >> > > > > Ignite >> > > > > > > 2.x? >> > > > > > > > > > >>>>>>> >> > > > > > > > > > >>>>>>> We have many features that have to evolve. >> > > > > > > > > > >>>>>>> Snapshots, rebalance, tooling, tracing, >> zookeeper >> > > > > support, >> > > > > > > etc. >> > > > > > > > > > >>>>>>> We have bugs and issues that can be fixed in 2.x >> > > > without >> > > > > > > > breaking >> > > > > > > > > > >>>>>> backward >> > > > > > > > > > >>>>>>> compatibility. >> > > > > > > > > > >>>>>>> We have many users who are happy with the 2.x >> with >> > > all >> > > > > it’s >> > > > > > > > > issues. >> > > > > > > > > > >>>>>>> >> > > > > > > > > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov < >> > > > > a...@apache.org >> > > > > > > >> > > > > > > > > > >>>>> написал(а): >> > > > > > > > > > >>>>>>>> >> > > > > > > > > > >>>>>>>> Alexey, >> > > > > > > > > > >>>>>>>> >> > > > > > > > > > >>>>>>>> Do we have any estimates of how fast we'll be >> able >> > > to >> > > > > gain >> > > > > > > > > > >>>>>>> production-ready >> > > > > > > > > > >>>>>>>> AI 3.0 in case of a "new repo" choice? >> > > > > > > > > > >>>>>>>> >> > > > > > > > > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey >> Goncharuk < >> > > > > > > > > > >>>>>>> alexey.goncha...@gmail.com> >> > > > > > > > > > >>>>>>>> wrote: >> > > > > > > > > > >>>>>>>> >> > > > > > > > > > >>>>>>>>> Nikolay, >> > > > > > > > > > >>>>>>>>> >> > > > > > > > > > >>>>>>>>> What new features are we planning to >> implement for >> > > > > Ignite >> > > > > > > > 2.x? >> > > > > > > > > I >> > > > > > > > > > >>>>> think >> > > > > > > > > > >>>>>>> once >> > > > > > > > > > >>>>>>>>> we commence working on Ignite 3.0, we should >> > > > gradually >> > > > > > > cease >> > > > > > > > > the >> > > > > > > > > > >>>>>>> activity >> > > > > > > > > > >>>>>>>>> on Ignite 2.x to mere bugfixes because such >> > > parallel >> > > > > > > > > development >> > > > > > > > > > >>>>> will >> > > > > > > > > > >>>>>> be >> > > > > > > > > > >>>>>>>>> overwhelming regardless of how we choose to >> > > proceed. >> > > > > > > > > > >>>>>>>>> >> > > > > > > > > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov < >> > > > > > > > > > nizhi...@apache.org >> > > > > > > > > > >>>>> : >> > > > > > > > > > >>>>>>>>> >> > > > > > > > > > >>>>>>>>>> To be clear: >> > > > > > > > > > >>>>>>>>>> >> > > > > > > > > > >>>>>>>>>>> I would suggest creating a new repository >> for >> > > > Ignite >> > > > > > 3.0 >> > > > > > > > > > >>>>> (perhaps, a >> > > > > > > > > > >>>>>>>>> new >> > > > > > > > > > >>>>>>>>>> clean branch, but a new repo looks nicer to >> me) >> > > and >> > > > a >> > > > > > new >> > > > > > > > > Ignite >> > > > > > > > > > >>>>> 3.0 >> > > > > > > > > > >>>>>>>>>> TeamCity project. >> > > > > > > > > > >>>>>>>>>> >> > > > > > > > > > >>>>>>>>>> +1 for new Team City project. >> > > > > > > > > > >>>>>>>>>> +1 for new branch for Ignite3. >> > > > > > > > > > >>>>>>>>>> -1 for new repo. >> > > > > > > > > > >>>>>>>>>> >> > > > > > > > > > >>>>>>>>>> >> > > > > > > > > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov < >> > > > > > > > > > >>>> nizhikov....@gmail.com >> > > > > > > > > > >>>>>> >> > > > > > > > > > >>>>>>>>>> написал(а): >> > > > > > > > > > >>>>>>>>>>> >> > > > > > > > > > >>>>>>>>>>> Hello, Alexey. >> > > > > > > > > > >>>>>>>>>>> >> > > > > > > > > > >>>>>>>>>>> I think it will hurt our project more than >> help. >> > > > > > > > > > >>>>>>>>>>> Developing new features for 2 separate >> branches >> > > > with >> > > > > > the >> > > > > > > > > > >>>> different >> > > > > > > > > > >>>>>>> APIs >> > > > > > > > > > >>>>>>>>>> and internal structure is overwhelming >> > > > > > > > > > >>>>>>>>>>> >> > > > > > > > > > >>>>>>>>>>> Maybe we should relax a bit requirements for >> > > > Ignite3? >> > > > > > > > > > >>>>>>>>>>> Maybe we should move step by step and make >> > > Ignite3 >> > > > > with >> > > > > > > new >> > > > > > > > > > >>>>>>>>>> configuration than Ignite4 with new >> transactions, >> > > > etc? >> > > > > > > > > > >>>>>>>>>>> >> > > > > > > > > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk >> < >> > > > > > > > > > >>>>>>>>> alexey.goncha...@gmail.com> >> > > > > > > > > > >>>>>>>>>> написал(а): >> > > > > > > > > > >>>>>>>>>>>> >> > > > > > > > > > >>>>>>>>>>>> Igniters, >> > > > > > > > > > >>>>>>>>>>>> >> > > > > > > > > > >>>>>>>>>>>> I wanted to pitch a rather radical idea >> > > regarding >> > > > > the >> > > > > > > > Ignite >> > > > > > > > > > >>>> 3.0 >> > > > > > > > > > >>>>>>>>>>>> development which has occurred to me some >> time >> > > > ago. >> > > > > > > > > > >>>>>>>>>>>> >> > > > > > > > > > >>>>>>>>>>>> We already have several IEPs targeted to >> Ignite >> > > > 3.0 >> > > > > > > which >> > > > > > > > > > imply >> > > > > > > > > > >>>>>> major >> > > > > > > > > > >>>>>>>>>>>> changes to the codebase (the change in >> > > replication >> > > > > > > > protocol >> > > > > > > > > > and >> > > > > > > > > > >>>>>> thus >> > > > > > > > > > >>>>>>>>>>>> transactions, change in binary format, >> updated >> > > > > > > > metastorage, >> > > > > > > > > > >>>> etc). >> > > > > > > > > > >>>>>> We >> > > > > > > > > > >>>>>>>>>> also >> > > > > > > > > > >>>>>>>>>>>> planned significant changes in public APIs: >> > > > > > > configuration >> > > > > > > > > > >>>> format >> > > > > > > > > > >>>>>>>>> change, >> > > > > > > > > > >>>>>>>>>>>> improvements in cache APIs, SQL APIs, >> > > transaction >> > > > > mode >> > > > > > > > > rework. >> > > > > > > > > > >>>>> The >> > > > > > > > > > >>>>>>>>>> wishlist >> > > > > > > > > > >>>>>>>>>>>> of changes for Ignite 3.0 is huge. >> > > > > > > > > > >>>>>>>>>>>> >> > > > > > > > > > >>>>>>>>>>>> So, I was wondering whether it makes sense >> to >> > > try >> > > > to >> > > > > > > > change >> > > > > > > > > > the >> > > > > > > > > > >>>>> old >> > > > > > > > > > >>>>>>>>>>>> codebase, or start with a new baseline and >> move >> > > > old >> > > > > > > pieces >> > > > > > > > > of >> > > > > > > > > > >>>>> code >> > > > > > > > > > >>>>>>>>> that >> > > > > > > > > > >>>>>>>>>> do >> > > > > > > > > > >>>>>>>>>>>> not require significant rework. >> Personally, I >> > > > would >> > > > > go >> > > > > > > > with >> > > > > > > > > > the >> > > > > > > > > > >>>>>>> second >> > > > > > > > > > >>>>>>>>>>>> option for the following reasons: >> > > > > > > > > > >>>>>>>>>>>> >> > > > > > > > > > >>>>>>>>>>>> - We have a chance to shift the development >> > > > paradigm >> > > > > > in >> > > > > > > > the >> > > > > > > > > > >>>>> project >> > > > > > > > > > >>>>>>>>> and >> > > > > > > > > > >>>>>>>>>>>> introduce the practice of true unit-tests. >> In >> > > the >> > > > > new >> > > > > > > > > baseline >> > > > > > > > > > >>>> at >> > > > > > > > > > >>>>>> the >> > > > > > > > > > >>>>>>>>>>>> beginning there will be no ability to run >> an >> > > > > > end-to-end >> > > > > > > > > > >>>> scenario, >> > > > > > > > > > >>>>>>>>> thus >> > > > > > > > > > >>>>>>>>>> we >> > > > > > > > > > >>>>>>>>>>>> will be forced to write unit-tests. So >> far, such >> > > > > > > practice >> > > > > > > > > was >> > > > > > > > > > >>>>> hard >> > > > > > > > > > >>>>>> to >> > > > > > > > > > >>>>>>>>>>>> implement because of tight coupling between >> > > Ignite >> > > > > > > > > components >> > > > > > > > > > >>>> and >> > > > > > > > > > >>>>>>>>>> inability >> > > > > > > > > > >>>>>>>>>>>> to instantiate components without an >> instance of >> > > > > > > > > > KernalContext. >> > > > > > > > > > >>>>> For >> > > > > > > > > > >>>>>>>>>>>> example, we should be able to thoroughly >> test >> > > > > internal >> > > > > > > > > > >>>>> primitives, >> > > > > > > > > > >>>>>>>>>> such as >> > > > > > > > > > >>>>>>>>>>>> replication protocol (without actual >> > > > communication), >> > > > > > > > > > >>>> distributed >> > > > > > > > > > >>>>>>>>>>>> metastorage contracts, persistence layer, >> etc. >> > > > > > > > > > >>>>>>>>>>>> - We will significantly reduce the >> development >> > > > cycle >> > > > > > in >> > > > > > > > the >> > > > > > > > > > >>>>>> beginning >> > > > > > > > > > >>>>>>>>>>>> (right now the RunAll takes two hours of >> > > > > astronomical >> > > > > > > time >> > > > > > > > > > with >> > > > > > > > > > >>>>>> empty >> > > > > > > > > > >>>>>>>>>> TC; >> > > > > > > > > > >>>>>>>>>>>> in the new approach developer will be able >> to >> > > run >> > > > > ALL >> > > > > > > > tests >> > > > > > > > > > >>>>> locally >> > > > > > > > > > >>>>>>>>> in >> > > > > > > > > > >>>>>>>>>> a >> > > > > > > > > > >>>>>>>>>>>> matter of minutes) >> > > > > > > > > > >>>>>>>>>>>> - We can get rid of TC bot and enforce >> green TC >> > > by >> > > > > > > > > integrating >> > > > > > > > > > >>>> TC >> > > > > > > > > > >>>>>>>>> build >> > > > > > > > > > >>>>>>>>>>>> results with GitHub PRs (the same way >> Travis is >> > > > > > > currently >> > > > > > > > > > >>>>>> integrated >> > > > > > > > > > >>>>>>>>>> to PR >> > > > > > > > > > >>>>>>>>>>>> check). We should restrict PR merge >> without a TC >> > > > > check >> > > > > > > > > > >>>>>>>>>>>> - We will still have to re-write all >> tests, but >> > > > only >> > > > > > > once. >> > > > > > > > > If >> > > > > > > > > > >>>> we >> > > > > > > > > > >>>>>> try >> > > > > > > > > > >>>>>>>>> to >> > > > > > > > > > >>>>>>>>>>>> modify the old codebase, we would need to >> modify >> > > > all >> > > > > > the >> > > > > > > > > tests >> > > > > > > > > > >>>>> for >> > > > > > > > > > >>>>>>>>>> every >> > > > > > > > > > >>>>>>>>>>>> major change (public API change, >> configuration >> > > > > change) >> > > > > > > > > > >>>>>>>>>>>> - We will have fewer conflicts when working >> > > > > together. >> > > > > > > For >> > > > > > > > > > >>>>> example, >> > > > > > > > > > >>>>>> I >> > > > > > > > > > >>>>>>>>>>>> cannot imagine how one would merge two >> changes >> > > of >> > > > > > > getting >> > > > > > > > > rid >> > > > > > > > > > >>>> of >> > > > > > > > > > >>>>>>>>>>>> IgniteFuture and changes in replication >> > > protocol, >> > > > > for >> > > > > > > > > example >> > > > > > > > > > >>>>>>>>>>>> >> > > > > > > > > > >>>>>>>>>>>> Technically, I would suggest creating a new >> > > > > repository >> > > > > > > for >> > > > > > > > > > >>>> Ignite >> > > > > > > > > > >>>>>> 3.0 >> > > > > > > > > > >>>>>>>>>>>> (perhaps, a new clean branch, but a new >> repo >> > > looks >> > > > > > nicer >> > > > > > > > to >> > > > > > > > > > me) >> > > > > > > > > > >>>>>> and a >> > > > > > > > > > >>>>>>>>>> new >> > > > > > > > > > >>>>>>>>>>>> Ignite 3.0 TeamCity project. >> > > > > > > > > > >>>>>>>>>>>> >> > > > > > > > > > >>>>>>>>>>>> While it may seem quite radical, I do >> believe >> > > that >> > > > > > this >> > > > > > > > > > >>>> approach >> > > > > > > > > > >>>>>> will >> > > > > > > > > > >>>>>>>>>> give >> > > > > > > > > > >>>>>>>>>>>> us more benefits than trying to make such >> major >> > > > > > changes >> > > > > > > in >> > > > > > > > > the >> > > > > > > > > > >>>>>>>>> existing >> > > > > > > > > > >>>>>>>>>>>> codebase. If needed, let's schedule a >> discord >> > > chat >> > > > > > like >> > > > > > > > > before >> > > > > > > > > > >>>> to >> > > > > > > > > > >>>>>>>>>> discuss >> > > > > > > > > > >>>>>>>>>>>> this. >> > > > > > > > > > >>>>>>>>>>>> >> > > > > > > > > > >>>>>>>>>>>> WDYT? >> > > > > > > > > > >>>>>>>>>>> >> > > > > > > > > > >>>>>>>>>> >> > > > > > > > > > >>>>>>>>>> >> > > > > > > > > > >>>>>>>>> >> > > > > > > > > > >>>>>>> >> > > > > > > > > > >>>>>>> >> > > > > > > > > > >>>>>> >> > > > > > > > > > >>>>> >> > > > > > > > > > >>>> >> > > > > > > > > > >>>> >> > > > > > > > > > >>>> -- >> > > > > > > > > > >>>> Best regards, >> > > > > > > > > > >>>> Andrey V. Mashenkov >> > > > > > > > > > >>>> >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >> > -- >> > >> > Best regards, >> > Alexei Scherbakov >> >
