Igniters, should we have a call for this topic?
> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <ptupit...@apache.org> написал(а):
>
>> not intend to rewrite everything from scratch
>
>> Every single test from Ignite 2.x should be moved to Ignite 3
>> regardless of how we choose to proceed.
>
> Alexey, thank you for the explanation, this addresses all of my concerns.
>
>
>
>
>
> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <andrey.mashen...@gmail.com>
> wrote:
>
>> Hi, Igniters.
>>
>> * AFAIU, we need a new repo if we want to apply different restrictions to
>> pull requests,
>> otherwise I see no difference for myself.
>> E.g. make static analysis (do we have?), compile, styles, and javadoc
>> checks mandatory.
>>
>> I think that relaxed requirements here will lead to bad product quality.
>>
>> * Agree with Pavel, we should 'keep' integrations tests somehow.
>> During active development tests will be broken most of time, so,
>> I'd port them e.g. suite-by-suite once we will have a stable and featured
>> environment to run them and of course make test's code clear and avoid
>> bad/non-relevant ones.
>>
>> * I like bottom-up approach.
>> With it we could make a better framework. I mean clear component lifecycle,
>> component wiring mechanics, general methods to approach core components
>> such as exchange/communication
>> to avoid code mess like we have in ExchangeFuture with all these custom
>> callbacks for each component, interfaces like
>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
>> a pack of start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
>> and so on in various unexpected places.
>> Hope, we will be able to port most of the good code to the new framework
>> version.
>>
>>
>>
>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
>> alexey.goncha...@gmail.com>
>> wrote:
>>
>>> Nikolay, Pavel,
>>>
>>> Thanks for the feedback! First of all, I wanted to stress that I do not
>>> intend to rewrite everything from scratch (I never used this phrase).
>> There
>>> are significant parts of code that would be moved with minimal
>>> modifications. Second, I never said that we will get rid of the old tests
>>> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
>>> regardless of how we choose to proceed.
>>>
>>> My point is that for some parts of the code a clean bottom-up
>>> implementation will be cheaper in many ways. Let me give you a few
>> concrete
>>> examples:
>>>
>>> - I think no one can object that we need a cleanly separated
>> persistence
>>> layer for Ignite. There is a very raw draft IEP for this already. On
>> the
>>> other hand, I think we also can agree that we need a split-brain
>>> resistant
>>> replication protocol for caches. There is also an IEP for this.
>> Neither
>>> of
>>> the changes is a good fit for 2.x because they are likely to introduce
>>> breaking changes in the persistence layer, configuration and behavior.
>>> Additionally, these components are now tightly coupled, so there is no
>>> way
>>> these two changes can be implemented in parallel and then merged
>>> together
>>> easily. So what we will end up with is having to implement these
>> changes
>>> sequentially, fixing all existing tests twice, and essentially
>> throwing
>>> away half of the work done because the other part of the change is
>>> re-implemented
>>> - Similar example goes with getting rid of IgniteInternalFuture and
>>> replacing it with CompletableFuture, and any other change that touches
>>> the
>>> asynchronous part of the code.
>>>
>>> Third, I do not see how this choice affects the UX of Ignite. The end
>> user
>>> experience must be fixed in the IEP regardless of the development process
>>> and the fact that we have gaps in this area in Ignite 2.x just confirms
>>> that.
>>>
>>> Pavel, agree that a repo/branch is a technicality, I guess if
>> reformulate,
>>> my point is that we might agree to have a single development master
>> branch
>>> with 'disassembled' end-to-end functionality for some period of time to
>>> speed up development, and re-assemble the core features after having
>>> submodules tested independently.
>>>
>>> Nikolay,
>>>> We have many features that have to evolve.
>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
>>> This is not very specific. In the end, resources are limited and we will
>>> not be able to drive both tracks simultaneously, especially after a
>> couple
>>> of features having been implemented for Ignite 3.0. If there are indeed
>>> some major changes that we want to do in Ignite 2.x instead of putting
>>> effort into 3.0 - let's discuss them. I am just not aware of any, that's
>>> why I am eager to move forward with Ignite 3.0.
>>>
>>>> We have bugs and issues that can be fixed in 2.x without breaking
>> backward
>>> compatibility.
>>>> We have many users who are happy with the 2.x with all it’s issues.
>>> These changes will be covered by end-to-end tests and migrated to Ignite
>>> 3.0, so I see no issues here.
>>>
>>> Finally, Anton & Nikolay
>>> I do not have an estimate for this simply because the activity is
>>> community-driven and it depends on the number of people willing to
>>> contribute. With the current pace, I would hope to have an RC of Ignite
>> 3.0
>>> to be ready by the end of 2021. My gut feeling is that by moving with
>>> incremental changes, we will not be able to implement even half of the
>>> wishlist by that time.
>>> I doubt that releasing several major releases with breaking changes will
>>> make Ignite users happy either because each upgrade will cost Ignite
>> users
>>> money, so the fewer major versions we release, the better. Thus my wish
>> to
>>> include all breaking changes in one release.
>>>
>>> I'll be now quiet for a while, let's see what other community members
>>> think.
>>>
>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <ptupit...@apache.org>:
>>>
>>>> 1. Rewriting from scratch is never a good idea.
>>>> We don't want to follow the path of Netscape and lose all our users
>>>> by the time we have a working 3.0 [1]
>>>>
>>>> 2. Not sure about new repo - seems like some pain and no gain, what's
>> the
>>>> problem with a branch?
>>>>
>>>> 3. We should keep existing integration tests when possible.
>>>> We have accumulated a lot of edge case knowledge over the years,
>>>> it is not a good idea to send all of that down the drain.
>>>> Yes, integration tests are slow, but they are the most valuable.
>>>> I think we can move more stuff into nightly runs and have a fast and
>>> modern
>>>> basic suite.
>>>>
>>>>
>>>> Alexey, you are much more familiar with the Ignite core codebase than
>>> most
>>>> of us,
>>>> can you please explain in more detail which particular feature, in your
>>>> opinion,
>>>> mandates this "start from scratch" approach?
>>>> Is it really not possible at all to follow a less radical way?
>>>>
>>>>
>>>> [1]
>>>>
>>>>
>>>
>> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
>>>>
>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <nizhi...@apache.org>
>>>> wrote:
>>>>
>>>>> Hello, Alexey.
>>>>>
>>>>> I think that «rewriting from scratch» approach has a high risk to
>> make
>>>> new
>>>>> features unusable.
>>>>> At the time Ignite2 was started no-one wants to do bad UX or bad
>>>> features.
>>>>> Nevertheless, it happen.
>>>>>
>>>>> I think we can avoid it with the Ignite3 and successors if we will
>> move
>>>>> step by step without keeping backward compatibility
>>>>> With the step by step approach, we can focus on each component
>>>> separately.
>>>>>
>>>>>> What new features are we planning to implement for Ignite 2.x?
>>>>>
>>>>> We have many features that have to evolve.
>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
>>>>> We have bugs and issues that can be fixed in 2.x without breaking
>>>> backward
>>>>> compatibility.
>>>>> We have many users who are happy with the 2.x with all it’s issues.
>>>>>
>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <a...@apache.org>
>>> написал(а):
>>>>>>
>>>>>> Alexey,
>>>>>>
>>>>>> Do we have any estimates of how fast we'll be able to gain
>>>>> production-ready
>>>>>> AI 3.0 in case of a "new repo" choice?
>>>>>>
>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
>>>>> alexey.goncha...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Nikolay,
>>>>>>>
>>>>>>> What new features are we planning to implement for Ignite 2.x? I
>>> think
>>>>> once
>>>>>>> we commence working on Ignite 3.0, we should gradually cease the
>>>>> activity
>>>>>>> on Ignite 2.x to mere bugfixes because such parallel development
>>> will
>>>> be
>>>>>>> overwhelming regardless of how we choose to proceed.
>>>>>>>
>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <nizhi...@apache.org
>>> :
>>>>>>>
>>>>>>>> To be clear:
>>>>>>>>
>>>>>>>>> I would suggest creating a new repository for Ignite 3.0
>>> (perhaps, a
>>>>>>> new
>>>>>>>> clean branch, but a new repo looks nicer to me) and a new Ignite
>>> 3.0
>>>>>>>> TeamCity project.
>>>>>>>>
>>>>>>>> +1 for new Team City project.
>>>>>>>> +1 for new branch for Ignite3.
>>>>>>>> -1 for new repo.
>>>>>>>>
>>>>>>>>
>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
>> nizhikov....@gmail.com
>>>>
>>>>>>>> написал(а):
>>>>>>>>>
>>>>>>>>> Hello, Alexey.
>>>>>>>>>
>>>>>>>>> I think it will hurt our project more than help.
>>>>>>>>> Developing new features for 2 separate branches with the
>> different
>>>>> APIs
>>>>>>>> and internal structure is overwhelming
>>>>>>>>>
>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
>>>>>>>>> Maybe we should move step by step and make Ignite3 with new
>>>>>>>> configuration than Ignite4 with new transactions, etc?
>>>>>>>>>
>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
>>>>>>> alexey.goncha...@gmail.com>
>>>>>>>> написал(а):
>>>>>>>>>>
>>>>>>>>>> Igniters,
>>>>>>>>>>
>>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite
>> 3.0
>>>>>>>>>> development which has occurred to me some time ago.
>>>>>>>>>>
>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which imply
>>>> major
>>>>>>>>>> changes to the codebase (the change in replication protocol and
>>>> thus
>>>>>>>>>> transactions, change in binary format, updated metastorage,
>> etc).
>>>> We
>>>>>>>> also
>>>>>>>>>> planned significant changes in public APIs: configuration
>> format
>>>>>>> change,
>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
>>> The
>>>>>>>> wishlist
>>>>>>>>>> of changes for Ignite 3.0 is huge.
>>>>>>>>>>
>>>>>>>>>> So, I was wondering whether it makes sense to try to change the
>>> old
>>>>>>>>>> codebase, or start with a new baseline and move old pieces of
>>> code
>>>>>>> that
>>>>>>>> do
>>>>>>>>>> not require significant rework. Personally, I would go with the
>>>>> second
>>>>>>>>>> option for the following reasons:
>>>>>>>>>>
>>>>>>>>>> - We have a chance to shift the development paradigm in the
>>> project
>>>>>>> and
>>>>>>>>>> introduce the practice of true unit-tests. In the new baseline
>> at
>>>> the
>>>>>>>>>> beginning there will be no ability to run an end-to-end
>> scenario,
>>>>>>> thus
>>>>>>>> we
>>>>>>>>>> will be forced to write unit-tests. So far, such practice was
>>> hard
>>>> to
>>>>>>>>>> implement because of tight coupling between Ignite components
>> and
>>>>>>>> inability
>>>>>>>>>> to instantiate components without an instance of KernalContext.
>>> For
>>>>>>>>>> example, we should be able to thoroughly test internal
>>> primitives,
>>>>>>>> such as
>>>>>>>>>> replication protocol (without actual communication),
>> distributed
>>>>>>>>>> metastorage contracts, persistence layer, etc.
>>>>>>>>>> - We will significantly reduce the development cycle in the
>>>> beginning
>>>>>>>>>> (right now the RunAll takes two hours of astronomical time with
>>>> empty
>>>>>>>> TC;
>>>>>>>>>> in the new approach developer will be able to run ALL tests
>>> locally
>>>>>>> in
>>>>>>>> a
>>>>>>>>>> matter of minutes)
>>>>>>>>>> - We can get rid of TC bot and enforce green TC by integrating
>> TC
>>>>>>> build
>>>>>>>>>> results with GitHub PRs (the same way Travis is currently
>>>> integrated
>>>>>>>> to PR
>>>>>>>>>> check). We should restrict PR merge without a TC check
>>>>>>>>>> - We will still have to re-write all tests, but only once. If
>> we
>>>> try
>>>>>>> to
>>>>>>>>>> modify the old codebase, we would need to modify all the tests
>>> for
>>>>>>>> every
>>>>>>>>>> major change (public API change, configuration change)
>>>>>>>>>> - We will have fewer conflicts when working together. For
>>> example,
>>>> I
>>>>>>>>>> cannot imagine how one would merge two changes of getting rid
>> of
>>>>>>>>>> IgniteFuture and changes in replication protocol, for example
>>>>>>>>>>
>>>>>>>>>> Technically, I would suggest creating a new repository for
>> Ignite
>>>> 3.0
>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
>>>> and a
>>>>>>>> new
>>>>>>>>>> Ignite 3.0 TeamCity project.
>>>>>>>>>>
>>>>>>>>>> While it may seem quite radical, I do believe that this
>> approach
>>>> will
>>>>>>>> give
>>>>>>>>>> us more benefits than trying to make such major changes in the
>>>>>>> existing
>>>>>>>>>> codebase. If needed, let's schedule a discord chat like before
>> to
>>>>>>>> discuss
>>>>>>>>>> this.
>>>>>>>>>>
>>>>>>>>>> WDYT?
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Best regards,
>> Andrey V. Mashenkov
>>
