Guys,
I've tried to build release candidate using TC [1]. But:
1. I can't choose a branch in this TC task (there is no such field).
2. On the "default" branch I got an error: Failed to collect changes,
error: List remote refs failed:
org.apache.http.conn.ConnectTimeoutException: Connect to
gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed: connect
timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
internal id=74, parent id=GitBoxAsfIgnite, description: "
https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
Is there some problem with this TC task? What am I doing wrong?
[1] :
https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <alexey.goncha...@gmail.com>:
> Saikat, Nikolay,
>
> We have migrated a bunch of modules to ignite-extensions recently. Given
> that these modules will not be available in Ignite 2.9 anymore (will
> they?), should we also plan to release the extensions?
>
> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <plehanov.a...@gmail.com>:
>
> > Igniters,
> >
> > I've prepared release notes for the 2.9 release [1]. Can anyone review
> it,
> > please?
> >
> > [1]: https://github.com/apache/ignite/pull/8273
> >
> > вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <plehanov.a...@gmail.com>:
> >
> > > Guys,
> > >
> > > I've filled the ticket with reproducer [1] for the discovery bug. This
> > bug
> > > caused by [2] ticket. We discussed with Vladimir Steshin privately and
> > > decided to revert this ticket. I will do it today (after TC bot visa)
> if
> > > there are no objections.
> > >
> > > [1]: https://issues.apache.org/jira/browse/IGNITE-13465
> > > [2]: https://issues.apache.org/jira/browse/IGNITE-13134
> > >
> > > пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <plehanov.a...@gmail.com>:
> > >
> > >> Guys,
> > >>
> > >> During internal testing, we've found a critical bug with
> > >> discovery (cluster falls apart if two nodes segmented sequentially).
> > This
> > >> problem is not reproduced in 2.8.1. I think we should fix it
> > >> before release. Under investigation now. I'll let you know when we get
> > >> something.
> > >>
> > >> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <ag...@apache.org>:
> > >>
> > >>> > So what do you think? Should we proceed with a 'hacked' version of
> > the
> > >>> message factory in 2.9 and go for the runtime message generation in
> > later
> > >>> release, or keep the code clean and fix the regression in the next
> > releases?
> > >>> > Andrey, can you take a look at my change? I think it is fairly
> > >>> straightforward and does not change the semantics, just skips the
> > factory
> > >>> closures for certain messages.
> > >>>
> > >>> IMHO 2.5% isn't too much especially because it isn't actual for all
> > >>> workloads (I didn't get any significant drops during benchmarking).
> So
> > >>> I prefer the runtime generation in later releases.
> > >>>
> > >>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
> > >>> <alexey.goncha...@gmail.com> wrote:
> > >>> >
> > >>> > Alexey, Andrey, Igniters,
> > >>> >
> > >>> > So what do you think? Should we proceed with a 'hacked' version of
> > the
> > >>> message factory in 2.9 and go for the runtime message generation in
> > later
> > >>> release, or keep the code clean and fix the regression in the next
> > releases?
> > >>> > Andrey, can you take a look at my change? I think it is fairly
> > >>> straightforward and does not change the semantics, just skips the
> > factory
> > >>> closures for certain messages.
> > >>> >
> > >>> > Personally, I would prefer fixing the regression given that we also
> > >>> introduced tracing in this release.
> > >>> >
> > >>> >
> > >>> >
> > >>> > пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
> plehanov.a...@gmail.com
> > >:
> > >>> >>
> > >>> >> Alexey,
> > >>> >>
> > >>> >> We've benchmarked by yardstick commits 130376741bf vs ed52559eb95
> > and
> > >>> the performance of ed52559eb95 is better for about 2.5% on average on
> > our
> > >>> environment (about the same results we got benchmarking 65c30ec6 vs
> > >>> 0606f03d). We've made 24 runs for each commit of
> > >>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
> > >>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers, 5
> > clients
> > >>> 50 threads each. Yardstick results for this configuration:
> > >>> >> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
> > >>> >> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
> > >>> >>
> > >>> >> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> > >>> a.budnikov.ign...@gmail.com>:
> > >>> >>>
> > >>> >>> Hi Everyone,
> > >>> >>>
> > >>> >>> I posted an instruction on how to publish the docs on
> > >>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you can
> > >>> update the docs by following the instruction. Unfortunately, I won't
> be
> > >>> able to spend any time on this project any longer. You can send your
> > pull
> > >>> requests and questions about the documentation to Denis Magda.
> > >>> >>>
> > >>> >>> -Artem
> > >>> >>>
> > >>> >>> [1] :
> > >>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> > >>> >>>
> > >>> >>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> > >>> alexey.goncha...@gmail.com> wrote:
> > >>> >>>>
> > >>> >>>> Alexey,
> > >>> >>>>
> > >>> >>>> I've tried to play with message factories locally, but
> > >>> unfortunately, I
> > >>> >>>> cannot spot the difference between old and new implementation in
> > >>> >>>> distributed benchmarks. I pushed an implementation of
> > >>> MessageFactoryImpl
> > >>> >>>> with the old switch statement to the ignite-2.9-revert-12568
> > branch
> > >>> >>>> (discussed this with Andrey Gura, the change should be
> compatible
> > >>> with the
> > >>> >>>> new metrics as we still use the register() mechanics).
> > >>> >>>>
> > >>> >>>> Can you check if this change makes any difference
> performance-wise
> > >>> in your
> > >>> >>>> environment? If yes, we can go with runtime code generation in
> the
> > >>> long
> > >>> >>>> term: register classes and generate a dynamic message factory
> with
> > >>> a switch
> > >>> >>>> statement once all messages are registered (not in 2.9 though,
> > >>> obviously).
> > >>> >>>>
> > >>> >>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
> > plehanov.a...@gmail.com
> > >>> >:
> > >>> >>>>
> > >>> >>>> > Hello guys,
> > >>> >>>> >
> > >>> >>>> > I've tried to optimize tracing implementation (ticket [1]), it
> > >>> reduced the
> > >>> >>>> > drop, but not completely removed it.
> > >>> >>>> > Ivan Rakov, Alexander Lapin, can you please review the patch?
> > >>> >>>> > Ivan Artiukhov, can you please benchmark the patch [2] against
> > >>> 2.8.1
> > >>> >>>> > release on your environment?
> > >>> >>>> > With this patch on our environment, it's about a 3% drop left,
> > >>> it's close
> > >>> >>>> > to measurement error and I think such a drop is not a
> > >>> showstopper. Guys,
> > >>> >>>> > WDYT?
> > >>> >>>> >
> > >>> >>>> > Also, I found that compatibility is broken for JDBC thin
> driver
> > >>> between 2.8
> > >>> >>>> > and 2.9 versions (ticket [3]). I think it's a blocker and
> should
> > >>> be
> > >>> >>>> > fixed in 2.9. I've prepared the patch.
> > >>> >>>> > Taras Ledkov, can you please review this patch?
> > >>> >>>> >
> > >>> >>>> > And one more ticket I propose to include into 2.9 [4] (NIO
> > message
> > >>> >>>> > send problem in some circumstances). I will cherry-pick it if
> > >>> there is no
> > >>> >>>> > objection.
> > >>> >>>> >
> > >>> >>>> > [1] https://issues.apache.org/jira/browse/IGNITE-13411
> > >>> >>>> > [2] https://github.com/apache/ignite/pull/8223
> > >>> >>>> > [3] https://issues.apache.org/jira/browse/IGNITE-13414
> > >>> >>>> > [4] https://issues.apache.org/jira/browse/IGNITE-13361
> > >>> >>>> >
> > >>> >>>> > пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
> mmu...@apache.org
> > >:
> > >>> >>>> >
> > >>> >>>> > > Alexey,
> > >>> >>>> > >
> > >>> >>>> > > I propose to include [1] issue to the 2.9 release. Since
> this
> > >>> issue is
> > >>> >>>> > > related to the new master key change functionality which
> > >>> haven't been
> > >>> >>>> > > released yet I think it will be safe to cherry-pick commit
> to
> > >>> the
> > >>> >>>> > > release branch.
> > >>> >>>> > >
> > >>> >>>> > > [1] https://issues.apache.org/jira/browse/IGNITE-13390
> > >>> >>>> > >
> > >>> >>>> > > On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
> > >>> nizhi...@apache.org>
> > >>> >>>> > wrote:
> > >>> >>>> > > >
> > >>> >>>> > > > Hello, Igniters.
> > >>> >>>> > > >
> > >>> >>>> > > > Alexey, please, include one more Python thin client fix
> [1]
> > >>> into the
> > >>> >>>> > 2.9
> > >>> >>>> > > release
> > >>> >>>> > > > It fixes kinda major issue - "Python client returns fields
> > in
> > >>> wrong
> > >>> >>>> > > order since the 2 row when fields_count>10"
> > >>> >>>> > > >
> > >>> >>>> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12809
> > >>> >>>> > > > [2]
> > >>> >>>> > >
> > >>> >>>> >
> > >>>
> >
> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
> > >>> >>>> > > >
> > >>> >>>> > > > > 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
> > >>> >>>> > alexey.goncha...@gmail.com>
> > >>> >>>> > > написал(а):
> > >>> >>>> > > > >
> > >>> >>>> > > > > Alexey, thanks, got it. I am not sure we can optimize
> > >>> anything out of
> > >>> >>>> > > the
> > >>> >>>> > > > > message factory with suppliers (at least I have no ideas
> > >>> right now),
> > >>> >>>> > so
> > >>> >>>> > > > > most likely the only move here is to switch back to the
> > >>> switch
> > >>> >>>> > approach
> > >>> >>>> > > > > somehow preserving the metrics part. Probably, inlining
> > the
> > >>> Ignite
> > >>> >>>> > > messages
> > >>> >>>> > > > > to the IgniteMessageFactoryImpl should do the trick. Let
> > me
> > >>> explore
> > >>> >>>> > the
> > >>> >>>> > > > > code a bit.
> > >>> >>>> > > > >
> > >>> >>>> > > > > P.S. I am surprised by the impact this part makes for
> the
> > >>> >>>> > performance.
> > >>> >>>> > > > > Message creation is indeed on the hot path, but a single
> > >>> virtual call
> > >>> >>>> > > > > should not make that much of a difference given the
> amount
> > >>> of other
> > >>> >>>> > > work
> > >>> >>>> > > > > happening during the message processing.
> > >>> >>>> > > > >
> > >>> >>>> > > > > пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
> > >>> plehanov.a...@gmail.com
> > >>> >>>> > >:
> > >>> >>>> > > > >
> > >>> >>>> > > > >> Alexey, sorry, I wrongly interpreted our benchmark
> > results.
> > >>> >>>> > Actually,
> > >>> >>>> > > we
> > >>> >>>> > > > >> were looking for a drop using bi-sect in the range
> > between
> > >>> e6a7f93
> > >>> >>>> > > (first
> > >>> >>>> > > > >> commit in the 2.9 branch after 2.8 branch cut) and
> > >>> 6592dfa5 (last
> > >>> >>>> > > commit in
> > >>> >>>> > > > >> the 2.9 branch). And we found these two problematic
> > >>> commits.
> > >>> >>>> > > > >>
> > >>> >>>> > > > >> Perhaps only IGNITE-13060 (Tracing) is responsible for
> a
> > >>> drop
> > >>> >>>> > between
> > >>> >>>> > > > >> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
> > >>> reverted
> > >>> >>>> > > IGNITE-13060
> > >>> >>>> > > > >> now and performance looks the same)
> > >>> >>>> > > > >>
> > >>> >>>> > > > >> Ticket IGNITE-12568 (MessageFactory refactoring) is not
> > >>> related to
> > >>> >>>> > > drop
> > >>> >>>> > > > >> between 2.8.1 and 2.9, but still has some performance
> > >>> problem, and
> > >>> >>>> > we
> > >>> >>>> > > can
> > >>> >>>> > > > >> win back IGNITE-13060 drop by this ticket.
> > >>> >>>> > > > >>
> > >>> >>>> > > > >> Do we need more investigation on IGNITE-13060 or we
> leave
> > >>> it as is?
> > >>> >>>> > > > >>
> > >>> >>>> > > > >> What should we do with IGNITE-12568 (MessageFactory
> > >>> refactoring)?
> > >>> >>>> > > > >>
> > >>> >>>> > > > >> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
> > >>> >>>> > > alexey.goncha...@gmail.com
> > >>> >>>> > > > >>> :
> > >>> >>>> > > > >>
> > >>> >>>> > > > >>> Alexey,
> > >>> >>>> > > > >>>
> > >>> >>>> > > > >>> While investigating, I found that IGNITE-12568 has an
> > >>> incorrect fix
> > >>> >>>> > > > >>> version and is actually present in ignite-2.8.1 branch
> > >>> [1], so it
> > >>> >>>> > > cannot be
> > >>> >>>> > > > >>> the source of the drop against 2.8.1.
> > >>> >>>> > > > >>>
> > >>> >>>> > > > >>> P.S. Looks like we need to enforce a more accurate
> work
> > >>> with fix
> > >>> >>>> > > versions
> > >>> >>>> > > > >>> or develop some sort of tooling to verify the fix
> > >>> versions.
> > >>> >>>> > > > >>>
> > >>> >>>> > > > >>> --AG
> > >>> >>>> > > > >>>
> > >>> >>>> > > > >>> [1]
> > >>> >>>> > > > >>>
> > >>> >>>> > >
> > >>> >>>> >
> > >>>
> >
> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
> > >>> >>>> > > > >>>
> > >>> >>>> > > > >>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
> > >>> >>>> > > alexey.goncha...@gmail.com
> > >>> >>>> > > > >>>> :
> > >>> >>>> > > > >>>
> > >>> >>>> > > > >>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
> > >>> >>>> > plehanov.a...@gmail.com
> > >>> >>>> > > >:
> > >>> >>>> > > > >>>>
> > >>> >>>> > > > >>>>> Guys,
> > >>> >>>> > > > >>>>>
> > >>> >>>> > > > >>>>> We have benchmarked 2.9 without IGNITE-13060 and
> > >>> IGNITE-12568
> > >>> >>>> > > (reverted
> > >>> >>>> > > > >>>>> it
> > >>> >>>> > > > >>>>> locally) and got the same performance as on 2.8.1
> > >>> >>>> > > > >>>>>
> > >>> >>>> > > > >>>>> IGNITE-13060 (Tracing) - some code was added to hot
> > >>> paths, to
> > >>> >>>> > trace
> > >>> >>>> > > > >>>>> these
> > >>> >>>> > > > >>>>> hot paths, it's clear why we have performance drop
> > here.
> > >>> >>>> > > > >>>>>
> > >>> >>>> > > > >>>>> IGNITE-12568 (MessageFactory refactoring) -
> > switch/case
> > >>> block was
> > >>> >>>> > > > >>>>> refactored to an array of message suppliers. The
> > >>> message factory
> > >>> >>>> > > is on
> > >>> >>>> > > > >>>>> the
> > >>> >>>> > > > >>>>> hot path, which explains why this commit has an
> impact
> > >>> on total
> > >>> >>>> > > > >>>>> performance.
> > >>> >>>> > > > >>>>> I've checked JIT assembly output, done some JMH
> > >>> microbenchmarks,
> > >>> >>>> > > and
> > >>> >>>> > > > >>>>> found
> > >>> >>>> > > > >>>>> that old implementation of MessageFactory.create()
> > >>> about 30-35%
> > >>> >>>> > > faster
> > >>> >>>> > > > >>>>> than
> > >>> >>>> > > > >>>>> the new one. The reason - approach with switch/case
> > can
> > >>> >>>> > effectively
> > >>> >>>> > > > >>>>> inline
> > >>> >>>> > > > >>>>> message creation code, but with an array of
> suppliers
> > >>> relatively
> > >>> >>>> > > heavy
> > >>> >>>> > > > >>>>> "invokeinterface" cannot be skipped. I've tried to
> > >>> rewrite the
> > >>> >>>> > code
> > >>> >>>> > > > >>>>> using
> > >>> >>>> > > > >>>>> an abstract class for suppliers instead of an
> > interface
> > >>> (to
> > >>> >>>> > > > >>>>> replace "invokeinterface" with the "invokevirtual"),
> > >>> but it gives
> > >>> >>>> > > back
> > >>> >>>> > > > >>>>> only
> > >>> >>>> > > > >>>>> 10% of method performance and in this case, code
> looks
> > >>> ugly
> > >>> >>>> > > (lambdas
> > >>> >>>> > > > >>>>> can't
> > >>> >>>> > > > >>>>> be used). Currently, I can't find any more ways to
> > >>> optimize the
> > >>> >>>> > > current
> > >>> >>>> > > > >>>>> approach (except return to the switch/case block).
> > >>> Andrey Gura,
> > >>> >>>> > as
> > >>> >>>> > > the
> > >>> >>>> > > > >>>>> author of IGNITE-12568, maybe you have some ideas
> > about
> > >>> >>>> > > optimization?
> > >>> >>>> > > > >>>>>
> > >>> >>>> > > > >>>>> Perhaps we should revert IGNITE-12568, but there are
> > >>> some metrics
> > >>> >>>> > > > >>>>> already
> > >>> >>>> > > > >>>>> created, which can't be rewritten using old message
> > >>> factory
> > >>> >>>> > > > >>>>> implementation
> > >>> >>>> > > > >>>>> (IGNITE-12756). Guys, WDYT?
> > >>> >>>> > > > >>>>>
> > >>> >>>> > > > >>>>
> > >>> >>>> > > > >>>> Alexey,
> > >>> >>>> > > > >>>>
> > >>> >>>> > > > >>>> I see that IGNITE-12756 (metrics improvements) is
> > >>> already released
> > >>> >>>> > > in
> > >>> >>>> > > > >>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
> > >>> only present
> > >>> >>>> > > in Ignite
> > >>> >>>> > > > >>>> 2.9. Let's revert both IGNITE-12568 and whichever new
> > >>> metrics
> > >>> >>>> > > created for
> > >>> >>>> > > > >>>> 2.9 that depend on the new message factory to unblock
> > >>> the release
> > >>> >>>> > > and deal
> > >>> >>>> > > > >>>> with the optimizations in 2.10?
> > >>> >>>> > > > >>>>
> > >>> >>>> > > > >>>
> > >>> >>>> > > >
> > >>> >>>> > >
> > >>> >>>> >
> > >>>
> > >>
> >
>
