Re: IEP-61 Technical discussion
Hi Ivan, Unfortunately, the earliest window available for us is 12:00 MSK (1 hour slot), or after 14:30 MSK. Let me know what time works best for you. ср, 25 нояб. 2020 г. в 21:38, Ivan Daschinsky : > Alexey, I kindly ask you to move the meeting a little bit earlier, ideal > variant -- in the morning. > > ср, 25 нояб. 2020 г. в 20:10, Alexey Goncharuk >: > > > Folks, let's have the call on Friday, Nov 27th at 18:00 MSK? We can use > the > > following waiting room link: > > https://zoom.us/j/99450012496?pwd=RWZmOGhCNWlRK0ZpamdOOTZsYTJ0dz09 > > > > Let me know if this time works for everybody. > > > > ср, 25 нояб. 2020 г. в 16:42, Alexey Goncharuk < > alexey.goncha...@gmail.com > > >: > > > > > Folks, > > > > > > I've made some edits in IEP-61 [1] regarding the group membership > service > > > and transaction protocol interaction with the replication > infrastructure, > > > please take a look before our Friday call. > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-61%3A+Common+Replication+Infrastructure > > > > > > пн, 23 нояб. 2020 г. в 13:28, Alexey Goncharuk < > > alexey.goncha...@gmail.com > > > >: > > > > > >> Thanks, Ivan, > > >> > > >> Another protocol for group membership worth checking out is RAPID [1] > (a > > >> recent one). Not sure though if there are any available > implementations > > for > > >> it already. > > >> > > >> [1] > > https://www.usenix.org/system/files/conference/atc18/atc18-suresh.pdf > > >> > > >> пн, 23 нояб. 2020 г. в 10:46, Ivan Daschinsky : > > >> > > >>> Also, here is some interesting reading about gossip, SWIM etc. > > >>> > > >>> 1 -- > > >>> > http://www.cs.cornell.edu/Info/Projects/Spinglass/public_pdfs/SWIM.pdf > > >>> 2 -- > > >>> > > >>> > > > http://www.antonkharenko.com/2015/09/swim-distributed-group-membership.html > > >>> 3 -- https://github.com/hashicorp/memberlist (Foundation library of > > >>> hashicorp serf) > > >>> 4 -- https://github.com/scalecube/scalecube-cluster -- (Java > > >>> implementation > > >>> of SWIM) > > >>> > > >>> чт, 19 нояб. 2020 г. в 16:35, Ivan Daschinsky : > > >>> > > >>> > >> Friday, Nov 27th work for you? If ok, let's have an open call > > then. > > >>> > Yes, great > > >>> > >> As for the protocol port - we will not be dealing with the > > >>> > concurrency... > > >>> > >>Judging by the Rust port, it seems fairly straightforward. > > >>> > Yes, they chose split transport and logic. But original Go package > > from > > >>> > etcd (see raft/node.go) contains some heartbeats mechanism etc. > > >>> > I agree with you, this seems not to be a huge deal to port. > > >>> > > > >>> > чт, 19 нояб. 2020 г. в 16:13, Alexey Goncharuk < > > >>> alexey.goncha...@gmail.com > > >>> > >: > > >>> > > > >>> >> Ivan, > > >>> >> > > >>> >> Agree, let's have a call to discuss the IEP. I have some more > > thoughts > > >>> >> regarding how the replication infrastructure works with > > >>> >> atomic/transactional caches, will put this info to the IEP. Does > > next > > >>> >> Friday, Nov 27th work for you? If ok, let's have an open call > then. > > >>> >> > > >>> >> As for the protocol port - we will not be dealing with the > > concurrency > > >>> >> model if we choose this way, this is what I like about their code > > >>> >> structure. Essentially, the raft module is a single-threaded > > automata > > >>> >> which > > >>> >> has a callback to process a message, process a tick (timeout) and > > >>> produces > > >>> >> messages that should be sent and log entries that should be > > persisted. > > >>> >> Judging by the Rust port, it seems fairly straightforward. Will be > > >>> happy > > >>> >> to > > >>> >> discuss this and other alternatives on the call as well. > > >>> >> > > >>> >> чт, 19 нояб. 2020 г. в 14:41, Ivan Daschinsky < > ivanda...@gmail.com > > >: > > >>> >> > > >>> >> > > Any existing library that can be used to avoid re-implementing > > the > > >>> >> > protocol ourselves? Perhaps, porting the existing implementation > > to > > >>> Java > > >>> >> > Personally, I like this idea. Go libraries (either raft module > of > > >>> etcd > > >>> >> or > > >>> >> > serf by Hashicorp) are famous for clean code, good design, > > >>> stability, > > >>> >> not > > >>> >> > enormous size. > > >>> >> > But, on other side, Go has different model for concurrency and > > >>> porting > > >>> >> > probably will not be so straightforward. > > >>> >> > > > >>> >> > > > >>> >> > > > >>> >> > чт, 19 нояб. 2020 г. в 13:48, Ivan Daschinsky < > > ivanda...@gmail.com > > >>> >: > > >>> >> > > > >>> >> > > I'd suggest to discuss this IEP and technical details in open > > ZOOM > > >>> >> > > meeting. > > >>> >> > > > > >>> >> > > чт, 19 нояб. 2020 г. в 13:47, Ivan Daschinsky < > > >>> ivanda...@gmail.com>: > > >>> >> > > > > >>> >> > >> > > >>> >> > >> > > >>> >> > >> -- Forwarded message - > > >>> >> > >> От: Ivan Daschinsky > > >>> >> > >> Date: чт, 19 нояб. 2020 г. в 13:02 > > >>> >> > >> Subject: Re: IEP-61 Technical
Re: IEP-61 Technical discussion
Alexey, is it possible to manage call at 16:00 MSK? чт, 26 нояб. 2020 г. в 12:30, Alexey Goncharuk : > Hi Ivan, > > Unfortunately, the earliest window available for us is 12:00 MSK (1 hour > slot), or after 14:30 MSK. Let me know what time works best for you. > > ср, 25 нояб. 2020 г. в 21:38, Ivan Daschinsky : > > > Alexey, I kindly ask you to move the meeting a little bit earlier, ideal > > variant -- in the morning. > > > > ср, 25 нояб. 2020 г. в 20:10, Alexey Goncharuk < > alexey.goncha...@gmail.com > > >: > > > > > Folks, let's have the call on Friday, Nov 27th at 18:00 MSK? We can use > > the > > > following waiting room link: > > > https://zoom.us/j/99450012496?pwd=RWZmOGhCNWlRK0ZpamdOOTZsYTJ0dz09 > > > > > > Let me know if this time works for everybody. > > > > > > ср, 25 нояб. 2020 г. в 16:42, Alexey Goncharuk < > > alexey.goncha...@gmail.com > > > >: > > > > > > > Folks, > > > > > > > > I've made some edits in IEP-61 [1] regarding the group membership > > service > > > > and transaction protocol interaction with the replication > > infrastructure, > > > > please take a look before our Friday call. > > > > > > > > [1] > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-61%3A+Common+Replication+Infrastructure > > > > > > > > пн, 23 нояб. 2020 г. в 13:28, Alexey Goncharuk < > > > alexey.goncha...@gmail.com > > > > >: > > > > > > > >> Thanks, Ivan, > > > >> > > > >> Another protocol for group membership worth checking out is RAPID > [1] > > (a > > > >> recent one). Not sure though if there are any available > > implementations > > > for > > > >> it already. > > > >> > > > >> [1] > > > https://www.usenix.org/system/files/conference/atc18/atc18-suresh.pdf > > > >> > > > >> пн, 23 нояб. 2020 г. в 10:46, Ivan Daschinsky >: > > > >> > > > >>> Also, here is some interesting reading about gossip, SWIM etc. > > > >>> > > > >>> 1 -- > > > >>> > > http://www.cs.cornell.edu/Info/Projects/Spinglass/public_pdfs/SWIM.pdf > > > >>> 2 -- > > > >>> > > > >>> > > > > > > http://www.antonkharenko.com/2015/09/swim-distributed-group-membership.html > > > >>> 3 -- https://github.com/hashicorp/memberlist (Foundation library > of > > > >>> hashicorp serf) > > > >>> 4 -- https://github.com/scalecube/scalecube-cluster -- (Java > > > >>> implementation > > > >>> of SWIM) > > > >>> > > > >>> чт, 19 нояб. 2020 г. в 16:35, Ivan Daschinsky >: > > > >>> > > > >>> > >> Friday, Nov 27th work for you? If ok, let's have an open call > > > then. > > > >>> > Yes, great > > > >>> > >> As for the protocol port - we will not be dealing with the > > > >>> > concurrency... > > > >>> > >>Judging by the Rust port, it seems fairly straightforward. > > > >>> > Yes, they chose split transport and logic. But original Go > package > > > from > > > >>> > etcd (see raft/node.go) contains some heartbeats mechanism etc. > > > >>> > I agree with you, this seems not to be a huge deal to port. > > > >>> > > > > >>> > чт, 19 нояб. 2020 г. в 16:13, Alexey Goncharuk < > > > >>> alexey.goncha...@gmail.com > > > >>> > >: > > > >>> > > > > >>> >> Ivan, > > > >>> >> > > > >>> >> Agree, let's have a call to discuss the IEP. I have some more > > > thoughts > > > >>> >> regarding how the replication infrastructure works with > > > >>> >> atomic/transactional caches, will put this info to the IEP. Does > > > next > > > >>> >> Friday, Nov 27th work for you? If ok, let's have an open call > > then. > > > >>> >> > > > >>> >> As for the protocol port - we will not be dealing with the > > > concurrency > > > >>> >> model if we choose this way, this is what I like about their > code > > > >>> >> structure. Essentially, the raft module is a single-threaded > > > automata > > > >>> >> which > > > >>> >> has a callback to process a message, process a tick (timeout) > and > > > >>> produces > > > >>> >> messages that should be sent and log entries that should be > > > persisted. > > > >>> >> Judging by the Rust port, it seems fairly straightforward. Will > be > > > >>> happy > > > >>> >> to > > > >>> >> discuss this and other alternatives on the call as well. > > > >>> >> > > > >>> >> чт, 19 нояб. 2020 г. в 14:41, Ivan Daschinsky < > > ivanda...@gmail.com > > > >: > > > >>> >> > > > >>> >> > > Any existing library that can be used to avoid > re-implementing > > > the > > > >>> >> > protocol ourselves? Perhaps, porting the existing > implementation > > > to > > > >>> Java > > > >>> >> > Personally, I like this idea. Go libraries (either raft module > > of > > > >>> etcd > > > >>> >> or > > > >>> >> > serf by Hashicorp) are famous for clean code, good design, > > > >>> stability, > > > >>> >> not > > > >>> >> > enormous size. > > > >>> >> > But, on other side, Go has different model for concurrency and > > > >>> porting > > > >>> >> > probably will not be so straightforward. > > > >>> >> > > > > >>> >> > > > > >>> >> > > > > >>> >> > чт, 19 нояб. 2020 г. в 13:48, Ivan Daschinsky < > > > ivanda...@gmail.com > > > >>> >: > > > >>> >> > > > > >>> >> > >
Re: IEP-54: Schema-first approach for 3.0
A little bit my thoughts about unsigned types: 1. Seems we may support unsign types 2. It requires adding new types to the internal representation, protocol, e.t.c. 3. internal representation should be the same as we keep sign types. So it will not requires more memory 4. User should be aware of specifics such types for platforms which not support unsigned types. For example, a user could derive -6 value in Java for 250 unsigned byte value (from bits perspective will be right). I think We shouldn't use more wide type for such cases, especially it will be bad for unsigned long when we require returns BigInteger type. 5. Possible it requires some suffix/preffix for new types like a '250u' - it means that 250 is an unsigned value type. 6. It requires a little bit more expensive comparison logic for indexes 7. It requires new comparison logic for expressions. I think it not possible for the current H2 engine and probably possible for the new Calcite engine. Need clarification from anybody who involved in this part WDYT? вт, 24 нояб. 2020 г. в 18:36, Alexey Goncharuk : > Actually, we can support comparisons in 3.0: once we the actual type > information, we can make proper runtime adjustments and conversions to > treat those values as unsigned - it will be just a bit more expensive. > > вт, 24 нояб. 2020 г. в 18:32, Pavel Tupitsyn : > > > > SQL range queries it will break > > > WHERE x > y may return wrong results > > > > Yes, range queries, inequality comparisons and so on are broken > > for unsigned data types, I think I mentioned this somewhere above. > > > > Again, in my opinion, we can document that SQL is not supported on those > > types, > > end of story. > > > > On Tue, Nov 24, 2020 at 6:25 PM Alexey Goncharuk < > > alexey.goncha...@gmail.com> > > wrote: > > > > > Folks, I think this is a reasonable request. I thought about this when > I > > > was drafting the IEP, but hesitated to add these types right away. > > > > > > > That is how it works in Ignite since the beginning with .NET and C++ > :) > > > I have some doubts that it actually works as expected, it needs some > > > checking (will be glad if my concerns are false): > > > > > >- It's true that equality check works properly, but for SQL range > > >queries it will break unless some special care is taken on Java > side: > > > for > > >u8 255 > 10, but in Java (byte)255 will be converted to -1, which > will > > >break the comparison. Since we don't have unsigned types now, I > doubt > > it > > >works. > > >- There is an obvious cross-platform data loss when "intuitive" type > > >mapping is used by a user (u8 corresponds to byte type in .NET, but > to > > >avoid values loss, a user will have to use short type in Java, and > > > Ignite > > >will also need to take care of the range check during > serialization). > > I > > >think we can even allow to try to deserialize a value into arbitrary > > > type, > > >but throw an exception if the range is out of bounds. > > > > > > Overall, I agree with Andrey's comments. > > > Andrey, do you mind updating the IEP once all the details are settled > > here? > > > > > > вт, 24 нояб. 2020 г. в 18:19, Andrey Mashenkov < > > andrey.mashen...@gmail.com > > > >: > > > > > > > Pavel, > > > > > > > > I believe uLong values beyond 2^63 can't be treated correctly for now > > > > (WHERE x > y may return wrong results) > > > > > > > > I think we could make "true" support for unsigned types, but they > will > > > have > > > > limitations on the Java side. > > > > Thus, the one will not be able to map uint64 to Java long primitive, > > but > > > to > > > > BigInteger only. > > > > As for indices, we could read uint64 to Java long, but treat negative > > > > values in a different way to preserve correct ordering. > > > > > > > > These limitations will affect only mixed environments when .Net and > > Java > > > > used to access the data. > > > > Will this solution address your issues? > > > > > > > > > > > > On Tue, Nov 24, 2020 at 5:45 PM Pavel Tupitsyn > > > > > wrote: > > > > > > > > > > That way is impossible. > > > > > > > > > > That is how it works in Ignite since the beginning with .NET and > C++ > > :) > > > > > You can use unsigned primitives as cache keys and values, as fields > > and > > > > > properties, > > > > > and in SQL queries (even in WHERE x=y clauses) - it works > > transparently > > > > for > > > > > the users. > > > > > Java side knows nothing and treats those values as corresponding > > signed > > > > > types. > > > > > > > > > > However, this abstraction leaks in some cases only because there > are > > no > > > > > corresponding type ids. > > > > > That is why I'm proposing a very simple change to the protocol - > add > > > type > > > > > ids, but handle them the same way as signed counterparts. > > > > > > > > > > > > > > > On Tue, Nov 24, 2020 at 5:00 PM Andrey Mashenkov < > > > > > andrey.mashen...@gmail.com> > > > > > wrote: > > > > > > > > > > > Pavel,
Re: [DISCUSS] Use GridNioServer in Java thin client
PR is ready for review [1] I've added a simple put/get benchmark, there is some performance improvement over existing implementation, see results in the PR description. [1] https://github.com/apache/ignite/pull/8483 On Fri, Nov 20, 2020 at 10:39 AM Pavel Tupitsyn wrote: > Since there are no objections, I've updated the IEP accordingly [1] > and started working on it [2] > > [1] > https://cwiki.apache.org/confluence/display/IGNITE/IEP-60%3A+Java+Thin+Client+Non-Blocking+Async+IO > [2] https://github.com/apache/ignite/pull/8483 > > On Mon, Nov 9, 2020 at 4:07 PM Ivan Daschinsky > wrote: > >> I suppose that the best variant -- ability to switch to netty if this lib >> is in classpath >> >> пн, 9 нояб. 2020 г. в 15:58, Igor Sapego : >> >> > Sounds like a good idea to me. >> > >> > Best Regards, >> > Igor >> > >> > >> > On Mon, Nov 9, 2020 at 3:32 PM Alex Plehanov >> > wrote: >> > >> > > +1 for using GridNioServer as java thin client communication layer. >> > > >> > > вс, 8 нояб. 2020 г. в 19:12, Pavel Tupitsyn : >> > > >> > > > Igniters, >> > > > >> > > > This is a continuation of "Use Netty for Java thin client" [1], >> > > > I'm starting a new thread for better visibility. >> > > > >> > > > The problems with current Java thin client are: >> > > > * Socket writes block user threads >> > > > * Every connection uses a separate listener thread (with partition >> > > > awareness there is a thread for every server node within a single >> > > > IgniteClient) >> > > > >> > > > GridNioServer can work in client mode and solves both of these >> > problems. >> > > > It is the most practical choice as well at the moment - no extra >> > > > dependencies required. >> > > > >> > > > A potential drawback is increased coupling between thin client and >> core >> > > > code, >> > > > which I'm going to mitigate by abstracting GridNioServer behind a >> > simpler >> > > > facade, >> > > > so we can replace it with Netty or something else easier if we >> decide >> > to >> > > > split the code. >> > > > >> > > > Thoughts, objections? >> > > > >> > > > [1] >> > > > >> > > > >> > > >> > >> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Use-Netty-for-Java-thin-client-td49732.html >> > > > >> > > >> > >> >> >> -- >> Sincerely yours, Ivan Daschinskiy >> >
Re: [DISCUSS] Use GridNioServer in Java thin client
Pavel, good job and great benchmark results! чт, 26 нояб. 2020 г. в 15:01, Pavel Tupitsyn : > PR is ready for review [1] > > I've added a simple put/get benchmark, there is some performance > improvement over existing implementation, see results in the PR > description. > > [1] https://github.com/apache/ignite/pull/8483 > > On Fri, Nov 20, 2020 at 10:39 AM Pavel Tupitsyn > wrote: > > > Since there are no objections, I've updated the IEP accordingly [1] > > and started working on it [2] > > > > [1] > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-60%3A+Java+Thin+Client+Non-Blocking+Async+IO > > [2] https://github.com/apache/ignite/pull/8483 > > > > On Mon, Nov 9, 2020 at 4:07 PM Ivan Daschinsky > > wrote: > > > >> I suppose that the best variant -- ability to switch to netty if this > lib > >> is in classpath > >> > >> пн, 9 нояб. 2020 г. в 15:58, Igor Sapego : > >> > >> > Sounds like a good idea to me. > >> > > >> > Best Regards, > >> > Igor > >> > > >> > > >> > On Mon, Nov 9, 2020 at 3:32 PM Alex Plehanov > > >> > wrote: > >> > > >> > > +1 for using GridNioServer as java thin client communication layer. > >> > > > >> > > вс, 8 нояб. 2020 г. в 19:12, Pavel Tupitsyn : > >> > > > >> > > > Igniters, > >> > > > > >> > > > This is a continuation of "Use Netty for Java thin client" [1], > >> > > > I'm starting a new thread for better visibility. > >> > > > > >> > > > The problems with current Java thin client are: > >> > > > * Socket writes block user threads > >> > > > * Every connection uses a separate listener thread (with partition > >> > > > awareness there is a thread for every server node within a single > >> > > > IgniteClient) > >> > > > > >> > > > GridNioServer can work in client mode and solves both of these > >> > problems. > >> > > > It is the most practical choice as well at the moment - no extra > >> > > > dependencies required. > >> > > > > >> > > > A potential drawback is increased coupling between thin client and > >> core > >> > > > code, > >> > > > which I'm going to mitigate by abstracting GridNioServer behind a > >> > simpler > >> > > > facade, > >> > > > so we can replace it with Netty or something else easier if we > >> decide > >> > to > >> > > > split the code. > >> > > > > >> > > > Thoughts, objections? > >> > > > > >> > > > [1] > >> > > > > >> > > > > >> > > > >> > > >> > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Use-Netty-for-Java-thin-client-td49732.html > >> > > > > >> > > > >> > > >> > >> > >> -- > >> Sincerely yours, Ivan Daschinskiy > >> > > > -- Sincerely yours, Ivan Daschinskiy
Re: IEP-61 Technical discussion
Done чт, 26 нояб. 2020 г. в 13:18, Ivan Daschinsky : > Alexey, is it possible to manage call at 16:00 MSK? > > чт, 26 нояб. 2020 г. в 12:30, Alexey Goncharuk >: > > > Hi Ivan, > > > > Unfortunately, the earliest window available for us is 12:00 MSK (1 hour > > slot), or after 14:30 MSK. Let me know what time works best for you. > > > > ср, 25 нояб. 2020 г. в 21:38, Ivan Daschinsky : > > > > > Alexey, I kindly ask you to move the meeting a little bit earlier, > ideal > > > variant -- in the morning. > > > > > > ср, 25 нояб. 2020 г. в 20:10, Alexey Goncharuk < > > alexey.goncha...@gmail.com > > > >: > > > > > > > Folks, let's have the call on Friday, Nov 27th at 18:00 MSK? We can > use > > > the > > > > following waiting room link: > > > > https://zoom.us/j/99450012496?pwd=RWZmOGhCNWlRK0ZpamdOOTZsYTJ0dz09 > > > > > > > > Let me know if this time works for everybody. > > > > > > > > ср, 25 нояб. 2020 г. в 16:42, Alexey Goncharuk < > > > alexey.goncha...@gmail.com > > > > >: > > > > > > > > > Folks, > > > > > > > > > > I've made some edits in IEP-61 [1] regarding the group membership > > > service > > > > > and transaction protocol interaction with the replication > > > infrastructure, > > > > > please take a look before our Friday call. > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-61%3A+Common+Replication+Infrastructure > > > > > > > > > > пн, 23 нояб. 2020 г. в 13:28, Alexey Goncharuk < > > > > alexey.goncha...@gmail.com > > > > > >: > > > > > > > > > >> Thanks, Ivan, > > > > >> > > > > >> Another protocol for group membership worth checking out is RAPID > > [1] > > > (a > > > > >> recent one). Not sure though if there are any available > > > implementations > > > > for > > > > >> it already. > > > > >> > > > > >> [1] > > > > > https://www.usenix.org/system/files/conference/atc18/atc18-suresh.pdf > > > > >> > > > > >> пн, 23 нояб. 2020 г. в 10:46, Ivan Daschinsky < > ivanda...@gmail.com > > >: > > > > >> > > > > >>> Also, here is some interesting reading about gossip, SWIM etc. > > > > >>> > > > > >>> 1 -- > > > > >>> > > > http://www.cs.cornell.edu/Info/Projects/Spinglass/public_pdfs/SWIM.pdf > > > > >>> 2 -- > > > > >>> > > > > >>> > > > > > > > > > > http://www.antonkharenko.com/2015/09/swim-distributed-group-membership.html > > > > >>> 3 -- https://github.com/hashicorp/memberlist (Foundation library > > of > > > > >>> hashicorp serf) > > > > >>> 4 -- https://github.com/scalecube/scalecube-cluster -- (Java > > > > >>> implementation > > > > >>> of SWIM) > > > > >>> > > > > >>> чт, 19 нояб. 2020 г. в 16:35, Ivan Daschinsky < > ivanda...@gmail.com > > >: > > > > >>> > > > > >>> > >> Friday, Nov 27th work for you? If ok, let's have an open > call > > > > then. > > > > >>> > Yes, great > > > > >>> > >> As for the protocol port - we will not be dealing with the > > > > >>> > concurrency... > > > > >>> > >>Judging by the Rust port, it seems fairly straightforward. > > > > >>> > Yes, they chose split transport and logic. But original Go > > package > > > > from > > > > >>> > etcd (see raft/node.go) contains some heartbeats mechanism > etc. > > > > >>> > I agree with you, this seems not to be a huge deal to port. > > > > >>> > > > > > >>> > чт, 19 нояб. 2020 г. в 16:13, Alexey Goncharuk < > > > > >>> alexey.goncha...@gmail.com > > > > >>> > >: > > > > >>> > > > > > >>> >> Ivan, > > > > >>> >> > > > > >>> >> Agree, let's have a call to discuss the IEP. I have some more > > > > thoughts > > > > >>> >> regarding how the replication infrastructure works with > > > > >>> >> atomic/transactional caches, will put this info to the IEP. > Does > > > > next > > > > >>> >> Friday, Nov 27th work for you? If ok, let's have an open call > > > then. > > > > >>> >> > > > > >>> >> As for the protocol port - we will not be dealing with the > > > > concurrency > > > > >>> >> model if we choose this way, this is what I like about their > > code > > > > >>> >> structure. Essentially, the raft module is a single-threaded > > > > automata > > > > >>> >> which > > > > >>> >> has a callback to process a message, process a tick (timeout) > > and > > > > >>> produces > > > > >>> >> messages that should be sent and log entries that should be > > > > persisted. > > > > >>> >> Judging by the Rust port, it seems fairly straightforward. > Will > > be > > > > >>> happy > > > > >>> >> to > > > > >>> >> discuss this and other alternatives on the call as well. > > > > >>> >> > > > > >>> >> чт, 19 нояб. 2020 г. в 14:41, Ivan Daschinsky < > > > ivanda...@gmail.com > > > > >: > > > > >>> >> > > > > >>> >> > > Any existing library that can be used to avoid > > re-implementing > > > > the > > > > >>> >> > protocol ourselves? Perhaps, porting the existing > > implementation > > > > to > > > > >>> Java > > > > >>> >> > Personally, I like this idea. Go libraries (either raft > module > > > of > > > > >>> etcd > > > > >>> >> or > > > > >>> >> > serf by Hashicorp) are famous for clean code,
[jira] [Created] (IGNITE-13766) Add API for checking network connectivity between all nodes in a cluster
Semyon Danilov created IGNITE-13766: --- Summary: Add API for checking network connectivity between all nodes in a cluster Key: IGNITE-13766 URL: https://issues.apache.org/jira/browse/IGNITE-13766 Project: Ignite Issue Type: Task Components: control.sh Affects Versions: 2.9 Reporter: Semyon Danilov Assignee: Semyon Danilov Fix For: 2.9.1 Lack of connectivity from Server to client nodes could be a huge problem for cluster stability. The client will be able to connect to the server and the server will be able to answer to the client as long as the connection established by the client will live, but if it will be closed and the server will try to establish it's own connection, it will just hang(along with an operation, which requires a message to be sent to the client node, it could be topology change or some snapshot stage change, etc). We should introduce an API, which will be available from control.sh and will allow checking the connectivity between all nodes in the cluster. NOTE that nodes should always try to establish it's own connection instead of just reusing the connection that was established by another node. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: 2.9.1 release scope and dates
Yaroslav, It seems we've reached the proposed voting date - Nov. 26th Do you need any help with the release preparation? On Fri, 20 Nov 2020 at 14:46, Steshin Vladimir wrote: > > Yaroslav, Hi. > > > I suggest to merge minor fix of javadoc: [1]. It should have appeared in > 2.9. Commits in master: > > d3e5b7c11ed037670700eea75851e619d5d1b6b1 > > and > > 1654e9fac61842424c08d26a08ef67569f74746a > > > [1] https://github.com/apache/ignite/pull/8448 > > > > 19.11.2020 17:15, Ivan Daschinsky пишет: > > Hi! > > Yaroslav, Max -- I have another ticket that will be nice to have in 2.9.1 > > https://issues.apache.org/jira/browse/IGNITE-13699 > > > > пт, 13 нояб. 2020 г. в 15:08, Yaroslav Molochkov : > > > >> Igniters, hello! > >> > >> I think the scope of 2.9.1 is finalized. > >> > >>> On 9 Nov 2020, at 12:04, Yaroslav Molochkov > >> wrote: > >>> Ivan, thanks! > >>> > >>> Added it to the list. > >>> > On 8 Nov 2020, at 14:13, Ivan Daschinsky wrote: > > Yaroslav, there is another bug for 2.9.1 release > https://issues.apache.org/jira/browse/IGNITE-13572 > > чт, 5 нояб. 2020 г., 19:23 Yaroslav Molochkov : > > > Ivan, hi! > > Sure. > > > > UPD: i am the release manager and will be doing this with Maxim's help > > (since i don't have some user permissions) > > > >> On Thu, Nov 5, 2020 at 6:24 PM Ivan Daschinsky > >> wrote: > >> > >> Hi. I'd suggest to add this issue. This is a usability improvement > >> for zk > >> discovery, and also this patch incorporates fixes for JMX metrics > >> concurrency issues > >> > >> [1] -- https://issues.apache.org/jira/browse/IGNITE-13577 > >> > >> чт, 5 нояб. 2020 г., 16:20 Yaroslav Molochkov >>> : > >>> Igniters! > >>> > >>> I'd like to help with the 2.9.1 release. The scope of this release > >> includes > >>> following issues: > >>> > >>> > >> https://issues.apache.org/jira/browse/IGNITE-13676?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.9.1 > >>> Maxim Muzafarov agreed to help me with the process and he will be the > >>> release manager. > >>> > >>> Scope freeze: Nov. 12th > >>> Code freeze: Nov. 19th > >>> Voting date: Nov. 26th > >>> Release date: Nov. 31st > >>> > >>> Tickets that were added (or to be added) to the scope don't bring new > >>> features but various bug fixes. > >>> > >
[GitHub] [ignite-3] AMashenkov opened a new pull request #2: IGNITE-13748: Schema configuration public API.
AMashenkov opened a new pull request #2: URL: https://github.com/apache/ignite-3/pull/2 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: 2.9.1 release scope and dates
Maxim, Yeah, I guess I did all I could with my access rights, all there’s to do is to merge my PR in the branch and publish necessary artifacts And, of course, sorry that I’m a bit behind the schedule > On 26 Nov 2020, at 21:14, Maxim Muzafarov wrote: > > Yaroslav, > > > It seems we've reached the proposed voting date - Nov. 26th > Do you need any help with the release preparation? > >> On Fri, 20 Nov 2020 at 14:46, Steshin Vladimir wrote: >> >> Yaroslav, Hi. >> >> >> I suggest to merge minor fix of javadoc: [1]. It should have appeared in >> 2.9. Commits in master: >> >> d3e5b7c11ed037670700eea75851e619d5d1b6b1 >> >> and >> >> 1654e9fac61842424c08d26a08ef67569f74746a >> >> >> [1] https://github.com/apache/ignite/pull/8448 >> >> >> >> 19.11.2020 17:15, Ivan Daschinsky пишет: >>> Hi! >>> Yaroslav, Max -- I have another ticket that will be nice to have in 2.9.1 >>> https://issues.apache.org/jira/browse/IGNITE-13699 >>> >>> пт, 13 нояб. 2020 г. в 15:08, Yaroslav Molochkov : >>> Igniters, hello! I think the scope of 2.9.1 is finalized. > On 9 Nov 2020, at 12:04, Yaroslav Molochkov wrote: > Ivan, thanks! > > Added it to the list. > >> On 8 Nov 2020, at 14:13, Ivan Daschinsky wrote: >> >> Yaroslav, there is another bug for 2.9.1 release >> https://issues.apache.org/jira/browse/IGNITE-13572 >> >> чт, 5 нояб. 2020 г., 19:23 Yaroslav Molochkov : >> >>> Ivan, hi! >>> Sure. >>> >>> UPD: i am the release manager and will be doing this with Maxim's help >>> (since i don't have some user permissions) >>> On Thu, Nov 5, 2020 at 6:24 PM Ivan Daschinsky wrote: Hi. I'd suggest to add this issue. This is a usability improvement for zk discovery, and also this patch incorporates fixes for JMX metrics concurrency issues [1] -- https://issues.apache.org/jira/browse/IGNITE-13577 чт, 5 нояб. 2020 г., 16:20 Yaroslav Molochkov : > Igniters! > > I'd like to help with the 2.9.1 release. The scope of this release includes > following issues: > > https://issues.apache.org/jira/browse/IGNITE-13676?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.9.1 > Maxim Muzafarov agreed to help me with the process and he will be the > release manager. > > Scope freeze: Nov. 12th > Code freeze: Nov. 19th > Voting date: Nov. 26th > Release date: Nov. 31st > > Tickets that were added (or to be added) to the scope don't bring new > features but various bug fixes. > >>>
Re: [DISCUSS] Missed (non-suited) tests
Hi, Ilya! So, I've updated PR, fixed comments and removed Core* prefixes. MTCGA shows no blockers, but it was 2 weeks ago, so I've started it again. If PR is OK then there are some suites that should be updated on TC. Could you please tell me how we can do it? 1. Add ignite-cassandra-serializers suite: 1. org.apache.ignite.tests.SerializerSuite 2. Add ignite-core to Queries* TC suite: 1. org.apache.ignite.client.IgniteClientTestSuite 2. org.apache.ignite.suites.IgniteBinaryCacheQueryTestSuite 3. org.apache.ignite.suites.IgniteBinaryCacheQueryTestSuite2 4. org.apache.ignite.suites.IgniteCacheQuerySelfTestSuite3 5. org.apache.ignite.suites.IgniteCacheQuerySelfTestSuite4 6. org.apache.ignite.suites.IgniteCacheQuerySelfTestSuite5 7. org.apache.ignite.suites.IgniteCacheQuerySelfTestSuite6 8. org.apache.ignite.suites.IgnitePdsWithIndexingCoreTestSuite 9. org.apache.ignite.suites.IgniteCacheMvccSqlTestSuite 3. Remove ignite-indexing from TC suites: 1. org.apache.ignite.testsuites.IgniteCacheQuerySelfTestSuite3 2. org.apache.ignite.testsuites.IgniteCacheQuerySelfTestSuite4 3. org.apache.ignite.testsuites.IgniteCacheQuerySelfTestSuite5 4. Add ignite-core to Spring* TC suite: 1. org.apache.ignite.testsuites.IgniteSpringTestSuite 5. Add ignite-core suite (depends on uri-deployment module): 1. org.apache.ignite.testsuites.IgniteUriDeploymentTestSuite 6. Add ignite-core suite to Zookeeper TC suite: 1. org.apache.ignite.testsuites.ZookeeperDiscoverySpiTestSuite3 7. Remove ignite-zookeeper test suite: 1. org.apache.ignite.testsuites.ZookeeperDiscoverySpiTestSuite3 8. Add ignite-ml test suites: 1. org.apache.ignite.ml.math.distances.DistancesTestSuite 2. org.apache.ignite.ml.naivebayes.NaiveBayesTestSuite On Wed, Nov 25, 2020 at 4:26 PM Ilya Kasnacheev wrote: > Hello! > > Yes, we have such tests which depend on ignite-indexing or ignite-spring. > They just need to be included in Spring* or Queries* test suite. Then they > will be executed on TC in the correct context. You can also run these tests > from IDEA by specifying other module as classpath. No need to move the > classes around. > > I will check the PR. > > Regards, > -- > Ilya Kasnacheev > > > ср, 25 нояб. 2020 г. в 00:22, Max Timonin : > > > Ilya, Anton, Ivan, hi! > > > > I fix some comments you leave in the PR. Also I checked some test suites > > and found that some tests are written in the core module but depend on > the > > indexing module (or other modules). Some of such test classes contain > tests > > that are related to the core functionality, but some to indexing. I'm not > > sure if it is correct to move a whole suite with all tests from the > > indexing module to the core, as it will hide some core tests from the > core > > module. > > > > I believe that the correct solution is to investigate every such test and > > move it to the right module. But I think this work will take a lot of > time > > and should be performed in a separate ticket, I will do it in the > > background. > > > > I think currently we should proceed with a way I introduced in PR: > > 1. Create fake suites for all such tests (written in core, suited in > other > > modules: indexing/spring/zookeeper/etc) in the core module. I named such > > suites with prefix Core*. > > 2. Replace tests in modules with links to fake suites. > > 3. Create an umbrella Jira ticket to discover every fake suite and > replace > > it with a new one in the right module. > > 4. Merge this PR for introducing a new travis check to avoid losing > > new tests. > > > > WDYT? > > > > List of such mixed suites: > > > > 1. suite IgniteBinaryCacheQueryTestSuite > > > > test GridCacheQueryIndexingDisabledSelfTest > > test IgniteCacheBinaryObjectsScanSelfTest > > test IgniteCacheBinaryObjectsScanWithEventsSelfTest) > > > > > > 2. suite IgniteCacheQuerySelfTestSuite3 > > > > test GridCacheContinuousQueryNodesFilteringTest > > > > > > 3. suite IgniteCacheQuerySelfTestSuite5 > > > > test ContinuousQueryRemoteFilterMissingInClassPathSelfTest > > > > 4. suite IgniteCacheQuerySelfTestSuite6 > > > > test CacheContinuousQueryOperationP2PTest > > > > test CacheContinuousQueryFilterDeploymentFailedTest > > > > > > 5. all tests in suite IgnitePdsWithIndexingCoreTestSuite > > > > > > 6. and some others. > > > > On Wed, Nov 18, 2020 at 12:38 PM Max Timonin > > wrote: > > > > > Hi Ilya! Thank you for up the topic. I will come back with fixes and > > > comments in a couple of days. > > > > > > On Tue, Nov 17, 2020 at 4:26 PM Ilya Kasnacheev < > > ilya.kasnach...@gmail.com> > > > wrote: > > > > > >> Hello! > > >> > > >> I have left some comments and there's also more discussion there. Can > > you > > >> please look? > > >> > > >> Thanks, > > >> -- > > >> Ilya Kasnacheev > > >> > > >> > > >> вт, 3 нояб. 2020 г. в 00:03, Max Timonin : > > >> > > >> > Hi! > > >> > > > >> > I've updated PR: https://github.com/apache/ignite/pull/8367. Anton, > > >> Ivan, > > >> >
