Hello, Maxim. Thanks for the update.
Please, let me know when PR will be ready. I will do the review. В Вт, 02/04/2019 в 14:52 +0300, Maxim Muzafarov пишет: > Ivan, > > I tend to agree with you. Let's keep changes as an internal Apache Ignite API. > > Igniters, > > I need your support by reviewing the issue [1] with my changes of > CommunicationSpi about establishing direct socket connections between > cluster nodes. I've prepared the draft PR [2] with such changes. I'll > add more JUnit tests to this PR and I'll provide all the > implementation details. > Can anyone help me with the review? > > The issue [1] is related to my work over IEP-28 of rebalancing cache > data by sending cache partition files (the persistence must be > enabled). You can find the rebalancing process details on the IEP-28 > page [3]. In general, the whole rebalance by partition files process > can be divided on five components: > - establishing socket connections ~ 3000 lines (CommunicationSpi) > - create consistent cache partition file ~ 6000 lines (Checkpointer) > - downloading, uploading partition files ~ 4000 lines > (CachePartitionUploader, CachePartitionDownloader) > - handling cache puts during the partition file download ~ 3000 lines > (CachupWALManager) > - rebuilding indexes ~ 2000 lines > > All these changes together (single PR) are big and complex, so better > to review them step by step. And I'd like to find the support of the > community and ask the review of changes in CommunicationSpi module of > the Apache Ignite first. > > [1] https://issues.apache.org/jira/browse/IGNITE-10619 > [2] https://github.com/apache/ignite/pull/5619 > [3] > https://cwiki.apache.org/confluence/display/IGNITE/IEP-28%3A+Cluster+peer-2-peer+balancing > > On Tue, 5 Mar 2019 at 17:52, Павлухин Иван <vololo...@gmail.com> wrote: > > > > Maxim, > > > > My humble opinion. If there is no convenient means to implement > > partition file sending today then we should introduce something. And > > keeping such facility private is much easier, because introduction of > > new public API is a significantly more complex task. > > > > пт, 1 мар. 2019 г. в 19:44, Maxim Muzafarov <maxmu...@gmail.com>: > > > > > > Igniters, > > > > > > Apache Ignite has a very suitable messaging user interface [1] for > > > topic-based communication between nodes (or a specific group of nodes > > > within a cluster). The messaging functionality in Ignite is provided > > > via IgniteMessaging interface. It allows: > > > - send a message to a certain topic > > > - register local\remote listeners > > > > > > I really like this feature, but the disadvantage here is when the user > > > wants to transfer a large amount of binary data (e.g. files) between > > > nodes he must create a complex logic to wrap it into messages. I think > > > Ignite could have an interface e.g. IgniteChannels which will allow: > > > - register local\remote listeners for channel created\destroy events. > > > - create a channel connection (a wrapped socket channel) to a certain > > > node\group of nodes and the desired topic > > > > > > As another suitable case where such a feature can be applied is > > > internal usage for Apache Ignite needs. I can mention here the task of > > > cluster rebalancing by sending cache partition files between nodes. > > > I've posted a small description of it on the IEP-28 page [2]. > > > > > > > > > WDYT about it? > > > > > > --- > > > > > > API (assumed) > > > > > > IgniteChannels chnls = ignite0.channels(); > > > chnls.remoteListen(TOPIC.MY_TOPIC, new RemoteListener()); > > > > > > IgniteSocketChannel ch0 = chnls.channel(node, TOPIC.MY_TOPIC); > > > ch0.writeInt(bigFile.size()); > > > ch0.transferTo(FileChannel.open(bigFile.path(), StandardOpenOption.READ)) > > > > > > > > > /** */ > > > > > > private class RemoteListener > > > implements IgniteBiPredicate<UUID, IgniteSocketChannel> { > > > > > > @IgniteInstanceResource > > > private Ignite ignite; > > > > > > @Override public boolean apply( > > > UUID nodeId, > > > IgniteSocketChannel ch > > > ) { > > > int size = ch.readInt(); > > > ignite.fileSystem("base") > > > .create("bigfile.mpg") > > > .transferFrom(ch, size); > > > return true; > > > } > > > } > > > > > > > > > [1] https://apacheignite.readme.io/docs/messaging > > > [2] > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-28%3A+Cluster+peer-2-peer+balancing#IEP-28:Clusterpeer-2-peerbalancing-CommunicationSpi > > > > > > > > -- > > Best regards, > > Ivan Pavlukhin
signature.asc
Description: This is a digitally signed message part
