Re: Thin client: compute support
Hello, Igniters. I think we should support full compute API for the thin clients(at least for the java clients) I think our roadmap should be the following: 1. Execution existing compute tasks by id (as Alex suggested). 2. Deploy and execution arbitrary compute tasks from the java thin client 3. Research possibility to create API for deployment and execution compute tasks in any language that implemented thin client protocol.(python as a first candidate). > 22 нояб. 2019 г., в 10:19, Alex Plehanov написал(а): > > Denis, the primary motivation is to enable execution of deployed to server > java tasks from thin clients (java and other languages). > > пт, 22 нояб. 2019 г. в 00:03, Denis Magda : > >> Alex, what is the primary motivation for this improvement? Are you looking >> to enable the compute APIs for languages different from Java? >> >> - >> Denis >> >> >> On Wed, Nov 20, 2019 at 11:58 PM Alex Plehanov >> wrote: >> >>> Hello, Igniters! >>> >>> I have plans to start implementation of Compute interface for Ignite thin >>> client and want to discuss features that should be implemented. >>> >>> We already have Compute implementation for binary-rest clients >>> (GridClientCompute), which have the following functionality: >>> - Filtering cluster nodes (projection) for compute >>> - Executing task by the name >>> >>> I think we can implement this functionality in a thin client as well. >>> >>> First of all, we need some operation types to request a list of all >>> available nodes and probably node attributes (by a list of nodes). Node >>> attributes will be helpful if we will decide to implement analog of >>> ClusterGroup#forAttribute or ClusterGroup#forePredicate methods in the >> thin >>> client. Perhaps they can be requested lazily. >>> >>> From the protocol point of view there will be two new operations: >>> >>> OP_CLUSTER_GET_NODES >>> Request: empty >>> Response: long topologyVersion, int minorTopologyVersion, int nodesCount, >>> for each node set of node fields (UUID nodeId, Object or String >>> consistentId, long order, etc) >>> >>> OP_CLUSTER_GET_NODE_ATTRIBUTES >>> Request: int nodesCount, for each node: UUID nodeId >>> Response: int nodesCount, for each node: int attributesCount, for each >> node >>> attribute: String name, Object value >>> >>> To execute tasks we need something like these methods in the client API: >>> Object execute(String task, Object arg) >>> Future executeAsync(String task, Object arg) >>> Object affinityExecute(String task, String cache, Object key, Object arg) >>> Future affinityExecuteAsync(String task, String cache, Object >> key, >>> Object arg) >>> >>> Which can be mapped to protocol operations: >>> >>> OP_COMPUTE_EXECUTE_TASK >>> Request: UUID nodeId, String taskName, Object arg >>> Response: Object result >>> >>> OP_COMPUTE_EXECUTE_TASK_AFFINITY >>> Request: String cacheName, Object key, String taskName, Object arg >>> Response: Object result >>> >>> The second operation is needed because we sometimes can't calculate and >>> connect to affinity node on the client-side (affinity awareness can be >>> disabled, custom affinity function can be used or there can be no >>> connection between client and affinity node), but we can make best effort >>> to send request to target node if affinity awareness is enabled. >>> >>> Currently, on the server-side requests always processed synchronously and >>> responses are sent right after request was processed. To execute long >> tasks >>> async we should whether change this logic or introduce some kind two-way >>> communication between client and server (now only one-way requests from >>> client to server are allowed). >>> >>> Two-way communication can also be useful in the future if we will send >> some >>> server-side generated events to clients. >>> >>> In case of two-way communication there can be new operations introduced: >>> >>> OP_COMPUTE_EXECUTE_TASK (from client to server) >>> Request: UUID nodeId, String taskName, Object arg >>> Response: long taskId >>> >>> OP_COMPUTE_TASK_FINISHED (from server to client) >>> Request: taskId, Object result >>> Response: empty >>> >>> The same for affinity requests. >>> >>> Also, we can implement not only execute task operation, but some other >>> operations from IgniteCompute (broadcast, run, call), but it will be >> useful >>> only for java thin client. And even with java thin client we should >> whether >>> implement peer-class-loading for thin clients (this also requires two-way >>> client-server communication) or put classes with executed closures to the >>> server locally. >>> >>> What do you think about proposed protocol changes? >>> Do we need two-way requests between client and server? >>> Do we need support of compute methods other than "execute task"? >>> What do you think about peer-class-loading for thin clients? >>> >>
Re: Thin client: compute support
So, guys: Cluster API (get nodes and get node attributes) will be implemented in scope of IGNITE-12385 ticket by Aleksandr. For compute API as a first step only the ability to execute the task by task name operation will be implemented, which will require: - Two-way requests (from clients to server and from server to clients) - Four new operation types: OP_COMPUTE_EXECUTE_TASK (client-server), OP_COMPUTE_EXECUTE_TASK_AFFINITY (client-server), OP_COMPUTE_CANCEL_TASK (client-server), OP_COMPUTE_TASK_FINISHED (server-client) - Option in node configuration to enable compute for thin clients (disabled by default) I think we should create IEP to describe protocol changes and new operations format. I will create it. Aleksandr, can you provide some details about your implementation? What types of new operations will be introduced? Can you describe the format of new operations? I will add this information to IEP. As a next step we can implement class deployment from thin clients (will work only for java) or some kind of domain specific language. It will be decided later do we really need this. WDYT? пт, 22 нояб. 2019 г. в 10:19, Alex Plehanov : > Denis, the primary motivation is to enable execution of deployed to server > java tasks from thin clients (java and other languages). > > пт, 22 нояб. 2019 г. в 00:03, Denis Magda : > >> Alex, what is the primary motivation for this improvement? Are you looking >> to enable the compute APIs for languages different from Java? >> >> - >> Denis >> >> >> On Wed, Nov 20, 2019 at 11:58 PM Alex Plehanov >> wrote: >> >> > Hello, Igniters! >> > >> > I have plans to start implementation of Compute interface for Ignite >> thin >> > client and want to discuss features that should be implemented. >> > >> > We already have Compute implementation for binary-rest clients >> > (GridClientCompute), which have the following functionality: >> > - Filtering cluster nodes (projection) for compute >> > - Executing task by the name >> > >> > I think we can implement this functionality in a thin client as well. >> > >> > First of all, we need some operation types to request a list of all >> > available nodes and probably node attributes (by a list of nodes). Node >> > attributes will be helpful if we will decide to implement analog of >> > ClusterGroup#forAttribute or ClusterGroup#forePredicate methods in the >> thin >> > client. Perhaps they can be requested lazily. >> > >> > From the protocol point of view there will be two new operations: >> > >> > OP_CLUSTER_GET_NODES >> > Request: empty >> > Response: long topologyVersion, int minorTopologyVersion, int >> nodesCount, >> > for each node set of node fields (UUID nodeId, Object or String >> > consistentId, long order, etc) >> > >> > OP_CLUSTER_GET_NODE_ATTRIBUTES >> > Request: int nodesCount, for each node: UUID nodeId >> > Response: int nodesCount, for each node: int attributesCount, for each >> node >> > attribute: String name, Object value >> > >> > To execute tasks we need something like these methods in the client API: >> > Object execute(String task, Object arg) >> > Future executeAsync(String task, Object arg) >> > Object affinityExecute(String task, String cache, Object key, Object >> arg) >> > Future affinityExecuteAsync(String task, String cache, Object >> key, >> > Object arg) >> > >> > Which can be mapped to protocol operations: >> > >> > OP_COMPUTE_EXECUTE_TASK >> > Request: UUID nodeId, String taskName, Object arg >> > Response: Object result >> > >> > OP_COMPUTE_EXECUTE_TASK_AFFINITY >> > Request: String cacheName, Object key, String taskName, Object arg >> > Response: Object result >> > >> > The second operation is needed because we sometimes can't calculate and >> > connect to affinity node on the client-side (affinity awareness can be >> > disabled, custom affinity function can be used or there can be no >> > connection between client and affinity node), but we can make best >> effort >> > to send request to target node if affinity awareness is enabled. >> > >> > Currently, on the server-side requests always processed synchronously >> and >> > responses are sent right after request was processed. To execute long >> tasks >> > async we should whether change this logic or introduce some kind two-way >> > communication between client and server (now only one-way requests from >> > client to server are allowed). >> > >> > Two-way communication can also be useful in the future if we will send >> some >> > server-side generated events to clients. >> > >> > In case of two-way communication there can be new operations introduced: >> > >> > OP_COMPUTE_EXECUTE_TASK (from client to server) >> > Request: UUID nodeId, String taskName, Object arg >> > Response: long taskId >> > >> > OP_COMPUTE_TASK_FINISHED (from server to client) >> > Request: taskId, Object result >> > Response: empty >> > >> > The same for affinity requests. >> > >> > Also, we can implement not only execute task operation, but some other >> > operations from Igni
Re: [RESULT] Ignite PMC Chair Vote
Hi Igniters, Thanks to everyone for all of the congratulations! I’m very honored to serve in PMC Chair role. Sincerely, Dmitriy Pavlov пт, 22 нояб. 2019 г. в 09:00, Ivan Pavlukhin : > Hooray! > > пт, 22 нояб. 2019 г. в 05:13, Saikat Maitra : > > > > Congrats, Dmitry! > > > > Regards, > > Saikat > > > > On Thu, Nov 21, 2019 at 3:05 PM Denis Magda wrote: > > > > > Igniters, > > > > > > The board has officially approved and appointed Dmitry for the role of > > > Ignite PMC Chair. > > > > > > Congrats, Dmitry! Well deserved! Let's move on, there are many things > for > > > all of us to do together ;) > > > > > > - > > > Denis > > > > > > > > > On Thu, Nov 7, 2019 at 1:39 PM Dmitriy Pavlov > wrote: > > > > > > > Hi Folks, > > > > > > > > Thank you, it would be a great honor for me to serve in this role. > And I > > > > really surprised by this community decision (other PMC members have a > > > huge > > > > contribution to the project in different aspects and they > > > > all deserve recognition). > > > > > > > > But it is a little bit early for congrats since the ASF Board > appoints > > > new > > > > Chair. It is an advisory decision of project PMC who will be the next > > > > Chair. > > > > > > > > AFAIK next Board meeting is scheduled for November 20. Let's wait for > > > their > > > > decision first. > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > > > чт, 7 нояб. 2019 г. в 13:44, Alexey Zinoviev >: > > > > > > > > > Dmitry, congrats, we will wait the Community roadmap from you and > will > > > > be > > > > > happy to start discussion about that In asf slack or on dev-list! > > > > > > > > > > чт, 7 нояб. 2019 г., 12:27 Petr Ivanov : > > > > > > > > > > > Dmitriy, congratulations! > > > > > > > > > > > > > > > > > > > On 7 Nov 2019, at 12:06, Николай Кулагин < > zzzadruga@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > Dmitry, my congratulations! > > > > > > > > > > > > > > This is a clear sign that you are moving in the right > direction. > > > > > > > Good luck! > > > > > > > > > > > > > > чт, 7 нояб. 2019 г. в 11:12, Вячеслав Коптилин < > > > > > slava.kopti...@gmail.com > > > > > > >: > > > > > > > > > > > > > >> Hello Igniters, Dmitriy, > > > > > > >> > > > > > > >> Congrats, Dmitriy! > > > > > > >> I hope that a board's approval is just a formal step :) > > > > > > >> > > > > > > >> Thanks, > > > > > > >> S. > > > > > > >> > > > > > > >> чт, 7 нояб. 2019 г. в 10:52, Ivan Pavlukhin < > vololo...@gmail.com > > > >: > > > > > > >> > > > > > > >>> Great news! > > > > > > >>> > > > > > > >>> Dmitriy my congratulations! I really believe that it is the > > > proper > > > > > > >>> decision today and will make a great positive impact on > Ignite > > > > > > >>> Community. > > > > > > >>> > > > > > > >>> Hooray! > > > > > > >>> > > > > > > >>> ср, 6 нояб. 2019 г. в 22:28, Denis Magda >: > > > > > > > > > > > > Igniters, > > > > > > > > > > > > As early head-ups for you. Ignite PMCs came to a consensus > and > > > > > > selected > > > > > > Dmitry Pavlov as the next Chair. Next, we need to get a > board's > > > > > > >> approval. > > > > > > Our request will be reviewed during the next board meeting > that > > > > > takes > > > > > > >>> place > > > > > > this month: https://www.apache.org/dev/pmc.html#newchair > > > > > > > > > > > > - > > > > > > Denis > > > > > > > > > > > > > > > > > > On Fri, Nov 1, 2019 at 12:32 PM Denis Magda < > dma...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > Ignite community, the votes were spread this way. > > > > > > > > > > > > > > Binding (votes of Ignite PMC members): > > > > > > > • Alexey Goncharuk - 3 > > > > > > > • Dmitry Pavlov - 2 > > > > > > > • Nikolay Izhikov - 3 > > > > > > > • Pavel Tupitsyn - 1 > > > > > > > > > > > > > > Non-binding (votes of other community members): > > > > > > > • Alexey Goncharuk - 2 > > > > > > > • Dmitry Pavlov - 11 > > > > > > > • Nikolay Izhikov - 1 > > > > > > > • Pavel Tupitsyn - 0 > > > > > > > > > > > > > > As long as only the binding votes can be counted towards > the > > > > > > > election results, we have two candidates who got equal > votes - > > > > > Alexey > > > > > > >>> and > > > > > > > Nikolay. Please give Ignite PMC some time to figure out > how to > > > > > > >> proceed > > > > > > > further. > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Denis > > > > > > > > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> -- > > > > > > >>> Best regards, > > > > > > >>> Ivan Pavlukhin > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Best regards, > Ivan Pavlukhin >
Re: Thin client: compute support
Hello! When doing this, I think we should also consider Service invocations to make their API/implementation as close as possible. Regards, -- Ilya Kasnacheev пт, 22 нояб. 2019 г. в 11:17, Alex Plehanov : > So, guys: > > Cluster API (get nodes and get node attributes) will be implemented in > scope of IGNITE-12385 ticket by Aleksandr. > > For compute API as a first step only the ability to execute the task by > task name operation will be implemented, which will require: > - Two-way requests (from clients to server and from server to clients) > - Four new operation types: OP_COMPUTE_EXECUTE_TASK (client-server), > OP_COMPUTE_EXECUTE_TASK_AFFINITY (client-server), OP_COMPUTE_CANCEL_TASK > (client-server), OP_COMPUTE_TASK_FINISHED (server-client) > - Option in node configuration to enable compute for thin clients (disabled > by default) > > I think we should create IEP to describe protocol changes and new > operations format. I will create it. > > Aleksandr, can you provide some details about your implementation? What > types of new operations will be introduced? Can you describe the format of > new operations? I will add this information to IEP. > > As a next step we can implement class deployment from thin clients (will > work only for java) or some kind of domain specific language. It will be > decided later do we really need this. > > WDYT? > > пт, 22 нояб. 2019 г. в 10:19, Alex Plehanov : > > > Denis, the primary motivation is to enable execution of deployed to > server > > java tasks from thin clients (java and other languages). > > > > пт, 22 нояб. 2019 г. в 00:03, Denis Magda : > > > >> Alex, what is the primary motivation for this improvement? Are you > looking > >> to enable the compute APIs for languages different from Java? > >> > >> - > >> Denis > >> > >> > >> On Wed, Nov 20, 2019 at 11:58 PM Alex Plehanov > > >> wrote: > >> > >> > Hello, Igniters! > >> > > >> > I have plans to start implementation of Compute interface for Ignite > >> thin > >> > client and want to discuss features that should be implemented. > >> > > >> > We already have Compute implementation for binary-rest clients > >> > (GridClientCompute), which have the following functionality: > >> > - Filtering cluster nodes (projection) for compute > >> > - Executing task by the name > >> > > >> > I think we can implement this functionality in a thin client as well. > >> > > >> > First of all, we need some operation types to request a list of all > >> > available nodes and probably node attributes (by a list of nodes). > Node > >> > attributes will be helpful if we will decide to implement analog of > >> > ClusterGroup#forAttribute or ClusterGroup#forePredicate methods in the > >> thin > >> > client. Perhaps they can be requested lazily. > >> > > >> > From the protocol point of view there will be two new operations: > >> > > >> > OP_CLUSTER_GET_NODES > >> > Request: empty > >> > Response: long topologyVersion, int minorTopologyVersion, int > >> nodesCount, > >> > for each node set of node fields (UUID nodeId, Object or String > >> > consistentId, long order, etc) > >> > > >> > OP_CLUSTER_GET_NODE_ATTRIBUTES > >> > Request: int nodesCount, for each node: UUID nodeId > >> > Response: int nodesCount, for each node: int attributesCount, for each > >> node > >> > attribute: String name, Object value > >> > > >> > To execute tasks we need something like these methods in the client > API: > >> > Object execute(String task, Object arg) > >> > Future executeAsync(String task, Object arg) > >> > Object affinityExecute(String task, String cache, Object key, Object > >> arg) > >> > Future affinityExecuteAsync(String task, String cache, Object > >> key, > >> > Object arg) > >> > > >> > Which can be mapped to protocol operations: > >> > > >> > OP_COMPUTE_EXECUTE_TASK > >> > Request: UUID nodeId, String taskName, Object arg > >> > Response: Object result > >> > > >> > OP_COMPUTE_EXECUTE_TASK_AFFINITY > >> > Request: String cacheName, Object key, String taskName, Object arg > >> > Response: Object result > >> > > >> > The second operation is needed because we sometimes can't calculate > and > >> > connect to affinity node on the client-side (affinity awareness can be > >> > disabled, custom affinity function can be used or there can be no > >> > connection between client and affinity node), but we can make best > >> effort > >> > to send request to target node if affinity awareness is enabled. > >> > > >> > Currently, on the server-side requests always processed synchronously > >> and > >> > responses are sent right after request was processed. To execute long > >> tasks > >> > async we should whether change this logic or introduce some kind > two-way > >> > communication between client and server (now only one-way requests > from > >> > client to server are allowed). > >> > > >> > Two-way communication can also be useful in the future if we will send > >> some > >> > server-side generated events to clients. > >> > > >> > In case of two-way com
Re: Thin client: compute support
Hi On Thu, Nov 21, 2019 at 12:48 PM Denis Garus wrote: > Hello, Sergey! > > > >> 3. Ignite doesn't have roles/authorization functionality for now. > > > I can't agree with you. > My bad, you're right. But the IEP page provided by you is limited and there's no clear how it works for thin clients > We already have authorization functionality in Ignite and for a thin > client too [1]. > > But, compute support for a thin client requires some additional efforts to > get an appropriate SecurityContext on a remote node. > > The list of tasks allowed for subjects, including thin clients, is the > area of responsibility of GridSecurityProcessor [2]. > > > >1. > > org.apache.ignite.internal.processors.security.client.ThinClientPermissionCheckTest >2. org.apache.ignite.internal.processors.security.GridSecurityProcessor > > > чт, 21 нояб. 2019 г. в 12:41, Pavel Tupitsyn : > >> Good points, Sergey. >> Maybe you are right, and Java-based compute without peer deployment is a >> good first step for thin clients. >> >> On Thu, Nov 21, 2019 at 12:32 PM Sergey Kozlov >> wrote: >> >> > Hi Pavel >> > >> > On Thu, Nov 21, 2019 at 11:30 AM Pavel Tupitsyn >> > wrote: >> > >> > > 1. I believe that Cluster operations for Thin Client protocol are >> already >> > > in the works >> > > by Alexandr Shapkin. Can't find the ticket though. >> > > Alexandr, can you please confirm and attach the ticket number? >> > > >> > > 2. Proposed changes will work only for Java tasks that are already >> > deployed >> > > on server nodes. >> > > This is mostly useless for other thin clients we have (Python, PHP, >> .NET, >> > > C++). >> > > >> > >> > I don't guess so. The task (execution) is a way to implement own layer >> for >> > the thin client application. >> > >> > >> > > We should think of a way to make this useful for all clients. >> > > For example, we may allow sending tasks in some scripting language >> like >> > > Javascript. >> > > Thoughts? >> > > >> > >> > The arbitrary code execution from a remote client must be protected >> > from malicious code. >> > I don't know how it could be designed but without that we open the hole >> to >> > kill cluster. >> > >> > >> > > >> > > On Thu, Nov 21, 2019 at 11:21 AM Sergey Kozlov >> > > wrote: >> > > >> > > > Hi Alex >> > > > >> > > > The idea is great. But I have some concerns that probably should be >> > taken >> > > > into account for design: >> > > > >> > > >1. We need to have the ability to stop a task execution, smth >> like >> > > >OP_COMPUTE_CANCEL_TASK operation (client to server) >> > > >2. What's about task execution timeout? It may help to the >> cluster >> > > >survival for buggy tasks >> > > >3. Ignite doesn't have roles/authorization functionality for now. >> > But >> > > a >> > > >task is the risky operation for cluster (for security reasons). >> > Could >> > > we >> > > >add for Ignite configuration new options: >> > > > - Explicit turning on for compute task support for thin >> protocol >> > > > (disabled by default) for whole cluster >> > > > - Explicit turning on for compute task support for a node >> > > > - The list of task names (classes) allowed to execute by thin >> > > client. >> > > >4. Support the labeling for task that may help to investigate >> issues >> > > on >> > > >cluster (the idea from IEP-34 [1]) >> > > > >> > > > 1. >> > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-34+Thin+client%3A+transactions+support >> > > > >> > > > >> > > > >> > > > On Thu, Nov 21, 2019 at 10:58 AM Alex Plehanov < >> > plehanov.a...@gmail.com> >> > > > wrote: >> > > > >> > > > > Hello, Igniters! >> > > > > >> > > > > I have plans to start implementation of Compute interface for >> Ignite >> > > thin >> > > > > client and want to discuss features that should be implemented. >> > > > > >> > > > > We already have Compute implementation for binary-rest clients >> > > > > (GridClientCompute), which have the following functionality: >> > > > > - Filtering cluster nodes (projection) for compute >> > > > > - Executing task by the name >> > > > > >> > > > > I think we can implement this functionality in a thin client as >> well. >> > > > > >> > > > > First of all, we need some operation types to request a list of >> all >> > > > > available nodes and probably node attributes (by a list of nodes). >> > Node >> > > > > attributes will be helpful if we will decide to implement analog >> of >> > > > > ClusterGroup#forAttribute or ClusterGroup#forePredicate methods in >> > the >> > > > thin >> > > > > client. Perhaps they can be requested lazily. >> > > > > >> > > > > From the protocol point of view there will be two new operations: >> > > > > >> > > > > OP_CLUSTER_GET_NODES >> > > > > Request: empty >> > > > > Response: long topologyVersion, int minorTopologyVersion, int >> > > nodesCount, >> > > > > for each node set of node fields (UUID nodeId, Object or String >> > > > > consistentI
[jira] [Created] (IGNITE-12388) The testReconnectServersRestart_3 test is flacky
Amelchev Nikita created IGNITE-12388:
Summary: The testReconnectServersRestart_3 test is flacky
Key: IGNITE-12388
URL: https://issues.apache.org/jira/browse/IGNITE-12388
Project: Ignite
Issue Type: Test
Reporter: Amelchev Nikita
Assignee: Amelchev Nikita
Fix For: 2.8
The testReconnectServersRestart_3 test is flacky. [TC
history.|https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=534893108987486091&tab=testDetails]
Sometimes it fails by timeout. The reason is the incorrect client mode
condition during nodes starts:
{noformat}
helper.clientModeThreadLocal(threadIdx == srvIdx ||
ThreadLocalRandom.current().nextBoolean())
{noformat}
There is no server node to connect when all started nodes are clients:
{noformat}
[2019-11-22 15:42:24,998][WARN ][start-node-5][ZookeeperDiscoveryImpl] Waiting
for local join event [nodeId=74a33ea3-6d7c-4ab4-b86c-3d9305c4,
name=internal.ZookeeperDiscoveryClientReconnectTest4]
[2019-11-22 15:42:25,010][WARN ][start-node-1][ZookeeperDiscoveryImpl] Waiting
for local join event [nodeId=b113bb09-5d6a-48b8-a656-d22cb920,
name=internal.ZookeeperDiscoveryClientReconnectTest0]
[2019-11-22 15:42:25,015][WARN ][start-node-4][ZookeeperDiscoveryImpl] Waiting
for local join event [nodeId=121ca03d-31de-4de4-a81c-20bd1373,
name=internal.ZookeeperDiscoveryClientReconnectTest3]
[2019-11-22 15:42:25,017][WARN ][start-node-2][ZookeeperDiscoveryImpl] Waiting
for local join event [nodeId=4a6b5c7f-fc06-4251-b78f-c49277f1,
name=internal.ZookeeperDiscoveryClientReconnectTest1]
[2019-11-22 15:42:25,020][WARN ][start-node-3][ZookeeperDiscoveryImpl] Waiting
for local join event [nodeId=b0468b08-7d57-4b29-b2fb-6d142592,
name=internal.ZookeeperDiscoveryClientReconnectTest2]
{noformat}
Then the test fails by timeout (5min).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Thin client: compute support
> Two-way requests (from clients to server and from server to clients) Alex, can you please clarify the need for server->client requests? > compute tasks in any language that implemented thin client protocol (python as a first candidate) Nikolay, do you have anything in mind here? Why Python? On Fri, Nov 22, 2019 at 3:13 PM Sergey Kozlov wrote: > Hi > > On Thu, Nov 21, 2019 at 12:48 PM Denis Garus wrote: > > > Hello, Sergey! > > > > > > >> 3. Ignite doesn't have roles/authorization functionality for now. > > > > > > I can't agree with you. > > > > My bad, you're right. But the IEP page provided by you is limited and > there's no clear how it works for thin clients > > > > We already have authorization functionality in Ignite and for a thin > > client too [1]. > > > > But, compute support for a thin client requires some additional efforts > to > > get an appropriate SecurityContext on a remote node. > > > > The list of tasks allowed for subjects, including thin clients, is the > > area of responsibility of GridSecurityProcessor [2]. > > > > > > > >1. > > > org.apache.ignite.internal.processors.security.client.ThinClientPermissionCheckTest > >2. > org.apache.ignite.internal.processors.security.GridSecurityProcessor > > > > > > чт, 21 нояб. 2019 г. в 12:41, Pavel Tupitsyn : > > > >> Good points, Sergey. > >> Maybe you are right, and Java-based compute without peer deployment is a > >> good first step for thin clients. > >> > >> On Thu, Nov 21, 2019 at 12:32 PM Sergey Kozlov > >> wrote: > >> > >> > Hi Pavel > >> > > >> > On Thu, Nov 21, 2019 at 11:30 AM Pavel Tupitsyn > > >> > wrote: > >> > > >> > > 1. I believe that Cluster operations for Thin Client protocol are > >> already > >> > > in the works > >> > > by Alexandr Shapkin. Can't find the ticket though. > >> > > Alexandr, can you please confirm and attach the ticket number? > >> > > > >> > > 2. Proposed changes will work only for Java tasks that are already > >> > deployed > >> > > on server nodes. > >> > > This is mostly useless for other thin clients we have (Python, PHP, > >> .NET, > >> > > C++). > >> > > > >> > > >> > I don't guess so. The task (execution) is a way to implement own layer > >> for > >> > the thin client application. > >> > > >> > > >> > > We should think of a way to make this useful for all clients. > >> > > For example, we may allow sending tasks in some scripting language > >> like > >> > > Javascript. > >> > > Thoughts? > >> > > > >> > > >> > The arbitrary code execution from a remote client must be protected > >> > from malicious code. > >> > I don't know how it could be designed but without that we open the > hole > >> to > >> > kill cluster. > >> > > >> > > >> > > > >> > > On Thu, Nov 21, 2019 at 11:21 AM Sergey Kozlov < > skoz...@gridgain.com> > >> > > wrote: > >> > > > >> > > > Hi Alex > >> > > > > >> > > > The idea is great. But I have some concerns that probably should > be > >> > taken > >> > > > into account for design: > >> > > > > >> > > >1. We need to have the ability to stop a task execution, smth > >> like > >> > > >OP_COMPUTE_CANCEL_TASK operation (client to server) > >> > > >2. What's about task execution timeout? It may help to the > >> cluster > >> > > >survival for buggy tasks > >> > > >3. Ignite doesn't have roles/authorization functionality for > now. > >> > But > >> > > a > >> > > >task is the risky operation for cluster (for security reasons). > >> > Could > >> > > we > >> > > >add for Ignite configuration new options: > >> > > > - Explicit turning on for compute task support for thin > >> protocol > >> > > > (disabled by default) for whole cluster > >> > > > - Explicit turning on for compute task support for a node > >> > > > - The list of task names (classes) allowed to execute by > thin > >> > > client. > >> > > >4. Support the labeling for task that may help to investigate > >> issues > >> > > on > >> > > >cluster (the idea from IEP-34 [1]) > >> > > > > >> > > > 1. > >> > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-34+Thin+client%3A+transactions+support > >> > > > > >> > > > > >> > > > > >> > > > On Thu, Nov 21, 2019 at 10:58 AM Alex Plehanov < > >> > plehanov.a...@gmail.com> > >> > > > wrote: > >> > > > > >> > > > > Hello, Igniters! > >> > > > > > >> > > > > I have plans to start implementation of Compute interface for > >> Ignite > >> > > thin > >> > > > > client and want to discuss features that should be implemented. > >> > > > > > >> > > > > We already have Compute implementation for binary-rest clients > >> > > > > (GridClientCompute), which have the following functionality: > >> > > > > - Filtering cluster nodes (projection) for compute > >> > > > > - Executing task by the name > >> > > > > > >> > > > > I think we can implement this functionality in a thin client as > >> well. > >> > > > > > >> > > > > First of all, we need some operation types to request a list of > >> all
Re: Thin client: compute support
Alex,
I think you can create a new IEP page and I will fill it with the Cluster
API details.
In short, I’ve introduced several new codes:
Cluster API is pretty straightforward:
OP_CLUSTER_IS_ACTIVE = 5000
OP_CLUSTER_CHANGE_STATE = 5001
OP_CLUSTER_CHANGE_WAL_STATE = 5002
OP_CLUSTER_GET_WAL_STATE = 5003
Cluster group codes:
OP_CLUSTER_GROUP_GET_NODE_IDS = 5100
OP_CLUSTER_GROUP_GET_NODE_INFO = 5101
The underlying implementation is based on the thick client logic.
For every request, we provide a known topology version and if it has
changed,
a client updates it firstly and then re-sends the filtering request.
Alongside the topVer a client sends a serialized nodes projection object
that could be considered as a code to value mapping.
Consider: [{Code = 1, Value= [“DotNet”, “MyAttribute”}, {Code=2, Value=1}]
Where “1” stands for Attribute filtering and “2” – serverNodesOnly flag.
As a result of request processing, a server sends nodeId UUIDs and a
current topVer.
When a client obtains nodeIds, it can perform a NODE_INFO call to get a
serialized ClusterNode object. In addition there should be a different API
method for accessing/updating node metrics.
чт, 21 нояб. 2019 г. в 12:32, Sergey Kozlov :
> Hi Pavel
>
> On Thu, Nov 21, 2019 at 11:30 AM Pavel Tupitsyn
> wrote:
>
> > 1. I believe that Cluster operations for Thin Client protocol are already
> > in the works
> > by Alexandr Shapkin. Can't find the ticket though.
> > Alexandr, can you please confirm and attach the ticket number?
> >
> > 2. Proposed changes will work only for Java tasks that are already
> deployed
> > on server nodes.
> > This is mostly useless for other thin clients we have (Python, PHP, .NET,
> > C++).
> >
>
> I don't guess so. The task (execution) is a way to implement own layer for
> the thin client application.
>
>
> > We should think of a way to make this useful for all clients.
> > For example, we may allow sending tasks in some scripting language like
> > Javascript.
> > Thoughts?
> >
>
> The arbitrary code execution from a remote client must be protected
> from malicious code.
> I don't know how it could be designed but without that we open the hole to
> kill cluster.
>
>
> >
> > On Thu, Nov 21, 2019 at 11:21 AM Sergey Kozlov
> > wrote:
> >
> > > Hi Alex
> > >
> > > The idea is great. But I have some concerns that probably should be
> taken
> > > into account for design:
> > >
> > >1. We need to have the ability to stop a task execution, smth like
> > >OP_COMPUTE_CANCEL_TASK operation (client to server)
> > >2. What's about task execution timeout? It may help to the cluster
> > >survival for buggy tasks
> > >3. Ignite doesn't have roles/authorization functionality for now.
> But
> > a
> > >task is the risky operation for cluster (for security reasons).
> Could
> > we
> > >add for Ignite configuration new options:
> > > - Explicit turning on for compute task support for thin protocol
> > > (disabled by default) for whole cluster
> > > - Explicit turning on for compute task support for a node
> > > - The list of task names (classes) allowed to execute by thin
> > client.
> > >4. Support the labeling for task that may help to investigate issues
> > on
> > >cluster (the idea from IEP-34 [1])
> > >
> > > 1.
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-34+Thin+client%3A+transactions+support
> > >
> > >
> > >
> > > On Thu, Nov 21, 2019 at 10:58 AM Alex Plehanov <
> plehanov.a...@gmail.com>
> > > wrote:
> > >
> > > > Hello, Igniters!
> > > >
> > > > I have plans to start implementation of Compute interface for Ignite
> > thin
> > > > client and want to discuss features that should be implemented.
> > > >
> > > > We already have Compute implementation for binary-rest clients
> > > > (GridClientCompute), which have the following functionality:
> > > > - Filtering cluster nodes (projection) for compute
> > > > - Executing task by the name
> > > >
> > > > I think we can implement this functionality in a thin client as well.
> > > >
> > > > First of all, we need some operation types to request a list of all
> > > > available nodes and probably node attributes (by a list of nodes).
> Node
> > > > attributes will be helpful if we will decide to implement analog of
> > > > ClusterGroup#forAttribute or ClusterGroup#forePredicate methods in
> the
> > > thin
> > > > client. Perhaps they can be requested lazily.
> > > >
> > > > From the protocol point of view there will be two new operations:
> > > >
> > > > OP_CLUSTER_GET_NODES
> > > > Request: empty
> > > > Response: long topologyVersion, int minorTopologyVersion, int
> > nodesCount,
> > > > for each node set of node fields (UUID nodeId, Object or String
> > > > consistentId, long order, etc)
> > > >
> > > > OP_CLUSTER_GET_NODE_ATTRIBUTES
> > > > Request: int nodesCount, for each node: UUID nodeId
> > > > Response: int nodesCount, for each node: int attributesCount, for
[jira] [Created] (IGNITE-12389) Make GridCacheQueryFutureAdapter.enqueue() use parameter of List type instead Collection
Yuriy Shuliha created IGNITE-12389:
---
Summary: Make GridCacheQueryFutureAdapter.enqueue() use parameter
of List type instead Collection
Key: IGNITE-12389
URL: https://issues.apache.org/jira/browse/IGNITE-12389
Project: Ignite
Issue Type: Improvement
Affects Versions: 2.7.6
Reporter: Yuriy Shuliha
Fix For: 2.8
Make {{GridCacheQueryFutureAdapter.enqueue(Collection col)}} use parameter
of List type instead Collection
This will allow to avoid new {{ArrayList}} creation and directly call
performance tolerant {{List.subList()}} method .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)
Dear Igniters, The first part of TextQuery improvement - a result limit - was developed and merged. Now we have to develop most important functionality here - proper sorting of Lucene index response and correct reducing of them for distributed queries. *There are two Lucene based aspects* 1. In case of using no sorting fields, the documents in response are still ordered by relevance. Actually this is ScoreDoc.score value. In order to reduce the distributed results correctly, the score should be passed with response. 2. When sorting by conventional fields, then Lucene should have these fields properly indexed and corresponding Sort object should be applied to Lucene's search call. In order to mark those fields a new annotation like '@SortField' may be introduced. *Reducing on Ignite * The obvious point of distributed response reduction is class GridCacheDistributedQueryFuture. Though, @Ivan Pavlukhin mentioned class with similar functionality: ReduceIndexSorted What I see here, that it is tangled with H2 related classes ( org.h2.result.Row) and might not be unified with TextQuery reduction. Still need a support here. Overall, the goal of this letter is to initiate discussion on TextQuery Sorting implementation and come closer to ticket creation. BR, Yuriy Shuliha вт, 22 жовт. 2019 о 13:31 Andrey Mashenkov пише: > Hi Dmitry, Yuriy. > > I've found GridCacheQueryFutureAdapter has newly added AtomicInteger > 'total' field and 'limit; field as primitive int. > > Both fields are used inside synchronized block only. > So, we can make both private and downgrade AtomicInteger to primitive int. > > Most likely, these fields can be replaced with one field. > > > > On Mon, Oct 21, 2019 at 10:01 PM Dmitriy Pavlov > wrote: > > > Hi Andrey, > > > > I've checked this ticket comments, and there is a TC Bot visa (with no > > blockers). > > > > Do you have any concerns related to this patch? > > > > Sincerely, > > Dmitriy Pavlov > > > > чт, 17 окт. 2019 г. в 16:43, Yuriy Shuliga : > > > >> Andrey, > >> > >> Per you request, I created ticket > >> https://issues.apache.org/jira/browse/IGNITE-12291 linked to > >> https://issues.apache.org/jira/projects/IGNITE/issues/IGNITE-12189 > >> > >> Could you please proceed with PR merge ? > >> > >> BR, > >> Yuriy Shuliha > >> > >> ср, 9 жовт. 2019 о 12:52 Andrey Mashenkov > >> пише: > >> > >> > Hi Yuri, > >> > > >> > To get access to TC Bot you should register as TeamCity user [1], if > you > >> > didn't do this already. > >> > Then you will be able to authorize on Ignite TC Bot page with same > >> > credentials. > >> > > >> > [1] https://ci.ignite.apache.org/registerUser.html > >> > > >> > On Fri, Oct 4, 2019 at 3:10 PM Yuriy Shuliga > wrote: > >> > > >> >> Andrew, > >> >> > >> >> I have corrected PR according to your notes. Please review. > >> >> What will be the next steps in order to merge in? > >> >> > >> >> Y. > >> >> > >> >> чт, 3 жовт. 2019 о 17:47 Andrey Mashenkov < > andrey.mashen...@gmail.com> > >> >> пише: > >> >> > >> >> > Yuri, > >> >> > > >> >> > I've done with review. > >> >> > No crime found, but trivial compatibility bug. > >> >> > > >> >> > On Thu, Oct 3, 2019 at 3:54 PM Yuriy Shuliga > >> wrote: > >> >> > > >> >> > > Denis, > >> >> > > > >> >> > > Thank you for your attention to this. > >> >> > > as for now, the > https://issues.apache.org/jira/browse/IGNITE-12189 > >> >> > ticket > >> >> > > is still pending review. > >> >> > > Do we have a chance to move it forward somehow? > >> >> > > > >> >> > > BR, > >> >> > > Yuriy Shuliha > >> >> > > > >> >> > > пн, 30 вер. 2019 о 23:35 Denis Magda пише: > >> >> > > > >> >> > > > Yuriy, > >> >> > > > > >> >> > > > I've seen you opening a pull-request with the first changes: > >> >> > > > https://issues.apache.org/jira/browse/IGNITE-12189 > >> >> > > > > >> >> > > > Alex Scherbakov and Ivan are you the right guys to do the > review? > >> >> > > > > >> >> > > > - > >> >> > > > Denis > >> >> > > > > >> >> > > > > >> >> > > > On Fri, Sep 27, 2019 at 8:48 AM Павлухин Иван < > >> vololo...@gmail.com> > >> >> > > wrote: > >> >> > > > > >> >> > > > > Yuriy, > >> >> > > > > > >> >> > > > > Thank you for providing details! Quite interesting. > >> >> > > > > > >> >> > > > > Yes, we already have support of distributed limit and merging > >> >> sorted > >> >> > > > > subresults for SQL queries. E.g. ReduceIndexSorted and > >> >> > > > > MergeStreamIterator are used for merging sorted streams. > >> >> > > > > > >> >> > > > > Could you please also clarify about score/relevance? Is it > >> >> provided > >> >> > by > >> >> > > > > Lucene engine for each query result? I am thinking how to do > >> >> sorted > >> >> > > > > merge properly in this case. > >> >> > > > > > >> >> > > > > ср, 25 сент. 2019 г. в 18:56, Yuriy Shuliga < > shul...@gmail.com > >> >: > >> >> > > > > > > >> >> > > > > > Ivan, > >> >> > > > > > > >> >> > > > > > Thank you for interesting question! > >> >> > > > > > > >> >> > > > > > Text searches (or full
[jira] [Created] (IGNITE-12390) .NET: Add automatic NuGet verification as part of the release process
Pavel Tupitsyn created IGNITE-12390: --- Summary: .NET: Add automatic NuGet verification as part of the release process Key: IGNITE-12390 URL: https://issues.apache.org/jira/browse/IGNITE-12390 Project: Ignite Issue Type: Bug Components: platforms Reporter: Pavel Tupitsyn Assignee: Pavel Tupitsyn Fix For: 2.8 Use dotnet CLI to verify that resulting nupkg file can be installed and Ignite node starts (dotnet new console, dotnet add package, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
Great! Limit parameter is supported for Ignite
Yuriy, congrats with the first contribution and thank you! Finally, someone took over the TextQuery component of Ignite! https://issues.apache.org/jira/browse/IGNITE-12189 Ivan, Andrey, Alexey, thanks for guiding Yuriy and helping him to complete the first significant task ;) - Denis
Re: [DISCUSS] Pub/Sub Streamer Implementation
Hi Emmanouil, Do you have any questions or need any support from the community? - Denis On Sun, Nov 10, 2019 at 3:07 PM Saikat Maitra wrote: > Hi Emmanouil, > > Can you please take a looks at dev utils, if this is something you are > looking for https://github.com/apache/ignite/tree/master/modules/dev-utils > ? > > IMO, if you can release Pub/Sub server in maven and then use it as > dependency, that would be great. > > Regards, > Saikat > > On Sun, Nov 10, 2019 at 5:00 PM Saikat Maitra > wrote: > > > Hi Emmanouil, > > > > The master branch has been initialized. I have raised a PR to migrate > > Flink module to ignite-extensions. > > > > PR https://github.com/apache/ignite-extensions/pull/1 > > Jira : https://issues.apache.org/jira/browse/IGNITE-12356 > > > > Regards, > > Saikat > > > > > > On Sun, Nov 10, 2019 at 3:39 PM Emmanouil Gkatziouras < > > gkatzio...@gmail.com> wrote: > > > >> Hi all! > >> > >> This is an update. The integration is tested with mocks based on the > >> original Ignite repo [1]. > >> I am currently waiting for the first commits on the new integration > >> project > >> to take place in order to create my pull request accordingly to the > >> initial > >> commits. > >> In order to simulate the Pub/Sub server in a test I used a test server > >> implementation [2]. Unfortunately they don't have this util uploaded in > a > >> maven repo, thus it needs to be hosted on a maven repo (Apache License > >> 2.0). Is there a repo for utilities that Ignite uses for the test/build? > >> I have also created a ticket [3] > >> > >> Kind regards > >> > >> [1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub > >> [2] https://github.com/GoogleCloudPlatform/kafka-pubsub-emulator > >> [3] https://issues.apache.org/jira/browse/IGNITE-12262 > >> > >> *Emmanouil Gkatziouras* > >> https://egkatzioura.com/ | > >> https://www.linkedin.com/in/gkatziourasemmanouil/ > >> https://github.com/gkatzioura > >> > >> > >> On Fri, 25 Oct 2019 at 03:35, Saikat Maitra > >> wrote: > >> > >> > Hello Ilya, > >> > > >> > Thank you for your email. Yes, before we remove any integration we > will > >> > move them to another repository and release. > >> > > >> > We are thinking we will start with Pub/Sub Streamer implementation and > >> > continue to move other integration as mentioned in the following doc. > >> > > >> > > >> > > >> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > >> > > >> > Regards, > >> > Saikat > >> > > >> > > >> > On Thu, Oct 24, 2019 at 11:00 AM Ilya Kasnacheev < > >> > ilya.kasnach...@gmail.com> > >> > wrote: > >> > > >> > > Hello! > >> > > > >> > > My take on this: we want to offload some of integrations, but please > >> > commit > >> > > and release them to another project first, and only then we will > >> decide > >> > to > >> > > remove them from our repository. > >> > > > >> > > Please also make sure they are available with same artifact name and > >> API. > >> > > If this has to be changed, let's wait for AI 3.0 > >> > > > >> > > Regards, > >> > > -- > >> > > Ilya Kasnacheev > >> > > > >> > > > >> > > ср, 23 окт. 2019 г. в 03:23, Saikat Maitra >: > >> > > > >> > > > Hello Emmanouil, > >> > > > > >> > > > As discussed earlier, I discussed with Apache Bahir community and > >> they > >> > > are > >> > > > interested to have Apache Ignite extensions as part of Apache > Bahir > >> > > > project. > >> > > > > >> > > > Can you please share Pub/Sub streamer implementation details with > >> > Apache > >> > > > Bahir community and request for feedback. > >> > > > > >> > > > https://bahir.apache.org/ > >> > > > > >> > > > I have also requested for contributor access in Jira for Apache > >> Bahir > >> > > > project so that I can create issues and assign to myself. > >> > > > > >> > > > I can help with code reviews as well. > >> > > > > >> > > > Here is the email thread for reference > >> > > > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html > >> > > > > >> > > > Regards, > >> > > > Saikat > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra < > >> saikat.mai...@gmail.com > >> > > > >> > > > wrote: > >> > > > > >> > > > > Hello, > >> > > > > > >> > > > > As discussed I have created following discussion threads on our > >> > > migration > >> > > > > proposals for Apache Ignite extensions: > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html > >> > > > > > >> > > > > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html > >> > > > > > >> > > > > I will share update as I hear more information. > >> > > > > > >> > > > > Regards, > >> > > > > Saikat > >> > > > > > >> > > > > > >> > > > > > >> > > > > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra < > >> >
Re: [DISCUSS] Pub/Sub Streamer Implementation
Hi all! I am sorry for being late on that. I was trying to refactor the test in order not to be in need of any external tools or spinning up a server. I did forked the new repo and indeed my changes there, so a pull request is a matter of time! Kind regards *Emmanouil Gkatziouras* https://egkatzioura.com/ | https://www.linkedin.com/in/gkatziourasemmanouil/ https://github.com/gkatzioura On Fri, 22 Nov 2019 at 20:45, Denis Magda wrote: > Hi Emmanouil, > > Do you have any questions or need any support from the community? > > - > Denis > > > On Sun, Nov 10, 2019 at 3:07 PM Saikat Maitra > wrote: > >> Hi Emmanouil, >> >> Can you please take a looks at dev utils, if this is something you are >> looking for >> https://github.com/apache/ignite/tree/master/modules/dev-utils? >> >> IMO, if you can release Pub/Sub server in maven and then use it as >> dependency, that would be great. >> >> Regards, >> Saikat >> >> On Sun, Nov 10, 2019 at 5:00 PM Saikat Maitra >> wrote: >> >> > Hi Emmanouil, >> > >> > The master branch has been initialized. I have raised a PR to migrate >> > Flink module to ignite-extensions. >> > >> > PR https://github.com/apache/ignite-extensions/pull/1 >> > Jira : https://issues.apache.org/jira/browse/IGNITE-12356 >> > >> > Regards, >> > Saikat >> > >> > >> > On Sun, Nov 10, 2019 at 3:39 PM Emmanouil Gkatziouras < >> > gkatzio...@gmail.com> wrote: >> > >> >> Hi all! >> >> >> >> This is an update. The integration is tested with mocks based on the >> >> original Ignite repo [1]. >> >> I am currently waiting for the first commits on the new integration >> >> project >> >> to take place in order to create my pull request accordingly to the >> >> initial >> >> commits. >> >> In order to simulate the Pub/Sub server in a test I used a test server >> >> implementation [2]. Unfortunately they don't have this util uploaded >> in a >> >> maven repo, thus it needs to be hosted on a maven repo (Apache License >> >> 2.0). Is there a repo for utilities that Ignite uses for the >> test/build? >> >> I have also created a ticket [3] >> >> >> >> Kind regards >> >> >> >> [1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub >> >> [2] https://github.com/GoogleCloudPlatform/kafka-pubsub-emulator >> >> [3] https://issues.apache.org/jira/browse/IGNITE-12262 >> >> >> >> *Emmanouil Gkatziouras* >> >> https://egkatzioura.com/ | >> >> https://www.linkedin.com/in/gkatziourasemmanouil/ >> >> https://github.com/gkatzioura >> >> >> >> >> >> On Fri, 25 Oct 2019 at 03:35, Saikat Maitra >> >> wrote: >> >> >> >> > Hello Ilya, >> >> > >> >> > Thank you for your email. Yes, before we remove any integration we >> will >> >> > move them to another repository and release. >> >> > >> >> > We are thinking we will start with Pub/Sub Streamer implementation >> and >> >> > continue to move other integration as mentioned in the following doc. >> >> > >> >> > >> >> > >> >> >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >> >> > >> >> > Regards, >> >> > Saikat >> >> > >> >> > >> >> > On Thu, Oct 24, 2019 at 11:00 AM Ilya Kasnacheev < >> >> > ilya.kasnach...@gmail.com> >> >> > wrote: >> >> > >> >> > > Hello! >> >> > > >> >> > > My take on this: we want to offload some of integrations, but >> please >> >> > commit >> >> > > and release them to another project first, and only then we will >> >> decide >> >> > to >> >> > > remove them from our repository. >> >> > > >> >> > > Please also make sure they are available with same artifact name >> and >> >> API. >> >> > > If this has to be changed, let's wait for AI 3.0 >> >> > > >> >> > > Regards, >> >> > > -- >> >> > > Ilya Kasnacheev >> >> > > >> >> > > >> >> > > ср, 23 окт. 2019 г. в 03:23, Saikat Maitra < >> saikat.mai...@gmail.com>: >> >> > > >> >> > > > Hello Emmanouil, >> >> > > > >> >> > > > As discussed earlier, I discussed with Apache Bahir community and >> >> they >> >> > > are >> >> > > > interested to have Apache Ignite extensions as part of Apache >> Bahir >> >> > > > project. >> >> > > > >> >> > > > Can you please share Pub/Sub streamer implementation details with >> >> > Apache >> >> > > > Bahir community and request for feedback. >> >> > > > >> >> > > > https://bahir.apache.org/ >> >> > > > >> >> > > > I have also requested for contributor access in Jira for Apache >> >> Bahir >> >> > > > project so that I can create issues and assign to myself. >> >> > > > >> >> > > > I can help with code reviews as well. >> >> > > > >> >> > > > Here is the email thread for reference >> >> > > > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html >> >> > > > >> >> > > > Regards, >> >> > > > Saikat >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra < >> >> saikat.mai...@gmail.com >> >> > > >> >> > > > wrote: >> >> > > > >> >> > > > > Hello, >> >> > > > > >> >> > > > > As discussed I have created following
Re: [DISCUSS] Pub/Sub Streamer Implementation
Awesome, ping us whenever you're ready! - Denis On Fri, Nov 22, 2019 at 12:52 PM Emmanouil Gkatziouras wrote: > Hi all! > > I am sorry for being late on that. I was trying to refactor the test in > order not to be in need of any external tools or spinning up a server. > I did forked the new repo and indeed my changes there, so a pull request is > a matter of time! > > Kind regards > > *Emmanouil Gkatziouras* > https://egkatzioura.com/ | > https://www.linkedin.com/in/gkatziourasemmanouil/ > https://github.com/gkatzioura > > > On Fri, 22 Nov 2019 at 20:45, Denis Magda wrote: > > > Hi Emmanouil, > > > > Do you have any questions or need any support from the community? > > > > - > > Denis > > > > > > On Sun, Nov 10, 2019 at 3:07 PM Saikat Maitra > > wrote: > > > >> Hi Emmanouil, > >> > >> Can you please take a looks at dev utils, if this is something you are > >> looking for > >> https://github.com/apache/ignite/tree/master/modules/dev-utils? > >> > >> IMO, if you can release Pub/Sub server in maven and then use it as > >> dependency, that would be great. > >> > >> Regards, > >> Saikat > >> > >> On Sun, Nov 10, 2019 at 5:00 PM Saikat Maitra > >> wrote: > >> > >> > Hi Emmanouil, > >> > > >> > The master branch has been initialized. I have raised a PR to migrate > >> > Flink module to ignite-extensions. > >> > > >> > PR https://github.com/apache/ignite-extensions/pull/1 > >> > Jira : https://issues.apache.org/jira/browse/IGNITE-12356 > >> > > >> > Regards, > >> > Saikat > >> > > >> > > >> > On Sun, Nov 10, 2019 at 3:39 PM Emmanouil Gkatziouras < > >> > gkatzio...@gmail.com> wrote: > >> > > >> >> Hi all! > >> >> > >> >> This is an update. The integration is tested with mocks based on the > >> >> original Ignite repo [1]. > >> >> I am currently waiting for the first commits on the new integration > >> >> project > >> >> to take place in order to create my pull request accordingly to the > >> >> initial > >> >> commits. > >> >> In order to simulate the Pub/Sub server in a test I used a test > server > >> >> implementation [2]. Unfortunately they don't have this util uploaded > >> in a > >> >> maven repo, thus it needs to be hosted on a maven repo (Apache > License > >> >> 2.0). Is there a repo for utilities that Ignite uses for the > >> test/build? > >> >> I have also created a ticket [3] > >> >> > >> >> Kind regards > >> >> > >> >> [1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub > >> >> [2] https://github.com/GoogleCloudPlatform/kafka-pubsub-emulator > >> >> [3] https://issues.apache.org/jira/browse/IGNITE-12262 > >> >> > >> >> *Emmanouil Gkatziouras* > >> >> https://egkatzioura.com/ | > >> >> https://www.linkedin.com/in/gkatziourasemmanouil/ > >> >> https://github.com/gkatzioura > >> >> > >> >> > >> >> On Fri, 25 Oct 2019 at 03:35, Saikat Maitra > > >> >> wrote: > >> >> > >> >> > Hello Ilya, > >> >> > > >> >> > Thank you for your email. Yes, before we remove any integration we > >> will > >> >> > move them to another repository and release. > >> >> > > >> >> > We are thinking we will start with Pub/Sub Streamer implementation > >> and > >> >> > continue to move other integration as mentioned in the following > doc. > >> >> > > >> >> > > >> >> > > >> >> > >> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > >> >> > > >> >> > Regards, > >> >> > Saikat > >> >> > > >> >> > > >> >> > On Thu, Oct 24, 2019 at 11:00 AM Ilya Kasnacheev < > >> >> > ilya.kasnach...@gmail.com> > >> >> > wrote: > >> >> > > >> >> > > Hello! > >> >> > > > >> >> > > My take on this: we want to offload some of integrations, but > >> please > >> >> > commit > >> >> > > and release them to another project first, and only then we will > >> >> decide > >> >> > to > >> >> > > remove them from our repository. > >> >> > > > >> >> > > Please also make sure they are available with same artifact name > >> and > >> >> API. > >> >> > > If this has to be changed, let's wait for AI 3.0 > >> >> > > > >> >> > > Regards, > >> >> > > -- > >> >> > > Ilya Kasnacheev > >> >> > > > >> >> > > > >> >> > > ср, 23 окт. 2019 г. в 03:23, Saikat Maitra < > >> saikat.mai...@gmail.com>: > >> >> > > > >> >> > > > Hello Emmanouil, > >> >> > > > > >> >> > > > As discussed earlier, I discussed with Apache Bahir community > and > >> >> they > >> >> > > are > >> >> > > > interested to have Apache Ignite extensions as part of Apache > >> Bahir > >> >> > > > project. > >> >> > > > > >> >> > > > Can you please share Pub/Sub streamer implementation details > with > >> >> > Apache > >> >> > > > Bahir community and request for feedback. > >> >> > > > > >> >> > > > https://bahir.apache.org/ > >> >> > > > > >> >> > > > I have also requested for contributor access in Jira for Apache > >> >> Bahir > >> >> > > > project so that I can create issues and assign to myself. > >> >> > > > > >> >> > > > I can help with code reviews as well. > >> >> > > > > >> >> > > > Here is the email thread for
Re: Contribution to Apache Ignite
Lev, congrats with your first contribution! Let us know if you're ready to carry on and finish the second ticket. As long as Sergey has stepped in, I would say that you're in good hands and the second contribution will go smoothly ;) - Denis On Wed, Nov 20, 2019 at 6:37 AM Sergey Chugunov wrote: > Hello Lev, > > My name is Sergey, I'm from Apache Ignite community. As I can see you > successfully completed your first ticket but there are some review comments > on your second one [1]. > > Do you need any assistance with resolving them? > > Also if you're interested in more challenging tasks, there are plenty of > them and we could figure out what to pick up next. > > Anyway, thank you for your interest to our project and community! > > [1] https://issues.apache.org/jira/browse/IGNITE-11312 > > -- > Best regards, > Sergey Chugunov. >
[jira] [Created] (IGNITE-12391) Rename "collocated" to "colocated" in APIs and Docs
Denis A. Magda created IGNITE-12391: --- Summary: Rename "collocated" to "colocated" in APIs and Docs Key: IGNITE-12391 URL: https://issues.apache.org/jira/browse/IGNITE-12391 Project: Ignite Issue Type: Bug Components: cache, documentation, sql Affects Versions: 2.7.6 Reporter: Denis A. Magda Fix For: 3.0 Rename all the "collocated" occurrences in APIs and docs. Check discussion here for more details: https://issues.apache.org/jira/browse/IGNITE-12382 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Pub/Sub Streamer Implementation
Hi all, I made my first pull request [1]. Since this project is brand new (no parent poms, licensing), I reckoned it was best to use Saikat's branch on Flink. I suppose the Flink branch will be merged first. If not please give me guidelines on how I should proceed next. Kind regards Emmanouil [1] https://github.com/apache/ignite-extensions/pull/2 *Emmanouil Gkatziouras* https://egkatzioura.com/ | https://www.linkedin.com/in/gkatziourasemmanouil/ https://github.com/gkatzioura On Fri, 22 Nov 2019 at 20:55, Denis Magda wrote: > Awesome, ping us whenever you're ready! > > - > Denis > > > On Fri, Nov 22, 2019 at 12:52 PM Emmanouil Gkatziouras < > gkatzio...@gmail.com> > wrote: > > > Hi all! > > > > I am sorry for being late on that. I was trying to refactor the test in > > order not to be in need of any external tools or spinning up a server. > > I did forked the new repo and indeed my changes there, so a pull request > is > > a matter of time! > > > > Kind regards > > > > *Emmanouil Gkatziouras* > > https://egkatzioura.com/ | > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > https://github.com/gkatzioura > > > > > > On Fri, 22 Nov 2019 at 20:45, Denis Magda wrote: > > > > > Hi Emmanouil, > > > > > > Do you have any questions or need any support from the community? > > > > > > - > > > Denis > > > > > > > > > On Sun, Nov 10, 2019 at 3:07 PM Saikat Maitra > > > > wrote: > > > > > >> Hi Emmanouil, > > >> > > >> Can you please take a looks at dev utils, if this is something you are > > >> looking for > > >> https://github.com/apache/ignite/tree/master/modules/dev-utils? > > >> > > >> IMO, if you can release Pub/Sub server in maven and then use it as > > >> dependency, that would be great. > > >> > > >> Regards, > > >> Saikat > > >> > > >> On Sun, Nov 10, 2019 at 5:00 PM Saikat Maitra < > saikat.mai...@gmail.com> > > >> wrote: > > >> > > >> > Hi Emmanouil, > > >> > > > >> > The master branch has been initialized. I have raised a PR to > migrate > > >> > Flink module to ignite-extensions. > > >> > > > >> > PR https://github.com/apache/ignite-extensions/pull/1 > > >> > Jira : https://issues.apache.org/jira/browse/IGNITE-12356 > > >> > > > >> > Regards, > > >> > Saikat > > >> > > > >> > > > >> > On Sun, Nov 10, 2019 at 3:39 PM Emmanouil Gkatziouras < > > >> > gkatzio...@gmail.com> wrote: > > >> > > > >> >> Hi all! > > >> >> > > >> >> This is an update. The integration is tested with mocks based on > the > > >> >> original Ignite repo [1]. > > >> >> I am currently waiting for the first commits on the new integration > > >> >> project > > >> >> to take place in order to create my pull request accordingly to the > > >> >> initial > > >> >> commits. > > >> >> In order to simulate the Pub/Sub server in a test I used a test > > server > > >> >> implementation [2]. Unfortunately they don't have this util > uploaded > > >> in a > > >> >> maven repo, thus it needs to be hosted on a maven repo (Apache > > License > > >> >> 2.0). Is there a repo for utilities that Ignite uses for the > > >> test/build? > > >> >> I have also created a ticket [3] > > >> >> > > >> >> Kind regards > > >> >> > > >> >> [1] > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub > > >> >> [2] https://github.com/GoogleCloudPlatform/kafka-pubsub-emulator > > >> >> [3] https://issues.apache.org/jira/browse/IGNITE-12262 > > >> >> > > >> >> *Emmanouil Gkatziouras* > > >> >> https://egkatzioura.com/ | > > >> >> https://www.linkedin.com/in/gkatziourasemmanouil/ > > >> >> https://github.com/gkatzioura > > >> >> > > >> >> > > >> >> On Fri, 25 Oct 2019 at 03:35, Saikat Maitra < > saikat.mai...@gmail.com > > > > > >> >> wrote: > > >> >> > > >> >> > Hello Ilya, > > >> >> > > > >> >> > Thank you for your email. Yes, before we remove any integration > we > > >> will > > >> >> > move them to another repository and release. > > >> >> > > > >> >> > We are thinking we will start with Pub/Sub Streamer > implementation > > >> and > > >> >> > continue to move other integration as mentioned in the following > > doc. > > >> >> > > > >> >> > > > >> >> > > > >> >> > > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > >> >> > > > >> >> > Regards, > > >> >> > Saikat > > >> >> > > > >> >> > > > >> >> > On Thu, Oct 24, 2019 at 11:00 AM Ilya Kasnacheev < > > >> >> > ilya.kasnach...@gmail.com> > > >> >> > wrote: > > >> >> > > > >> >> > > Hello! > > >> >> > > > > >> >> > > My take on this: we want to offload some of integrations, but > > >> please > > >> >> > commit > > >> >> > > and release them to another project first, and only then we > will > > >> >> decide > > >> >> > to > > >> >> > > remove them from our repository. > > >> >> > > > > >> >> > > Please also make sure they are available with same artifact > name > > >> and > > >> >> API. > > >> >> > > If this has to be changed, let's wait for AI 3.0 > > >> >> > > > > >> >> > > Regards, > > >> >> > > -- > > >> >> > > Ily
