Re: ignitevisorcmd.sh does not connect to grid
Hello, Ali. Looks like this is known issue and it was fixed by [1] Nikolay, can you confirm it? [1] https://github.com/apache/ignite/commit/d0bae90bce33dbed8b9f3cd52663ce5efa545d97 > 28 марта 2020 г., в 12:30, ali hassan написал(а): > > Hi, > I failed to connect ignitevisorcmd.sh to my grid. The result of open is: > > +-+ > > | Status | Disconnected | > > | Ignite instance name | | > > | Config path | | > > | Uptime | | > > +-+ > > visor> > > > > even if I choose the right conf, and I repeat open many times. > > > > The grid is running, and I can perform queries normally. > > There is no exception, nor indication in the logs. > > I used –v in both ignite.sh and ignitevisorcmd.sh. > > > > I appreciate your help. > > > > Best Regards, > > Ali Hassan > > Senior Scala SE > > Senior BigData SE
Re: Apache Ignite 2.8.0 spring services configuration null fields
Thank you for answer Nikolay, I receive various null pointer exceptions @ runtime on TestServiceImpl for all fields extended from Test class My class TestServiceImpl class is defined by the following pice of code /public class TestServiceImpl extends Test implements Service, TestService/ Test - contains all my specific fields according with TestService This scenario works well in 2.7.6 but not in 2.8.0 -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Apache Ignite 2.8.0 spring services configuration null fields
Hello. Sill lack of information. Please, send stack trace and piece of code that produce the error. > 30 марта 2020 г., в 18:33, myset написал(а): > > Thank you for answer Nikolay, > I receive various null pointer exceptions @ runtime on TestServiceImpl for > all fields extended from Test class > My class TestServiceImpl class is defined by the following pice of code > > /public class TestServiceImpl extends Test implements Service, TestService/ > > Test - contains all my specific fields according with TestService > This scenario works well in 2.7.6 but not in 2.8.0 > > > > > > > > -- > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Vladimir,
@param forceDeactivation If {@code true}, cluster deactivation will be
> forced.
It's true that it's possible to infer semantics of forced deactivation from
other parts of API. I just wanted to highlight that exactly this
description explains something that can be guessed by the parameter name.
I suppose to shorten the lookup path and shed a light on deactivation
semantics a bit:
> @param forceDeactivation If {@code true}, cluster will be deactivated even
> if running in-memory caches are present. All data in the corresponding
> caches will be vanished as a result.
Does this make sense?
On Fri, Mar 27, 2020 at 12:00 PM Vladimir Steshin
wrote:
> Ivan, hi.
>
>
> 1) >>> Is it correct? If we are on the same page, let's proceed this way
>
> It is correct.
>
>
> 2) - In many places in the code I can see the following javadoc
>
> > @param forceDeactivation If {@code true}, cluster deactivation will be
> forced.
>
> In the internal params/flags. You can also find /@see
> ClusterState#INACTIVE/ and full description with several public APIs (
> like /Ignite.active(boolean)/ ):
>
> //
>
> /* /
>
> //
>
> /* NOTE:/
>
> //
>
> /* Deactivation clears in-memory caches (without persistence) including
> the system caches./
>
> Should be enough. Is not?
>
>
> 27.03.2020 10:51, Ivan Rakov пишет:
> > Vladimir, Igniters,
> >
> > Let's emphasize our final plan.
> >
> > We are going to add --force flags that will be necessary to pass for a
> > deactivation if there are in-memory caches to:
> > 1) Rest API (already implemented in [1])
> > 2) Command line utility (already implemented in [1])
> > 3) JMX bean (going to be implemented in [2])
> > We are *not* going to change IgniteCluster or any other thick Java API,
> > thus we are *not* going to merge [3].
> > We plan to *fully rollback* [1] and [2] once cache data survival after
> > activation-deactivation cycle will be implemented.
> >
> > Is it correct? If we are on the same page, let's proceed this way.
> > I propose to:
> > - Create a JIRA issue for in-memory-data-safe deactivation (possibly,
> > without IEP and detailed design so far)
> > - Describe in the issue description what exact parts of API should be
> > removed under the issue scope.
> >
> > Also, a few questions on already merged [1]:
> > - We have removed GridClientClusterState#state(ClusterState) from Java
> > client API. Is it a legitimate thing to do? Don't we have to support API
> > compatibility for thin clients as well?
> > - In many places in the code I can see the following javadoc
> >
> >> @param forceDeactivation If {@code true}, cluster deactivation will
> be forced.
> >>
> >> As for me, this javadoc doesn't clarify anything. I'd suggest to
> describe
> > in which cases deactivation won't happen unless it's forced and which
> > impact forced deactivation will bring on the system.
> >
> > [1]: https://issues.apache.org/jira/browse/IGNITE-12701
> > [2]: https://issues.apache.org/jira/browse/IGNITE-12779
> > [3]: https://issues.apache.org/jira/browse/IGNITE-12614
> >
> > --
> > Ivan
> >
> > On Tue, Mar 24, 2020 at 7:18 PM Vladimir Steshin
> wrote:
> >
> >> Hi, Igniters.
> >>
> >> I'd like to remind you that cluster can be deactivated by user with 3
> >> utilities: control.sh, *JMX and the REST*. Proposed in [1] solution is
> >> not about control.sh. It suggests same approach regardless of the
> >> utility user executes. The task touches *only* *API of the user calls*,
> >> not the internal APIs.
> >>
> >> The reasons why flag “--yes” and confirmation prompt hasn’t taken into
> >> account for control.sh are:
> >>
> >> -Various commands widely use “--yes” just to start. Even not dangerous
> >> ones require “--yes” to begin. “--force” is dedicated for *harmless
> >> actions*.
> >>
> >> -Checking of probable data erasure works after command start and
> >> “--force” may not be required at all.
> >>
> >> -There are also JMX and REST. They have no “—yes” but should work alike.
> >>
> >> To get the deactivation safe I propose to merge last ticket with
> >> the JMX fixes [2]. In future releases, I believe, we should estimate
> >> jobs and fix memory erasure in general. For now, let’s prevent it. WDYT?
> >>
> >>
> >> [1] https://issues.apache.org/jira/browse/IGNITE-12614
> >>
> >> [2] https://issues.apache.org/jira/browse/IGNITE-12779
> >>
> >>
> >> 24.03.2020 15:55, Вячеслав Коптилин пишет:
> >>> Hello Nikolay,
> >>>
> >>> I am talking about the interactive mode of the control utility, which
> >>> requires explicit confirmation from the user.
> >>> Please take a look at DeactivateCommand#prepareConfirmation and its
> >> usages.
> >>> It seems to me, this mode has the same aim as the forceDeactivation
> flag.
> >>> We can change the message returned by
> >> DeactivateCommand#confirmationPrompt
> >>> as follows:
> >>> "Warning: the command will deactivate the cluster nnn and clear
> >>> in-memory caches (without persistence) including system caches."
> >>>
> >>> What do you think?
> >>
JSR 381 (Visual Recognition in Java) and Apache Ignite
Hi Alexey, Igniters, Let me introduce you Heather, Zoran, and Frank. Heather is the Chair of the JCP. Zoran and Frank are JSR 381 spec leads. They are interested in discussing the upcoming visual recognition specification with the Apache Ignite community, to understand whether the community has any interest in implementing it. Zoran and Frank, Please meet Alexey Zinoviev, who is the principal maintainer of the Apache ML module. I hope he will be able to help you with your questions. Regards, Vladimir.
Re: ignitevisorcmd.sh does not connect to grid
Ali, yes, this problem has been fixed. пн, 30 мар. 2020 г. в 11:44, Nikolay Izhikov : > Hello, Ali. > > Looks like this is known issue and it was fixed by [1] > > Nikolay, can you confirm it? > > [1] > https://github.com/apache/ignite/commit/d0bae90bce33dbed8b9f3cd52663ce5efa545d97 > > > 28 марта 2020 г., в 12:30, ali hassan написал(а): > > > > Hi, > > I failed to connect ignitevisorcmd.sh to my grid. The result of open is: > > > > +-+ > > > > | Status | Disconnected | > > > > | Ignite instance name | | > > > > | Config path | | > > > > | Uptime | | > > > > +-+ > > > > visor> > > > > > > > > even if I choose the right conf, and I repeat open many times. > > > > > > > > The grid is running, and I can perform queries normally. > > > > There is no exception, nor indication in the logs. > > > > I used –v in both ignite.sh and ignitevisorcmd.sh. > > > > > > > > I appreciate your help. > > > > > > > > Best Regards, > > > > Ali Hassan > > > > Senior Scala SE > > > > Senior BigData SE > >
Re: Apache Ignite 2.8.0 spring services configuration null fields
Please see the code bellow: *Ignite service definition ... ... *TestService interface* *Test class* *TestServiceImpl class* *Exceptions @ runtime with ignite 2.8.0* Thank you -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Apache Ignite 2.8.0 spring services configuration null fields
Hello. Sorry, I can’t see any code. Can you create a gist and sent us link to it? > 30 марта 2020 г., в 20:19, myset написал(а): > > Please see the code bellow: > > *Ignite service definition > ... > > ... > *TestService interface* > > > *Test class* > > > *TestServiceImpl class* > > > > *Exceptions @ runtime with ignite 2.8.0* > > > > Thank you > > > > > -- > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Apache Ignite 2.8.0 spring services configuration null fields
Stripped ... Please check the gist with code snippets https://gist.github.com/myset/696461e135d9c8a32588ff2a534d7eb8 Thank you Nikolay -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Unclear to use methods in the DiscoverySpiCustomMessage interface
Hi, Igniters. Could someone review my changes, please? [1] [2] I have removed the method from the internal interface (DiscoveryCustomMessage) and marked deprecated in the public interface (DiscoverySpiCustomMessage). [1] https://issues.apache.org/jira/browse/IGNITE-12400 [2] https://github.com/apache/ignite/pull/7268/files чт, 16 янв. 2020 г. в 16:40, Nikita Amelchev : > > Hi, Igniters. > > I have created the issue [1] and prepared PR [2] to remove the > stopProcess method from interfaces. > > I have removed the method from the internal interface > (DiscoveryCustomMessage) and marked deprecated in the public interface > (DiscoverySpiCustomMessage). > > Could anyone review my changes, please? > > [1] https://issues.apache.org/jira/browse/IGNITE-12400 > [2] https://github.com/apache/ignite/pull/7268/files > > пт, 25 окт. 2019 г. в 11:24, Nikolay Izhikov : > > > > Nikita, please, feel free: > > > > 1. To create a ticket to remove `stopProcess` method from the interface. > > This improvement looks obvious to me. > > Seems we can implement it easily. > > > > 2. To make all Discovery messages immutable. > > This one more complex. > > I think we should discuss every single mutable message separately. > > Lets' do it in a separate discussion. > > > > WDYT? > > > > В Пт, 25/10/2019 в 11:10 +0300, Ivan Pavlukhin пишет: > > > There are at least a couple more mutable messages: > > > StartRoutineDiscoveryMessage and SchemaProposeDiscoveryMessage. > > > > > > I agree that here that making all messages immutable are highly > > > desired (e.g. it will allow implementation of new discovery > > > protocols). But so far I do not understand why marker interface is a > > > better approach. Pavel, could you please elaborate? > > > > > > чт, 24 окт. 2019 г. в 17:55, Pavel Kovalenko : > > > > > > > > Hi Nikita, > > > > > > > > 1. As I can understand, this functionality is needed if a custom > > > > message is > > > > needed to be processed only on the coordinator node. I'm not following > > > > for > > > > what purposes it was implemented and I see the real usage only in tests. > > > > As far as this functionality is not used in production code, I suggest > > > > to > > > > remove it. > > > > 2. This method needs if we want to perform a message mutation during > > > > sending it over the ring. Each node after message receive can mutate its > > > > state and resend to another node. I know only 2 mechanisms where this > > > > functionality is needed: > > > > binary metadata propagation and cluster-wide metrics collecting. From my > > > > side of view, this is poor approach and we should avoid discovery > > > > messages > > > > mutation and use only immutable messages. It can be removed from the > > > > custom > > > > message interface and marker interface should be introduced instead. > > > > > > > > чт, 24 окт. 2019 г. в 16:35, Nikita Amelchev : > > > > > > > > > Hi, Igniters. > > > > > > > > > > The discovery custom message interface has two methods the use of > > > > > which is not clear to developers: > > > > > > > > > > 1. The stopProcess() method. Currently, it works only if the zookeeper > > > > > discovery configured. It doesn't work in TcpDiscoverySpi. I did not > > > > > find any usages of this method except tests. I suggest to remove it > > > > > from the discovery custom message interface. > > > > > > > > > > 2. The isMutable() method. It works in TcpDiscoverySpi but not in the > > > > > zookeeper SPI. Using this method is opaque to the developer when he > > > > > wants to add new features to the project. In general, it cannot be > > > > > used because may not be supported depending on the configuration. > > > > > Perhaps some functionality that uses it is not working correctly with > > > > > zookeeper configured. It seems we should avoid it in the custom > > > > > message interface too. > > > > > > > > > > WDYT? > > > > > > > > > > -- > > > > > Best wishes, > > > > > Amelchev Nikita > > > > > > > > > > > > > > > > > > -- > Best wishes, > Amelchev Nikita -- Best wishes, Amelchev Nikita
[jira] [Created] (IGNITE-12845) GridNioServer can infinitely lose some events
Aleksey Plekhanov created IGNITE-12845:
--
Summary: GridNioServer can infinitely lose some events
Key: IGNITE-12845
URL: https://issues.apache.org/jira/browse/IGNITE-12845
Project: Ignite
Issue Type: Bug
Reporter: Aleksey Plekhanov
With enabled optimization (IGNITE_NO_SELECTOR_OPTS = false, by default)
{{GridNioServer}} can lose some events for a channel (depending on JDK version
and OS). It can lead to connected applications hang. Reproducer:
{code:java}
public void testConcurrentLoad() throws Exception {
startGrid(0);
try (IgniteClient client = Ignition.startClient(new
ClientConfiguration().setAddresses("127.0.0.1:10800"))) {
ClientCache cache =
client.getOrCreateCache(DEFAULT_CACHE_NAME);
GridTestUtils.runMultiThreaded(
() -> {
for (int i = 0; i < 1000; i++)
cache.put(i, i);
}, 5, "run-async");
}
}
{code}
This reproducer hangs eventually on MacOS (tested with JDK 8, 11, 12, 13, 14),
hangs on some Linux environments (for example passed more than 100 times on
desktop Linux system with JDK 8, but hangs on team-city agents with JDK 8, 11)
and never hanged (passed more than 100 times) on windows system, but passes on
all systems and JDK versions when system property {{IGNITE_NO_SELECTOR_OPTS =
true}} is set.
The root cause: optimized {{SelectedSelectionKeySet}} always returns {{false}}
for {{contains()}} method. The {{contains()}} method used by
{{sun.nio.ch.SelectorImpl.processReadyEvents()}} method:
{code:java}
if (selectedKeys.contains(ski)) {
if (ski.translateAndUpdateReadyOps(rOps)) {
return 1;
}
} else {
ski.translateAndSetReadyOps(rOps);
if ((ski.nioReadyOps() & ski.nioInterestOps()) != 0) {
selectedKeys.add(ski);
return 1;
}
}
{code}
So, for fair implementation, if a selection key is contained in the selected
keys set, then ready operations flags are updated, but for
{{SelectedSelectionKeySet}} ready operations flags will be always overridden
and new selector key will be added even if it's already contained in the set.
Some {{SelectorImpl}} implementations can pass several events for one selector
key to {{processReadyEvents }}method (for example, MacOs implementation
{{KQueueSelectorImpl}} works in such a way). In this case duplicated selector
keys will be added to {{selectedKeys}} and all events except last will be lost.
Two bad things happen in {{GridNioServer}} due to described above reasons:
# Some event flags are lost and the worker doesn't process corresponding
action (for attached reproducer "channel is ready for reading" event is lost
and the workers never read the channel after some point in time).
# Duplicated selector keys with the same event flags (for attached reproducer
it's "channel is ready for writing" event, this duplication leads to wrong
processing of {{GridSelectorNioSessionImpl#procWrite}} flag, which will be
{{false}} in some cases, but at the same time selector key's {{interestedOps}}
will contain {{OP_WRITE}} operation and this operation never be excluded)
Possible solutions:
* Fair implementation of {{SelectedSelectionKeySet.contains}} method (this
will solve all problems but can be resource consuming)
* Always set {{GridSelectorNioSessionImpl#procWrite}} to {{true}} when adding
{{OP_WRITE}} to {{interestedOps}} (for example in
{{AbstractNioClientWorker.registerWrite()}} method). In this case, some
"channel is ready for reading" events (but not data) still can be lost, but not
infinitely, and eventually data will be read.
* Exclude {{OP_WRITE}} from {{interestedOps}} even if
{{GridSelectorNioSessionImpl#procWrite}} is {{false}} when there are no write
requests in the queue (see {{GridNioServer.stopPollingForWrite()}} method).
This solution has the same shortcomings as the previous one.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Thin client: compute support
Guys, I've finished IEP implementation for server-side and java thin client side (ticket: [1], PR: [2]). Can you please review the patch? [1]: https://issues.apache.org/jira/browse/IGNITE-12835 [2]: https://github.com/apache/ignite/pull/7572 пт, 27 мар. 2020 г. в 19:11, Pavel Tupitsyn : > Agree with Igor, let's set deployment aside for now, it is out of scope of > this IEP. > > > > On Fri, Mar 27, 2020 at 6:52 PM Igor Sapego wrote: > > > Hi guys, > > > > I like the proposal in general. > > > > Now for the task deployment - I think we should have separate API > > for it (and separate IEP I believe). Also, I'm not sure that this API > > should be a part of the API of any thin client as it seems weird to me > > to use Python client to deploy Java tasks. Control.sh or visor proposal > > sounds much better. > > > > Best Regards, > > Igor > > > > > > On Thu, Mar 26, 2020 at 10:56 PM Denis Magda wrote: > > > > > > > > > > Deployment API definitely needed as one of the next steps. Currently, > > we > > > > are talking only about the first step (execution of already deployed > > > > tasks). > > > > Also, I'm not sure about automatic redeploy and peer-class-loading > for > > > thin > > > > clients, I think it's better to have more control here and provide > API > > to > > > > explicitly deploy classes or jar files. WDYT? > > > > > > > > > Alex, agree that automatic redeployment is better suited for the > > management > > > APIs. How about adding this capability to our command-line tool > > > (control.sh, or visor cmd, or one new holistic tool). > > > > > > - > > > Denis > > > > > > > > > On Wed, Mar 25, 2020 at 1:04 PM Alex Plehanov > > > > wrote: > > > > > > > Pavel, > > > > > > > > 1. Actually it can be solved on the client-side (and already solved > in > > > PoC > > > > implementation). But I agreed it brings extra complexity for > > client-side > > > > implementation, will try to provide such guarantees on the > server-side. > > > > 2. ComputeTask has also "reduce" step which is executed on the > > initiator > > > > node. Binary-rest client implementation, for example, has such > affinity > > > > methods (to execute the task by name). I'm ok with removing it. At > > least > > > if > > > > someone will need it we can implement it again at any time in the > > future > > > > without protocol change. > > > > I've fixed IEP. > > > > > > > > Denis, > > > > > > > > Deployment API definitely needed as one of the next steps. Currently, > > we > > > > are talking only about the first step (execution of already deployed > > > > tasks). > > > > Also, I'm not sure about automatic redeploy and peer-class-loading > for > > > thin > > > > clients, I think it's better to have more control here and provide > API > > to > > > > explicitly deploy classes or jar files. WDYT? > > > > > > > > ср, 25 мар. 2020 г. в 21:17, Denis Magda : > > > > > > > > > Alex, thanks for preparing the outline. > > > > > > > > > > I'd like us to discuss an approach for compute tasks update with no > > > > > downtimes on the servers' end. For instance, let's assume that a > > > > > Python/C++/Node.JS developer requested to update a compute task he > > > called > > > > > from the app. Should we introduce some system level API to the > binary > > > > > protocol that can take a jar file (or class) and redeploy it > > > > automatically > > > > > with the usage of peer-class-loading? > > > > > > > > > > - > > > > > Denis > > > > > > > > > > > > > > > On Wed, Mar 25, 2020 at 5:47 AM Alex Plehanov < > > plehanov.a...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > Hello guys. > > > > > > > > > > > > I've implemented PoC and created IEP [1] for thin client compute > > grid > > > > > > functionality. Please have a look. > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-42+Thin+client%3A+compute+support > > > > > > > > > > > > пт, 24 янв. 2020 г. в 16:56, Alex Plehanov < > > plehanov.a...@gmail.com > > > >: > > > > > > > > > > > > > We've discussed thin client compute protocol with Pavel > Tupitsyn > > > and > > > > > Igor > > > > > > > Sapego and come to the conclusion that approach with two-way > > > requests > > > > > > > should be used: client generates taskId and send a request to > the > > > > > server > > > > > > to > > > > > > > execute a task. The server responds that the request has been > > > > accepted. > > > > > > > After task has finished the server notifies the client (send a > > > > request > > > > > > > without waiting for a response). The client can cancel the task > > by > > > > > > sending > > > > > > > a corresponding request to the server. > > > > > > > > > > > > > > Also, a node list should be passed (optionally) with a request > to > > > > limit > > > > > > > nodes to execute the task. > > > > > > > > > > > > > > I will create IEP and file detailed protocol changes shortly. > > > > > > > > > > > > > > вт, 21 янв. 2020 г. в 18:46, Alex Pleh
