Re: Ignite YARN deployment - how to use TCP IP Discovery?

Hi, I guess you can try collect all IP addresses from all nodes that managed by YARN (where you are going to start Ignite) and add them all to addresses into TcpDiscoveryVmIpFinder part of Ignite configuration. Also you should provide the guarantee that each such hosts will be able to connec

Re: Ignite 2.7.0 YARN deployment - IGNITE_JVM_OPTS doesn't seem to work

* > > Platform Content Technology, Bangalore > > [image: cid:image001.gif@01C95541.6801BF70] > > *Voice:* +91-98806 72987 +91-80-67492572 > > > > *From:* Seshan, Manoj N. (TR Tech, Content & Ops) > *Sent:* Tuesday, November 05, 2019 11:45 PM > *To:* user@ignite.apache.org &

Ignite 2.7.0 YARN deployment - IGNITE_JVM_OPTS doesn't seem to work

ite.apache.org Subject: Ignite YARN deployment - how to use TCP IP Discovery? We are using Ignite as a Distributed In-Memory cache, deployed using YARN on a Hadoop Cluster. We have configured Zookeeper Discovery, and this is working fine. Given this is a small 20 node Ignite cluster, Zookeeper D

Ignite YARN deployment - how to use TCP IP Discovery?

We are using Ignite as a Distributed In-Memory cache, deployed using YARN on a Hadoop Cluster. We have configured Zookeeper Discovery, and this is working fine. Given this is a small 20 node Ignite cluster, Zookeeper Discovery seems overkill. Would it be possible to switch to TCP Discovery? Mu

Re: Persistent Store Not enabled in Ignite Yarn Deployment

Hello! Unfortunately it's hard to tell why the node would stop without looking at client & server logs. Can you share these somewhere? Maybe you should also set memory policy for these nodes, to the values that your Yarn configuration expect them to have: https://apacheignite.readme.io/docs/memor

Re: Persistent Store Not enabled in Ignite Yarn Deployment

Thanks for your time !!! 1) Changed the logic of generating keys. Now added a query to get the maximum key from cache. New record will be max key + 1. This has resolved the issue of count mismatch. Thank you 2) After splitting data into more batches and writing them to grid in many iterations,

Re: Persistent Store Not enabled in Ignite Yarn Deployment

clients are started and stopped in different Iterations. As > we could see the client count becomes 0 and 1 whereas the servers count > remain the same as 10 > > 3) /tmp is a HDFS path where we have configured for Ignite and provided in > cluster.properties. We could cha

Re: Persistent Store Not enabled in Ignite Yarn Deployment

HDFS path where we have configured for Ignite and provided in cluster.properties. We could change this to any path. It would be helpful if there is a way to enable persistence in YARN deployment. Thank you. Best Regards, Raghav -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Persistent Store Not enabled in Ignite Yarn Deployment

I can see two options here: - Between iteration 1 and iteration 2 some nodes were stopped. Perhaps some new nodes were started. Data on stopped nodes became unavailab.e - Cache key collisions between iterations 1 and 2 so that 80% keys are identical and only 20% are distinct the second time. I ex

Re: Persistent Store Not enabled in Ignite Yarn Deployment

Hi, 1) Load data to cache var cacheConf: CacheConfiguration[Long, Data] = new CacheConfiguration[Long, Data]("DataCache") cacheConf.setCacheMode(CacheMode.PARTITIONED) cacheConf.setIndexedTypes(classOf[Long], classOf[Data]) val cache = ignite.getOrCreateCache(cacheConf) var dataMap = ge

Re: Persistent Store Not enabled in Ignite Yarn Deployment

to enable Ignite Native Persistence in Ignite Yarn Deployment. > Purpose of this is to have no eviction of data at all from Ignite grids. > Whenever the memory is full the data should get stored in disc. > > But when I try to add large number of records to Ignite Grid, the data is > gett

Persistent Store Not enabled in Ignite Yarn Deployment

Hello, Am trying to enable Ignite Native Persistence in Ignite Yarn Deployment. Purpose of this is to have no eviction of data at all from Ignite grids. Whenever the memory is full the data should get stored in disc. But when I try to add large number of records to Ignite Grid, the data is

Re: Ignite YARN deployment mode issues

Hello Ray, Yes, please try it and tell if it helps. Regards, -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite YARN deployment mode issues

Hi Ilya Do you mean modify the original config from to if my understanding is correct? Thanks -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite YARN deployment mode issues

folder can be picked up by ignite. > But for YARN deployment, the issue still exists. > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >

Re: Ignite YARN deployment mode issues

older can be picked up by ignite. But for YARN deployment, the issue still exists. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite YARN deployment mode issues

Hello Ray! -- Ilya Kasnacheev 2017-09-19 10:51 GMT+03:00 Ray : > I'm using ignite 2.1, my ignite config xml is as follows. > > I tried to put the log4j2.xml file under IGNITE_USERS_LIBS, still not > working. > I guess it should be relative to IGNITE_HOME, or absolute path (I expect it's no-no

Re: Ignite YARN deployment mode issues

Figured the second question out by myself. The node I'm running ./control.sh does not seem to have ignite grid started. Here's the yarn log. [08:09:30,866][SEVERE][main][IgniteKernal] Exception during start processors, node will be stopped and close connections class org.apache.ignite.IgniteExce

Ignite YARN deployment mode issues

I'm using ignite 2.1, my ignite config xml is as follows. And in standalone mode, ignite can read the log4j2.xml in local directory and works ok. When deployed in YARN, ignite says it can fi

Re: Ignite Yarn Deployment

e deployment so that each node picks that > config file. say for example I want to pass a 'log4j.properties' > > How do I upload/deploy config files as part of Ignite/Yarn deployment > > > Thanking you > Raja > > > > > -- > View this message in cont

Ignite Yarn Deployment

Hi, I have deployed Ignite on a Hadoop Yarn cluster and I wanted to pass a configuration file as part of the deployment so that each node picks that config file. say for example I want to pass a 'log4j.properties' How do I upload/deploy config files as part of Ignite/Yarn deployment

Re: Yarn deployment for static TcpDiscoverySpi issues:Urgent In Production

I have already fixed my issues, and add classpth://srping-bean.jar to load native jar NOT internet. Thanks -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Yarn-deployment-for-static-TcpDiscoverySpi-issues-Urgent-In-Production-tp7205p7289.html Sent from

Re: Yarn deployment for static TcpDiscoverySpi issues:Urgent In Production

http://apache-ignite-users.70518.x6.nabble.com/Yarn-deployment-for-static-TcpDiscoverySpi-issues-Urgent-In-Production-tp7205p7284.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Yarn deployment for memory capacity make a bigger than before: Urgent!!!

ology, is this correct? Ithink that this is not correct.* -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Yarn-deployment-for-memory-capacity-make-a-bigger-than-before-Urgent-tp7275p7282.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Yarn deployment for memory capacity make a bigger than before: Urgent!!!

, > addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.16.186.200], > sockAddrs=[sparkup1/172.16.186.200:47501, /0:0:0:0:0:0:0:1%1:47501, > /127.0.0.1:47501, /172.16.186.200:47501], discPort=47501, order=6, > intOrder=6, lastExchangeTime=1472048226549, loc=false, > ver=1.6.0#20160518-sha1:0b22c45b, isClient=fals

Yarn deployment for memory capacity make a bigger than before: Urgent!!!

7:17:11 INFO scheduler.TaskSetManager: Starting task 6.0 in stage 0.0 (TID 7, sparkup1, partition 6,NODE_LOCAL, 1967 bytes) 16/08/24 07:17:11 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 13763 ms on sparkup1 (3/1024) 16/08/24 07:17:11 INFO scheduler.TaskSetManager: Fini

Re: Yarn deployment for static TcpDiscoverySpi issues:Urgent In Production

anDefinitions(XmlBeanDefinitionReader.java:303) at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379) ... 21 more -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Yarn-deploym

Re: Yarn deployment for static TcpDiscoverySpi issues:Urgent In Production

Thanks Val, I have fixed my issues for above, thanks for your great help on this issue. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Yarn-deployment-for-static-TcpDiscoverySpi-issues-Urgent-In-Production-tp7205p7253.html Sent from the Apache Ignite Users

Re: Yarn deployment for static TcpDiscoverySpi issues:Urgent In Production

Configuration file is required on all nodes where Ignite will start. On executors the same path that you provide will be used. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Yarn-deployment-for-static-TcpDiscoverySpi-issues-Urgent-In-Production

Re: Yarn deployment for static TcpDiscoverySpi issues:Urgent In Production

unt) 2) my ignite default.xml configuration is static IP NOT mulicast -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Yarn-deployment-for-static-TcpDiscoverySpi-issues-Urgent-In-Production-tp7205p7238.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Yarn deployment for static TcpDiscoverySpi issues:Urgent In Production

active=0, idle=48, qSize=0] ^-- System thread pool [active=0, idle=48, qSize=0] ^-- Outbound messages queue [size=0] [17:08:34,136][INFO][grid-timeout-worker-#97%null%][IgniteKernal] Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=28cf1142, nam

Re: Yarn deployment for static TcpDiscoverySpi issues:Urgent In Production

re still an issue? I see in the latest log that the node is successfully started. P.S. Please also use 'attach' to provide large log files. They are pretty much unreadable when copy-pasted here. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.co

Re: Yarn deployment for static TcpDiscoverySpi issues:Urgent In Production

cessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$ > deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit

Yarn deployment for static TcpDiscoverySpi issues:Urgent In Production

k-70fba93f-d31c-44d7-ada1-5fe0b9dae5cc 16/08/22 05:46:03 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 16/08/22 05:46:03 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 16/08/22 05:46:03 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-70fba93f-d31c-44d7-ada1-5fe0b9dae5cc/httpd-91da8e86-bee0-4c5a-868a-ea7d35a2536d [root@sparkup1 config]# -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Yarn-deployment-for-static-TcpDiscoverySpi-issues-Urgent-In-Production-tp7205.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: yarn deployment

Yes, that worked. Thank you. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/yarn-deployment-tp6843p7142.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Re:Re: Ignite for Spark on YARN Deployment

s.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p7075.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Re:Re: Ignite for Spark on YARN Deployment

ot;\t paris => " + sharedRDD.partitions.size) println("=>" + sharedRDD.filter(_._2 > 5).count) *final result is 4 NOT 5, this is happened also on embed integration with spark.* -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Sp

Re: Re:Re: Ignite for Spark on YARN Deployment

Hello, Nikolai, YES, I can ping yarn cluster from driver machine. I try to telnet localhost 50075 on driver machine, but failed, is this root cause for this issues? thanks!!! -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment

Re: Re:Re: Ignite for Spark on YARN Deployment

t; nodes(ClusterGroupAdapter.java:288) > at > org.apache.ignite.internal.processors.cluster.ClusterProcessor$ > UpdateNotifierTimerTask.safeRun(ClusterProcessor.java:224) > at org.apache.ignite.internal.util.GridTimerTask.run( > GridTimerTask.java:34) > at java.util.

Re: Re:Re: Ignite for Spark on YARN Deployment

feRun(ClusterProcessor.java:224) at org.apache.ignite.internal.util.GridTimerTask.run(GridTimerTask.java:34) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) -- View this message in context: http://apache-ignite-users.7

Re: Re:Re: Ignite for Spark on YARN Deployment

Can anyone tell me? thanks -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p7007.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Re:Re: Ignite for Spark on YARN Deployment

Hello, Nikolai, I'm a new ignite, can you please provide more detail steps for this? thanks again!!! -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6986.html Sent from the Apache Ignite Users mailing list archi

Re: Re:Re: Ignite for Spark on YARN Deployment

> > 3) I'm running spark-shell on driver machine NOT yarn cluster? > ignite-spark starts Ignite client node which should have direct access to YARN ignite cluster (have network access, open ports and etc).

Re: Re:Re: Ignite for Spark on YARN Deployment

=8.0GB] 2)spark-shell hangs there 3) I'm running spark-shell on driver machine NOT yarn cluster? Thanks!!! -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6979.html Sent from the Apache Ignite Users mailin

Re: Re:Re: Ignite for Spark on YARN Deployment

.xml") > > > it also hanging, > > Can I miss some files to be changed? thanks!!! > > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6974.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >

Re: Re:Re: Ignite for Spark on YARN Deployment

anging, Can I miss some files to be changed? thanks!!! -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6974.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Re:Re: Ignite for Spark on YARN Deployment

pache.spark.deploy.SparkSubmit$.submit( > SparkSubmit.scala:206) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit. > scala:121) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.net.MalformedURLException: no protocol: > /ignite/release16/apache-ignite-fabric-1.6.0-bin/config/default-config.xml > at java.net.URL.(URL.java:585) > at java.net.URL.(URL.java:482) > at java.net.URL.(URL.java:431) > at > org.apache.ignite.internal.util.IgniteUtils.resolveSpringUrl(IgniteUtils. > java:3571) > ... 58 more > > > Thanks > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6971.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >

Re: Re:Re: Ignite for Spark on YARN Deployment

at java.net.URL.(URL.java:431) at org.apache.ignite.internal.util.IgniteUtils.resolveSpringUrl(IgniteUtils.java:3571) ... 58 more Thanks -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6971.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Re:Re: Ignite for Spark on YARN Deployment

spark:1.6.0,org.apache.ignite:ignite-spring:1.6.0 > > 2、SQL context available as sqlContext. > > scala> import org.apache.ignite.spark._ > import org.apache.ignite.spark._ > > scala> import org.apache.ignite.configuration._ > import org.apache.ignite.configuration._ > > scala> val ic = new IgniteContext[Integer, Integer](sc, () => new > IgniteConfiguration()) > > > > > the spark-shell hang here > > Thanks again > > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6961.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >

Re: Ignite for Spark on YARN Deployment

-- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6963.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Re:Re: Ignite for Spark on YARN Deployment

ation._ scala> val ic = new IgniteContext[Integer, Integer](sc, () => new IgniteConfiguration()) the spark-shell hang here Thanks again -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6961.html

Re:Re: Ignite for Spark on YARN Deployment

... 12 more 16/08/10 22:48:34 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm1 Can anyone help me on this issue? I'm vary urgent for this issues, thanks again!!!1 -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6941.html S

Re: Ignite for Spark on YARN Deployment

ethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod( > RetryInvocationHandler.java:187) > at > org.apache.hadoop.io

Re: yarn deployment

rm? > > -Val > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/yarn-deployment-tp6843p6939.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >

Ignite for Spark on YARN Deployment

http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tt6910.html

Re: Ignite for Spark on YARN Deployment

this issues, thanks again!!!1 -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6941.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: yarn deployment

To my knowledge you can upload the ZIP file and provide the path to it. No need to unzip it manually. Nikolai, can you please confirm? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/yarn-deployment-tp6843p6939.html Sent from the Apache Ignite Users

Re: yarn deployment

I downloaded the binary from here https://ignite.apache.org/download.cgi Do I have to copy the entire unzipped folder to HDFS and provide the root folder location of that? Thank you! -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/yarn-deployment-tp6843p6938

Re: Ignite for Spark on YARN Deployment

his issues? thanks! Can you attach the log files from the server nodes? Are they discovering each other? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6935.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: yarn deployment

Hi, YARN could not download Ignite build. It's might be related with configuration your access to internet. You can configure IGNITE_PATH property. In this case ignite yarn will take a apache ignite build from hdfs [1]. [1] http://apacheignite.gridgain.org/v1.6/docs/yarn-deployment On Tue

Re: yarn deployment

bump... -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/yarn-deployment-tp6843p6885.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

yarn deployment

I am trying to follow instructions here https://apacheignite.readme.io/docs/yarn-deployment for version 1.7.0 I get the following error when I try to run hadoop jar ignite-yarn-.jar Exception in thread "main" java.lang.RuntimeException: Got unexpected response code. Response

Re: Ignite for Spark on YARN Deployment

Hi Hongmei Zong, Could you use the following configuration for Ignite [1]? You need to use the same configuration file for YARN integration and IgniteContext. The configuration uses TcpDiscoveryMulticastIpFinder ip finder which will be more acceptable for your case. You can get more details about

Re: Ignite for Spark on YARN Deployment

Hi Nikolai, I ran the command to start the Spark-shell and Spark started successfully Then I import two classes and create a new IgniteContext, I got the error as posted below. Any good suggestions??? Thank you very much! Hongmei First run: /usr/bin/spark-shell --jars /u/hongmei/apache-igni

Re: Ignite for Spark on YARN Deployment

Hi Nikolay, Thank you very much for your solution. I think the ignite started successfully with your updated version of configuration file!!! :) I think so far ignite running ok on YARN, right? Next I will try to Test ignite from Spark-shell. Should I follow the link below: https://apacheigni

Re: Ignite for Spark on YARN Deployment

Could you download a configuration file ( https://gist.github.com/ntikhonov/d2a2ede2faca7b533dc643e0da475959) and put it to /user/hongmei/ignite/config/ignite-default-config.xml? Also change IGNITE_XML_CONFIG from /user/hongmei/ignite/config/ to /user/hongmei/ignite/config/ignite-default-config.xml

Re: Ignite for Spark on YARN Deployment

Hi Nikolai, Here is the log of one of the container: Logs for container_e24_1464374946035_32114_01_14 Showing 4096 bytes. Click here for full log SpringHelperImp

Re: Ignite for Spark on YARN Deployment

Hi Hongmei Zong! Could you show logs from other containers (container_e24_1464374946035_ 29722_01_15) which was completed? On Thu, Jun 9, 2016 at 6:29 PM, Hongmei Zong wrote: > Hi Nikolay, > > After I changed the value of IGNITE_XML_CONFIG=/user/hongmei/ignite/config/ > (a HDFS path). Igni

Re: Ignite for Spark on YARN Deployment

Hi Nikolay, After I changed the value of IGNITE_XML_CONFIG=/user/hongmei/ignite/config/ (a HDFS path). Ignite RARN is running now. I use the Hadoop UI console to check the log of the application, the attached is the *stderr *log information about containers: It looks like that the containers are

Re: Ignite for Spark on YARN Deployment

Hi Nikolay, I tried to comment the line IGNITE_XML_CONFIG= ,but the Ignite YARN failed. [hongmei@c5hdpe001 config]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite

Re: Ignite for Spark on YARN Deployment

Hi Nikolay, I tried to comment the line IGNITE_XML_CONFIG= ,but the Ignite YARN failed. [hongmei@c5hdpe001 config]$ hadoop jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar /u/hongmei/apache-ignite

Re: Ignite for Spark on YARN Deployment

Thank you very much Nikolay, I will give it a try! :) Hongmei On Thu, Jun 9, 2016 at 10:21 AM, Nikolay Tikhonov wrote: > You set wrong value to IGNITE_XML_CONFIG property. The property should > contains path to ignite configuration file. For example > > IGNITE_XML_CONFIG=/u/hongmei/apache-ignit

Re: Ignite for Spark on YARN Deployment

You set wrong value to IGNITE_XML_CONFIG property. The property should contains path to ignite configuration file. For example IGNITE_XML_CONFIG=/u/hongmei/apache-ignite/config/default-config.xml I think you can comment this line in property file and ignite will start with default configuration.

Re: Ignite for Spark on YARN Deployment

Hi nikolai, Thank you very much for prompt reply! I did not find the ignite-config.xml file under my ignite home directory( /u/hongmei/apache-ignite/ ). I find a "default-config.xml" at the path: /u/hongmei/apache-ignite/config/default-config.xml http://www.springframework.org/schema/be

Re: Ignite for Spark on YARN Deployment

It seems that your ignite configuration invalid. Could you share ignite-config.xml and /u/hongmei/apache-ignite/config/cluster.properties? On Thu, Jun 9, 2016 at 4:35 PM, Hongmei Zong wrote: > Thank you very much Nikolai ! > > I found another issue regarding my Ignite YARN Integration: > > I ru

Re: Ignite for Spark on YARN Deployment

Thank you very much Nikolai ! I found another issue regarding my Ignite YARN Integration: I run Ignite YARN application on one client server machine 'c5hdpe001', screenshot as the following: Then I log onto one of the container node and the log is as the following: [hongmei@c5hdpe001 apache-ign

Re: Ignite for Spark on YARN Deployment

> > *your_address1:47500..47510,your_address2:47500..47510 > and your_address3:47500..47510 are the YARN master_host address, right?* > No, this addresses hosts on which deploy YARN cluster. For example, you have YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In this case you will

Re: Ignite for Spark on YARN Deployment

der(ipFinder); > > cfg.setDiscoverySpi(spi); > > On Wed, Jun 8, 2016 at 6:31 PM, Hongmei Zong <mailto:hong...@kayak.com>> wrote: > Hi Denis, > > I tried testing Ignite as the following steps: > > Background information: > 1. Our Spark is running on YARN d

Re: Ignite for Spark on YARN Deployment

our_address2:47500..47510", "your_address3:47500..47510")); spi.setIpFinder(ipFinder); cfg.setDiscoverySpi(spi); On Wed, Jun 8, 2016 at 6:31 PM, Hongmei Zong wrote: > Hi Denis, > > I tried testing Ignite as the following steps: > > Background information: > 1. Our S

Re: Ignite for Spark on YARN Deployment

Hi Denis, I tried testing Ignite as the following steps: Background information: 1. Our Spark is running on YARN deployment; There are three Master hosts and many Worker nodes and three client nodes in the Spark clusters. 2. I installed Ignite on one of the client node and can launch Ignite

Re: Ignite for Spark on YARN Deployment

states can be used for later jobs. For Shared > Deployment, > > the document only offer two ways to deploy Ignite cluster. First is the > > standalone deployment, second is MESOS deployment. But Our Spark clusters > > are running on YARN. My question is: is it possible to run

Re: Ignite for Spark on YARN Deployment

offer two ways to deploy Ignite cluster. First is the > standalone deployment, second is MESOS deployment. But Our Spark clusters > are running on YARN. My question is: is it possible to run Ignite for Spark > on YARN deployment??? > > I downloaded and installed Ignite on my machine

Ignite for Spark on YARN Deployment

Our Spark clusters are running on YARN. My question is: is it possible to run Ignite for Spark on YARN deployment??? I downloaded and installed Ignite on my machine. Next, I referenced the link below for YARN Deployment. http://apacheignite.gridgain.org/docs/yarn-deployment I created the cluster.