Re: Calling Ignite from Spark leads to "Failed to start SPI: TcpDiscoverySpi"

Антон Wed, 22 Sep 2021 02:52:20 -0700

Hi,

also if it could be helpful here is output for spark-shell (full output in attachment):

$ spark-shell --jars=/opt/apache-ignite-2.10.0-bin/libs/ignite-spring/*.jar,/opt/apache-ignite-2.10.0-bin/libs/optional/ignite-spark/ignite-*.jar,/opt/apache-ignite-2.10.0-bin/libs/*.jar,/opt/apache-ignite-2.10.0-bin/libs/ignite-indexing/*.jar,/opt/apache-ignite-2.10.0-bin/libs/optional/ignite-log4j/*.jar

scala> val igniteSession = IgniteSparkSession.builder().appName("Spark Ignite example").igniteConfig("/opt/apache-ignite-2.10.0-bin/config.xml").getOrCreate();

21/09/21 23:42:03 WARN IgniteSparkSession$IgniteBuilder: Using an existing SparkSession; some configuration may not take effect.

21/09/21 23:42:04 WARN G: Ignite work directory is not provided, automatically resolved to: /opt/apache-ignite-2.10.0-bin/work

21/09/21 23:42:04 WARN IgniteKernal: Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed environments.

21/09/21 23:42:05 WARN TcpCommunicationSpi: Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides.

21/09/21 23:42:05 WARN NoopCheckpointSpi: Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation)

21/09/21 23:42:05 WARN GridCollisionManager: Collision resolution is disabled (all jobs will be activated upon arrival).

21/09/21 23:42:06 WARN GridCacheProcessor: Persistent Store is not supported on client nodes (Persistent Store's configuration will be ignored).

21/09/21 23:42:06 WARN IgniteCacheDatabaseSharedManager: DataRegionConfiguration.maxWalArchiveSize instead DataRegionConfiguration.walHistorySize would be used for removing old archive wal files

21/09/21 23:42:06 WARN IgniteH2Indexing: Serialization of Java objects in H2 was enabled.

21/09/21 23:42:07 ERROR IgniteKernal: Failed to start manager: GridManagerAdapter [enabled=true, name=o.a.i.i.managers.discovery.GridDiscoveryManager]

class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@34a82874], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, soLinger=0, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false]

at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:281)

at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:981)

at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1968)

at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1324)

at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2112)

at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1758)

at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1143)

at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:663)

...

21.09.2021, 16:45, "Антон" <anton...@yandex.ru>:

Hi. I'm trying to work with Ignite using Spark. I'm starting session with:
pyspark --jars /opt/apache-ignite-2.10.0-bin/libs/ignite-spring/*.jar,/opt/apache-ignite-2.10.0-bin/libs/optional/ignite-spark/ignite-*.jar,/opt/apache-ignite-2.10.0-bin/libs/*.jar,/opt/apache-ignite-2.10.0-bin/libs/ignite-indexing,/opt/apache-ignite-2.10.0-bin/libs/optional/ignite-log4j/*.jar
Then I'm trying to run simple query:
>>> input=spark.read.format("ignite").option("table","city").option("config","/opt/apache-ignite-2.10.0-bin/config.xml").load()
But it ends with:
ERROR IgniteKernal: Failed to start manager: GridManagerAdapter [enabled=true, name=o.a.i.i.managers.discovery.GridDiscoveryManager]
class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@752d80e6], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, soLinger=0, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false]
Full output and config file is in the attachment. How to resolve this problem?

spark-shell 
--jars=/opt/apache-ignite-2.10.0-bin/libs/ignite-spring/*.jar,/opt/apache-ignite-2.10.0-bin/libs/optional/ignite-spark/ignite-*.jar,/opt/apache-ignite-2.10.0-bin/libs/*.jar,/opt/apache-ignite-2.10.0-bin/libs/ignite-indexing/*.jar,/opt/apache-ignite-2.10.0-bin/libs/optional/ignite-log4j/*.jar


scala> val igniteSession = IgniteSparkSession.builder().appName("Spark Ignite 
example").igniteConfig("/opt/apache-ignite-2.10.0-bin/config.xml").getOrCreate();
21/09/21 23:42:03 WARN IgniteSparkSession$IgniteBuilder: Using an existing 
SparkSession; some configuration may not take effect.
21/09/21 23:42:04 WARN G: Ignite work directory is not provided, automatically 
resolved to: /opt/apache-ignite-2.10.0-bin/work
21/09/21 23:42:04 WARN IgniteKernal: Please set system property 
'-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed 
environments.
21/09/21 23:42:05 WARN TcpCommunicationSpi: Message queue limit is set to 0 
which may lead to potential OOMEs when running cache operations in FULL_ASYNC 
or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides.
21/09/21 23:42:05 WARN NoopCheckpointSpi: Checkpoints are disabled (to enable 
configure any GridCheckpointSpi implementation)
21/09/21 23:42:05 WARN GridCollisionManager: Collision resolution is disabled 
(all jobs will be activated upon arrival).
21/09/21 23:42:06 WARN GridCacheProcessor: Persistent Store is not supported on 
client nodes (Persistent Store's configuration will be ignored).
21/09/21 23:42:06 WARN IgniteCacheDatabaseSharedManager: 
DataRegionConfiguration.maxWalArchiveSize instead 
DataRegionConfiguration.walHistorySize would be used for removing old archive 
wal files
21/09/21 23:42:06 WARN IgniteH2Indexing: Serialization of Java objects in H2 
was enabled.
21/09/21 23:42:07 ERROR IgniteKernal: Failed to start manager: 
GridManagerAdapter [enabled=true, 
name=o.a.i.i.managers.discovery.GridDiscoveryManager]
class org.apache.ignite.IgniteCheckedException: Failed to start SPI: 
TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, 
marsh=JdkMarshaller 
[clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@34a82874], 
reconCnt=10, reconDelay=2000, maxAckTimeout=600000, soLinger=0, 
forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, 
skipAddrsRandomization=false]
        at 
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:281)
        at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:981)
        at 
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1968)
        at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1324)
        at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2112)
        at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1758)
        at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1143)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:663)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:604)
        at org.apache.ignite.Ignition.getOrStart(Ignition.java:419)
        at org.apache.ignite.spark.IgniteContext.ignite(IgniteContext.scala:150)
        at org.apache.ignite.spark.IgniteContext.<init>(IgniteContext.scala:63)
        at org.apache.ignite.spark.IgniteContext.<init>(IgniteContext.scala:99)
        at 
org.apache.spark.sql.ignite.IgniteSparkSession$IgniteBuilder.getOrCreate(IgniteSparkSession.scala:254)
        at $line27.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:29)
        at $line27.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:34)
        at $line27.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:36)
        at $line27.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:38)
        at $line27.$read$$iw$$iw$$iw$$iw.<init>(<console>:40)
        at $line27.$read$$iw$$iw$$iw.<init>(<console>:42)
        at $line27.$read$$iw$$iw.<init>(<console>:44)
        at $line27.$read$$iw.<init>(<console>:46)
        at $line27.$read.<init>(<console>:48)
        at $line27.$read$.<init>(<console>:52)
        at $line27.$read$.<clinit>(<console>)
        at $line27.$eval$.$print$lzycompute(<console>:7)
        at $line27.$eval$.$print(<console>:6)
        at $line27.$eval.$print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:793)
        at 
scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1054)
        at 
scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:645)
        at 
scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:644)
        at 
scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
        at 
scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
        at 
scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:644)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:576)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:572)
        at 
scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:819)
        at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:691)
        at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:404)
        at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:425)
        at 
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:244)
        at 
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:141)
        at 
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:141)
        at 
scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:141)
        at org.apache.spark.repl.Main$.doMain(Main.scala:76)
        at org.apache.spark.repl.Main$.main(Main.scala:56)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
        at 
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to 
deserialize object with given class loader: 
sun.misc.Launcher$AppClassLoader@33909752
        at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:2052)
        at 
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1991)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at 
org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:317)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
21/09/21 23:42:07 ERROR IgniteKernal: Got exception while starting (will 
rollback startup routine).
class org.apache.ignite.IgniteCheckedException: Failed to start manager: 
GridManagerAdapter [enabled=true, 
name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
        at 
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1973)
        at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1324)
        at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2112)
        at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1758)
        at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1143)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:663)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:604)
        at org.apache.ignite.Ignition.getOrStart(Ignition.java:419)
        at org.apache.ignite.spark.IgniteContext.ignite(IgniteContext.scala:150)
        at org.apache.ignite.spark.IgniteContext.<init>(IgniteContext.scala:63)
        at org.apache.ignite.spark.IgniteContext.<init>(IgniteContext.scala:99)
        at 
org.apache.spark.sql.ignite.IgniteSparkSession$IgniteBuilder.getOrCreate(IgniteSparkSession.scala:254)
        at $line27.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:29)
        at $line27.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:34)
        at $line27.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:36)
        at $line27.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:38)
        at $line27.$read$$iw$$iw$$iw$$iw.<init>(<console>:40)
        at $line27.$read$$iw$$iw$$iw.<init>(<console>:42)
        at $line27.$read$$iw$$iw.<init>(<console>:44)
        at $line27.$read$$iw.<init>(<console>:46)
        at $line27.$read.<init>(<console>:48)
        at $line27.$read$.<init>(<console>:52)
        at $line27.$read$.<clinit>(<console>)
        at $line27.$eval$.$print$lzycompute(<console>:7)
        at $line27.$eval$.$print(<console>:6)
        at $line27.$eval.$print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:793)
        at 
scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1054)
        at 
scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:645)
        at 
scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:644)
        at 
scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
        at 
scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
        at 
scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:644)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:576)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:572)
        at 
scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:819)
        at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:691)
        at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:404)
        at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:425)
        at 
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:244)
        at 
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:141)
        at 
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:141)
        at 
scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:141)
        at org.apache.spark.repl.Main$.doMain(Main.scala:76)
        at org.apache.spark.repl.Main$.main(Main.scala:56)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
        at 
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: 
TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, 
marsh=JdkMarshaller 
[clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@34a82874], 
reconCnt=10, reconDelay=2000, maxAckTimeout=600000, soLinger=0, 
forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, 
skipAddrsRandomization=false]
        at 
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:281)
        at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:981)
        at 
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1968)
        ... 60 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to 
deserialize object with given class loader: 
sun.misc.Launcher$AppClassLoader@33909752
        at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:2052)
        at 
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1991)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at 
org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:317)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
21/09/21 23:42:07 ERROR IgniteContext: Failed to start Ignite.
class org.apache.ignite.IgniteException: Failed to start manager: 
GridManagerAdapter [enabled=true, 
name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
        at 
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1098)
        at org.apache.ignite.Ignition.getOrStart(Ignition.java:422)
        at org.apache.ignite.spark.IgniteContext.ignite(IgniteContext.scala:150)
        at org.apache.ignite.spark.IgniteContext.<init>(IgniteContext.scala:63)
        at org.apache.ignite.spark.IgniteContext.<init>(IgniteContext.scala:99)
        at 
org.apache.spark.sql.ignite.IgniteSparkSession$IgniteBuilder.getOrCreate(IgniteSparkSession.scala:254)
        at $line27.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:29)
        at $line27.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:34)
        at $line27.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:36)
        at $line27.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:38)
        at $line27.$read$$iw$$iw$$iw$$iw.<init>(<console>:40)
        at $line27.$read$$iw$$iw$$iw.<init>(<console>:42)
        at $line27.$read$$iw$$iw.<init>(<console>:44)
        at $line27.$read$$iw.<init>(<console>:46)
        at $line27.$read.<init>(<console>:48)
        at $line27.$read$.<init>(<console>:52)
        at $line27.$read$.<clinit>(<console>)
        at $line27.$eval$.$print$lzycompute(<console>:7)
        at $line27.$eval$.$print(<console>:6)
        at $line27.$eval.$print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:793)
        at 
scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1054)
        at 
scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:645)
        at 
scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:644)
        at 
scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
        at 
scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
        at 
scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:644)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:576)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:572)
        at 
scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:819)
        at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:691)
        at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:404)
        at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:425)
        at 
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:244)
        at 
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:141)
        at 
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:141)
        at 
scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:141)
        at org.apache.spark.repl.Main$.doMain(Main.scala:76)
        at org.apache.spark.repl.Main$.main(Main.scala:56)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
        at 
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start 
manager: GridManagerAdapter [enabled=true, 
name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
        at 
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1973)
        at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1324)
        at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2112)
        at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1758)
        at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1143)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:663)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:604)
        at org.apache.ignite.Ignition.getOrStart(Ignition.java:419)
        ... 53 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: 
TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, 
marsh=JdkMarshaller 
[clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@34a82874], 
reconCnt=10, reconDelay=2000, maxAckTimeout=600000, soLinger=0, 
forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, 
skipAddrsRandomization=false]
        at 
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:281)
        at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:981)
        at 
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1968)
        ... 60 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to 
deserialize object with given class loader: 
sun.misc.Launcher$AppClassLoader@33909752
        at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:2052)
        at 
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1991)
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at 
org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:317)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
org.apache.ignite.IgniteException: Failed to start manager: GridManagerAdapter 
[enabled=true, 
name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
  at 
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1098)
  at org.apache.ignite.Ignition.getOrStart(Ignition.java:422)
  at org.apache.ignite.spark.IgniteContext.ignite(IgniteContext.scala:150)
  at org.apache.ignite.spark.IgniteContext.<init>(IgniteContext.scala:63)
  at org.apache.ignite.spark.IgniteContext.<init>(IgniteContext.scala:99)
  at 
org.apache.spark.sql.ignite.IgniteSparkSession$IgniteBuilder.getOrCreate(IgniteSparkSession.scala:254)
  ... 49 elided
Caused by: org.apache.ignite.IgniteCheckedException: Failed to start manager: 
GridManagerAdapter [enabled=true, 
name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
  at 
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1973)
  at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1324)
  at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2112)
  at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1758)
  at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1143)
  at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:663)
  at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:604)
  at org.apache.ignite.Ignition.getOrStart(Ignition.java:419)
  ... 53 more
Caused by: org.apache.ignite.IgniteCheckedException: Failed to start SPI: 
TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, 
marsh=JdkMarshaller 
[clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@34a82874], 
reconCnt=10, reconDelay=2000, maxAckTimeout=600000, soLinger=0, 
forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, 
skipAddrsRandomization=false]
  at 
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:281)
  at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:981)
  at 
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1968)
  ... 60 more
Caused by: org.apache.ignite.spi.IgniteSpiException: Failed to deserialize 
object with given class loader: sun.misc.Launcher$AppClassLoader@33909752
  at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:2052)
  at 
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1991)
  at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
  at org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:317)
  at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)

Re: Calling Ignite from Spark leads to "Failed to start SPI: TcpDiscoverySpi"

Reply via email to