Hi,
I/m trying to run Spark 1.6.0 on HDP 2.2
Everything was fine until I tried to turn on dynamic allocation.
According to instruction I need to add shuffle service to yarn classpath.
The problem is that HDP 2.2 has jackson 2.2.3 and Spark is using 2.4.4.
So connecting it gives error:
2016-01-11 16:56:51,222 INFO containermanager.AuxServices
(AuxServices.java:addService(72)) - Adding auxiliary service
spark_shuffle, "spark_shuffle"
2016-01-11 16:56:51,439 FATAL nodemanager.NodeManager
(NodeManager.java:initAndStartNodeManager(465)) - Error starting
NodeManager
java.lang.NoSuchMethodError:
com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z
at
com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:457)
at
com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:379)
at
org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.<clinit>(ExternalShuffleBlockResolver.java:57)
at
org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.<init>(ExternalShuffleBlockHandler.java:56)
at
org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:128)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:143)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:237)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:253)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:462)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:509)
What can I do ?
I have following ideas:
1) Compile Spark 1.6.0 with modified pom.xml (change jackson version
to 2.2.3). I'm not sure if this will be working
2) I tried to put shuffle service from different version of Spark.
1.4.1 works on HDP 2.2.
Is it possible to run shuffle service from 1.4.1 with Spark 1.6.0 ?
3) Other ideas ?
Regards,
--
Maciek Bryński
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
