I would suggest trying option #1 first. Thanks
> On Jan 13, 2016, at 2:12 AM, Maciej Bryński <mac...@brynski.pl> wrote: > > Hi, > I/m trying to run Spark 1.6.0 on HDP 2.2 > Everything was fine until I tried to turn on dynamic allocation. > According to instruction I need to add shuffle service to yarn classpath. > The problem is that HDP 2.2 has jackson 2.2.3 and Spark is using 2.4.4. > So connecting it gives error: > > 2016-01-11 16:56:51,222 INFO containermanager.AuxServices > (AuxServices.java:addService(72)) - Adding auxiliary service > spark_shuffle, "spark_shuffle" > 2016-01-11 16:56:51,439 FATAL nodemanager.NodeManager > (NodeManager.java:initAndStartNodeManager(465)) - Error starting > NodeManager > java.lang.NoSuchMethodError: > com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z > at > com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:457) > at > com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:379) > at > org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.<clinit>(ExternalShuffleBlockResolver.java:57) > at > org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.<init>(ExternalShuffleBlockHandler.java:56) > at > org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:128) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:143) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:237) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:253) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:462) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:509) > > > What can I do ? > I have following ideas: > 1) Compile Spark 1.6.0 with modified pom.xml (change jackson version > to 2.2.3). I'm not sure if this will be working > 2) I tried to put shuffle service from different version of Spark. > 1.4.1 works on HDP 2.2. > Is it possible to run shuffle service from 1.4.1 with Spark 1.6.0 ? > 3) Other ideas ? > > Regards, > -- > Maciek Bryński > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
