Thanks. I successfully compiled Spark 1.6.0 with Jackson 2.2.3 from source.
I'll try to using it. 2016-01-13 11:25 GMT+01:00 Ted Yu <yuzhih...@gmail.com>: > I would suggest trying option #1 first. > > Thanks > >> On Jan 13, 2016, at 2:12 AM, Maciej Bryński <mac...@brynski.pl> wrote: >> >> Hi, >> I/m trying to run Spark 1.6.0 on HDP 2.2 >> Everything was fine until I tried to turn on dynamic allocation. >> According to instruction I need to add shuffle service to yarn classpath. >> The problem is that HDP 2.2 has jackson 2.2.3 and Spark is using 2.4.4. >> So connecting it gives error: >> >> 2016-01-11 16:56:51,222 INFO containermanager.AuxServices >> (AuxServices.java:addService(72)) - Adding auxiliary service >> spark_shuffle, "spark_shuffle" >> 2016-01-11 16:56:51,439 FATAL nodemanager.NodeManager >> (NodeManager.java:initAndStartNodeManager(465)) - Error starting >> NodeManager >> java.lang.NoSuchMethodError: >> com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z >> at >> com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:457) >> at >> com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:379) >> at >> org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.<clinit>(ExternalShuffleBlockResolver.java:57) >> at >> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.<init>(ExternalShuffleBlockHandler.java:56) >> at >> org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:128) >> at >> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:143) >> at >> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >> at >> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:237) >> at >> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >> at >> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) >> at >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:253) >> at >> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >> at >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:462) >> at >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:509) >> >> >> What can I do ? >> I have following ideas: >> 1) Compile Spark 1.6.0 with modified pom.xml (change jackson version >> to 2.2.3). I'm not sure if this will be working >> 2) I tried to put shuffle service from different version of Spark. >> 1.4.1 works on HDP 2.2. >> Is it possible to run shuffle service from 1.4.1 with Spark 1.6.0 ? >> 3) Other ideas ? >> >> Regards, >> -- >> Maciek Bryński >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> For additional commands, e-mail: dev-h...@spark.apache.org >> -- Maciek Bryński --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
