Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to ` /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar`?
On Mon, Jun 18, 2018 at 4:02 PM Garvit Sharma <garvit...@gmail.com> wrote: > I don't think I can access core-default as it comes with Hadoop jar > > On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann <trohrm...@apache.org> > wrote: > >> Hmm, could you check whether core-default.xml contains any suspicious >> entries? Apparently xerces:2.9.1 cannot read it. >> >> On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma <garvit...@gmail.com> >> wrote: >> >>> Hi, >>> >>> After putting the following log in my code, I can see that the Xerces >>> version is - Xerces version : Xerces-J 2.9.1 >>> >>> log.info("Xerces version : {}", >>> org.apache.xerces.impl.Version.getVersion()); >>> >>> Also, following is the response of *$* *locate xerces* command on the >>> server - >>> >>> >>> /usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar >>> >>> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar >>> >>> /usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar >>> >>> >>> /usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar >>> >>> /usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar >>> >>> /usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar >>> >>> /usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar >>> >>> /usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar >>> >>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar >>> >>> /usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar >>> >>> >>> /usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar >>> >>> /usr/hdp/2.6.3.0-235/oozie/libserver/xercesImpl-2.10.0.jar >>> >>> /usr/hdp/2.6.3.0-235/oozie/libtools/xercesImpl-2.10.0.jar >>> >>> >>> /usr/hdp/2.6.3.0-235/ranger-admin/ews/webapp/WEB-INF/lib/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.3.0-235/slider/lib/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.3.0-235/spark2/jars/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.3.0-235/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar >>> >>> /usr/hdp/2.6.3.0-235/zookeeper/lib/xercesMinimal-1.9.6.2.jar >>> >>> /usr/hdp/share/hst/hst-common/lib/xercesImpl-2.9.1.jar >>> >>> Now, I can say that the version of xerces are same. >>> >>> >>> So, what is causing this issue if Xerces version is in sync? >>> >>> >>> I am very excited to discover the issue :) >>> >>> >>> Thanks, >>> >>> On Mon, Jun 18, 2018 at 6:27 PM Till Rohrmann <trohrm...@apache.org> >>> wrote: >>> >>>> Could you check which xerces version you have on your classpath? >>>> Apparently, it cannot read core-default.xml as Ted pointed out. This might >>>> be the root cause for the failure. >>>> >>>> Cheers, >>>> Till >>>> >>>> On Mon, Jun 18, 2018 at 1:31 PM Garvit Sharma <garvit...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> Sorry for the confusion, but the yarn is running on Hadoop version 2.7 >>>>> only and hence I am using Flink 1.5 Hadoop 2.7 binary. >>>>> >>>>> Below are the details provided by Yarn version command : >>>>> >>>>> Hadoop 2.7.3.2.6.3.0-235 >>>>> Subversion g...@github.com:hortonworks/hadoop.git -r >>>>> 45bfd33bba8acadfa0e6024c80981c023b28d454 >>>>> Compiled by jenkins on 2017-10-30T02:31Z >>>>> Compiled with protoc 2.5.0 >>>>> From source with checksum cd1a4a466ef450f547c279989f3aa3 >>>>> This command was run using >>>>> /usr/hdp/2.6.3.0-235/hadoop/hadoop-common-2.7.3.2.6.3.0-235.jar >>>>> >>>>> Please let me know if you have found the resolution to my issue :) >>>>> >>>>> Thanks, >>>>> >>>>> >>>>> On Mon, Jun 18, 2018 at 4:50 PM Till Rohrmann <trohrm...@apache.org> >>>>> wrote: >>>>> >>>>>> Which Hadoop version have you installed? It looks as if Flink has >>>>>> been build with Hadoop 2.7 but I see /usr/hdp/2.6.3.0-235 in the class >>>>>> path. If you want to run Flink on Hadoop 2.6, then try to use the Hadoop >>>>>> free Flink binaries or the one built for Hadoop 2.6. >>>>>> >>>>>> Cheers, >>>>>> Till >>>>>> >>>>>> On Mon, Jun 18, 2018 at 10:48 AM Garvit Sharma <garvit...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Ok, I have attached the log file. >>>>>>> >>>>>>> Please check and let me know. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> On Mon, Jun 18, 2018 at 2:07 PM Amit Jain <aj201...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Gravit, >>>>>>>> >>>>>>>> I think Till is interested to know about classpath details present >>>>>>>> at the start of JM and TM logs e.g. following logs provide classpath >>>>>>>> details used by TM in our case. >>>>>>>> >>>>>>>> 2018-06-17 19:01:30,656 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>>>>>> -------------------------------------------------------------------------------- >>>>>>>> 2018-06-17 19:01:30,658 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>>>>>> Starting >>>>>>>> YARN TaskExecutor runner (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 >>>>>>>> @ >>>>>>>> 14:54:44 UTC) >>>>>>>> 2018-06-17 19:01:30,659 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - OS >>>>>>>> current user: yarn >>>>>>>> 2018-06-17 19:01:31,662 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>>>>>> Current >>>>>>>> Hadoop/Kerberos user: hadoop >>>>>>>> 2018-06-17 19:01:31,663 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM: >>>>>>>> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10 >>>>>>>> 2018-06-17 19:01:31,663 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>>>>>> Maximum >>>>>>>> heap size: 6647 MiBytes >>>>>>>> 2018-06-17 19:01:31,663 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>>>>>> JAVA_HOME: /usr/lib/jvm/java-openjdk >>>>>>>> 2018-06-17 19:01:31,664 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>>>>>> Hadoop >>>>>>>> version: 2.8.3 >>>>>>>> 2018-06-17 19:01:31,664 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM >>>>>>>> Options: >>>>>>>> 2018-06-17 19:01:31,665 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>>>>>> -Xms6936m >>>>>>>> 2018-06-17 19:01:31,665 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>>>>>> -Xmx6936m >>>>>>>> 2018-06-17 19:01:31,665 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>>>>>> -XX:MaxDirectMemorySize=4072m >>>>>>>> 2018-06-17 19:01:31,665 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>>>>>> -Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log >>>>>>>> 2018-06-17 19:01:31,665 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>>>>>> -Dlogback.configurationFile=file:./logback.xml >>>>>>>> 2018-06-17 19:01:31,665 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>>>>>> -Dlog4j.configuration=file:./log4j.properties >>>>>>>> 2018-06-17 19:01:31,665 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>>>>>> Program >>>>>>>> Arguments: >>>>>>>> 2018-06-17 19:01:31,665 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>>>>>> --configDir >>>>>>>> 2018-06-17 19:01:31,665 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - . >>>>>>>> *2018-06-17 19:01:31,666 INFO >>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>>>>>> Classpath: >>>>>>>> lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........* >>>>>>>> >>>>>>>> -- >>>>>>>> Thanks, >>>>>>>> Amit >>>>>>>> >>>>>>>> On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <garvit...@gmail.com >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Please refer to my previous mail for complete logs. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann < >>>>>>>>> trohrm...@apache.org> wrote: >>>>>>>>> >>>>>>>>>> Could you also please share the complete log file with us. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Till >>>>>>>>>> >>>>>>>>>> On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <yuzhih...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> The error for core-default.xml is interesting. >>>>>>>>>>> >>>>>>>>>>> Flink doesn't have this file. Probably it came with Yarn. Please >>>>>>>>>>> check the hadoop version Flink was built with versus the hadoop >>>>>>>>>>> version in >>>>>>>>>>> your cluster. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> -------- Original message -------- >>>>>>>>>>> From: Garvit Sharma <garvit...@gmail.com> >>>>>>>>>>> Date: 6/16/18 7:23 AM (GMT-08:00) >>>>>>>>>>> To: trohrm...@apache.org >>>>>>>>>>> Cc: Chesnay Schepler <ches...@apache.org>, user@flink.apache.org >>>>>>>>>>> Subject: Re: Exception while submitting jobs through Yarn >>>>>>>>>>> >>>>>>>>>>> I am not able to figure out, got stuck badly in this since last >>>>>>>>>>> 1 week. Any little help would be appreciated. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2018-06-16 19:25:10,523 DEBUG >>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator >>>>>>>>>>> - Parallelism set: 1 for 8 >>>>>>>>>>> >>>>>>>>>>> 2018-06-16 19:25:10,578 DEBUG >>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator >>>>>>>>>>> - Parallelism set: 1 for 1 >>>>>>>>>>> >>>>>>>>>>> 2018-06-16 19:25:10,588 DEBUG >>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator >>>>>>>>>>> - CONNECTED: KeyGroupStreamPartitioner - 1 -> 8 >>>>>>>>>>> >>>>>>>>>>> 2018-06-16 19:25:10,591 DEBUG >>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator >>>>>>>>>>> - Parallelism set: 1 for 5 >>>>>>>>>>> >>>>>>>>>>> 2018-06-16 19:25:10,597 DEBUG >>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator >>>>>>>>>>> - CONNECTED: KeyGroupStreamPartitioner - 5 -> 8 >>>>>>>>>>> >>>>>>>>>>> 2018-06-16 19:25:10,618 FATAL >>>>>>>>>>> org.apache.hadoop.conf.Configuration - >>>>>>>>>>> error parsing conf core-default.xml >>>>>>>>>>> >>>>>>>>>>> javax.xml.parsers.ParserConfigurationException: Feature ' >>>>>>>>>>> http://apache.org/xml/features/xinclude' is not recognized. >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown >>>>>>>>>>> Source) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.conf.Configuration.get(Configuration.java:1188) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49) >>>>>>>>>>> >>>>>>>>>>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096) >>>>>>>>>>> >>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>>>> >>>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096) >>>>>>>>>>> >>>>>>>>>>> 2018-06-16 19:25:10,620 WARN >>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor >>>>>>>>>>> - Error while getting queue information from YARN: >>>>>>>>>>> null >>>>>>>>>>> >>>>>>>>>>> 2018-06-16 19:25:10,621 DEBUG >>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - >>>>>>>>>>> Error details >>>>>>>>>>> >>>>>>>>>>> java.lang.ExceptionInInitializerError >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096) >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Garvit Sharma >>>>>>>>> github.com/garvitlnmiit/ >>>>>>>>> >>>>>>>>> No Body is a Scholar by birth, its only hard work and strong >>>>>>>>> determination that makes him master. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Garvit Sharma >>>>>>> github.com/garvitlnmiit/ >>>>>>> >>>>>>> No Body is a Scholar by birth, its only hard work and strong >>>>>>> determination that makes him master. >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> >>>>> Garvit Sharma >>>>> github.com/garvitlnmiit/ >>>>> >>>>> No Body is a Scholar by birth, its only hard work and strong >>>>> determination that makes him master. >>>>> >>>> >>> >>> -- >>> >>> Garvit Sharma >>> github.com/garvitlnmiit/ >>> >>> No Body is a Scholar by birth, its only hard work and strong >>> determination that makes him master. >>> >> -- > > Garvit Sharma > github.com/garvitlnmiit/ > > No Body is a Scholar by birth, its only hard work and strong determination > that makes him master. >