You might want to double check your Hadoop config files. From the stack trace it looks like this is happening when simply trying to load configuration (XML files). Make sure they're well formed.
On Thu, May 7, 2020 at 6:12 AM Hrishikesh Mishra <sd.hri...@gmail.com> wrote: > Hi > > I am getting out of memory error in worker log in streaming jobs in every > couple of hours. After this worker dies. There is no shuffle, no > aggression, no. caching in job, its just a transformation. > I'm not able to identify where is the problem, driver or executor. And why > worker getting dead after the OOM streaming job should die. Am I missing > something. > > Driver Memory: 2g > Executor memory: 4g > > Spark Version: 2.4 > Kafka Direct Stream > Spark Standalone Cluster. > > > 20/05/06 12:52:20 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(root); groups > with view permissions: Set(); users with modify permissions: Set(root); > groups with modify permissions: Set() > > 20/05/06 12:53:03 ERROR SparkUncaughtExceptionHandler: Uncaught exception > in thread Thread[ExecutorRunner for app-20200506124717-10226/0,5,main] > > java.lang.OutOfMemoryError: Java heap space > > at org.apache.xerces.util.XMLStringBuffer.append(Unknown Source) > > at org.apache.xerces.impl.XMLEntityScanner.scanData(Unknown Source) > > at org.apache.xerces.impl.XMLScanner.scanComment(Unknown Source) > > at > org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanComment(Unknown > Source) > > at > org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown > Source) > > at > org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown > Source) > > at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) > > at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) > > at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) > > at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) > > at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) > > at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150) > > at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2480) > > at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2468) > > at > org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2539) > > at > org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2492) > > at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2405) > > at org.apache.hadoop.conf.Configuration.set(Configuration.java:1143) > > at org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) > > at > org.apache.spark.deploy.SparkHadoopUtil$.org$apache$spark$deploy$SparkHadoopUtil$$appendS3AndSparkHadoopConfigurations(SparkHadoopUtil.scala:464) > > at > org.apache.spark.deploy.SparkHadoopUtil$.newConfiguration(SparkHadoopUtil.scala:436) > > at > org.apache.spark.deploy.SparkHadoopUtil.newConfiguration(SparkHadoopUtil.scala:114) > > at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:114) > > at org.apache.spark.deploy.worker.ExecutorRunner.org > $apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:149) > > at > org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73) > > 20/05/06 12:53:38 INFO DriverRunner: Worker shutting down, killing driver > driver-20200505181719-1187 > > 20/05/06 12:53:38 INFO DriverRunner: Killing driver process! > > > > > Regards > Hrishi >
