I am ruuning streaming job. It is running in local,yarn-client mode but
when I am running it in yarn-cluster mode it is failing with PermGen space.
Can any one help me out.
Spark version : 1.5.0
Hadoop: 2.6.0
Java : 1.7
16/10/06 15:08:40 INFO scheduler.JobScheduler: Added jobs for time
1475746720000 ms
16/10/06 15:08:40 INFO scheduler.JobGenerator: Checkpointing graph for
time 1475746720000 ms
16/10/06 15:08:40 INFO streaming.DStreamGraph: Updating checkpoint
data for time 1475746720000 ms
16/10/06 15:08:40 INFO streaming.DStreamGraph: Updated checkpoint data
for time 1475746720000 ms
Exception in thread "JobGenerator" java.lang.OutOfMemoryError: PermGen space
at sun.misc.Unsafe.defineClass(Native Method)
at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
at
sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399)
at
sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395)
at
sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:113)
at
sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:331)
at
java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1376)
at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:493)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at
java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:440)
at
org.apache.spark.streaming.dstream.DStreamCheckpointData$$anonfun$writeObject$1.apply$mcV$sp(DStreamCheckpointData.scala:128)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1160)
at
org.apache.spark.streaming.dstream.DStreamCheckpointData.writeObject(DStreamCheckpointData.scala:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
--
Thanks and Regards,
Saurav Sinha
Contact: 9742879062
