Hi All,
I have written simple java code that read data using Hive and transform using
Table API (Blink Planner) and Flink 1.11.0 on HDP cluster. I am encountering
"java.lang.Error: Failed to find GC Cleaner among available providers” error.
Full error stack is at end of the email.
Do anyone encounter the same issue or any solutions/suggestions ?
Cluster Config:
* Hadoop Version: 2.7.3
* Java Version: 1.8.0_40
* Flink 1.11.0 ( built from source)
* Hive 1.2.1
Thanks
Murali Pusala
Caused by: java.lang.Error: Failed to find GC Cleaner among available
providers: [Legacy (before Java 9) cleaner provider, New Java 9+ cleaner
provider]
at
org.apache.flink.util.JavaGcCleanerWrapper.findGcCleanerManager(JavaGcCleanerWrapper.java:149)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper.<clinit>(JavaGcCleanerWrapper.java:56)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.core.memory.MemoryUtils.createMemoryGcCleaner(MemoryUtils.java:111)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.core.memory.MemorySegmentFactory.allocateOffHeapUnsafeMemory(MemorySegmentFactory.java:175)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.runtime.memory.MemoryManager.lambda$allocatePages$0(MemoryManager.java:237)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.runtime.memory.MemoryManager$$Lambda$188/978816879.apply(Unknown
Source) ~[?:?]
at
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
~[?:1.8.0_40]
at
org.apache.flink.runtime.memory.MemoryManager.allocatePages(MemoryManager.java:233)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:82)
~[flink-table-uber-blink_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.runtime.io.disk.SimpleCollectingOutputView.<init>(SimpleCollectingOutputView.java:49)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.table.runtime.operators.aggregate.BytesHashMap$RecordArea.<init>(BytesHashMap.java:522)
~[flink-table-uber-blink_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.<init>(BytesHashMap.java:184)
~[flink-table-uber-blink_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.<init>(BytesHashMap.java:148)
~[flink-table-uber-blink_2.11-1.11.0.jar:1.11.0]
at LocalHashAggregateWithKeys$100.open(Unknown Source) ~[?:?]
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:291)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:473)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask$$Lambda$176/1800425807.run(Unknown
Source) ~[?:?]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:469)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_40]
Caused by: org.apache.flink.util.FlinkRuntimeException: Legacy (before Java 9)
cleaner provider: Failed to find Reference#tryHandlePending method
at
org.apache.flink.util.JavaGcCleanerWrapper$ReflectionUtils.findMethod(JavaGcCleanerWrapper.java:398)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper$ReflectionUtils.access$1300(JavaGcCleanerWrapper.java:376)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper$PendingCleanersRunnerProvider.createPendingCleanersRunner(JavaGcCleanerWrapper.java:326)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper$PendingCleanersRunnerProvider.access$800(JavaGcCleanerWrapper.java:303)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper$CleanerProvider.createCleanerManager(JavaGcCleanerWrapper.java:180)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper$CleanerProvider.access$400(JavaGcCleanerWrapper.java:162)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper.findGcCleanerManager(JavaGcCleanerWrapper.java:140)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper.<clinit>(JavaGcCleanerWrapper.java:56)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.core.memory.MemoryUtils.createMemoryGcCleaner(MemoryUtils.java:111)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.core.memory.MemorySegmentFactory.allocateOffHeapUnsafeMemory(MemorySegmentFactory.java:175)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.runtime.memory.MemoryManager.lambda$allocatePages$0(MemoryManager.java:237)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.runtime.memory.MemoryManager$$Lambda$188/978816879.apply(Unknown
Source) ~[?:?]
at
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
~[?:1.8.0_40]
at
org.apache.flink.runtime.memory.MemoryManager.allocatePages(MemoryManager.java:233)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:82)
~[flink-table-uber-blink_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.runtime.io.disk.SimpleCollectingOutputView.<init>(SimpleCollectingOutputView.java:49)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.table.runtime.operators.aggregate.BytesHashMap$RecordArea.<init>(BytesHashMap.java:522)
~[flink-table-uber-blink_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.<init>(BytesHashMap.java:184)
~[flink-table-uber-blink_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.<init>(BytesHashMap.java:148)
~[flink-table-uber-blink_2.11-1.11.0.jar:1.11.0]
at LocalHashAggregateWithKeys$100.open(Unknown Source) ~[?:?]
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:291)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:473)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask$$Lambda$176/1800425807.run(Unknown
Source) ~[?:?]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:469)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_40]
Suppressed: org.apache.flink.util.FlinkRuntimeException: New Java 9+
cleaner provider: Failed to find java class
at
org.apache.flink.util.JavaGcCleanerWrapper$ReflectionUtils.findClass(JavaGcCleanerWrapper.java:388)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper$ReflectionUtils.access$1200(JavaGcCleanerWrapper.java:376)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper$CleanerFactoryProvider.createCleanerFactory(JavaGcCleanerWrapper.java:244)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper$CleanerFactoryProvider.access$700(JavaGcCleanerWrapper.java:217)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper$CleanerProvider.createCleanerManager(JavaGcCleanerWrapper.java:179)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper$CleanerProvider.access$400(JavaGcCleanerWrapper.java:162)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper.findGcCleanerManager(JavaGcCleanerWrapper.java:140)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper.<clinit>(JavaGcCleanerWrapper.java:56)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.core.memory.MemoryUtils.createMemoryGcCleaner(MemoryUtils.java:111)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.core.memory.MemorySegmentFactory.allocateOffHeapUnsafeMemory(MemorySegmentFactory.java:175)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.runtime.memory.MemoryManager.lambda$allocatePages$0(MemoryManager.java:237)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.runtime.memory.MemoryManager$$Lambda$188/978816879.apply(Unknown
Source) ~[?:?]
at
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
~[?:1.8.0_40]
at
org.apache.flink.runtime.memory.MemoryManager.allocatePages(MemoryManager.java:233)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:82)
~[flink-table-uber-blink_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.runtime.io.disk.SimpleCollectingOutputView.<init>(SimpleCollectingOutputView.java:49)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.table.runtime.operators.aggregate.BytesHashMap$RecordArea.<init>(BytesHashMap.java:522)
~[flink-table-uber-blink_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.<init>(BytesHashMap.java:184)
~[flink-table-uber-blink_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.<init>(BytesHashMap.java:148)
~[flink-table-uber-blink_2.11-1.11.0.jar:1.11.0]
at LocalHashAggregateWithKeys$100.open(Unknown Source) ~[?:?]
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:291)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:473)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask$$Lambda$176/1800425807.run(Unknown
Source) ~[?:?]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:469)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_40]
Caused by: java.lang.ClassNotFoundException: java.lang.ref.Cleaner
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
~[?:1.8.0_40]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
~[?:1.8.0_40]
at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[?:1.8.0_40]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
~[?:1.8.0_40]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_40]
at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_40]
at
org.apache.flink.util.JavaGcCleanerWrapper$ReflectionUtils.findClass(JavaGcCleanerWrapper.java:385)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
... 28 more
Caused by: java.lang.NoSuchMethodException:
java.lang.ref.Reference.tryHandlePending(boolean)
at java.lang.Class.getDeclaredMethod(Class.java:2130) ~[?:1.8.0_40]
at
org.apache.flink.util.JavaGcCleanerWrapper$ReflectionUtils.findMethod(JavaGcCleanerWrapper.java:395)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper$ReflectionUtils.access$1300(JavaGcCleanerWrapper.java:376)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper$PendingCleanersRunnerProvider.createPendingCleanersRunner(JavaGcCleanerWrapper.java:326)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper$PendingCleanersRunnerProvider.access$800(JavaGcCleanerWrapper.java:303)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper$CleanerProvider.createCleanerManager(JavaGcCleanerWrapper.java:180)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper$CleanerProvider.access$400(JavaGcCleanerWrapper.java:162)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper.findGcCleanerManager(JavaGcCleanerWrapper.java:140)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.util.JavaGcCleanerWrapper.<clinit>(JavaGcCleanerWrapper.java:56)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.core.memory.MemoryUtils.createMemoryGcCleaner(MemoryUtils.java:111)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.core.memory.MemorySegmentFactory.allocateOffHeapUnsafeMemory(MemorySegmentFactory.java:175)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.runtime.memory.MemoryManager.lambda$allocatePages$0(MemoryManager.java:237)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.runtime.memory.MemoryManager$$Lambda$188/978816879.apply(Unknown
Source) ~[?:?]
at
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
~[?:1.8.0_40]
at
org.apache.flink.runtime.memory.MemoryManager.allocatePages(MemoryManager.java:233)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:82)
~[flink-table-uber-blink_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.runtime.io.disk.SimpleCollectingOutputView.<init>(SimpleCollectingOutputView.java:49)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.table.runtime.operators.aggregate.BytesHashMap$RecordArea.<init>(BytesHashMap.java:522)
~[flink-table-uber-blink_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.<init>(BytesHashMap.java:184)
~[flink-table-uber-blink_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.<init>(BytesHashMap.java:148)
~[flink-table-uber-blink_2.11-1.11.0.jar:1.11.0]
at LocalHashAggregateWithKeys$100.open(Unknown Source) ~[?:?]
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:291)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:473)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask$$Lambda$176/1800425807.run(Unknown
Source) ~[?:?]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:469)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
~[flink-dist_2.11-1.11.0.jar:1.11.0]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_40]
