[
https://issues.apache.org/jira/browse/FLINK-31092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17695530#comment-17695530
]
Shengkai Fang edited comment on FLINK-31092 at 3/2/23 8:07 AM:
---------------------------------------------------------------
Yes, Gateway will try to cancel the execution of the running operation by
explicitly interrupting the thread in
[Operation:377|https://github.com/apache/flink/blob/9201f1e3684b130c3d665114f28208f248848b46/flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/service/operation/OperationManager.java#L377]
and wait all task finish. However, the Gateway seems close the resource before
all task finishes. This confuses me a lot. BTW, I think we can propose a quick
fix like this to prevent endless loading here. WDYT [~mapohl] ,[~Leonard] ?
{code:java}
static <T> List<LoadResult<T>> load(Class<T> clazz, ClassLoader classLoader) {
List<LoadResult<T>> loadResults = new ArrayList<>();
Iterator<T> serviceLoaderIterator = ServiceLoader.load(clazz,
classLoader).iterator();
while (true) {
try {
T next = serviceLoaderIterator.next();
loadResults.add(new LoadResult<>(next));
} catch (NoSuchElementException e) {
break;
} catch (Throwable t) {
// check whether the throwable is as expected
if (!clazz.isInstance(t)) {
throw t;
}
loadResults.add(new LoadResult<>(t));
}
}
return loadResults;
}
{code}
was (Author: fsk119):
Yes, Gateway will try to cancel the execution of the running operation by
explicitly interrupting the thread in
[Operation:377|https://github.com/apache/flink/blob/9201f1e3684b130c3d665114f28208f248848b46/flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/service/operation/OperationManager.java#L377]
and wait all task finish. However, the Gateway seems close the resource before
all task finishes. This confuses me a lot. BTW, I think we can propose a quick
fix like this to prevent endless loading here.
{code:java}
static <T> List<LoadResult<T>> load(Class<T> clazz, ClassLoader classLoader) {
List<LoadResult<T>> loadResults = new ArrayList<>();
Iterator<T> serviceLoaderIterator = ServiceLoader.load(clazz,
classLoader).iterator();
while (true) {
try {
T next = serviceLoaderIterator.next();
loadResults.add(new LoadResult<>(next));
} catch (NoSuchElementException e) {
break;
} catch (Throwable t) {
// check whether the throwable is as expected
if (!clazz.isInstance(t)) {
throw t;
}
loadResults.add(new LoadResult<>(t));
}
}
return loadResults;
}
{code}
> Hive ITCases fail with OutOfMemoryError
> ---------------------------------------
>
> Key: FLINK-31092
> URL: https://issues.apache.org/jira/browse/FLINK-31092
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
> Affects Versions: 1.17.0
> Reporter: Matthias Pohl
> Assignee: luoyuxia
> Priority: Blocker
> Labels: test-stability
> Attachments:
> -__w-2-s-flink-connectors-flink-connector-hive-target-surefire-reports-2023-02-15T05-01-18_982-jvmRun4.dump,
> VisualVM-FLINK-31092.png
>
>
> We're experiencing a OutOfMemoryError where the heap space reaches the upper
> limit:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46161&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=995c650b-6573-581c-9ce6-7ad4cc038461&l=23142
> {code}
> Feb 15 05:05:14 [INFO] Running
> org.apache.flink.table.catalog.hive.HiveCatalogITCase
> Feb 15 05:05:17 [INFO] java.lang.OutOfMemoryError: Java heap space
> Feb 15 05:05:17 [INFO] Dumping heap to java_pid9669.hprof ...
> Feb 15 05:05:28 [INFO] Heap dump file created [1957090051 bytes in 11.718
> secs]
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.maven.surefire.booter.ForkedBooter.cancelPingScheduler(ForkedBooter.java:209)
> at
> org.apache.maven.surefire.booter.ForkedBooter.acknowledgedExit(ForkedBooter.java:419)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:186)
> at
> org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
