What could also cause the problem is that the metaspace memory budget is configured too tightly. Here is a pointer to increasing the metaspace size [1].
[1] https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_trouble.html#outofmemoryerror-metaspace Cheers, Till On Mon, Aug 24, 2020 at 1:49 PM Till Rohrmann <trohrm...@apache.org> wrote: > Hi, > > could you share with us the Flink cluster logs? This would help answering > a lot of questions around your setup and the Flink version you are using. > Thanks a lot! > > Cheers, > Till > > On Mon, Aug 24, 2020 at 10:48 AM 耿延杰 <gyj199...@qq.com> wrote: > >> Still failed after every 12 tasks. >> And the exception stack of failed tasks is different. >> >> >> such as the recent failed tasks's exception info: >> Caused by: java.lang.OutOfMemoryError: Metaspace >> at java.lang.ClassLoader.defineClass1(Native >> Method) >> at >> java.lang.ClassLoader.defineClass(ClassLoader.java:757) >> at >> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >> at >> java.net.URLClassLoader.defineClass(URLClassLoader.java:468) >> at >> java.net.URLClassLoader.access$100(URLClassLoader.java:74) >> at >> java.net.URLClassLoader$1.run(URLClassLoader.java:369) >> at >> java.net.URLClassLoader$1.run(URLClassLoader.java:363) >> at >> java.security.AccessController.doPrivileged(Native Method) >> at >> java.net.URLClassLoader.findClass(URLClassLoader.java:362) >> at >> org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:66) >> at >> java.lang.ClassLoader.loadClass(ClassLoader.java:352) >> at >> org.apache.http.impl.client.CloseableHttpClient.determineTarget(CloseableHttpClient.java:93) >> at >> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) >> at >> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) >> at >> ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:614) >> at >> ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:117) >> at >> ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:100) >> at >> ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:95) >> at >> ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:90) >> at >> ru.yandex.clickhouse.ClickHouseConnectionImpl.initTimeZone(ClickHouseConnectionImpl.java:94) >> at >> ru.yandex.clickhouse.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:80) >> at >> ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:55) >> at >> ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:47) >> at >> ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:29) >> at >> java.sql.DriverManager.getConnection(DriverManager.java:664) >> at >> java.sql.DriverManager.getConnection(DriverManager.java:270) >> at org.apache.flink.api.java.io >> .jdbc.AbstractJDBCOutputFormat.establishConnection(AbstractJDBCOutputFormat.java:68) >> at >> com.xxx.clickhouse.ClickHouseJDBCOutputFormat.open(ClickHouseJDBCOutputFormat.java:53) >> at >> org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:205) >> at >> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707) >> at >> org.apache.flink.runtime.taskmanager.Task.run(Task.java:532) >> at java.lang.Thread.run(Thread.java:748) >> >> >> >> >> is different with the exception info in last email. >> >> >> So analyse the dump file is the key. >> >> >> >> >> >> >> ------------------ 原始邮件 ------------------ >> 发件人: >> "耿延杰" >> < >> gyj199...@qq.com>; >> 发送时间: 2020年8月24日(星期一) 下午4:33 >> 收件人: "dev"<dev@flink.apache.org>; >> >> 主题: 回复：OutOfMemoryError: Metaspace on Batch Task When Write into >> Clickhouse >> >> >> >> Additional info: >> >> >> The exception info in Flink Manager Page: >> >> >> Caused by: java.lang.OutOfMemoryError: Metaspace >> at java.lang.ClassLoader.defineClass1(Native Method) >> at >> java.lang.ClassLoader.defineClass(ClassLoader.java:757) >> at >> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >> at >> java.net.URLClassLoader.defineClass(URLClassLoader.java:468) >> at >> java.net.URLClassLoader.access$100(URLClassLoader.java:74) >> at >> java.net.URLClassLoader$1.run(URLClassLoader.java:369) >> at >> java.net.URLClassLoader$1.run(URLClassLoader.java:363) >> at java.security.AccessController.doPrivileged(Native >> Method) >> at >> java.net.URLClassLoader.findClass(URLClassLoader.java:362) >> at >> org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:66) >> at >> java.lang.ClassLoader.loadClass(ClassLoader.java:352) >> at >> org.apache.http.impl.client.CloseableHttpClient.determineTarget(CloseableHttpClient.java:93) >> at >> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) >> at >> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) >> at >> ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:614) >> at >> ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:117) >> at >> ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:100) >> at >> ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:95) >> at >> ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:90) >> at >> ru.yandex.clickhouse.ClickHouseConnectionImpl.initTimeZone(ClickHouseConnectionImpl.java:94) >> at >> ru.yandex.clickhouse.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:80) >> at >> ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:55) >> at >> ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:47) >> at >> ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:29) >> at >> java.sql.DriverManager.getConnection(DriverManager.java:664) >> at >> java.sql.DriverManager.getConnection(DriverManager.java:270) >> at org.apache.flink.api.java.io >> .jdbc.AbstractJDBCOutputFormat.establishConnection(AbstractJDBCOutputFormat.java:68) >> at >> com.xx.xx.xx.ClickHouseJDBCOutputFormat.open(ClickHouseJDBCOutputFormat.java:53) >> at >> org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:205) >> at >> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707) >> at >> org.apache.flink.runtime.taskmanager.Task.run(Task.java:532) >> at java.lang.Thread.run(Thread.java:748) >> >> >> >> >> >> >> ------------------ 原始邮件 ------------------ >> 发件人: >> "耿延杰" >> < >> gyj199...@qq.com>; >> 发送时间: 2020年8月24日(星期一) 下午4:20 >> 收件人: "dev"<dev@flink.apache.org>; >> >> 主题: OutOfMemoryError: Metaspace on Batch Task When Write into >> Clickhouse >> >> >> >> Hi, >> >> >> I catch "OutOfMemoryError: Metaspace" on Batch Task When Write into >> Clickhouse. >> Attached *.java file is my task code. >> >> And I find that, after running 12 tasks, the 13th task will be failed. >> And the exception always is "OutOfMemoryError: Metaspace". see >> "task-failed.png" >> >> >> I conf -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/hprofFile >> and dump the hprof file. >> I analyse this hprof file. And find this error occurs may not caused by >> my user-code. >> So I came here ask for your help. To confirm whether the memory leak >> should be caused by Flink. >> >> >> Attached file "java_pid29294.hprof" is the dump file. >> >> >> Thanks. > >
