[
https://issues.apache.org/jira/browse/FLINK-26335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17610148#comment-17610148
]
Austin Cawley-Edwards commented on FLINK-26335:
-----------------------------------------------
Hey [~mlnogueira], thanks for the report. Do you know roughly how many metrics
your job was trying to export (# of labels, # of custom metrics, total size
(hundreds of mbs? gbs?)) when this happened? Were you able to resolve it by
giving the job more resources?
> prometheus/metrics - java.lang.OutOfMemoryError: Java heap space
> ----------------------------------------------------------------
>
> Key: FLINK-26335
> URL: https://issues.apache.org/jira/browse/FLINK-26335
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Metrics
> Affects Versions: 1.14.3
> Reporter: Marcelo Nogueira
> Priority: Major
>
>
> {code:java}
> 2022-02-22 14:39:33.782 INFO
> [org.apache.flink.streaming.runtime.operators.sink.AbstractStreamingCommitterHandler]
> - Committing the state for checkpoint 4
> 2022-02-22 14:39:47.367 ERROR
> [org.apache.flink.runtime.util.ClusterUncaughtExceptionHandler] - WARNING:
> Thread 'prometheus-http-1-6' produced an uncaught exception. If you want to
> fail on uncaught exceptions, then configure
> cluster.uncaught-exception-handling accordingly
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Unknown Source) ~[?:?]
> at java.io.ByteArrayOutputStream.grow(Unknown Source) ~[?:?]
> at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source) ~[?:?]
> at java.io.ByteArrayOutputStream.write(Unknown Source) ~[?:?]
> at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[?:?]
> at sun.nio.cs.StreamEncoder.implWrite(Unknown Source) ~[?:?]
> at sun.nio.cs.StreamEncoder.implWrite(Unknown Source) ~[?:?]
> at sun.nio.cs.StreamEncoder.write(Unknown Source) ~[?:?]
> at sun.nio.cs.StreamEncoder.write(Unknown Source) ~[?:?]
> at java.io.OutputStreamWriter.write(Unknown Source) ~[?:?]
> at java.io.Writer.append(Unknown Source) ~[?:?]
> at
> io.prometheus.client.exporter.common.TextFormat.writeEscapedLabelValue(TextFormat.java:88)
> ~[flink-metrics-prometheus-1.14.3.jar:1.14.3]
> at
> io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:42)
> ~[flink-metrics-prometheus-1.14.3.jar:1.14.3]
> at
> io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:60)
> ~[flink-metrics-prometheus-1.14.3.jar:1.14.3]
> at com.sun.net.httpserver.Filter$Chain.doFilter(Unknown Source)
> ~[jdk.httpserver:?]
> at sun.net.httpserver.AuthFilter.doFilter(Unknown Source)
> ~[jdk.httpserver:?]
> at com.sun.net.httpserver.Filter$Chain.doFilter(Unknown Source)
> ~[jdk.httpserver:?]
> at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(Unknown
> Source) ~[jdk.httpserver:?]
> at com.sun.net.httpserver.Filter$Chain.doFilter(Unknown Source)
> ~[jdk.httpserver:?]
> at sun.net.httpserver.ServerImpl$Exchange.run(Unknown Source)
> ~[jdk.httpserver:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> ~[?:?]
> at java.lang.Thread.run(Unknown Source) [?:?]
> 2022-02-22 14:40:32.668 INFO
> [org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer] - Flushing
> new partitions// code placeholder
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
