Usually that means the remote task manager, where the slot locates, is lost.
You will need to look into the log of that task manager to find out what's
wrong with it.
Thank you~
Xintong Song
On Fri, Jun 12, 2020 at 4:13 PM Ramya Ramamurthy wrote:
> Yes ... the image was on Heap Committed me
Yes ... the image was on Heap Committed metrics.
And i have not yet faced this issue now, post changing the memory.
I seem to get one more frequent error: org.apache.flink.util.FlinkException:
The assigned slot d9d4db5cc747bcbd374888d97e81945b_0 was removed.
When are we likely to get this ??
Tha
BTW, the image you previously attached cannot be displayed. So I assume you
are talking about the "Heap Committed" displayed on Flink's webui?
Thank you~
Xintong Song
On Fri, Jun 12, 2020 at 2:30 PM Xintong Song wrote:
> Do you still run into the "java.lang.OutOfMemoryError: Java heap space"
Do you still run into the "java.lang.OutOfMemoryError: Java heap space"?
If not, then you don't really need to worry about the committed memory.
It is the maximum that really matters. The committed memory should increase
automatically when it's needed.
Thank you~
Xintong Song
On Fri, Jun 12,
Hi Xintong,
Thanks for the quick response.
I have kept my task manager memory to be 1.5GB. But still seeing the Heap
committed metric to be around 54MB or so. Why does this happen ? Should I
configure any memory fraction configurations here ?
Thanks.
On Fri, Jun 12, 2020 at 10:58 AM Xintong Son
Hi Ramya,
Increasing the memory of your pod will not give you more JVM heap space.
You will need to configure Flink so it launches the JVM process with more
memory.
In Flink 1.7, this could be achieved by configuring 'jobmanager.heap.size'
& 'taskmanager.heap.size' in your 'flink-conf.yaml'. Both
Thanks Till.
Actually, i have around 5GB pods for each TM, and each pod with only one
slot.
But the metrics i have pulled is as below, which is slightly confusing.
It says only ~50MB of Heap is committed for the tasks. Would you be able to
point me to the right configuration to be set.
Thanks
~Ram
Hi Ramya,
it looks as if you should give your Flink pods and also the Flink process a
bit more memory as the process fails with an out of memory error. You could
also try Flink's latest version which comes with native Kubernetes support.
Cheers,
Till
On Tue, Jun 9, 2020 at 8:45 AM Ramya Ramamurt
Hi,
My flink jobs are constantly going down beyond an hour with the below
exception.
This is Flink 1.7 on kubes, with checkpoints to Google storage.
AsynchronousException{java.lang.Exception: Could not materialize
checkpoint 21 for operator Source: Kafka011TableSource(sid, _zpsbd3,
_zpsbd4, _zpsb
