Hi Nicholas,
Try reducing
containerized.heap-cutoff-min
from the default 600MB to a lower number.
Still, I think RPi is way too under-powered to run any serious Flink apps.
Regards,
Kien
On 4/28/2018 4:38 PM, Nicholas Walton wrote:
Hi,
Hope this is the right place to ask, but I have a 5 Raspberry Pi 3
cluster running Flink on which I’m hitting memory issues. Each Pi has
1Gb and is running with a 256Gb USB drive, however I don’t seem to be
able to configure Java to use more than 256M of heap. The memory
problem, I’m reasonably sure, lies in in the key store backed by the
filestore (state.backend: filesystem). Either it is not caching to
disk or has insufficient RAM to function properly. How can I set the
configuration to increase the heap size?
The task manger pane shows the following for each of the five PIs
akka.tcp://flink@wright:35869/user/taskmanager 456052018-04-28,
10:35:25224923 MB224 MB108 MB
My flink configuration is
#—————————————————————————————————
jobmanager.rpc.address: gandalf
# The RPC port where the JobManager is reachable.
jobmanager.rpc.port: 6123
# The heap size for the JobManager JVM, I have to comment these out
# otherwise Flink refuses to run on the Pi
# jobmanager.heap.mb: 1024
# The heap size for the TaskManager JVM
# taskmanager.heap.mb: 1024
# The number of task slots that each TaskManager offers. Each slot
runs one parallel pipeline.
taskmanager.numberOfTaskSlots: 2
# Specify whether TaskManager memory should be allocated when starting
up (true) or when
# memory is required in the memory manager (false)
# Important Note: For pure streaming setups, we highly recommend to
set this value to `false`
# as the default state backends currently do not use the managed memory.
taskmanager.memory.preallocate: false
# The parallelism used for programs that did not specify and other
parallelism.
parallelism.default: 2
….
state.backend: filesystem
….
#—————————————————————————————————
