Which version of Fllink are you using?
Is the high CPU usage caused by the browser or the JVM?
One thing we should investigate is how many requests the UI sends to the
JobManager,
but I don't know of a good way to do that without modifying the source.
On 06.12.2017 01:20, Yuta Morisawa wrote:
Hi Timo
I execute streaming job without checkpointing and I don't configure
any state backend, so it may be "MemoryStateBackend".
Actually, my streaming app just reads data from kafka and writes it to
an external DB. Its not so complicated.
Regards,
Yuta
On 2017/12/05 19:55, Timo Walther wrote:
I had some profiling tool like jvisualvm in mind. Are you executing
streaming or batch jobs? If streaming, is checkpointing enabled and
which type of statebackend?
@Chesnay do you have experience with slow behavior of the Web UI?
Regards,
Timo
Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa:
Hi Timo
Thank you for your early reply.
These are commands which I run my apps.
./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10
./bin/flink run -p 100 <jar1> <jar1 parameters>
./bin/flink run -p 100 <jar2> <jar2 parameters>
So, JobManager Heap Memory = 6000 MB and it manages 2 jobs.
> Maybe you can use a profiler and find out which component consumes so
> much CPU resources?
You mean Java Flight Recorder or JITWatch?
Or, Flink has original profiler?
https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html
Regards,
Yuta
On 2017/12/05 18:02, Timo Walther wrote:
Hi Yuta,
as far as I know you cannot assign more cores to a JobManager.
Can you tell us a bit more about your environment? How many jobs
does the JobManager has to manage? How much heap memory is assigned
to the JobManager?
Maybe you can use a profiler and find out which component consumes
so much CPU resources?
Regards,
Timo
Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa:
Hi
Now I am looking for the way to increase the number of allocated
CPU cores because my JobManagaer WEBUI is very heavy and sometimes
freeze.
I think this is caused by the resource shortage of JobManager.
How can I increase the number of CPU for JobManager in YARN mode?
Thanks
Yuta
