Re: CPU Cores of JobManager

Thank you for information. I will investigate my cluster more carefully. Anyway, at this moment, the WEBUI works well(I don't know why). If I get some updates, I will inform you. By the way, I have another problem... When I run multiple Flink jobs in YARN, the job-logs are mixed. cf) yarn logs

Re: CPU Cores of JobManager

That said, metrics could still be the issue, with FLINK-7368 being the prime suspect. This was fixed for 1.3.3 and 1.4 (both unreleased though). On 06.12.2017 10:36, Chesnay Schepler wrote: This can only have an effect if "*taskmanager.net.deta

Re: CPU Cores of JobManager

This can only have an effect if "*taskmanager.net.detailed-metrics*" is enabled (by default it is disabled). Otherwise an increase in sub-channels will neither increase the number of metrics nor the number of polls by the UI. On 06.12.2017 10:22, Stephan Ewen wrote: A 'rebalance()' should no im

Re: CPU Cores of JobManager

A 'rebalance()' should no impose load on the JM itself. There are more sub-channels in that case, but the JobManager is not coordinating anything for them unless there is a failure/recovery. Each subchannel generates additional metrics, though - might that be an issue? On Wed, Dec 6, 2017 at 9:50

Re: CPU Cores of JobManager

> Which version of Fllink are you using? I'm using Flink 1.3.0. > Is the high CPU usage caused by the browser or the JVM? The CPU usage of browser is not high, the JVM is not also. I found that if I remove the rebalance function in my code, the WEBUI works well. I don't know why but I think reb

Re: CPU Cores of JobManager

Which version of Fllink are you using? Is the high CPU usage caused by the browser or the JVM? One thing we should investigate is how many requests the UI sends to the JobManager, but I don't know of a good way to do that without modifying the source. On 06.12.2017 01:20, Yuta Morisawa wrote:

Re: CPU Cores of JobManager

Hi Timo I execute streaming job without checkpointing and I don't configure any state backend, so it may be "MemoryStateBackend". Actually, my streaming app just reads data from kafka and writes it to an external DB. Its not so complicated. Regards, Yuta On 2017/12/05 19:55, Timo Walther w

Re: CPU Cores of JobManager

I had some profiling tool like jvisualvm in mind. Are you executing streaming or batch jobs? If streaming, is checkpointing enabled and which type of statebackend? @Chesnay do you have experience with slow behavior of the Web UI? Regards, Timo Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa: H

Re: CPU Cores of JobManager

Hi Timo Thank you for your early reply. These are commands which I run my apps. ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10 ./bin/flink run -p 100 ./bin/flink run -p 100 So, JobManager Heap Memory = 6000 MB and it manages 2 jobs. > Maybe you can use a profiler and find out which c

Re: CPU Cores of JobManager

Hi Yuta, as far as I know you cannot assign more cores to a JobManager. Can you tell us a bit more about your environment? How many jobs does the JobManager has to manage? How much heap memory is assigned to the JobManager? Maybe you can use a profiler and find out which component consumes s

CPU Cores of JobManager

Hi Now I am looking for the way to increase the number of allocated CPU cores because my JobManagaer WEBUI is very heavy and sometimes freeze. I think this is caused by the resource shortage of JobManager. How can I increase the number of CPU for JobManager in YARN mode? Thanks Yuta --