Re: Flip 6 mesos support

o to file a bug in jira. > > > > Ted Yu 于 2018年3月23日周五 下午7:16写道： > > > > > Renjie:The log didn't go through. > > > Consider logging a JIRA and attach the log there. > > > Thanks > > > Original message From: Renjie Liu < >

Re: Flip 6 mesos support

; > Original message From: Renjie Liu < > > liurenjie2...@gmail.com> Date: 3/23/18 1:38 AM (GMT-08:00) To: > > dev@flink.apache.org Subject: Re: Flip 6 mesos support > > Hi, Till:Attached is my log. > > I'm also looking into this, could you please assign t

Re: Flip 6 mesos support

--From: Renjie Liu < > liurenjie2...@gmail.com> Date: 3/23/18 1:38 AM (GMT-08:00) To: > dev@flink.apache.org Subject: Re: Flip 6 mesos support > Hi, Till:Attached is my log. > I'm also looking into this, could you please assign this bug to me? I'm > also trying

Re: Flip 6 mesos support

Renjie:The log didn't go through. Consider logging a JIRA and attach the log there.  Thanks Original message From: Renjie Liu Date: 3/23/18 1:38 AM (GMT-08:00) To: dev@flink.apache.org Subject: Re: Flip 6 mesos support Hi, Till:Attached is my log. I'm also looking

Re: Flip 6 mesos support

Hi, Till: Attached is my log. I'm also looking into this, could you please assign this bug to me? I'm also trying to contribute to flink. On Fri, Mar 23, 2018 at 4:11 PM Till Rohrmann wrote: > HI Renjie, could you share the logs with us? This sounds like a bug we > should fix. > > Cheers, > Til

Re: Flip 6 mesos support

HI Renjie, could you share the logs with us? This sounds like a bug we should fix. Cheers, Till On Fri, Mar 23, 2018 at 4:42 AM, Renjie Liu wrote: > Hi, Till: > Has anybody succeeded to deploy flip 6 mode on mesos? > > I'm testing flip 6 using the master branch and I just can't run jobs. The >

Re: Flip 6 mesos support

Hi, Till: Has anybody succeeded to deploy flip 6 mode on mesos? I'm testing flip 6 using the master branch and I just can't run jobs. The following are my configurations: *jobmanager.rpc.address: qt9ss.prod.mediav.com * *jobmanager.rpc.port: 6123* *jobmanager.heap.mb

Re: Flip 6 mesos support

The resources consumed by the JobMaster can be specified by `jobmanager.heap.mb`. Cheers, Till On Wed, Mar 21, 2018 at 3:20 PM, Renjie Liu wrote: > Hi, Till: > > In fact, I want to ask the resources consume by job manager > > Till Rohrmann 于 2018年3月21日周三 下午8:17写道： > > > As many as the applicat

Re: Flip 6 mesos support

Hi, Till: In fact, I want to ask the resources consume by job manager Till Rohrmann 于 2018年3月21日周三 下午8:17写道： > As many as the application needs to run. If you start a job with > parallelism 10 then it will ask for 10 slots (assuming slot sharing). > > On Wed, Mar 21, 2018 at 12:04 PM, Renjie Li

Re: Flip 6 mesos support

As many as the application needs to run. If you start a job with parallelism 10 then it will ask for 10 slots (assuming slot sharing). On Wed, Mar 21, 2018 at 12:04 PM, Renjie Liu wrote: > So how many slots a job manager may consume? > > On Wed, Mar 21, 2018 at 6:50 PM Till Rohrmann > wrote: >

Re: Flip 6 mesos support

So how many slots a job manager may consume? On Wed, Mar 21, 2018 at 6:50 PM Till Rohrmann wrote: > At the moment this is not possible. In order to do this, you will have to > use the per-job mode and run each job on a dedicated Flink cluster. > > On Wed, Mar 21, 2018 at 11:33 AM, Renjie Liu >

Re: Flip 6 mesos support

At the moment this is not possible. In order to do this, you will have to use the per-job mode and run each job on a dedicated Flink cluster. On Wed, Mar 21, 2018 at 11:33 AM, Renjie Liu wrote: > For example, we have 2 jobs. > For job 1, I want to start job manger with 1 CPU and 100M memory. Job

Re: Flip 6 mesos support

For example, we have 2 jobs. For job 1, I want to start job manger with 1 CPU and 100M memory. Job 1 need s10 slots, and I want to deploy these 10 slots in 2 task managers, each with 5 cores and 1G memory. For job 2, I want to start job manager with 2 CPU and 200M memory. Job 2 needs 100 slots and

Re: Flip 6 mesos support

Hi Renjie, what do you mean with specifying different JM and TM resources for different jobs exactly? Cheers, Till On Wed, Mar 21, 2018 at 10:55 AM, Renjie Liu wrote: > Hi, Till: > > How to specify job manager and task manager resources for different jobs in > session mode? > > On Sun, Mar 18,

Re: Flip 6 mesos support

Hi, Till: How to specify job manager and task manager resources for different jobs in session mode? On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann wrote: > Hi Shuyi, > > best if you look at the other e2e tests in the flink-end-to-end-tests > module. For example the Kafka e2e test under > flink/f

Re: Flip 6 mesos support

Hi Shuyi, best if you look at the other e2e tests in the flink-end-to-end-tests module. For example the Kafka e2e test under flink/flink-end-to-end-tests/test-scripts/test_streaming_kafka010.sh. Cheers, Till On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen wrote: > Hi Till, > > For FLINK-8562, the

Re: Flip 6 mesos support

Hi Till, For FLINK-8562, the test is passing now because it's not really checking the right thing. Yes, I can help with the Kerberos integration ticket. Is there an example on how the e2e test should be structured and invoked? Thanks Shuyi On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann wrote:

Re: Flip 6 mesos support

Hi Shuyi, thanks for the working on FLINK-8562. Once this issue is fixed, it will automatically be executed on the Flip-6 components. In fact it is already being executed on Flip-6. But what you could help the community with is setting up an automated end-to-end test for the Kerberos integration

Re: Flip 6 mesos support

Hi Till, This is Shuyi :) Thanks a lot. In FLINK-8562, I already sent a PR to resolve the issue, your help to take a look will be great. Please let me know what I can help to test the Kerberos authentication, I am decently familiar with the Kerberos and YARN security part in Flink. As a starting

Re: Flip 6 mesos support

Hi Renjie, thanks for the pointer with the YARNSessionFIFOSecuredITCase. You're right that we should fix this test. There is FLINK-8562 which seems to address the problem. Will take a look. Additionally, we want to test Kerberos authentication explicitly as part of the release testing for Flink 1

Re: Flip 6 mesos support

Thanks for the clarification On Thu, Mar 15, 2018 at 2:30 PM 周思华 wrote: > Hi Renjie, > if I am not misunderstand, you just need to start the cluster as normal as > before. The dispatcher and resourcemanager are spawned by ClusterEntryPoint > (you can have a look at yarn-session.sh & FlinkYarnSes

Re: Flip 6 mesos support

Hi Renjie, if I am not misunderstand, you just need to start the cluster as normal as before. The dispatcher and resourcemanager are spawned by ClusterEntryPoint (you can have a look at yarn-session.sh & FlinkYarnSessionCli & YarnSessionClusterEntrypoint), and the TM are spawned by ResourceManag

Re: Flip 6 mesos support

Hi, Till: In fact I'm asking how to deploy other components such as dispatcher, etc. Till Rohrmann 于 2018年3月15日周四 上午12:17写道： > Hi Renjie, > > in the current master and release-1.5 branch flip-6 is activated by > default. If you want to turn it off you have to add `mode: old` to your > flink-conf

Re: Flip 6 mesos support

Hi Till, have we tested the YARN kerberos integration in flip6? AFAI remember, YARNSessionFIFOSecuredITCase is not functioning (FLINK-8562 ), do we have similar integration test for flip6? Also, Flink yarn kerberos integration in the old deployment

Re: Flip 6 mesos support

Hi Renjie, in the current master and release-1.5 branch flip-6 is activated by default. If you want to turn it off you have to add `mode: old` to your flink-conf.yaml. I'm really happy that you want to test it out :-) Cheers, Till On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu wrote: > Hi Till: >

Re: Flip 6 mesos support

Hi Till: Is there any doc on deploying flink in flip6 mode? We want to help testing it. Till Rohrmann 于 2018年3月14日周三 下午7:08写道： > Hi Renjie, > > in order to make Mesos work, we only needed to implement a Mesos specific > ResourceManager. Look at MesosResourceManager for more details. As > dispatc

Re: Flip 6 mesos support

Hi Renjie, in order to make Mesos work, we only needed to implement a Mesos specific ResourceManager. Look at MesosResourceManager for more details. As dispatcher, we use the StandaloneDispatcher which is spawned by the MesosSessionClusterEntrypoint. Cheers, Till On Wed, Mar 14, 2018 at 9:32 AM,

Flip 6 mesos support

Hi all: I'm reading the source code and it seems that flip6 does not support mesos? According to the design, client send job graph to dispatcher and dispatcher spawn job mananger and resource manager for job execution. But I can't find dispatcher implementation for mesos. -- Liu, Renjie Software E