Hi, Till: In fact, I want to ask the resources consume by job manager
Till Rohrmann <trohrm...@apache.org> 于 2018年3月21日周三 下午8:17写道： > As many as the application needs to run. If you start a job with > parallelism 10 then it will ask for 10 slots (assuming slot sharing). > > On Wed, Mar 21, 2018 at 12:04 PM, Renjie Liu <liurenjie2...@gmail.com> > wrote: > > > So how many slots a job manager may consume? > > > > On Wed, Mar 21, 2018 at 6:50 PM Till Rohrmann <trohrm...@apache.org> > > wrote: > > > > > At the moment this is not possible. In order to do this, you will have > to > > > use the per-job mode and run each job on a dedicated Flink cluster. > > > > > > On Wed, Mar 21, 2018 at 11:33 AM, Renjie Liu <liurenjie2...@gmail.com> > > > wrote: > > > > > > > For example, we have 2 jobs. > > > > For job 1, I want to start job manger with 1 CPU and 100M memory. > Job 1 > > > > need s10 slots, and I want to deploy these 10 slots in 2 task > managers, > > > > each with 5 cores and 1G memory. > > > > > > > > For job 2, I want to start job manager with 2 CPU and 200M memory. > Job > > 2 > > > > needs 100 slots and I want to deploy these 100 slot in 10 task > > managers, > > > > each with 10 cores and 2G memory. > > > > > > > > Is this possible? > > > > > > > > On Wed, Mar 21, 2018 at 6:19 PM Till Rohrmann <trohrm...@apache.org> > > > > wrote: > > > > > > > > > Hi Renjie, > > > > > > > > > > what do you mean with specifying different JM and TM resources for > > > > > different jobs exactly? > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Wed, Mar 21, 2018 at 10:55 AM, Renjie Liu < > > liurenjie2...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi, Till: > > > > > > > > > > > > How to specify job manager and task manager resources for > different > > > > jobs > > > > > in > > > > > > session mode? > > > > > > > > > > > > On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann < > > trohrm...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > Hi Shuyi, > > > > > > > > > > > > > > best if you look at the other e2e tests in the > > > flink-end-to-end-tests > > > > > > > module. For example the Kafka e2e test under > > > > > > > flink/flink-end-to-end-tests/test-scripts/test_streaming_ > > > > kafka010.sh. > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen < > suez1...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > For FLINK-8562, the test is passing now because it's not > really > > > > > > > > checking the right thing. > > > > > > > > > > > > > > > > Yes, I can help with the Kerberos integration ticket. > > > > > > > > > > > > > > > > Is there an example on how the e2e test should be structured > > and > > > > > > invoked? > > > > > > > > > > > > > > > > Thanks > > > > > > > > Shuyi > > > > > > > > > > > > > > > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann < > > > > trohrm...@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Shuyi, > > > > > > > > > > > > > > > > > > thanks for the working on FLINK-8562. Once this issue is > > fixed, > > > > it > > > > > > will > > > > > > > > > automatically be executed on the Flip-6 components. In fact > > it > > > is > > > > > > > already > > > > > > > > > being executed on Flip-6. > > > > > > > > > > > > > > > > > > But what you could help the community with is setting up an > > > > > automated > > > > > > > > > end-to-end test for the Kerberos integration if you want: > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > > > > > > > > > > > > > > > The Flink community is currently working on automating more > > and > > > > > more > > > > > > > > tests > > > > > > > > > in order to facilitate faster releases and improve the test > > > > > coverage. > > > > > > > You > > > > > > > > > can find more about this effort here: > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Till > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen < > > > suez1...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already > > sent > > > a > > > > PR > > > > > > to > > > > > > > > > > resolve the issue, your help to take a look will be > great. > > > > > > > > > > > > > > > > > > > > Please let me know what I can help to test the Kerberos > > > > > > > > authentication, I > > > > > > > > > > am decently familiar with the Kerberos and YARN security > > part > > > > in > > > > > > > Flink. > > > > > > > > > > > > > > > > > > > > As a starting point, I'd suggest to add an integration > test > > > > > similar > > > > > > > to > > > > > > > > > > YARNSessionFIFOSecuredITCase > > > > > > > > > > for flip6. > > > > > > > > > > > > > > > > > > > > Shuyi > > > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann < > > > > > > trohrm...@apache.org > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > > > > > thanks for the pointer with the > > > YARNSessionFIFOSecuredITCase. > > > > > > > You're > > > > > > > > > > right > > > > > > > > > > > that we should fix this test. There is FLINK-8562 which > > > seems > > > > > to > > > > > > > > > address > > > > > > > > > > > the problem. Will take a look. > > > > > > > > > > > > > > > > > > > > > > Additionally, we want to test Kerberos authentication > > > > > explicitly > > > > > > as > > > > > > > > > part > > > > > > > > > > of > > > > > > > > > > > the release testing for Flink 1.5. I will shortly send > > > > around a > > > > > > > mail > > > > > > > > > > where > > > > > > > > > > > I will lay out the ongoing testing efforts and where > more > > > is > > > > > > > needed. > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu < > > > > > > > liurenjie2...@gmail.com > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Thanks for the clarification > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 < > > summerle...@163.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > if I am not misunderstand, you just need to start > the > > > > > cluster > > > > > > > as > > > > > > > > > > normal > > > > > > > > > > > > as > > > > > > > > > > > > > before. The dispatcher and resourcemanager are > > spawned > > > by > > > > > > > > > > > > ClusterEntryPoint > > > > > > > > > > > > > (you can have a look at yarn-session.sh & > > > > > > FlinkYarnSessionCli & > > > > > > > > > > > > > YarnSessionClusterEntrypoint), and the TM are > spawned > > > by > > > > > > > > > > > ResourceManager > > > > > > > > > > > > > lazily (ResourceManager will setup TM according to > > the > > > > > > > submitted > > > > > > > > > job) > > > > > > > > > > > or > > > > > > > > > > > > > spawned by the setup script (you can have a look at > > > > > > > > > > start-cluster.sh). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > > > Sihua Zhou > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 发自网易邮箱大师 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 03/15/2018 10:14，Renjie Liu< > > liurenjie2...@gmail.com > > > > > > > > > > wrote： > > > > > > > > > > > > > Hi, Till: > > > > > > > > > > > > > In fact I'm asking how to deploy other components > > such > > > as > > > > > > > > > dispatcher, > > > > > > > > > > > > etc. > > > > > > > > > > > > > > > > > > > > > > > > > > Till Rohrmann <trohrm...@apache.org> 于 > 2018年3月15日周四 > > > > > > 上午12:17写道： > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > > > > > > > > > in the current master and release-1.5 branch flip-6 > > is > > > > > > > activated > > > > > > > > by > > > > > > > > > > > > > default. If you want to turn it off you have to add > > > > `mode: > > > > > > old` > > > > > > > > to > > > > > > > > > > your > > > > > > > > > > > > > flink-conf.yaml. I'm really happy that you want to > > test > > > > it > > > > > > out > > > > > > > > :-) > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu < > > > > > > > > > liurenjie2...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Till: > > > > > > > > > > > > > Is there any doc on deploying flink in flip6 mode? > We > > > > want > > > > > to > > > > > > > > help > > > > > > > > > > > > > testing > > > > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > Till Rohrmann <trohrm...@apache.org> 于 > 2018年3月14日周三 > > > > > > 下午7:08写道： > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > > > > > > > > > in order to make Mesos work, we only needed to > > > implement > > > > a > > > > > > > Mesos > > > > > > > > > > > > > specific > > > > > > > > > > > > > ResourceManager. Look at MesosResourceManager for > > more > > > > > > details. > > > > > > > > As > > > > > > > > > > > > > dispatcher, we use the StandaloneDispatcher which > is > > > > > spawned > > > > > > by > > > > > > > > > > > > > the MesosSessionClusterEntrypoint. > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu < > > > > > > > > > liurenjie2...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all: > > > > > > > > > > > > > I'm reading the source code and it seems that flip6 > > > does > > > > > not > > > > > > > > > support > > > > > > > > > > > > > mesos? > > > > > > > > > > > > > According to the design, client send job graph to > > > > > dispatcher > > > > > > > and > > > > > > > > > > > > > dispatcher > > > > > > > > > > > > > spawn job mananger and resource manager for job > > > > execution. > > > > > > But > > > > > > > I > > > > > > > > > > > > > can't > > > > > > > > > > > > > find > > > > > > > > > > > > > dispatcher implementation for mesos. > > > > > > > > > > > > > -- > > > > > > > > > > > > > Liu, Renjie > > > > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > Liu, Renjie > > > > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > Liu, Renjie > > > > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Liu, Renjie > > > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > "So you have to trust that the dots will somehow connect > in > > > > your > > > > > > > > future." > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > "So you have to trust that the dots will somehow connect in > > your > > > > > > future." > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Liu, Renjie > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > -- > > > > Liu, Renjie > > > > Software Engineer, MVAD > > > > > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > -- Liu, Renjie Software Engineer, MVAD
