Re: on YARN question

I am not a Flink expert. Just out of curiosity, I am seeing “YARN application has been deployed successfully“ Does it not mean it’s working properly? Best, Ethan > On Apr 9, 2020, at 23:01, 罗杰 wrote: > >  > Hello, could you please tell me how to solve the problem that when I use > yarn

Re: How to enforce ACLs on Flink JobManager/ApplicationMaster URL on Yarn

lows teams to spin up clusters really > easily, so we can get away with saying that the machine network rules can > map to the actual access control rules any given job/team might need. > > Aaron > > On Mon, Mar 30, 2020 at 12:53 PM Ethan Li > wrote: > >> Hi Team, &g

How to enforce ACLs on Flink JobManager/ApplicationMaster URL on Yarn

Hi Team, I am evaluating Flink on yarn. I can submit a flink job to a secured Yarn cluster and the job can run correctly. But flink jobmanager UI seems accessibly by everyone. Is there anyway in Flink or Yarn to secure it with ACLs? Thanks, Ethan

Re: [Question] Failed to submit flink job to secure yarn cluster

Sorry forgot to update on this. I figured it out. KMS is not set up correctly in my environment. ResourceManager is also missing key provider config. PE is fixing it. Thanks for looking into this Ethan Li > On Jan 13, 2020, at 21:38, Yang Wang wrote: > >  > I am not f

Re: [Question] Failed to submit flink job to secure yarn cluster

ects/flink/flink-docs-release-1.9/ops/config.html#security-kerberos-login-contexts > > Best, > Yangze Guo > > On Fri, Jan 10, 2020 at 10:37 AM Ethan Li wrote: >> >> Hello >> >> I was following >> https://ci.apache.org/projects/flink/flink-docs-releas

[Question] Failed to submit flink job to secure yarn cluster

Hello I was following https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/deployment/yarn_setup.html#run-a-flink-job-on-yarn and trying to submit a flink job on yar

Re: Questions about taskmanager.memory.off-heap and taskmanager.memory.preallocate

researching git history, I found the description of > "taskmanager.memory.preallocate" was written by @Chesnay Schepler > <mailto:ches...@apache.org> (from 1.8 branch). So maybe he can give more > context or information. Correct me, if I am wrong. > Best, > Vino. > &

Re: Questions about taskmanager.memory.off-heap and taskmanager.memory.preallocate

ed, > but will have to wait until the GC to be truly released. > > Thank you~ > Xintong Song > > > On Tue, Dec 17, 2019 at 12:30 PM Ethan Li <mailto:ethanopensou...@gmail.com>> wrote: > Thank you very much Xintong! It’s much clear to me now. > > I am stil

Questions about taskmanager.memory.off-heap and taskmanager.memory.preallocate

Hi Community, I have a question about the taskmanager.memory.preallocate config in the doc https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmanager-memory-preallocate

Multiple Taskmanagers per node for standalone cluster

Hello, Does/did anyone try to set up a standalone cluster with multiple TaskManagers per node? We are working on moving to flink-on-yarn solution. But before that happens, I am thinking about the following setup to get jobs isolated from each other 1) multiple taskmanager per host 2) 1 taskS

Re: TaskManager gets confused after the JobManager restarts

gt; > Cheers, > Till > > On Thu, Feb 14, 2019 at 5:06 PM Ethan Li <mailto:ethanopensou...@gmail.com>> wrote: > The related job manager log is > https://gist.github.com/Ethanlm/86a10e786ad9025ddaa27c113c536da8 > <https://gist.github.com/Ethanlm/86a10e786ad9025ddaa2

Re: TaskManager gets confused after the JobManager restarts

The related job manager log is https://gist.github.com/Ethanlm/86a10e786ad9025ddaa27c113c536da8 > On Feb 14, 2019, at 9:40 AM, Ethan Li wrote: > > Hello, > > I have a standalone flink-1.4.2 cluster with one JobManager, one TaskManager, > and zookeeper. I first started JM

TaskManager gets confused after the JobManager restarts

Hello, I have a standalone flink-1.4.2 cluster with one JobManager, one TaskManager, and zookeeper. I first started JM and TM and waited for them to be stable. Then I restarted JM. It’s when the TM got confused. TM got notified that Leader node has changed and it tried to register to the new

Re: One TaskManager per node or multiple TaskManager per node

if you don't run > multiple jobs in the same TM/JVM. > > In terms of the TM memory usage you can set that up by configuring it in the > flink-conf.yaml file. The config key you are looking or is > taskmanager.heap.size: > https://ci.apache.org/projects/flink/flink-d

Re: One TaskManager per node or multiple TaskManager per node

7/ops/config.html#taskmanager-heap-size > > <https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html#taskmanager-heap-size> > > > On Mon, Jan 14, 2019 at 8:05 AM Ethan Li <mailto:ethanopensou...@gmail.com>> wrote: > Hello, > > I am

One TaskManager per node or multiple TaskManager per node

Hello, I am setting up a standalone flink cluster and I am wondering what’s the best way to distribute TaskManagers. Do we usually launch one TaskManager (with many slots) per node or multiple TaskManagers per node (with smaller number of slots per tm) ? Also with one TaskManager per node, I