Thank you, yes, it can be done externally, if not supported within Flink. So the way to spawn multiple task managers would be to list the same slave machines N times as necessary in the slaves file?
On Sat, Jul 2, 2016 at 11:22 AM, Ufuk Celebi <u...@apache.org> wrote: > No, not inside of Flink. That sounds like something like the OS or > resource manager should handle. > > On Sat, Jul 2, 2016 at 5:12 PM, Saliya Ekanayake <esal...@gmail.com> > wrote: > > That's great, so is there support to pin task managers to sockets as > well? > > > > On Sat, Jul 2, 2016 at 11:08 AM, Ufuk Celebi <u...@apache.org> wrote: > >> > >> Regarding 2) if you don't manually configure something else, that > >> should happen always. > >> > >> Yes, you can run more than one task manager per node depending on the > >> process isolation you want. Within a task manager, there are multiple > >> threads for each slot. For example, if you have 2 task managers with 2 > >> slots each and submit a job with parallelism 4, each task manager will > >> execute 2 sub tasks in separate Threads. > >> > >> > >> On Sat, Jul 2, 2016 at 3:26 AM, Saliya Ekanayake <esal...@gmail.com> > >> wrote: > >> > Hi Ufuk, > >> > > >> > Looking at the document you sent it seems only 1 task manager per node > >> > exist > >> > and within that you have multiple slots. Is it possible to run more > than > >> > 1 > >> > task manager per node? Also, within a task manager is the parallelism > >> > done > >> > through threads or processes? > >> > > >> > Thank you, > >> > Saliya > >> > > >> > On Thu, Jun 30, 2016 at 5:27 PM, Saliya Ekanayake <esal...@gmail.com> > >> > wrote: > >> >> > >> >> Thank you, I'll check these. > >> >> > >> >> In 2.) you said they are likely to exchange through memory. Is there > a > >> >> case why they wouldn't? > >> >> > >> >> On Thu, Jun 30, 2016 at 5:03 AM, Ufuk Celebi <u...@apache.org> wrote: > >> >>> > >> >>> On Thu, Jun 30, 2016 at 1:44 AM, Saliya Ekanayake < > esal...@gmail.com> > >> >>> wrote: > >> >>> > 1. What parameters are available to control parallelism within a > >> >>> > node? > >> >>> > >> >>> Task Manager processing slots: > >> >>> > >> >>> > >> >>> > https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#configuring-taskmanager-processing-slots > >> >>> > >> >>> > 2. Does Flink support shared memory-based messaging within a node > >> >>> > (without > >> >>> > doing TCP calls)? > >> >>> > >> >>> Yes, local exchanges happen via memory and not TCP, for example if > you > >> >>> have a map-reduce, map subtask 1 and reduce subtask 1 are likely to > >> >>> exchange data locally. > >> >>> > >> >>> > 3. Is there support for Infiniband interconnect? > >> >>> > >> >>> No, not that I'm aware of. > >> >>> > >> >>> – Ufuk > >> >> > >> >> > >> >> > >> >> > >> >> -- > >> >> Saliya Ekanayake > >> >> Ph.D. Candidate | Research Assistant > >> >> School of Informatics and Computing | Digital Science Center > >> >> Indiana University, Bloomington > >> >> > >> > > >> > > >> > > >> > -- > >> > Saliya Ekanayake > >> > Ph.D. Candidate | Research Assistant > >> > School of Informatics and Computing | Digital Science Center > >> > Indiana University, Bloomington > >> > > > > > > > > > > > -- > > Saliya Ekanayake > > Ph.D. Candidate | Research Assistant > > School of Informatics and Computing | Digital Science Center > > Indiana University, Bloomington > > > -- Saliya Ekanayake Ph.D. Candidate | Research Assistant School of Informatics and Computing | Digital Science Center Indiana University, Bloomington
