Just to clarify: Shinhyung is running one a single node with 4 CPUs, each having 16 cores.
On Mon, Apr 4, 2016 at 10:32 AM, Robert Metzger <rmetz...@apache.org> wrote: > Hi, > > usually it doesn't make sense to run multiple task managers on a single > machine to get more slots. > Your machine has only 4 CPU cores, so you are just putting a lot of pressure > on the cpu scheduler.. > > On Thu, Mar 31, 2016 at 7:16 PM, Shinhyung Yang <shinhyung.y...@gmail.com> > wrote: >> >> Thank you for replying! >> >> I am trying to do this on a single machine in fact. Since it has 64 >> cores, it would be interesting to look at the performance in that >> regard. >> >> > How many machines are you using for this? >> > >> > The fact that you are giving 64 slots to each TaskManager means that a >> > single TaskManager may end up executing all 64 pipelines. That would >> > heavily >> > overload that TaskManager and cause heavy degradation. >> >> Does it make sense if I run multiple TaskManagers on a single machine >> if 64 slots are too many for a TaskManager? >> >> > If, for example, you use 16 machines, then give each machine 4 task >> > slots >> > (total of 64 slots across all machines) >> > That way, the final run (parallelism 64) will be guaranteed to be spread >> > across all machines. >> >> My intention for the experiment at the moment is to try to scale the >> application up on a single machine to its maximum before moving on to >> run the experiment on multiple machines. >> >> Thank you again! >> With best regards, >> Shinhyung Yang > >
