Are you sure you have the parallelism set to 448? You can see the parallelism of operators in the web UI.
On Fri, Sep 23, 2016 at 12:15 AM, amir bahmanyari <amirto...@yahoo.com> wrote: > Hi Again & sorry to take your time. But am puzzled by what I cannot > explain why. > The parallelism is set to 448. There are 112 tasks per TM. > Why is Flink NOT allocating ALL 448 slots? It allocates only 1/2 of it. > I also bumped up the # buffers to equate a 2GiB in each TM & see no > difference :-( > So I incremented my total-slots = 448. Kafka topic also has 448 partitions. > Why am I having such a bad luck with this!!!!!!!??? LOL!! > Thanks for your attention Aljoscha. > > [image: Inline image] > > > > ------------------------------ > *From:* amir bahmanyari <amirto...@yahoo.com> > *To:* Aljoscha Krettek <aljos...@apache.org>; User <user@flink.apache.org> > > *Sent:* Thursday, September 22, 2016 10:10 AM > > *Subject:* Re: How can I prove .... > > Hi Again, following is from the dashboard while wverything is supposedlt > running. > No real-time change in send/received/#of records...but one node is > definitely producing a *.out file... > And all TMs are reporting in their *.log files. And the process will > eventually end , but very slow. > Thanks again Aljoscha. > [image: Inline image] > > > > ------------------------------ > *From:* amir bahmanyari <amirto...@yahoo.com> > *To:* Aljoscha Krettek <aljos...@apache.org>; User <user@flink.apache.org> > > *Sent:* Thursday, September 22, 2016 9:16 AM > *Subject:* Re: How can I prove .... > > Thanks Aljoscha, > Thats why I am wondering about this. I dont see send/receive columns > change at all....just 0's all the time. > The only thing that changes is time stamp. > Is this an indication that the nodes in the cluster are not participating > in execution of the data? > Thanks again. > Amir- > > > ------------------------------ > *From:* Aljoscha Krettek <aljos...@apache.org> > *To:* amir bahmanyari <amirto...@yahoo.com>; User <user@flink.apache.org> > *Sent:* Thursday, September 22, 2016 5:01 AM > *Subject:* Re: How can I prove .... > > Hi, > depending on the data source you might not be able to stress CPU/MEM > because the source might be to slow. As long as you see the numbers > increasing in the Flink Dashboard for all operators you should be good. > > Cheers, > Aljoscha > > On Thu, 22 Sep 2016 at 00:26 amir bahmanyari <amirto...@yahoo.com> wrote: > > That all nodes in a Flink Cluster are involved simultaneously in > processing the data? > Programmatically, graphically... > I need to stress CPU , MEM and all resources to their max. > How can I guarantee this is happening in Flink Cluster? > Out of 4 nodes, this is the highest resource usage I see from "top"... > Everything else is not even close... > t > op - 22:22:45 up 41 days, 2:39, 1 user, load average: 1.76, 1.55, 1.28 > Tasks: 344 total, 1 running, 343 sleeping, 0 stopped, 0 zombie > %Cpu(s): 5.4 us, 1.0 sy, 0.0 ni, 93.5 id, 0.1 wa, 0.0 hi, 0.0 si, > 0.0 st > KiB Mem: 11551564+total, 65702020 used, 49813632 free, 115072 buffers > KiB Swap: 0 total, 0 used, 0 free. 3148420 cached Mem > > > I am pretty sure I can push FlinkRunner to way more extent than this.... > And thats where true realistic perf numbers start showing up. > Thanks+regards, > Amir- > > > > > > > >
