Hi, I’m not entirely sure what you are testing. I have looked at your code (only the constant straggler scenario) and please correct me if’m wrong, in your job you are basically measuring throughput of `Thread.sleep(straggler.waitMillis)`.
In the first RichMap task (`subTaskId == 0`), per every record you do the sleep(50ms), so after filling in all of the network buffers your whole job will be bottlenecked by this throughput cap of 20 records / second. Every so often when this struggling task will be able to process and free up some buffer from the backlog. This briefly unblocks other three tasks (which are capped at 133 records / second). Apart from those short stints, those other tasks can not process constant 133 records / seconds, because records are evenly distributed by the source between all of those tasks. Which is I think clearly visible on the charts and every system would behave in exactly the same way. But what scenario are you really trying to simulate? A data skew when one task is 6.65 (133 / 20 ) times more overloaded/processing heavier records than the others? Yes, this is expected behaviour, but your benchmark is testing this in a bit convoluted way. A failing machine which has 6.65 times less performance? With keyed network exchanges there is again very little that you can do (except of the speculative execution). Without keyed network exchanges, OK, I agree. In this case, randomly/evenly distributing the records is not the optimal shuffling strategy and there is some room for the improvement in Flink (we could distribute records not randomly but to the less busy machines). However this is a pretty much niche feature (failing machine + non keyed exchanges) and you are not saying anywhere that this is what you are testing for. Piotrek > On 8 Oct 2019, at 18:10, Owen Rees-Hayward <owe...@googlemail.com> wrote: > > Hi, > > I am having a few issues with the Flink (v1.8.1) backpressure default > settings, which lead to poor throughput in a comparison I am doing between > Storm, Spark and Flink. > > I have a setup that simulates a progressively worse straggling task that > Storm and Spark cope with the relatively well. Flink not so much. Code can be > found here - https://github.com/owenrh/flink-variance > <https://github.com/owenrh/flink-variance>. > > See this throughput chart for the an idea of how badly - > https://owenrh.me.uk/assets/images/blog/smackdown/flink-constant-straggler.png > > <https://owenrh.me.uk/assets/images/blog/smackdown/flink-constant-straggler.png> > > I do not have any production experience with Flink, but I have had a look at > the Flink docs and there is nothing in there that jumps out at me to explain > or address this. I presume I am missing something, as I cannot believe Flink > is this weak in the face of stragglers. It must be configuration right? > > Would appreciate any help on this. I've got a draft blog post that I will > publish in a day or two, and don't want to criticise the Flink backpressure > implementation for what seems most likely some default configuration issue. > > Thanks in advance, Owen > > -- > Owen Rees-Hayward > 07912 876046 > twitter.com/owen4d <http://twitter.com/owen4d>
