[
https://issues.apache.org/jira/browse/FLINK-9597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
swy updated FLINK-9597:
-----------------------
Attachment: (was: JM.png)
> Flink fail to scale!
> --------------------
>
> Key: FLINK-9597
> URL: https://issues.apache.org/jira/browse/FLINK-9597
> Project: Flink
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.5.0
> Reporter: swy
> Priority: Major
> Attachments: flink_app_parser_git.zip, scaleNotWork.png
>
>
> Hi, we found that our Flink application with simple logic, which using
> process function is not scale-able when scale from 8 parallelism onward even
> though with sufficient resources. Below it the result which is capped at
> ~250k TPS. No matter how we tune the parallelism of the operators it just not
> scale, same to increase source parallelism.
> Please refer to "scaleNotWork.png",
> 1. fixed source parallelism 4, other operators parallelism 8
> 2. fixed source parallelism 4, other operators parallelism 16
> 3. fixed source parallelism 4, other operators parallelism 32
> 4. fixed source parallelism 6, other operators parallelism 8
> 5. fixed source parallelism 6, other operators parallelism 16
> 6. fixed source parallelism 6, other operators parallelism 32
> 7. fixed source parallelism 6, other operators parallelism 64 performance
> worse than parallelism 32.
> Sample source code attached(flink_app_parser_git.zip). It is a simple
> program, parsing json record into object, and pass it to a empty logic
> Flink's process function. Rocksdb is in used, and the source is generated by
> the program itself. This could be reproduce easily.
> We choose Flink because of it scalability, but this is not the case now,
> appreciated if anyone could help as this is impacting our projects! thank you.
> To run the program, sample parameters,
> "aggrinterval=6000000 loop=7500000 statsd=1 psrc=4 pJ2R=32 pAggr=72
> URL=do36.comptel.com:8127"
> * aggrinterval: time in ms for timer to trigger
> * loop: how many row of data to feed
> * statsd: to send result to statsd
> * psrc: source parallelism
> * pJ2R: parallelism of map operator(JsonRecTranslator)
> * pAggr: parallelism of process+timer operator(AggregationDuration) !JM.png!
> !TM.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
