[
https://issues.apache.org/jira/browse/FLINK-30193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17638336#comment-17638336
]
Gen Luo commented on FLINK-30193:
---------------------------------
HI [~landlord],
We did some work like this, which makes rebalance partitioner able to
distribute the records according the load of downstream operators, and we are
considering that maybe we would contribute the code to the community when we
can make the time. Maybe we can discuss about the feature in the mail list if
you'd like. The jira should be created after the discuss in the mail list
reaches a consensus.
> could flink has dynamic rebalance according to the load of downstream
> operators
> ---------------------------------------------------------------------------------
>
> Key: FLINK-30193
> URL: https://issues.apache.org/jira/browse/FLINK-30193
> Project: Flink
> Issue Type: New Feature
> Components: API / DataStream
> Affects Versions: 1.15.2
> Reporter: Lu
> Priority: Major
>
> Different records cause different loads to the downstream and different task
> slot has different compute power. So rebalancing according to the load of
> downstream operators will greatly reduce data skew and improve resource
> utilization.
> I thaught the point is how to gain the load of downstream. I have seen the
> StreamPartitioner , and it dosen't has any context or method to get the load
> information of downstream operators. So ordinary developers cannot easily
> implement such functions.
> finally，Could flink provide such a function?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
