[
https://issues.apache.org/jira/browse/FLINK-15670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17019907#comment-17019907
]
Jiangjie Qin edited comment on FLINK-15670 at 1/21/20 7:41 AM:
---------------------------------------------------------------
[~maguowei] Yes, that would be an option. However, doing that may have
significant performance impact. Also, IIUC {{blockingConnectionBetweenChains}}
only exists batch jobs, which already have multiple stages as failure boundary.
The issue we are trying to address here seems more about streaming jobs. So
most of the edges will be pipeline.
was (Author: becket_qin):
[~maguowei] Yes, that would be an option. However, IIUC
{{blockingConnectionBetweenChains}} only exists batch jobs, which already have
multiple stages as failure boundary. The issue we are trying to address here
seems more about streaming jobs. So most of the edges will be pipeline.
> Provide a Kafka Source/Sink pair that aligns Kafka's Partitions and Flink's
> KeyGroups
> -------------------------------------------------------------------------------------
>
> Key: FLINK-15670
> URL: https://issues.apache.org/jira/browse/FLINK-15670
> Project: Flink
> Issue Type: New Feature
> Components: API / DataStream, Connectors / Kafka
> Reporter: Stephan Ewen
> Priority: Major
> Labels: usability
> Fix For: 1.11.0
>
>
> This Source/Sink pair would serve two purposes:
> 1. You can read topics that are already partitioned by key and process them
> without partitioning them again (avoid shuffles)
> 2. You can use this to shuffle through Kafka, thereby decomposing the job
> into smaller jobs and independent pipelined regions that fail over
> independently.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
