Exactly, at least it's worth mentioning the partitioner used by default in case 
none was specified in the javadoc, because the default behavior might not seem 
obvious.


Kind Regards,
Mike Pryakhin


> On 3 Dec 2017, at 22:08, Stephan Ewen <se...@apache.org> wrote:
> 
> Sounds like adding a round robin partitioner to the set of readily available 
> partitioners would make sense.
> 
> On Fri, Dec 1, 2017 at 5:16 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org 
> <mailto:tzuli...@apache.org>> wrote:
> Hi Mike,
> 
> The rationale behind implementing the FlinkFixedPartitioner as the default
> is so that each Flink sink partition (i.e. one sink parallel subtask) maps
> to a single Kafka partition.
> 
> One other thing to clarify:
> By setting the partitioner to null, the partitioning is based on a hash of
> the record's attached key (the key retrieved from the
> `SerializationSchema`), not round-robin.
> To use round-robin partitioning, a custom partitioner should be provided.
> Note however, a round-robin partitioner will create a lot of network
> connections to all Kafka brokers on all Flink sink parallel subtasks, which
> can be quite a lot.
> 
> To conclude this, I think the appropriate partitioning scheme depends on the
> actual case.
> For example, for some simple Flink job that only does some filtering of data
> and has no aggregation within the pipeline, the key hash based partitioning
> would probably be more ideal.
> For more complex pipelines that partitions the computation by key already,
> it could make sense that a direct mapping of a Flink sink partition to Kafka
> partition would do.
> 
> On the other hand, considering that the key for each record is always
> "re-calculated" by the `SerializationSchema` in each Flink Kafka Producer
> sink partition, it might make sense to actually make the key hash
> partitioner as the default instead.
> 
> Cheers,
> Gordon
> 
> 
> 
> --
> Sent from: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ 
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
> 

Attachment: smime.p7s
Description: S/MIME cryptographic signature

Reply via email to