If I can throw in my 2 cents, I agree with what Elias says. Without that feature (not partitioning already partitioned Kafka data), Flink is in bad position for common simpler processing, that don't involve shuffling at all, for example simple readKafka-enrich-writeKafka . The systems like the new Kafka Streams processing system, that leverage Kafka partitioning, will probably win with Flink in performance (of course, it's just an intuition).
Are you planning to provide such feature? Is it simple to do with Flink current engine and API? czw., 14.04.2016 o 03:11 użytkownik Elias Levy <fearsome.lucid...@gmail.com> napisał: > On Wed, Apr 13, 2016 at 2:10 AM, Stephan Ewen <se...@apache.org> wrote: > >> If you want to use Flink's internal key/value state, however, you need to >> let Flink re-partition the data by using "keyBy()". That is because Flink's >> internal sharding of state (including the re-sharding to adjust parallelism >> we are currently working on) follows a dedicated hashing scheme which is >> with all likelihood different from the partition function that writes the >> key/value pairs to the Kafka Topics. >> > > That is interesting, if somewhat disappointing. I was hoping that > performing a keyBy from a Kafka source would perform no reshuffling if you > used the same value as you used for the Kafka message key. But it makes > sense if you are using different hash functions. > > It may be useful to have a variant of keyBy() that converts the stream to > a KeyedStream but performs no shuffling if the caller is certain that the > DataStream is already partitioned by the given key. > > >
