One more question, If I only need watermark's logic, not keyedStream, why not provide methods such as writeDataStream and readDataStream. It uses the similar methods for kafka producer sink records and broadcast watermark to partitions and then kafka consumers read it and regenerate the watermark. I think it will be more general? In this way, the kafka consumer reads the stream from kafka, and can continue to call keyBy to get a keyedStream. I don't know why KafkaShuffle only considers the 'keyedStream' case.
Piotr Nowojski <pnowoj...@apache.org> 于2021年3月4日周四 下午3:54写道： > Great :) > > Just one more note. Currently FlinkKafkaShuffle has a critical bug [1] > that probably will prevent you from using it directly. I hope it will be > fixed in some next release. In the meantime you can just inspire your > solution with the source code. > > Best, > Piotrek > > > [1] https://issues.apache.org/jira/browse/FLINK-21317 > > czw., 4 mar 2021 o 03:48 yidan zhao <hinobl...@gmail.com> napisał(a): > >> Yes, you are right and thank you. I take a brief look at what >> FlinkKafkaShuffle is doing, it seems what I need and I will have a try. >> >>>
