This depends a little bit on your requirements.
If it just about reading data from HDFS and writing it into Kafka, then
it should be possible to simply wrap a KafkaProducer in a
RichMapFunction that you use as a sink in your DataSet program.
However you could also use the Streaming API for that.
On 05.06.2018 00:39, Oleksandr Nitavskyi wrote:
Hello Squirrels,
Flink has a wonderful Kafka connector. We need to move data from HDFS
to Kafka. Confluent is proposing to use Kafka-connect for this, but
probably it can be easier to use Flink for such task, much higher
abstraction, less details to manage, easier for our context.
Do you know is there a way to output data into the Kafka using the
Batch approach?
Thanks
Kind Regards
Oleksandr Nitavskyi
