Hi, Flink works very well with Kafka if you wish to stream data. Following is how I am streaming data with Kafka and Flink.
FlinkKafkaConsumer08<Event> kafkaConsumer = new FlinkKafkaConsumer08<>(KAFKA_AVRO_TOPIC, avroSchema, properties); DataStream<Event> messageStream = env.addSource(kafkaConsumer); Is there a way to do a micro batch operation on the data coming from Kafka? What I want to do is to reduce or aggregate the events coming from Kafka. For instance I am getting 40000 events per second from Kafka and what I want is to group 2000 events into one and send it to my microservice for further processing. Can I use the Flink DataSet API for this or should I go with Spark or some other framework? Thanks & Regards Zeeshan Alam
