Re: Flink Batch Processing with Kafka

2016-08-03 Thread Prabhu V
If your environment is not kerberized (or if you can offord to restart the job every 7 days), a checkpoint enabled, flink job with windowing and the count trigger, would be ideal for your requirement. Check the api's on flink windows. I had something like this that worked stream.keyBy(0).countW

Flink Batch Processing with Kafka

2016-08-03 Thread Alam, Zeeshan
Hi, Flink works very well with Kafka if you wish to stream data. Following is how I am streaming data with Kafka and Flink. FlinkKafkaConsumer08 kafkaConsumer = new FlinkKafkaConsumer08<>(KAFKA_AVRO_TOPIC, avroSchema, properties); DataStream messageStream = env.addSource(kafkaConsumer); Is th