Hi Palle,
this sounds indeed like a good use case for Flink.
Depending on the complexity of the aggregated historical views, you can
implement a Flink DataStream program which builds the views on the fly,
i.e., you do not need to periodically trigger MR/Flink/Spark batch jobs to
compute the views
I see the flow to be as below:
LogStash->Log Stream->Flink ->Kafka->Live Model
|
Mongo/HBASE
The Live Model will again be Flink streaming data sets from Kakfa.
There you analyze the incoming stream for the certain value and
HI Palle,
I am a beginner in Flink.
However, I can say something about your other questions:
1. It is better to use Spark to create aggregate views. It is a lot
faster than MR. You could use either batch or streaming mode in spark based on
your needs.
2. If your a
