Hi Palle,
this sounds indeed like a good use case for Flink.
Depending on the complexity of the aggregated historical views, you can
implement a Flink DataStream program which builds the views on the fly,
i.e., you do not need to periodically trigger MR/Flink/Spark batch jobs to
compute the views
I see the flow to be as below:
LogStash->Log Stream->Flink ->Kafka->Live Model
|
Mongo/HBASE
The Live Model will again be Flink streaming data sets from Kakfa.
There you analyze the incoming stream for the certain value and
Grüßen / Sincères salutations
M. Lohith Samaga
-Original Message-
From: pa...@sport.dk [mailto:pa...@sport.dk
Sent: Friday, May 06, 2016 16.23
To: user@flink.apache.org
Subject: Where to put live model and business logic in Hadoop/Flink BigData
system
Hi there.
We are putting together
Hi there.
We are putting together some BigData components for handling a large amount of
incoming data from different log files and perform some analysis on the data.
All data being fed into the system will go into HDFS. We plan on using
Logstash, Kafka and Flink for bringing data from the log
