I agree with Alex, that a very short retention is probably a good solution, if you want to minimize data loss when Flink is restarting.
If some small degree of data loss is okay, you could also consider implementing a systlog connector for Flink. I think that it is probably quite easy to create a Flink data source that implements the syslog protocol. Flink's worker nodes would just look like a cluster of syslog servers you can send data to. On Sat, Mar 4, 2017 at 11:00 PM, Alex Ott <alex...@gmail.com> wrote: > Why not to use Kafka with very short retention time? > > On Sat, 4 Mar 2017 at 14:16, trang anh (Joey) <anhtv010...@gmail.com> > wrote: > > > Hi Alex, > > > > That is one of solutions we are looking right now, but the problem with > us > > is the our system will be processing the huge data, few TB daily basis, > > > > So that we don’t want store the data in our KAFKA cluster, we just want > > get event data, do some processing and emitting to downstream systems > > > > Please look at this link for more detail of our challenges and our > current > > approach for that > > > > Any advice would be grateful > > > > > > http://stackoverflow.com/questions/42525680/best- > practice-for-flink-streaming-processing > > < > > http://stackoverflow.com/questions/42525680/best- > practice-for-flink-streaming-processing > > > > > > > > > > > > > > On 4 Mar 2017, at 5:36 PM, Alex Ott <alex...@gmail.com> wrote: > > > > > > Why not to send syslog data to Kafka, via rsyslog, or something like? > > From > > > my point of view this could be more flexible solution > > > > > > On Sat, Mar 4, 2017 at 1:39 AM, trang anh (Joey) < > anhtv010...@gmail.com> > > > wrote: > > > > > >> Hi, > > >> > > >> I am looking for the Flink syslog source connector which can source > > data > > >> directly from syslog event as input data stream to Flink > > >> > > >> I have checked the source connectors list, but seems there no such > thing > > >> available. > > >> > > >> Appreciate any advise? > > >> > > >> Regards > > >> > > >> Joey > > > > > > > > > > > > > > > -- > > > With best wishes, Alex Ott > > > http://alexott.net/ > > > Twitter: alexott_en (English), alexott (Russian) > > > Skype: alex.ott > > > > -- > With best wishes, Alex Ott > http://alexott.net/ > Twitter: alexott_en (English), alexott (Russian) > Skype: alex.ott >
