Have you looked at Pinterest Secor? (
http://engineering.pinterest.com/post/84276775924/introducing-pinterest-secor
)
Cheers, Robert
On Mon, Jun 16, 2014 at 5:17 AM, Mark Godfrey wrote:
> There is Bifrost, which archives Kafka data to S3:
> https://github.com/uswitch/bifrost
>
> Obviously tha
There is Bifrost, which archives Kafka data to S3:
https://github.com/uswitch/bifrost
Obviously that's a fairly specific archive solution, but it might work for
you.
Mark.
On Mon, Jun 16, 2014 at 11:02 AM, Anatoly Deyneka
wrote:
> Hi all,
>
> I'm looking for the way of archiving data.
> The d
You should do this as a consumer (i.e. "archiveDataConsumer")
Take a look at the AWS section of the eco system
https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem (e.g.
https://github.com/pinterest/secor ).
Also the tools is a good place to check out
https://cwiki.apache.org/confluence/di
Hi all,
I'm looking for the way of archiving data.
The data is hot for few days in our system.
After that it can rarely be used. Speed is not so important for archive.
Lets say we have kafka cluster and storage system.
It would be great if kafka supported moving data to storage system instead
of
