The following wiki describes the operational part of Kafka. https://cwiki.apache.org/confluence/display/KAFKA/Operations
To get your log into Kafka, if this log4j data, you may consider adding a KafkaLog4jAppender. Otherwise, you can probably use ConsoleProducer. You will still need to deal with things like log rolling yourself though. Thanks, Jun On Thu, Jan 10, 2013 at 9:59 AM, Ron Tsoref <chie...@gmail.com> wrote: > Hi. > > I currently have a couple of servers running a reverse proxy software that > creates access logs in the squid log format (Here is a > screenshot<http://www.arnut.com/pics/itpro/LogFile22.jpg>showing the > file's formation). > > As I understand it, Kafka is a good solution for handling the connection > between the proxies (that actually create the logs) and Storm ( in order to > analyze them in real-time). > > Right now, I'm looking for a way to gather the logs from each server with > Kafka and how to configure this Kafka instances. > > I would appreciate any recommendation on how to do this, or any other > source regarding this kind of setup. > > Is there any production-ready producer that can handle this log aggregating > task? Basically, the ideal solution for me would be to generate a message > for each one of the lines in the logs for each server, and then analyzing > with Storm shouldn't be a big problem. > > Thanks, > > Ron >
