Ron, The best way of doing this would be to use the ConsoleProducer. Basically, it reads data from the console and parses it using the message "reader" which by default is the LineReader. In this case, you can either write your own SquidMessageReader that understands the Squid access format [1] and sends JSON data to Kafka or use the inbuilt LineReader al though that wouldn't attach any structure to your data.
If you do end up writing a squid message reader, would you mind putting it up on github. I can see it being useful to the community - https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem Thanks, Neha 1. http://wiki.squid-cache.org/Features/LogFormat On Thu, Jan 10, 2013 at 10:24 AM, Jun Rao <jun...@gmail.com> wrote: > The following wiki describes the operational part of Kafka. > https://cwiki.apache.org/confluence/display/KAFKA/Operations > > To get your log into Kafka, if this log4j data, you may consider adding a > KafkaLog4jAppender. Otherwise, you can probably use ConsoleProducer. You > will still need to deal with things like log rolling yourself though. > > Thanks, > > Jun > > On Thu, Jan 10, 2013 at 9:59 AM, Ron Tsoref <chie...@gmail.com> wrote: > > > Hi. > > > > I currently have a couple of servers running a reverse proxy software > that > > creates access logs in the squid log format (Here is a > > screenshot<http://www.arnut.com/pics/itpro/LogFile22.jpg>showing the > > file's formation). > > > > As I understand it, Kafka is a good solution for handling the connection > > between the proxies (that actually create the logs) and Storm ( in order > to > > analyze them in real-time). > > > > Right now, I'm looking for a way to gather the logs from each server with > > Kafka and how to configure this Kafka instances. > > > > I would appreciate any recommendation on how to do this, or any other > > source regarding this kind of setup. > > > > Is there any production-ready producer that can handle this log > aggregating > > task? Basically, the ideal solution for me would be to generate a message > > for each one of the lines in the logs for each server, and then analyzing > > with Storm shouldn't be a big problem. > > > > Thanks, > > > > Ron > > >
