Fernando, have a look -
http://blog.sematext.com/2014/10/06/top-5-most-popular-log-shippers/
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Wed, Jan 28, 2015 at 1:39 PM, Fernando O. wrote:
> Hi all,
> I'
Thanks David. This looks to be interesting. Will definitely test this out
to see whether this solves our problem.
On Thu, Jan 29, 2015 at 8:29 AM, David Morales wrote:
> Existing "tail" source is not the best choice in your scenario, as you have
> pointed out.
>
> SpoolDir could be a solution if
Existing "tail" source is not the best choice in your scenario, as you have
pointed out.
SpoolDir could be a solution if your log file rotation policy is very low
(5 minutes, for example), but then you have to deal with a huge number of
files in the folder (slower listings).
There is a proposal f
We have been using Flume to solve a very similar usecase. Our servers write
the log files to a local file system, and then we have flume agent which
ships the data to kafka.
Flume you can use as exec source running tail. Though the exec source runs
well with tail, there are issues if the agent goe
The big syslog daemons support Kafka since a while back.
rsyslog:
http://www.rsyslog.com/doc/master/configuration/modules/omkafka.html
syslog-ng:
https://czanik.blogs.balabit.com/2015/01/syslog-ng-kafka-destination-support/#more-1013
And Bruce might be of interest aswell:
https://github.com/tagg
Logstash
--
Colin Clark
+1 612 859 6129
Skype colin.p.clark
> On Jan 28, 2015, at 10:47 AM, Gwen Shapira wrote:
>
> It sounds like you are describing Flume, with SpoolingDirectory source
> (or exec source running tail) and Kafka channel.
>
>> On Wed, Jan 28, 2015 at 10:39 AM, Fernando O. wro
It sounds like you are describing Flume, with SpoolingDirectory source
(or exec source running tail) and Kafka channel.
On Wed, Jan 28, 2015 at 10:39 AM, Fernando O. wrote:
> Hi all,
> I'm evaluating using Kafka.
>
> I liked this thing of Facebook scribe that you log to your own machine and
>
Something like Heka but lightweight :D
On Wed, Jan 28, 2015 at 3:39 PM, Fernando O. wrote:
> Hi all,
> I'm evaluating using Kafka.
>
> I liked this thing of Facebook scribe that you log to your own machine and
> then there's a separate process that forwards messages to the central
> logger.
On Tue, Jan 15, 2013 at 3:12 PM, Corbin Hoenes wrote:
> +1 how about posting yours to GitHub?
> Sounds like a good contrib project.
>
There is nothing to post at the moment as we're currently in the
requirements gathering phase :) Potentially, we might have a contrib
project along the lines
of
Jay,
Thanks for your insight! More comments are below.
On Tue, Jan 15, 2013 at 3:18 PM, Jay Kreps wrote:
> I can't speak for all users, but at LinkedIn we don't do this. We just run
> Kafka as a high-availability system (i.e. something not allowed to be
> down). These kind of systems require
I can't speak for all users, but at LinkedIn we don't do this. We just run
Kafka as a high-availability system (i.e. something not allowed to be
down). These kind of systems require more care, but we already have a
number of such data systems. We chose this approach because local queuing
leads to d
+1 how about posting yours to GitHub?
Sounds like a good contrib project.
Sent from my iPhone
On Jan 15, 2013, at 12:29 PM, Stan Rosenberg wrote:
> Hi,
>
> In out current data ingestion system, producers are resilient in the sense
> that if data cannot be reliably published (e.g., network is
12 matches
Mail list logo
