Set up scripts to send alerts sooner ? There isn't a built in way in flume so
you will have to setup monitoring separately
On Mon, Feb 27, 2017 at 8:57 AM +0530, "Suresh V" wrote:
Hello,
Is there a way to set up an alert mechanism by email immediately when a flume
agent fails d
Not sure I understood the question correctly but you might be able to use
the regex interceptor to do what you want.
--
Sharninder
On Tue, Feb 16, 2016 at 1:56 PM, Gonzalo Herreros
wrote:
> I don't think that's possible without writing/reusing custom code.
> You would need a
Monitoring requirements are different for different teams but have you looked
at collectd? That with graphite/grafana could work for you.
> On 20-Nov-2015, at 10:00 AM, Hemanth Abbina wrote:
>
> Just a follow up question.
>
> We need a remote Linux server monitoring tool, to monitor our clu
ransaction will surely help withe reducing IO on the flume server.
Thanks a lot.
--
Sharninder
On Sun, Sep 27, 2015 at 3:16 PM, Gonzalo Herreros
wrote:
> There are subtle but significant differences.
>
> When you configure in the sink: "batchSize" you are specifying how m
Anyone ?
> On 25-Sep-2015, at 3:51 PM, Sharninder wrote:
>
> Hi,
>
> We want to move to the built-in kafka sink from our own custom implementation
> and I have a question about the batchsize config parameter.
>
> Looking at the code of the sink, I can tell that
is the difference between this variable and the kafka
batch.num.messages parameter?
Is the flume parameter necessary ?
--
Sharninder
;,
> "StopTime": "0",
> "RollbackCount": "0",
> "StartTime": "1443053580579",
> "BatchUnderflowCount": "0"
> },
> "SOURCE.my-kafka-source": {
> "KafkaEventGetTimer": "886",
> "OpenConnectionCount": "0",
> "Type": "SOURCE",
> "AppendBatchAcceptedCount": "0",
> "AppendBatchReceivedCount": "0",
> "EventAcceptedCount": "6500",
> "AppendReceivedCount": "0",
> "StopTime": "0",
> "StartTime": "1443053581472",
> "EventReceivedCount": "7000",
> "KafkaCommitTimer": "246",
> "AppendAcceptedCount": "0"
> }
> }
>
>
> Thanks!
>
--
--
Sharninder
gt; On Fri, Sep 18, 2015 at 10:28 PM, Sharninder wrote:
>
>> Use the Kafka console utility to see what's being fed to Kafka.
>>
>>
>> On Saturday, September 19, 2015, Doug McClure wrote:
>>
>>> Ideally as they appear passing through broker in topic and
being collected in the spooldir
>> source and sent via kafka sink, specifically the values of fileHeader and
>> basenameHeader data.
>>
>> Tks!
>>
>> Doug
>>
>
>
--
--
Sharninder
implemented and,
honestly, it shouldn't be more than a couple of hours of work, most likely
less.
--
Sharninder
On Fri, Sep 4, 2015 at 11:34 PM, Sutanu Das wrote:
> Huge THANKS Hari.
>
>
>
> I just did this per your recommendation/docs – and it worked !!!, I can
> n
e generic enough. The file channel is
durable but depends on a lot of things including the kind of disks etc. The
type of messages being sent might affect the throughput too. In short you'll
have to test for your use case and your hardware.
--
Sharninder
The kind of messages and your particular use case will really define the amount
of heap you need. Why start with 200m. Set both Xms and Xmx to one value, say
1g and test from there.
Sharninder
_
From: mahendran m
Sent: Friday, April 24, 2015 9:39 am
Subject
yes a source. My bad :)
On Fri, Oct 17, 2014 at 9:54 PM, Anthony Biacco wrote:
> You mean a custom source, right? Not a sink.
>
> -Tony
>
>
> On Thu, Oct 16, 2014 at 10:47 PM, Sharninder wrote:
>
>> You need to write a custom sink using the flume sdk. I'm not su
You need to write a custom sink using the flume sdk. I'm not sure what are
the APIs provided by the gnip4 jar but it shouldn't be too difficult to
write the flume part.
--
Sharninder
On Fri, Oct 17, 2014 at 2:08 AM, Anthony Biacco wrote:
>
> We were using flume with twitter&#
Thanks Jeff. I'll take a look at the multipart source too.
On Thu, Oct 16, 2014 at 8:52 PM, Jeff Lord wrote:
> You will get better perf out of the multiport syslog source
>
>
> On Wednesday, October 15, 2014, Sharninder wrote:
>
>> I just looked at the existing syslog
I just looked at the existing syslogtcp source and it seems it does take
pains to parse the hostname from the message and I think that is the best
bet for me. Ofcourse, it might fail for a few devices, but I'll just have
to think of something else for those.
--
Sharninder
On Thu, Oct 16,
Yes Jeff. That's a possiblity but I'm not sure (actually pretty sure) that
there would be a some random device which will not send their logs in the
proper format and my regex will break. This is the way I'll implement it if
I can't find anything better.
Thanks,
Sharninder
Yes, I did think of that but that just seems like a hack and doesn't scale
too much.
Ideally, I should be able to just look at the remote host from the tcp
headers somewhere and add that info to the flume event header.
--
Sharninder
On Thu, Oct 16, 2014 at 10:17 AM, Hari Shreed
eader somehow? I can then use
either an interceptor to read the header or even a custom sink to deal with
events based on the hostname.
--
Sharninder
Implementing a new sink isn't that difficult really so if your requirements
aren't met with the current file sink, by all means write your own. Take a look
at the existing code and the Dev guide.
> On 26-Sep-2014, at 5:47 pm, jean garutti wrote:
>
> Hello,
> For differents purposes, i want to
Have you increased the JVM heap size or running with the default 20M only?
--
Sharninder
> On 21-Sep-2014, at 3:08 am, terreyshih wrote:
>
> BTW, no exceptions are thrown. It just started the agent-shutdown-hook.
>
> thx
>
>> On Sep 20, 2014, at 2:33 PM, terr
g else in the interceptor/morphline.
--
Sharninder
On Fri, Sep 5, 2014 at 8:23 AM, Kevin Warner
wrote:
> Hello All,
> We have the following configuration:
> Source->Channel->Sink
>
> Now, the source is pointing to a folder that has lots of json files. The
> channel is file ba
Yes, you can compose avro messages in php. As to your question about which
is a better performances wise, I'm not really sure about that, but since
you already have a script generating syslog, why not just start using it.
To add header values to each event you can add an interceptor to the
topology
I'm not sure without looking at the exact usecase, but maybe you can use
something like haproxy?
--
Sharninder
On Thu, Aug 14, 2014 at 4:08 PM, Mohit Durgapal
wrote:
> Hi Sharninder,
>
> Thanks for the response. The load balancing is not based on header. To
> simplify, le
To add headers to the events, you can either send proper avro formatted
packets (which have a header) to an avro source, or implement a custom
interceptor to add headers after they're received by the syslog source.
There is a static interceptor bundled with flume that you can use. The
problem with
The purpose of a sink is to take events from a channel and pass it on to a
further destination. That destination can be anything, including your
another program listening on rpc.
On Tue, Aug 12, 2014 at 1:26 PM, terreyshih wrote:
> Hi,
>
> It seems to me that the intention of the flume is to be
Liu, you first need to figure out what TCP data you want to collect. Is
there a possibility that this data can be collected at some central
router/gateway using SNMP?
If not SNMP then you can definitely run something like wireshark or write
up your own tool using a library like libpcap and collect
As far as I know, there is no (open source) implementation of an S3 source,
so yes, you'll have to implement your own. You'll have to implement a
Pollable source and the dev documentation has an outline that you can use.
You can also look at the existing Execsource and work your way up.
As far as
here any way to find that the transfer is
> complete?
>
> Thanks,
> Anand.
>
>
> On 07/26/2014 01:38 PM, Sharninder wrote:
>
> If you really want to add files to HDFS, use the spool directory source
> which is much more reliable. If you do want to use the exec source, n
If you really want to add files to HDFS, use the spool directory source
which is much more reliable. If you do want to use the exec source, no
point using cat since that's as good as cp'ing the file the HDFS, use tail
-f rather.
--
Sharninder
On Sat, Jul 26, 2014 at 9:34 AM,
I did look at the developer guide and was a little confused since there seem to
be a couple of omissions in it. In particular, the even based source isn't
explained very well. I'm new to Java (and flume) so maybe that's just me.
Your explanation helps.
--
Sharninder
> On
will I be opening another connection again for the second event?
When is stop() called?
Has anyone implemented a websocket sink that I could take a look at?
--
Sharninder
use-case.
The reason the author might have moved to kafka, and I'm just speculating
here, is that kafka provides him better buffering support for exactly the
case I've written above.
HTH
Sharninder
On Mon, Jun 30, 2014 at 7:57 PM, Máté Gulyás wrote:
> Hi!
>
> I would l
ble ?
Hope my description was clear enough?
--
Sharninder
oming from is not mentioned, or needed.
--
Sharninder
On Wed, Jun 25, 2014 at 5:03 PM, wrote:
> My question is where is that input avro format file mentioned in the
> configuration file.
>
> Sent from my iPhone
>
> On Jun 25, 2014, at 2:06 PM, Sharninder wrote:
>
>
Did you try using the spooldir source with an hdfs sink? What problems did
you face?
--
Sharninder
On Wed, Jun 25, 2014 at 12:15 PM, kishore alajangi <
alajangikish...@gmail.com> wrote:
> Hi Flume Experts,
>
> Could anybody help me to store avro files located in my localfilesys
That is an avro source, which means the producer or the app producing
events needs to send it's data in the avro format. This is also used for
building a multi-hop flume pipeline. Is this what you're asking?
On Wed, Jun 25, 2014 at 12:37 PM, kishore alajangi <
alajangikish...@gmail.com> wrote:
;>
>>> tier1.channels.channel1.capacity = 10*
>>> tier1.sources.source1.deserializer.maxLineLength = 32768
>>>
>>>
>>>
>>> the command I used is
>>>
>>>
>>>
>>> ./flume-ng agent --conf ./conf/ -f bin/e
e, take a look at the spool directory source
or the exec source and you should be able to put something together that'll
push data through flume to hadoop.
--
Sharninder
slightly unrelated but I've seen the flume file source also "hang" a few
times on me. Restarting solved my problem. Can't really say more since I
don't have a lot of data but flume failing on you is a problem and is the
reason a lot of people develop their own "custom collectors".
On a related not
40 matches
Mail list logo
