Have a look at Flume Client SDK, this would make implementation very easy.
Basically you implement a wrapper for libpcap, run it anywhere and you can
send Avro events to Avro source.
On Fri, Aug 1, 2014 at 1:27 PM, Blade Liu wrote:
> Hi Sharninder and Ashnish,
>
> Thanks for your nice suggestio
Hi Sharninder and Ashnish,
Thanks for your nice suggestions. I agree one good solution would be
writing some tools to glue libpcap, Avro and Flume.
2014-08-01 14:27 GMT+08:00 Sharninder :
> Liu, you first need to figure out what TCP data you want to collect. Is
> there a possibility that this d
Liu, you first need to figure out what TCP data you want to collect. Is
there a possibility that this data can be collected at some central
router/gateway using SNMP?
If not SNMP then you can definitely run something like wireshark or write
up your own tool using a library like libpcap and collect
Can you please clarify a bit more on "made on fly". I interpret this as
running in promiscuous mode to listen for data and create events out of it.
Is this correct or I am completely off the track?
On Fri, Aug 1, 2014 at 11:25 AM, Liu Blade wrote:
> Hi folks,
>
> Sorry didn't clarify my problem
Hi folks,
Sorry didn't clarify my problem. The problem has two folds: (1) use which
way to collect incoming TCP streams from external connections, and it must
be made on the fly; (2)use which method as Flume source, e.g., syslogTcp,
Avro.
It seems syslog is unable to tap into TCP connections. Lo
Mohit,
Thanks for your reply. The problem here is that we can only capture TCP
data on the fly, not on hosts. Besides, if the server is Windows, it may
not provide syslog-like utility.
2014-08-01 12:04 GMT+08:00 Mohit Durgapal :
> I am not sure if that's exactly what you need but have you tried
I am not sure if that's exactly what you need but have you tried the syslog
tcp source? It can listen and consume events from a tcp queue on a specific
host & port.
On Fri, Aug 1, 2014 at 8:47 AM, Liu Blade wrote:
> Dear all,
>
> The scenario is we want to collect data over TCP connection which
I've used a utility 'tcpflow' in the past to capture streams. It writes
them to files, could potentially read those into flume?
On Jul 31, 2014 8:18 PM, "Liu Blade" wrote:
> Dear all,
>
> The scenario is we want to collect data over TCP connection which is send
> to backend database server. But i
