Liu, you first need to figure out what TCP data you want to collect. Is there a possibility that this data can be collected at some central router/gateway using SNMP?
If not SNMP then you can definitely run something like wireshark or write up your own tool using a library like libpcap and collect data passing through the network card. I'm not sure if this is what you want. Once you've decided on the data that you want to collect, it is definitely possible to use flume to collect it and the easiest would be to write a utility to consume that data and convert it to avro and then use the avro source on the flume side. That's my suggestion. Write your own tool to collect data, bundle it into avro events and pass them on to flume. On Fri, Aug 1, 2014 at 11:25 AM, Liu Blade <hafzc...@gmail.com> wrote: > Hi folks, > > Sorry didn't clarify my problem. The problem has two folds: (1) use which > way to collect incoming TCP streams from external connections, and it must > be made on the fly; (2)use which method as Flume source, e.g., syslogTcp, > Avro. > > It seems syslog is unable to tap into TCP connections. Look forward to > your opinions. > > Thanks, > > > > 2014-08-01 11:17 GMT+08:00 Liu Blade <hafzc...@gmail.com>: > > Dear all, >> >> The scenario is we want to collect data over TCP connection which is send >> to backend database server. But it is not possible to use an intrusive way, >> which means we would not collect data on servers. >> >> Is that possible to use libpcap/winpcap to tap into TCP stream, convert >> it to Avro/Thrift, and then send to Flume source? >> >> Very appreciate your suggestions. Please indicate if there are better >> options. >> >> Cheers, >> Blade >> >> > >
