Kafka Connect can definitely be used for this -- it's one of the reasons we designed it with standalone mode ( http://docs.confluent.io/3.0.0/connect/userguide.html#workers). For the specific connector, we include a very simple File connector with Kafka which will just take each line and send it to Kafka with a simple String schema. However, it's intentionally very limited (as an easy reference for starting new connectors), so it doesn't do any log parsing, multi-line handling, multiple files, tracking rolling files, etc. For that you'd need to write a new connector. The closest one I'm familiar with is a syslog connector, which you can find listed on connectors.confluent.io
-Ewen On Tue, May 3, 2016 at 2:07 PM, David Birdsong <david.birds...@gmail.com> wrote: > On Tue, May 3, 2016 at 1:00 PM Banias H <banias4sp...@gmail.com> wrote: > > > I should add Flume is not an option for various reasons. > > > > On Tue, May 3, 2016 at 2:49 PM, Banias H <banias4sp...@gmail.com> wrote: > > > > > We use Kafka (0.9.x) internally in our pipeline and now we would like > to > > > ingest application logs sitting in local file system of servers > external > > to > > > the Kafka cluster. > > > > > > We could write a producer program running on the application servers to > > > push files to Kafka. However we wonder if we can leverage Kafka Connect > > to > > > pull files into Kafka. It requires a connector to access local file > > systems > > > and I am not sure if I can make use of existing connectors or I need to > > > write one. > > > > > > Any thoughts or reference would be helpful. > > > > heka - great at reading arbitrary inputs, files being a first-class input, > can write to kafka. next release should see use of librdkafka which should > handle leader changeovers more smoothly than the existing go library. > > > > > > > > > Thanks, > > > B > > > > > > -- Thanks, Ewen
