Maybe you need a different KafkaConnect source? I think this one may fit your needs better: https://github.com/jcustenborder/kafka-connect-spooldir
It was built to copy data from files in directory into Kafka... On Wed, Aug 10, 2016 at 8:56 AM, Jeyhun Karimov <je.kari...@gmail.com> wrote: > Hi community, > > I am using Kafka-Connect to import text files to kafka. Because I have to > give exact file name (like ./data.txt) and not the parent directory (./*) > as an input to Kafka-Connect, how can I import files in particular > directory as they are created in runtime? The solutions that come to my > mind are below but I don't know about their efficiency. > - I may create a new file in particular directory (after the kafka-connect > started already) and start new connect instance to point that file > -or I may write to only one file (just one ./data.txt) all the time (and no > other data files created) and kafka-connect will transfer the data only > from one file and it will from where it left (as data writes are made) > -or any other way to efficiently handle this use case. > > My second question is, is it safe to override the data transfer methods of > kafka-connect? For example I want to put thread.sleeps in kafka-streams > side while transferring data and see the behaviour in kafka side or in > application side. You can think of as simulation of load. > > Cheers > Jeyhun > > > -- > -Cheers > > Jeyhun -- Gwen Shapira Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter | blog
