Redirecting to a file will not work, we would not be able to create a different filenames after some time or filesize. HDFS connect sink already has those options , also i could use some minor transformations while using the connect
On Tue, May 14, 2019 at 7:28 AM Hans Jespersen <h...@confluent.io> wrote: > Can you just use kafka-console-consumer and just redirect the output into a > file? > > -hans > > > On Mon, May 13, 2019 at 1:55 PM Vinay Jain <vina...@gmail.com> wrote: > > > Hi > > > > The data needs to be transferred to some other system in other network, > and > > due to some security reasons, the other systems cannot be exposed . So > the > > available mechanism is file based integration. Is there a production > ready > > Kafka connect adapter which can create files in local directory. > > > > Regards > > Vinay Jain > > > > On Mon, May 13, 2019 at 3:41 PM Robin Moffatt <ro...@confluent.io> > wrote: > > > > > Can you explain more about why you're writing a file with the data? > > > Presumably, this is for another application to consume; could it not > take > > > the data from Kafka directly, whether with a native client or over the > > REST > > > proxy? > > > Oftentimes local files are unnecessary 'duck tape' for integration that > > can > > > be done in a better way. > > > > > > > > > -- > > > > > > Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff > > > > > > > > > On Mon, 13 May 2019 at 01:35, Vinay Jain <vina...@gmail.com> wrote: > > > > > > > Hi > > > > > > > > I would like to consume a Topic and save the AVRO messages in local > > > > directory in the AVRO file formats. As per Kafka Connect File Stream > > > > documentation, it is not for production use. > > > > > > > > Other Option I am thinking to use Kafka Connect - HDFS Sink but I am > > not > > > > sure if it can also write to the Local directory if we pass in the > > > variable > > > > hdfs.url the URL for local file system instead of HDFS path. > > > > > > > > Will this work or are there any other ready made options which can be > > > used > > > > for the same. > > > > > > > > Regards > > > > > > > > Vinay > > > > > > > > > >
