Appreciate if somebody has experience on the above and respond to the same. Logically it should work if we have a property set hdfs.url = file:///home/user/data
On Wed, May 15, 2019 at 5:06 AM Vinay Jain <vina...@gmail.com> wrote: > Redirecting to a file will not work, we would not be able to create a > different filenames after some time or filesize. HDFS connect sink already > has those options , also i could use some minor transformations while using > the connect > > On Tue, May 14, 2019 at 7:28 AM Hans Jespersen <h...@confluent.io> wrote: > >> Can you just use kafka-console-consumer and just redirect the output into >> a >> file? >> >> -hans >> >> >> On Mon, May 13, 2019 at 1:55 PM Vinay Jain <vina...@gmail.com> wrote: >> >> > Hi >> > >> > The data needs to be transferred to some other system in other network, >> and >> > due to some security reasons, the other systems cannot be exposed . So >> the >> > available mechanism is file based integration. Is there a production >> ready >> > Kafka connect adapter which can create files in local directory. >> > >> > Regards >> > Vinay Jain >> > >> > On Mon, May 13, 2019 at 3:41 PM Robin Moffatt <ro...@confluent.io> >> wrote: >> > >> > > Can you explain more about why you're writing a file with the data? >> > > Presumably, this is for another application to consume; could it not >> take >> > > the data from Kafka directly, whether with a native client or over the >> > REST >> > > proxy? >> > > Oftentimes local files are unnecessary 'duck tape' for integration >> that >> > can >> > > be done in a better way. >> > > >> > > >> > > -- >> > > >> > > Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff >> > > >> > > >> > > On Mon, 13 May 2019 at 01:35, Vinay Jain <vina...@gmail.com> wrote: >> > > >> > > > Hi >> > > > >> > > > I would like to consume a Topic and save the AVRO messages in local >> > > > directory in the AVRO file formats. As per Kafka Connect File Stream >> > > > documentation, it is not for production use. >> > > > >> > > > Other Option I am thinking to use Kafka Connect - HDFS Sink but I am >> > not >> > > > sure if it can also write to the Local directory if we pass in the >> > > variable >> > > > hdfs.url the URL for local file system instead of HDFS path. >> > > > >> > > > Will this work or are there any other ready made options which can >> be >> > > used >> > > > for the same. >> > > > >> > > > Regards >> > > > >> > > > Vinay >> > > > >> > > >> > >> >
