Re: Kafka Connect - HDFS or FileStream

2019-05-15 Thread Vinay Jain
Appreciate if somebody has experience on the above and respond to the same. Logically it should work if we have a property set hdfs.url = file:///home/user/data On Wed, May 15, 2019 at 5:06 AM Vinay Jain wrote: > Redirecting to a file will not work, we would not be able to creat

Re: Kafka Connect - HDFS or FileStream

2019-05-14 Thread Vinay Jain
Can you just use kafka-console-consumer and just redirect the output into a > file? > > -hans > > > On Mon, May 13, 2019 at 1:55 PM Vinay Jain wrote: > > > Hi > > > > The data needs to be transferred to some other system in other network, > and > > due

Re: Kafka Connect - HDFS or FileStream

2019-05-13 Thread Vinay Jain
. Regards Vinay Jain On Mon, May 13, 2019 at 3:41 PM Robin Moffatt wrote: > Can you explain more about why you're writing a file with the data? > Presumably, this is for another application to consume; could it not take > the data from Kafka directly, whether with a native client or

Kafka Connect - HDFS or FileStream

2019-05-12 Thread Vinay Jain
Hi I would like to consume a Topic and save the AVRO messages in local directory in the AVRO file formats. As per Kafka Connect File Stream documentation, it is not for production use. Other Option I am thinking to use Kafka Connect - HDFS Sink but I am not sure if it can also write to the Local