Thanks for your suggestions. I think if kafka connect provides the same functionality as flume and storm, why should we go for another infrastructure investment.
Kafka Connect effectively copies data from Kafka topic to HDFS through connector. It supports avro as well as parquet, I am looking if we can use it to load plain text data. Cheers Pari On 22 June 2016 at 12:34, Lohith Samaga M <lohith.sam...@mphasis.com> wrote: > Hi, > You can use Storm also, Here you have the option of rotating the > file. You can also write to Hive directly. > > Best regards / Mit freundlichen Grüßen / Sincères salutations > M. Lohith Samaga > > > > > -----Original Message----- > From: Mudit Kumar [mailto:mudit.ku...@askme.in] > Sent: Wednesday, June 22, 2016 12.32 > To: users@kafka.apache.org; d...@kafka.apache.org > Subject: Re: Kafka HDFS Connector > > I think you can use flume also. > > Thanks, > Mudit > > > > > On 6/22/16, 12:29 PM, "Pariksheet Barapatre" <pari.data...@gmail.com> > wrote: > > >Anybody have any idea on this? > > > >Thanks > >Pari > > > >On 20 June 2016 at 14:36, Pariksheet Barapatre <pari.data...@gmail.com> > >wrote: > > > >> Hello All, > >> > >> I have data coming from sensors into kafka cluster in text format > >> delimited by comma. > >> > >> How to offload this data to Hive periodically from Kafka. I guess, > >> Kafka Connect should solve my problem but when I checked > >> documentation, examples have only avro formatted data. Can you please > >> provide some knowledge on this. > >> > >> Many Thanks > >> Pari > >> > > Information transmitted by this e-mail is proprietary to Mphasis, its > associated companies and/ or its customers and is intended > for use only by the individual or entity to which it is addressed, and may > contain information that is privileged, confidential or > exempt from disclosure under applicable law. If you are not the intended > recipient or it appears that this mail has been forwarded > to you without proper authority, you are notified that any use or > dissemination of this information in any manner is strictly > prohibited. In such cases, please notify us immediately at > mailmas...@mphasis.com and delete this mail from your records. >
