Yes, I believe what you're looking for is what Dave described. Here's the source of that interface https://github.com/confluentinc/kafka-connect-hdfs/blob/master/src/main/java/io/confluent/connect/hdfs/Format.java There already exists a StringConverter that should handle the conversion in and out of the connect data format in your case https://kafka.apache.org/0100/javadoc/org/apache/kafka/connect/storage/StringConverter.html. I think that's what you are looking for in terms of a Converter. It looks like your bigger need is the output format for HDFS.
FYI -- you are most welcome to add your request at the GitHub issues page for the HDFS Connector https://github.com/confluentinc/kafka-connect-hdfs/issues On Wed, Jun 22, 2016 at 1:26 PM, Tauzell, Dave <dave.tauz...@surescripts.com > wrote: > I don't see any built-in support for this but I think that you can write a > class that implements io.confluent.connect.hdfs.Format > > public interface Format { > RecordWriterProvider getRecordWriterProvider(); > SchemaFileReader getSchemaFileReader(AvroData avroData); > HiveUtil getHiveUtil(HdfsSinkConnectorConfig config, AvroData avroData, > HiveMetaStore hiveMetaStore); > } > > You would still have to register a schema in the Schema Registry and the > "SchemaFileReader" that you return would have to return the same Schema. > > -Dave > > Dave Tauzell | Senior Software Engineer | Surescripts > O: 651.855.3042 | www.surescripts.com | dave.tauz...@surescripts.com > Connect with us: Twitter I LinkedIn I Facebook I YouTube > > > -----Original Message----- > From: Pariksheet Barapatre [mailto:pari.data...@gmail.com] > Sent: Wednesday, June 22, 2016 11:49 AM > To: users@kafka.apache.org > Cc: d...@kafka.apache.org > Subject: Re: Kafka HDFS Connector > > Hi Dustin, > > I am looking for option 1. > > Looking at Kafka Connect code, I guess we need to write converter code if > not available. > > > Thanks in advance. > > Regards > Pari > > > On 22 June 2016 at 18:50, Dustin Cote <dus...@confluent.io> wrote: > > > Hi Pari, > > > > Can you clarify which scenario you are looking to implement? > > 1) plaintext Kafka data --> plaintext HDFS data readable by hive > > 2) plaintext Kafka data --> avro/parquet HDFS data readable by hive > > > > Regards, > > > > > > > > On Wed, Jun 22, 2016 at 6:02 AM, Pariksheet Barapatre < > > pari.data...@gmail.com> wrote: > > > > > Thanks for your suggestions. I think if kafka connect provides the > > > same functionality as flume and storm, why should we go for another > > > infrastructure investment. > > > > > > Kafka Connect effectively copies data from Kafka topic to HDFS > > > through connector. It supports avro as well as parquet, I am looking > > > if we can > > use > > > it to load plain text data. > > > > > > Cheers > > > Pari > > > > > > > > > > > > On 22 June 2016 at 12:34, Lohith Samaga M > > > <lohith.sam...@mphasis.com> > > > wrote: > > > > > > > Hi, > > > > You can use Storm also, Here you have the option of > > > > rotating > > the > > > > file. You can also write to Hive directly. > > > > > > > > Best regards / Mit freundlichen Grüßen / Sincères salutations M. > > > > Lohith Samaga > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Mudit Kumar [mailto:mudit.ku...@askme.in] > > > > Sent: Wednesday, June 22, 2016 12.32 > > > > To: users@kafka.apache.org; d...@kafka.apache.org > > > > Subject: Re: Kafka HDFS Connector > > > > > > > > I think you can use flume also. > > > > > > > > Thanks, > > > > Mudit > > > > > > > > > > > > > > > > > > > > On 6/22/16, 12:29 PM, "Pariksheet Barapatre" > > > > <pari.data...@gmail.com> > > > > wrote: > > > > > > > > >Anybody have any idea on this? > > > > > > > > > >Thanks > > > > >Pari > > > > > > > > > >On 20 June 2016 at 14:36, Pariksheet Barapatre < > > pari.data...@gmail.com> > > > > >wrote: > > > > > > > > > >> Hello All, > > > > >> > > > > >> I have data coming from sensors into kafka cluster in text > > > > >> format delimited by comma. > > > > >> > > > > >> How to offload this data to Hive periodically from Kafka. I > > > > >> guess, Kafka Connect should solve my problem but when I checked > > > > >> documentation, examples have only avro formatted data. Can you > > please > > > > >> provide some knowledge on this. > > > > >> > > > > >> Many Thanks > > > > >> Pari > > > > >> > > > > > > > > Information transmitted by this e-mail is proprietary to Mphasis, > > > > its associated companies and/ or its customers and is intended for > > > > use only by the individual or entity to which it is addressed, and > > > may > > > > contain information that is privileged, confidential or exempt > > > > from disclosure under applicable law. If you are not the > > intended > > > > recipient or it appears that this mail has been forwarded to you > > > > without proper authority, you are notified that any use or > > > > dissemination of this information in any manner is strictly > > > > prohibited. In such cases, please notify us immediately at > > > > mailmas...@mphasis.com and delete this mail from your records. > > > > > > > > > > > > > > > -- > > Dustin Cote > > confluent.io > > > This e-mail and any files transmitted with it are confidential, may > contain sensitive information, and are intended solely for the use of the > individual or entity to whom they are addressed. If you have received this > e-mail in error, please notify the sender by reply e-mail immediately and > destroy all copies of the e-mail and any attachments. > -- Dustin Cote confluent.io
