Hi Michael, You'd probably be interested in the discussion for this KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP- 75+-+Add+per-connector+Converters
For now, you'd have to run different Connect instances, but KIP-75 plans to let you have control over converters at a connector level. Regards, On Mon, Aug 22, 2016 at 1:32 PM, Michael Sklyar <mikesk...@gmail.com> wrote: > I am looking into Kafka Connect and Confluent HDFSSinkConnector. > > The goal is to save data from various topics to HDFS. > We have at least two different formats of the data in Kafka - raw data > (JSON) - that we want to save as SequenceFile and normalized data > (Protobuf) that we want to save as Parquet. > > (I understand that Confluent expects to use Avro but I succeeded with > writing my custom converters and RecordWriters that work fine without Avro > and ShemaRegistry). > > Question: Is there a specific reason that key.converter value,converter are > defined per Kafka Connect cluster and not per a specific connector? > > It means that all the data in Kafka(in all the topics) should be stored in > the same format - or I will need two different clusters: one with > value.converter = MyCustomJsonConverter and another with > MyCustomProtobufConverter. > > It becomes even worse in case of Protobuf - every topic has a different > Protobuf schema - therefore needs a different converter and having a dozen > of Kafka clusters sounds like a very bad option. > > Wouldn't it make more sense to have the key.converter and value.converter > defined on the specific Connector level? > > Any other suggestions? > -- *Dustin Cote* Customer Operations Engineer | Confluent Follow us: Twitter <https://twitter.com/ConfluentInc> | blog <http://www.confluent.io/blog>
