Hi Ewen, Thank you for your advise. I reached a following compromise: My application sends records in the Avro Binary encoding to Kafka directly using Confluent's serializers(io.confluent/kafka-avro-serializer), which integrated with schema registry, and also does logging in the Avro JSON encoding to a local file for easy reading. An advantage of transferring records in log files to Kafka is that the log files can be used as buffering(nice disaster tolerance). I think KAFKA-1955[0] will solve this problem in a better way.
[0]: [KAFKA-1955] Explore disk-based buffering in new Kafka Producer - ASF JIRA https://issues.apache.org/jira/browse/KAFKA-1955 -Takahiro 2016-11-27 11:00 GMT+09:00 Ewen Cheslack-Postava <e...@confluent.io>: > Avro JSON encoding is a wire-level format. The AvroConverter accepts Java > runtime data (e.g. primitive types like Strings & Integers, Maps, Arrays, > and Connect Structs). > > The component that most closely matches your needs is Confluent's REST > proxy, which supports the Avro JSON encoding when receiving a payload (via > the vnd.kafka.avro.v1+json format). However, since the REST proxy is push, > you'd need another process to send requests to it. > > Alternatively, you could build a source connector which pulls Avro > JSON-encoded data and emits generic Connect data. By combining that with > the AvroConverters, you'd end up generating Avro binary-serialized data > into Kafka (albeit via a slightly indirect route). > > -Ewen > > On Mon, Oct 17, 2016 at 6:14 AM, Takahiro Hozumi <fat...@googlemail.com> > wrote: > > > Hi all, > > > > My application will emit log files in avro json encoding so that humans > > easily can read and grep records. > > I need to transfer this logs into Kafka as Avro binary encoding. > > And I want to use confluent schema registry in order to prepare schema > > evolution. > > > > After some research, I think Kafka connect in standalone mode might be > the > > easiest solution. > > The avro-converter included in kafka connect interacts with schema > registry > > under the hood. > > But I am not sure avro-converter accepts Avro json encoding as input. > > Do I need to create a json-encoding version of the avro-converter? > > > > And I don't find a documentation of Converter interface. > > I think converting job is just byte[] to byte[] job, but the interface > > seems to require internal data object. > > > > http://docs.confluent.io/3.0.1/connect/javadocs/index.html? > > org/apache/kafka/connect/storage/Converter.html > > > > Any advise or guidance would be greatly appreciated! > > > > Thanks, > > -- > > Takahiro > > > > > > -- > Thanks, > Ewen >
