It actually is possible to do so if you adapt the Connect Converter API to
streams. There are a couple of good reasons why we shouldn't require
everyone to just use the same schema:
1. Efficiency
Connect favors a little bit of inefficiency (translating byte[] ->
serialization runtime format -> Co
just out of curiosity, why does kafka streams not use this runtime data api
defined in kafka connect?
On Wed, Jul 26, 2017 at 3:10 AM, Ewen Cheslack-Postava
wrote:
> Stephen's explanation is great and accurate :)
>
> One of the design goals for Kafka Connect was to not rely on any specific
> ser
Stephen's explanation is great and accurate :)
One of the design goals for Kafka Connect was to not rely on any specific
serialization format since that is really orthogonal to getting/sending
data from/to other systems. We define the generic *runtime* data API, which
is what you'll find in the Ka
Ah, sorry, I have never used the JsonConverter, so didn't know that was
actually a thing. Looking at the code it looks like the converter can
handle json with or without the schema [1]. Take a look at the json
envelope code to get an idea of how the schema is passed along with the
message (also in
thanks for that explanation.
i use json instead of avro should i use the json serialization that
serializes both schema and data, so that the schema travels with the data
from source to sink? so set key.converter.schemas.enable=true and
value.converter.schemas.enable=true?
is it a correct assumpt
I'll try to answer this for you. I'm going to assume you are using the
pre-packaged kafka connect distro from confluent.
org.apache.kafka.connect.data.Schema is an abstraction of the type
definition for the data being passed around. How that is defined
generally falls onto the connector being used
i see kafka connect invented its own runtime data type system in
org.apache.kafka.connect.data
however i struggle to understand how this is used. the payload in kafka is
bytes. kafka does not carry any "schema" metadata. so how does connect know
what the schema is of a ConnectRecord?
if i write j
