@Neha, not sure what you mean by using base64 encoded strings. base64
encoding takes bytes and gives you ASCII text. We need to go from
arbitrarily structured offsets data to bytes (e.g. user has given us a
record (with schema they have defined) containing db name + table name for
the key, and anot
Yeah, I agree that if we have the ser/de we can do anything :)
I'd actually feel more comfortable if the users *have* to go through our
APIs to get to the metadata (which again, is kind of internal to Copycat).
If they start writing their own code that depends on this data, who knows
what we may a
Ewen,
I meant we use format X to store offsets, whether you serialize your data
with Y or Z and we don't expose it as something that can be configured. As
far as the serialization format goes, I was suggesting just going with
simple base64 encoded strings (maybe there is a reason you are saying th
On Fri, Aug 14, 2015 at 6:35 PM, Gwen Shapira wrote:
> Yeah, I missed the option to match serialization of offsets to data, which
> solves the configuration overhead.
>
> It still doesn't give us the ability to easily evolve the metadata messages
> or to use them in monitoring tools.
>
> And I am
Yeah, I missed the option to match serialization of offsets to data, which
solves the configuration overhead.
It still doesn't give us the ability to easily evolve the metadata messages
or to use them in monitoring tools.
And I am still not clear of the benefits of using user-defined
serializatio
I'm not sure the existing discussion is clear about how the format of
offset data is decided. One possibility is that we choose one fixed format
and that is what we use internally to store offsets no matter what
serializer you choose. This would be similar to how the __offsets topic is
currently ha
Copycat enables streaming data in and out of Kafka. Connector writers need
to define the serde of the data as it is different per system. Metadata
should be entirely hidden by the copycat framework and isn't something
users or connector implementors need to serialize differently as long as we
provi
Hi Team Kafka,
As you know from KIP-26 and PR-99, when you will use Copycat to move data
from an external system to Kafka, in addition to storing the data itself,
Copycat will also need to store some metadata.
This metadata is currently offsets on the source system (say, SCN# from
Oracle redo log
