Hi Team Kafka, As you know from KIP-26 and PR-99, when you will use Copycat to move data from an external system to Kafka, in addition to storing the data itself, Copycat will also need to store some metadata.
This metadata is currently offsets on the source system (say, SCN# from Oracle redo log), but I can imagine to store a bit more. When storing data, we obviously want pluggable serializers, so users will get the data in a format they like. But the metadata seems internal. i.e users shouldn't care about it and if we want them to read or change anything, we want to provide them with tools to do it. Moreover, by controlling the format we can do three important things: * Read the metadata for monitoring / audit purposes * Evolve / modify it. If users serialize it in their own format, and actually write clients to use this metadata, we don't know if its safe to evolve. * Keep configuration a bit simpler. This adds at least 4 new configuration items... What do you guys think? Gwen
