Hi, I'm excited to see the community expanding Connect in this direction! Headers + Transforms == Fun message routing.
I like how clean the proposal is, but I'm concerned that it kinda deviates from how Connect handles data elsewhere. Unlike Kafka, Connect doesn't look at all data as byte-arrays, we have converters that take data in specific formats (JSON, Avro) and turns it into Connect data types (defined in the data api). I think it will be more consistent for connector developers to also get headers as some kind of structured or semi-structured data (and to expand the converters to handle header conversions as well). This will allow for Connect's separation of concerns - Connector developers don't worry about data formats (because they get the internal connect objects) and Converters do all the data format work. Another thing, in my experience, APIs work better if they are put into use almost immediately - so difficulties in using the APIs are immediately surfaced. Are you planning any connectors that will use this feature (not necessarily in Kafka, just in general)? Or perhaps we can think of a way to expand Kafka's file connectors so they'll use headers somehow (can't think of anything, but maybe?). Gwen On Sat, Apr 29, 2017 at 12:12 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi All, > > Now KIP-82 is committed I would like to discuss extending the work to > expose it in Kafka Connect, its primary focus being so connectors that may > do similar tasks as MirrorMakers, either Kafka->Kafka or JMS-Kafka would be > able to replicate the headers. > It would be ideal but not mandatory for this to go in 0.11 release so is > available on day one of headers being available. > > Please find the KIP here: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 145+-+Expose+Record+Headers+in+Kafka+Connect > > Please find an initial implementation as a PR here: > https://github.com/apache/kafka/pull/2942 > > Kind Regards > Mike > The information contained in this email is strictly confidential and for > the use of the addressee only, unless otherwise indicated. If you are not > the intended recipient, please do not read, copy, use or disclose to others > this message or any attachment. Please also notify the sender by replying > to this email or by telephone (+44(020 7896 0011) and then delete the email > and any copies of it. Opinions, conclusion (etc) that do not relate to the > official business of this company shall be understood as neither given nor > endorsed by it. IG is a trading name of IG Markets Limited (a company > registered in England and Wales, company number 04008957) and IG Index > Limited (a company registered in England and Wales, company number > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG > Index Limited (register number 114059) are authorised and regulated by the > Financial Conduct Authority. > -- *Gwen Shapira* Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter <https://twitter.com/ConfluentInc> | blog <http://www.confluent.io/blog>
