+1 for {namespace, plugin}, would definitely cut down on registration
overhead.On Tue, Nov 8, 2016 at 5:54 PM, Gwen Shapira <g...@confluent.io> wrote: > Thank you so much for this clear and fair summary of the arguments. > > I'm in favor of ints. Not a deal-breaker, but in favor. > > Even more in favor of Magnus's decentralized suggestion with Roger's > tweak: add a namespace for headers. This will allow each app to just > use whatever IDs it wants internally, and then let the admin deploying > the app figure out an available namespace ID for the app to live in. > So io.confluent.schema-registry can be namespace 0x01 on my deployment > and 0x57 on yours, and the poor guys developing the app don't need to > worry about that. > > Gwen > > On Tue, Nov 8, 2016 at 4:23 PM, radai <radai.rosenbl...@gmail.com> wrote: > > +1 for sean's document. it covers pretty much all the trade-offs and > > provides concrete figures to argue about :-) > > (nit-picking - used the same xkcd twice, also trove has been superceded > for > > purposes of high performance collections: look at > > https://github.com/leventov/Koloboke) > > > > so to sum up the string vs int debate: > > > > performance - you can do 140k ops/sec _per thread_ with string headers. > you > > could do x2-3 better with ints. there's no arguing the relative diff > > between the two, there's only the question of whether or not _the rest of > > kafka_ operates fast enough to care. if we want to make choices solely > > based on performance we need ints. if we are willing to settle/compromise > > for a nicer (to some) API than strings are good enough for the current > > state of affairs. > > > > message size - with batching and compression it comes down to a ~5% > > difference (internal testing, not in the doc. maybe would help adding if > > this becomes a point of contention?). this means it wont really affect > > kafka in "throughput mode" (large, compressed batches). in "low latency" > > mode (meaning less/no batching and compression) the difference can be > > extreme (it'll easily be an order of magnitude with small payloads like > > stock ticks and header keys of the form > > "com.acme.infraTeam.kafka.hiMom.auditPlugin"). we have a few such > topics at > > linkedin where actual payloads are ~2 ints and are eclipsed by our > in-house > > audit "header" which is why we liked ints to begin with. > > > > "ease of use" - strings would probably still require _some_ degree of > > partitioning by convention (imagine if everyone used the key "infra"...) > > but its very intuitive for java devs to do anyway (reverse-domain is > > ingrained into java developers at a young age :-) ). also most java devs > > find Map<String, whatever> more intuitive than Map<Integer, whatever> - > > probably because of other text-based protocols like http. ints would > > require a number registry. if you think number registries are hard just > > look at the wiki page for KIPs (specifically the number for next > available > > KIP) and think again - we are probably talking about the same volume of > > requests. also this would only be "required" (good citizenship, more > like) > > if you want to publish your plugin for others to use. within your org do > > whatever you want - just know that if you use [some "reserved" range] > and a > > future kafka update breaks it its your problem. RTFM. > > > > personally im in favor of ints. > > > > having said that (and like nacho) I will settle if int vs string remains > > the only obstacle to this. > > > > On Tue, Nov 8, 2016 at 3:53 PM, Nacho Solis <nso...@linkedin.com.invalid > > > > wrote: > > > >> I think it's well known I've been pushing for ints (and I could switch > to > >> 16 bit shorts if pressed). > >> > >> - efficient (space) > >> - efficient (processing) > >> - easily partitionable > >> > >> > >> However, if the only thing that is keeping us from adopting headers is > the > >> use of strings vs ints as keys, then I would cave in and accept > strings. If > >> we do so, I would like to limit string keys to 128 bytes in length. > This > >> way 1) I could use a 3 letter string if I wanted (effectively using 4 > total > >> bytes), 2) limit overall impact of possible keys (don't really want > people > >> to send a 16K header string key). > >> > >> Nacho > >> > >> > >> On Tue, Nov 8, 2016 at 3:35 PM, Gwen Shapira <g...@confluent.io> wrote: > >> > >> > Forgot to mention: Thank you for quantifying the trade-off - it is > >> > helpful and important regardless of what we end up deciding. > >> > > >> > On Tue, Nov 8, 2016 at 3:12 PM, Sean McCauliff > >> > <smccaul...@linkedin.com.invalid> wrote: > >> > > On Tue, Nov 8, 2016 at 2:15 PM, Gwen Shapira <g...@confluent.io> > >> wrote: > >> > > > >> > >> Since Kafka specifically targets high-throughput, low-latency > >> > >> use-cases, I don't think we should trade them off that easily. > >> > >> > >> > > > >> > > I find these kind of design goals not to be really helpful unless > it's > >> > > quantified in someway. Because it's always possible to argue > against > >> > > something as either being not performant or just an implementation > >> > detail. > >> > > > >> > > This is a single threaded benchmarks so all the measurements are per > >> > > thread. > >> > > > >> > > For 1M messages/s/thread if header keys are int and you had even a > >> > single > >> > > header key, value pair then it's still about 2^-2 microseconds which > >> > means > >> > > you only have another 0.75 microseconds to do everything else you > want > >> to > >> > > do with a message (1M messages/s means 1 micro second per message). > >> With > >> > > string header keys there is still 0.5 micro seconds to process a > >> message. > >> > > > >> > > > >> > > > >> > > I love strings as much as the next guy (we had them in Flume), but I > >> > >> was convinced by Magnus/Michael/Radai that strings don't actually > have > >> > >> strong benefits as opposed to ints (you'll need a string registry > >> > >> anyway - otherwise, how will you know what does the "profile_id" > >> > >> header refers to?) and I want to keep closer to our original design > >> > >> goals for Kafka. > >> > >> > >> > > > >> > > "confluent.profile_id" > >> > > > >> > > > >> > >> > >> > >> If someone likes strings in the headers and doesn't do millions of > >> > >> messages a sec, they probably have lots of other systems they can > use > >> > >> instead. > >> > >> > >> > > > >> > > None of them will scale like Kafka. Horizontal scaling is still > good. > >> > > > >> > > > >> > >> > >> > >> > >> > >> On Tue, Nov 8, 2016 at 1:22 PM, Sean McCauliff > >> > >> <smccaul...@linkedin.com.invalid> wrote: > >> > >> > +1 for String keys. > >> > >> > > >> > >> > I've been doing some bechmarking and it seems like the speedup > for > >> > using > >> > >> > integer keys is about 2-5 depending on the length of the strings > and > >> > what > >> > >> > collections are being used. The overall amount of time spent > >> parsing > >> > a > >> > >> set > >> > >> > of header key, value pairs probably does not matter unless you > are > >> > >> getting > >> > >> > close to 1M messages per consumer. In which case probably don't > use > >> > >> > headers. There is also the option to use very short strings; > some > >> > that > >> > >> are > >> > >> > even shorter than integers. > >> > >> > > >> > >> > Partitioning the string key space will be easier than > partitioning > >> an > >> > >> > integer key space. We won't need a global registry. Kafka > >> internally > >> > can > >> > >> > reserve some prefix like "_" as its namespace. Everyone else can > >> use > >> > >> their > >> > >> > company or project name as namespace prefix and life should be > good. > >> > >> > > >> > >> > Here's the link to some of the benchmarking info: > >> > >> > https://docs.google.com/document/d/1tfT- > >> > 6SZdnKOLyWGDH82kS30PnUkmgb7nPL > >> > >> dw6p65pAI/edit?usp=sharing > >> > >> > > >> > >> > > >> > >> > > >> > >> > -- > >> > >> > Sean McCauliff > >> > >> > Staff Software Engineer > >> > >> > Kafka > >> > >> > > >> > >> > smccaul...@linkedin.com > >> > >> > linkedin.com/in/sean-mccauliff-b563192 > >> > >> > > >> > >> > On Mon, Nov 7, 2016 at 11:51 PM, Michael Pearce < > >> > michael.pea...@ig.com> > >> > >> > wrote: > >> > >> > > >> > >> >> +1 on this slimmer version of our proposal > >> > >> >> > >> > >> >> I def think the Id space we can reduce from the proposed > >> > int32(4bytes) > >> > >> >> down to int16(2bytes) it saves on space and as headers we > wouldn't > >> > >> expect > >> > >> >> the number of headers being used concurrently being that high. > >> > >> >> > >> > >> >> I would wonder if we should make the value byte array length > still > >> > int32 > >> > >> >> though as This is the standard Max array length in Java saying > that > >> > it > >> > >> is a > >> > >> >> header and I guess limiting the size is sensible and would work > for > >> > all > >> > >> the > >> > >> >> use cases we have in mind so happy with limiting this. > >> > >> >> > >> > >> >> Do people generally concur on Magnus's slimmer version? Anyone > see > >> > any > >> > >> >> issues if we moved from int32 to int16? > >> > >> >> > >> > >> >> Re configurable ids per plugin over a global registry also would > >> work > >> > >> for > >> > >> >> us. As such if this has better concensus over the proposed > global > >> > >> registry > >> > >> >> I'd be happy to change that. > >> > >> >> > >> > >> >> I was already sold on ints over strings for keys ;) > >> > >> >> > >> > >> >> Cheers > >> > >> >> Mike > >> > >> >> > >> > >> >> ________________________________________ > >> > >> >> From: Magnus Edenhill <mag...@edenhill.se> > >> > >> >> Sent: Monday, November 7, 2016 10:10:21 PM > >> > >> >> To: dev@kafka.apache.org > >> > >> >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers > >> > >> >> > >> > >> >> Hi, > >> > >> >> > >> > >> >> I'm +1 for adding generic message headers, but I do share the > >> > concerns > >> > >> >> previously aired on this thread and during the KIP meeting. > >> > >> >> > >> > >> >> So let me propose a slimmer alternative that does not require > any > >> > sort > >> > >> of > >> > >> >> global header registry, does not affect broker performance or > >> > >> operations, > >> > >> >> and adds as little overhead as possible. > >> > >> >> > >> > >> >> > >> > >> >> Message > >> > >> >> ------------ > >> > >> >> The protocol Message type is extended with a Headers array > consting > >> > of > >> > >> >> Tags, where a Tag is defined as: > >> > >> >> int16 Id > >> > >> >> int16 Len // binary_data length > >> > >> >> binary_data[Len] // opaque binary data > >> > >> >> > >> > >> >> > >> > >> >> Ids > >> > >> >> --- > >> > >> >> The Id space is not centrally managed, so whenever an > application > >> > needs > >> > >> to > >> > >> >> add headers, or use an eco-system plugin that does, its Id > >> allocation > >> > >> will > >> > >> >> need to be manually configured. > >> > >> >> This moves the allocation concern from the global space down to > >> > >> >> organization level and avoids the risk for id conflicts. > >> > >> >> Example pseudo-config for some app: > >> > >> >> sometrackerplugin.tag.sourcev3.id=1000 > >> > >> >> dbthing.tag.tablename.id=1001 > >> > >> >> myschemareg.tag.schemaname.id=1002 > >> > >> >> myschemareg.tag.schemaversion.id=1003 > >> > >> >> > >> > >> >> > >> > >> >> Each header-writing or header-reading plugin must provide means > >> > >> (typically > >> > >> >> through configuration) to specify the tag for each header it > uses. > >> > >> Defaults > >> > >> >> should be avoided. > >> > >> >> A consumer silently ignores tags it does not have a mapping for > >> > (since > >> > >> the > >> > >> >> binary_data can't be parsed without knowing what it is). > >> > >> >> > >> > >> >> Id range 0..999 is reserved for future use by the broker and > must > >> > not be > >> > >> >> used by plugins. > >> > >> >> > >> > >> >> > >> > >> >> > >> > >> >> Broker > >> > >> >> --------- > >> > >> >> The broker does not process the tags (other than the standard > >> > protocol > >> > >> >> syntax verification), it simply stores and forwards them as > opaque > >> > data. > >> > >> >> > >> > >> >> Standard message translation (removal of Headers) kicks in for > >> older > >> > >> >> clients. > >> > >> >> > >> > >> >> > >> > >> >> Why not string ids? > >> > >> >> ------------------------- > >> > >> >> String ids might seem like a good idea, but: > >> > >> >> * does not really solve uniqueness > >> > >> >> * consumes a lot of space (2 byte string length + string, per > >> > header) > >> > >> to > >> > >> >> be meaningful > >> > >> >> * doesn't really say anything how to parse the tag's data, so > it > >> is > >> > in > >> > >> >> effect useless on its own. > >> > >> >> > >> > >> >> > >> > >> >> Regards, > >> > >> >> Magnus > >> > >> >> > >> > >> >> > >> > >> >> > >> > >> >> > >> > >> >> 2016-11-07 18:32 GMT+01:00 Michael Pearce < > michael.pea...@ig.com>: > >> > >> >> > >> > >> >> > Hi Roger, > >> > >> >> > > >> > >> >> > Thanks for the support. > >> > >> >> > > >> > >> >> > I think the key thing is to have a common key space to make an > >> > >> ecosystem, > >> > >> >> > there does have to be some level of contract for people to > play > >> > >> nicely. > >> > >> >> > > >> > >> >> > Having map<String, byte[]> or as per current proposed in kip > of > >> > >> having a > >> > >> >> > numerical key space of map<int, byte[]> is a level of the > >> contract > >> > >> that > >> > >> >> > most people would expect. > >> > >> >> > > >> > >> >> > I think the example in a previous comment someone else made > >> > linking to > >> > >> >> AWS > >> > >> >> > blog and also implemented api where originally they didn’t > have a > >> > >> header > >> > >> >> > space but not they do, where keys are uniform but the value > can > >> be > >> > >> >> string, > >> > >> >> > int, anything is a good example. > >> > >> >> > > >> > >> >> > Having a custom MetadataSerializer is something we had played > >> with, > >> > >> but > >> > >> >> > discounted the idea, as if you wanted everyone to work the > same > >> > way in > >> > >> >> the > >> > >> >> > ecosystem, having to have this also customizable makes it a > bit > >> > >> harder. > >> > >> >> > Think about making the whole message record custom > serializable, > >> > this > >> > >> >> would > >> > >> >> > make it fairly tricky (though it would not be impossible) to > have > >> > made > >> > >> >> work > >> > >> >> > nicely. Having the value customizable we thought is a > reasonable > >> > >> tradeoff > >> > >> >> > here of flexibility over contract of interaction between > >> different > >> > >> >> parties. > >> > >> >> > > >> > >> >> > Is there a particular case or benefit of having serialization > >> > >> >> customizable > >> > >> >> > that you have in mind? > >> > >> >> > > >> > >> >> > Saying this it is obviously something that could be > implemented, > >> if > >> > >> there > >> > >> >> > is a need. If we did go this avenue I think a defaulted > >> serializer > >> > >> >> > implementation should exist so for the 80:20 rule, people can > >> just > >> > >> have > >> > >> >> the > >> > >> >> > broker and clients get default behavior. > >> > >> >> > > >> > >> >> > Cheers > >> > >> >> > Mike > >> > >> >> > > >> > >> >> > On 11/6/16, 5:25 PM, "radai" <radai.rosenbl...@gmail.com> > wrote: > >> > >> >> > > >> > >> >> > making header _key_ serialization configurable potentially > >> > >> undermines > >> > >> >> > the > >> > >> >> > board usefulness of the feature (any point along the path > >> must > >> > be > >> > >> >> able > >> > >> >> > to > >> > >> >> > read the header keys. the values may be whatever and > require > >> > more > >> > >> >> > intimate > >> > >> >> > knowledge of the code that produced specific headers, but > >> keys > >> > >> should > >> > >> >> > be > >> > >> >> > universally readable). > >> > >> >> > > >> > >> >> > it would also make it hard to write really portable > plugins - > >> > say > >> > >> i > >> > >> >> > wrote a > >> > >> >> > large message splitter/combiner - if i rely on key > >> > "largeMessage" > >> > >> and > >> > >> >> > values of the form "1/20" someone who uses (contrived > >> example) > >> > >> >> > Map<Byte[], > >> > >> >> > Double> wouldnt be able to re-use my code. > >> > >> >> > > >> > >> >> > not the end of a the world within an organization, but > >> > >> problematic if > >> > >> >> > you > >> > >> >> > want to enable an ecosystem > >> > >> >> > > >> > >> >> > On Thu, Nov 3, 2016 at 2:04 PM, Roger Hoover < > >> > >> roger.hoo...@gmail.com > >> > >> >> > > >> > >> >> > wrote: > >> > >> >> > > >> > >> >> > > As others have laid out, I see strong reasons for a > common > >> > >> message > >> > >> >> > > metadata structure for the Kafka ecosystem. In > particular, > >> > I've > >> > >> >> > seen that > >> > >> >> > > even within a single organization, infrastructure teams > >> often > >> > >> own > >> > >> >> the > >> > >> >> > > message metadata while application teams own the > >> > >> application-level > >> > >> >> > data > >> > >> >> > > format. Allowing metadata and content to have different > >> > >> structure > >> > >> >> > and > >> > >> >> > > evolve separately is very helpful for this. Also, I > think > >> > >> there's > >> > >> >> a > >> > >> >> > lot of > >> > >> >> > > value to having a common metadata structure shared > across > >> the > >> > >> Kafka > >> > >> >> > > ecosystem so that tools which leverage metadata can more > >> > easily > >> > >> be > >> > >> >> > shared > >> > >> >> > > across organizations and integrated together. > >> > >> >> > > > >> > >> >> > > The question is, where does the metadata structure > belong? > >> > >> Here's > >> > >> >> > my take: > >> > >> >> > > > >> > >> >> > > We change the Kafka wire and on-disk format to from a > (key, > >> > >> value) > >> > >> >> > model to > >> > >> >> > > a (key, metadata, value) model where all three are byte > >> > arrays > >> > >> from > >> > >> >> > the > >> > >> >> > > brokers point of view. The primary reason for this is > that > >> > it > >> > >> >> > provides a > >> > >> >> > > backward compatible migration path forward. Producers > can > >> > start > >> > >> >> > populating > >> > >> >> > > metadata fields before all consumers understand the > >> metadata > >> > >> >> > structure. > >> > >> >> > > For people who already have custom envelope structures, > >> they > >> > can > >> > >> >> > populate > >> > >> >> > > their existing structure and the new structure for a > while > >> as > >> > >> they > >> > >> >> > make the > >> > >> >> > > transition. > >> > >> >> > > > >> > >> >> > > We could stop there and let the clients plug in a > >> > KeySerializer, > >> > >> >> > > MetadataSerializer, and ValueSerializer but I think it > is > >> > also > >> > >> be > >> > >> >> > useful to > >> > >> >> > > have a default MetadataSerializer that implements a > >> key-value > >> > >> model > >> > >> >> > similar > >> > >> >> > > to AMQP or HTTP headers. Or we could go even further > and > >> > >> >> prescribe a > >> > >> >> > > Map<String, byte[]> or Map<String, String> data model > for > >> > >> headers > >> > >> >> in > >> > >> >> > the > >> > >> >> > > clients (while still allowing custom serialization of > the > >> > header > >> > >> >> data > >> > >> >> > > model). > >> > >> >> > > > >> > >> >> > > I think this would address Radai's concerns: > >> > >> >> > > 1. All client code would not need to be updated to know > >> about > >> > >> the > >> > >> >> > > container. > >> > >> >> > > 2. Middleware friendly clients would have a standard > header > >> > data > >> > >> >> > model to > >> > >> >> > > work with. > >> > >> >> > > 3. KIP is required both b/c of broker changes and > because > >> of > >> > >> client > >> > >> >> > API > >> > >> >> > > changes. > >> > >> >> > > > >> > >> >> > > Cheers, > >> > >> >> > > > >> > >> >> > > Roger > >> > >> >> > > > >> > >> >> > > > >> > >> >> > > On Wed, Nov 2, 2016 at 4:38 PM, radai < > >> > >> radai.rosenbl...@gmail.com> > >> > >> >> > wrote: > >> > >> >> > > > >> > >> >> > > > my biggest issues with a "standard" wrapper format: > >> > >> >> > > > > >> > >> >> > > > 1. _ALL_ client _CODE_ (as opposed to kafka lib > version) > >> > must > >> > >> be > >> > >> >> > updated > >> > >> >> > > to > >> > >> >> > > > know about the container, because any old naive code > >> > trying to > >> > >> >> > directly > >> > >> >> > > > deserialize its own payload would keel over and die > (it > >> > needs > >> > >> to > >> > >> >> > know to > >> > >> >> > > > deserialize a container, and then dig in there for its > >> > >> payload). > >> > >> >> > > > 2. in order to write middleware-friendly clients that > >> > utilize > >> > >> >> such > >> > >> >> > a > >> > >> >> > > > container one would basically have to write their own > >> > >> >> > producer/consumer > >> > >> >> > > API > >> > >> >> > > > on top of the open source kafka one. > >> > >> >> > > > 3. if you were going to go with a wrapper format you > >> really > >> > >> dont > >> > >> >> > need to > >> > >> >> > > > bother with a kip (just open source your own client > stack > >> > >> from #2 > >> > >> >> > above > >> > >> >> > > so > >> > >> >> > > > others could stop re-inventing it) > >> > >> >> > > > > >> > >> >> > > > On Wed, Nov 2, 2016 at 4:25 PM, James Cheng < > >> > >> >> wushuja...@gmail.com> > >> > >> >> > > wrote: > >> > >> >> > > > > >> > >> >> > > > > How exactly would this work? Or maybe that's out of > >> scope > >> > >> for > >> > >> >> > this > >> > >> >> > > email. > >> > >> >> > > > > >> > >> >> > > > >> > >> >> > > >> > >> >> > > >> > >> >> > The information contained in this email is strictly > confidential > >> > and > >> > >> for > >> > >> >> > the use of the addressee only, unless otherwise indicated. If > you > >> > are > >> > >> not > >> > >> >> > the intended recipient, please do not read, copy, use or > disclose > >> > to > >> > >> >> others > >> > >> >> > this message or any attachment. Please also notify the sender > by > >> > >> replying > >> > >> >> > to this email or by telephone (+44(020 7896 0011) and then > delete > >> > the > >> > >> >> email > >> > >> >> > and any copies of it. Opinions, conclusion (etc) that do not > >> > relate to > >> > >> >> the > >> > >> >> > official business of this company shall be understood as > neither > >> > given > >> > >> >> nor > >> > >> >> > endorsed by it. IG is a trading name of IG Markets Limited (a > >> > company > >> > >> >> > registered in England and Wales, company number 04008957) and > IG > >> > Index > >> > >> >> > Limited (a company registered in England and Wales, company > >> number > >> > >> >> > 01190902). Registered address at Cannon Bridge House, 25 > Dowgate > >> > Hill, > >> > >> >> > London EC4R 2YA. Both IG Markets Limited (register number > 195355) > >> > and > >> > >> IG > >> > >> >> > Index Limited (register number 114059) are authorised and > >> > regulated by > >> > >> >> the > >> > >> >> > Financial Conduct Authority. > >> > >> >> > > >> > >> >> The information contained in this email is strictly confidential > >> and > >> > for > >> > >> >> the use of the addressee only, unless otherwise indicated. If > you > >> are > >> > >> not > >> > >> >> the intended recipient, please do not read, copy, use or > disclose > >> to > >> > >> others > >> > >> >> this message or any attachment. Please also notify the sender by > >> > >> replying > >> > >> >> to this email or by telephone (+44(020 7896 0011) and then > delete > >> the > >> > >> email > >> > >> >> and any copies of it. Opinions, conclusion (etc) that do not > relate > >> > to > >> > >> the > >> > >> >> official business of this company shall be understood as neither > >> > given > >> > >> nor > >> > >> >> endorsed by it. IG is a trading name of IG Markets Limited (a > >> company > >> > >> >> registered in England and Wales, company number 04008957) and IG > >> > Index > >> > >> >> Limited (a company registered in England and Wales, company > number > >> > >> >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate > >> > Hill, > >> > >> >> London EC4R 2YA. Both IG Markets Limited (register number > 195355) > >> > and IG > >> > >> >> Index Limited (register number 114059) are authorised and > regulated > >> > by > >> > >> the > >> > >> >> Financial Conduct Authority. > >> > >> >> > >> > >> > >> > >> > >> > >> > >> > >> -- > >> > >> Gwen Shapira > >> > >> Product Manager | Confluent > >> > >> 650.450.2760 | @gwenshap > >> > >> Follow us: Twitter | blog > >> > >> > >> > > >> > > >> > > >> > -- > >> > Gwen Shapira > >> > Product Manager | Confluent > >> > 650.450.2760 | @gwenshap > >> > Follow us: Twitter | blog > >> > > >> > >> > >> > >> -- > >> Nacho (Ignacio) Solis > >> Kafka > >> nso...@linkedin.com > >> > > > > -- > Gwen Shapira > Product Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter | blog >
