"...whether the broker would ever want/need to act on non-standard
headers..."
core broker code? no. never (which is why i suggested lazily parsing the
headers on 1st call from a bytebuffer slice() somewhere earlier in the
discussion).
broker-side plugins however ...
so here's something off the top of my head: as a kafka infra provider, i
dont want _anyone_ producing into one of "my" brokers without audit data.
it messes with my reporting/billing systems.
i cant go chasing after (literally) hundreds of internal developers, each
copy-pasting bits and pieces of code from stack overflow, and they only
come to me when said copy-pasted code doesnt work :-)
i would like to write a plugin that just drops messages without an audit
header on the floor (think of this as the equivalent of a
content-inspecting firewall).
On Tue, Nov 8, 2016 at 6:41 PM, radai <radai.rosenbl...@gmail.com> wrote:
> +1 for {namespace, plugin}, would definitely cut down on registration
> overhead.
>
> On Tue, Nov 8, 2016 at 5:54 PM, Gwen Shapira <g...@confluent.io> wrote:
>
>> Thank you so much for this clear and fair summary of the arguments.
>>
>> I'm in favor of ints. Not a deal-breaker, but in favor.
>>
>> Even more in favor of Magnus's decentralized suggestion with Roger's
>> tweak: add a namespace for headers. This will allow each app to just
>> use whatever IDs it wants internally, and then let the admin deploying
>> the app figure out an available namespace ID for the app to live in.
>> So io.confluent.schema-registry can be namespace 0x01 on my deployment
>> and 0x57 on yours, and the poor guys developing the app don't need to
>> worry about that.
>>
>> Gwen
>>
>> On Tue, Nov 8, 2016 at 4:23 PM, radai <radai.rosenbl...@gmail.com> wrote:
>> > +1 for sean's document. it covers pretty much all the trade-offs and
>> > provides concrete figures to argue about :-)
>> > (nit-picking - used the same xkcd twice, also trove has been superceded
>> for
>> > purposes of high performance collections: look at
>> > https://github.com/leventov/Koloboke)
>> >
>> > so to sum up the string vs int debate:
>> >
>> > performance - you can do 140k ops/sec _per thread_ with string headers.
>> you
>> > could do x2-3 better with ints. there's no arguing the relative diff
>> > between the two, there's only the question of whether or not _the rest
>> of
>> > kafka_ operates fast enough to care. if we want to make choices solely
>> > based on performance we need ints. if we are willing to
>> settle/compromise
>> > for a nicer (to some) API than strings are good enough for the current
>> > state of affairs.
>> >
>> > message size - with batching and compression it comes down to a ~5%
>> > difference (internal testing, not in the doc. maybe would help adding if
>> > this becomes a point of contention?). this means it wont really affect
>> > kafka in "throughput mode" (large, compressed batches). in "low latency"
>> > mode (meaning less/no batching and compression) the difference can be
>> > extreme (it'll easily be an order of magnitude with small payloads like
>> > stock ticks and header keys of the form
>> > "com.acme.infraTeam.kafka.hiMom.auditPlugin"). we have a few such
>> topics at
>> > linkedin where actual payloads are ~2 ints and are eclipsed by our
>> in-house
>> > audit "header" which is why we liked ints to begin with.
>> >
>> > "ease of use" - strings would probably still require _some_ degree of
>> > partitioning by convention (imagine if everyone used the key "infra"...)
>> > but its very intuitive for java devs to do anyway (reverse-domain is
>> > ingrained into java developers at a young age :-) ). also most java devs
>> > find Map<String, whatever> more intuitive than Map<Integer, whatever> -
>> > probably because of other text-based protocols like http. ints would
>> > require a number registry. if you think number registries are hard just
>> > look at the wiki page for KIPs (specifically the number for next
>> available
>> > KIP) and think again - we are probably talking about the same volume of
>> > requests. also this would only be "required" (good citizenship, more
>> like)
>> > if you want to publish your plugin for others to use. within your org do
>> > whatever you want - just know that if you use [some "reserved" range]
>> and a
>> > future kafka update breaks it its your problem. RTFM.
>> >
>> > personally im in favor of ints.
>> >
>> > having said that (and like nacho) I will settle if int vs string remains
>> > the only obstacle to this.
>> >
>> > On Tue, Nov 8, 2016 at 3:53 PM, Nacho Solis <nso...@linkedin.com.invalid
>> >
>> > wrote:
>> >
>> >> I think it's well known I've been pushing for ints (and I could switch
>> to
>> >> 16 bit shorts if pressed).
>> >>
>> >> - efficient (space)
>> >> - efficient (processing)
>> >> - easily partitionable
>> >>
>> >>
>> >> However, if the only thing that is keeping us from adopting headers is
>> the
>> >> use of strings vs ints as keys, then I would cave in and accept
>> strings. If
>> >> we do so, I would like to limit string keys to 128 bytes in length.
>> This
>> >> way 1) I could use a 3 letter string if I wanted (effectively using 4
>> total
>> >> bytes), 2) limit overall impact of possible keys (don't really want
>> people
>> >> to send a 16K header string key).
>> >>
>> >> Nacho
>> >>
>> >>
>> >> On Tue, Nov 8, 2016 at 3:35 PM, Gwen Shapira <g...@confluent.io>
>> wrote:
>> >>
>> >> > Forgot to mention: Thank you for quantifying the trade-off - it is
>> >> > helpful and important regardless of what we end up deciding.
>> >> >
>> >> > On Tue, Nov 8, 2016 at 3:12 PM, Sean McCauliff
>> >> > <smccaul...@linkedin.com.invalid> wrote:
>> >> > > On Tue, Nov 8, 2016 at 2:15 PM, Gwen Shapira <g...@confluent.io>
>> >> wrote:
>> >> > >
>> >> > >> Since Kafka specifically targets high-throughput, low-latency
>> >> > >> use-cases, I don't think we should trade them off that easily.
>> >> > >>
>> >> > >
>> >> > > I find these kind of design goals not to be really helpful unless
>> it's
>> >> > > quantified in someway. Because it's always possible to argue
>> against
>> >> > > something as either being not performant or just an implementation
>> >> > detail.
>> >> > >
>> >> > > This is a single threaded benchmarks so all the measurements are
>> per
>> >> > > thread.
>> >> > >
>> >> > > For 1M messages/s/thread if header keys are int and you had even a
>> >> > single
>> >> > > header key, value pair then it's still about 2^-2 microseconds
>> which
>> >> > means
>> >> > > you only have another 0.75 microseconds to do everything else you
>> want
>> >> to
>> >> > > do with a message (1M messages/s means 1 micro second per message).
>> >> With
>> >> > > string header keys there is still 0.5 micro seconds to process a
>> >> message.
>> >> > >
>> >> > >
>> >> > >
>> >> > > I love strings as much as the next guy (we had them in Flume), but
>> I
>> >> > >> was convinced by Magnus/Michael/Radai that strings don't actually
>> have
>> >> > >> strong benefits as opposed to ints (you'll need a string registry
>> >> > >> anyway - otherwise, how will you know what does the "profile_id"
>> >> > >> header refers to?) and I want to keep closer to our original
>> design
>> >> > >> goals for Kafka.
>> >> > >>
>> >> > >
>> >> > > "confluent.profile_id"
>> >> > >
>> >> > >
>> >> > >>
>> >> > >> If someone likes strings in the headers and doesn't do millions of
>> >> > >> messages a sec, they probably have lots of other systems they can
>> use
>> >> > >> instead.
>> >> > >>
>> >> > >
>> >> > > None of them will scale like Kafka. Horizontal scaling is still
>> good.
>> >> > >
>> >> > >
>> >> > >>
>> >> > >>
>> >> > >> On Tue, Nov 8, 2016 at 1:22 PM, Sean McCauliff
>> >> > >> <smccaul...@linkedin.com.invalid> wrote:
>> >> > >> > +1 for String keys.
>> >> > >> >
>> >> > >> > I've been doing some bechmarking and it seems like the speedup
>> for
>> >> > using
>> >> > >> > integer keys is about 2-5 depending on the length of the
>> strings and
>> >> > what
>> >> > >> > collections are being used. The overall amount of time spent
>> >> parsing
>> >> > a
>> >> > >> set
>> >> > >> > of header key, value pairs probably does not matter unless you
>> are
>> >> > >> getting
>> >> > >> > close to 1M messages per consumer. In which case probably
>> don't use
>> >> > >> > headers. There is also the option to use very short strings;
>> some
>> >> > that
>> >> > >> are
>> >> > >> > even shorter than integers.
>> >> > >> >
>> >> > >> > Partitioning the string key space will be easier than
>> partitioning
>> >> an
>> >> > >> > integer key space. We won't need a global registry. Kafka
>> >> internally
>> >> > can
>> >> > >> > reserve some prefix like "_" as its namespace. Everyone else
>> can
>> >> use
>> >> > >> their
>> >> > >> > company or project name as namespace prefix and life should be
>> good.
>> >> > >> >
>> >> > >> > Here's the link to some of the benchmarking info:
>> >> > >> > https://docs.google.com/document/d/1tfT-
>> >> > 6SZdnKOLyWGDH82kS30PnUkmgb7nPL
>> >> > >> dw6p65pAI/edit?usp=sharing
>> >> > >> >
>> >> > >> >
>> >> > >> >
>> >> > >> > --
>> >> > >> > Sean McCauliff
>> >> > >> > Staff Software Engineer
>> >> > >> > Kafka
>> >> > >> >
>> >> > >> > smccaul...@linkedin.com
>> >> > >> > linkedin.com/in/sean-mccauliff-b563192
>> >> > >> >
>> >> > >> > On Mon, Nov 7, 2016 at 11:51 PM, Michael Pearce <
>> >> > michael.pea...@ig.com>
>> >> > >> > wrote:
>> >> > >> >
>> >> > >> >> +1 on this slimmer version of our proposal
>> >> > >> >>
>> >> > >> >> I def think the Id space we can reduce from the proposed
>> >> > int32(4bytes)
>> >> > >> >> down to int16(2bytes) it saves on space and as headers we
>> wouldn't
>> >> > >> expect
>> >> > >> >> the number of headers being used concurrently being that high.
>> >> > >> >>
>> >> > >> >> I would wonder if we should make the value byte array length
>> still
>> >> > int32
>> >> > >> >> though as This is the standard Max array length in Java saying
>> that
>> >> > it
>> >> > >> is a
>> >> > >> >> header and I guess limiting the size is sensible and would
>> work for
>> >> > all
>> >> > >> the
>> >> > >> >> use cases we have in mind so happy with limiting this.
>> >> > >> >>
>> >> > >> >> Do people generally concur on Magnus's slimmer version? Anyone
>> see
>> >> > any
>> >> > >> >> issues if we moved from int32 to int16?
>> >> > >> >>
>> >> > >> >> Re configurable ids per plugin over a global registry also
>> would
>> >> work
>> >> > >> for
>> >> > >> >> us. As such if this has better concensus over the proposed
>> global
>> >> > >> registry
>> >> > >> >> I'd be happy to change that.
>> >> > >> >>
>> >> > >> >> I was already sold on ints over strings for keys ;)
>> >> > >> >>
>> >> > >> >> Cheers
>> >> > >> >> Mike
>> >> > >> >>
>> >> > >> >> ________________________________________
>> >> > >> >> From: Magnus Edenhill <mag...@edenhill.se>
>> >> > >> >> Sent: Monday, November 7, 2016 10:10:21 PM
>> >> > >> >> To: dev@kafka.apache.org
>> >> > >> >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>> >> > >> >>
>> >> > >> >> Hi,
>> >> > >> >>
>> >> > >> >> I'm +1 for adding generic message headers, but I do share the
>> >> > concerns
>> >> > >> >> previously aired on this thread and during the KIP meeting.
>> >> > >> >>
>> >> > >> >> So let me propose a slimmer alternative that does not require
>> any
>> >> > sort
>> >> > >> of
>> >> > >> >> global header registry, does not affect broker performance or
>> >> > >> operations,
>> >> > >> >> and adds as little overhead as possible.
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> Message
>> >> > >> >> ------------
>> >> > >> >> The protocol Message type is extended with a Headers array
>> consting
>> >> > of
>> >> > >> >> Tags, where a Tag is defined as:
>> >> > >> >> int16 Id
>> >> > >> >> int16 Len // binary_data length
>> >> > >> >> binary_data[Len] // opaque binary data
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> Ids
>> >> > >> >> ---
>> >> > >> >> The Id space is not centrally managed, so whenever an
>> application
>> >> > needs
>> >> > >> to
>> >> > >> >> add headers, or use an eco-system plugin that does, its Id
>> >> allocation
>> >> > >> will
>> >> > >> >> need to be manually configured.
>> >> > >> >> This moves the allocation concern from the global space down to
>> >> > >> >> organization level and avoids the risk for id conflicts.
>> >> > >> >> Example pseudo-config for some app:
>> >> > >> >> sometrackerplugin.tag.sourcev3.id=1000
>> >> > >> >> dbthing.tag.tablename.id=1001
>> >> > >> >> myschemareg.tag.schemaname.id=1002
>> >> > >> >> myschemareg.tag.schemaversion.id=1003
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> Each header-writing or header-reading plugin must provide means
>> >> > >> (typically
>> >> > >> >> through configuration) to specify the tag for each header it
>> uses.
>> >> > >> Defaults
>> >> > >> >> should be avoided.
>> >> > >> >> A consumer silently ignores tags it does not have a mapping for
>> >> > (since
>> >> > >> the
>> >> > >> >> binary_data can't be parsed without knowing what it is).
>> >> > >> >>
>> >> > >> >> Id range 0..999 is reserved for future use by the broker and
>> must
>> >> > not be
>> >> > >> >> used by plugins.
>> >> > >> >>
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> Broker
>> >> > >> >> ---------
>> >> > >> >> The broker does not process the tags (other than the standard
>> >> > protocol
>> >> > >> >> syntax verification), it simply stores and forwards them as
>> opaque
>> >> > data.
>> >> > >> >>
>> >> > >> >> Standard message translation (removal of Headers) kicks in for
>> >> older
>> >> > >> >> clients.
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> Why not string ids?
>> >> > >> >> -------------------------
>> >> > >> >> String ids might seem like a good idea, but:
>> >> > >> >> * does not really solve uniqueness
>> >> > >> >> * consumes a lot of space (2 byte string length + string, per
>> >> > header)
>> >> > >> to
>> >> > >> >> be meaningful
>> >> > >> >> * doesn't really say anything how to parse the tag's data, so
>> it
>> >> is
>> >> > in
>> >> > >> >> effect useless on its own.
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> Regards,
>> >> > >> >> Magnus
>> >> > >> >>
>> >> > >> >>
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> 2016-11-07 18:32 GMT+01:00 Michael Pearce <
>> michael.pea...@ig.com>:
>> >> > >> >>
>> >> > >> >> > Hi Roger,
>> >> > >> >> >
>> >> > >> >> > Thanks for the support.
>> >> > >> >> >
>> >> > >> >> > I think the key thing is to have a common key space to make
>> an
>> >> > >> ecosystem,
>> >> > >> >> > there does have to be some level of contract for people to
>> play
>> >> > >> nicely.
>> >> > >> >> >
>> >> > >> >> > Having map<String, byte[]> or as per current proposed in kip
>> of
>> >> > >> having a
>> >> > >> >> > numerical key space of map<int, byte[]> is a level of the
>> >> contract
>> >> > >> that
>> >> > >> >> > most people would expect.
>> >> > >> >> >
>> >> > >> >> > I think the example in a previous comment someone else made
>> >> > linking to
>> >> > >> >> AWS
>> >> > >> >> > blog and also implemented api where originally they didn’t
>> have a
>> >> > >> header
>> >> > >> >> > space but not they do, where keys are uniform but the value
>> can
>> >> be
>> >> > >> >> string,
>> >> > >> >> > int, anything is a good example.
>> >> > >> >> >
>> >> > >> >> > Having a custom MetadataSerializer is something we had played
>> >> with,
>> >> > >> but
>> >> > >> >> > discounted the idea, as if you wanted everyone to work the
>> same
>> >> > way in
>> >> > >> >> the
>> >> > >> >> > ecosystem, having to have this also customizable makes it a
>> bit
>> >> > >> harder.
>> >> > >> >> > Think about making the whole message record custom
>> serializable,
>> >> > this
>> >> > >> >> would
>> >> > >> >> > make it fairly tricky (though it would not be impossible) to
>> have
>> >> > made
>> >> > >> >> work
>> >> > >> >> > nicely. Having the value customizable we thought is a
>> reasonable
>> >> > >> tradeoff
>> >> > >> >> > here of flexibility over contract of interaction between
>> >> different
>> >> > >> >> parties.
>> >> > >> >> >
>> >> > >> >> > Is there a particular case or benefit of having serialization
>> >> > >> >> customizable
>> >> > >> >> > that you have in mind?
>> >> > >> >> >
>> >> > >> >> > Saying this it is obviously something that could be
>> implemented,
>> >> if
>> >> > >> there
>> >> > >> >> > is a need. If we did go this avenue I think a defaulted
>> >> serializer
>> >> > >> >> > implementation should exist so for the 80:20 rule, people can
>> >> just
>> >> > >> have
>> >> > >> >> the
>> >> > >> >> > broker and clients get default behavior.
>> >> > >> >> >
>> >> > >> >> > Cheers
>> >> > >> >> > Mike
>> >> > >> >> >
>> >> > >> >> > On 11/6/16, 5:25 PM, "radai" <radai.rosenbl...@gmail.com>
>> wrote:
>> >> > >> >> >
>> >> > >> >> > making header _key_ serialization configurable
>> potentially
>> >> > >> undermines
>> >> > >> >> > the
>> >> > >> >> > board usefulness of the feature (any point along the path
>> >> must
>> >> > be
>> >> > >> >> able
>> >> > >> >> > to
>> >> > >> >> > read the header keys. the values may be whatever and
>> require
>> >> > more
>> >> > >> >> > intimate
>> >> > >> >> > knowledge of the code that produced specific headers, but
>> >> keys
>> >> > >> should
>> >> > >> >> > be
>> >> > >> >> > universally readable).
>> >> > >> >> >
>> >> > >> >> > it would also make it hard to write really portable
>> plugins -
>> >> > say
>> >> > >> i
>> >> > >> >> > wrote a
>> >> > >> >> > large message splitter/combiner - if i rely on key
>> >> > "largeMessage"
>> >> > >> and
>> >> > >> >> > values of the form "1/20" someone who uses (contrived
>> >> example)
>> >> > >> >> > Map<Byte[],
>> >> > >> >> > Double> wouldnt be able to re-use my code.
>> >> > >> >> >
>> >> > >> >> > not the end of a the world within an organization, but
>> >> > >> problematic if
>> >> > >> >> > you
>> >> > >> >> > want to enable an ecosystem
>> >> > >> >> >
>> >> > >> >> > On Thu, Nov 3, 2016 at 2:04 PM, Roger Hoover <
>> >> > >> roger.hoo...@gmail.com
>> >> > >> >> >
>> >> > >> >> > wrote:
>> >> > >> >> >
>> >> > >> >> > > As others have laid out, I see strong reasons for a
>> common
>> >> > >> message
>> >> > >> >> > > metadata structure for the Kafka ecosystem. In
>> particular,
>> >> > I've
>> >> > >> >> > seen that
>> >> > >> >> > > even within a single organization, infrastructure teams
>> >> often
>> >> > >> own
>> >> > >> >> the
>> >> > >> >> > > message metadata while application teams own the
>> >> > >> application-level
>> >> > >> >> > data
>> >> > >> >> > > format. Allowing metadata and content to have
>> different
>> >> > >> structure
>> >> > >> >> > and
>> >> > >> >> > > evolve separately is very helpful for this. Also, I
>> think
>> >> > >> there's
>> >> > >> >> a
>> >> > >> >> > lot of
>> >> > >> >> > > value to having a common metadata structure shared
>> across
>> >> the
>> >> > >> Kafka
>> >> > >> >> > > ecosystem so that tools which leverage metadata can
>> more
>> >> > easily
>> >> > >> be
>> >> > >> >> > shared
>> >> > >> >> > > across organizations and integrated together.
>> >> > >> >> > >
>> >> > >> >> > > The question is, where does the metadata structure
>> belong?
>> >> > >> Here's
>> >> > >> >> > my take:
>> >> > >> >> > >
>> >> > >> >> > > We change the Kafka wire and on-disk format to from a
>> (key,
>> >> > >> value)
>> >> > >> >> > model to
>> >> > >> >> > > a (key, metadata, value) model where all three are byte
>> >> > arrays
>> >> > >> from
>> >> > >> >> > the
>> >> > >> >> > > brokers point of view. The primary reason for this is
>> that
>> >> > it
>> >> > >> >> > provides a
>> >> > >> >> > > backward compatible migration path forward. Producers
>> can
>> >> > start
>> >> > >> >> > populating
>> >> > >> >> > > metadata fields before all consumers understand the
>> >> metadata
>> >> > >> >> > structure.
>> >> > >> >> > > For people who already have custom envelope structures,
>> >> they
>> >> > can
>> >> > >> >> > populate
>> >> > >> >> > > their existing structure and the new structure for a
>> while
>> >> as
>> >> > >> they
>> >> > >> >> > make the
>> >> > >> >> > > transition.
>> >> > >> >> > >
>> >> > >> >> > > We could stop there and let the clients plug in a
>> >> > KeySerializer,
>> >> > >> >> > > MetadataSerializer, and ValueSerializer but I think it
>> is
>> >> > also
>> >> > >> be
>> >> > >> >> > useful to
>> >> > >> >> > > have a default MetadataSerializer that implements a
>> >> key-value
>> >> > >> model
>> >> > >> >> > similar
>> >> > >> >> > > to AMQP or HTTP headers. Or we could go even further
>> and
>> >> > >> >> prescribe a
>> >> > >> >> > > Map<String, byte[]> or Map<String, String> data model
>> for
>> >> > >> headers
>> >> > >> >> in
>> >> > >> >> > the
>> >> > >> >> > > clients (while still allowing custom serialization of
>> the
>> >> > header
>> >> > >> >> data
>> >> > >> >> > > model).
>> >> > >> >> > >
>> >> > >> >> > > I think this would address Radai's concerns:
>> >> > >> >> > > 1. All client code would not need to be updated to know
>> >> about
>> >> > >> the
>> >> > >> >> > > container.
>> >> > >> >> > > 2. Middleware friendly clients would have a standard
>> header
>> >> > data
>> >> > >> >> > model to
>> >> > >> >> > > work with.
>> >> > >> >> > > 3. KIP is required both b/c of broker changes and
>> because
>> >> of
>> >> > >> client
>> >> > >> >> > API
>> >> > >> >> > > changes.
>> >> > >> >> > >
>> >> > >> >> > > Cheers,
>> >> > >> >> > >
>> >> > >> >> > > Roger
>> >> > >> >> > >
>> >> > >> >> > >
>> >> > >> >> > > On Wed, Nov 2, 2016 at 4:38 PM, radai <
>> >> > >> radai.rosenbl...@gmail.com>
>> >> > >> >> > wrote:
>> >> > >> >> > >
>> >> > >> >> > > > my biggest issues with a "standard" wrapper format:
>> >> > >> >> > > >
>> >> > >> >> > > > 1. _ALL_ client _CODE_ (as opposed to kafka lib
>> version)
>> >> > must
>> >> > >> be
>> >> > >> >> > updated
>> >> > >> >> > > to
>> >> > >> >> > > > know about the container, because any old naive code
>> >> > trying to
>> >> > >> >> > directly
>> >> > >> >> > > > deserialize its own payload would keel over and die
>> (it
>> >> > needs
>> >> > >> to
>> >> > >> >> > know to
>> >> > >> >> > > > deserialize a container, and then dig in there for
>> its
>> >> > >> payload).
>> >> > >> >> > > > 2. in order to write middleware-friendly clients that
>> >> > utilize
>> >> > >> >> such
>> >> > >> >> > a
>> >> > >> >> > > > container one would basically have to write their own
>> >> > >> >> > producer/consumer
>> >> > >> >> > > API
>> >> > >> >> > > > on top of the open source kafka one.
>> >> > >> >> > > > 3. if you were going to go with a wrapper format you
>> >> really
>> >> > >> dont
>> >> > >> >> > need to
>> >> > >> >> > > > bother with a kip (just open source your own client
>> stack
>> >> > >> from #2
>> >> > >> >> > above
>> >> > >> >> > > so
>> >> > >> >> > > > others could stop re-inventing it)
>> >> > >> >> > > >
>> >> > >> >> > > > On Wed, Nov 2, 2016 at 4:25 PM, James Cheng <
>> >> > >> >> wushuja...@gmail.com>
>> >> > >> >> > > wrote:
>> >> > >> >> > > >
>> >> > >> >> > > > > How exactly would this work? Or maybe that's out of
>> >> scope
>> >> > >> for
>> >> > >> >> > this
>> >> > >> >> > > email.
>> >> > >> >> > > >
>> >> > >> >> > >
>> >> > >> >> >
>> >> > >> >> >
>> >> > >> >> > The information contained in this email is strictly
>> confidential
>> >> > and
>> >> > >> for
>> >> > >> >> > the use of the addressee only, unless otherwise indicated.
>> If you
>> >> > are
>> >> > >> not
>> >> > >> >> > the intended recipient, please do not read, copy, use or
>> disclose
>> >> > to
>> >> > >> >> others
>> >> > >> >> > this message or any attachment. Please also notify the
>> sender by
>> >> > >> replying
>> >> > >> >> > to this email or by telephone (+44(020 7896 0011) and then
>> delete
>> >> > the
>> >> > >> >> email
>> >> > >> >> > and any copies of it. Opinions, conclusion (etc) that do not
>> >> > relate to
>> >> > >> >> the
>> >> > >> >> > official business of this company shall be understood as
>> neither
>> >> > given
>> >> > >> >> nor
>> >> > >> >> > endorsed by it. IG is a trading name of IG Markets Limited (a
>> >> > company
>> >> > >> >> > registered in England and Wales, company number 04008957)
>> and IG
>> >> > Index
>> >> > >> >> > Limited (a company registered in England and Wales, company
>> >> number
>> >> > >> >> > 01190902). Registered address at Cannon Bridge House, 25
>> Dowgate
>> >> > Hill,
>> >> > >> >> > London EC4R 2YA. Both IG Markets Limited (register number
>> 195355)
>> >> > and
>> >> > >> IG
>> >> > >> >> > Index Limited (register number 114059) are authorised and
>> >> > regulated by
>> >> > >> >> the
>> >> > >> >> > Financial Conduct Authority.
>> >> > >> >> >
>> >> > >> >> The information contained in this email is strictly
>> confidential
>> >> and
>> >> > for
>> >> > >> >> the use of the addressee only, unless otherwise indicated. If
>> you
>> >> are
>> >> > >> not
>> >> > >> >> the intended recipient, please do not read, copy, use or
>> disclose
>> >> to
>> >> > >> others
>> >> > >> >> this message or any attachment. Please also notify the sender
>> by
>> >> > >> replying
>> >> > >> >> to this email or by telephone (+44(020 7896 0011) and then
>> delete
>> >> the
>> >> > >> email
>> >> > >> >> and any copies of it. Opinions, conclusion (etc) that do not
>> relate
>> >> > to
>> >> > >> the
>> >> > >> >> official business of this company shall be understood as
>> neither
>> >> > given
>> >> > >> nor
>> >> > >> >> endorsed by it. IG is a trading name of IG Markets Limited (a
>> >> company
>> >> > >> >> registered in England and Wales, company number 04008957) and
>> IG
>> >> > Index
>> >> > >> >> Limited (a company registered in England and Wales, company
>> number
>> >> > >> >> 01190902). Registered address at Cannon Bridge House, 25
>> Dowgate
>> >> > Hill,
>> >> > >> >> London EC4R 2YA. Both IG Markets Limited (register number
>> 195355)
>> >> > and IG
>> >> > >> >> Index Limited (register number 114059) are authorised and
>> regulated
>> >> > by
>> >> > >> the
>> >> > >> >> Financial Conduct Authority.
>> >> > >> >>
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >> --
>> >> > >> Gwen Shapira
>> >> > >> Product Manager | Confluent
>> >> > >> 650.450.2760 | @gwenshap
>> >> > >> Follow us: Twitter | blog
>> >> > >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Gwen Shapira
>> >> > Product Manager | Confluent
>> >> > 650.450.2760 | @gwenshap
>> >> > Follow us: Twitter | blog
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Nacho (Ignacio) Solis
>> >> Kafka
>> >> nso...@linkedin.com
>> >>
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>>
>
>
