My bad, the link should be this:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Enriched+Message+Metadata
On Fri, Nov 21, 2014 at 5:29 PM, Timothy Chen wrote:
> Hi Guozhang,
>
> I don't think that is publically accessible, can you update it to the
> Kafka wiki?
>
> Tim
>
> On Fri, Nov
Hi Guozhang,
I don't think that is publically accessible, can you update it to the
Kafka wiki?
Tim
On Fri, Nov 21, 2014 at 5:24 PM, Guozhang Wang wrote:
> Hi all,
>
> I have updated the wiki page (
> https://iwww.corp.linkedin.com/wiki/cf/display/ENGS/Kafka+Enriched+Message+Metadata)
> accordin
Hi all,
I have updated the wiki page (
https://iwww.corp.linkedin.com/wiki/cf/display/ENGS/Kafka+Enriched+Message+Metadata)
according to people's comments and discussions offline.
Guozhang
On Thu, Nov 13, 2014 at 9:43 AM, Guozhang Wang wrote:
> Hi Jun,
>
> Sorry for the delay on your comments
Hi Jun,
Sorry for the delay on your comments in the wiki page as well as this
thread; quite swamped now. I will get back to you as soon as I find some
time.
Guozhang
On Tue, Nov 11, 2014 at 6:26 PM, Jun Rao wrote:
> Thinking about this a bit more. For adding the auditing support, I am not
> su
Thinking about this a bit more. For adding the auditing support, I am not
sure if we need to change the message format by adding the application
tags. An alternative way to do that is to add it in the producer client.
For example, for each message payload (doesn't matter what the
serialization mech
Hi Jun,
Regarding 4) in your comment, after thinking it for a while I cannot come
up a way to it along with log compaction without adding new fields into the
current format on message set. Do you have a better way that do not require
protocol changes?
Guozhang
On Mon, Oct 20, 2014 at 9:53 AM, Gu
I have updated the wiki page incorporating received comments. We can
discuss some more details on:
1. How we want to do audit? Whether we want to have in-built auditing on
brokers or even MMs or use an audit consumer to fetch all messages from
just brokers.
2. How we can avoid de-/re-compression
8. Perhaps I need to see the details of the alternative solution. During
the startup of a follower, in general, it's not enough for the follower to
just see the latest generation of the current leader, since the follower
can be several generations behind. So, if the controlled message only
contains
Thanks for the detailed comments Jun! Some replies inlined.
On Sun, Oct 19, 2014 at 7:42 PM, Jun Rao wrote:
> Hi, Guozhang,
>
> Thanks for the writeup.
>
> A few high level comments.
>
> 1. Associating (versioned) schemas to a topic can be a good thing overall.
> Yes, this could add a bit more m
A couple notes on this
1 / 2 - Message format should not be important to Kafka or the common
infrastructure (audit, mirror maker). As I have noted elsewhere, having to
ensure the data contract just to provide a general messaging service is
additional overhead that is not always needed. It eith
Hi, Guozhang,
Thanks for the writeup.
A few high level comments.
1. Associating (versioned) schemas to a topic can be a good thing overall.
Yes, this could add a bit more management overhead in Kafka. However, it
makes sure that the data format contract between a producer and a consumer
is kept
I think the tags are a useful concept to have in that they do for
applications, what the additional metadata does for brokers. i.e.,
avoiding decompression and recompression of an entire message-set. I
agree that we should not place any "core" fields (i.e., those used
internally by Kafka) in tags a
Let me add my view on #2 in less delicate terms than Guozhang did :)
When you¹re trying to run Kafka as a service, having to care about the
format of the message sucks. I have plenty of users who are just fine
using the Avro standard and play nice. Then I have a bunch of users who
don¹t want to us
Thanks Joe,
I think we now have a few open questions to discuss around this topic:
1. Shall we make core Kafka properties as first class fields in message
header or put them as tags?
The pros of the first approach is more compacted format and hence less
message header overhead; the cons are that
I think we could add schemaId(binary) to the MessageAndMetaData
With the schemaId you can implement different downstream software pattern
on the messages reliably. I wrote up more thoughts on this use
https://cwiki.apache.org/confluence/display/KAFKA/Schema+based+topics it
should strive to encompa
Hi Jay,
Thanks for the comments. Replied inline.
Guozhang
On Mon, Oct 13, 2014 at 11:11 AM, Jay Kreps wrote:
> I need to take more time to think about this. Here are a few off-the-cuff
> remarks:
>
> - To date we have tried really, really hard to keep the data model for
> message simple since
I need to take more time to think about this. Here are a few off-the-cuff
remarks:
- To date we have tried really, really hard to keep the data model for
message simple since after all you can always add whatever you like inside
the message body.
- For system tags, why not just make these fields
Thanks Guozhang! This is an excellent write-up and the approach nicely
consolidates a number of long-standing issues. It would be great if
everyone can review this carefully and give feedback.
Also, wrt discussion in the past we have used a mix of wiki comments
and the mailing list. Personally, I
Hello all,
I put some thoughts on enhancing our current message metadata format to
solve a bunch of existing issues:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Enriched+Message+Metadata
This wiki page is for kicking off some discussions about the feasibility of
adding more info into
19 matches
Mail list logo
