Re: [Proposal] - Kafka Input Format for headers, key and payload parsing

2021-09-21 Thread Gian Merlino
Hey Lokesh, Thanks for the details. To me it makes more sense to have the user specify the entire timestamp and key field name (it seems weird to have a "timestamp prefix" and "key prefix" that are only used for single fields). I just wrote that + a few comments on the PR itself: https://github.co

Re: [Proposal] - Kafka Input Format for headers, key and payload parsing

2021-09-17 Thread Lokesh Lingarajan
Hi Gian, Thanks for the your reply, please find below are my comments 1) How is the timestamp exposed exactly? I see there is a recordTimestampLabelPrefix, but what is that a prefix to? Also: what do you think about accepting the entire name of the timestamp field instead? Finally: in the docs it

Re: [Proposal] - Kafka Input Format for headers, key and payload parsing

2021-09-16 Thread Gian Merlino
Lokesh, it looks like you got dropped from the thread, so I'm adding you back. Please check out the previous message for some comments. By the way, by default, replies to the dev list go back to the dev list only, which can cause you to miss some replies. If you join the list you will be sure to g

Re: [Proposal] - Kafka Input Format for headers, key and payload parsing

2021-09-14 Thread Gian Merlino
Hey Lokesh, The concept and API looks solid to me! Thank you for writing this up. I agree with Ben's comment. This will be really useful functionality. I have a few questions about how it would work: 1) How is the timestamp exposed exactly? I see there is a recordTimestampLabelPrefix, but what i

Re: [Proposal] - Kafka Input Format for headers, key and payload parsing

2021-09-14 Thread Lokesh Lingarajan
Any updates/comments ? -Lokesh On Tue, Sep 7, 2021 at 9:29 AM Lokesh Lingarajan wrote: > Hope everyone had a good long weekend. Any updates/comments ? > > -Lokesh > > > On Mon, Aug 30, 2021 at 2:43 PM Lokesh Lingarajan < > llingara...@confluent.io> wrote: > >> Motivation >> >> Today we ingest

Re: [Proposal] - Kafka Input Format for headers, key and payload parsing

2021-09-07 Thread Ben Krug
I'm not a coder, but wanted to say that I have heard other druid users ask for this functionality, so I think it would be useful. Thank you! On Tue, Sep 7, 2021 at 10:09 AM Lokesh Lingarajan wrote: > Hope everyone had a good long weekend. Any updates/comments ? > > -Lokesh > > > On Mon, Aug 30,

Re: [Proposal] - Kafka Input Format for headers, key and payload parsing

2021-09-07 Thread Lokesh Lingarajan
Hope everyone had a good long weekend. Any updates/comments ? -Lokesh On Mon, Aug 30, 2021 at 2:43 PM Lokesh Lingarajan wrote: > Motivation > > Today we ingest a number of high cardinality metrics into Druid across > dimensions. These metrics are rolled up on a per minute basis, and are very >