De : Bruno Cadonna
Envoyé : jeudi 19 décembre 2024 09:31
À : dev@kafka.apache.org
Objet : [EXT] Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams
Warning External sender Do not click on any links or open any attachments
unless you trust the sender and know the content is
ks
Sébastien
De : Matthias J. Sax
Envoyé : samedi 14 décembre 2024 05:29
À : dev@kafka.apache.org
Objet : [EXT] Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams
Warning External sender Do not click on any links or open any
attachments unless you trust the sender and know the con
after stopping." If this is acceptable, I completely agree.
cheers
____________
De : Bruno Cadonna
Envoyé : mercredi 11 décembre 2024 11:49
À : dev@kafka.apache.org
Objet : [EXT] Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka
Streams
Warning External sender Do not click
, while
minimizing potential confusion or edge cases.
Thanks
Sébastien
De : Matthias J. Sax
Envoyé : samedi 14 décembre 2024 05:29
À : dev@kafka.apache.org
Objet : [EXT] Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams
Warning External sender Do not
quot;resume" instead of "continue," but I interpreted it as "let's
continue after stopping." If this is acceptable, I completely agree.
cheers
____________
De : Bruno Cadonna
Envoyé : mercredi 11 décembre 2024 11:49
À : dev@kafka.apache.org
Objet : [EXT] Re: [DISCUSS] KIP-1034:
ut I interpreted it as "let's
continue after stopping." If this is acceptable, I completely agree.
cheers
________________
De : Bruno Cadonna
Envoyé : mercredi 11 décembre 2024 11:49
À : dev@kafka.apache.org
Objet : [EXT] Re: [DISCUSS] KIP-1034: Dead letter q
C9
I wasn’t entirely happy with my naming either. I considered using
"resume" instead of "continue," but I interpreted it as "let's
continue after stopping." If this is acceptable, I completely agree.
cheers
________________
De : Bruno Cadonna
Envo
uno,
We have planned a meeting for next friday to discuss it with
Loic and Damien.
We will be able to restart discussions about it soon.
regards
De : Bruno Cadonna
Envoyé : lundi 26 août 2024 11:32
À : dev@kafka.apache.org
Objet : [EXT] Re: [DISCUSS] KIP-103
De : Bruno Cadonna
Envoyé : mercredi 11 décembre 2024 11:49
À : dev@kafka.apache.org
Objet : [EXT] Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams
Warning External sender Do not click on any links or open any attachments
unless you trust the sender and know the content is safe.
e have planned a meeting for next friday to discuss it with
Loic and Damien.
We will be able to restart discussions about it soon.
regards
____
De : Bruno Cadonna
Envoyé : lundi 26 août 2024 11:32
À : dev@kafka.apache.org
Objet : [EXT] Re: [DISCUSS] KIP-1034: D
27 Aug 2024 at 15:37, Sebastien Viale
wrote:
Hi Bruno,
We have planned a meeting for next friday to discuss it with Loic and Damien.
We will be able to restart discussions about it soon.
regards
De : Bruno Cadonna
Envoyé : lundi 26 août 2024 11:32
À : dev@kafk
re readable.
> >>
> >>
> >> Best,
> >> Bruno
> >>
> >> On 8/30/24 3:37 PM, Damien Gasparina wrote:
> >>> Hi everyone,
> >>>
> >>> We just updated KIP-1034, we changed the following:
> >>> - We
We will be able to restart discussions about it soon.
regards
De : Bruno Cadonna
Envoyé : lundi 26 août 2024 11:32
À : dev@kafka.apache.org
Objet : [EXT] Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams
Warning External sender Do not click on any links or
> >>>> B3
> >>>> What is exactly the trigger to send a record to the dead letter queue?
> >>>> Is setting ERRORS_DEADLETTERQUEUE_TOPIC_NAME_CONFIG or is it adding a
> >>>> record to the return value of the exception handler?
> >>>> Wha
runo Cadonna
Envoyé : lundi 26 août 2024 11:32
À : dev@kafka.apache.org
Objet : [EXT] Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams
Warning External sender Do not click on any links or open any attachments
unless you trust the sender and know the content is safe.
Hi Loïc, Sebastie
> De : Bruno Cadonna
> Envoyé : lundi 26 août 2024 11:32
> À : dev@kafka.apache.org
> Objet : [EXT] Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams
>
> Warning External sender Do not click on any links or open any attachments
> unless you trus
erResponse.CONTINUE
>>>>> .withDeadLetterQueueRecord(record, "dlq-topic-c");
>>>>> }
>>>>>
>>>>> S5B:
>>>>>> I was having a bit of trouble understanding what the behavior would be
>>>>>> if someone configured a "error
De : Lucas Brutschy
Envoyé : lundi 22 avril 2024 14:36
À : dev@kafka.apache.org
Objet : [EXT] Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams
Warning External sender Do not click on any links or open any attachments
unless you trust the sender and know the content
nt to DLQ or not.
> >>> The "errors.deadletterqueue.topic.name" takes place to:
> >>>
> >>> * Specifying if the provided handlers should or should not send records
> >>> to DLQ.
> >>> * If the value is empty, the handlers should not send records to DLQ.
> >>>
d(record, "dlq-topic")
* CONTINUE.withDeadLetterQueueRecord(record, "dlq-topic")
cheers !
Sébastien
De : Lucas Brutschy
Envoyé : lundi 22 avril 2024 14:36
À : dev@kafka.apache.org
Objet : [EXT] Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams
Warning E
FAIL
> > * CONTINUE
> > * FAIL.withDeadLetterQueueRecord(record, "dlq-topic")
> > * CONTINUE.withDeadLetterQueueRecord(record, "dlq-topic")
> >
> > A DLQ topic name is currently required using the two last response types.
> > I am wo
etterQueueRecord(record, "dlq-topic")
>
> A DLQ topic name is currently required using the two last response types.
> I am wondering if it could benefit users to ease the use of the default DLQ
> topic "errors.deadletterqueue.topic.name" when implementing custom handler
uot;errors.deadletterqueue.topic.name" when implementing custom handlers,
with such kind of implementation:
* FAIL.withDefaultDeadLetterQueueRecord(record)
* CONTINUE.withDefaultDeadLetterQueueRecord(record)
Regards,
Loïc
De : Damien Gasparina
Envoyé : dimanche 14 avril 2024 2
Hi Sophie,
Thanks a lot for your feedback and your detailed comments.
S1.
> I'm confused -- are you saying that we're introducing a new kind of
ProducerRecord class for this?
Sorry for the poor wording, that's not what I meant. While writing the
KIP, I was hesitating between 1. leveraging the Ka
Thanks for the KIP, this will make a lot of people very happy.
Wanted to chime in on a few points that have been raised so far and add
some of my own (numbering with an S to distinguish my points from the
previous ones)
S1.
> 1.a I really meant ProducerRecord, that's the class used to forward to
Hi Andrew,
Thanks a lot for your review, plenty of good points!
11. Typo fixed, good cach.
12. I do agree with you and Nick also mentioned it, I updated the KIP
to mention that context headers should be forwarded.
13. Good catch, to be consistent with KIP-298, and without a strong
opinion from
Hi Nick,
1. Good point, that's less impactful than a custom interface, I just
updated the KIP with the new signature.
1.a I really meant ProducerRecord, that's the class used to forward to
downstream processors in the PAPI. The only information missing in
this class is the topic name. I also cons
Hi Damien, Sebastien and Loic,
Thanks for the KIP. The DLQ pattern is well established and bringing this to
Kafka Streams is a good improvement. I do plan to add DLQ support to share
groups in the future, once KIP-932 is complete. Having broad support in Kafka
for DLQs is great.
I have a few comme
Hi Damien and Sebastien,
1.
I think you can just add a `String topic` argument to the existing
`withDeadLetterQueueRecord(ProducerRecord
deadLetterQueueRecord)` method, and then the implementation of the
exception handler could choose the topic to send records to using whatever
logic the user desi
Hi Nick,
Thanks a lot for your review and your useful comments!
1. It is a good point, as you mentioned, I think it would make sense
in some use cases to have potentially multiple DLQ topics, so we
should provide an API to let users do it.
Thinking out-loud here, maybe it is a better approach to
Oh, and one more thing:
5.
Whenever you take a record out of the stream, and then potentially
re-introduce it at a later date, you introduce the potential for record
ordering issues. For example, that record could have been destined for a
Window that has been closed by the time it's re-processed.
Hi Damien,
Thanks for the KIP! Dead-letter queues are something that I think a lot of
users would like.
I think there are a few points with this KIP that concern me:
1.
It looks like you can only define a single, global DLQ for the entire Kafka
Streams application? What about applications that w
In a general way, if the user does not configure the right ACL, that
would be a security issue, but that's true for any topic.
This KIP allows users to configure a Dead Letter Queue without writing
custom Java code in Kafka Streams, not at the topic level.
A lot of applications are already impleme
My concern is that someone would create a dead letter queue on a sensitive
topic and not get the ACL correct from the start. Thus causing potential
confidential data leak. Is there anything in the proposal that would
prevent that from happening? If so I did not recognize it as such.
On Fri, Apr
Hi Claude,
In this KIP, the Dead Letter Queue is materialized by a standard and
independant topic, thus normal ACL applies to it like any other topic.
This should not introduce any security issues, obviously, the right
ACL would need to be provided to write to the DLQ if configured.
Cheers,
Dami
I am new to the Kafka codebase so please excuse any ignorance on my part.
When a dead letter queue is established is there a process to ensure that
it at least is defined with the same ACL as the original queue? Without
such a guarantee at the start it seems that managing dead letter queues
will
36 matches
Mail list logo
