Hi everyone,
>From my understanding, if a topic has n partitions, we can create up to n
>tasks for both the source and sink connectors to achieve the maximum
>parallelism. Adding more tasks would not be beneficial, as they would remain
>idle and be limited to the number of partitions of the t
Hi all,
Sink connectors support Dear Letter Queues[1], but Source connectors don't seem
to
What is the reason that we decided to do that?
In my data pipeline, I'd like to apply some transformations to the messages
before they are sink, but that leaves me vulnerable to failures as I need to
ei
Hi team,
In KIP-449[1], we introduced a "connector context" that allows us to get worker
context in the logs such as "[my-connector|task-0|offsets]"
Are you aware of any built-in mechanism to extract only the connector name from
these logs?
In an ideal world, I would like to build a lo
Hello everyone,
Is there any mechanism to force Kafka Connect to ingest at a given rate per
second as opposed to tasks?
I am operating in a shared environment where the ingestion rate needs to be as
low as possible (for example, 5 requests/second as an upper limit), and as far
as I can tel
Hi all,
This might be a common question, but unfortunately, I couldn't find a reliable
answer or documentation to guide me. There are various conflicting ideas.
If a producer tries to ingest at a faster rate than the configuration set in
the topic, what will happen?
Example:
- Topic size: 10
che/kafka/pull/6791
[2] - https://issues.apache.org/jira/browse/KAFKA-8404
Cheers,
Chris
On Fri, Oct 6, 2023 at 10:14 PM Yeikel Santana <mailto:em...@yeikel.com> wrote:
> Hello everyone,
>
> I'm currently running Kafka Connect behind a firewall that mandates the
Hello everyone,
I'm currently running Kafka Connect behind a firewall that mandates the
inclusion of a specific header. This situation becomes particularly challenging
when forwarding requests among multiple workers, as it appears that only the
Authorization header is included in the request.
f68e4f2
On Tue, Sep 26, 2023 at 8:25 AM Yeikel Santana <mailto:em...@yeikel.com> wrote:
> Thank you, Yash. Your explanation makes sense
>
> To clarify, who initiates the step that assigns a connector to a specific
> worker? If this process is controlled by the leader, wouldn
essentially useless.
Thanks,
Yash
On Mon, Sep 25, 2023 at 7:51 PM Yeikel Santana <mailto:em...@yeikel.com> wrote:
> Thank you, Nikhil.
>
> I did notice that challenge you're describing with the REST updates when I
> had more than one worker within the same
-Nikhil
On Sun, 24 Sept 2023 at 06:36, Yeikel Santana wrote:
> Hello everyone,I'm currently designing a new Kafka Connect cluster, and
> I'm trying to understand how connectivity functions among workers.In my
> setup, I have a single Kafka Connect cluster connected to the sam
Hello everyone,I'm currently designing a new Kafka Connect cluster, and I'm
trying to understand how connectivity functions among workers.In my setup, I
have a single Kafka Connect cluster connected to the same Kafka topics and
Kafka cluster. However, the workers are deployed in geographically s
Hello all,
>From what I've seen so far, with
>https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%3A+Externalizing+Secrets+for+Connect+Configurations,
> Kafka provides a mechanism to externalize secrets, which was later extended
>with
>https://cwiki.apache.org/confluence/display/KAFKA/KIP
12 matches
Mail list logo
