I see there are no "Rejected Alternatives" in the KIP -- What is the reason
you can't run a distributed herder as a single node? Is the goal here to
run multiple standalone workers with the same group.id?
Long wall of text:
---
Hi Ewen
> There are 2 aspects to this. The first is rebalancing. There's actually
nothing that fundamentally requires every rebalance operation to be
stop-the-world. I've had some thoughts about allowing partial and
incremental rebalancing floating around for awhile.
Hi Ewen
thanks for the response, I totally agree that this KIP touches upon
multiple rather orthogonal issues, I had a hard time deciding on the name
and scope for it :)
Let me sum up the biggest pain points that we're facing with distributed
mode:
* By far the biggest issue we have is the lack o
Hey all,
I think think this is a great discussion, and is helping to clarify the
real pain points as well as explore a few more options than just what was
initially proposed.
Stephane, I think why you're ending up in "grand redesign" state is because
you're highlighting (and the KIP's motivation
Hi,
thanks for the feedback, you have some valid points that I did not consider
thoroughly.
Now that I think about it, my current proposal is indeed a bit hacky,
namely the dichotomy between running "light"/"heavy" connectors in
standalone/distributed mode: indeed there should be no difference, o
Fat fingered too... "Connect source should be able to achieve the same
centrality of offsets "
On Thu., 17 May 2018, 10:27 pm Stephane Maarek, <
steph...@simplemachines.com.au> wrote:
> Say you have 50 connectors all with different ACLs and service account.
> That's 50 connect clusters to maintai
Say you have 50 connectors all with different ACLs and service account.
That's 50 connect clusters to maintain. So 50*3 internal connect topics to
maintain (they can't share the same connect topics because they're
different clusters). At default config we're talking 1500 partitions which
is a lot f
First sentence fat fingered.
“Just curious as to why there’s an issue with the backing topics for Kafka
Connect.”
--
Rahul Singh
rahul.si...@anant.us
Anant Corporation
On May 17, 2018, 6:17 AM -0400, Stephane Maarek
, wrote:
> Hi Salius
>
> I think you're on the money, but you're not pushing
Just curious as to why there’s a pr been for the backing topics for Kafka
Connect.
I’m a fan of decoupling and love the world that operates in containers.
I’m just wondering specifically what the issue is for having topics storing the
connect states for source / sinks.
Wouldn’t it make sense t
Hi Salius
I think you're on the money, but you're not pushing things too far.
This is something I've hoped for a long time.
Let's talk Kafka Connect v2
Kafka Connect Cluster, as you said, are not convenient to work with (the
KIP details drawbacks well). I'm all about containerisation just like
st
Hi,
the only real usecase for the REST interface I can see is providing
health/liveness checks for mesos/kubernetes. It's also true that the API
can be left as is and e.g. not exposed publicly on the platform level, but
this would still leave opportunities to accidentally mess something up
interna
Hi,
What do you plan to use the read-only REST interface for? Is there
something what you cannot get through metrics interface? Otherwise it might
be easier to just disable the REST interface (either in the code, or just
on the platform level - e.g. in Kubernetes).
Also, I do not know what is the
Hi,
I'd like to start a discussion on the following KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-304%3A+Connect+runtime+mode+improvements+for+container+platforms
Basically the idea is to make it easier to run separate instances of Kafka
Connect hosting isolated connectors on contain
13 matches
Mail list logo
