Hi Greg,
1. Sorry, I don't exactly understand your question, mainly the cattle/pet
reference :). I think naming may not need to be connected to access. I'd
rather use ACLs to limit access of non-VC users to VC-linked topics.
2. Yes, with UUID-first generation prefix ACLs are unusable and I think
Hi all,
I'm coming around to this design now.
If we require users in some cluster (the root cluster) to be able to refer
to resources in virtual clusters for administration, and are not willing to
add protocol-specific support for this, it needs to be done through name
generation of some kind. I
Hey Stan,
1. You're right, most of the broker-level configs shouldn't be modified,
such as thread pools, loggers, etc.. In all honesty, I was mainly thinking
of group and topic resources here. Overall I think cluster admins should be
able to restrict VC admins through ACLs. I think we should be ab
This is a super cool proposal. Thank you for it!
1) The KIP mentions VCs are allowed ALTER_CONFIG, but doesn't mention what
configs will be allowed to get modified and what not. There are a few
broker-level configs that shouldn't be modifiable by any single VC imo -
https://github.com/apache/ka
Luke,
Agree that we provide tooling for this migration.
I added a draft about migration tooling, let me know what you think. The
metrics part is coming up sometime next week, I hope.
Best,
Viktor
On Fri, Mar 7, 2025 at 5:44 PM Viktor Somogyi-Vass <
viktor.somo...@cloudera.com> wrote:
> Adminis
Administrators would be able to put a cap on the quota of a virtual
cluster, so they can prevent virtual-clusters from monopolizing resources.
This can be thought of as a certain kind of isolation but I'm not sure if
you meant more than that? I'll add a revised version of the answer to David
to the
Hi,
I'll answer your questions below.
1. Sure, I amended the KIP with a diagram.
2. Topics scale out as they did before, there will be no changes. One can
increase the replication factor or partition count as they did before.
3. Virtual clusters aren't physical entities, there won't be any broker
Hi Greg,
Thank you for your comments!
1.
> Can you provide an example of a situation where a client in a VC needs to
> refer to resources in a child VC using the existing APIs?
I think topic creation/topic listing are both good examples, but I'd argue
that a cluster admin might need to be able
Hi Victor,
> Yes, we didn't plan for this originally, so it would be the
administrator's
responsibility to migrate existing ACLs, however thinking about it, the
migration could be quite simple (essentially we would just need to use the
same ACLs but move them to under VC1 so they're not global). T
Hi Greg,
3.
It sounds like some of the complication you're mentioning comes from the
physical topic names and physical location on disk; that's sorta why I
brought up (2).
If we transform the log layer to store things by UUID rather than by topic
name, we are then free to develop whatever mapping
Hey Daniel and Viktor,
Thanks so much for your answers!
1.
> The issue we encountered here is how can clients inside a VC can
> refer to topics linked in nested VCs? An admin of a VC needs a way to
refer
> to the resources inside the VC and the child VCs.
I would think the only operations referr
Hi Greg,
Replying to the rest of your points:
3. Well, after thinking about it a lot, I think that overall it would
complicate the feature without adding much usefulness. If we assume that a
topic is inside a directory hierarchy, then it wouldn't be transparent
anymore to users. Right now we say
Hi Greg,
Trying to respond to some of your questions/points:
1. We started out with the exact model you propose here - if VCs can be
nested, the root VC can be a special one, and everything else falls into
place. The issue we encountered here is how can clients inside a VC can
refer to topics lin
Hi All,
Thanks for this amazing KIP! I'm glad that Kafka is getting more accessible
multi-tenancy features, beyond the primitives that are already available.
1. I think one concept for hierarchical VCs could be to make it so there is
a "root VC" corresponding to the physical cluster, and define t
Hi Luke,
1. ACL after migrating to VC:
Suppose we have an ACL entry to user Bob, after Bob joining VC1, how does
the ACL migration work?
Will Bob still have the ACL in cluster-wide level?
Yes, we didn't plan for this originally, so it would be the administrator's
responsibility to migrate existin
Hi YengTao,
Yes, we left out the naming mechanism but not intentionally. I'll add it
after answering your feedback.
- Will this leverage Kafka's existing UUID mechanism, or are there plans
to implement a new naming mechanism?
- Will there be standardized formats for the length and structu
Hi Viktor,
This is a very interesting KIP!
It keeps most of the existing concept and design to create a new virtual
cluster concept. Nice!
Some questions:
1. ACL after migrating to VC:
Suppose we have an ACL entry to user Bob, after Bob joining VC1, how does
the ACL migration work?
Will Bob still
Hi Viktor,
Thank you and all the authors for this great KIP!
I've taken a quick pass and have some questions about the topic-generated
name mechanism within virtual clusters.
The KIP mentions that topics created within virtual clusters will have
generated names to avoid conflicts. However, it do
Hi David,
Thanks for reading it through in advance. So I think there are 3 options
for resource isolation today, let me go through them for context. The
4th is how virtual clusters help the 3rd option :).
1. We can create completely separate Kafka clusters and host them
separately without any othe
Thanks for the KIP, Viktor (et al)!
This certainly looks like an interesting feature. I had one initial
question upon my first cursory readthrough: how does a virtual cluster help
with isolation? I see this mentioned as a motivation, but it's not
immediately obvious to me how it changes resource i
Hi all,
I'd like to start a discussion on our new feature proposal: virtual
clusters in Kafka. In this KIP we propose the leveling up of multi-tenancy
to better support organizations who manage their deployments centrally for
multiple teams.
Please take a look if you're interested and I'm happy to
21 matches
Mail list logo
