Hi Viktor, This is a very interesting KIP! It keeps most of the existing concept and design to create a new virtual cluster concept. Nice!
Some questions: 1. ACL after migrating to VC: Suppose we have an ACL entry to user Bob, after Bob joining VC1, how does the ACL migration work? Will Bob still have the ACL in cluster-wide level? 2. What does topic linking mean? I don't understand what it means with a topic link to a VC. If a topic foo links to all VC1, VC2, VC3, does that mean all users under VC 1, 2, 3 can write/read data from it? It's like a shared topic for all these VCs? If so, will a consumer group consist of consumer1 in VC1, consumer2 in VC2...? 3. I think a VC inside a VC is not permitted, right? 4. The view of the VC users. Supposedly, when a VC user creates a topic "foo", it should be able to see the topic "foo" created. But now it creates a "UUID-foo" topic. Will we do something here to help VC users using virtual-clusters.sh CLI? 5. Monitoring How do we monitor the resources for a specific VC? Do we ever think about it? Thanks. Luke On Sat, Feb 15, 2025 at 4:46 PM TengYao Chi <kiting...@gmail.com> wrote: > Hi Viktor, > > Thank you and all the authors for this great KIP! > > I've taken a quick pass and have some questions about the topic-generated > name mechanism within virtual clusters. > > The KIP mentions that topics created within virtual clusters will have > generated names to avoid conflicts. However, it doesn't seem to mention or > define specifications for this name-generation pattern. > > Specifically: > > > - Will this leverage Kafka's existing UUID mechanism, or are there plans > to implement a new naming mechanism? > - Will there be standardized formats for the length and structure of > these generated names? > > Thank you for answering my questions :) > > Best Regards, > TengYao > > > > Viktor Somogyi-Vass <viktor.somo...@cloudera.com.invalid> 於 2025年2月14日 週五 > 下午10:11寫道： > > > Hi David, > > > > Thanks for reading it through in advance. So I think there are 3 options > > for resource isolation today, let me go through them for context. The > > 4th is how virtual clusters help the 3rd option :). > > 1. We can create completely separate Kafka clusters and host them > > separately without any other software running on those hosts. In this > > scenario you are completely in control of the hardware (cpu, network, > disk > > utilization). This however may be the most expensive solution too because > > hosting your own servers can be expensive for many users. Also you're > > likely left with unused capacity. > > 2. Then if we think of more economic solutions, we can choose to host > Kafka > > clusters together in a virtualized environment. There you should be able > to > > control the hardware resources but from an operation point of view you'll > > have a harder time as you'll need to manage more Kafka clusters (need to > > set up ACLs, quotas and configuration and keep them in sync if needed). > > Also, if there is a need for sharing data between these, it's not easy > > because then you'll need to set up replication between those two > clusters. > > 3.To overcome the cluster management overhead, it's better to host one > big > > Kafka cluster, and this is what larger organizations tend to do. Then a > > central team manages the installation (if self-hosted and they're not > using > > a cloud service) and provides Kafka as a service for other teams. This > > often isn't perfect either, because they need to know about every new > > client or user to set proper network quotas (besides the catch-all > > default). > > 4. With virtual clusters we improve this situation by quotas for virtual > > clusters. You would set virtual cluster specific quotas that wouldn't be > > possible to exceed for all the clients running together on that VC. > Besides > > this, virtual cluster admins could be defined who may have better > knowledge > > about their cluster and their team and set quotas in the clusters > > appropriately. > > > > Viktor > > > > On Fri, Feb 14, 2025 at 12:50 AM David Arthur <mum...@gmail.com> wrote: > > > > > Thanks for the KIP, Viktor (et al)! > > > > > > This certainly looks like an interesting feature. I had one initial > > > question upon my first cursory readthrough: how does a virtual cluster > > help > > > with isolation? I see this mentioned as a motivation, but it's not > > > immediately obvious to me how it changes resource isolation. > > > > > > I'll do a more thorough review and come back with more questions :) > > > > > > -David A > > > > > > > > > On Thu, Feb 13, 2025 at 12:11 PM Viktor Somogyi-Vass > > > <viktor.somo...@cloudera.com.invalid> wrote: > > > > > > > Hi all, > > > > > > > > I'd like to start a discussion on our new feature proposal: virtual > > > > clusters in Kafka. In this KIP we propose the leveling up of > > > multi-tenancy > > > > to better support organizations who manage their deployments > centrally > > > for > > > > multiple teams. > > > > Please take a look if you're interested and I'm happy to take any > > > feedback > > > > on it. > > > > > > > > The link to the KIP: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1134%3A+Virtual+Clusters+in+Kafka > > > > > > > > Thanks, > > > > Viktor > > > > > > > > > > > > > -- > > > David Arthur > > > > > >
