Hi Luke, 1. ACL after migrating to VC: Suppose we have an ACL entry to user Bob, after Bob joining VC1, how does the ACL migration work? Will Bob still have the ACL in cluster-wide level?
Yes, we didn't plan for this originally, so it would be the administrator's responsibility to migrate existing ACLs, however thinking about it, the migration could be quite simple (essentially we would just need to use the same ACLs but move them to under VC1 so they're not global). Therefore if you think it could be useful, we could perhaps provide tooling for this. 2. What does topic linking mean? I don't understand what it means with a topic link to a VC. If a topic foo links to all VC1, VC2, VC3, does that mean all users under VC 1, 2, 3 can write/read data from it? It's like a shared topic for all these VCs? If so, will a consumer group consist of consumer1 in VC1, consumer2 in VC2...? Yes, topic linking is basically creating virtual topics that exist within the context of a virtual cluster but they point to a real topic. Also yes to your example, we wanted topics to be shareable between virtual clusters, so VC1, VC2 and VC3 all could have a link to topic "foo" and clients in those VCs could read/write them. For consumer groups, I wouldn't provide this sharing capability. Consumer groups created by users in VCs would be prefixed by the VC name and their offsets would be stored in __consumer-offsets by this name, like VC1.consumer1. They would be able to consume from topics in other VCs via topic sharing. 3. I think a VC inside a VC is not permitted, right? Yes, correct. We in fact originally started with a concept of VC hierarchies which I think is similar in lots of ways, but after quite a few brainstorming sessions we discarded this idea as it makes the whole design very complex vs the current simpler design where we would get the same features cheaper. To summarize it we found the following main problems: - Topics could be visible just in the VC they're created in or in the sub-VCs (child VCs?). Both can be arguable and have use-cases, yet the implementation really should decide on one, therefore it's too restricting. If we created a topic sharing feature in this environment, then that would lead to complicated graphs we don't want as it's hard to work with them. - To refer to topics that in a VC that's inside another VC can be complicated similarly to a hierarchy. We can choose to have absolute naming, relative naming or just keep physical naming. It's complicated to design this. If we allow relative or absolute naming, then we would need to introduce a path separator character in clients, which makes them incompatible, so users would need to upgrade all their clients too if they wanted to take advantage of VCs. If we don't allow referencing into sub-VCs, then they lose some part of their usefulness. They could still be used as configuration templates or ACL templates where all resources within them inherit the same defaults, but I think without topic shareability they'd lose their attractiveness. - Implementing ACLs can be complicated as well. First, whatever we decided in the previous points, ACLs would have to support that, so its complexity really depends on the previous points. Then there is an argument whether ACLs are transparent or not. In the current design, they're transparent, which means that being part of a VC wouldn't inherently prevent you from creating a topic if you have the correct permissions on the "kafka-cluster" resource. But in case of sub-VCs this may or may not be beneficial, depending on your use-case: do you want sub-VCs to inherit ACLs or not. This is again a design decision where we may need to sacrifice a part of valid use-cases. With the current design this problem doesn't exist as it's transparent and we don't have VCs. Users would have the freedom to write tools using the Admin API that would set the ACLs properly if a new VC is created. 4. The view of the VC users. Supposedly, when a VC user creates a topic "foo", it should be able to see the topic "foo" created. But now it creates a "UUID-foo" topic. Will we do something here to help VC users using virtual-clusters.sh CLI? VC users will always see the links or virtual topics instead of the real ones. So the broker will know if a user is assigned to a VC and translates "UUID.foo" to "foo" and returns that in its response. This naming pattern would be totally transparent for the clients inside a VC. This is true for all protocols (produce, consume etc.). For clients outside, we don't really know better but to return the physical name of the topic which is UUID.foo . 5. Monitoring How do we monitor the resources for a specific VC? Do we ever think about it? This is a good point, we haven't gotten to it yet. I'll circle back to you in a few days. Do you have any ideas in mind? I'd be happy to consider them. Thanks, Viktor On Tue, Feb 18, 2025 at 2:34 PM Viktor Somogyi-Vass < viktor.somo...@cloudera.com> wrote: > Hi YengTao, > > Yes, we left out the naming mechanism but not intentionally. I'll add it > after answering your feedback. > > - Will this leverage Kafka's existing UUID mechanism, or are there plans > to implement a new naming mechanism? > - Will there be standardized formats for the length and structure of > these generated names? > > We planned to leverage the UUID naming mechanism for topics that are > created by users. Initially the thought was that we could completely > replace topic names with UUIDs but unfortunately it can't be done because > topic names are deeply embedded in Kafka. So we came to the conclusion that > we should just prefix topic names with UUIDs, like > "abcd-1234-dcba-3412.mytopic"). This is also advantageous because from the > debugging perspective, it's easier to identify topics and their assumed > content if their names contain a human-readable part. It also has to be a > valid topic name so that non-VC clients can access them if needed (backward > compatibility, we don't want to make VC topics incompatible with non-VC > clients). Therefore we could either separate topics by ".", "_" or "-". We > choose "." but it could be any other. > Also, topic names that have been created before virtual clusters (or > outside virtual clusters but later linked to them), would behave > normally, so if a user named it "mytopic", then we wouldn't prefix it with > anything. > > Thanks, > Viktor > > On Mon, Feb 17, 2025 at 9:22 AM Luke Chen <show...@gmail.com> wrote: > >> Hi Viktor, >> >> This is a very interesting KIP! >> It keeps most of the existing concept and design to create a new virtual >> cluster concept. Nice! >> >> Some questions: >> 1. ACL after migrating to VC: >> Suppose we have an ACL entry to user Bob, after Bob joining VC1, how does >> the ACL migration work? >> Will Bob still have the ACL in cluster-wide level? >> >> 2. What does topic linking mean? >> I don't understand what it means with a topic link to a VC. >> If a topic foo links to all VC1, VC2, VC3, does that mean all users under >> VC 1, 2, 3 can write/read data from it? >> It's like a shared topic for all these VCs? >> If so, will a consumer group consist of consumer1 in VC1, consumer2 in >> VC2...? >> >> 3. I think a VC inside a VC is not permitted, right? >> >> 4. The view of the VC users. >> Supposedly, when a VC user creates a topic "foo", it should be able to see >> the topic "foo" created. >> But now it creates a "UUID-foo" topic. Will we do something here to help >> VC >> users using virtual-clusters.sh CLI? >> >> 5. Monitoring >> How do we monitor the resources for a specific VC? >> Do we ever think about it? >> >> Thanks. >> Luke >> >> On Sat, Feb 15, 2025 at 4:46 PM TengYao Chi <kiting...@gmail.com> wrote: >> >> > Hi Viktor, >> > >> > Thank you and all the authors for this great KIP! >> > >> > I've taken a quick pass and have some questions about the >> topic-generated >> > name mechanism within virtual clusters. >> > >> > The KIP mentions that topics created within virtual clusters will have >> > generated names to avoid conflicts. However, it doesn't seem to mention >> or >> > define specifications for this name-generation pattern. >> > >> > Specifically: >> > >> > >> > - Will this leverage Kafka's existing UUID mechanism, or are there >> plans >> > to implement a new naming mechanism? >> > - Will there be standardized formats for the length and structure of >> > these generated names? >> > >> > Thank you for answering my questions :) >> > >> > Best Regards, >> > TengYao >> > >> > >> > >> > Viktor Somogyi-Vass <viktor.somo...@cloudera.com.invalid> 於 2025年2月14日 >> 週五 >> > 下午10:11寫道： >> > >> > > Hi David, >> > > >> > > Thanks for reading it through in advance. So I think there are 3 >> options >> > > for resource isolation today, let me go through them for context. The >> > > 4th is how virtual clusters help the 3rd option :). >> > > 1. We can create completely separate Kafka clusters and host them >> > > separately without any other software running on those hosts. In this >> > > scenario you are completely in control of the hardware (cpu, network, >> > disk >> > > utilization). This however may be the most expensive solution too >> because >> > > hosting your own servers can be expensive for many users. Also you're >> > > likely left with unused capacity. >> > > 2. Then if we think of more economic solutions, we can choose to host >> > Kafka >> > > clusters together in a virtualized environment. There you should be >> able >> > to >> > > control the hardware resources but from an operation point of view >> you'll >> > > have a harder time as you'll need to manage more Kafka clusters (need >> to >> > > set up ACLs, quotas and configuration and keep them in sync if >> needed). >> > > Also, if there is a need for sharing data between these, it's not easy >> > > because then you'll need to set up replication between those two >> > clusters. >> > > 3.To overcome the cluster management overhead, it's better to host one >> > big >> > > Kafka cluster, and this is what larger organizations tend to do. Then >> a >> > > central team manages the installation (if self-hosted and they're not >> > using >> > > a cloud service) and provides Kafka as a service for other teams. This >> > > often isn't perfect either, because they need to know about every new >> > > client or user to set proper network quotas (besides the catch-all >> > > default). >> > > 4. With virtual clusters we improve this situation by quotas for >> virtual >> > > clusters. You would set virtual cluster specific quotas that wouldn't >> be >> > > possible to exceed for all the clients running together on that VC. >> > Besides >> > > this, virtual cluster admins could be defined who may have better >> > knowledge >> > > about their cluster and their team and set quotas in the clusters >> > > appropriately. >> > > >> > > Viktor >> > > >> > > On Fri, Feb 14, 2025 at 12:50 AM David Arthur <mum...@gmail.com> >> wrote: >> > > >> > > > Thanks for the KIP, Viktor (et al)! >> > > > >> > > > This certainly looks like an interesting feature. I had one initial >> > > > question upon my first cursory readthrough: how does a virtual >> cluster >> > > help >> > > > with isolation? I see this mentioned as a motivation, but it's not >> > > > immediately obvious to me how it changes resource isolation. >> > > > >> > > > I'll do a more thorough review and come back with more questions :) >> > > > >> > > > -David A >> > > > >> > > > >> > > > On Thu, Feb 13, 2025 at 12:11 PM Viktor Somogyi-Vass >> > > > <viktor.somo...@cloudera.com.invalid> wrote: >> > > > >> > > > > Hi all, >> > > > > >> > > > > I'd like to start a discussion on our new feature proposal: >> virtual >> > > > > clusters in Kafka. In this KIP we propose the leveling up of >> > > > multi-tenancy >> > > > > to better support organizations who manage their deployments >> > centrally >> > > > for >> > > > > multiple teams. >> > > > > Please take a look if you're interested and I'm happy to take any >> > > > feedback >> > > > > on it. >> > > > > >> > > > > The link to the KIP: >> > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1134%3A+Virtual+Clusters+in+Kafka >> > > > > >> > > > > Thanks, >> > > > > Viktor >> > > > > >> > > > >> > > > >> > > > -- >> > > > David Arthur >> > > > >> > > >> > >> >
