Thanks for the response Joe. I think I understand where you are coming, but as I look at the current capabilities for putting limits on a namespace (backlog quotas, publish and subscribe rate limits, ttl, etc) there does appear to be a fair number of controls in place today that do allow for *some* control over what can be done within a given namespace such that I can reasonably delegate *some* responsibility. Notice that in the permissions I listed, I gave very little control to "namespace admins" for any namespace polices, even with just topic creation for namespace admins, I think that would be very beneficial to have that capability. I understand that usage of these resources aren't hierarchical (i.e. inherited from a tenant) and therefore I am still placing a fair amount of trust on tenant admins to set those values reasonably (that can have global impact) and to then delegate that permission to namespace admins. However, I would put forth that in many situations, it may be much less effort to monitor and alert on these sort of situations of using too many resources than it is to build external automation to "safely" create and manage resources or use a tenant strategy that doesn't map well to my business (where the visibility of multiple components that make up a product is very natural mapping to a namespace and a tenant)
In other words, I fully understand it isn't necessarily ideal, but I think it makes sense for me to be able to make those trade-offs as I implement Pulsar. I am positive this will improve over time using some of the longer term ideas discussed in this thread, but I would also advocate that iteration and a bit more flexibility today would have some real utility (with some trade-offs that need to be understood by operators of Pulsar). Pivoting a bit, I am curious to hear any opinions regarding a more "full featured" approach to defining policies, such as a policy language to allow arbitrary sets of permissions. I think that obviously gives the most flexibility, but does shift a large amount of complexity onto the user to figure out what policies are safe. Once again, thanks for the discussion and happy to add anything else if it is useful in helping figure out the right answer for Pulsar as well as seeing how we can help out in making it happen :) Addison On Tue, Nov 12, 2019 at 12:02 AM Joe F <j...@apache.org> wrote: > >Is there any appetite for considering a "namespace admin"? > > I think this is where there is an issue. As Sijie aptly pointed out, this > namespace "admin", as a concept, does not exist in Pulsar for resource > allocation. It will upend the whole design of how resources are managed > in Pulsar. While we started this PIP as a discussion on permissions, it is > not possible do many of these asks as simply permissions. Implicitly, it's > reworking the whole resource management model in Pulsar. And I think that > if we are attempting to re-work resource management, it should be > discussed as such, and not occur as an unnoticed side effect of attempting > to manage permissions . > > >I think that set of permissions (with some deletions or adjustments) could > go a long way to making the current permissions model work a fair bit > better for orgs trying to deploy pulsar. > > Ultimately, when you give someone access to a resource, they are going to > use it. And that use has a cost. Who pays the cost, and how is it managed > and accounted? Power to manage access is power to spend. To what level do > we do resource controls? > > I am a little lost on the idea of sub-tenants (namespace admins). This is > the point where I would think that the system model would be better off > having those namespace as first-class tenants, and not as namespaces. > Pulsar is, not as it is, designed to have sub-tenants. > > Joe > > > > > On Mon, Nov 11, 2019 at 9:35 AM Addison Higham <addis...@gmail.com> wrote: > > > Is there any appetite for considering a "namespace admin"? We could > really > > have use of this as currently we have to give more people than we would > > want tenant admin to ease administration, when many users basically just > > need permissions to create topics and grant permissions to those topics. > > > > From my reading of this chain, it is obvious that it probably won't be > > given a ton of permissions and it may be complex to add, but it seems > like > > starting with a conservative approach of permissions for a namespace > admin > > initially could be done fairly quickly and safely. > > > > I would say the following could make a a lot of sense to be in a > namespace > > admin permissions: > > > > Topics > > - compaction/compaction status > > - offload/offload-status > > - create/delete non partitioned topics > > - create/delete partitioned topics > > - list/grant/revoke permissions > > - create/delete/peek/seek subscriptions > > - stats calls > > > > Namespaces > > - list/grant/revoke permissions > > - set/get clusters (maybe not?) > > - get/set ttl > > - get/set persistence (maybe not?) > > > > I think that set of permissions (with some deletions or adjustments) > could > > go a long way to making the current permissions model work a fair bit > > better for orgs trying to deploy pulsar. > > > > Long term, it feels like the correct answer to this problem is probably > to > > introduce some sort of policy language with a role being tied to some > > policy that can grant a wide range of individual permissions, with the > > existing superuser/tenant-admin/etc being redefined in terms of some of > > these policies (perhaps using something like https://casbin.org/en/). If > > it > > seems like the focus should be on the more general solution, then I am > fine > > leaving this for now, but this will be something we need to tackle in the > > next 6 months or so for my org, so I would love to see any sort of > > direction of where things should be heading so we can plan to that and > > perhaps help where possible! > > > > > > > > On Mon, Nov 11, 2019 at 2:24 AM xiaolong ran <ranxiaolong...@gmail.com> > > wrote: > > > > > Hello everyone: > > > > > > I reorganize the relevant content of this PIP. PTAL again. > > > > > > > > > https://github.com/apache/pulsar/wiki/PIP-49:-Permission-levels-and-inheritance > > > < > > > > > > https://github.com/apache/pulsar/wiki/PIP-49:-Permission-levels-and-inheritance > > > > > > > > > > In this PIP, I only fix the unreasonable permissions in the current > > > command and shown in bold. > > > About specific permission of command, if there are different opinions > > > about this, pls let me know, thanks. > > > > > > -- > > > > > > Thanks > > > Xiaolong Ran > > > > > > > > > > 在 2019年11月8日，下午4:41，xiaolong ran <ranxiaolong...@gmail.com> 写道： > > > > > > > > Thanks Sijie, Joe F and Addison Highham feedback. > > > > > > > > As Joe said: > > > > > > > > > There are no simple answers here other than to understand the > effect > > > of the command. > > > > > > > > So in here, we can use sijie proposal, we only need to fix the > > > unreasonable permissions in the current command. > > > > > > > > I will reorganize this PIP, what do you think about this? > > > > > > > > -- > > > > > > > > Thanks > > > > Xiaolong Ran > > > > > > > > > > > >> 在 2019年11月8日，下午3:59，Sijie Guo <guosi...@gmail.com <mailto: > > > guosi...@gmail.com>> 写道： > > > >> > > > >> If the problem is complex enough, can we first reduce the scope of > > this > > > PIP > > > >> to focus on fixing the permissions of individual commands (e.g. > > schemas > > > >> don't require super-user permissions)? > > > >> As you said, there is no simple answers. We can go through the > > commands > > > one > > > >> by one and come up a table of commands and their permissions and > > > document > > > >> it. > > > >> It can help users understand how the permissions work for each > > command. > > > >> > > > >> We can revisit a larger scope till we add the capability of > allocation > > > >> resources at different levels. > > > >> > > > >> Does that make sense? > > > >> > > > >> - Sijie > > > >> > > > >> > > > >> On Fri, Nov 8, 2019 at 3:22 PM Joe F <j...@apache.org <mailto: > > > j...@apache.org>> wrote: > > > >> > > > >>> There are no simple answers here other than to understand the > effect > > > of the > > > >>> command. > > > >>> > > > >>> The resource limit/control command always addresses an object. But > > > giving > > > >>> control of that command to the object owner, just because the > command > > > >>> addresses an object, will break all controls about resource > > > utilization in > > > >>> a shared system. A filesystem like model may look simple and > elegant > > > but > > > >>> is fraught with all kind of unintended consequences if implemented > in > > > >>> Pulsar (Even in the filesystem rm, mv and chown, while all > > addressing > > > a > > > >>> file, treats the file owner differently) > > > >>> > > > >>> This is a complex, and complicated effort, and my concern is that > > > this PIP > > > >>> does not even come close to the scope of the work that is > required. > > > Some > > > >>> of them may be impossible to do (as allocating certain resources to > > > >>> namespace as compared to tenant) till such capabilities are added > to > > > the > > > >>> system. And some begs fundamental questions - when you start > > managing > > > a > > > >>> namespace like a tenant, does it even make sense to have that > > > namespace? > > > >>> Should't that namespace be a tenant in the first place? Why > have a > > > >>> namespace admin at all? > > > >>> > > > >>> Joe > > > >>> > > > >>> > > > >>> > > > >>> On Thu, Nov 7, 2019 at 8:16 PM Dave Fisher <wave4d...@comcast.net > > > <mailto:wave4d...@comcast.net>> wrote: > > > >>> > > > >>>> I’m not diving in but thinking about the logical implication of > this > > > >>>> dichotomy. For any object’s attributes some ought to be controlled > > by > > > >>>> object level permissions and others by sysadmin permissions. How > can > > > >>>> developers tell? > > > >>>> > > > >>>> Best Regards, > > > >>>> Dave > > > >>>> > > > >>>> > > > >>>> Sent from my iPhone > > > >>>> > > > >>>>> On Nov 7, 2019, at 8:02 PM, Joe F <j...@apache.org <mailto: > > > j...@apache.org>> wrote: > > > >>>>> > > > >>>>> This proposal has the same issues as the previous one . In > general > > > it > > > >>> is > > > >>>>> not correct to think of commands as being controlled by owner of > > the > > > >>>>> object on which the command operates. Eg: It will be an error to > > > >>> assing > > > >>>>> control of all namespace commands to the namespace owner > > > >>>>> > > > >>>>> For eg: set subscription-dispatch-rate operates on a > subscription > > > rate > > > >>>> and > > > >>>>> set-max-producers-per-topic These operate on namespaces. A naive > > > >>> approach > > > >>>>> is to think that this would be controlled by the namespace owner. > > > But > > > >>>>> essentially both these are resource management operations. That > > > should > > > >>>> be > > > >>>>> controlled by the system admin, unless you want to deal with > every > > > >>>>> namespace owner having the power to destroy your SLAs fo the > > cluster. > > > >>>>> > > > >>>>> I just picked 2 examples to indicate the drawback of approaching > > > >>>>> permissions in the proposed manner. > > > >>>>> > > > >>>>> There are many such cases in the proposal, too many to list out > > > here. I > > > >>>>> suggest completely ditching this approach of equating objects > with > > > >>> admin > > > >>>>> capability of object owner > > > >>>>> > > > >>>>> Joe > > > >>>>> > > > >>>>> > > > >>>>>> On Wed, Nov 6, 2019 at 2:14 AM xiaolong ran < > > > ranxiaolong...@gmail.com <mailto:ranxiaolong...@gmail.com> > > > >>>> > > > >>>>>> wrote: > > > >>>>>> > > > >>>>>> Hello all committers: > > > >>>>>> > > > >>>>>> About this PIP, do you have any other good ideas or propose. > > > >>>>>> > > > >>>>>> > > > >>>>>> -- > > > >>>>>> > > > >>>>>> Thanks > > > >>>>>> Xioalong Ran > > > >>>>>> > > > >>>>>> > > > >>>>>>>> 在 2019年10月30日，下午7:49，xiaolong ran <ranxiaolong...@gmail.com > > > <mailto:ranxiaolong...@gmail.com>> 写道： > > > >>>>>>> > > > >>>>>>> Hello all: > > > >>>>>>> > > > >>>>>>> When using pulsar-admin, I found that the current permission > > > >>>>>> verification mechanism has some problems. > > > >>>>>>> > > > >>>>>>> We are starting a proposal about permission levels and > > inheritance > > > in > > > >>>>>> Apache Pulsar. > > > >>>>>>> > > > >>>>>>> The proposal is in: > > > >>>>>>> > > > >>>>>> > > > >>>> > > > >>> > > > > > > https://github.com/apache/pulsar/wiki/PIP-49%3A-Permission-levels-and-inheritance > > > < > > > > > > https://github.com/apache/pulsar/wiki/PIP-49%3A-Permission-levels-and-inheritance > > > > > > > >>>>>> < > > > >>>>>> > > > >>>> > > > >>> > > > > > > https://github.com/apache/pulsar/wiki/PIP-49:-Permission-levels-and-inheritance > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> To ensure compatibility, these changes will be made in v4's > admin > > > >>> api. > > > >>>>>> About the v4 admin API, see: > > > >>>>>>> > > > >>>>>>> > > > >>> > > https://github.com/apache/pulsar/wiki/PIP-48%3A-hierarchical-admin-api > > > >>>> < > > > >>>>>> > > > https://github.com/apache/pulsar/wiki/PIP-48:-hierarchical-admin-api> > > > >>>>>>> > > > >>>>>>> Looking forward to any feedback. > > > >>>>>> > > > >>>>>> > > > >>>> > > > >>>> > > > >>> > > > > > > > > > > > > >
