On Wed, Feb 15, 2023 at 4:36 PM <mattisonc...@gmail.com> wrote: > Hi, All > > First of all, I want to list all of the system topics as follows. That > Yunze has mentioned before. > > Namespace level: > > • pulsar/system > • transaction_coordinator_assign > • __transaction_log_ > • resource-usage > • pulsar/<host> > • healthcheck > > Topic level: > > • __change_events > • __transaction_buffer_snapshot_segment > • ... > > We can check the details here. [0] > > Secondly. I think we need focus on the system topic name prefix. we have > some options as follows: > > • __SYSTEM__ > • __system__ > > > Both them make sense for me. As there are already two people prefer > `__SYSTEM`. if no other people have concern, I will chose it. >
I vote `__system__` since I think upper casing is not that common for internal names (for me it's "shouting" and requires an extra key press to write it). Examples: Cassandra System tables <https://docs.datastax.com/en/dse/5.1/cql/cql/cql_using/useQuerySystem.html> : * batches Postgres database catalog <https://www.postgresql.org/docs/current/catalogs.html>: * pg_database <https://www.postgresql.org/docs/current/catalog-pg-database.html> OpenSearch system indices <https://opensearch.org/docs/1.0/security-plugin/configuration/system-indices/> : * .opendistro-alerting-config So I think we should stick to `__system__` > > Best, > Mattison > > -[0] > https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/naming/SystemTopicNames.java > On Feb 15, 2023, 11:36 +0800, Michael Marshall <mmarsh...@apache.org>, > wrote: > > I support this PIP, thank you for driving it forward Mattison. > > > > I support using the prefix __SYSTEM__ for system topics, and I agree > > that DLQ and RETRY topics do not qualify as system topics because they > > are produced to and consumed by regular clients. > > > > I think we might benefit from a generic definition for a system topic. > > I provided a loose definition for system topics in this email [0]. I > > copy it here: > > > > A system topic is a topic that is completely internal to Pulsar > > components. Internally, it is a normal topic. It requires elevated > > permission to produce/consume when authorization is enabled, even if > > topic level policies are not enabled. Generic calls like > > `clearNamespaceBacklog` should not affect system topics. Deleting a > > namespace or tenant should delete the system topics within it. > > > > Thanks, > > Michael > > > > [0] https://lists.apache.org/thread/sr01hvqmdrnk4lxwfwzcpg7y21psj6tt > > > > > > On Tue, Feb 14, 2023 at 9:55 AM <mattisonc...@gmail.com> wrote: > > > > > > > > Could you list all existing system topic names that are used so we > canunderstand the rule better? > > > Yes, sure. I will list it later. > > > > > And I saw that this proposal only forbids the creation of > thesetopics. What about writing messages to them? I think it's better notto > allow Pulsar clients to write messages to them. The Pulsar clientshould > configure something to get the permission to write messages tothem. To keep > the compatibility, maybe we can make use of the protocolversion. > > > Sure, I agree with your point, because the current proposal just want > to have topic name restriction. maybe we can revise the system topic > permission and etc in the next proposal. (relate to system topic) > > > > > > Best, > > > Mattison > > > On Feb 14, 2023, 21:22 +0800, Yunze Xu <y...@streamnative.io.invalid>, > wrote: > > > > > Could you list all existing system topic names that are used so we > can > > > > > understand the rule better? Such as the "-RETRY" and "-DLQ" topics > > > > > mentioned above. > > > > > > > > > > And I saw that this proposal only forbids the creation of these > > > > > topics. What about writing messages to them? I think it's better > not > > > > > to allow Pulsar clients to write messages to them. The Pulsar > client > > > > > should configure something to get the permission to write messages > to > > > > > them. To keep the compatibility, maybe we can make use of the > protocol > > > > > version. > > > > > > > > > > Thanks > > > > > Yunze > > > > > > > > > > On Tue, Feb 14, 2023 at 5:38 PM Yubiao Feng > > > > > <yubiao.f...@streamnative.io.invalid> wrote: > > > > > > > > > > > > > > Hi Qiang > > > > > > > > > > > > > > ### System topic name > > > > > > > '__system__<name>' I think this format is clearer. > > > > > > > Now the system automatically creates topics of type retry > consumer letters > > > > > > > and dead letters. > > > > > > > These topics all end in uppercase letters, such as `-RETRY,` > `-DLQ.` > > > > > > > Is it better to define the system topic name in uppercase( > > > > > > > '__SYSTEM__<name>' )? > > > > > > > > > > > > > > ### Another comment > > > > > > > If you now redesign the topic name restrictions, should we > make `-RETRY` > > > > > > > and `-DLQ` keywords? > > > > > > > > > > > > > > Thanks > > > > > > > Yubiao Feng > > > > > > > > > > > > > > On Tue, Feb 14, 2023 at 5:06 PM Asaf Mesika < > asaf.mes...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > On Tue, Feb 14, 2023 at 3:46 AM < > mattisonc...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, Asaf > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Welcome to join this discussion. > > > > > > > > > > > > > > > > > > > You mean that allows the *system* to > use it when it's a partitioned > > > > > > > > > > > > > > > topic? > > > > > > > > > > > > > > > Sorry, I didn't get your point. What do you > mean by *system*? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This sentence was a reply to: > > > > > > > > > > > > > > > > > > > > > > 2. Make the `-partition-` string the keyword. That > allows the user to use > > > > > > > > > > > > > > > it when it's a partitioned topic. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I wanted to say that this sentence should be: > > > > > > > > > > > Make the `-partition-` string the keyword, that allows > the *system* to use > > > > > > > > > > > it when it's a partitioned topic. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Why postfix of `__`?Why uppercase > ?Maybe `__system__<name>`? > > > > > > > > > > > > > > > Yes, That is a key point that I want to > discuss in this > > > > > > > > > > > > > > > thread. `__system__<name>` is good for me. > > > > > > > > > > > > > > > > > > > Can you please elaborate what it means > to make it dynamic exactly? > > > > > > > > > > > > > > > Sorry, I will refine it. it means we can > update this configuration > > > > > > > > > > > > > > > dynamically. (using rest api or sth) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm unfamiliar with how Pulsar supports dynamic > configuration. I would > > > > > > > > > > > love it if you can share a link or explain it briefly, > thus explaining what > > > > > > > > > > > exactly you are going to change to support dynamic > configuration. > > > > > > > > > > > > > > > > > > > > > > > > > > General question: In the last thread you said > something about > > > > > > > > > > > > > > > configurablerules, etc? You decided not to use > this idea? > > > > > > > > > > > > > > > IMO, That idea is an advanced feature. we may > need more time to discuss > > > > > > > > > > > > > > > the details and for the topic name > restriction, maybe we don't have > > > > > > > > > > > strong > > > > > > > > > > > > > > > reason to use that. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We can introduce this advanced feature when we > have a need for it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I agree. Future PIP and discussion. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > Mattison > > > > > > > > > > > > > > > On Feb 13, 2023, 22:21 +0800, Asaf Mesika < > asaf.mes...@gmail.com>, > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > You mean that allows the *system* to > use it when it's a partitioned > > > > > > > > > > > > > > > topic? > > > > > > > > > > > > > > > > > > > > > > > > > > >
