Re: [DISCUSS] Releasing Pulsar-client-go 0.9.0

2022-06-29 Thread PengHui Li
+1
On Jun 29, 2022, 14:08 +0800, Zixuan Liu , wrote:
> +1
>
> On 2022/06/29 06:06:01 Rui Fu wrote:
> > Hello Everyone:
> >
> > I hope you’ve all been doing well. In the past couple of months, we have 
> > fixed many bugs and added some new features for the pulsar-client-go. For 
> > more information, refer to:
> >
> > https://github.com/apache/pulsar-client-go/milestone/10?closed=1
> >
> > For that reason, let us start the release of v0.9.0.
> >
> > Best,
> >
> > Rui Fu
> >


Re: [DISCUSS] Releasing Pulsar-client-go 0.9.0

2022-06-29 Thread mattison chao
+1

On Wed, 29 Jun 2022 at 15:44, PengHui Li  wrote:

> +1
> On Jun 29, 2022, 14:08 +0800, Zixuan Liu , wrote:
> > +1
> >
> > On 2022/06/29 06:06:01 Rui Fu wrote:
> > > Hello Everyone:
> > >
> > > I hope you’ve all been doing well. In the past couple of months, we
> have fixed many bugs and added some new features for the pulsar-client-go.
> For more information, refer to:
> > >
> > > https://github.com/apache/pulsar-client-go/milestone/10?closed=1
> > >
> > > For that reason, let us start the release of v0.9.0.
> > >
> > > Best,
> > >
> > > Rui Fu
> > >
>


Re: [DISCUSS] Releasing Pulsar-client-go 0.9.0

2022-06-29 Thread ZhangJian He
+1

Thanks
ZhangJian He

mattison chao  于2022年6月29日周三 15:57写道:

> +1
>
> On Wed, 29 Jun 2022 at 15:44, PengHui Li  wrote:
>
> > +1
> > On Jun 29, 2022, 14:08 +0800, Zixuan Liu , wrote:
> > > +1
> > >
> > > On 2022/06/29 06:06:01 Rui Fu wrote:
> > > > Hello Everyone:
> > > >
> > > > I hope you’ve all been doing well. In the past couple of months, we
> > have fixed many bugs and added some new features for the
> pulsar-client-go.
> > For more information, refer to:
> > > >
> > > > https://github.com/apache/pulsar-client-go/milestone/10?closed=1
> > > >
> > > > For that reason, let us start the release of v0.9.0.
> > > >
> > > > Best,
> > > >
> > > > Rui Fu
> > > >
> >
>


[DISCUSS] PIP-181: Provide new load balance placement strategy implementation for ModularLoadManagerStrategy

2022-06-29 Thread 叶韵
Hi Pulsar community:
I open a pip to discuss "Shadow Topic, an alternative way to support
readonly topic ownership."
Proposal Link:

   - issue link: https://github.com/apache/pulsar/issues/16274
   - pr link: https://github.com/apache/pulsar/pull/16281

---
## Motivation
The modular load manager, implemented in `ModularLoadManagerImpl`, is a
flexible alternative to the previously implemented load manager, which
attempts to simplify how load is managed while also providing abstractions
so that complex load management strategies may be implemented.

The load management component determines the criteria for unloading bundles
and contains the following load shedding strategy: `OverloadShedder` and
`ThresholdShedder` and `UniformLoadShedder`. (default is
`ThresholdShedder`since 2.10.0)
- `OverloadShedder`: This strategy attempts to shed exactly one bundle on
brokers which are overloaded
- `ThresholdShedder`: This strategy unloads any broker that exceeds the
average resource utilization of all brokers by a configured threshold.
- `UniformLoadShedder`:This strategy tends to distribute load uniformly
across all brokers.

However, the bundle placement strategy contains only one:
`LeastLongTermMessageRate`, which selects a broker based on which one has
the least long term message rate.

The load management in our pulsar cluster use `ThresholdShedder` as load
shedding strategy, and use `LeastLongTermMessageRate` as bundle placement
strategy, which does not work well.
Some broker nodes have a high load when the traffic of some topics are
relatively large. The load shedding strategy will unload some bundles in
any broker that exceeds the average resource utilization of all brokers by
a configured threshold. And the bundles will be transferred to the next
broker node. However it causes the load of the next broker node exceed the
average resource utilization. Therefore, the load balancing will occur
again on the current broker node due to high load. Worse yet, this scenario
keeps popping up.

The load shedding strategy configuration is as follows
```
# load shedding strategy, support OverloadShedder and ThresholdShedder,
default is OverloadShedder
loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder

# The broker resource usage threshold.
# When the broker resource usage is greater than the pulsar cluster average
resource usage,
# the threshold shedder will be triggered to offload bundles from the
broker.
# It only takes effect in the ThresholdShedder strategy.
loadBalancerBrokerThresholdShedderPercentage=10

# When calculating new resource usage, the history usage accounts for.
# It only takes effect in the ThresholdShedder strategy.
loadBalancerHistoryResourcePercentage=0.9

# The BandWithIn usage weight when calculating new resource usage.
# It only takes effect in the ThresholdShedder strategy.
loadBalancerBandwithInResourceWeight=1.0

# The BandWithOut usage weight when calculating new resource usage.
# It only takes effect in the ThresholdShedder strategy.
loadBalancerBandwithOutResourceWeight=1.0

# The CPU usage weight when calculating new resource usage.
# It only takes effect in the ThresholdShedder strategy.
loadBalancerCPUResourceWeight=1.0

# The heap memory usage weight when calculating new resource usage.
# It only takes effect in the ThresholdShedder strategy.
loadBalancerMemoryResourceWeight=0.1

# The direct memory usage weight when calculating new resource usage.
# It only takes effect in the ThresholdShedder strategy.
loadBalancerDirectMemoryResourceWeight=0.1

# Bundle unload minimum throughput threshold (MB), avoiding bundle unload
frequently.
# It only takes effect in the ThresholdShedder strategy.
loadBalancerBundleUnloadMinThroughputThreshold=0.1
```
The following screenshots are the status of the cluster:
Problem 1. Load balancing took a long time 10 hours and over 400 times, and
it has been unloading if there is a large traffic.
https://user-images.githubusercontent.com/4970972/176341641-b85f8258-e973-4b14-8875-16be573dcbda.png
">
https://user-images.githubusercontent.com/4970972/176341676-ed81b465-10fc-4051-8353-42e6306d4210.png
">

Problem 2. The effect of cpu balancing is poor.
https://user-images.githubusercontent.com/4970972/176341746-d3b28234-11ef-48c4-9f91-2fdf7bcde74b.png
">
https://user-images.githubusercontent.com/4970972/176341792-b77a0691-b402-4fa0-a7aa-ac15c890613a.png
">

The load shedding strategy `ThresholdShedder` work well, but not the bundle
placement strategyLeastLongTermMessageRate .
There are 3 possible reasons for the problems.
1. Although the cluster has many brokers with low load, there are fewer
brokers to be considered for assignment.
https://user-images.githubusercontent.com/4970972/176341873-6da69749-3c1d-49cf-9e83-b942a8327db0.png
">

Some brokers with lower load but more bundles can not be candidate due to
distributing bundles evenly in LoadManager by force. Most of brokers are
filtered out by the strategy, only 1 or 2 broke

Re: [DISCUSS] Reject partitioned topic creation when the topic name contains the `partition` keyword.

2022-06-29 Thread Dave Maughan
Hi Mattison,

1. Should we reject the partitioned topic metadata creation when the topic
> name contains the `-partition-` keywords?


I'm not familiar with all the details but my naive opinion would be that we
should avoid placing constraints on users if we can. In this case that
would mean removing any technical issues that arise due to `-partition-`
being in the topic name, rather than rejecting topics with that string in
the name.

Regards,
Dave



On Thu, Jun 16, 2022 at 4:20 AM mattison chao 
wrote:

> Hello, everyone.
>
> I want to start a discussion about *Reject partitioned topic creation when
> the topic name contains the `-partition-` keyword.*
> Please feel free to express your opinion.
>
> *Background*
>
> We first found this problem two months ago. When the user sets the
> *allowAutoTopicCreation* policy as below:
>
> > {
> > allowAutoTopicCreation: true,
> > allowAutoTopicCreationType: "partitioned",
> > defaultNumPartitions: 1
> >  }
>
>
> This policy will make brokers automatically create the partitioned topic
> when the topic does not exist. But we don't check if users use topic
> names like "persistent://public/default/-partition-YYY".
> In our logic, this topic will pass the automatically created
> partition topic metadata check. We will make the partitioned metadata like
> this:
>
> Z-NODE:
>
>  paritioned-topics/public/defualt/persistent/-partition-YYY
>
> {"partitions": 1}
>
>
> The things here stand at the broker's side. Everything looks like we
> expect. Even the topic name includes "-partition-" keywords.
>
> And then, we continue the process. The Pulsar Client(Java) will get this
> metadata and then try to create separate topics with `-partition-{index}`.
> But there have small trick things,
> The client try to invoke
> `TopicName.get(topicName).getPartition(partitionIndex).toString();` method
> to get paritioned topic name. however, we have `partition` keywords check
> in the method `TopicName.get(topicName).getPartition(partitionIndex)`[1].
> If the topic name includes `-partition-` keywords, it will return the
> original topic name directly. So, The client will use this topic name to
> create the consumer.
>
> Here we go. Let's summarize:
>
> 1. We make the topic `persistent://public/default/-partition-YYY` a
> partitioned topic.
> 2. The client tries to create the topic using
> `persistent://public/default/-partition-YYY` not
> `persistent://public/default/-partition-YYY-partition-0`
>
> In this condition, we have s problem for users. If they want to delete this
> topic, awkward things happen.
>
> - If the users want to use delete-partition-topic admin API, The broker
> will return the error "Partitioned Topic Name should not contain
> '-partition-'".
> - If the users want to use delete-topic admin API, The broker will delete
> this topic, but the broker doesn't remove the partition topic metadata.
>
> So, maybe there have many methods to solve this problem. It looks like
> removing the `partition` keywords check for the `delete-partition` admin
> API. But it seems not to make sense. So, we chose to reject this topic
> creation directly and push a PR for it[2].
>
> Things are not over yet. In the past few days. After discussing with
> Penghui Li and Baodi Shi. We found another problem that will affect the
> dead letter queue(DLQ) If we reject topic creation directly. So, I push a
> PR[3] to revert it soon. And then draft this discussion.
>
> Please remember the phenomenon described above and take it into DLQ with
> the partitioned topic.
>
> When we want to create a Share model subscription, for example, that topic
> name is "persistent://public/default/test-topic". And we enabled DLQ.
> The client will try to create a partitioned topic that looks like
> `persistent://public/default/test-topic-partition-0`. And then, when we
> need to use DLQ, the client will use a new producer that topic name follows
> the DLQ default naming format:
>
> topicname-subscriptionname-DLQ =>
> > persistent://public/default/test-topic-partition-0-test-sub-DLQ
>
>
> With the allowAutoTopicCreation policy, we mentioned before. the topic
> `persistent://public/default/test-topic-partition-0-test-sub-DLQ` will
> create the partitioned topic metadata.
>
> Z-NODE:
>
>
>  
> paritioned-topics/public/defualt/persistent/test-topic-partition-0-test-sub-DLQ
>
> {"partitions": 1}
>
>
> And the client will also do the same things we mentioned above.
>
> So, there have some problems:
>
> - The DLQ topic is partitioned, but the client doesn't use
>
> `persistent://public/default/test-topic-partition-0-test-sub-DLQ-partition-0`
> to create this topic.
> - If the auto-creation policy `defaultNumPartitions` > 1, and the
> subscription type is `Exclusive`. the error will occur. Because the
> consumer already exists. Please check this logic.[4]
> - If the auto-creation policy `defaultNumPartitions` > 1, and the
> subscription type is `Shared`. we will create many consumers/producers to
> the same 

Re: [VOTE] Pulsar Release 2.9.3 Candidate 1

2022-06-29 Thread mattison chao
The docker images:

https://hub.docker.com/layers/244373555/mattison/pulsar-all/2.9.3-rc-1/images/sha256-a7cb36d64817e73ee5dbac38d5435b4584bbbd8639d41449d21adff5138729d9?context=repo

https://hub.docker.com/layers/243611358/mattison/pulsar/2.9.3-rc-1/images/sha256-79132930a35ec093ac428567bcbaf406baac9daa84009699ed00e7968bc84e48?context=repo

On Tue, 28 Jun 2022 at 20:23, mattison chao  wrote:

> I found one problem with this candidate.
>> - When I import `pulsar-broker` and `managed-ledger` into my
>> project, I found the project build failed with the following
>> exceptions.
>
>
> Hi, Hang.
>
> According to the release 2.10.1 vote email, it looks like this problem
> gets solved and won't block the release. Right?
>
> Best,
> Mattison
>
> On Tue, 28 Jun 2022 at 20:21, mattison chao 
> wrote:
>
>> @mattison It's better to upload the image under your org first so that we
>>> can verify the image
>>
>>
>> Sure, I will do it later.
>>
>> Best,
>> Mattison
>>
>> On Tue, 28 Jun 2022 at 15:57, Enrico Olivelli 
>> wrote:
>>
>>> Hang,
>>>
>>> Il giorno lun 27 giu 2022 alle ore 04:35 Hang Chen
>>>  ha scritto:
>>> >
>>> > Hi Mattison,
>>> > Thanks for your great work!
>>> >
>>> > I found one problem with this candidate.
>>> > - When I import `pulsar-broker` and `managed-ledger` into my
>>> > project, I found the project build failed with the following
>>> > exceptions.
>>>
>>> You can fix this problem by importing the Pulsar BOM like we do in
>>> Pulsar Adapters
>>> https://github.com/apache/pulsar-adapters/blob/master/pom.xml#L163
>>>
>>> Enrico
>>>
>>> >
>>> > ```
>>> > [ERROR] Failed to execute goal on project test_pulsar_lib: Could not
>>> > resolve dependencies for project
>>> > org.example:test_pulsar_lib:jar:1.0-SNAPSHOT: Failed to collect
>>> > dependencies for org.example:test_pulsar_lib:jar:1.0-SNAPSHOT: Could
>>> > not resolve version conflict among
>>> > [org.apache.pulsar:pulsar-broker:jar:2.9.3 ->
>>> > org.apache.pulsar:pulsar-zookeeper-utils:jar:2.9.3 ->
>>> > org.apache.bookkeeper:stream-storage-server:jar:4.14.5 ->
>>> > org.apache.bookkeeper:stream-storage-java-client:jar:4.14.5 ->
>>> > io.grpc:grpc-core:jar:1.42.1,
>>> > org.apache.pulsar:pulsar-broker:jar:2.9.3 ->
>>> > org.apache.pulsar:pulsar-zookeeper-utils:jar:2.9.3 ->
>>> > org.apache.bookkeeper:stream-storage-server:jar:4.14.5 ->
>>> > org.apache.bookkeeper:stream-storage-java-client:jar:4.14.5 ->
>>> > io.grpc:grpc-testing:jar:1.42.1 ->
>>> > io.grpc:grpc-core:jar:[1.42.1,1.42.1],
>>> > org.apache.pulsar:pulsar-broker:jar:2.9.3 ->
>>> > org.apache.pulsar:pulsar-zookeeper-utils:jar:2.9.3 ->
>>> > io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-core:jar:[1.45.1,1.45.1],
>>> > org.apache.pulsar:pulsar-broker:jar:2.9.3 ->
>>> > org.apache.pulsar:pulsar-zookeeper-utils:jar:2.9.3 ->
>>> > io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-grpclb:jar:1.45.1 ->
>>> > io.grpc:grpc-core:jar:[1.45.1,1.45.1],
>>> > org.apache.pulsar:pulsar-broker:jar:2.9.3 ->
>>> > org.apache.pulsar:pulsar-zookeeper-utils:jar:2.9.3 ->
>>> > io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-netty:jar:1.45.1 ->
>>> > io.grpc:grpc-core:jar:[1.45.1,1.45.1],
>>> > org.apache.pulsar:pulsar-broker:jar:2.9.3 ->
>>> > org.apache.pulsar:pulsar-zookeeper-utils:jar:2.9.3 ->
>>> > io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-rls:jar:1.45.1 ->
>>> > io.grpc:grpc-core:jar:[1.45.1,1.45.1],
>>> > org.apache.pulsar:pulsar-broker:jar:2.9.3 ->
>>> > org.apache.pulsar:pulsar-zookeeper-utils:jar:2.9.3 ->
>>> > io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-services:jar:1.45.1 ->
>>> > io.grpc:grpc-core:jar:[1.45.1,1.45.1],
>>> > org.apache.pulsar:pulsar-broker:jar:2.9.3 ->
>>> > org.apache.pulsar:pulsar-zookeeper-utils:jar:2.9.3 ->
>>> > io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-xds:jar:1.45.1 ->
>>> > io.grpc:grpc-core:jar:1.45.1,
>>> > org.apache.pulsar:pulsar-broker:jar:2.9.3 ->
>>> > org.apache.pulsar:pulsar-zookeeper-utils:jar:2.9.3 ->
>>> > io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-xds:jar:1.45.1 ->
>>> > io.grpc:grpc-netty-shaded:jar:[1.45.1,1.45.1] ->
>>> > io.grpc:grpc-core:jar:[1.45.1,1.45.1]] -> [Help 1]
>>> > ```
>>> >
>>> > There is a PR: https://github.com/apache/pulsar/pull/16109 removed
>>> > pulsar-zookeeper-utils, which fixes this issue. But in branch 2.9, we
>>> > still need this module, we can not cherry-pick this pr into
>>> > branch-2.9. We can exclude the `io.grpc:grpc-core` in one dependency
>>> > to fix this issue.
>>> >
>>> > Thanks,
>>> > Hang
>>> >
>>> > PengHui Li  于2022年6月27日周一 09:51写道:
>>> > >
>>> > > +1 (binding)
>>> > >
>>> > > - verify gpg signatures
>>> > > - run standalone
>>> > > - pub/sub with pulsar-perf
>>> > > - verify connector (Cassandra)
>>> > > - verify stateful function
>>> > > - build image and test internally for the integration tests (all the
>>> tests
>>> > > get passed)
>>> > >
>>> > > @mattison It's better to upload the image under your org first so
>>> that we
>>> > > can verify the image
>>> > >
>>> > > Thanks,
>>> > > Penghui
>>> > >
>>> 

[VOTE] [PIP-179] Support the admin API to check unknown request parameters

2022-06-29 Thread Yubiao Feng
Hi Pulsar Community

I would like to start a VOTE on "Support the admin API to check unknown
request parameters" (PIP-179).

The proposal can be read at https://github.com/apache/pulsar/issues/16135

and the discussion thread is available at
https://lists.apache.org/thread/m8vkxl46njm7sh0r1mqsn25jggq9v8kb

Voting will stay open for at least 48h.

Thanks
Yubiao Feng


[GitHub] [pulsar-site] urfreespace merged pull request #128: fix: update build script

2022-06-29 Thread GitBox


urfreespace merged PR #128:
URL: https://github.com/apache/pulsar-site/pull/128


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[GitHub] [pulsar-site] urfreespace merged pull request #129: feat: add support for force build by parse commit msg

2022-06-29 Thread GitBox


urfreespace merged PR #129:
URL: https://github.com/apache/pulsar-site/pull/129


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [VOTE] [PIP-179] Support the admin API to check unknown request parameters

2022-06-29 Thread PengHui Li
+1 (binding)

Penghui

On Thu, Jun 30, 2022 at 12:40 AM Yubiao Feng
 wrote:

> Hi Pulsar Community
>
> I would like to start a VOTE on "Support the admin API to check unknown
> request parameters" (PIP-179).
>
> The proposal can be read at https://github.com/apache/pulsar/issues/16135
>
> and the discussion thread is available at
> https://lists.apache.org/thread/m8vkxl46njm7sh0r1mqsn25jggq9v8kb
>
> Voting will stay open for at least 48h.
>
> Thanks
> Yubiao Feng
>


Re: [VOTE] [PIP-179] Support the admin API to check unknown request parameters

2022-06-29 Thread Qiang Huang
+1

Yubiao Feng  于2022年6月30日周四 00:40写道:

> Hi Pulsar Community
>
> I would like to start a VOTE on "Support the admin API to check unknown
> request parameters" (PIP-179).
>
> The proposal can be read at https://github.com/apache/pulsar/issues/16135
>
> and the discussion thread is available at
> https://lists.apache.org/thread/m8vkxl46njm7sh0r1mqsn25jggq9v8kb
>
> Voting will stay open for at least 48h.
>
> Thanks
> Yubiao Feng
>


-- 
BR,
Qiang Huang


Re: [DISCUSS] [PIP-179] Support the admin API to check unknown request parameters

2022-06-29 Thread Michael Marshall
I think this optional configuration is a good idea. I agree that a 204
is misleading for a malformed request.

Additionally, I think we should go further and add this configuration
option to the function worker and possibly the proxy (which handles a
single post call) as well, since they also handle http requests with
parameters.

Thanks,
Michael

On Mon, Jun 27, 2022 at 12:16 AM Zike Yang  wrote:
>
> +1
>
> Zike Yang
>
> On Wed, Jun 22, 2022 at 11:26 AM PengHui Li  wrote:
>
> > +1
> >
> > Penghui
> >
> > On Wed, Jun 22, 2022 at 10:53 AM Yubiao Feng
> >  wrote:
> >
> > > Hi, Pulsar community:
> > >
> > > I open a pip to discuss "Support the admin API to check unknown request
> > > parameters"
> > >
> > > Proposal Link: https://github.com/apache/pulsar/issues/16135
> > >
> > > ### Motivation
> > >
> > > The design of the Admin API is now such that if an incorrect parameter
> > name
> > > is submitted, this property (if not required) will be ignored, then
> > > execution continues, and the response is “204 Success”. This will trick
> > the
> > > user into thinking the setup succeeded when it didn't correctly as
> > expected
> > > in some cases, as shown below:
> > >
> > > User POST request to /{tenant}/{namespace}/{topic}/retention" with
> > > incorrect parameter:
> > > ```json
> > > {"retention_size_in_mb":-1,"retention_time_in_minutes":40320}
> > > ```
> > >
> > > Which should have been this:
> > >
> > > ```json
> > > {"retentionSizeInMB":-1,"retentionTimeInMinutes":40320}
> > > ```
> > >
> > > Response:
> > >
> > > ```http
> > > HTTP/1.1 204 No Content
> > > Date: Mon, 20 Jun 2022 02:54:25 GMT
> > > broker-address: 127.0.0.1
> > > Server: Jetty(9.4.44.v20210927)
> > > ```
> > >
> > > We can provide an optional mechanism: "fail (HTTP status 400 bad
> > requests)
> > > on unknown request parameters".
> > >
> > > ## Goal
> > >
> > > - scope:
> > >   - ~~Path variables~~(no need for change):  This represents the domain.
> > > The current API has been validated, so no additional modifications are
> > > required.
> > >   - ~~Query params~~(no support on this proposal):  I haven't found an
> > > elegant way to do it yet, so this proposal does not include Query Param
> > > validation.
> > >   - *Entity properties*:  This proposal only handles requests whose
> > > Content-type is "application/json" (in fact, this is the only type in our
> > > project).
> > > - Configurable(Support dynamic switching).
> > >
> > >
> > > ## Approach
> > >
> > > When parsing the request body, any unknown property is considered a bad
> > > request. The [Jackson unknown property rule](
> > >
> > >
> > https://github.com/FasterXML/jackson-databind/blob/de3d0ecbc1fd0a1a6b061e62a198b3ba0d0d163e/src/main/java/com/fasterxml/jackson/databind/DeserializationFeature.java#L121
> > > )
> > > is adopted:
> > >
> > > - Case sensitive.
> > > - Special characters are not ignored.
> > > - Do not trim Spaces.
> > >
> > > If the check fails,  return a text/plain response with 400 code. Like
> > this:
> > >
> > > ```http
> > > HTTP/1.1 400 Bad Request
> > > Date: Mon, 20 Jun 2022 03:52:10 GMT
> > > broker-address: 127.0.0.1
> > > Content-Type: text/plain
> > > Content-Length: 432
> > > Server: Jetty(9.4.44.v20210927)
> > >
> > > Unrecognized field "retention_size_in_mb" (class
> > > org.apache.pulsar.common.policies.data.RetentionPolicies known
> > properties:
> > > "retentionSizeInMB", "retentionTimeInMinutes"])
> > > ```
> > >
> > > ## Configuration Changes
> > >
> > > broker.conf
> > >
> > > ```properties
> > > # Admin API fail on unknown request parameter in request-body. see
> > PIP-178.
> > > Setting this to blank means that this feature is turned off.
> > > httpRequestsFailOnUnknownPropertiesEnabled=false
> > > ```
> > >
> > > ## Dynamic switching
> > > Enabling this feature affects all of the broker's HTTP services,
> > including
> > > the following:
> > >
> > > - /status.html (no post-entity request)
> > > - /admin [v2,v3]
> > > - /lookup (no post-entity request)
> > > - /topics (http client)
> > > - /metrics (no post-entity request)
> > >
> > > Because of the number of apis involved, we provide dynamic configuration.
> > > When a user discovers any problem, it can be turned on and off
> > dynamically
> > > using the Admin API(without restarting Broker), which can reduce impact.
> > >
> > > Note: Since admin api v1 is no longer maintained, this feature does not
> > > affect this part of the functionality.
> > >
> > > ```shell
> > > pulsar-admin brokers update-dynamic-config --config
> > > httpRequestsFailOnUnknownPropertiesEnabled --value [boolean]
> > > ```
> > >
> > > Thanks
> > > Yubiao Feng
> > >
> >


Re: [VOTE] [PIP-179] Support the admin API to check unknown request parameters

2022-06-29 Thread Michael Marshall
+1

- Michael

On Wed, Jun 29, 2022 at 9:29 PM Qiang Huang  wrote:
>
> +1
>
> Yubiao Feng  于2022年6月30日周四 00:40写道:
>
> > Hi Pulsar Community
> >
> > I would like to start a VOTE on "Support the admin API to check unknown
> > request parameters" (PIP-179).
> >
> > The proposal can be read at https://github.com/apache/pulsar/issues/16135
> >
> > and the discussion thread is available at
> > https://lists.apache.org/thread/m8vkxl46njm7sh0r1mqsn25jggq9v8kb
> >
> > Voting will stay open for at least 48h.
> >
> > Thanks
> > Yubiao Feng
> >
>
>
> --
> BR,
> Qiang Huang


Re: [VOTE] [PIP-179] Support the admin API to check unknown request parameters

2022-06-29 Thread mattison chao
+1(non-binding)

Best,
Mattison

On Thu, 30 Jun 2022 at 12:35, Michael Marshall  wrote:

> +1
>
> - Michael
>
> On Wed, Jun 29, 2022 at 9:29 PM Qiang Huang 
> wrote:
> >
> > +1
> >
> > Yubiao Feng  于2022年6月30日周四 00:40写道:
> >
> > > Hi Pulsar Community
> > >
> > > I would like to start a VOTE on "Support the admin API to check unknown
> > > request parameters" (PIP-179).
> > >
> > > The proposal can be read at
> https://github.com/apache/pulsar/issues/16135
> > >
> > > and the discussion thread is available at
> > > https://lists.apache.org/thread/m8vkxl46njm7sh0r1mqsn25jggq9v8kb
> > >
> > > Voting will stay open for at least 48h.
> > >
> > > Thanks
> > > Yubiao Feng
> > >
> >
> >
> > --
> > BR,
> > Qiang Huang
>


[DISCUSS] Move all generated documentation to apache/pulsar-site repo

2022-06-29 Thread Michael Marshall
Hi Pulsar Community,

I would like to discuss changing where our generated docs are stored.

Currently, we keep some generated docs, like the python client docs
and the swagger docs, in our apache/pulsar repo. There are other docs
that are generated and stored in our apache/pulsar-site repo.

I propose that we move all generated docs to be stored in the
apache/pulsar-site repo. This change will primarily affect release
managers since they will need to push commits to the pulsar-site repo.
No one else should need to modify generated docs.

My primary motivation is that I'd like to add new generated Javadoc
pages for all of our recent releases so that we no longer only serve
"-SNAPSHOT" versions of the generated docs. Those new docs will
increase the size of our apache/pulsar repo. I prefer to keep the
apache/pulsar repo small.

I plan to open a PR to move these docs from the apache/pulsar repo to
the apache/pulsar-site repo later this week or early next week, unless
anyone has strong objections. I'll also update our release manager
documentation for the new process.

Thanks,
Michael


Re: [VOTE] [PIP-179] Support the admin API to check unknown request parameters

2022-06-29 Thread Zike Yang
+1

Zike Yang

On Thu, Jun 30, 2022 at 12:38 PM mattison chao 
wrote:

> +1(non-binding)
>
> Best,
> Mattison
>
> On Thu, 30 Jun 2022 at 12:35, Michael Marshall 
> wrote:
>
> > +1
> >
> > - Michael
> >
> > On Wed, Jun 29, 2022 at 9:29 PM Qiang Huang 
> > wrote:
> > >
> > > +1
> > >
> > > Yubiao Feng  于2022年6月30日周四
> 00:40写道:
> > >
> > > > Hi Pulsar Community
> > > >
> > > > I would like to start a VOTE on "Support the admin API to check
> unknown
> > > > request parameters" (PIP-179).
> > > >
> > > > The proposal can be read at
> > https://github.com/apache/pulsar/issues/16135
> > > >
> > > > and the discussion thread is available at
> > > > https://lists.apache.org/thread/m8vkxl46njm7sh0r1mqsn25jggq9v8kb
> > > >
> > > > Voting will stay open for at least 48h.
> > > >
> > > > Thanks
> > > > Yubiao Feng
> > > >
> > >
> > >
> > > --
> > > BR,
> > > Qiang Huang
> >
>