[GitHub] [pulsar-site] SignorMercurio opened a new pull request, #169: Move config docs to /reference and improve docs gen scripts
SignorMercurio opened a new pull request, #169: URL: https://github.com/apache/pulsar-site/pull/169 @urfreespace PTAL carefully. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] Technoboy- commented on a diff in pull request #167: [Don't Merge] Pulsar 2.11.0 release notes
Technoboy- commented on code in PR #167: URL: https://github.com/apache/pulsar-site/pull/167#discussion_r950852597 ## site2/website/release-notes.md: ## @@ -1,6 +1,486 @@ ## Apache Pulsar Release Notes +### 2.11.0 + 2022-08-16 + +### Important notice +- Rename Pulsar txn metrics to specify OpenMetrics [#16581](https://github.com/apache/pulsar/pull/16581) +- Rename Pulsar schema metrics to specify OpenMetrics [#16610](https://github.com/apache/pulsar/pull/16610) +- Rename Pulsar lb metrics to specify OpenMetrics [#16611](https://github.com/apache/pulsar/pull/16611) +- Remove jul-to-slf4j #16320 [#16320](https://github.com/apache/pulsar/pull/16320) +- Fix get non-persistent topics issue in Namespaces [#16170](https://github.com/apache/pulsar/pull/16170) +- Fix producer/consume permission can’t get schema [#15956](https://github.com/apache/pulsar/pull/15956) +- Add verification for configured default backlog quota and retention [#15441](https://github.com/apache/pulsar/pull/15441) +- Fix broker LoadBalance uneffective [#15314](https://github.com/apache/pulsar/pull/15314) +- Ensure the deletion consistency of topic and schema [#14608](https://github.com/apache/pulsar/pull/14608) +- Transaction cumulative ack redeliver change [#14371](https://github.com/apache/pulsar/pull/14371) +- Bump prometheus client version from 0.5.0 to 0.15.0 [#13785](https://github.com/apache/pulsar/pull/13785) +- [PIP-158] Split client TLS transport encryption from authentication [#15634](https://github.com/apache/pulsar/pull/15634) +- [PIP-155] Removed Python 2 support [#15376](https://github.com/apache/pulsar/pull/15376) +- [PIP-156] Build Pulsar Server on Java 17 [#15264](https://github.com/apache/pulsar/pull/15264) +- [PIP-162] Enable system topic by default [#15619](https://github.com/apache/pulsar/pull/15619) +- [PIP-174] New managed ledger entry cache implementation [#15955](https://github.com/apache/pulsar/pull/15955) + +### PIPs +- [PIP 37] Support chunking with Shared subscription [#16202](https://github.com/apache/pulsar/pull/16202) +- [PIP-74] Pulsar client memory limits [#14400](https://github.com/apache/pulsar/pull/14400) +- [PIP-81] Split createNewMetadataLedger into multiple methods for reuse [#15425](https://github.com/apache/pulsar/pull/15425) +- [PIP 97] Update Authentication Interfaces to Include Async Authentication Methods [#12104](https://github.com/apache/pulsar/pull/12104) +- [PIP-100] Support pluggable topic factory [#12235](https://github.com/apache/pulsar/pull/12235) +- [PIP-105] Store Subscription properties [#15757](https://github.com/apache/pulsar/pull/15757) +- [PIP-117] Change Pulsar standalone defaults [#15478](https://github.com/apache/pulsar/pull/15478) +- [PIP-132] Include message header size when check maxMessageSize for non-batch message on the client side. [#14007](https://github.com/apache/pulsar/pull/14007) +- [PIP-136] Sync Pulsar metadata across multiple clouds [#16425](https://github.com/apache/pulsar/pull/16425) +- [PIP-143] Support split bundle by specified boundaries [#13796](https://github.com/apache/pulsar/pull/13796) +- [PIP-145] Improve performance of regex subscriptions [#14505](https://github.com/apache/pulsar/pull/14505) +- [PIP-146] ManagedCursorInfo compression [#14542](https://github.com/apache/pulsar/pull/14542) +- [PIP 149] Making the REST Admin API fully async [#14365](https://github.com/apache/pulsar/issues/14365) +- [PIP-152] Support subscription level dispatch rate limiter setting [#15295](https://github.com/apache/pulsar/issues/15295) +- [PIP-153] Optimize metadataPositions in MLPendingAckStore [#15137](https://github.com/apache/pulsar/pull/15137) +- [PIP-160] Make transactions work more efficiently by aggregation operation for transaction log and pending ack store [#15370](https://github.com/apache/pulsar/issues/15370) +- [PIP-161] Exclusive Producer: ability to fence out an existing Producer [#15488](https://github.com/apache/pulsar/pull/15488) +- [PIP-163] Add lowWaterMark check before appending entry to TB [#15424](https://github.com/apache/pulsar/pull/15424) +- [PIP-166] Function add MANUAL delivery semantics [#16279](https://github.com/apache/pulsar/pull/16279) +- [PIP-179] Support the admin API to check unknown request parameters [#16135](https://github.com/apache/pulsar/issues/16135) +- [PIP-181] Pulsar Shell [#16250](https://github.com/apache/pulsar/issues/16250) +- [PIP-184] Topic specific consumer priorityLevel [#16715](https://github.com/apache/pulsar/issues/16715) +- [PIP-187] Add API to analyze a subscription backlog and provide a accurate value [#16545](https://github.com/apache/pulsar/issues/16545) + +### Broker +- [PIP 81] Split the individual acknowledgments into multiple entries + - Add range set wrapper to support record dirty ledgers [#15607](https://github.com/apache/pulsar/pull/15607) + - Notifications for faster topic discovery [#16062](https://github.com/apache/pulsar/pull/1
[GitHub] [pulsar] tisonkun edited a discussion: Strange failure when build Pulsar Docker image on M1
GitHub user tisonkun edited a discussion: Strange failure when build Pulsar Docker image on M1 These issues should be all answered by https://github.com/apache/pulsar/issues/12944#issuecomment-1103444708. When debugging https://github.com/apache/pulsar/issues/12944, I encounter several strange failures. Here I'll share them and see if others encounter them also. NOTE: These failures don't exist in CI. Thus I think they're strange. 1. `cannot create /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security: Directory nonexistent` ``` [INFO] DOCKER> Step 14/25 : ENV JAVA_HOME /usr/lib/jvm/java-17-openjdk-amd64 [INFO] DOCKER> [INFO] DOCKER> ---> Running in a47a209de148 [INFO] DOCKER> Removing intermediate container a47a209de148 [INFO] DOCKER> ---> 6a9d5c8543b8 [INFO] DOCKER> Step 15/25 : RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security [INFO] DOCKER> [INFO] DOCKER> ---> Running in d55c15e90ed1 [INFO] DOCKER> [91m/bin/sh: 1: cannot create /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security: Directory nonexistent [INFO] DOCKER> Removing intermediate container d55c15e90ed1 [ERROR] DOCKER> Unable to build image [apachepulsar/java-test-image] : "The command '/bin/sh -c echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security' returned a non-zero code: 2" ["The command '/bin/sh -c echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security' returned a non-zero code: 2" ] ``` After add `RUN mkdir -p /usr/lib/jvm/java-17-openjdk-amd64/conf/security` before `RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security` in `docker/pulsar/Dockerfile` and `test/docker-images/java-test-image/Dockerfile`, this issue vanished. 2. `FileNotFoundError: [Errno 2] No such file or directory: 'cc'` ``` File "/tmp/pip-install-od2wltzl/grpcio/setup.py", line 149, in check_linker_need_libatomic cc_test = subprocess.Popen(['cc', '-x', 'c++', '-std=c++11', '-'], File "/usr/lib/python3.8/subprocess.py", line 858, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'cc' ``` See details in https://github.com/apache/pulsar/issues/12944#issuecomment-1219300666. Does `ubuntu:20.04` contain `cc`? If so, why does it fail on my local env. If not, why does it succeed in CI? 3. After I add `gcc` in the `apt install` list, it can find `cc`, but later it fails with `Python.h not found`. 4. After I add `python-dev` in the `apt install` list, it still fails with `Python.h not found`. cc @lhotari @nodece @michaeljmarshall GitHub link: https://github.com/apache/pulsar/discussions/17166 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun edited a discussion: Strange failure when build Pulsar Docker image on M1
GitHub user tisonkun edited a discussion: Strange failure when build Pulsar Docker image on M1 UPDATED: These issues should be all answered by https://github.com/apache/pulsar/issues/12944#issuecomment-1103444708. When debugging https://github.com/apache/pulsar/issues/12944, I encounter several strange failures. Here I'll share them and see if others encounter them also. NOTE: These failures don't exist in CI. Thus I think they're strange. 1. `cannot create /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security: Directory nonexistent` ``` [INFO] DOCKER> Step 14/25 : ENV JAVA_HOME /usr/lib/jvm/java-17-openjdk-amd64 [INFO] DOCKER> [INFO] DOCKER> ---> Running in a47a209de148 [INFO] DOCKER> Removing intermediate container a47a209de148 [INFO] DOCKER> ---> 6a9d5c8543b8 [INFO] DOCKER> Step 15/25 : RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security [INFO] DOCKER> [INFO] DOCKER> ---> Running in d55c15e90ed1 [INFO] DOCKER> [91m/bin/sh: 1: cannot create /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security: Directory nonexistent [INFO] DOCKER> Removing intermediate container d55c15e90ed1 [ERROR] DOCKER> Unable to build image [apachepulsar/java-test-image] : "The command '/bin/sh -c echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security' returned a non-zero code: 2" ["The command '/bin/sh -c echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security' returned a non-zero code: 2" ] ``` After add `RUN mkdir -p /usr/lib/jvm/java-17-openjdk-amd64/conf/security` before `RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security` in `docker/pulsar/Dockerfile` and `test/docker-images/java-test-image/Dockerfile`, this issue vanished. 2. `FileNotFoundError: [Errno 2] No such file or directory: 'cc'` ``` File "/tmp/pip-install-od2wltzl/grpcio/setup.py", line 149, in check_linker_need_libatomic cc_test = subprocess.Popen(['cc', '-x', 'c++', '-std=c++11', '-'], File "/usr/lib/python3.8/subprocess.py", line 858, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'cc' ``` See details in https://github.com/apache/pulsar/issues/12944#issuecomment-1219300666. Does `ubuntu:20.04` contain `cc`? If so, why does it fail on my local env. If not, why does it succeed in CI? 3. After I add `gcc` in the `apt install` list, it can find `cc`, but later it fails with `Python.h not found`. 4. After I add `python-dev` in the `apt install` list, it still fails with `Python.h not found`. cc @lhotari @nodece @michaeljmarshall GitHub link: https://github.com/apache/pulsar/discussions/17166 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun edited a discussion: Strange failure when build Pulsar Docker image on M1
GitHub user tisonkun edited a discussion: Strange failure when build Pulsar Docker image on M1 **UPDATED: These issues should be all answered by https://github.com/apache/pulsar/issues/12944#issuecomment-1103444708.** --- When debugging https://github.com/apache/pulsar/issues/12944, I encounter several strange failures. Here I'll share them and see if others encounter them also. NOTE: These failures don't exist in CI. Thus I think they're strange. 1. `cannot create /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security: Directory nonexistent` ``` [INFO] DOCKER> Step 14/25 : ENV JAVA_HOME /usr/lib/jvm/java-17-openjdk-amd64 [INFO] DOCKER> [INFO] DOCKER> ---> Running in a47a209de148 [INFO] DOCKER> Removing intermediate container a47a209de148 [INFO] DOCKER> ---> 6a9d5c8543b8 [INFO] DOCKER> Step 15/25 : RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security [INFO] DOCKER> [INFO] DOCKER> ---> Running in d55c15e90ed1 [INFO] DOCKER> [91m/bin/sh: 1: cannot create /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security: Directory nonexistent [INFO] DOCKER> Removing intermediate container d55c15e90ed1 [ERROR] DOCKER> Unable to build image [apachepulsar/java-test-image] : "The command '/bin/sh -c echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security' returned a non-zero code: 2" ["The command '/bin/sh -c echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security' returned a non-zero code: 2" ] ``` After add `RUN mkdir -p /usr/lib/jvm/java-17-openjdk-amd64/conf/security` before `RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security` in `docker/pulsar/Dockerfile` and `test/docker-images/java-test-image/Dockerfile`, this issue vanished. 2. `FileNotFoundError: [Errno 2] No such file or directory: 'cc'` ``` File "/tmp/pip-install-od2wltzl/grpcio/setup.py", line 149, in check_linker_need_libatomic cc_test = subprocess.Popen(['cc', '-x', 'c++', '-std=c++11', '-'], File "/usr/lib/python3.8/subprocess.py", line 858, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'cc' ``` See details in https://github.com/apache/pulsar/issues/12944#issuecomment-1219300666. Does `ubuntu:20.04` contain `cc`? If so, why does it fail on my local env. If not, why does it succeed in CI? 3. After I add `gcc` in the `apt install` list, it can find `cc`, but later it fails with `Python.h not found`. 4. After I add `python-dev` in the `apt install` list, it still fails with `Python.h not found`. cc @lhotari @nodece @michaeljmarshall GitHub link: https://github.com/apache/pulsar/discussions/17166 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
Performance issue in Pulsar with large fan-outs since Pulsar 2.8.2
Hi all, I'd like to get some more eyes on this long outstanding performance issue with large fan-outs (a large number of consumers for a single topic). The broker cache does not work as expected due to invalid changes introduced in version 2.8.2 by PR https://github.com/apache/pulsar/pull/12045. I have reported the issue with broker cache over 2 months ago with https://github.com/apache/pulsar/issues/16054 . Please take a look. The workaround seems to be to use the feature introduced in PR https://github.com/apache/pulsar/pull/14985, "Add a cache eviction policy：Evicting cache data by the slowest markDeletedPosition", by setting cacheEvictionByMarkDeletedPosition=true . One notable detail is that the cacheEvictionByMarkDeletedPosition entry is not included in conf/broker.conf or conf/standalone.conf files. Therefore in the Pulsar Helm chart, it is necessary to use "PULSAR_PREFIX_cacheEvictionByMarkDeletedPosition: true" to activate the setting. However, the problem with the cacheEvictionByMarkDeletedPosition solution is that the cache could be filled with irrelevant entries when there's a lot of topics and consumers on a single broker since the cache expiration is based on markDeletedPosition and not the farthest behind read position for active consumers. My assumption is also that there wouldn't have been a need to add https://github.com/apache/pulsar/pull/14985 at all if the broker cache would have worked as expected. PIP-174: Provide new implementation for broker dispatch cache ( https://github.com/apache/pulsar/issues/15954 / https://github.com/apache/pulsar/pull/15955) will provide a new broker cache implementation where this issue hopefully goes away, but we will regardless need to make the fix for current maintenance versions. I'm looking forward to feedback of original contributors of the broker cache and the related changes so that we finally get this issue resolved. Thanks, Lari
[GitHub] [pulsar-site] Anonymitaet commented on a diff in pull request #167: [Don't Merge] Pulsar 2.11.0 release notes
Anonymitaet commented on code in PR #167: URL: https://github.com/apache/pulsar-site/pull/167#discussion_r950948327 ## site2/website/release-notes.md: ## @@ -1,6 +1,486 @@ ## Apache Pulsar Release Notes +### 2.11.0 + 2022-08-16 + +### Important notice +- Rename Pulsar txn metrics to specify OpenMetrics [#16581](https://github.com/apache/pulsar/pull/16581) +- Rename Pulsar schema metrics to specify OpenMetrics [#16610](https://github.com/apache/pulsar/pull/16610) +- Rename Pulsar lb metrics to specify OpenMetrics [#16611](https://github.com/apache/pulsar/pull/16611) +- Remove jul-to-slf4j #16320 [#16320](https://github.com/apache/pulsar/pull/16320) +- Fix get non-persistent topics issue in Namespaces [#16170](https://github.com/apache/pulsar/pull/16170) +- Fix producer/consume permission can’t get schema [#15956](https://github.com/apache/pulsar/pull/15956) +- Add verification for configured default backlog quota and retention [#15441](https://github.com/apache/pulsar/pull/15441) +- Fix broker LoadBalance uneffective [#15314](https://github.com/apache/pulsar/pull/15314) +- Ensure the deletion consistency of topic and schema [#14608](https://github.com/apache/pulsar/pull/14608) +- Transaction cumulative ack redeliver change [#14371](https://github.com/apache/pulsar/pull/14371) +- Bump prometheus client version from 0.5.0 to 0.15.0 [#13785](https://github.com/apache/pulsar/pull/13785) +- [PIP-158] Split client TLS transport encryption from authentication [#15634](https://github.com/apache/pulsar/pull/15634) +- [PIP-155] Removed Python 2 support [#15376](https://github.com/apache/pulsar/pull/15376) +- [PIP-156] Build Pulsar Server on Java 17 [#15264](https://github.com/apache/pulsar/pull/15264) +- [PIP-162] Enable system topic by default [#15619](https://github.com/apache/pulsar/pull/15619) +- [PIP-174] New managed ledger entry cache implementation [#15955](https://github.com/apache/pulsar/pull/15955) + +### PIPs +- [PIP 37] Support chunking with Shared subscription [#16202](https://github.com/apache/pulsar/pull/16202) +- [PIP-74] Pulsar client memory limits [#14400](https://github.com/apache/pulsar/pull/14400) +- [PIP-81] Split createNewMetadataLedger into multiple methods for reuse [#15425](https://github.com/apache/pulsar/pull/15425) +- [PIP 97] Update Authentication Interfaces to Include Async Authentication Methods [#12104](https://github.com/apache/pulsar/pull/12104) +- [PIP-100] Support pluggable topic factory [#12235](https://github.com/apache/pulsar/pull/12235) +- [PIP-105] Store Subscription properties [#15757](https://github.com/apache/pulsar/pull/15757) +- [PIP-117] Change Pulsar standalone defaults [#15478](https://github.com/apache/pulsar/pull/15478) +- [PIP-132] Include message header size when check maxMessageSize for non-batch message on the client side. [#14007](https://github.com/apache/pulsar/pull/14007) +- [PIP-136] Sync Pulsar metadata across multiple clouds [#16425](https://github.com/apache/pulsar/pull/16425) +- [PIP-143] Support split bundle by specified boundaries [#13796](https://github.com/apache/pulsar/pull/13796) +- [PIP-145] Improve performance of regex subscriptions [#14505](https://github.com/apache/pulsar/pull/14505) +- [PIP-146] ManagedCursorInfo compression [#14542](https://github.com/apache/pulsar/pull/14542) +- [PIP 149] Making the REST Admin API fully async [#14365](https://github.com/apache/pulsar/issues/14365) +- [PIP-152] Support subscription level dispatch rate limiter setting [#15295](https://github.com/apache/pulsar/issues/15295) +- [PIP-153] Optimize metadataPositions in MLPendingAckStore [#15137](https://github.com/apache/pulsar/pull/15137) +- [PIP-160] Make transactions work more efficiently by aggregation operation for transaction log and pending ack store [#15370](https://github.com/apache/pulsar/issues/15370) +- [PIP-161] Exclusive Producer: ability to fence out an existing Producer [#15488](https://github.com/apache/pulsar/pull/15488) +- [PIP-163] Add lowWaterMark check before appending entry to TB [#15424](https://github.com/apache/pulsar/pull/15424) +- [PIP-166] Function add MANUAL delivery semantics [#16279](https://github.com/apache/pulsar/pull/16279) +- [PIP-179] Support the admin API to check unknown request parameters [#16135](https://github.com/apache/pulsar/issues/16135) +- [PIP-181] Pulsar Shell [#16250](https://github.com/apache/pulsar/issues/16250) +- [PIP-184] Topic specific consumer priorityLevel [#16715](https://github.com/apache/pulsar/issues/16715) +- [PIP-187] Add API to analyze a subscription backlog and provide a accurate value [#16545](https://github.com/apache/pulsar/issues/16545) + +### Broker +- [PIP 81] Split the individual acknowledgments into multiple entries + - Add range set wrapper to support record dirty ledgers [#15607](https://github.com/apache/pulsar/pull/15607) + - Notifications for faster topic discovery [#16062](https://github.com/apache/pulsar/pull/
[GitHub] [pulsar-site] Anonymitaet commented on a diff in pull request #167: [Don't Merge] Pulsar 2.11.0 release notes
Anonymitaet commented on code in PR #167: URL: https://github.com/apache/pulsar-site/pull/167#discussion_r948753697 ## site2/website/release-notes.md: ## @@ -1,6 +1,486 @@ ## Apache Pulsar Release Notes +### 2.11.0 + 2022-08-16 + +### Important notice +- Rename Pulsar txn metrics to specify OpenMetrics [#16581](https://github.com/apache/pulsar/pull/16581) +- Rename Pulsar schema metrics to specify OpenMetrics [#16610](https://github.com/apache/pulsar/pull/16610) +- Rename Pulsar lb metrics to specify OpenMetrics [#16611](https://github.com/apache/pulsar/pull/16611) +- Remove jul-to-slf4j #16320 [#16320](https://github.com/apache/pulsar/pull/16320) +- Fix get non-persistent topics issue in Namespaces [#16170](https://github.com/apache/pulsar/pull/16170) +- Fix producer/consume permission can’t get schema [#15956](https://github.com/apache/pulsar/pull/15956) +- Add verification for configured default backlog quota and retention [#15441](https://github.com/apache/pulsar/pull/15441) +- Fix broker LoadBalance uneffective [#15314](https://github.com/apache/pulsar/pull/15314) +- Ensure the deletion consistency of topic and schema [#14608](https://github.com/apache/pulsar/pull/14608) +- Transaction cumulative ack redeliver change [#14371](https://github.com/apache/pulsar/pull/14371) +- Bump prometheus client version from 0.5.0 to 0.15.0 [#13785](https://github.com/apache/pulsar/pull/13785) +- [PIP-158] Split client TLS transport encryption from authentication [#15634](https://github.com/apache/pulsar/pull/15634) +- [PIP-155] Removed Python 2 support [#15376](https://github.com/apache/pulsar/pull/15376) +- [PIP-156] Build Pulsar Server on Java 17 [#15264](https://github.com/apache/pulsar/pull/15264) +- [PIP-162] Enable system topic by default [#15619](https://github.com/apache/pulsar/pull/15619) +- [PIP-174] New managed ledger entry cache implementation [#15955](https://github.com/apache/pulsar/pull/15955) + +### PIPs +- [PIP 37] Support chunking with Shared subscription [#16202](https://github.com/apache/pulsar/pull/16202) +- [PIP-74] Pulsar client memory limits [#14400](https://github.com/apache/pulsar/pull/14400) +- [PIP-81] Split createNewMetadataLedger into multiple methods for reuse [#15425](https://github.com/apache/pulsar/pull/15425) +- [PIP 97] Update Authentication Interfaces to Include Async Authentication Methods [#12104](https://github.com/apache/pulsar/pull/12104) +- [PIP-100] Support pluggable topic factory [#12235](https://github.com/apache/pulsar/pull/12235) +- [PIP-105] Store Subscription properties [#15757](https://github.com/apache/pulsar/pull/15757) +- [PIP-117] Change Pulsar standalone defaults [#15478](https://github.com/apache/pulsar/pull/15478) +- [PIP-132] Include message header size when check maxMessageSize for non-batch message on the client side. [#14007](https://github.com/apache/pulsar/pull/14007) +- [PIP-136] Sync Pulsar metadata across multiple clouds [#16425](https://github.com/apache/pulsar/pull/16425) +- [PIP-143] Support split bundle by specified boundaries [#13796](https://github.com/apache/pulsar/pull/13796) +- [PIP-145] Improve performance of regex subscriptions [#14505](https://github.com/apache/pulsar/pull/14505) +- [PIP-146] ManagedCursorInfo compression [#14542](https://github.com/apache/pulsar/pull/14542) +- [PIP 149] Making the REST Admin API fully async [#14365](https://github.com/apache/pulsar/issues/14365) +- [PIP-152] Support subscription level dispatch rate limiter setting [#15295](https://github.com/apache/pulsar/issues/15295) +- [PIP-153] Optimize metadataPositions in MLPendingAckStore [#15137](https://github.com/apache/pulsar/pull/15137) +- [PIP-160] Make transactions work more efficiently by aggregation operation for transaction log and pending ack store [#15370](https://github.com/apache/pulsar/issues/15370) +- [PIP-161] Exclusive Producer: ability to fence out an existing Producer [#15488](https://github.com/apache/pulsar/pull/15488) +- [PIP-163] Add lowWaterMark check before appending entry to TB [#15424](https://github.com/apache/pulsar/pull/15424) +- [PIP-166] Function add MANUAL delivery semantics [#16279](https://github.com/apache/pulsar/pull/16279) +- [PIP-179] Support the admin API to check unknown request parameters [#16135](https://github.com/apache/pulsar/issues/16135) +- [PIP-181] Pulsar Shell [#16250](https://github.com/apache/pulsar/issues/16250) +- [PIP-184] Topic specific consumer priorityLevel [#16715](https://github.com/apache/pulsar/issues/16715) +- [PIP-187] Add API to analyze a subscription backlog and provide a accurate value [#16545](https://github.com/apache/pulsar/issues/16545) + +### Broker +- [PIP 81] Split the individual acknowledgments into multiple entries + - Add range set wrapper to support record dirty ledgers [#15607](https://github.com/apache/pulsar/pull/15607) + - Notifications for faster topic discovery [#16062](https://github.com/apache/pulsar/pull/
[GitHub] [pulsar-site] Anonymitaet commented on a diff in pull request #167: [Don't Merge] Pulsar 2.11.0 release notes
Anonymitaet commented on code in PR #167: URL: https://github.com/apache/pulsar-site/pull/167#discussion_r950948327 ## site2/website/release-notes.md: ## @@ -1,6 +1,486 @@ ## Apache Pulsar Release Notes +### 2.11.0 + 2022-08-16 + +### Important notice +- Rename Pulsar txn metrics to specify OpenMetrics [#16581](https://github.com/apache/pulsar/pull/16581) +- Rename Pulsar schema metrics to specify OpenMetrics [#16610](https://github.com/apache/pulsar/pull/16610) +- Rename Pulsar lb metrics to specify OpenMetrics [#16611](https://github.com/apache/pulsar/pull/16611) +- Remove jul-to-slf4j #16320 [#16320](https://github.com/apache/pulsar/pull/16320) +- Fix get non-persistent topics issue in Namespaces [#16170](https://github.com/apache/pulsar/pull/16170) +- Fix producer/consume permission can’t get schema [#15956](https://github.com/apache/pulsar/pull/15956) +- Add verification for configured default backlog quota and retention [#15441](https://github.com/apache/pulsar/pull/15441) +- Fix broker LoadBalance uneffective [#15314](https://github.com/apache/pulsar/pull/15314) +- Ensure the deletion consistency of topic and schema [#14608](https://github.com/apache/pulsar/pull/14608) +- Transaction cumulative ack redeliver change [#14371](https://github.com/apache/pulsar/pull/14371) +- Bump prometheus client version from 0.5.0 to 0.15.0 [#13785](https://github.com/apache/pulsar/pull/13785) +- [PIP-158] Split client TLS transport encryption from authentication [#15634](https://github.com/apache/pulsar/pull/15634) +- [PIP-155] Removed Python 2 support [#15376](https://github.com/apache/pulsar/pull/15376) +- [PIP-156] Build Pulsar Server on Java 17 [#15264](https://github.com/apache/pulsar/pull/15264) +- [PIP-162] Enable system topic by default [#15619](https://github.com/apache/pulsar/pull/15619) +- [PIP-174] New managed ledger entry cache implementation [#15955](https://github.com/apache/pulsar/pull/15955) + +### PIPs +- [PIP 37] Support chunking with Shared subscription [#16202](https://github.com/apache/pulsar/pull/16202) +- [PIP-74] Pulsar client memory limits [#14400](https://github.com/apache/pulsar/pull/14400) +- [PIP-81] Split createNewMetadataLedger into multiple methods for reuse [#15425](https://github.com/apache/pulsar/pull/15425) +- [PIP 97] Update Authentication Interfaces to Include Async Authentication Methods [#12104](https://github.com/apache/pulsar/pull/12104) +- [PIP-100] Support pluggable topic factory [#12235](https://github.com/apache/pulsar/pull/12235) +- [PIP-105] Store Subscription properties [#15757](https://github.com/apache/pulsar/pull/15757) +- [PIP-117] Change Pulsar standalone defaults [#15478](https://github.com/apache/pulsar/pull/15478) +- [PIP-132] Include message header size when check maxMessageSize for non-batch message on the client side. [#14007](https://github.com/apache/pulsar/pull/14007) +- [PIP-136] Sync Pulsar metadata across multiple clouds [#16425](https://github.com/apache/pulsar/pull/16425) +- [PIP-143] Support split bundle by specified boundaries [#13796](https://github.com/apache/pulsar/pull/13796) +- [PIP-145] Improve performance of regex subscriptions [#14505](https://github.com/apache/pulsar/pull/14505) +- [PIP-146] ManagedCursorInfo compression [#14542](https://github.com/apache/pulsar/pull/14542) +- [PIP 149] Making the REST Admin API fully async [#14365](https://github.com/apache/pulsar/issues/14365) +- [PIP-152] Support subscription level dispatch rate limiter setting [#15295](https://github.com/apache/pulsar/issues/15295) +- [PIP-153] Optimize metadataPositions in MLPendingAckStore [#15137](https://github.com/apache/pulsar/pull/15137) +- [PIP-160] Make transactions work more efficiently by aggregation operation for transaction log and pending ack store [#15370](https://github.com/apache/pulsar/issues/15370) +- [PIP-161] Exclusive Producer: ability to fence out an existing Producer [#15488](https://github.com/apache/pulsar/pull/15488) +- [PIP-163] Add lowWaterMark check before appending entry to TB [#15424](https://github.com/apache/pulsar/pull/15424) +- [PIP-166] Function add MANUAL delivery semantics [#16279](https://github.com/apache/pulsar/pull/16279) +- [PIP-179] Support the admin API to check unknown request parameters [#16135](https://github.com/apache/pulsar/issues/16135) +- [PIP-181] Pulsar Shell [#16250](https://github.com/apache/pulsar/issues/16250) +- [PIP-184] Topic specific consumer priorityLevel [#16715](https://github.com/apache/pulsar/issues/16715) +- [PIP-187] Add API to analyze a subscription backlog and provide a accurate value [#16545](https://github.com/apache/pulsar/issues/16545) + +### Broker +- [PIP 81] Split the individual acknowledgments into multiple entries + - Add range set wrapper to support record dirty ledgers [#15607](https://github.com/apache/pulsar/pull/15607) + - Notifications for faster topic discovery [#16062](https://github.com/apache/pulsar/pull/
RE: Enable "Update branch" Github button in pull requests
+1 Best, Zixuan On 2022/07/12 16:06:43 Nicolò Boschi wrote: > Hi all, > > I'd like to propose to enable the Github button "update branch" in the pull > requests. > > The main reason is that it helps when you want to rebase your pull to the > current master. > For instance today a fix for a failing job in the CI has been committed and > you were forced to close/reopen the pull or manual rebase your branch. With > this button is much easier and it will retrigger the CI as well. > > This is the guide to enable it: > https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/keeping-your-pull-request-in-sync-with-the-base-branch > > This is an example where I activated the button in my own Pulsar fork: > https://github.com/nicoloboschi/pulsar/pull/4 > > Thanks, > Nicolò Boschi >
RE: [DISCUSS] Enable non-mandatory updating PR branches
+1 Best, Zixuan On 2022/08/18 13:01:58 tison wrote: > Hello, > > The short version > = > > Vote if you agree on enabling the non-mandatory updating PR branches > button, i.e., the "Always suggest updating pull request branches" GitHub > settings. > > The full version > > > Pulsar is under rapid development and numerous fixes are pushed to master > every time. Since we are still suffering from quite a few flaky tests, > merge master and retest is a hotspot to verify the patch once more. > > However, we should pull the nightly master locally, check out the PR > branch, perform the merge and push to remote. It's a bit awkward especially > when a developer works on multiple branches. > > GitHub provides a button "Always suggest updating pull request branches" > with the description "Whenever there are new changes available in the base > branch, present an “update branch” option in the pull request."[1] > > It can simplify the workflow with one button click. > > To clarify, this is different from the branch protection rule "Require > branches to be up to date before merging" - it's non-mandatory and just > provides the "update branch" button. It means we don't force every PR to > catch up with the latest master before merged, which can cause exextremely > high unnecessary traffic. Since we already allow PR authors to retrigger > tests with the pulsorbot (or even contributors can push an empty commit), > providing such a button does no harm. > > I post this thread here to collect feedback, especially from the PMC > members. Previously I asked the INFRA team to turn on this option for > Apache Kvrocks (Incubating)[2] and I believe the INFRA team would be happy > to see an explicit community consensus. > > Best, > tison. > > [1] > https://github.blog/changelog/2022-02-03-more-ways-to-keep-your-pull-request-branch-up-to-date/ > [2] https://issues.apache.org/jira/browse/INFRA-23432 >
[GitHub] [pulsar-site] urfreespace merged pull request #170: fix version replace
urfreespace merged PR #170: URL: https://github.com/apache/pulsar-site/pull/170 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BohanZhang0222 created a discussion: Flink Connector Exception: Failed to list subscribed topic partitions due to
GitHub user BohanZhang0222 created a discussion: Flink Connector Exception:
Failed to list subscribed topic partitions due to
## Env
Pulsar Version: 2.9.2
Flink Version: 1.14
##Source Conf
`{"topicNames":["persistent://autopilot/x01/autopilot-filter-iot-signal-canlog-x01-testtwo-partition-0"],"topicsPattern":null,"subscriptionName":"ontest-lyt","subscriptionType":"Key_Shared","subscriptionMode":"Durable","receiverQueueSize":1000,"acknowledgementsGroupTimeMicros":10,"negativeAckRedeliveryDelayMicros":6000,"maxTotalReceiverQueueSizeAcrossPartitions":5,"consumerName":null,"ackTimeoutMillis":0,"tickDurationMillis":1000,"priorityLevel":0,"maxPendingChunkedMessage":10,"autoAckOldestChunkedMessageOnQueueFull":false,"expireTimeOfIncompleteChunkedMessageMillis":6,"cryptoFailureAction":"FA
IL","properties":{},"readCompacted":false,"subscriptionInitialPosition":"Latest","patternAutoDiscoveryPeriod":60,"regexSubscriptionMode":"PersistentOnly","deadLetterPolicy":null,"retryEnable":false,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"replicateSubscriptionState":false,"resetIncludeHead":false,"keySharedPolicy":{"keySharedMode":"STICKY","allowOutOfOrderDelivery":false,"ranges":[{"start":0,"end":65535}],"hashRangeTotal":65536},"batchIndexAckEnabled":false,"ackReceiptEnabled":false,"poolMessages":false,"maxPendingChuckedMessage":10}
2022-08-21 16:56:45,407 INFO
org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl [] - Pulsar client
config:
{"serviceUrl":"pulsar://bigdata-pulsar-ontest.dip.chj.cloud:6650","authPluginClassName":null,"authParams":null,"authParamMap":null,"operationTimeoutMs":3,"lookupTimeoutMs":3,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"useTcpNoDelay":true,"useTls":false,"tlsTrustCertsFilePath":"","
tlsAllowInsecureConnection":false,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":5,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":1,"requestTimeoutMs":6,"initialBackoffIntervalNanos":1,"maxBackoffIntervalNanos":600,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null}`
## Exception
`2022-08-21 17:11:14,147 ERROR
org.apache.flink.runtime.source.coordinator.SourceCoordinatorContext [] -
Exception while handling result from async call in SourceCoordinator-Source:
Pulsar Source. Triggering job failover.
org.apache.flink.util.FlinkRuntimeException: Failed to list subscribed topic
partitions due to
at
org.apache.flink.connector.pulsar.source.enumerator.PulsarSourceEnumerator.checkPartitionChanges(PulsarSourceEnumerator.java:232)
~[mylogpack-jar-with-dependencies.jar:?]
at
org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$null$4(ExecutorNotifier.java:133)
~[mylogpack-jar-with-dependencies.jar:?]
at
org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:40)
[flink-dist_2.11-1.14.3.jar:1.14.3]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_162]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_162]
at java.util.concurrent.ScheduledThr
eadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_162]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[?:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException:
org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException:
HTTP 502 Bad Gateway
at
org.apache.flink.connector.pulsar.source.enumerator.subscriber.impl.BasePulsarSubscriber.queryTopicMetadata(BasePulsarSubscriber.java:53)
~[mylogpack-jar-with-dependencies.jar:?]
at
org.apache.flink.connector.pulsar.source.enumerator.subscriber.impl.TopicListSubscriber.lambda$getSubscribedTopicPartitions$0(TopicListSubscriber.java:48)
~[mylogpack-jar-with-dependencies.jar:?]
at java.util.stre
am.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_162]
at
java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
~[?:1.8.0_162]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
~[?:1.8.0_162]
at
java.util.
[GitHub] [pulsar] BohanZhang0222 edited a discussion: Flink Connector Exception: Failed to list subscribed topic partitions due to
GitHub user BohanZhang0222 edited a discussion: Flink Connector Exception:
Failed to list subscribed topic partitions due to
## Env
Pulsar Version: 2.9.2
Flink Version: 1.14
##Source Conf
`{"topicNames":["persistent://autopilot/x01/autopilot-filter-iot-signal-canlog-x01-testtwo-partition-0"],"topicsPattern":null,"subscriptionName":"ontest-lyt","subscriptionType":"Key_Shared","subscriptionMode":"Durable","receiverQueueSize":1000,"acknowledgementsGroupTimeMicros":10,"negativeAckRedeliveryDelayMicros":6000,"maxTotalReceiverQueueSizeAcrossPartitions":5,"consumerName":null,"ackTimeoutMillis":0,"tickDurationMillis":1000,"priorityLevel":0,"maxPendingChunkedMessage":10,"autoAckOldestChunkedMessageOnQueueFull":false,"expireTimeOfIncompleteChunkedMessageMillis":6,"cryptoFailureAction":"FA
IL","properties":{},"readCompacted":false,"subscriptionInitialPosition":"Latest","patternAutoDiscoveryPeriod":60,"regexSubscriptionMode":"PersistentOnly","deadLetterPolicy":null,"retryEnable":false,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"replicateSubscriptionState":false,"resetIncludeHead":false,"keySharedPolicy":{"keySharedMode":"STICKY","allowOutOfOrderDelivery":false,"ranges":[{"start":0,"end":65535}],"hashRangeTotal":65536},"batchIndexAckEnabled":false,"ackReceiptEnabled":false,"poolMessages":false,"maxPendingChuckedMessage":10}
2022-08-21 16:56:45,407 INFO
org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl [] - Pulsar client
config:
{"serviceUrl":"pulsar://bigdata-pulsar-ontest.dip.chj.cloud:6650","authPluginClassName":null,"authParams":null,"authParamMap":null,"operationTimeoutMs":3,"lookupTimeoutMs":3,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"useTcpNoDelay":true,"useTls":false,"tlsTrustCertsFilePath":"","
tlsAllowInsecureConnection":false,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":5,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":1,"requestTimeoutMs":6,"initialBackoffIntervalNanos":1,"maxBackoffIntervalNanos":600,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null}`
## Exception
`2022-08-21 17:11:14,147 ERROR
org.apache.flink.runtime.source.coordinator.SourceCoordinatorContext [] -
Exception while handling result from async call in SourceCoordinator-Source:
Pulsar Source. Triggering job failover.
org.apache.flink.util.FlinkRuntimeException: Failed to list subscribed topic
partitions due to
at
org.apache.flink.connector.pulsar.source.enumerator.PulsarSourceEnumerator.checkPartitionChanges(PulsarSourceEnumerator.java:232)
~[mylogpack-jar-with-dependencies.jar:?]
at
org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$null$4(ExecutorNotifier.java:133)
~[mylogpack-jar-with-dependencies.jar:?]
at
org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:40)
[flink-dist_2.11-1.14.3.jar:1.14.3]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_162]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_162]
at java.util.concurrent.ScheduledThr
eadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_162]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[?:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException:
org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException:
HTTP 502 Bad Gateway
at
org.apache.flink.connector.pulsar.source.enumerator.subscriber.impl.BasePulsarSubscriber.queryTopicMetadata(BasePulsarSubscriber.java:53)
~[mylogpack-jar-with-dependencies.jar:?]
at
org.apache.flink.connector.pulsar.source.enumerator.subscriber.impl.TopicListSubscriber.lambda$getSubscribedTopicPartitions$0(TopicListSubscriber.java:48)
~[mylogpack-jar-with-dependencies.jar:?]
at java.util.stre
am.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_162]
at
java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
~[?:1.8.0_162]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
~[?:1.8.0_162]
at
java.util.s
Re: [Discussion] PIP 198 - How to define [type] and [scope]?
Thank you tison and Zixuan! Agree on the following aspects: # 1. Remove 3 [scope]s - Remove [workflow] since it can be replaced with other scopes eg. "[feat][workflow] Add instructions for previewing website changes" can be written as "[feat][doc] Add instructions for previewing website changes" - Remove [depoly] since changes to deployment can be represented by other [scope]s. - Remove [pkg]since it refers to package API [1], which belongs to [admin]. # 2. Update 4 [scope]s - Add [meta], which refers to changes to metadata. - Add [storage], which refers to changes to managed ledger. - Rename [ts] to [offloaded], which refers to changes to tiered storage. - Rename [func] to [fn], which refers to changes to function. # 3. Remain the same These formats are fine to go: - Submit breaking changes [feat][broker]! Support xx - Submit PIP changes [feat][broker] PIP-198: Support xx Feel free to comment, thank you! [1] https://pulsar.apache.org/docs/next/admin-api-packages Yu and Zixuan On Fri, Aug 19, 2022 at 6:21 PM Zixuan Liu wrote: > +1 for fcn -> fn > +1 for ts -> offloader > > +1 * ci - CI workflow changes or debugging. > +1 * build - all things related to the build system, including tools, > deployment logic, maven changes, packaging logics, docker image, > buildscripts. > > `pkg` should belong to the `admin` scope, so suggest using the `admin` > instead `pkg`. > > `tool` is pulsar-admin, pulsar, pulsar-client, and so on cli, so keep using > the `tool`. > > deploy should belong to the `build` scope`, so suggest using the `build` > instead `deploy`. > > > tison 于2022年8月19日周五 17:46写道： > > > BTW, how can I sort changes for the metadata store? > > > > Best, > > tison. > > > > > > tison 于2022年8月19日周五 17:44写道： > > > > > To proposal a workable solution, I suggest: > > > > > > replace > > > > > > * pkg > > > * tool > > > * deploy > > > * ci > > > * workflow > > > * build > > > > > > with > > > > > > * ci - CI workflow changes or debugging. > > > * build - all things related to the build system, including tools, > > > deployment logic, maven changes, packaging logics, docker image, build > > > scripts. > > > > > > Best, > > > tison. > > > > > > > > > tison 于2022年8月19日周五 17:41写道： > > > > > >> > I intended to mean changes to "process / standard / guide" [2] > rather > > >> than "CI workflow", but it still causes confusion. > > >> > > >> How can a PR be relevant to these things? I think the result should be > > >> either CI workflow changes or document updates. We don't need a > > dedicated > > >> "workflow" in such situations. > > >> > > >> > build system or external dependencies. > > >> > > >> So, it overlaps with deps. While I can regard it as Maven/Docker/Build > > >> script related changes, we may not have "pkg", "tool", "deploy" that > can > > >> overlap with this. > > >> > > >> Best, > > >> tison. > > >> > > >> > > >> tison 于2022年8月19日周五 17:38写道： > > >> > > >>> As for the type candidates: > > >>> > > >>> LGTM. No comment here. > > >>> > > >>> As for the scope candidates: > > >>> > > >>> +1 for dep -> deps > > >>> +1 for fcn -> fn > > >>> Among most communities and language conventions, the abbr of > function > > >>> is fn (Rust), fun (Erlang), or func (Golang). No fcn IIRC. I'd prefer > > the > > >>> short one, fn. > > >>> +1 for ts -> offloader > > >>> If I get it right, tiered storage is offloader. We can save one > word > > >>> while keeping semantic. > > >>> > > >>> I don't know clearly what is: > > >>> * pkg > > >>> * tool > > >>> * deploy > > >>> * ci > > >>> * workflow > > >>> * build > > >>> They look quite similar or overlapping. > > >>> > > >>> Rest LGTM. > > >>> > > >>> As for the remaining issues: > > >>> > > >>> - Submit breaking changes > > >>> [feat][broker]! Support xx > > >>> > > >>> This is fine. Since we don't verify what follows the [type][scope] > > >>> section, it doesn't block the proposal. > > >>> > > >>> - Submit PIP changes > > >>> [feat][broker] PIP-198: Support xx > > >>> > > >>> The same as before. Contributors can name whatever they like. We > don't > > >>> set too complex rules. > > >>> > > >>> - Cherry pick changes [4] > > >>> Choice A: [fix][broker][branch-2.9] xxx > > >>> Choice B: [fix][broker] xxx. And add "cherry pick xxx to branch-2.9" > in > > >>> the > > >>> PR description. > > >>> > > >>> I'd prefer [fix][broker][BP-2.9] to save some letters. This is how > the > > >>> Flink community does. BP means backport. But yes, it's not a > > requirement > > >>> but a suggestion. > > >>> > > >>> > > >>> Best, > > >>> tison. > > >>> > > >> > > >
[DISCUSS] Use Github project to track the multi clients development
Hi, all Currently, many features or bug fixes of some clients like the C++ client and Python client have not caught up with the Java client. For better tracking of these features or bug fixes, I have created a Github project called `Apache Pulsar / Multi Clients`[0] under the pulsar main repo. It can be removed if we don't like it this way. There are six columns in this Github project: * Backlog: Low priority issues or issues(or cards) that have not been initially investigated. * Todo: Newly added issues and issues suitable for contributors to pick first. * In progress: Work in progress issues or PRs. * Review in progress: The PRs or issues that are waiting for review. * Reviewer approved: The PRs or issues that are waiting to merge. * Done: Merged PRs and Solved issues. This not only helps to track the status of other clients' features and bug fixes but also helps contributors who want to participate in other language client feature development to pick the task. This Project is currently only for clients maintained by the main pulsar repository, such as C++ and Python clients. I have created a PR [1] to add C++ and Python client issues and PRs to the Multi Clients project. Please let me know what you think. Thanks! [0] https://github.com/apache/pulsar/projects/12 [1] https://github.com/apache/pulsar/pull/17201 Best, Zike Yang
Re: [Discussion] PIP 198 - How to define [type] and [scope]?
Hi tison, Thanks for your suggestions! We have several questions on [build]: > build - all things related to the build system, including tools, deployment logic, maven changes, packaging logics, docker image, build scripts. # 1. "tools" What do you refer to? Plugins? Besides, the existing scope, [tool], refers to Pulsar CLI tools [1]. We're considering to rename it to [cli] since: a. "cli" is more clear and short b. Save the word "tool" for future use Does it make sense? # 2. "deployment logic" Seems that it's an obsolete module and has not been updated for a long while. If so, can we ignore this? # 3. Does "packaging logics" belong to [admin]? [2] # 4. How about defining [build] refer to the following? - Dependency (Maven) In this way, we do not have the scope [dependency] since the changes to dependency belong to [build]. - Docker - Build or release script Thank you for your reply! [1] https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.khz275ok35u5 [2] https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.nnekhkthmwlh Yu and Zixuan On Mon, Aug 22, 2022 at 12:40 PM Yu wrote: > Thank you tison and Zixuan! > > Agree on the following aspects: > > > > # 1. Remove 3 [scope]s > > - Remove [workflow] since it can be replaced with other scopes > eg. > "[feat][workflow] Add instructions for previewing website changes" > can be written as > "[feat][doc] Add instructions for previewing website changes" > > - Remove [depoly] since changes to deployment can be represented by other > [scope]s. > > - Remove [pkg]since it refers to package API [1], which belongs to [admin]. > > > > # 2. Update 4 [scope]s > > - Add [meta], which refers to changes to metadata. > > - Add [storage], which refers to changes to managed ledger. > > - Rename [ts] to [offloaded], which refers to changes to tiered storage. > > - Rename [func] to [fn], which refers to changes to function. > > > > # 3. Remain the same > > These formats are fine to go: > > - Submit breaking changes > [feat][broker]! Support xx > > - Submit PIP changes > [feat][broker] PIP-198: Support xx > > > > Feel free to comment, thank you! > > > > [1] https://pulsar.apache.org/docs/next/admin-api-packages > > Yu and Zixuan > > On Fri, Aug 19, 2022 at 6:21 PM Zixuan Liu wrote: > >> +1 for fcn -> fn >> +1 for ts -> offloader >> >> +1 * ci - CI workflow changes or debugging. >> +1 * build - all things related to the build system, including tools, >> deployment logic, maven changes, packaging logics, docker image, >> buildscripts. >> >> `pkg` should belong to the `admin` scope, so suggest using the `admin` >> instead `pkg`. >> >> `tool` is pulsar-admin, pulsar, pulsar-client, and so on cli, so keep >> using >> the `tool`. >> >> deploy should belong to the `build` scope`, so suggest using the `build` >> instead `deploy`. >> >> >> tison 于2022年8月19日周五 17:46写道： >> >> > BTW, how can I sort changes for the metadata store? >> > >> > Best, >> > tison. >> > >> > >> > tison 于2022年8月19日周五 17:44写道： >> > >> > > To proposal a workable solution, I suggest: >> > > >> > > replace >> > > >> > > * pkg >> > > * tool >> > > * deploy >> > > * ci >> > > * workflow >> > > * build >> > > >> > > with >> > > >> > > * ci - CI workflow changes or debugging. >> > > * build - all things related to the build system, including tools, >> > > deployment logic, maven changes, packaging logics, docker image, build >> > > scripts. >> > > >> > > Best, >> > > tison. >> > > >> > > >> > > tison 于2022年8月19日周五 17:41写道： >> > > >> > >> > I intended to mean changes to "process / standard / guide" [2] >> rather >> > >> than "CI workflow", but it still causes confusion. >> > >> >> > >> How can a PR be relevant to these things? I think the result should >> be >> > >> either CI workflow changes or document updates. We don't need a >> > dedicated >> > >> "workflow" in such situations. >> > >> >> > >> > build system or external dependencies. >> > >> >> > >> So, it overlaps with deps. While I can regard it as >> Maven/Docker/Build >> > >> script related changes, we may not have "pkg", "tool", "deploy" that >> can >> > >> overlap with this. >> > >> >> > >> Best, >> > >> tison. >> > >> >> > >> >> > >> tison 于2022年8月19日周五 17:38写道： >> > >> >> > >>> As for the type candidates: >> > >>> >> > >>> LGTM. No comment here. >> > >>> >> > >>> As for the scope candidates: >> > >>> >> > >>> +1 for dep -> deps >> > >>> +1 for fcn -> fn >> > >>> Among most communities and language conventions, the abbr of >> function >> > >>> is fn (Rust), fun (Erlang), or func (Golang). No fcn IIRC. I'd >> prefer
Re: [Discussion] PIP 198 - How to define [type] and [scope]?
Hi developers, Two quick questions need your vote! Which do you prefer? # 1. Use "branch" or "BP"? Choice A: [fix][broker][branch-2.9] xxx Choice B: [fix][broker][BP-2.9] xxx # 2. for the [scope], use "misc" or "chore"? [1] Choice A: misc Choice B: chore Thank you all! [1] https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.58q1qxhu7pio Yu On Mon, Aug 22, 2022 at 12:44 PM Yu wrote: > Hi tison, > > Thanks for your suggestions! We have several questions on [build]: > > > build - all things related to the build system, including tools, > deployment logic, maven changes, packaging logics, docker image, build > scripts. > > > > # 1. "tools" > > What do you refer to? Plugins? > > Besides, the existing scope, [tool], refers to Pulsar CLI tools [1]. > We're considering to rename it to [cli] since: > a. "cli" is more clear and short > b. Save the word "tool" for future use > > Does it make sense? > > > > # 2. "deployment logic" > > Seems that it's an obsolete module and has not been updated for a long > while. > If so, can we ignore this? > > > > # 3. Does "packaging logics" belong to [admin]? [2] > > > > # 4. How about defining [build] refer to the following? > > - Dependency (Maven) > > In this way, we do not have the scope [dependency] since the changes to > dependency belong to [build]. > > - Docker > > - Build or release script > > > > Thank you for your reply! > > [1] > https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.khz275ok35u5 > [2] > https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.nnekhkthmwlh > > Yu and Zixuan > > On Mon, Aug 22, 2022 at 12:40 PM Yu wrote: > >> Thank you tison and Zixuan! >> >> Agree on the following aspects: >> >> >> >> # 1. Remove 3 [scope]s >> >> - Remove [workflow] since it can be replaced with other scopes >> eg. >> "[feat][workflow] Add instructions for previewing website changes" >> can be written as >> "[feat][doc] Add instructions for previewing website changes" >> >> - Remove [depoly] since changes to deployment can be represented by other >> [scope]s. >> >> - Remove [pkg]since it refers to package API [1], which belongs to >> [admin]. >> >> >> >> # 2. Update 4 [scope]s >> >> - Add [meta], which refers to changes to metadata. >> >> - Add [storage], which refers to changes to managed ledger. >> >> - Rename [ts] to [offloaded], which refers to changes to tiered storage. >> >> - Rename [func] to [fn], which refers to changes to function. >> >> >> >> # 3. Remain the same >> >> These formats are fine to go: >> >> - Submit breaking changes >> [feat][broker]! Support xx >> >> - Submit PIP changes >> [feat][broker] PIP-198: Support xx >> >> >> >> Feel free to comment, thank you! >> >> >> >> [1] https://pulsar.apache.org/docs/next/admin-api-packages >> >> Yu and Zixuan >> >> On Fri, Aug 19, 2022 at 6:21 PM Zixuan Liu wrote: >> >>> +1 for fcn -> fn >>> +1 for ts -> offloader >>> >>> +1 * ci - CI workflow changes or debugging. >>> +1 * build - all things related to the build system, including tools, >>> deployment logic, maven changes, packaging logics, docker image, >>> buildscripts. >>> >>> `pkg` should belong to the `admin` scope, so suggest using the `admin` >>> instead `pkg`. >>> >>> `tool` is pulsar-admin, pulsar, pulsar-client, and so on cli, so keep >>> using >>> the `tool`. >>> >>> deploy should belong to the `build` scope`, so suggest using the `build` >>> instead `deploy`. >>> >>> >>> tison 于2022年8月19日周五 17:46写道： >>> >>> > BTW, how can I sort changes for the metadata store? >>> > >>> > Best, >>> > tison. >>> > >>> > >>> > tison 于2022年8月19日周五 17:44写道： >>> > >>> > > To proposal a workable solution, I suggest: >>> > > >>> > > replace >>> > > >>> > > * pkg >>> > > * tool >>> > > * deploy >>> > > * ci >>> > > * workflow >>> > > * build >>> > > >>> > > with >>> > > >>> > > * ci - CI workflow changes or debugging. >>> > > * build - all things related to the build system, including tools, >>> > > deployment logic, maven changes, packaging logics, docker image, >>> build >>> > > scripts. >>> > > >>> > > Best, >>> > > tison. >>> > > >>> > > >>> > > tison 于2022年8月19日周五 17:41写道： >>> > > >>> > >> > I intended to mean changes to "process / standard / guide" [2] >>> rather >>> > >> than "CI workflow", but it still causes confusion. >>> > >> >>> > >> How can a PR be relevant to these things? I think the result should >>> be >>> > >> either CI workflow changes or document updates. We don't need
Re: [Discussion] PIP 198 - How to define [type] and [scope]?
A A Thanks Tao Jiuming > 2022年8月22日 下午12:47，Yu 写道： > > Hi developers, > > Two quick questions need your vote! > > Which do you prefer? > > > > # 1. Use "branch" or "BP"? > > Choice A: [fix][broker][branch-2.9] xxx > Choice B: [fix][broker][BP-2.9] xxx > > > > # 2. for the [scope], use "misc" or "chore"? [1] > > Choice A: misc > Choice B: chore > > > > Thank you all! > > [1] > https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.58q1qxhu7pio > > Yu > > On Mon, Aug 22, 2022 at 12:44 PM Yu wrote: > >> Hi tison, >> >> Thanks for your suggestions! We have several questions on [build]: >> >>> build - all things related to the build system, including tools, >> deployment logic, maven changes, packaging logics, docker image, build >> scripts. >> >> >> >> # 1. "tools" >> >> What do you refer to? Plugins? >> >> Besides, the existing scope, [tool], refers to Pulsar CLI tools [1]. >> We're considering to rename it to [cli] since: >> a. "cli" is more clear and short >> b. Save the word "tool" for future use >> >> Does it make sense? >> >> >> >> # 2. "deployment logic" >> >> Seems that it's an obsolete module and has not been updated for a long >> while. >> If so, can we ignore this? >> >> >> >> # 3. Does "packaging logics" belong to [admin]? [2] >> >> >> >> # 4. How about defining [build] refer to the following? >> >> - Dependency (Maven) >> >> In this way, we do not have the scope [dependency] since the changes to >> dependency belong to [build]. >> >> - Docker >> >> - Build or release script >> >> >> >> Thank you for your reply! >> >> [1] >> https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.khz275ok35u5 >> [2] >> https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.nnekhkthmwlh >> >> Yu and Zixuan >> >> On Mon, Aug 22, 2022 at 12:40 PM Yu wrote: >> >>> Thank you tison and Zixuan! >>> >>> Agree on the following aspects: >>> >>> >>> >>> # 1. Remove 3 [scope]s >>> >>> - Remove [workflow] since it can be replaced with other scopes >>> eg. >>> "[feat][workflow] Add instructions for previewing website changes" >>> can be written as >>> "[feat][doc] Add instructions for previewing website changes" >>> >>> - Remove [depoly] since changes to deployment can be represented by other >>> [scope]s. >>> >>> - Remove [pkg]since it refers to package API [1], which belongs to >>> [admin]. >>> >>> >>> >>> # 2. Update 4 [scope]s >>> >>> - Add [meta], which refers to changes to metadata. >>> >>> - Add [storage], which refers to changes to managed ledger. >>> >>> - Rename [ts] to [offloaded], which refers to changes to tiered storage. >>> >>> - Rename [func] to [fn], which refers to changes to function. >>> >>> >>> >>> # 3. Remain the same >>> >>> These formats are fine to go: >>> >>> - Submit breaking changes >>> [feat][broker]! Support xx >>> >>> - Submit PIP changes >>> [feat][broker] PIP-198: Support xx >>> >>> >>> >>> Feel free to comment, thank you! >>> >>> >>> >>> [1] https://pulsar.apache.org/docs/next/admin-api-packages >>> >>> Yu and Zixuan >>> >>> On Fri, Aug 19, 2022 at 6:21 PM Zixuan Liu wrote: >>> +1 for fcn -> fn +1 for ts -> offloader +1 * ci - CI workflow changes or debugging. +1 * build - all things related to the build system, including tools, deployment logic, maven changes, packaging logics, docker image, buildscripts. `pkg` should belong to the `admin` scope, so suggest using the `admin` instead `pkg`. `tool` is pulsar-admin, pulsar, pulsar-client, and so on cli, so keep using the `tool`. deploy should belong to the `build` scope`, so suggest using the `build` instead `deploy`. tison 于2022年8月19日周五 17:46写道： > BTW, how can I sort changes for the metadata store? > > Best, > tison. > > > tison 于2022年8月19日周五 17:44写道： > >> To proposal a workable solution, I suggest: >> >> replace >> >> * pkg >> * tool >> * deploy >> * ci >> * workflow >> * build >> >> with >> >> * ci - CI workflow changes or debugging. >> * build - all things related to the build system, including tools, >> deployment logic, maven changes, packaging logics, docker image, build >> scripts. >> >> Best, >> tison. >> >> >> tison 于2022年8月19日周五 17:41写道： >> I i
Re: [VOTE]PIP-189: No batching if only one message in batch.
+1 (binding) Regards Jiwei Guo (Tboy) On Mon, Aug 8, 2022 at 11:17 AM PengHui Li wrote: > +1 (binding) > > Penghui > > On Tue, Jul 26, 2022 at 10:23 PM Zike Yang wrote: > > > +1 (non-binding) > > Great work! > > > > Thanks > > Zike Yang > > > > On Tue, Jul 26, 2022 at 7:31 PM Enrico Olivelli > > wrote: > > > > > > +1 (binding) > > > > > > This is a small improvement but a great impact !! > > > > > > Enrico > > > > > > Il giorno mar 26 lug 2022 alle ore 12:48 Anon Hxy > > > ha scritto: > > > > > > > > Hi Zike > > > > > > > > Thanks for your reminding. I just forgot to update the issue and > have > > > > updated it now. I am not going to add the configuration > > > > `batchingSingleMessage ` to the producer. > > > > > > > > Thanks, > > > > Xiaoyu Hou > > > > > > > > Zike Yang 于2022年7月26日周二 18:16写道： > > > > > > > > > Hi, Anon > > > > > > > > > > So are you going to add the configuration `batchingSingleMessage ` > to > > > > > the producer? I saw that it was still in the PIP: > > > > > > > > > > > So this may cause ((BatchMessageIdImpl) messageId) throw > > > > > ClassCastException. we need to add a switch for the producer to > > enable or > > > > > disable this feature > > > > > > ProducerBuilder batchingSingleMessage(boolean > > > > > batchingSingleMessage); // default value is true > > > > > > > > > > > > > > > > > > > > > > > > > Zike Yang > > > > > > > > > > On Tue, Jul 26, 2022 at 6:05 PM Qiang Huang < > > qiang.huang1...@gmail.com> > > > > > wrote: > > > > > > > > > > > > +1(non-binding) > > > > > > > > > > > > BR, > > > > > > Qiang Huang > > > > > > > > > > > > mattison chao 于2022年7月25日周一 13:17写道： > > > > > > > > > > > > > +1(non-binding) > > > > > > > > > > > > > > Best, > > > > > > > Mattison > > > > > > > > > > > > > > On Mon, 25 Jul 2022 at 10:35, Haiting Jiang < > > jianghait...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > > +1 > > > > > > > > Thanks, > > > > > > > > Haiting > > > > > > > > > > > > > > > > On 2022/07/25 02:23:20 Anon Hxy wrote: > > > > > > > > > Dear Community, > > > > > > > > > > > > > > > > > > I would like to start a VOTE on "PIP-189: No batching if > > only one > > > > > > > message > > > > > > > > > in batch." > > > > > > > > > > > > > > > > > > The proposal can be read at [0] and the discussion thread > is > > > > > available > > > > > > > at > > > > > > > > > [1] and the PR link is available at [2] > > > > > > > > > > > > > > > > > > Voting will stay open for at least 48h. > > > > > > > > > > > > > > > > > > [0] https://github.com/apache/pulsar/issues/16619 > > > > > > > > > [1] > > > > > https://lists.apache.org/thread/dbq1lrv03bhtk0lr5nwm5txo9ndjplv0 > > > > > > > > > [2] https://github.com/apache/pulsar/pull/16605 > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Xiaoyu > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > BR, > > > > > > Qiang Huang > > > > > > > >
Re: [Discussion] PIP 198 - How to define [type] and [scope]?
A, A Thanks, Yunze > 2022年8月22日 12:47，Yu 写道： > > Hi developers, > > Two quick questions need your vote! > > Which do you prefer? > > > > # 1. Use "branch" or "BP"? > > Choice A: [fix][broker][branch-2.9] xxx > Choice B: [fix][broker][BP-2.9] xxx > > > > # 2. for the [scope], use "misc" or "chore"? [1] > > Choice A: misc > Choice B: chore > > > > Thank you all! > > [1] > https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.58q1qxhu7pio > > Yu > > On Mon, Aug 22, 2022 at 12:44 PM Yu wrote: > >> Hi tison, >> >> Thanks for your suggestions! We have several questions on [build]: >> >>> build - all things related to the build system, including tools, >> deployment logic, maven changes, packaging logics, docker image, build >> scripts. >> >> >> >> # 1. "tools" >> >> What do you refer to? Plugins? >> >> Besides, the existing scope, [tool], refers to Pulsar CLI tools [1]. >> We're considering to rename it to [cli] since: >> a. "cli" is more clear and short >> b. Save the word "tool" for future use >> >> Does it make sense? >> >> >> >> # 2. "deployment logic" >> >> Seems that it's an obsolete module and has not been updated for a long >> while. >> If so, can we ignore this? >> >> >> >> # 3. Does "packaging logics" belong to [admin]? [2] >> >> >> >> # 4. How about defining [build] refer to the following? >> >> - Dependency (Maven) >> >> In this way, we do not have the scope [dependency] since the changes to >> dependency belong to [build]. >> >> - Docker >> >> - Build or release script >> >> >> >> Thank you for your reply! >> >> [1] >> https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.khz275ok35u5 >> [2] >> https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.nnekhkthmwlh >> >> Yu and Zixuan >> >> On Mon, Aug 22, 2022 at 12:40 PM Yu wrote: >> >>> Thank you tison and Zixuan! >>> >>> Agree on the following aspects: >>> >>> >>> >>> # 1. Remove 3 [scope]s >>> >>> - Remove [workflow] since it can be replaced with other scopes >>> eg. >>> "[feat][workflow] Add instructions for previewing website changes" >>> can be written as >>> "[feat][doc] Add instructions for previewing website changes" >>> >>> - Remove [depoly] since changes to deployment can be represented by other >>> [scope]s. >>> >>> - Remove [pkg]since it refers to package API [1], which belongs to >>> [admin]. >>> >>> >>> >>> # 2. Update 4 [scope]s >>> >>> - Add [meta], which refers to changes to metadata. >>> >>> - Add [storage], which refers to changes to managed ledger. >>> >>> - Rename [ts] to [offloaded], which refers to changes to tiered storage. >>> >>> - Rename [func] to [fn], which refers to changes to function. >>> >>> >>> >>> # 3. Remain the same >>> >>> These formats are fine to go: >>> >>> - Submit breaking changes >>> [feat][broker]! Support xx >>> >>> - Submit PIP changes >>> [feat][broker] PIP-198: Support xx >>> >>> >>> >>> Feel free to comment, thank you! >>> >>> >>> >>> [1] https://pulsar.apache.org/docs/next/admin-api-packages >>> >>> Yu and Zixuan >>> >>> On Fri, Aug 19, 2022 at 6:21 PM Zixuan Liu wrote: >>> +1 for fcn -> fn +1 for ts -> offloader +1 * ci - CI workflow changes or debugging. +1 * build - all things related to the build system, including tools, deployment logic, maven changes, packaging logics, docker image, buildscripts. `pkg` should belong to the `admin` scope, so suggest using the `admin` instead `pkg`. `tool` is pulsar-admin, pulsar, pulsar-client, and so on cli, so keep using the `tool`. deploy should belong to the `build` scope`, so suggest using the `build` instead `deploy`. tison 于2022年8月19日周五 17:46写道： > BTW, how can I sort changes for the metadata store? > > Best, > tison. > > > tison 于2022年8月19日周五 17:44写道： > >> To proposal a workable solution, I suggest: >> >> replace >> >> * pkg >> * tool >> * deploy >> * ci >> * workflow >> * build >> >> with >> >> * ci - CI workflow changes or debugging. >> * build - all things related to the build system, including tools, >> deployment logic, maven changes, packaging logics, docker image, build >> scripts. >> >> Best, >> tison. >> >> >> tison 于2022年8月19日周五 17:41写道： >> I int
Re: [VOTE]PIP-189: No batching if only one message in batch.
Thank you all. Close the vote with 3 (+1) bindings and 4 (+1) non-bindings. Thanks, Xiaoyu Hou guo jiwei 于2022年8月22日周一 13:18写道： > +1 (binding) > > > Regards > Jiwei Guo (Tboy) > > > On Mon, Aug 8, 2022 at 11:17 AM PengHui Li wrote: > > > +1 (binding) > > > > Penghui > > > > On Tue, Jul 26, 2022 at 10:23 PM Zike Yang wrote: > > > > > +1 (non-binding) > > > Great work! > > > > > > Thanks > > > Zike Yang > > > > > > On Tue, Jul 26, 2022 at 7:31 PM Enrico Olivelli > > > wrote: > > > > > > > > +1 (binding) > > > > > > > > This is a small improvement but a great impact !! > > > > > > > > Enrico > > > > > > > > Il giorno mar 26 lug 2022 alle ore 12:48 Anon Hxy > > > > ha scritto: > > > > > > > > > > Hi Zike > > > > > > > > > > Thanks for your reminding. I just forgot to update the issue and > > have > > > > > updated it now. I am not going to add the configuration > > > > > `batchingSingleMessage ` to the producer. > > > > > > > > > > Thanks, > > > > > Xiaoyu Hou > > > > > > > > > > Zike Yang 于2022年7月26日周二 18:16写道： > > > > > > > > > > > Hi, Anon > > > > > > > > > > > > So are you going to add the configuration `batchingSingleMessage > ` > > to > > > > > > the producer? I saw that it was still in the PIP: > > > > > > > > > > > > > So this may cause ((BatchMessageIdImpl) messageId) throw > > > > > > ClassCastException. we need to add a switch for the producer to > > > enable or > > > > > > disable this feature > > > > > > > ProducerBuilder batchingSingleMessage(boolean > > > > > > batchingSingleMessage); // default value is true > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Zike Yang > > > > > > > > > > > > On Tue, Jul 26, 2022 at 6:05 PM Qiang Huang < > > > qiang.huang1...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > +1(non-binding) > > > > > > > > > > > > > > BR, > > > > > > > Qiang Huang > > > > > > > > > > > > > > mattison chao 于2022年7月25日周一 13:17写道： > > > > > > > > > > > > > > > +1(non-binding) > > > > > > > > > > > > > > > > Best, > > > > > > > > Mattison > > > > > > > > > > > > > > > > On Mon, 25 Jul 2022 at 10:35, Haiting Jiang < > > > jianghait...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > Thanks, > > > > > > > > > Haiting > > > > > > > > > > > > > > > > > > On 2022/07/25 02:23:20 Anon Hxy wrote: > > > > > > > > > > Dear Community, > > > > > > > > > > > > > > > > > > > > I would like to start a VOTE on "PIP-189: No batching if > > > only one > > > > > > > > message > > > > > > > > > > in batch." > > > > > > > > > > > > > > > > > > > > The proposal can be read at [0] and the discussion thread > > is > > > > > > available > > > > > > > > at > > > > > > > > > > [1] and the PR link is available at [2] > > > > > > > > > > > > > > > > > > > > Voting will stay open for at least 48h. > > > > > > > > > > > > > > > > > > > > [0] https://github.com/apache/pulsar/issues/16619 > > > > > > > > > > [1] > > > > > > https://lists.apache.org/thread/dbq1lrv03bhtk0lr5nwm5txo9ndjplv0 > > > > > > > > > > [2] https://github.com/apache/pulsar/pull/16605 > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Xiaoyu > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > BR, > > > > > > > Qiang Huang > > > > > > > > > > > >
Re: [DISCUSS] [PIP-201] Extensions mechanism for Pulsar Admin CLI tools
The motivation and goal LGTM, but the API changes look very simple and
hard to use. Do we have to implement all these interfaces for an admin
extension? If yes, could you show an example in the proposal as a
guidance?
For example, if I just want to implement a simple command:
```bash
./bin/pulsar-admin kafka create-topic --partitions
```
How should I implement these interfaces?
Thanks,
Yunze
> 2022年8月18日 16:23，Enrico Olivelli 写道：
>
> Hello,
> I have drafted a PIP around this proposal.
>
> PIP link: https://github.com/apache/pulsar/issues/17155
>
> I am preparing an official PR, I already have a working prototype.
>
> Copy of the contents of the GH issue is attached for discussion here
> on the Mailing list.
>
> Motivation
>
> There are many projects that are in the Pulsar ecosystem like Protocol
> Handlers (Kafka, MQTT, RabbitMQ) and libraries (JMS…) that need
> additional tools for operating Pulsar following specific conventions
> adopted by each project and to handle custom domain objects (like JMS
> queues, Kafka Consumer Groups...).
>
> Some examples:
>
> Kafka: tools for inspecting internal systems, SchemaRegistry,
> Transaction Manager, Consumers Groups
> JMS: tools to handling Subscriptions and Selectors
> RabbitMQ: tools to handle Pulsar topics and subscription following the
> convention
>
> This is very important as it is hard to follow the conventions of each
> project using pulsar-admin and the administrator may inadvertently
> break the system.
>
> This feature will enhance the UX of the Pulsar Admin CLI tools for the
> benefit of the whole ecosystem and users.
>
> Goal
>
> As we do for many other components in Pulsar, we need a way to enhance
> the CLI tools, pulsar-admin and pulsar-shell, with additional commands
> that are specific to the additional features.
>
> The proposal here is to add an extension mechanism to the pulsar-admin
> (and so pulsar-shell) tool.
> Following the usual conventions for extensions the extension will be
> bundled in a .nar file that may contain additional third party
> libraries.
>
> The extension will be able to provide new top level commands
>
> pulsar-admin my-command-group my-command arguments…
>
> The extension will be able to access the PulsarAdmin API provided by
> the environment.
>
> The extension must not depend directly on the JCommander library but
> we will provide an API to declare the parameters and the other
> metadata necessary to document and execute the command.
> This is very important because during the lifecycle of Pulsar the
> project may decide to upgrade JCommander to an incompatible version or
> to drop the dependency at all.
>
> API Changes
>
> We will introduce a new Maven module pulsar-tools-api that contains
> the public API that can be used by implementations of the custom
> commands.
>
> The implementation will be bundled in a .nar file with a descriptor
> with the following fields:
>
> factoryClass: x.CommandFactory
> name: extension-name
> description: Description...
>
> There are the new classes:
>
> /**
> Access to the environment
> */
> public interface CommandExecutionContext {
>PulsarAdmin getPulsarAdmin();
>Properties getConfiguration();
> }
>
>
> /**
> * Custom command implementation
> */
> public interface CustomCommand {
>String name();
>String description();
>List parameters();
>boolean execute(Map parameters,
> CommandExecutionContext context) throws Exception;
> }
>
> /**
> * A group of commands.
> */
> public interface CustomCommandGroup {
>String name();
>String description();
>List commands(CommandExecutionContext context);
> }
>
> /**
> * Main entry point of the extension
> */
> public interface CustomCommandFactory {
>
>/**
> * Generate the available command groups.
> */
>List commandGroups(CommandExecutionContext context);
> }
>
> @Builder
> @Getter
> public final class ParameterDescriptor {
>@Builder.Default
>private String name = "";
>@Builder.Default
>private String description = "";
>private ParameterType type = ParameterType.STRING;
>private boolean required = false;
> }
Re: Performance issue in Pulsar with large fan-outs since Pulsar 2.8.2
Thank you for starting this important discussion, Lari. It seems the issue is in the way the `heap` list in the ManagedCursorContainer class is ordered. The heap is ordered by mark delete position, while cache eviction is determined by the slowest read position [0]. The PR [1] essentially says that the cursor with the oldest mark delete position is also the one with the slowest read position. That assumption is not always true. We need to iterate over active cursors to find the slowest read position. We likely want to revert [1]. It enabled caching for non-durable cursors, which is probably helpful, but it also started tracking non durable cursors within the ManagedCursorContainer, which is unnecessary. > My assumption is also that there wouldn't have been a need to add > https://github.com/apache/pulsar/pull/14985 at all if the broker cache > would have worked as expected. With read position based cache eviction, a consumer requesting redelivery of an entry likely triggers a read to bookkeeper because the entry is eligible to be evicted when it is first sent to the consumer. With this PR [2] enabled, entries are cached until passed by the mark delete position, which could be helpful for certain workloads that expect negative acks and want the benefit of caching. Given that there are tradeoffs to this extra cache retention, I think it could make sense to keep the configuration option. However, I do agree that this feature would not work efficiently with high numbers of individually acknowledged messages. Thanks, Michael [0] https://github.com/apache/pulsar/blob/81593682aa94c6ac1e526216a0ec134d1d5c5481/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorContainer.java#L109 [1] https://github.com/apache/pulsar/pull/12045 [2] https://github.com/apache/pulsar/pull/14985 On Sun, Aug 21, 2022 at 5:47 PM Lari Hotari wrote: > > Hi all, > > I'd like to get some more eyes on this long outstanding performance issue > with large fan-outs (a large number of consumers for a single topic). The > broker cache does not work as expected due to invalid changes introduced in > version 2.8.2 by PR https://github.com/apache/pulsar/pull/12045. > > I have reported the issue with broker cache over 2 months ago with > https://github.com/apache/pulsar/issues/16054 . Please take a look. > > The workaround seems to be to use the feature introduced in PR > https://github.com/apache/pulsar/pull/14985, "Add a cache eviction > policy：Evicting cache data by the slowest markDeletedPosition", > by setting cacheEvictionByMarkDeletedPosition=true . > One notable detail is that the cacheEvictionByMarkDeletedPosition entry is > not included in conf/broker.conf or conf/standalone.conf files. Therefore > in the Pulsar Helm chart, it is necessary to use > "PULSAR_PREFIX_cacheEvictionByMarkDeletedPosition: true" to activate the > setting. > > However, the problem with the cacheEvictionByMarkDeletedPosition solution > is that the cache could be filled with irrelevant entries when there's a > lot of topics and consumers on a single broker since the cache expiration > is based on markDeletedPosition and not the farthest behind read position > for active consumers. > My assumption is also that there wouldn't have been a need to add > https://github.com/apache/pulsar/pull/14985 at all if the broker cache > would have worked as expected. > > PIP-174: Provide new implementation for broker dispatch cache ( > https://github.com/apache/pulsar/issues/15954 / > https://github.com/apache/pulsar/pull/15955) will provide a new broker > cache implementation where this issue hopefully goes away, but we will > regardless need to make the fix for current maintenance versions. > > I'm looking forward to feedback of original contributors of the broker > cache and the related changes so that we finally get this issue resolved. > > Thanks, > > Lari
