Re: [VOTE] PIP-239: Retry Mechanism per Message title
+1 (non-binding) Thanks, Zike Yang On Sun, Jan 8, 2023 at 3:22 PM Haiting Jiang wrote: > > +1 binding > > Although I think we should use more unique keys for system properties in > pulsar, > e.g. reserve all properties prefixed with "PULSAR_" for system internal usage. > But it's another topic as we already have "RECONSUMETIMES". > > > On Sun, Jan 8, 2023 at 12:27 PM Nitin Goyal wrote: > > > > Hello community, > > > > I'm starting the VOTE for PIP-239: Retry Mechanism per Message title. > > > > Motivation: We are working on a project where each message has their retry > > as per the requirements. like one message 100 times and other messages 5 > > times and so on. This feature also adds an extra functionality while > > comparing existing queueing services like RabbitMQ or SQS etc. > > > > For more details please find the detailed PIP at: > > https://github.com/apache/pulsar/issues/19136 > > > > Discussion Threads: > > 1. https://lists.apache.org/thread/rn6114xxtx1j57yy2jbmdv4xzt80o1hz > > 2. https://lists.apache.org/thread/b9rfv6t307z439xx3zt2ym4p140qzp06 > > > > Proposed changes in pulsar go client library. > > https://github.com/apache/pulsar-client-go/pull/939 > > > > Thanks > > Nitin Goyal
Re: [VOTE] PIP-239: Retry Mechanism per Message title
+1 (binding) Thanks, Yunze On Sun, Jan 8, 2023 at 4:46 PM Zike Yang wrote: > > +1 (non-binding) > > Thanks, > Zike Yang > > On Sun, Jan 8, 2023 at 3:22 PM Haiting Jiang wrote: > > > > +1 binding > > > > Although I think we should use more unique keys for system properties in > > pulsar, > > e.g. reserve all properties prefixed with "PULSAR_" for system internal > > usage. > > But it's another topic as we already have "RECONSUMETIMES". > > > > > > On Sun, Jan 8, 2023 at 12:27 PM Nitin Goyal > > wrote: > > > > > > Hello community, > > > > > > I'm starting the VOTE for PIP-239: Retry Mechanism per Message title. > > > > > > Motivation: We are working on a project where each message has their retry > > > as per the requirements. like one message 100 times and other messages 5 > > > times and so on. This feature also adds an extra functionality while > > > comparing existing queueing services like RabbitMQ or SQS etc. > > > > > > For more details please find the detailed PIP at: > > > https://github.com/apache/pulsar/issues/19136 > > > > > > Discussion Threads: > > > 1. https://lists.apache.org/thread/rn6114xxtx1j57yy2jbmdv4xzt80o1hz > > > 2. https://lists.apache.org/thread/b9rfv6t307z439xx3zt2ym4p140qzp06 > > > > > > Proposed changes in pulsar go client library. > > > https://github.com/apache/pulsar-client-go/pull/939 > > > > > > Thanks > > > Nitin Goyal
Re: [VOTE] PIP-239: Retry Mechanism per Message title
+1 (non-binding) Thanks, Zixuan Yunze Xu 于2023年1月8日周日 22:38写道： > +1 (binding) > > Thanks, > Yunze > > On Sun, Jan 8, 2023 at 4:46 PM Zike Yang wrote: > > > > +1 (non-binding) > > > > Thanks, > > Zike Yang > > > > On Sun, Jan 8, 2023 at 3:22 PM Haiting Jiang > wrote: > > > > > > +1 binding > > > > > > Although I think we should use more unique keys for system properties > in pulsar, > > > e.g. reserve all properties prefixed with "PULSAR_" for system > internal usage. > > > But it's another topic as we already have "RECONSUMETIMES". > > > > > > > > > On Sun, Jan 8, 2023 at 12:27 PM Nitin Goyal > wrote: > > > > > > > > Hello community, > > > > > > > > I'm starting the VOTE for PIP-239: Retry Mechanism per Message title. > > > > > > > > Motivation: We are working on a project where each message has their > retry > > > > as per the requirements. like one message 100 times and other > messages 5 > > > > times and so on. This feature also adds an extra functionality while > > > > comparing existing queueing services like RabbitMQ or SQS etc. > > > > > > > > For more details please find the detailed PIP at: > > > > https://github.com/apache/pulsar/issues/19136 > > > > > > > > Discussion Threads: > > > > 1. https://lists.apache.org/thread/rn6114xxtx1j57yy2jbmdv4xzt80o1hz > > > > 2. https://lists.apache.org/thread/b9rfv6t307z439xx3zt2ym4p140qzp06 > > > > > > > > Proposed changes in pulsar go client library. > > > > https://github.com/apache/pulsar-client-go/pull/939 > > > > > > > > Thanks > > > > Nitin Goyal >
Re: [DISCUSS] PIP-236: Upload AUTO_CONSUME SchemaType to Broker
Sorry for the late reply.
We can leverage the `ProtocolVersion` [1] to handle the compatibility issue.
It looks like only if the protocol_version >= 21, subscribe with the
auto_consume schema
IMO, both the new key-value of the subscribe command, and a specific
representative are
API changes. It's just that some have modified the definition of the API,
and some have modified the behavior of the API
I prefer the intuitive way. And from the perspective of API-based
developers, we should
try to provide a simple and clear API with no hidden rules. The client just
uploads the schema
that it has except the byte[] schema. The broker knows how to handle the
different schemas,
such as AUTO_PRODUCE, and AUTO_CONSUME. And this should not be the burden
of the
client developer to learn the details of the schema implementation. They
should work according
to the API spec.
If we can resolve the compatibility issue with uploading the AUTO_CONSUME
schema with
subscribe command, do you see any apparent cons?
Best,
Penghui
[1]
https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/proto/PulsarApi.proto#L241-L266
On Fri, Jan 6, 2023 at 10:31 PM SiNan Liu wrote:
> Ok, I will update the PIP issue later.
>
> About my current design.
> When the consumer with AUTO_CONSUME schema subscribed to an "empty" topic,
> the schemaInfo will be null.
> ```
> public SchemaInfo getSchemaInfo(byte[] schemaVersion) {
> SchemaVersion sv = getSchemaVersion(schemaVersion);
> if (schemaMap.containsKey(sv)) {
> return schemaMap.get(sv).getSchemaInfo();
> }
> return null;
>
> }
>
> ```
> And checkSchemaCompatibility must be set in
> `org.apache.pulsar.common.protocol.Commands#newSubscribe`,
> and we need to know that this is an AUTO_CONSUME consumer subscribing. So
> we should set a "*default*" schemaInfo(schemaType = AUTO_CONSUME) for
> AutoConsumeSchema,
> this is because schemaInfo is also null when `si.getType` is
> SchemaType.BYTES or SchemaType.NONE.
> And checkSchemaCompatibility can be set in
> `org.apache.pulsar.common.protocol.Commands#newSubscribe`. The most
> important thing is clearSchema, which does not carry the wrong schema to
> the broker.
>
>
> Yunze Xu 于2023年1月6日周五 12:57写道：
>
> > You only need to describe what's added to the PulsarApi.proto, i.e.
> > you don't need to paste all definitions of `CommandSubscribe` in the
> > proposal. Take PIP-54 [1] for example, it only pastes the new field
> > `ack_set` and does not paste the whole `MessageIdData` definition.
> >
> > The implementations section involves too much code and just looks like
> > an actual PR. Take PIP-194 [2] for example, you should only talk about
> > the implementations from a high level.
> >
> > Let's talk back to your current design, when the schema type is
> > AUTO_CONSUME, you clear the schema in CommandSubscribe. It seems that
> > adding a SchemaInfo to the AutoConsumeSchema is meaningless.
> >
> > [1]
> >
> https://github.com/apache/pulsar/wiki/PIP-54:-Support-acknowledgment-at-batch-index-level#changes
> > [2] https://github.com/apache/pulsar/issues/16757
> >
> > On Fri, Jan 6, 2023 at 12:17 AM SiNan Liu
> wrote:
> > >
> > > I just updated the PIP issue and title, you guys can have a look.
> > issue19113
> > >
> > > I added `check_schema_compatibility` in CommandSubscribe, and I also
> made
> > > many other changes.
> > >
> > > Yunze Xu 于2023年1月5日周四 14:33写道：
> > >
> > > > It's not related to the schema itself. When an AUTO_CONSUME consumer
> > > > subscribes to a topic, the option tells the broker that it's an
> > > > AUTO_CONSUME consumer so that the broker should not treat it as an
> > > > active consumer when performing schema compatibility check. If there
> > > > is a consumer that also wants to ignore the schema compatibility
> check
> > > > in future, this option can be reused.
> > > >
> > > > The other important reason is the breaking change by carrying the
> > > > schema info on an AUTO_CONSUMER consumer. (See my explanations in
> > > > GitHub and the mail list) If the consumer serves an old version
> > > > consumer, the schema could be uploaded into the registry and other
> > > > clients would be affected. So we should keep not carrying the schema
> > > > info in CommandSubscribe for an AUTO_CONSUMER consumer.
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Thu, Jan 5, 2023 at 11:55 AM SiNan Liu
> > wrote:
> > > > >
> > > > > I have modified pip issue and title last night. Yunze. You mean
> that
> > in
> > > > > PulsarApi.proto, take `optional bool is_auto_consume_schema = 6
> > [default
> > > > =
> > > > > false]; ` in CommandSubscribe instead of Schema? But shouldn't
> > > > > schema-related stuff be in Schema?
> > > > >
> > > > > Thanks,
> > > > > Sinan
> > > > >
> > > > > Yunze Xu 于 2023年1月5日周四 上午12:31写道：
> > > > >
> > > > > > I found a similar compatibility problem with my closed PR. We
> > should
> > > > > > not set the `Schema` field for AUTO
Re: [DISCUSS] Introduce oshi library to sensory OS resources
+1 Thanks, Ran Gao On 2022/12/21 09:38:01 Zike Yang wrote: > +1 > > Thanks, > Zike Yang > > On Wed, Dec 21, 2022 at 5:06 PM Yunze Xu wrote: > > > > +1 > > > > Thanks, > > Yunze > > > > On Wed, Dec 21, 2022 at 5:02 PM houxiaoyu wrote: > > > > > > +1 > > > > > > Yubiao Feng 于2022年12月21日周三 11:59写道： > > > > > > > +1 > > > > > > > > On Mon, Dec 19, 2022 at 6:19 PM wrote: > > > > > > > > > > > > > > Hi, All > > > > > > > > > > I would like to introduce a new library oshi[1] to help Apache Pulsar > > > > > sensory OS resources. It can help us to get away from the complex file > > > > > manipulation and cross-platform compatibility issues in some operating > > > > > systems. > > > > > > > > > > code example: https://github.com/apache/pulsar/pull/18984 > > > > > > > > > > Please feel free to left comments. > > > > > > > > > > > > > > > Best, > > > > > Mattison > > > > > > > > > > > > > > > [1] https://github.com/oshi/oshi > > > > > > > > > >
Re: [DISCUSS] PIP-236: Upload AUTO_CONSUME SchemaType to Broker
We should not use a negative enum number in PulsarApi.proto. It's
unnatural. If we decide to carry the AUTO_CONSUME schema in a
CommandSubscribe, it should not be treated as a negative schema type.
Thanks,
Yunze
On Mon, Jan 9, 2023 at 9:51 AM PengHui Li wrote:
>
> Sorry for the late reply.
>
> We can leverage the `ProtocolVersion` [1] to handle the compatibility issue.
> It looks like only if the protocol_version >= 21, subscribe with the
> auto_consume schema
>
> IMO, both the new key-value of the subscribe command, and a specific
> representative are
> API changes. It's just that some have modified the definition of the API,
> and some have modified the behavior of the API
>
> I prefer the intuitive way. And from the perspective of API-based
> developers, we should
> try to provide a simple and clear API with no hidden rules. The client just
> uploads the schema
> that it has except the byte[] schema. The broker knows how to handle the
> different schemas,
> such as AUTO_PRODUCE, and AUTO_CONSUME. And this should not be the burden
> of the
> client developer to learn the details of the schema implementation. They
> should work according
> to the API spec.
>
> If we can resolve the compatibility issue with uploading the AUTO_CONSUME
> schema with
> subscribe command, do you see any apparent cons?
>
> Best,
> Penghui
>
> [1]
> https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/proto/PulsarApi.proto#L241-L266
>
> On Fri, Jan 6, 2023 at 10:31 PM SiNan Liu wrote:
>
> > Ok, I will update the PIP issue later.
> >
> > About my current design.
> > When the consumer with AUTO_CONSUME schema subscribed to an "empty" topic,
> > the schemaInfo will be null.
> > ```
> > public SchemaInfo getSchemaInfo(byte[] schemaVersion) {
> > SchemaVersion sv = getSchemaVersion(schemaVersion);
> > if (schemaMap.containsKey(sv)) {
> > return schemaMap.get(sv).getSchemaInfo();
> > }
> > return null;
> >
> > }
> >
> > ```
> > And checkSchemaCompatibility must be set in
> > `org.apache.pulsar.common.protocol.Commands#newSubscribe`,
> > and we need to know that this is an AUTO_CONSUME consumer subscribing. So
> > we should set a "*default*" schemaInfo(schemaType = AUTO_CONSUME) for
> > AutoConsumeSchema,
> > this is because schemaInfo is also null when `si.getType` is
> > SchemaType.BYTES or SchemaType.NONE.
> > And checkSchemaCompatibility can be set in
> > `org.apache.pulsar.common.protocol.Commands#newSubscribe`. The most
> > important thing is clearSchema, which does not carry the wrong schema to
> > the broker.
> >
> >
> > Yunze Xu 于2023年1月6日周五 12:57写道：
> >
> > > You only need to describe what's added to the PulsarApi.proto, i.e.
> > > you don't need to paste all definitions of `CommandSubscribe` in the
> > > proposal. Take PIP-54 [1] for example, it only pastes the new field
> > > `ack_set` and does not paste the whole `MessageIdData` definition.
> > >
> > > The implementations section involves too much code and just looks like
> > > an actual PR. Take PIP-194 [2] for example, you should only talk about
> > > the implementations from a high level.
> > >
> > > Let's talk back to your current design, when the schema type is
> > > AUTO_CONSUME, you clear the schema in CommandSubscribe. It seems that
> > > adding a SchemaInfo to the AutoConsumeSchema is meaningless.
> > >
> > > [1]
> > >
> > https://github.com/apache/pulsar/wiki/PIP-54:-Support-acknowledgment-at-batch-index-level#changes
> > > [2] https://github.com/apache/pulsar/issues/16757
> > >
> > > On Fri, Jan 6, 2023 at 12:17 AM SiNan Liu
> > wrote:
> > > >
> > > > I just updated the PIP issue and title, you guys can have a look.
> > > issue19113
> > > >
> > > > I added `check_schema_compatibility` in CommandSubscribe, and I also
> > made
> > > > many other changes.
> > > >
> > > > Yunze Xu 于2023年1月5日周四 14:33写道：
> > > >
> > > > > It's not related to the schema itself. When an AUTO_CONSUME consumer
> > > > > subscribes to a topic, the option tells the broker that it's an
> > > > > AUTO_CONSUME consumer so that the broker should not treat it as an
> > > > > active consumer when performing schema compatibility check. If there
> > > > > is a consumer that also wants to ignore the schema compatibility
> > check
> > > > > in future, this option can be reused.
> > > > >
> > > > > The other important reason is the breaking change by carrying the
> > > > > schema info on an AUTO_CONSUMER consumer. (See my explanations in
> > > > > GitHub and the mail list) If the consumer serves an old version
> > > > > consumer, the schema could be uploaded into the registry and other
> > > > > clients would be affected. So we should keep not carrying the schema
> > > > > info in CommandSubscribe for an AUTO_CONSUMER consumer.
> > > > >
> > > > > Thanks,
> > > > > Yunze
> > > > >
> > > > > On Thu, Jan 5, 2023 at 11:55 AM SiNan Liu
> > > wrote:
> > > > > >
> > > > > > I have modified pip issue and title last
Re: [VOTE] PIP-239: Retry Mechanism per Message title
+1 (non-binding) Thanks, Ran Gao On 2023/01/08 15:05:31 Zixuan Liu wrote: > +1 (non-binding) > > Thanks, > Zixuan > > Yunze Xu 于2023年1月8日周日 22:38写道： > > > +1 (binding) > > > > Thanks, > > Yunze > > > > On Sun, Jan 8, 2023 at 4:46 PM Zike Yang wrote: > > > > > > +1 (non-binding) > > > > > > Thanks, > > > Zike Yang > > > > > > On Sun, Jan 8, 2023 at 3:22 PM Haiting Jiang > > wrote: > > > > > > > > +1 binding > > > > > > > > Although I think we should use more unique keys for system properties > > in pulsar, > > > > e.g. reserve all properties prefixed with "PULSAR_" for system > > internal usage. > > > > But it's another topic as we already have "RECONSUMETIMES". > > > > > > > > > > > > On Sun, Jan 8, 2023 at 12:27 PM Nitin Goyal > > wrote: > > > > > > > > > > Hello community, > > > > > > > > > > I'm starting the VOTE for PIP-239: Retry Mechanism per Message title. > > > > > > > > > > Motivation: We are working on a project where each message has their > > retry > > > > > as per the requirements. like one message 100 times and other > > messages 5 > > > > > times and so on. This feature also adds an extra functionality while > > > > > comparing existing queueing services like RabbitMQ or SQS etc. > > > > > > > > > > For more details please find the detailed PIP at: > > > > > https://github.com/apache/pulsar/issues/19136 > > > > > > > > > > Discussion Threads: > > > > > 1. https://lists.apache.org/thread/rn6114xxtx1j57yy2jbmdv4xzt80o1hz > > > > > 2. https://lists.apache.org/thread/b9rfv6t307z439xx3zt2ym4p140qzp06 > > > > > > > > > > Proposed changes in pulsar go client library. > > > > > https://github.com/apache/pulsar-client-go/pull/939 > > > > > > > > > > Thanks > > > > > Nitin Goyal > > >
Re: [DISCUSS] Introduce oshi library to sensory OS resources
+1 Thanks, Cong On 2022/12/19 10:19:07 mattisonc...@gmail.com wrote: > > Hi, All > > I would like to introduce a new library oshi[1] to help Apache Pulsar sensory > OS resources. It can help us to get away from the complex file manipulation > and cross-platform compatibility issues in some operating systems. > > code example: https://github.com/apache/pulsar/pull/18984 > > Please feel free to left comments. > > > Best, > Mattison > > > [1] https://github.com/oshi/oshi >
Re: [DISCUSS] PIP-236: Upload AUTO_CONSUME SchemaType to Broker
This design also has serious compatibility problems between old and new
pulsar clients and new and old brokers.
Thanks,
Sinan
PengHui Li 于 2023年1月9日周一 上午9:51写道：
> Sorry for the late reply.
>
> We can leverage the `ProtocolVersion` [1] to handle the compatibility
> issue.
> It looks like only if the protocol_version >= 21, subscribe with the
> auto_consume schema
>
> IMO, both the new key-value of the subscribe command, and a specific
> representative are
> API changes. It's just that some have modified the definition of the API,
> and some have modified the behavior of the API
>
> I prefer the intuitive way. And from the perspective of API-based
> developers, we should
> try to provide a simple and clear API with no hidden rules. The client just
> uploads the schema
> that it has except the byte[] schema. The broker knows how to handle the
> different schemas,
> such as AUTO_PRODUCE, and AUTO_CONSUME. And this should not be the burden
> of the
> client developer to learn the details of the schema implementation. They
> should work according
> to the API spec.
>
> If we can resolve the compatibility issue with uploading the AUTO_CONSUME
> schema with
> subscribe command, do you see any apparent cons?
>
> Best,
> Penghui
>
> [1]
>
> https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/proto/PulsarApi.proto#L241-L266
>
> On Fri, Jan 6, 2023 at 10:31 PM SiNan Liu wrote:
>
> > Ok, I will update the PIP issue later.
> >
> > About my current design.
> > When the consumer with AUTO_CONSUME schema subscribed to an "empty"
> topic,
> > the schemaInfo will be null.
> > ```
> > public SchemaInfo getSchemaInfo(byte[] schemaVersion) {
> > SchemaVersion sv = getSchemaVersion(schemaVersion);
> > if (schemaMap.containsKey(sv)) {
> > return schemaMap.get(sv).getSchemaInfo();
> > }
> > return null;
> >
> > }
> >
> > ```
> > And checkSchemaCompatibility must be set in
> > `org.apache.pulsar.common.protocol.Commands#newSubscribe`,
> > and we need to know that this is an AUTO_CONSUME consumer subscribing. So
> > we should set a "*default*" schemaInfo(schemaType = AUTO_CONSUME) for
> > AutoConsumeSchema,
> > this is because schemaInfo is also null when `si.getType` is
> > SchemaType.BYTES or SchemaType.NONE.
> > And checkSchemaCompatibility can be set in
> > `org.apache.pulsar.common.protocol.Commands#newSubscribe`. The most
> > important thing is clearSchema, which does not carry the wrong schema to
> > the broker.
> >
> >
> > Yunze Xu 于2023年1月6日周五 12:57写道：
> >
> > > You only need to describe what's added to the PulsarApi.proto, i.e.
> > > you don't need to paste all definitions of `CommandSubscribe` in the
> > > proposal. Take PIP-54 [1] for example, it only pastes the new field
> > > `ack_set` and does not paste the whole `MessageIdData` definition.
> > >
> > > The implementations section involves too much code and just looks like
> > > an actual PR. Take PIP-194 [2] for example, you should only talk about
> > > the implementations from a high level.
> > >
> > > Let's talk back to your current design, when the schema type is
> > > AUTO_CONSUME, you clear the schema in CommandSubscribe. It seems that
> > > adding a SchemaInfo to the AutoConsumeSchema is meaningless.
> > >
> > > [1]
> > >
> >
> https://github.com/apache/pulsar/wiki/PIP-54:-Support-acknowledgment-at-batch-index-level#changes
> > > [2] https://github.com/apache/pulsar/issues/16757
> > >
> > > On Fri, Jan 6, 2023 at 12:17 AM SiNan Liu
> > wrote:
> > > >
> > > > I just updated the PIP issue and title, you guys can have a look.
> > > issue19113
> > > >
> > > > I added `check_schema_compatibility` in CommandSubscribe, and I also
> > made
> > > > many other changes.
> > > >
> > > > Yunze Xu 于2023年1月5日周四 14:33写道：
> > > >
> > > > > It's not related to the schema itself. When an AUTO_CONSUME
> consumer
> > > > > subscribes to a topic, the option tells the broker that it's an
> > > > > AUTO_CONSUME consumer so that the broker should not treat it as an
> > > > > active consumer when performing schema compatibility check. If
> there
> > > > > is a consumer that also wants to ignore the schema compatibility
> > check
> > > > > in future, this option can be reused.
> > > > >
> > > > > The other important reason is the breaking change by carrying the
> > > > > schema info on an AUTO_CONSUMER consumer. (See my explanations in
> > > > > GitHub and the mail list) If the consumer serves an old version
> > > > > consumer, the schema could be uploaded into the registry and other
> > > > > clients would be affected. So we should keep not carrying the
> schema
> > > > > info in CommandSubscribe for an AUTO_CONSUMER consumer.
> > > > >
> > > > > Thanks,
> > > > > Yunze
> > > > >
> > > > > On Thu, Jan 5, 2023 at 11:55 AM SiNan Liu
> > > wrote:
> > > > > >
> > > > > > I have modified pip issue and title last night. Yunze. You mean
> > that
> > > in
> > > > > > PulsarApi.proto, take `optional
Re: [VOTE] Pulsar Release 2.11.0 Candidate-5
+1 (binding) - Checked the signature - Build from source - Checked license(server and shell) - Start standalone with zookeeper - Checked function - Checked Cassandra connector - Checked stateful function Regards Jiwei Guo (Tboy) On Wed, Jan 4, 2023 at 8:25 PM Nicolò Boschi wrote: > About Pulsar Shell, > 1) zip file is distributed as a commodity for Windows users since the > .tar.gz archive is more tricky to extract. If this is a problem, we can > discuss it in another thread > 2) The bug on Windows is not a blocker for the release since it's not a > regression. We could fix it in 2.11.1 > > Here's my vote: > > +1 non binding > > - Signature and checksum > - LICENSE on server and shell > > - Rat check passes > > - Compile from source with JDK 17 on MacOs Intel > > - Run Pulsar standalone and produce-consume from CLI > > - Tested K8S installation with Datastax Pulsar helm chart and verified TLS, > offloads, transactions and ElasticSearch sink > > - Pulsar Shell commands > > > Thanks, > Nicolò Boschi > > > Il giorno mer 4 gen 2023 alle ore 12:33 丛搏 ha scritto: > > > +1 (non-binding) > > > > system: mac os 12.6, Apple M1 > > maven: 3.8.5 > > java: OpenJDK 17.0.3 > > > > - Checked the signature > > - Checked LICENSE > > - Start standalone with zookeeper stream storage > > - Publish and consume messages > > - Verified Function and State Function > > - Verified Cassandra connector > > - Build from the source package > > - Run a simple transaction performance check > > > > Thanks, > > Bo > > > > Yunze Xu 于2023年1月4日周三 16:41写道： > > > > > > Okay, I will give my +1 (binding). > > > > > > Thanks, > > > Yunze > > > > > > On Wed, Jan 4, 2023 at 4:00 PM PengHui Li wrote: > > > > > > > > I think it's ok. > > > > 2.11.0 is the first release version of pulsar-shell > > > > So it's not a regression that was introduced in 2.11.0 and not > > > > critical security issues or license issues which will block users > move > > > > to the new version. > > > > > > > > Thanks, > > > > Penghui > > > > > > > > > > > > > > > > On Wed, Jan 4, 2023 at 3:36 PM Yunze Xu > > > > > wrote: > > > > > > > > > +0 (binding) > > > > > > > > > > - Checked the signature > > > > > - Build from source (Java 17, Ubuntu 20.04 WSL2) > > > > > - Start standalone with KoP branch-2.11 (ab9d761f) > > > > > - Verify Pulsar client 2.11.0, Kafka client 3.3.1 (through KoP) > > > > > - Verify pulsar-shell on Ubuntu 20.04 WSL2 and Windows: > > > > > 1. Use `client` command to produce and consume messages > > > > > 2. Use `admin` command to create, list and delete topics > > > > > > > > > > I didn't give +1 because I have to modify the > > > > > `bin/pulsar-admin-common.cmd` to make `pulsar-shell` work on > Windows. > > > > > I left a command here: > > > > > https://github.com/apache/pulsar/pull/17243#discussion_r1061199501 > . > > > > > Not sure if it's a blocker so I didn't give +1 or -1 until someone > > > > > could explain it. > > > > > > > > > > Thanks, > > > > > Yunze > > > > > > > > > > On Wed, Jan 4, 2023 at 2:37 PM PengHui Li > > wrote: > > > > > > > > > > > > > I found there are both apache-pulsar-shell-2.11.0-bin.tar.gz > and > > > > > > apache-pulsar-shell-2.11.0-bin.zip, are they duplicates? > > > > > > > > > > > > I noticed here is the discussion about this part in Slack > > > > > > > > https://apache-pulsar.slack.com/archives/C5ZSVEN4E/p1672411025574369 > > > > > > > > > > > > Penghui > > > > > > > > > > > > On Wed, Jan 4, 2023 at 2:35 PM Yunze Xu > > > > > > > > wrote: > > > > > > > > > > > > > I found there are both apache-pulsar-shell-2.11.0-bin.tar.gz > and > > > > > > > apache-pulsar-shell-2.11.0-bin.zip, are they duplicates? > > > > > > > > > > > > > > Thanks, > > > > > > > Yunze > > > > > > > > > > > > > > On Mon, Jan 2, 2023 at 8:01 PM guo jiwei > > > wrote: > > > > > > > > > > > > > > > > This is the fifth release candidate for Apache Pulsar, > version > > > > > 2.11.0. > > > > > > > > > > > > > > > > This release contains 1617 commits by 61 contributors. > > > > > > > > > > > > > > > https://github.com/apache/pulsar/compare/v2.10.2...v2.11.0-candidate-5 > > > > > > > > > > > > > > > > CI for this release candidate > > > > > > > > https://github.com/Technoboy-/pulsar/pull/26 > > > > > > > > > > > > > > > > *** Please download, test and vote on this release. This vote > > will > > > > > stay > > > > > > > open > > > > > > > > for at least 72 hours *** > > > > > > > > > > > > > > > > Note that we are voting upon the source (tag), binaries are > > provided > > > > > for > > > > > > > > convenience. > > > > > > > > > > > > > > > > Source and binary files: > > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.11.0-candidate-5 > > > > > > > > > > > > > > > > SHA-512 checksums: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 5bf77f600ac23c7ec72696655a49801e0755d67fb4c59c7ab30e9449b061aacef61ca227f106e6b7f5f490fc2ac57412ac2af3ff064ba1e6a31c162d4ca2ba6d > > > > > > > > > > > > > > > > ./apache-p
Re: [DISCUSS] Introduce oshi library to sensory OS resources
Please create a plan about how to leverage it. The PR only adds the dependency Enrico Il giorno lun 9 gen 2023 alle ore 04:26 Cong Zhao ha scritto: > > +1 > > Thanks, > Cong > > On 2022/12/19 10:19:07 mattisonc...@gmail.com wrote: > > > > Hi, All > > > > I would like to introduce a new library oshi[1] to help Apache Pulsar > > sensory OS resources. It can help us to get away from the complex file > > manipulation and cross-platform compatibility issues in some operating > > systems. > > > > code example: https://github.com/apache/pulsar/pull/18984 > > > > Please feel free to left comments. > > > > > > Best, > > Mattison > > > > > > [1] https://github.com/oshi/oshi > >
Re: [VOTE] PIP-239: Retry Mechanism per Message title
I am sure that this is a good idea. -1 (binding) I will follow up on the discussion thread I am sorry I am catching up late on this thread Enrico Il giorno lun 9 gen 2023 alle ore 03:51 Ran Gao ha scritto: > > +1 (non-binding) > > Thanks, > Ran Gao > > On 2023/01/08 15:05:31 Zixuan Liu wrote: > > +1 (non-binding) > > > > Thanks, > > Zixuan > > > > Yunze Xu 于2023年1月8日周日 22:38写道： > > > > > +1 (binding) > > > > > > Thanks, > > > Yunze > > > > > > On Sun, Jan 8, 2023 at 4:46 PM Zike Yang wrote: > > > > > > > > +1 (non-binding) > > > > > > > > Thanks, > > > > Zike Yang > > > > > > > > On Sun, Jan 8, 2023 at 3:22 PM Haiting Jiang > > > wrote: > > > > > > > > > > +1 binding > > > > > > > > > > Although I think we should use more unique keys for system properties > > > in pulsar, > > > > > e.g. reserve all properties prefixed with "PULSAR_" for system > > > internal usage. > > > > > But it's another topic as we already have "RECONSUMETIMES". > > > > > > > > > > > > > > > On Sun, Jan 8, 2023 at 12:27 PM Nitin Goyal > > > > > > > > wrote: > > > > > > > > > > > > Hello community, > > > > > > > > > > > > I'm starting the VOTE for PIP-239: Retry Mechanism per Message > > > > > > title. > > > > > > > > > > > > Motivation: We are working on a project where each message has their > > > retry > > > > > > as per the requirements. like one message 100 times and other > > > messages 5 > > > > > > times and so on. This feature also adds an extra functionality while > > > > > > comparing existing queueing services like RabbitMQ or SQS etc. > > > > > > > > > > > > For more details please find the detailed PIP at: > > > > > > https://github.com/apache/pulsar/issues/19136 > > > > > > > > > > > > Discussion Threads: > > > > > > 1. https://lists.apache.org/thread/rn6114xxtx1j57yy2jbmdv4xzt80o1hz > > > > > > 2. https://lists.apache.org/thread/b9rfv6t307z439xx3zt2ym4p140qzp06 > > > > > > > > > > > > Proposed changes in pulsar go client library. > > > > > > https://github.com/apache/pulsar-client-go/pull/939 > > > > > > > > > > > > Thanks > > > > > > Nitin Goyal > > > > >
Re: [DISCUSS] PIP-239: Retry Mechanism per Message
I don't think that this is a good idea. Because IIUC we want to add a property per message that sets the maximum time of retries. Unfortunately in a real system sometimes you have to change the number of retries temporarily, for instance in case of major system failure you don't want to lose all your messages. If we allow you to set a property on a message then you won't be able to change it because the message is immutable. TTL (time to live) is a similar concept but it is related to the concept of "physical time", if you have message that represents a task to be executed within a given deadline then it makes sense to state it in the message metadata. But the "number of retries" depends on how you deal with the retries: - how much time do you wait ? - how often do you have a temporary failure to retry ? It would make more sense to have a QOS (quality of service) attribute on the message, like "important/"non-important"/"foo"/"bar" and have a way for the brokers and the clients to handle that. I am pretty sure that with interceptors you can already do something. I am against hard coding the behaviour described in the PIP (and I voted -1 in the VOTE thread) Enrico Il giorno ven 6 gen 2023 alle ore 09:11 Zike Yang ha scritto: > > Hi, > > This looks good to me. > +1 > > I was thinking if we could add a new API for `reconsumeLater` to let > users set the max retry times easily. But I saw that there are too > many reconsumeLater API and this will make the consumer more complex. > > Thanks, > Zike Yang > > > On Thu, Jan 5, 2023 at 3:58 PM Zixuan Liu wrote: > > > > +1 > > > > Thanks, > > Zixuan > > > > Nitin Goyal 于2023年1月5日周四 13:50写道： > > > > > Hi all, > > > > > > I made a PIP to discuss: https://github.com/apache/pulsar/issues/19136 > > > > > > Thanks, > > > Nitin Goyal > > >
Re: [VOTE] Pulsar Release 2.11.0 Candidate-5
+1 (binding) - checked signatures/digests/rat - built from source - run smoke tests on the built binaries Thanks Enrico Il giorno lun 9 gen 2023 alle ore 06:18 guo jiwei ha scritto: > > +1 (binding) > > - Checked the signature > - Build from source > - Checked license(server and shell) > - Start standalone with zookeeper > - Checked function > - Checked Cassandra connector > - Checked stateful function > > > Regards > Jiwei Guo (Tboy) > > > On Wed, Jan 4, 2023 at 8:25 PM Nicolò Boschi wrote: > > > About Pulsar Shell, > > 1) zip file is distributed as a commodity for Windows users since the > > .tar.gz archive is more tricky to extract. If this is a problem, we can > > discuss it in another thread > > 2) The bug on Windows is not a blocker for the release since it's not a > > regression. We could fix it in 2.11.1 > > > > Here's my vote: > > > > +1 non binding > > > > - Signature and checksum > > - LICENSE on server and shell > > > > - Rat check passes > > > > - Compile from source with JDK 17 on MacOs Intel > > > > - Run Pulsar standalone and produce-consume from CLI > > > > - Tested K8S installation with Datastax Pulsar helm chart and verified TLS, > > offloads, transactions and ElasticSearch sink > > > > - Pulsar Shell commands > > > > > > Thanks, > > Nicolò Boschi > > > > > > Il giorno mer 4 gen 2023 alle ore 12:33 丛搏 ha scritto: > > > > > +1 (non-binding) > > > > > > system: mac os 12.6, Apple M1 > > > maven: 3.8.5 > > > java: OpenJDK 17.0.3 > > > > > > - Checked the signature > > > - Checked LICENSE > > > - Start standalone with zookeeper stream storage > > > - Publish and consume messages > > > - Verified Function and State Function > > > - Verified Cassandra connector > > > - Build from the source package > > > - Run a simple transaction performance check > > > > > > Thanks, > > > Bo > > > > > > Yunze Xu 于2023年1月4日周三 16:41写道： > > > > > > > > Okay, I will give my +1 (binding). > > > > > > > > Thanks, > > > > Yunze > > > > > > > > On Wed, Jan 4, 2023 at 4:00 PM PengHui Li wrote: > > > > > > > > > > I think it's ok. > > > > > 2.11.0 is the first release version of pulsar-shell > > > > > So it's not a regression that was introduced in 2.11.0 and not > > > > > critical security issues or license issues which will block users > > move > > > > > to the new version. > > > > > > > > > > Thanks, > > > > > Penghui > > > > > > > > > > > > > > > > > > > > On Wed, Jan 4, 2023 at 3:36 PM Yunze Xu > > > > > > > wrote: > > > > > > > > > > > +0 (binding) > > > > > > > > > > > > - Checked the signature > > > > > > - Build from source (Java 17, Ubuntu 20.04 WSL2) > > > > > > - Start standalone with KoP branch-2.11 (ab9d761f) > > > > > > - Verify Pulsar client 2.11.0, Kafka client 3.3.1 (through KoP) > > > > > > - Verify pulsar-shell on Ubuntu 20.04 WSL2 and Windows: > > > > > > 1. Use `client` command to produce and consume messages > > > > > > 2. Use `admin` command to create, list and delete topics > > > > > > > > > > > > I didn't give +1 because I have to modify the > > > > > > `bin/pulsar-admin-common.cmd` to make `pulsar-shell` work on > > Windows. > > > > > > I left a command here: > > > > > > https://github.com/apache/pulsar/pull/17243#discussion_r1061199501 > > . > > > > > > Not sure if it's a blocker so I didn't give +1 or -1 until someone > > > > > > could explain it. > > > > > > > > > > > > Thanks, > > > > > > Yunze > > > > > > > > > > > > On Wed, Jan 4, 2023 at 2:37 PM PengHui Li > > > wrote: > > > > > > > > > > > > > > > I found there are both apache-pulsar-shell-2.11.0-bin.tar.gz > > and > > > > > > > apache-pulsar-shell-2.11.0-bin.zip, are they duplicates? > > > > > > > > > > > > > > I noticed here is the discussion about this part in Slack > > > > > > > > > > https://apache-pulsar.slack.com/archives/C5ZSVEN4E/p1672411025574369 > > > > > > > > > > > > > > Penghui > > > > > > > > > > > > > > On Wed, Jan 4, 2023 at 2:35 PM Yunze Xu > > > > > > > > > > wrote: > > > > > > > > > > > > > > > I found there are both apache-pulsar-shell-2.11.0-bin.tar.gz > > and > > > > > > > > apache-pulsar-shell-2.11.0-bin.zip, are they duplicates? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Yunze > > > > > > > > > > > > > > > > On Mon, Jan 2, 2023 at 8:01 PM guo jiwei > > > > > wrote: > > > > > > > > > > > > > > > > > > This is the fifth release candidate for Apache Pulsar, > > version > > > > > > 2.11.0. > > > > > > > > > > > > > > > > > > This release contains 1617 commits by 61 contributors. > > > > > > > > > > > > > > > > > > https://github.com/apache/pulsar/compare/v2.10.2...v2.11.0-candidate-5 > > > > > > > > > > > > > > > > > > CI for this release candidate > > > > > > > > > https://github.com/Technoboy-/pulsar/pull/26 > > > > > > > > > > > > > > > > > > *** Please download, test and vote on this release. This vote > > > will > > > > > > stay > > > > > > > > open > > > > > > > > > for at least 72 hours *** > > > > > > > > > > > > > > > > > > Note that we are voting upon the source (tag), binaries
