[DISCUSS] PIP-229: Add a common interface to get fields of MessageIdData

2022-12-15 Thread Yunze Xu
Hi all, I've opened a PIP to discuss: https://github.com/apache/pulsar/issues/18950 Currently the `MessageId` interface is not friendly to developers of Pulsar core and ecosystems. There is no abstraction of the `MessageIdData` defined in `PulsarApi.proto`. This proposal aims at solving this pro

Re: [VOTE] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs

2022-12-15 Thread Yunze Xu
Yeah, it's an implementation detail and I will keep the same semantics with the latest master when I push my PR. Thanks, Yunze On Fri, Dec 16, 2022 at 3:03 PM 丛搏 wrote: > > if you don't change this in PIP-229 or PIP-224, I will create a new > PIP to handle the `BatchMessageIdImpl` and `MessageId

[VOTE] Pulsar Client Python Release 3.0.0 Candidate 3

2022-12-15 Thread Yunze Xu
This is the third release candidate for Apache Pulsar Client Python, version 3.0.0. It fixes the following issues: https://github.com/apache/pulsar-client-python/milestone/1?closed=1 *** Please download, test and vote on this release. This vote will stay open for at least 72 hours *** Python whe

Re: Too many emails - Is there a better way to control or manage emails from GitBox

2022-12-15 Thread tison
And I can support the statement that either open-ended discussions or questions traffics are currently most on users@ bias. Best, tison. tison 于2022年12月16日周五 15:11写道: > Hi Michael, > > For my personal mailbox, on Gmail, I have a filter 'to:( > dev@pulsar.apache.org) subject:GitHub' and then al

Re: Too many emails - Is there a better way to control or manage emails from GitBox

2022-12-15 Thread tison
Hi Michael, For my personal mailbox, on Gmail, I have a filter 'to:( dev@pulsar.apache.org) subject:GitHub' and then all related emails are filtered out. It can produce false positives but not hurt so far. Best, tison. Michael Marshall 于2022年12月16日周五 14:35写道: > Do we provide users any guideli

Re: [VOTE] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs

2022-12-15 Thread 丛搏
if you don't change this in PIP-229 or PIP-224, I will create a new PIP to handle the `BatchMessageIdImpl` and `MessageIdImpl` `compareTo()` method, now I have no problem with this PIP +1 (non-binding) Sorry to bother this PIP vote. Thanks, Bo Yunze Xu 于2022年12月16日周五 11:58写道: > > If this breakin

Re: Remove unexpected created https://github.com/apache/pulsar-pulsar-client-cpp?

2022-12-15 Thread Michael Marshall
Oops, I missed those PRs. It is certainly a different case, and we shouldn't remove it! Thanks, Michael On Fri, Dec 16, 2022 at 12:30 AM Dave Fisher wrote: > > Hi Michael, > > I think that the ruby client is diiferent. There are open PRs. The question > is if the community wants to develop a ru

Re: Too many emails - Is there a better way to control or manage emails from GitBox

2022-12-15 Thread Michael Marshall
Do we provide users any guidelines for when to open an issue vs start a discussion? If we cannot control the discussion notifications to this list, I think we should prefer issues for now. The extra notifications are making it challenging for me to keep up with this list. In my opinion, open ended

Re: [DISCUSSIONS] Should we use AUTO_PRODUCE schema?

2022-12-15 Thread 丛搏
> When the input messages are raw bytes, we cannot guarantee the > validation always succeeds because the schema might change. The > exception is actually thrown in `TypedMessageBuilder#value`. > > But since these APIs are stable, we could only fix it by adding the > documents to describe in which

Re: Remove unexpected created https://github.com/apache/pulsar-pulsar-client-cpp?

2022-12-15 Thread Dave Fisher
Hi Michael, I think that the ruby client is diiferent. There are open PRs. The question is if the community wants to develop a ruby client. Best, Dave Sent from my iPhone > On Dec 15, 2022, at 9:37 PM, Michael Marshall wrote: > > Thanks Tison! > > I noticed another Git repo that is not in

[GitHub] [pulsar] michaeljmarshall added a comment to the discussion: Zero receiver queue size for multi topic consumers

2022-12-15 Thread GitBox
GitHub user michaeljmarshall added a comment to the discussion: Zero receiver queue size for multi topic consumers The documentation indicating that it cannot be done for a partitioned topic is going to provide the same justification: https://github.com/apache/pulsar/blame/master/site2/websit

Re: Remove unexpected created https://github.com/apache/pulsar-pulsar-client-cpp?

2022-12-15 Thread Michael Marshall
Thanks Tison! I noticed another Git repo that is not in use: https://github.com/apache/pulsar-client-ruby. That one is not invalid, per se, but it is not being used. Is it worth removing? Thanks, Michael On Thu, Dec 15, 2022 at 8:48 PM Zili Chen wrote: > > So here is the ticket: https://issues.

Re: [DISCUSSIONS] Should we use AUTO_PRODUCE schema?

2022-12-15 Thread Yunze Xu
> It is irresponsible behavior of the producer to leave everything to the > consumer. I agreed now. > I think what we need to do is describe the document clearly IMO, it's a code problem because there is no exception signature for `TypedMessageBuilder#value` and `Message#getValue`. The applicat

Re: [VOTE] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs

2022-12-15 Thread Yunze Xu
If this breaking change can pass the PMC votes, I will keep the new semantics in PIP-229. Otherwise, it would not make sense to adopt the new semantics in PIP-229. Thanks, Yunze On Fri, Dec 16, 2022 at 11:46 AM Yunze Xu wrote: > > I cannot find any confusing code from the proposal itself. Could

Re: [DISCUSS] Modify MessageIdImpl and BatchMessageIdImpl compareTo(MessageId o) method

2022-12-15 Thread Yunze Xu
In short, this change is okay for me. But since it's a breaking change, the Pulsar PMCs should be responsible to make the idea pass. Thanks, Yunze On Fri, Dec 16, 2022 at 11:53 AM Yunze Xu wrote: > > PIP is required when you want to make a breaking change. [1] What you > think is wrong might hav

Re: [DISCUSS] Modify MessageIdImpl and BatchMessageIdImpl compareTo(MessageId o) method

2022-12-15 Thread Yunze Xu
PIP is required when you want to make a breaking change. [1] What you think is wrong might have been applied in existing applications, just like the comments here [2] The only controversial point is that should we treat it as a breaking change? From [3] > A change in one part of a software system

Re: [VOTE] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs

2022-12-15 Thread Yunze Xu
I cannot find any confusing code from the proposal itself. Could you point it out? If you are mentioning the `legacyCompare` and `compare` methods in #18890 [1], it's not related to this proposal. And I have opened PIP-229 [2] for discussion. BTW, the PIP-229 itself doesn't mention the compare log

PIP-229: Add a common interface to get fields of MessageIdData

2022-12-15 Thread Yunze Xu
Hi all, I've opened a PIP to discuss: https://github.com/apache/pulsar/issues/18950 Currently the `MessageId` interface is not friendly to developers of Pulsar core and ecosystems. There is no abstraction of the `MessageIdData` defined in `PulsarApi.proto`. This proposal aims at solving this pro

Re: [VOTE] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs

2022-12-15 Thread 丛搏
Although unrelated, it adds a lot of confusing code. Thanks, Bo Yunze Xu 于2022年12月16日周五 08:05写道: > > This proposal is not related to the comparison logic between > BatchMessageIdImpl and MessageIdImpl. > > Thanks, > Yunze > > On Thu, Dec 15, 2022 at 12:58 PM 丛搏 wrote: > > > > -1 (non-binding)

Re: [DISCUSS] Modify MessageIdImpl and BatchMessageIdImpl compareTo(MessageId o) method

2022-12-15 Thread 丛搏
It doesn’t matter if PIP is open. can directly describe it clearly in https://lists.apache.org/thread/mbrpjsgrgwrlkdpvkk738jxnlk7rf4qk and modify it. But either newly opened PIP or in https://lists.apache.org/thread/mbrpjsgrgwrlkdpvkk738jxnlk7rf4q, my point is that we need to fix it, not compatible

Re: Remove unexpected created https://github.com/apache/pulsar-pulsar-client-cpp?

2022-12-15 Thread Zili Chen
So here is the ticket: https://issues.apache.org/jira/browse/INFRA-24010 On 2022/12/16 00:55:39 Matteo Merli wrote: > Yes, it was my mistake. I couldn't find a way to do it in Gitbox > though. :/ Probably it does require an INFRA ticket. > -- > Matteo Merli > > > On Thu, Dec 15, 2022 at 3:58 PM

Re: Remove unexpected created https://github.com/apache/pulsar-pulsar-client-cpp?

2022-12-15 Thread Dave Fisher
Yes, that would be an INFRA ticket. Refer to this thread. Best, Dave Sent from my iPhone > On Dec 15, 2022, at 4:56 PM, Matteo Merli wrote: > > Yes, it was my mistake. I couldn't find a way to do it in Gitbox > though. :/ Probably it does require an INFRA ticket. > -- > Matteo Merli > > >>

Re: Remove unexpected created https://github.com/apache/pulsar-pulsar-client-cpp?

2022-12-15 Thread Matteo Merli
Yes, it was my mistake. I couldn't find a way to do it in Gitbox though. :/ Probably it does require an INFRA ticket. -- Matteo Merli On Thu, Dec 15, 2022 at 3:58 PM tison wrote: > > I found this repo: https://github.com/apache/pulsar-pulsar-client-cpp > > It seems to be created with a typo. May

Re: [DISCUSS] Modify MessageIdImpl and BatchMessageIdImpl compareTo(MessageId o) method

2022-12-15 Thread Yunze Xu
Let's see another example that is considered as a "bug", not a breaking change. https://lists.apache.org/thread/88t1xxf68j092k09srdwyzj1tk4ml5n9 > I think that this is fixing a bug, if the topic does not exist we should > return "not found". The "bug" and "breaking change" are not in contrast.

Re: [VOTE] Pulsar Client Python Release 3.0.0 Candidate 2

2022-12-15 Thread Yunze Xu
Hi Dave, Thanks for your reminder. I will raise a new release candidate to include the source code. Thanks, Yunze On Thu, Dec 15, 2022 at 6:58 AM Dave Fisher wrote: > > -1 (binding) > > I don’t see a source release package in the dist.Apache.org directory. ASF > projects release source and eve

Re: [VOTE] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs

2022-12-15 Thread Yunze Xu
This proposal is not related to the comparison logic between BatchMessageIdImpl and MessageIdImpl. Thanks, Yunze On Thu, Dec 15, 2022 at 12:58 PM 丛搏 wrote: > > -1 (non-binding) > sorry, I have one question about the BatchMessageId compareTo() > method. the discussion mail : > https://lists.apach

Remove unexpected created https://github.com/apache/pulsar-pulsar-client-cpp?

2022-12-15 Thread tison
I found this repo: https://github.com/apache/pulsar-pulsar-client-cpp It seems to be created with a typo. May we remove it? I don't know whether and how I can perform such actions if desired, though. Best, tison.

[GitHub] [pulsar] congbobo184 added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user congbobo184 added a comment to the discussion: Cannot parse schema @robshep @dhinesherode91 could you please check if the topic has multiple schemas by the problem when it happens? or upgrade the pulsar version and test if this problem has been fixed @tisonkun it seems like a

[GitHub] [pulsar] tisonkun added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user tisonkun added a comment to the discussion: Cannot parse schema Moved to the Q&A category. If we find some bugs later, we can create issues correspondingly. GitHub link: https://github.com/apache/pulsar/discussions/18948#discussioncomment-4414978 This is an automatically s

[GitHub] [pulsar] nodece added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user nodece added a comment to the discussion: Cannot parse schema Thanks @robshep feedback. GitHub link: https://github.com/apache/pulsar/discussions/18948#discussioncomment-4414974 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an ema

[GitHub] [pulsar] nodece added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user nodece added a comment to the discussion: Cannot parse schema Do you have a stable way to reproduce this issue? GitHub link: https://github.com/apache/pulsar/discussions/18948#discussioncomment-4414972 This is an automatically sent email for dev@pulsar.apache.org. To unsubs

[GitHub] [pulsar] robshep added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user robshep added a comment to the discussion: Cannot parse schema I can't relaiably re-produce this, but the issue persists. Thanks @dhinesherode91 for adding your observations. We do not have this load. We have a single standalone running on ample VM (no cpu, memory , i/o or disk

[GitHub] [pulsar] tisonkun added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user tisonkun added a comment to the discussion: Cannot parse schema @nodece is this issue a bug in the Pulsar side or depending on how users use it? GitHub link: https://github.com/apache/pulsar/discussions/18948#discussioncomment-4414976 This is an automatically sent email for

[GitHub] [pulsar] nodece added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user nodece added a comment to the discussion: Cannot parse schema I did a test to reproduce this case: ```java public class Main { public static void main(String[] args) throws Exception { var pulsarClient = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build

[GitHub] [pulsar] dhinesherode91 added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user dhinesherode91 added a comment to the discussion: Cannot parse schema Thanks for the information. In my case, in my local I am running a standalone Pulsar and Consumer applications are Spring boot application. The above issue is not occurring in local. But in staging and produc

[GitHub] [pulsar] dhinesherode91 added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user dhinesherode91 added a comment to the discussion: Cannot parse schema @congbobo184 The issue is not yet resolved. We are using Pulsar 2.7.1. Both broker and Pulsar client are using the same version. Our consumer is subscribed to a single topic only. Ours is an AVRO schema based t

[GitHub] [pulsar] robshep added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user robshep added a comment to the discussion: Cannot parse schema Hi No, sadly I cannot recreate this issue in isolation. But It occurs on ~50% of redeploys and requires repeated restarts of the application to clear it. We use TestContainers as part of the integration tests to spin

[GitHub] [pulsar] congbobo184 added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user congbobo184 added a comment to the discussion: Cannot parse schema @dhinesherode91 hi, this issue problem solved? if not, could you please provide the broker version and client version. Do your consumers consume multiple topics? look at the source code, it does not store the `NON

[GitHub] [pulsar] dhinesherode91 added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user dhinesherode91 added a comment to the discussion: Cannot parse schema Hey, I am facing a similar issue. I am running pulsar 2.7.1 and I have a topic with AVRO schema where my producers push avro messages and consumer receives and process the avro data. My consumer application re

[GitHub] [pulsar] congbobo184 added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user congbobo184 added a comment to the discussion: Cannot parse schema if restart, this topic `persistent://myorg/myappA/ingress` not recover you can use admin api to check is the schema exist. GitHub link: https://github.com/apache/pulsar/discussions/18948#discussioncomment-4414963

[GitHub] [pulsar] robshep added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user robshep added a comment to the discussion: Cannot parse schema Thank you, Attached is the application log from the first time this happened. It isn't clear exaclty which topic the consumer finds a null schema, but you can see in this example it was both consumers. On this occ

[GitHub] [pulsar] robshep added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user robshep added a comment to the discussion: Cannot parse schema Thank you, If the components are generally stable then there could be issues with my setup? In this case I am grateful for your support, and will look you up on slack With thanks R Get Outlook for iOS

[GitHub] [pulsar] congbobo184 added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user congbobo184 added a comment to the discussion: Cannot parse schema @robshep Sorry to hear that, AvroSchema has been in the production environment for a long time. It is a stable feature. I think you can use one topic to test and use ```./pulsar-admin schemas get persistent://t

[GitHub] [pulsar] robshep added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user robshep added a comment to the discussion: Cannot parse schema Hi, This is still happening and we still need to keep restarting connected producers/consumers after updating our application. Occasionally if 20+restarts does not clear it, we destroy the (docker standalone) broker

[GitHub] [pulsar] codelipenghui added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user codelipenghui added a comment to the discussion: Cannot parse schema @robshep Could you please provide more information about the schema configuration? Which schema compatibility policy are you using? how the schema of the topic looks like now(You can use the pulsar-admin schemas

[GitHub] [pulsar] codelipenghui added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user codelipenghui added a comment to the discussion: Cannot parse schema @robshep Thanks, seems the schema definition is not the problem, I think maybe the problem is the reader cache in the StructSchema. @congbobo184 Could you please help take a look at this issue? GitHub link: ht

[GitHub] [pulsar] congbobo184 added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user congbobo184 added a comment to the discussion: Cannot parse schema @robshep could you please provide more detailed information from app3, log and which topic the consumer `Cannot parse schema` GitHub link: https://github.com/apache/pulsar/discussions/18948#discussioncomment-441

[GitHub] [pulsar] robshep added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user robshep added a comment to the discussion: Cannot parse schema Thank you, I'm afraid I don't know how to get the schema compatibility policy of a topic - please can you advise. I don't see a cli command to show this) On the few occasions this has happened, it typically works OK

[GitHub] [pulsar] robshep added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user robshep added a comment to the discussion: Cannot parse schema Hi, thanks for taking a look. There are 3 topics in use in this deployment. All have schemas. None of this code, or the Payload objects have changed for many months. I notice in App1 there is another topic `-RETRY`

[GitHub] [pulsar] congbobo184 added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user congbobo184 added a comment to the discussion: Cannot parse schema @robshep hello, I look up your log and are you consume more than one topic in your consumer? it look like one topic does not have schema. GitHub link: https://github.com/apache/pulsar/discussions/18948#discussio

[GitHub] [pulsar] robshep added a comment to the discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user robshep added a comment to the discussion: Cannot parse schema A log file showing the same issue encountered just now. A restart of the application recovered, but seemingly wasn't able to recover itself. [thingping.log](https://github.com/apache/pulsar/files/5490475/thingping.l

[GitHub] [pulsar] robshep created a discussion: Cannot parse schema

2022-12-15 Thread GitBox
GitHub user robshep created a discussion: Cannot parse schema **Describe the bug** Reconnecting an updated consumer application that had NOT changed any of the following: * Any connection or topic details * Any message object details used to generate the AvroSchema This system was restarted

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

2022-12-15 Thread tison
About the final status, you can take the inlong-website repo as a reference: https://github.com/apache/inlong-website Best, tison. tison 于2022年12月16日周五 07:07写道: > Hi, > > After several works around the build flow of our official > website[1][2][3], the content sync and site build flow is debug

[PROPOSAL] Website precommit and move the source of docs to the site repo

2022-12-15 Thread tison
Hi, After several works around the build flow of our official website[1][2][3], the content sync and site build flow is debuggable and reproducible now. However, compared to other Apache projects' websites' project layouts and workflow, we still meet two challenges on the Pulsar site: 1. We don'