Re: [VOTE] Pulsar Release 3.0.0 Candidate 3

[Christophe Bornet]

Good catch!
I'll trigger a new RC.
For the future we need to have a procedure to not allow this kind of issues.
Maybe we should remove "release/blocker" labels once they are merged
into the release branch and ensure there are 0 "release/blocker"
before cutting the release ?

Le jeu. 27 avr. 2023 à 08:25, Yunze Xu  a écrit :
>
> -1 (binding)
>
> There was a fix [1] for a regression that was not cherry-picked into
> branch-3.0 and here [2] is an example failure that was affected by the
> regression. This PR was labeled as "release/blocker" so I
> cherry-picked it into branch-3.0. Then we need to trigger another
> candidate release.
>
> [1] https://github.com/apache/pulsar/pull/20163
> [2] 
> https://github.com/streamnative/kop/actions/runs/4815818877/jobs/8574850990?pr=1822
>
> Thanks,
> Yunze
>
> On Thu, Apr 27, 2023 at 12:27 AM Matteo Merli  wrote:
> >
> > +1
> >
> > Checked signatures, standalone & docker images.
> >
> >
> > --
> > Matteo Merli
> > 
> >
> >
> > On Tue, Apr 25, 2023 at 7:38 AM Christophe Bornet 
> > wrote:
> >
> > > This is the third release candidate for Apache Pulsar, version 3.0.0.
> > >
> > > It fixes the following issues:
> > > https://github.com/apache/pulsar/milestone/34?closed=1
> > >
> > > *** Please download, test and vote on this release. This vote will
> > > stay open for at least 72 hours ***
> > >
> > > Note that we are voting upon the source (tag), binaries are provided
> > > for convenience.
> > >
> > > Source and binary files:
> > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.0.0-candidate-3/
> > >
> > > SHA-512 checksums:
> > >
> > > 6d50952ee14902d5a9b55df9f52716fec6ab5b960fbb07b088f0487754d4e8d85679b9604039de47c034f222d76dab2cfc35f8e60fdb76cfcab79c2cb8f510a3
> > > ./apache-pulsar-3.0.0-bin.tar.gz
> > >
> > > fa75e46014577f74b1af81205fe29662b2f388df18c2b15072353dc6bcf29729a96737f4ae8992bf0c418b3da76508f2e4d924105038f3617b1c2b7be68591fe
> > >  ./apache-pulsar-3.0.0-src.tar.gz
> > >
> > > Maven staging repo:
> > > https://repository.apache.org/content/repositories/orgapachepulsar-1231/
> > >
> > > The tag to be voted upon:
> > > v3.0.0-candidate-3 (e8261fcef8d76061d378d8a9f3b05a5d115013fe)
> > > https://github.com/apache/pulsar/releases/tag/v3.0.0-candidate-3
> > >
> > > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > >
> > > Docker images:
> > >
> > >
> > > https://hub.docker.com/layers/cbornet/pulsar/3.0.0/images/sha256-e38659f1f3a0e16ea13925342434513e73bdebaaae84d4e84011de64fcd47c45?context=repo
> > >
> > >
> > > https://hub.docker.com/layers/cbornet/pulsar-all/3.0.0/images/sha256-13aadf3dc46c9029fa434fa9c9f5278dedfe225775b4cacb5924abb8d8bfb626?context=repo
> > >
> > > Please download the source package, and follow the README to build and
> > > run the Pulsar standalone service.
> > > https://pulsar.apache.org/contribute/validate-release-candidate
> > >
> > > This release candidate contains docker images for both AMD64 and ARM64
> > > architectures. Please indicate which architecture you could test when
> > > casting your vote.
> > >
> > > Thanks
> > >
> > > Christophe Bornet
> > >

Re: [VOTE] Pulsar Release 3.0.0 Candidate 3

[Yunze Xu]

Yes, it should be documented how we should use the "release/blocker" label.
1. Add this label for the PR that should be included in the release candidate.
2. After it's merged, cherry-pick it into the release branch and
remove the label.
3. Ensure this label does not exist when starting a vote.

Thanks,
Yunze

On Thu, Apr 27, 2023 at 3:23 PM Christophe Bornet
 wrote:
>
> Good catch!
> I'll trigger a new RC.
> For the future we need to have a procedure to not allow this kind of issues.
> Maybe we should remove "release/blocker" labels once they are merged
> into the release branch and ensure there are 0 "release/blocker"
> before cutting the release ?
>
> Le jeu. 27 avr. 2023 à 08:25, Yunze Xu  a écrit 
> :
> >
> > -1 (binding)
> >
> > There was a fix [1] for a regression that was not cherry-picked into
> > branch-3.0 and here [2] is an example failure that was affected by the
> > regression. This PR was labeled as "release/blocker" so I
> > cherry-picked it into branch-3.0. Then we need to trigger another
> > candidate release.
> >
> > [1] https://github.com/apache/pulsar/pull/20163
> > [2] 
> > https://github.com/streamnative/kop/actions/runs/4815818877/jobs/8574850990?pr=1822
> >
> > Thanks,
> > Yunze
> >
> > On Thu, Apr 27, 2023 at 12:27 AM Matteo Merli  
> > wrote:
> > >
> > > +1
> > >
> > > Checked signatures, standalone & docker images.
> > >
> > >
> > > --
> > > Matteo Merli
> > > 
> > >
> > >
> > > On Tue, Apr 25, 2023 at 7:38 AM Christophe Bornet 
> > > wrote:
> > >
> > > > This is the third release candidate for Apache Pulsar, version 3.0.0.
> > > >
> > > > It fixes the following issues:
> > > > https://github.com/apache/pulsar/milestone/34?closed=1
> > > >
> > > > *** Please download, test and vote on this release. This vote will
> > > > stay open for at least 72 hours ***
> > > >
> > > > Note that we are voting upon the source (tag), binaries are provided
> > > > for convenience.
> > > >
> > > > Source and binary files:
> > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.0.0-candidate-3/
> > > >
> > > > SHA-512 checksums:
> > > >
> > > > 6d50952ee14902d5a9b55df9f52716fec6ab5b960fbb07b088f0487754d4e8d85679b9604039de47c034f222d76dab2cfc35f8e60fdb76cfcab79c2cb8f510a3
> > > > ./apache-pulsar-3.0.0-bin.tar.gz
> > > >
> > > > fa75e46014577f74b1af81205fe29662b2f388df18c2b15072353dc6bcf29729a96737f4ae8992bf0c418b3da76508f2e4d924105038f3617b1c2b7be68591fe
> > > >  ./apache-pulsar-3.0.0-src.tar.gz
> > > >
> > > > Maven staging repo:
> > > > https://repository.apache.org/content/repositories/orgapachepulsar-1231/
> > > >
> > > > The tag to be voted upon:
> > > > v3.0.0-candidate-3 (e8261fcef8d76061d378d8a9f3b05a5d115013fe)
> > > > https://github.com/apache/pulsar/releases/tag/v3.0.0-candidate-3
> > > >
> > > > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > > >
> > > > Docker images:
> > > >
> > > >
> > > > https://hub.docker.com/layers/cbornet/pulsar/3.0.0/images/sha256-e38659f1f3a0e16ea13925342434513e73bdebaaae84d4e84011de64fcd47c45?context=repo
> > > >
> > > >
> > > > https://hub.docker.com/layers/cbornet/pulsar-all/3.0.0/images/sha256-13aadf3dc46c9029fa434fa9c9f5278dedfe225775b4cacb5924abb8d8bfb626?context=repo
> > > >
> > > > Please download the source package, and follow the README to build and
> > > > run the Pulsar standalone service.
> > > > https://pulsar.apache.org/contribute/validate-release-candidate
> > > >
> > > > This release candidate contains docker images for both AMD64 and ARM64
> > > > architectures. Please indicate which architecture you could test when
> > > > casting your vote.
> > > >
> > > > Thanks
> > > >
> > > > Christophe Bornet
> > > >

Fwd: Pulsar Functions Lifecycle?

[Niclas Hedhman]

Hi,

I am noticing that the users@pulsar.a.o is turning into a ANNOUNCEMENT 
list, and not really an active forum to helpful users. Is there some 
other forum where this help is happening, or is this community only 
consisting of the developers of the product?


Cheers
Niclas


 Original Message 
Subject: Pulsar Functions Lifecycle?
Date: 2023-04-26 12:55
From: Niclas Hedhman 
To: us...@pulsar.apache.org
Reply-To: us...@pulsar.apache.org

Hi,

I have been using Pulsar quite successfully for a bit more than a year 
now, and quite happy with it.


Now, I would like to streamline my app a bit and use Pulsar Functions 
(on 2.11.x), but I need a little bit of guidance.


1. IIUIC, I have the option (among other) to run my functions (trusted) 
inside the same JVM as the Pulsar Broker itself, by choosing "Run 
function workers with brokers" and the "thread runtime". And if I want 
to run on the same VM as the broker, but in separate OS process, I 
simply follow "process runtime". Is that correct? I don't need K8s to 
run functions (I have not set up K8s)?



2. Lifecycle of functions is very unclear to me.
   a. How many instances are created?
   b. One per request?
   c. One per key?
   d. Are instances pooled?
   e. Do the functions need to be thread-safe?
   f. Can I control it?

3. Deployment
   a. When deploying with "pulsar-admin functions create", do I need to
  do that on each broker instance, or just once? What happens if my
  Ansible tries to do that in parallel on all instances?

   b. I assume that "pulsar-admin functions update" will let all
  functions complete before killing the thread/process. Right?


I guess there will be more questions once I dive deeper.


TIA
Niclas

Re: Pulsar Functions Lifecycle?

[tison]

Hi Niclas,

I think https://github.com/apache/pulsar/discussions/categories/q-a is the
correct place. Or you can find more contact info at
https://pulsar.apache.org/community/#section-discussions.

Best,
tison.


Niclas Hedhman  于2023年4月27日周四 16:59写道：

> Hi,
>
> I am noticing that the users@pulsar.a.o is turning into a ANNOUNCEMENT
> list, and not really an active forum to helpful users. Is there some
> other forum where this help is happening, or is this community only
> consisting of the developers of the product?
>
> Cheers
> Niclas
>
>
>  Original Message 
> Subject: Pulsar Functions Lifecycle?
> Date: 2023-04-26 12:55
>  From: Niclas Hedhman 
> To: us...@pulsar.apache.org
> Reply-To: us...@pulsar.apache.org
>
> Hi,
>
> I have been using Pulsar quite successfully for a bit more than a year
> now, and quite happy with it.
>
> Now, I would like to streamline my app a bit and use Pulsar Functions
> (on 2.11.x), but I need a little bit of guidance.
>
> 1. IIUIC, I have the option (among other) to run my functions (trusted)
> inside the same JVM as the Pulsar Broker itself, by choosing "Run
> function workers with brokers" and the "thread runtime". And if I want
> to run on the same VM as the broker, but in separate OS process, I
> simply follow "process runtime". Is that correct? I don't need K8s to
> run functions (I have not set up K8s)?
>
>
> 2. Lifecycle of functions is very unclear to me.
> a. How many instances are created?
> b. One per request?
> c. One per key?
> d. Are instances pooled?
> e. Do the functions need to be thread-safe?
> f. Can I control it?
>
> 3. Deployment
> a. When deploying with "pulsar-admin functions create", do I need to
>do that on each broker instance, or just once? What happens if my
>Ansible tries to do that in parallel on all instances?
>
> b. I assume that "pulsar-admin functions update" will let all
>functions complete before killing the thread/process. Right?
>
>
> I guess there will be more questions once I dive deeper.
>
>
> TIA
> Niclas
>

[VOTE] Pulsar Release 3.0.0 Candidate 4

[Christophe Bornet]

This is the fourth release candidate for Apache Pulsar, version 3.0.0.

It fixes the following issues:
https://github.com/apache/pulsar/milestone/34?closed=1

*** Please download, test and vote on this release. This vote will
stay open for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided
for convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.0.0-candidate-4/

SHA-512 checksums:
7c74599ff52606c34c7123619beee37eb89f86a1f78daeb827915fd2275d976aa05469914b7485a3f4c2f67f663f397677c25be615dded386f896c1dbf32ea29
 ./apache-pulsar-3.0.0-bin.tar.gz
e0e3919f4e8d79146dfabdef64986760339405c404ec2552db0dcb72081588f44d1b84b5cca9061956152892ff6d83aa94e6c2f2c5f3d722f0398a885721ed6e
 ./apache-pulsar-3.0.0-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1232/

The tag to be voted upon:
v3.0.0-candidate-4 (7636e8989f4d3fc24fce69a356d54e1c550945ed)
https://github.com/apache/pulsar/releases/tag/v3.0.0-candidate-4

Pulsar's KEYS file containing PGP keys you use to sign the release:
https://dist.apache.org/repos/dist/dev/pulsar/KEYS

Docker images:

https://hub.docker.com/layers/cbornet/pulsar/3.0.0/images/sha256-43ba59feeef43ce54f00ca80af53c075f3d98caf6d150215c988f78ccecc89c0?context=repo
https://hub.docker.com/layers/cbornet/pulsar-all/3.0.0/images/sha256-19bf90d5404d8bbdf181e9270c46b3423e8d06d8db3e23cc2a5bc86034482179?context=repo

Please download the source package, and follow the README to build and
run the Pulsar standalone service.
https://pulsar.apache.org/contribute/validate-release-candidate

This release candidate contains docker images for both AMD64 and ARM64
architectures. Please indicate which architecture you could test when
casting your vote.

Thanks

Christophe Bornet

[DISCUSS] PIP-264: Enhanced OTel-based metric system

[Asaf Mesika]

Hi all,

I'm very excited to release a PIP I've been working on in the past 11
months, which I think will be immensely valuable to Pulsar, which I like so
much.

PIP: https://github.com/apache/pulsar/issues/20197

I'm quoting here the preface:

=== QUOTE START ===

Roughly 11 months ago, I started working on solving the biggest issue with
Pulsar metrics: the lack of ability to monitor a pulsar broker with a large
topic count: 10k, 100k, and future support of 1M. This started by mapping
the existing functionality and then enumerating all the problems I saw (all
documented in this doc

).

This PIP is a parent PIP. It aims to gradually solve (using sub-PIPs) all
the current metric system's problems and provide the ability to monitor a
broker with a large topic count, which is currently lacking. As a parent
PIP, it will describe each problem and its solution at a high level,
leaving fine-grained details to the sub-PIPs. The parent PIP ensures all
solutions align and does not contradict each other.

The basic building block to solve the monitoring ability of large topic
count is aggregating internally (to topic groups) and adding fine-grained
filtering. We could have shoe-horned it into the existing metric system,
but we thought adding that to a system already ingrained with many problems
would be wrong and hard to do gradually, as so many things will break. This
is why the second-biggest design decision presented here is consolidating
all existing metric libraries into a single one - OpenTelemetry
. The parent PIP will explain why OpenTelemetry
was chosen out of existing solutions and why it far exceeds all other
options. I’ve been working closely with the OpenTelemetry community in the
past eight months: brain-storming this integration, and raising issues, in
an effort to remove serious blockers to make this migration successful.

I made every effort to summarize this document so that it can be concise
yet clear. I understand it is an effort to read it and, more so, provide
meaningful feedback on such a large document; hence I’m very grateful for
each individual who does so.

I think this design will help improve the user experience immensely, so it
is worth the time spent reading it.


=== QUOTE END ===


Thanks!

Asaf Mesika

[DISCUSS] Improve Pulsar Function Source Primitive Schema Mapping

[Neng Lu]

Hi All,

Based on [1], Pulsar has various primitive schema types and has a very
clear mapping between java classes to primitive schema types.

But in code [2], Pulsar Function Source only handles the byte and String
java classes primitive schema mapping while default all other primitive
types to JSON schema. Also for byte class types, the NONE schema is used
instead of the BYTES schema.

All these differences cause confusion for users trying to use Pulsar
Functions for the first time, and also make Pulsar Function not following
the Pulsar Schema official document.

Ideally, we should change the code [2], to make it following [1]. But such
changes may lead to breaking behaviors for existing users who adapted their
code to run the Pulsar Functions.

I would like to hear your thoughts on this and see how we should proceed.

Thank you! Regards

[1] https://pulsar.apache.org/docs/2.11.x/schema-understand/#primitive-type
[2]
https://github.com/apache/pulsar/blob/master/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/TopicSchema.java#L124

[DISCUSS] Is PIP required for small changes in metrics

[mattisonchao]

Hi, All

I've got two questions want to discuss with you guys.

1. I am wondering if we should draft PIP for small metrics changes, e.g: 
https://github.com/apache/pulsar/pull/20147
2. We haven't declear we should draft a PIP for configuration changes. why?  
Refer to: https://github.com/apache/pulsar/blob/master/wiki/proposals/PIP.md


I want to discuss with you all to come to a conclusion within the community, 
thanks a lot!

Best,
Mattison

Re: [DISCUSS] Is PIP required for small changes in metrics

[Rajan Dhabalia]

Hi,

Thank you Mattison for starting this thread because I am feeling some of
the community members are making new contributors' life difficult and
trying to enforce rules which were never discussed and discourage them from
contributing to Pulsar. I really don't want to see the Pulsar community
become like Confluent but the Pulsar community should welcome and encourage
contributors so, we can grow the Pulsar community, If we help and
encourage companies to contribute for their requirements then we can
increase the Pulsar adoption.
For example: our PIP requirement :
https://github.com/apache/pulsar/blob/master/wiki/proposals/PIP.md doesn't
talk about anything specific about config changes but that PR was blocked
because of undocumented reasons. There are many such PRs that might have
merged but it would not be fair to discourage new contributors by giving
undocumented and invalid reasons.
So, let's help and encourage community and new contributors so we can
increase Pulsar adoption across the industry which should help everyone to
grow together.

Thanks,
Rajan


On Thu, Apr 27, 2023 at 6:22 PM  wrote:

>
> Hi, All
>
> I've got two questions want to discuss with you guys.
>
> 1. I am wondering if we should draft PIP for small metrics changes, e.g:
> https://github.com/apache/pulsar/pull/20147
> 2. We haven't declear we should draft a PIP for configuration changes.
> why?  Refer to:
> https://github.com/apache/pulsar/blob/master/wiki/proposals/PIP.md
>
>
> I want to discuss with you all to come to a conclusion within the
> community, thanks a lot!
>
> Best,
> Mattison
>

Re: [DISCUSS] Is PIP required for small changes in metrics

[lifepuzzlefun]

Totally agree. I have seen some patches with little change worth to do, but 
stopped making progress, only being told an pip is needed.
I think if we have one fast-path for notice small changes like metrics or 
configuration. 


some pip to add configuration is also simple enough. like just add an 
configuration and set the parameter.
contribution will be easy and the separation of different kinds of pip is 
helpful for those who only what to know the feature change.


also i think an guide is needed to tell people which kind of patch need an pip, 
and what should he/she do to make pip making progress.



On 2023/04/28 01:41:09 Rajan Dhabalia wrote:
> Hi,
> 
> Thank you Mattison for starting this thread because I am feeling some of
> the community members are making new contributors' life difficult and
> trying to enforce rules which were never discussed and discourage them from
> contributing to Pulsar. I really don't want to see the Pulsar community
> become like Confluent but the Pulsar community should welcome and encourage
> contributors so, we can grow the Pulsar community, If we help and
> encourage companies to contribute for their requirements then we can
> increase the Pulsar adoption.
> For example: our PIP requirement :
> https://github.com/apache/pulsar/blob/master/wiki/proposals/PIP.md doesn't
> talk about anything specific about config changes but that PR was blocked
> because of undocumented reasons. There are many such PRs that might have
> merged but it would not be fair to discourage new contributors by giving
> undocumented and invalid reasons.
> So, let's help and encourage community and new contributors so we can
> increase Pulsar adoption across the industry which should help everyone to
> grow together.
> 
> Thanks,
> Rajan
> 
> 
> On Thu, Apr 27, 2023 at 6:22 PM  wrote:
> 
> >
> > Hi, All
> >
> > I've got two questions want to discuss with you guys.
> >
> > 1. I am wondering if we should draft PIP for small metrics changes, e.g:
> > https://github.com/apache/pulsar/pull/20147
> > 2. We haven't declear we should draft a PIP for configuration changes.
> > why?  Refer to:
> > https://github.com/apache/pulsar/blob/master/wiki/proposals/PIP.md
> >
> >
> > I want to discuss with you all to come to a conclusion within the
> > community, thanks a lot!
> >
> > Best,
> > Mattison
> >
> 

Re: [DISCUSS] Is PIP required for small changes in metrics

[Yunze Xu]

The problem is not related to whether the patch is simple. It is,
adding configurations were very casual in the past few years. Let's
look at how many configurations we have from a discussion [1] I
started a few months ago.

Adding a configuration is simple, removing it is hard.

Most of these configurations are never touched. They were added just
because the authors think it's flexible. Let's take a recent PR [2] as
example, this PR skips splitting bundles when there is one broker.
However, if the author tried to make it configurable and added a
config to determine whether to do that. Then, a new config that might
never be touched would be added. What's worse, a new config can be
applied to the namespace level of topic level policy.

More ridiculously, nearly every if statement could bring a new config.
To make the Pulsar community grow fast, new contributors can
contribute many PRs by adding this and that new configs. BUT, IS IT
REALLY GOOD FOR COMMUNITY?

The motivation of requiring a PIP for a new configuration is to notify
more developers and record the new configuration in the GH issue.

> but stopped making progress, only being told an pip is needed.

It's the contributor's responsibility to make progress once he's told
that a PIP is required. I think pushing the patch without any
discussion is bad. It's reasonable for contributors to ask why the PIP
is needed, what is the PIP. But it's not reasonable to just ignore the
comment and just go away.

[1] https://lists.apache.org/thread/j23ny19opmp8jww57gwk7g27b5dvl0ot
[2] https://github.com/apache/pulsar/pull/20190

Thanks,
Yunze

On Fri, Apr 28, 2023 at 11:03 AM lifepuzzlefun  wrote:
>
> Totally agree. I have seen some patches with little change worth to do, but 
> stopped making progress, only being told an pip is needed.
> I think if we have one fast-path for notice small changes like metrics or 
> configuration.
>
>
> some pip to add configuration is also simple enough. like just add an 
> configuration and set the parameter.
> contribution will be easy and the separation of different kinds of pip is 
> helpful for those who only what to know the feature change.
>
>
> also i think an guide is needed to tell people which kind of patch need an 
> pip, and what should he/she do to make pip making progress.
>
>
>
> On 2023/04/28 01:41:09 Rajan Dhabalia wrote:
> > Hi,
> >
> > Thank you Mattison for starting this thread because I am feeling some of
> > the community members are making new contributors' life difficult and
> > trying to enforce rules which were never discussed and discourage them from
> > contributing to Pulsar. I really don't want to see the Pulsar community
> > become like Confluent but the Pulsar community should welcome and encourage
> > contributors so, we can grow the Pulsar community, If we help and
> > encourage companies to contribute for their requirements then we can
> > increase the Pulsar adoption.
> > For example: our PIP requirement :
> > https://github.com/apache/pulsar/blob/master/wiki/proposals/PIP.md doesn't
> > talk about anything specific about config changes but that PR was blocked
> > because of undocumented reasons. There are many such PRs that might have
> > merged but it would not be fair to discourage new contributors by giving
> > undocumented and invalid reasons.
> > So, let's help and encourage community and new contributors so we can
> > increase Pulsar adoption across the industry which should help everyone to
> > grow together.
> >
> > Thanks,
> > Rajan
> >
> >
> > On Thu, Apr 27, 2023 at 6:22 PM  wrote:
> >
> > >
> > > Hi, All
> > >
> > > I've got two questions want to discuss with you guys.
> > >
> > > 1. I am wondering if we should draft PIP for small metrics changes, e.g:
> > > https://github.com/apache/pulsar/pull/20147
> > > 2. We haven't declear we should draft a PIP for configuration changes.
> > > why?  Refer to:
> > > https://github.com/apache/pulsar/blob/master/wiki/proposals/PIP.md
> > >
> > >
> > > I want to discuss with you all to come to a conclusion within the
> > > community, thanks a lot!
> > >
> > > Best,
> > > Mattison
> > >
> >

Re: [DISCUSS] Improve Pulsar Function Source Primitive Schema Mapping

[Pengcheng Jiang]

Hello Neng,

IMO, we should update code[2] to follow the doc, and for existing
functions, if they are in running status, they won't touch code[2]; and for
a new run, functions
will fail to start, and this will remind users to update their function

Regards,
Pengcheng Jiang

Neng Lu  于2023年4月28日周五 06:59写道：

> Hi All,
>
> Based on [1], Pulsar has various primitive schema types and has a very
> clear mapping between java classes to primitive schema types.
>
> But in code [2], Pulsar Function Source only handles the byte and String
> java classes primitive schema mapping while default all other primitive
> types to JSON schema. Also for byte class types, the NONE schema is used
> instead of the BYTES schema.
>
> All these differences cause confusion for users trying to use Pulsar
> Functions for the first time, and also make Pulsar Function not following
> the Pulsar Schema official document.
>
> Ideally, we should change the code [2], to make it following [1]. But such
> changes may lead to breaking behaviors for existing users who adapted their
> code to run the Pulsar Functions.
>
> I would like to hear your thoughts on this and see how we should proceed.
>
> Thank you! Regards
>
> [1]
> https://pulsar.apache.org/docs/2.11.x/schema-understand/#primitive-type
> [2]
>
> https://github.com/apache/pulsar/blob/master/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/TopicSchema.java#L124
>