date:20200416

Re: [DISCUSS] PIP: Producer Send Message with Different Schema

2020-04-16 Thread Sijie Guo

Yeah!

I don't think there is anyone picking this up yet. You are very welcome to
contribute to this feature. Can you start putting up a PIP for it?

Thanks,
Sijie

On Wed, Apr 15, 2020 at 9:35 PM Shivji Kumar Jha  wrote:

> Hi Sijie,
>
> If no one has picked this up, I would like to volunteer for this feature.
>
> Regards,
> Shivji Kumar Jha
> http://www.shivjijha.com/
> +91 8884075512
>
>
> On Thu, Apr 16, 2020 at 3:53 AM Sijie Guo  wrote:
>
> > I see. I wasn't sure that Raman is looking for this capability based on
> his
> > previous email.
> >
> > I do agree that decoupling the relationship between topic and schema can
> > drive more use cases. It is a great feature to add.
> >
> > We will pick this up and come up a PIP for introducing this capability.
> >
> > Thanks,
> > Sijie
> >
> > On Wed, Apr 15, 2020 at 4:25 AM Shivji Kumar Jha 
> > wrote:
> >
> > > Hi Sijie,
> > >
> > > I second with Raman. Apart from PIP-43 and PIP-44 which ease schema
> > > management, in my opinion, we should also loosely couple the
> association
> > > between topic and schema (or more precisely *type of data* on topic)
> > which
> > > is 1 to 1 as of now.
> > >
> > >1. The schema (or schema versions of one data type) could be grouped
> > >into what Kafka calls *subject*.
> > >2. The schema compatibility should then be done among schemas in the
> > >same subject only.
> > >3. One topic can associate with multiple schema subjects and have
> > their
> > >own evolution paths.
> > >4. Similarly, one subject can also associate to multiple topics.
> > >
> > > *Use case:*
> > > This feature would be handy when one needs different business models
> in a
> > > strictly ordered fashion. At the same time, these business models have
> > > their own evolution paths too. As an example, an event sourcing system
> > > could have events like customerCreated, customerAddressChanged,
> > > customerInvoicePaid events etc required in order.
> > >
> > > The ideas presented above are picked from here
> > > <
> https://martin.kleppmann.com/2018/01/18/event-types-in-kafka-topic.html
> > >.
> > >
> > > Regards,
> > > Shivji Kumar Jha
> > > http://www.shivjijha.com/
> > > +91 8884075512
> > >
> > >
> > > On Wed, Apr 15, 2020 at 2:27 AM Sijie Guo  wrote:
> > >
> > > > Hi Raman,
> > > >
> > > > The schema compatibility strategies were already there prior to
> PIP-43.
> > > >
> > > > PIP-44 enhances the schema compatibility strategy support.
> > > >
> > > > Both of the changes are already landed in 2.5.0 release.
> > > >
> > > > Did you see any issues when you tryout this feature?
> > > >
> > > > - Sijie
> > > >
> > > > On Tue, Apr 14, 2020 at 8:35 AM rocketra...@gmail.com <
> > > > rocketra...@gmail.com>
> > > > wrote:
> > > >
> > > > > Now that PIP-43 is released in 2.5.0, I wanted to follow up on the
> > > > > messages below.
> > > > >
> > > > > What is remaining to be done in Pulsar to support having multiple
> > > > > different types on one topic in Pulsar? Yi indicates below that
> > PIP-43
> > > > sets
> > > > > the stage for this, but that the schema compatibility
> implementation
> > > > still
> > > > > would need some work.
> > > > >
> > > > > Would this require another PIP, or just an issue to track the work?
> > > > >
> > > > > Regards,
> > > > > Raman
> > > > >
> > > > > On 2019/09/16 01:32:39, Yi Tang  wrote:
> > > > > > Hi rarma,
> > > > > >
> > > > > > It's a great and important feature, I think. This PIP requires
> the
> > > > > > compatibility check from bottom registry only and doesn't touch
> the
> > > > > > implementation detail. I think we should address this feature in
> > the
> > > > > > future, and this PIP provides the essential ability to implement
> > it.
> > > > > >
> > > > > > Thanks,
> > > > > > Yi
> > > > > >
> > > > > > rocketra...@gmail.com  于 2019年9月15日周日
> > > 22:36写道：
> > > > > >
> > > > > > > I see a mention of compatibility in the PIP but with no
> details.
> > > The
> > > > > docs
> > > > > > > about schema compatibility state this:
> > > > > > >
> > > > > > > > Consequently, those events need to go in the same Pulsar
> > > partition
> > > > to
> > > > > > > maintain order. This application can use ALWAYS_COMPATIBLE to
> > allow
> > > > > > > different kinds of events co-exist in the same topic.
> > > > > > >
> > > > > > > With this PIP, this limitation can be relaxed, and schema
> > > > compatibility
> > > > > > > should be able to be strengthened, since each type of message
> on
> > a
> > > > > topic
> > > > > > > can have its own schema, and compatibility can then be checked
> > > > against
> > > > > only
> > > > > > > other schemas for the same type. Kafka does this via the
> concept
> > of
> > > > > > > "subjects" in the schema registry, and subjects default to just
> > the
> > > > > topic
> > > > > > > name (plus a "-key" or "-value" suffix since keys and values
> can
> > > both
> > > > > have
> > > > > > > their own schemas), but can also include (via an injectable
> > > strategy)

Build failed in Jenkins: pulsar-website-build #966

2020-04-16 Thread Apache Jenkins Server

See 


Changes:

[github] [python client]Support generate pulsar-client for python3.8 (#6741)

[github] [pulsar-broker] avoid backpressure by skipping dispatching if consumer

[github] [Issue #6711]: add audience verify in AuthenticationProviderToken


--
[...truncated 3.64 MB...]
Extracting: '/website/translated_docs/zh-TW/version-2.3.2/administration-geo.md'
Extracting: '/website/translated_docs/zh-TW/version-2.3.2/concepts-clients.md'
Extracting: '/website/translated_docs/zh-TW/version-2.3.2/concepts-messaging.md'
Extracting: '/website/translated_docs/zh-TW/version-2.3.2/deploy-bare-metal.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.3.2/functions-guarantees.md'
Extracting: '/website/translated_docs/zh-TW/version-2.3.2/functions-worker.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.3.2/getting-started-clients.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.3.2/getting-started-docker.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.3.2/getting-started-standalone.md'
Extracting: '/website/translated_docs/zh-TW/version-2.3.2/io-connectors.md'
Extracting: '/website/translated_docs/zh-TW/version-2.3.2/io-redis.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.3.2/reference-cli-tools.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.3.2/reference-configuration.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.3.2/reference-pulsar-admin.md'
Extracting: '/website/translated_docs/zh-TW/version-2.3.2/security-kerberos.md'
Extracting: '/website/translated_docs/zh-TW/version-2.3.2/security-overview.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/adaptors-kafka.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/admin-api-overview.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.0/administration-zk-bk.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.0/client-libraries-go.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.0/client-libraries-java.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/concepts-messaging.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.0/concepts-tiered-storage.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.0/cookbooks-compaction.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.0/cookbooks-tiered-storage.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/deploy-bare-metal.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.0/developing-binary-protocol.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/developing-cpp.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/functions-api.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.0/functions-debugging.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/functions-overview.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.0/functions-quickstart.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/functions-runtime.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/functions-worker.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.0/getting-started-standalone.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/io-cdc-canal.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/io-cdc-debezium.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/io-managing.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/io-quickstart.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.0/reference-cli-tools.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.0/reference-configuration.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/reference-metrics.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.0/reference-pulsar-admin.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/security-jwt.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/security-kerberos.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.0/security-overview.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.0/security-tls-authentication.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.0/security-token-admin.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.0/sql-getting-started.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.1/admin-api-functions.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.1/admin-api-non-persistent-topics.md'
Extracting: '/website/translated_docs/zh-TW/version-2.4.1/admin-api-overview.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.1/admin-api-partitioned-topics.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.1/administration-dashboard.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2.4.1/administration-upgrade.md'
Extracting: 
'/website/translated_docs/zh-TW/version-2

Slack digest for #dev - 2020-04-16

2020-04-16 Thread Apache Pulsar Slack

2020-04-15 09:30:13 UTC - Frederic Tausch: @Frederic Tausch has joined the 
channel

2020-04-15 13:54:20 UTC - matt_innerspace.io: @Addison Higham where would i 
find that image?  not sure where to look.

2020-04-15 14:14:03 UTC - matt_innerspace.io: found it -

Re: [DISCUSS] PIP: Producer Send Message with Different Schema

2020-04-16 Thread Shivji Kumar Jha

Sure Sijie.

Regards,
Shivji Kumar Jha
http://www.shivjijha.com/
+91 8884075512


On Thu, Apr 16, 2020 at 2:22 PM Sijie Guo  wrote:

> Yeah!
>
> I don't think there is anyone picking this up yet. You are very welcome to
> contribute to this feature. Can you start putting up a PIP for it?
>
> Thanks,
> Sijie
>
> On Wed, Apr 15, 2020 at 9:35 PM Shivji Kumar Jha 
> wrote:
>
> > Hi Sijie,
> >
> > If no one has picked this up, I would like to volunteer for this feature.
> >
> > Regards,
> > Shivji Kumar Jha
> > http://www.shivjijha.com/
> > +91 8884075512
> >
> >
> > On Thu, Apr 16, 2020 at 3:53 AM Sijie Guo  wrote:
> >
> > > I see. I wasn't sure that Raman is looking for this capability based on
> > his
> > > previous email.
> > >
> > > I do agree that decoupling the relationship between topic and schema
> can
> > > drive more use cases. It is a great feature to add.
> > >
> > > We will pick this up and come up a PIP for introducing this capability.
> > >
> > > Thanks,
> > > Sijie
> > >
> > > On Wed, Apr 15, 2020 at 4:25 AM Shivji Kumar Jha 
> > > wrote:
> > >
> > > > Hi Sijie,
> > > >
> > > > I second with Raman. Apart from PIP-43 and PIP-44 which ease schema
> > > > management, in my opinion, we should also loosely couple the
> > association
> > > > between topic and schema (or more precisely *type of data* on topic)
> > > which
> > > > is 1 to 1 as of now.
> > > >
> > > >1. The schema (or schema versions of one data type) could be
> grouped
> > > >into what Kafka calls *subject*.
> > > >2. The schema compatibility should then be done among schemas in
> the
> > > >same subject only.
> > > >3. One topic can associate with multiple schema subjects and have
> > > their
> > > >own evolution paths.
> > > >4. Similarly, one subject can also associate to multiple topics.
> > > >
> > > > *Use case:*
> > > > This feature would be handy when one needs different business models
> > in a
> > > > strictly ordered fashion. At the same time, these business models
> have
> > > > their own evolution paths too. As an example, an event sourcing
> system
> > > > could have events like customerCreated, customerAddressChanged,
> > > > customerInvoicePaid events etc required in order.
> > > >
> > > > The ideas presented above are picked from here
> > > > <
> > https://martin.kleppmann.com/2018/01/18/event-types-in-kafka-topic.html
> > > >.
> > > >
> > > > Regards,
> > > > Shivji Kumar Jha
> > > > http://www.shivjijha.com/
> > > > +91 8884075512
> > > >
> > > >
> > > > On Wed, Apr 15, 2020 at 2:27 AM Sijie Guo 
> wrote:
> > > >
> > > > > Hi Raman,
> > > > >
> > > > > The schema compatibility strategies were already there prior to
> > PIP-43.
> > > > >
> > > > > PIP-44 enhances the schema compatibility strategy support.
> > > > >
> > > > > Both of the changes are already landed in 2.5.0 release.
> > > > >
> > > > > Did you see any issues when you tryout this feature?
> > > > >
> > > > > - Sijie
> > > > >
> > > > > On Tue, Apr 14, 2020 at 8:35 AM rocketra...@gmail.com <
> > > > > rocketra...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Now that PIP-43 is released in 2.5.0, I wanted to follow up on
> the
> > > > > > messages below.
> > > > > >
> > > > > > What is remaining to be done in Pulsar to support having multiple
> > > > > > different types on one topic in Pulsar? Yi indicates below that
> > > PIP-43
> > > > > sets
> > > > > > the stage for this, but that the schema compatibility
> > implementation
> > > > > still
> > > > > > would need some work.
> > > > > >
> > > > > > Would this require another PIP, or just an issue to track the
> work?
> > > > > >
> > > > > > Regards,
> > > > > > Raman
> > > > > >
> > > > > > On 2019/09/16 01:32:39, Yi Tang  wrote:
> > > > > > > Hi rarma,
> > > > > > >
> > > > > > > It's a great and important feature, I think. This PIP requires
> > the
> > > > > > > compatibility check from bottom registry only and doesn't touch
> > the
> > > > > > > implementation detail. I think we should address this feature
> in
> > > the
> > > > > > > future, and this PIP provides the essential ability to
> implement
> > > it.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Yi
> > > > > > >
> > > > > > > rocketra...@gmail.com  于 2019年9月15日周日
> > > > 22:36写道：
> > > > > > >
> > > > > > > > I see a mention of compatibility in the PIP but with no
> > details.
> > > > The
> > > > > > docs
> > > > > > > > about schema compatibility state this:
> > > > > > > >
> > > > > > > > > Consequently, those events need to go in the same Pulsar
> > > > partition
> > > > > to
> > > > > > > > maintain order. This application can use ALWAYS_COMPATIBLE to
> > > allow
> > > > > > > > different kinds of events co-exist in the same topic.
> > > > > > > >
> > > > > > > > With this PIP, this limitation can be relaxed, and schema
> > > > > compatibility
> > > > > > > > should be able to be strengthened, since each type of message
> > on
> > > a
> > > > > > topic
> > > > > > > > can have its own schema, and compa

#TGIPulsar Episode 006 - Lifecycle of a Pulsar Message

2020-04-16 Thread Sijie Guo

Hi all,

A lot of new Pulsar users might be confused about backlog vs storage size,
TTL vs retention, why the bookies' disk is still filled up even the topic
is already deleted, and etc. In this week's live stream #TGIPulsar, we will
talk about the lifecycle of a Pulsar Message and help you understand how a
Pulsar message is produced, stored, acknowledged, and eventually deleted.

If you have similar confusions or want to learn about the underlying
mechanism, join us at 1 PM Pacific Time on Friday (04/17).
https://www.youtube.com/channel/UCywxUI5HlIyc0VEKYR4X9Pg/live

Thanks,
Sijie

Re: #TGIPulsar Episode 006 - Lifecycle of a Pulsar Message

2020-04-16 Thread lan.liang





Cool ! This topic is useful in work.


Thank you Sijie .






- lan.liang
On 4/16/2020 20:00，Sijie Guo wrote：
Hi all,

A lot of new Pulsar users might be confused about backlog vs storage size,
TTL vs retention, why the bookies' disk is still filled up even the topic
is already deleted, and etc. In this week's live stream #TGIPulsar, we will
talk about the lifecycle of a Pulsar Message and help you understand how a
Pulsar message is produced, stored, acknowledged, and eventually deleted.

If you have similar confusions or want to learn about the underlying
mechanism, join us at 1 PM Pacific Time on Friday (04/17).
https://www.youtube.com/channel/UCywxUI5HlIyc0VEKYR4X9Pg/live

Thanks,
Sijie

Re: [Discuss] Replace cgo client in Pulsar repo with Native Go Client

2020-04-16 Thread Jia Zhai

+1,

According to current status, Maybe 2.7.0 is a better time.

Best Regards.


Jia Zhai

Beijing, China

Mobile: +86 15810491983




On Tue, Apr 14, 2020 at 7:34 PM xiaolong ran 
wrote:

> > I'd rather wait a bit more for that, given that apps are fetching
> directly from github. Eg: change the Docs but leave the code there for
> one more release cycle.
>
> Cool, good idea, agree with you. We can remove this in next
> major Apache Pulsar release. Eg: Apache Pulsar 2.6.0 or 2.7.0.
>
> --
> Thanks
> Xiaolong Ran
>
>
> > 在 2020年4月14日，上午6:03，Matteo Merli  写道：
> >
> >> Delete the  CGo client code from Pulsar repo
> >
> > I'd rather wait a bit more for that, given that apps are fetching
> > directly from github. Eg: change the Docs but leave the code there for
> > one more release cycle.
> >
> > --
> > Matteo Merli
> > 
> >
> > On Mon, Apr 13, 2020 at 2:37 PM Yuva raj  wrote:
> >>
> >> +1
> >>
> >> On Mon, Apr 13, 2020, 7:49 PM xiaolong ran 
> wrote:
> >>
> >>> Hello all:
> >>>
> >>>
> >>> Pulsar has provided [pulsar-client-go](
> >>> https://github.com/apache/pulsar/tree/master/pulsar-client-go <
> >>> https://github.com/apache/pulsar/tree/master/pulsar-client-go>)
> >>> based on the CPP. However, CGO has many problems, which causes
> performance
> >>> to degrade
> >>> easily, makes it difficult to debug, and makes Pulsar hard to support
> the
> >>> Go community better.
> >>> Consequently, we rewrite the [Go Client](
> >>> https://github.com/apache/pulsar-client-go <
> >>> https://github.com/apache/pulsar-client-go>) based on native
> >>> Go to replace the CGO client and allow Pulsar to better support the
> >>> applications and integrations
> >>> of the Go community.
> >>>
> >>> Currently, pulsar-client-go 0.1.0 is released with multiple features
> >>> (producer, consumer, reader, and so on).
> >>> Except for schema, pulsar-client-go 0.1.0 can entirely replace CGO
> >>> client, so may I suggest replacing
> >>> CGO client with pulsar-client-go 0.1.0 in the next major Pulsar
> release.
> >>>
> >>> If agree, we need to do the following tasks:
> >>> Update the Go client doc on the Pulsar website
> >>> Remove the related contents of CGo client and add the related contents
> of
> >>> CPP client in release validation
> >>> Delete the  CGo client code from Pulsar repo
> >>> Celebrate after finishing this project
> >>>
> >>> --
> >>> Thanks
> >>> Xiaolong Ran
>
>

Re: [Discuss] Replace cgo client in Pulsar repo with Native Go Client

2020-04-16 Thread xiaolong ran

It seems there are no objections to this proposal, I will update 
native go client docs to Apache Pulsar web site and remove 
the validate contents of CGo client.

--
Thanks
Xiaolong Ran

> 在 2020年4月13日，下午10:19，xiaolong ran  写道：
> 
> Hello all:
> 
> 
> Pulsar has provided 
> [pulsar-client-go](https://github.com/apache/pulsar/tree/master/pulsar-client-go
>  )
> based on the CPP. However, CGO has many problems, which causes performance to 
> degrade 
> easily, makes it difficult to debug, and makes Pulsar hard to support the Go 
> community better. 
> Consequently, we rewrite the [Go 
> Client](https://github.com/apache/pulsar-client-go 
> ) based on native 
> Go to replace the CGO client and allow Pulsar to better support the 
> applications and integrations
> of the Go community.
> 
> Currently, pulsar-client-go 0.1.0 is released with multiple features 
> (producer, consumer, reader, and so on).
>  Except for schema, pulsar-client-go 0.1.0 can entirely replace CGO client, 
> so may I suggest replacing 
> CGO client with pulsar-client-go 0.1.0 in the next major Pulsar release.
> 
> If agree, we need to do the following tasks:
> Update the Go client doc on the Pulsar website
> Remove the related contents of CGo client and add the related contents of CPP 
> client in release validation
> Delete the  CGo client code from Pulsar repo
> Celebrate after finishing this project
> 
> --
> Thanks
> Xiaolong Ran

[Webinar] Common architectural patterns on Apache Pulsar

2020-04-16 Thread Sijie Guo

Hi Pulsarers,

We have a new Pulsar webinar upcoming at 10 AM Pacific Time on 04/28. Devin
Bost from Overstock will share how Overstock has leveraged Pulsar in
production. This webinar will help engineers and architects better
understand common real-world use cases for Pulsar and learn about helpful
patterns that simplify solutions to some of the most common pain points
with Pulsar.

If you are interested in this topic, definitely sign up using the following
link! Come and hang out with Devin!

https://zoom.us/webinar/register/7515867672387/WN_eV-yBB-aQ0yU4uqLvPpgjg

- Sijie

Re: [DISCUSS] PIP: Producer Send Message with Different Schema

Build failed in Jenkins: pulsar-website-build #966

Slack digest for #dev - 2020-04-16

Re: [DISCUSS] PIP: Producer Send Message with Different Schema

#TGIPulsar Episode 006 - Lifecycle of a Pulsar Message

Re: #TGIPulsar Episode 006 - Lifecycle of a Pulsar Message

Re: [Discuss] Replace cgo client in Pulsar repo with Native Go Client

Re: [Discuss] Replace cgo client in Pulsar repo with Native Go Client

[Webinar] Common architectural patterns on Apache Pulsar

9 matches

Site Navigation

Mail list logo

Footer information