After many of my own attempts, research, digging through source code, and
speaking with folks in various channels in the community, I'm starting to
wonder if *anyone* has *ever* successfully gotten Avro to work with Python
Functions.
(I don't just mean ingesting a byte array with fastavro but actu
or which company?
>
> Can you take a look at my previous response to see if it answers the
> concern you raised?
>
> Thanks!
>
>
> On Wed, Jun 14, 2023 at 1:49 PM Devin Bost wrote:
>
> > > Hi,
> > >
> > > Are we proposing a change to break existing
+1 to this change.
> I am not sure enabling hostname verification > by default is a good idea
> because it will cause deployment failures for > many users. Hostname
> verification requires x509 cert changes and wildcard SNName in some cases
> and it's not easy when broker hostname patterns don't
> Hi,
>
> Are we proposing a change to break existing metrics compatibility
> (prometheus)? If that is the case then it's a big red flag as it will be a
> pain for any company to upgrade Pulsar as monitoring is THE most important
> part of the system and we don't even want to break compatibility fo
Great team work on all the flaky tests. It's much appreciated.
- Devin
On Wed, Apr 5, 2023, 4:39 PM Michael Marshall wrote:
> Thanks for fixing that test Heesung, and thanks for the report Nicolo!
>
> - Michael
>
> On Wed, Apr 5, 2023 at 2:17 PM Lari Hotari wrote:
> >
> > Thank you Heesung for
+1 since it can be pulled back up in git history if someone decides to do
something with it to improve it at a later time.
I also agree that it's a pain to maintain, and I don't know anyone using
it. I've gone through some of those code paths, and I was concerned about
divergence anyway.
- Devin
It looks like the purpose of the StreamingDispatcher was to improve
performance by implementing an improved readahead mechanism. Was it
actually able to accomplish this objective? If so, why do we need a
separate code path for it instead of updating the existing dispatcher to
use the improvement? I
Congrats!
Devin G. Bost
On Wed, Mar 29, 2023 at 6:38 AM ZhangJian He wrote:
> Congratulations!
>
> Thanks
> ZhangJian He
>
>
> On Wed, 29 Mar 2023 at 19:33, Haiting Jiang
> wrote:
>
> > Congratulations!
> >
> >
> > Haiting
> >
> > On Wed, Mar 29, 2023 at 5:29 PM Cong Zhao wrote:
> > >
> > >
+1
Devin G. Bost
On Tue, Mar 28, 2023 at 1:57 AM Enrico Olivelli wrote:
> +1 (binding)
>
> Enrico
>
> Il giorno mar 28 mar 2023 alle ore 04:59 Dave Fisher
> ha scritto:
> >
> > +1 (binding)
> >
> > Sent from my iPhone
> >
> > > On Mar 27, 2023, at 12:59 PM, Lari Hotari wrote:
> > >
> > > +1
> not in the client?
>
> The user experience around maintaining types/schemas between apps in
> > Pulsar is not good
>
>
> What are we comparing this to though? What would the ideal data developer
> workflow look like?
>
> Thanks,
>
> Elliot.
>
>
a DLQ or Retry
> topic because we'd like a way to handle content failures separately from
> other kinds of failures.
>
> Yes, I agree. It's not a field of DLQ.
>
> Thanks,
> Penghui
>
> On Thu, Nov 17, 2022 at 7:37 AM Devin Bost wrote:
>
> > I appreciat
ny of them and the
Pulsar Manager.
Thanks for bringing up this subject. You're making good points that we have
a gap in this area.
In your perspective, where do you think the biggest gaps are in terms of
functionality?
--
Devin Bost
Sent from mobile
Cell: 801-400-4602
On Mon, Feb 20, 2023,
Congrats!
--
Devin Bost
Sent from mobile
Cell: 801-400-4602
On Wed, Feb 1, 2023, 6:07 AM r...@apache.org
wrote:
> Congratulations!
>
> --
> Thanks
> Xiaolong Ran
>
> Qiang Huang 于2023年2月1日周三 10:16写道：
>
> > Congratulations !!!
> >
> > Zike Yang 于2023年
upport for schema, certainly at the edges, is a
> > > cornerstone for a data system. I also agree that it would be better to
> > > adopt existing standards rather than implement them in a bespoke
> manner.
> > >
> > > I'd be interested to hear your thought
ta problem wasn't necessarily caused by Pulsar.
What's the right way for us to address this problem?
--
Devin Bost
Sent from mobile
Cell: 801-400-4602
+1
On Wed, Oct 12, 2022, 7:49 AM Xiangying Meng wrote:
> +1
>
> Thanks,
> Xiangying
>
> On Wed, Oct 12, 2022 at 7:11 PM Haiting Jiang
> wrote:
>
> > +1
> >
> > Thanks,
> > Haiting
> >
> > On Wed, Oct 12, 2022 at 5:46 PM houxiaoyu wrote:
> > >
> > > +1
> > > Very useful.
> > >
> > > Michael Mar
; That sounds a lot of marketish. If you want to work on cloud-events
> integration, please do. I still don't see how's that related to Pulsar
> 3.0 discussion.
>
> --
> Matteo Merli
>
>
> On Tue, Oct 11, 2022 at 9:54 AM Devin Bost wrote:
> >
> > How abo
gt; > can
> > > lead to a fork. Are there certain kinds of breaking changes that are
> more
> > > acceptable than others?
> > >
> > > Also, if the forward looking plan is to never introduce breaking
> changes,
> > > when *would* we ever cut a
lease? Do we have any criteria on
what kinds of changes would necessitate cutting a new major release but
would still be considered acceptable by the community?
--
Devin Bost
Sent from mobile
Cell: 801-400-4602
On Sat, Oct 8, 2022, 2:14 PM Rajan Dhabalia wrote:
> This sounds like the curre
saved in the Pulsar
> Schema Registry in a way that non-Pulsar clients (like WebServices)
> can consume Pulsar messages
> 3) other
>
> I agree that supporting CloudEvents would be great in Pulsar and we
> should do something.
>
> If you have a real world use case to share
rties: Map
propertiesSet: bool
timestamp: long
--
Devin Bost
Sent from mobile
Cell: 801-400-4602
On Fri, Sep 2, 2022, 5:33 PM Devin Bost wrote:
> Hi recently discovered the discussion around creating a CloudEvent binding
> for Pulsar. https://github.com/cloudevents/spec/pull/237
>
>
vendors using it in
their products/services. "
As CloudEvents is gaining momentum within CNCF, this may become a problem.
Has their been any discussion around standardization and how we might meet
this requirement?
--
Devin Bost
Sent from mobile
Cell: 801-400-4602
So, based on this, it looks like we'd need to upgrade Pulsar to use BK
4.15.0 to get M1 compatibility?
What kind of changes will that introduce into Pulsar?
Devin G. Bost
On Fri, Apr 1, 2022 at 8:17 AM Nicolò Boschi wrote:
> Hi all,
>
> this is a follow-up discussion about the RocksDB upgrade
Thanks for all the feedback on this.
So, I'm looking into running functions in the Kubernetes runtime, and I'm
seeing another potential issue in terms of restricting function
authorization scope.
The function worker reads the token from the pulsar-admin call's
Authorization header when the functio
I support this change. Having different connectors in each project is
actually really confusing for new folks since it's not automatically clear
which adapter they should be using.
--
Devin G. Bost
On Thu, Feb 3, 2022, 1:17 PM Andrey Yegorov
wrote:
> Hello,
>
> I created a PR to remove Flink fr
Hi Everyone,
I was just informed that the first version of the Apache Beam PulsarIO
connector has been completed, and the contributor is looking for feedback.
This is a good opportunity for any feedback from the Pulsar community prior
to it getting merged into Beam master. Apache Beam is used for
I discovered that all functions running in process mode inherit the same
role from the token specified in the functions_worker.yml file. So, that
means that all functions in process mode have the same authorization scope,
which must be broad enough to ensure that all functions have the required
per
Congrats!! Well done, Lari. Definitely well deserved.
Devin Bost
(Sent from mobile)
From: Enrico Olivelli
Sent: Monday, January 17, 2022 3:07:25 PM
To: pulsar-users
Cc: dev
Subject: Re: [ANNOUNCE] New PMC Member - Lari Hotari
Congratulations!
Enrico
Il
s, but if some users
trust the Apache releases to be stable, that trust can be damaged if we're not
careful.
Devin Bost
(Sent from mobile)
From: Enrico Olivelli
Sent: Wednesday, December 15, 2021 5:29:53 AM
To: Dev
Subject: Re: Status of Pulsar 2.9.0 and st
how do we deal with this problem?
I'm also open to suggestions here.
--
Devin G. Bost
On Mon, Oct 25, 2021, 3:41 PM Enrico Olivelli wrote:
> Devin,
>
> Il Lun 25 Ott 2021, 23:05 Devin Bost ha scritto:
>
> > I've occasionally run into situations where I needed a list
I've occasionally run into situations where I needed a list of all the
topics in a cluster.
If I want to submit a PR to add something to one of the endpoints to return
a list of all topics in a cluster, what part of the API should that go
under?
e.g. bin/pulsar-admin tenants list-all-topics
+1 for Option 1, though I'd made the icons bigger and increase the color
contrast. Preferably, I'd like to see some additional minor color
variations of Option 1 (primarily to increase readability on small
devices), though I like the blue theme.
Also, these pictures only cover the desktop display.
Matteo,
Thanks for working on this. I've been wondering about this for a while now.
Is the expectation that a PR with proposed changes must already be created
before a PIP is added to the Wiki? I've felt a little confused about this
in the past when trying to start discussions or get feedback on
What are some of the most commonly asked questions or kinds of questions?
--
Devin G. Bost
On Mon, Jul 26, 2021, 1:34 AM Anonymitaet _
wrote:
> Hi all,
>
> Great idea.
>
> For the "contact" page, we can add "Pulsar community meeting info" (
> https://github.com/apache/pulsar/wiki/Community-Meet
I agree with this move.
Devin Bost
(Sent from mobile)
From: Sijie Guo
Sent: Friday, July 23, 2021 2:35:34 PM
To: Dev
Subject: Re: Moving questions from Slack to Stack Overflow
Aaron,
That is a very valuable point. +1
Just FYI, we have been encouraging
ma, so it's not clear to me if
they would be impacted.
Devin G. Bost
On Mon, Jul 19, 2021 at 12:41 PM Devin Bost wrote:
> > I think Sijie is referring to using KubernetesRuntime to deploy functions
> > where each function/source/sink runs as an independent statefulset in
> K8s
using.
Not everybody is using the KubernetesRuntime yet (especially since the Helm
charts aren't feature-complete), and it appears that those who aren't
running KubernetesRuntime would be impacted the most by this issue.
Devin G. Bost
On Mon, Jul 19, 2021 at 12:36 PM Devin Bost wrote:
&
> For example, if you are upgrading Flink from one version to the other
> version, you have to make a save point in the previous version for all
> the Flink jobs.
> Upgrade the Flink cluster and resume jobs in a new version.
>
>
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/
Is the client at least backwards compatible so that we could have all of
our teams upgrade functions before a broker upgrade is scheduled? We have
hundreds of functions running in production that are maintained by many
different teams and departments.
If upgrades of the functions need be all happe
+1.
I think these are great suggestions.
--
Devin G. Bost
On Mon, May 31, 2021, 2:30 AM Lari Hotari wrote:
> > The PMC can also assign members to a secur...@pulsar.apache.org mailing
> list.
>
> +1 for this plan.
>
> BR, Lari
>
>
> On Fri, May 28, 2021 at 2:24 AM Dave Fisher wrote:
>
> >
> >
e problem is ongoing. Multiple subsequent
> thread dump taken a few seconds apart are useful in detecting
> blocked/dead-locked threads. Heap dumps would also be useful, but those
> contain confidential details about the messages and environment
> credentials, so they shouldn't be
While researching https://github.com/apache/pulsar/issues/6054, I
discovered some key things that revealed a major performance issue that
manifests in high-velocity workloads:
1. The problem can be reproduced when batching is disabled on all Pulsar
functions
2. When a subscription is "frozen," it'
If we're going to use dependency injection on the Java side, I recommend
using Dagger2 since it's much easier to maintain than other Java DI
frameworks. I love DI, and I've thought about introducing it in Pulsar
already. However, with that said, DI doesn't necessarily remove
dependencies; it just m
Congrats!
Devin G. Bost
On Thu, May 6, 2021 at 12:36 PM Matteo Merli wrote:
> Hi everyone,
>
> The Project Management Committee (PMC) for Apache Pulsar has invited
> Lari Hotari (https://github.com/lhotari) to become a committer and we
> are pleased to announce that he has accepted.
>
> Please
to 2.7.2, so let's just ensure they're all included in the cut.
Devin G. Bost
On Fri, Apr 30, 2021 at 10:12 PM Devin Bost wrote:
> I just ran into a regression that appeared in the last 2-3 days of
> commits, and it's preventing my function from starting. I repo
fficial in case others are impacted. (Sometimes I wonder
if I'm the only one who runs into certain bugs, and it's possible that's
happening here, but we should double-check in case it affects a larger
audience.)
Devin G. Bost
On Fri, Apr 30, 2021 at 6:08 PM Devin Bost wrote:
>
If it's not too late, can we please also include:
https://github.com/apache/pulsar/pull/10389 ?
Devin G. Bost
On Fri, Apr 30, 2021 at 1:01 AM Enrico Olivelli wrote:
> Il giorno gio 29 apr 2021 alle ore 20:57 Sijie Guo
> ha scritto:
> >
> > Can you include these two issues? I have marked them
We definitely want https://github.com/apache/pulsar/pull/10413
included in 2.7.2 since it improves thread safety.
--
Devin G. Bost
On Fri, Apr 30, 2021, 1:01 AM Enrico Olivelli wrote:
> Il giorno gio 29 apr 2021 alle ore 20:57 Sijie Guo
> ha scritto:
> >
> > Can you include these two issues? I
I think it makes sense to add https://github.com/apache/pulsar/pull/10318
to 2.7.2 since it fixes an issue with a commit that's already part of 2.7.2
--
Devin G. Bost
On Fri, Apr 23, 2021, 2:29 AM Enrico Olivelli wrote:
> Hello all,
>
> I ran through the list of issues and PRs tagged for 2.7.2.
>> Could we add a system topic that has exactly one partition per broker?
> I think this depends on which type of topic we use for the events.
> If it is
> nonpersistent, I think this approach would work because the events
wouldn't
> outlive the broker.
So, what happens to events in a nonpersiste
> Could we add a system topic that has exactly one partition per broker?
Unfortunately, that just creates multiple single points of failure because
each partition has data that only exists on that partition. So, if any
partition fails, there's a gap, resulting in data loss.
--
Devin G. Bost
On Fr
>> > > Based on looking through the interceptor trait, I don't see a way to
>> > > trigger events based on auto created/deleted topics. For example,
>> when a
>> > > producer connects to a broker for a nonexistent topic (assuming auto
>> > topic
>> > > creation is allowed), a managed ledger, and th
Congrats!
--
Devin G. Bost
On Thu, Apr 22, 2021, 2:44 PM Enrico Olivelli wrote:
> Congratulations Addison!
>
>
> Enrico
>
> Il Gio 22 Apr 2021, 20:23 Sijie Guo ha scritto:
>
>> Hi everyone,
>>
>> On behalf of the PMC, I'm very happy to announce Addison Higham as a new
>> Pulsar committer.
>>
>
How computationally intensive is the responsibility of the Transaction
Coordinator?
--
Devin G. Bost
On Wed, Apr 21, 2021, 3:01 PM Sijie Guo wrote:
> Enrico - I was suggesting not even making it a standalone. TC is a
> component that really tied to Pulsar protocol and logic. We shouldn't do it
We've handled this with a function by just putting a sleep on incoming
messages.
For any flow that needs the pattern, we just drop the function into the
flow. So, Matteo has a good point.
--
Devin G. Bost
On Tue, Apr 20, 2021, 10:55 AM Matteo Merli wrote:
> The main point of the delayed deliver
I like the idea of doing this in the broker if it's feasible.
--
Devin G. Bost
On Mon, Apr 19, 2021, 7:42 AM Enrico Olivelli wrote:
> Penghui
>
> Il giorno lun 19 apr 2021 alle ore 15:11 PengHui Li
> ha scritto:
> >
> > Hi enrico,
> >
> > The delayed message feature is to support messages that
I wouldn't mind if we could include the debug statements from PR #10217 (
https://github.com/apache/pulsar/pull/10217) as part of this release,
though I realize that PR #10217 is currently tagged for 2.8.0.
I'm just waiting to get the flaky tests to clear. I expect them to clear
after a couple of a
Dear Pulsar community,
After inspecting many of the flaky tests in Pulsar and performing some
research on the issue, I discovered that there are some patterns that will
allow us to remedy many of our flaky tests. I bring your attention to this
publication:
Luo, Q., Hariri, F., Eloussi, L., & Mari
tests look good, I'll submit a PR to Apache Pulsar.
>
> Devin G. Bost
>
>
> On Mon, Apr 5, 2021 at 8:50 PM Devin Bost wrote:
>
>> It appears that the reason the router classes are singletons is to ensure
>> the clock behavior is consistent across producers.
&
PR to Apache Pulsar.
Devin G. Bost
On Mon, Apr 5, 2021 at 8:50 PM Devin Bost wrote:
> It appears that the reason the router classes are singletons is to ensure
> the clock behavior is consistent across producers.
> I wonder if we might be able to avoid creating a breaking change by
>
On Mon, Apr 5, 2021 at 6:01 PM Devin Bost wrote:
> Jerry,
>
> Thanks for the information and for the links!
> Thanks also for the suggestion to split the PR into two parts. That will
> make it easier to get some of this work completed in time for the 2.8.0
> release.
>
&
//github.com/apache/pulsar/pull/7159
>
> or more specifically what is done for functions here.
>
>
> https://github.com/apache/pulsar/pull/7159/files#diff-427711b7c1c3b9f17cd1c02bb9ceb942af454676dd97902a43e910f645febc6d
>
>
> Perhaps you can help revive this work!
>
>
REST call.
If an interceptor wasn't provided, during conversion of the ProducerConfig
to the protobuf ProducerSpec, the protobuf defaults will take effect (since
protobuf doesn't allow null primitives), which will then be picked up when
the producer is created.
Is that right?
Devin G. Bost
If we use the interceptor approach, does that mean a cluster-admin would
need to write custom code in order to override the function producer
settings? I worry about adding additional burden to cluster administration.
--
Devin G. Bost
On Sat, Apr 3, 2021, 8:26 AM Devin Bost wrote:
> Hi
Check out the instructions by Lari Hotari here about how to setup your
personal Github CI:
https://markmail.org/message/xapp7aguh44osqhm
I've been working through similar issues as I've been facing a
NullPointerException (in a PR of mine) that consistently occurs in Github
but never occurs locally
we will have some new fields to `Function.proto`.
>
> Best,
>
> Rui
> 在 2021年4月2日 +0800 AM10:13，Devin Bost ，写道：
> > What would be some of the additional benefits of using the interceptor
> > approach?
> >
> > --
> > Devin G. Bost
> >
> > On Thu,
n changes the function config defaults and wants
> existing functions to utilize those configs that have changed, the admin
> can just update those configs of the functions through the regular update
> mechanism for functions we have today.
>
>
>
> On Thu, Apr 1, 2021 at 4:5
Would it be better to just ignore WorkerConfig defaults during function
update? We could still update the function producer behavior through the
producerConfig passed as a REST parameter, but we'd avoid the edge case I
mentioned.
Devin G. Bost
On Thu, Apr 1, 2021 at 5:42 PM Devin Bost
customize
> the function config however you want.
>
> Best,
>
> Jerry
>
>
>
> On Thu, Apr 1, 2021 at 2:08 PM Devin Bost wrote:
>
> > *Cluster-Wide and Function-Specific Producer Defaults*
> >
> >
> >
> >
> > * - Status: Proposal- Author
erceptor method gets called. For example, the
> "regsterFunctionInterceptor()" method I provided above will be called
> towards the beginning of the registerFunction and allow you to customize
> the function config however you want.
>
> Best,
>
> Jerry
>
>
>
*Cluster-Wide and Function-Specific Producer Defaults*
* - Status: Proposal- Author: Devin Bost (with guidance from Jerry Peng)-
Pull Request: https://github.com/apache/pulsar/pull/9987
<https://github.com/apache/pulsar/pull/9987>- Mailing List discussion: -
Release: 2.8.0*
*Moti
Wow, nice work!
I'm currently working on adding a feature for Pulsar functions, so I'll be
sure to update the spreadsheet once I'm done with it.
--
Devin G. Bost
On Thu, Mar 25, 2021, 4:29 PM Sijie Guo wrote:
> Jennifer - Great work! Thank you for driving that!
>
> - Sijie
>
> On Tue, Mar 23,
ue
>> >>>> content on pulsar.apache.org <http://pulsar.apache.org/> and have it
>> >>> take
>> >>>> a 'Getting Started' approach.
>> >>>>
>> >>>>> On Feb 9, 2021, at 1:13 PM, Jonathan Ellis
>
Hi Jonathan,
Thanks for giving thought to this subject. I think we can all agree that
increasing public awareness of the momentum and community strength behind
Pulsar is a common goal among members of the Pulsar community. I'm sure
many of us have run into similar challenges with increasing adopti
Thanks for working on this.
--
Devin G. Bost
On Mon, Jan 25, 2021, 10:05 AM Lari Hotari wrote:
> Dear Pulsar community members,
>
> The current test suites in Apache Pulsar build contain flaky tests that
> make the builds fail very frequently.
>
> To address the issues caused by flaky tests, I
+1
Devin G. Bost
On Tue, Nov 10, 2020 at 10:43 PM Lan Liang wrote:
> +1
>
>
>
>
>
>
> Best Regards,
> Lan Liang
> On 11/11/2020 13:36，Shivji Kumar Jha wrote：
> +1
>
> Regards,
> Shivji Kumar Jha
> http://www.shivjijha.com/
> +91 8884075512
>
>
> On Wed, Nov 11, 2020 at 7:20 AM Jia Zhai wrote:
It occurred to me that we don't yet have a great solution for replicating
function workloads when running Pulsar with geo-replication.
There are a couple of concerns that I mention below, addressing failover
and varying function behavior.
1. Failover. There are challenges with running functions i
I apologize for not fully understanding the context here, but is the
concern about using the existing function architecture the complexity of
needing two sequential operations in a function flow to be synchronous with
respect to transactions, such as to avoid race conditions and issues with
paralle
This is awesome!
--
Devin G. Bost
On Thu, Feb 13, 2020 at 7:03 PM PengHui Li wrote:
> Great! 👍
> On Feb 14, 2020, 08:43 +0800, Sijie Guo , wrote:
> > Sorry for the late update here. INFRA just set up the TOKEN correctly for
> > us.
> >
> > So the `pulsarbot` is working as expected now. You can
What tools are the rest of you using to run the C++ tests locally?
I'm using IntelliJ as my IDE, but it doesn't appear that there's a good C++
plugin available.
Devin G. Bost
I've noticed that there are really two different types of discussions that
take place in the Pulsar #general channel. Most of them appear to be new
users, but that can make it tricky to have focused developer discussions
since the velocity of new user questions is high. The email groups are
already
Sijie,
Thanks for working on this!
--
Devin G. Bost
On Wed, Feb 5, 2020, 12:54 AM Sijie Guo wrote:
> Hi all,
>
> After we moved CI from Jenkins to Github Actions, only committers have the
> ability to trigger re-running the failed checks. It is inconvenient for
> contributors.
>
> As you might
I'm working on fixing the unstable tests (
https://github.com/apache/pulsar/issues/6137), and I've learned that many
of the unstable tests appear to be related to timing issues or the use of
Thread.sleep.
While investigating the issue, I noticed a commonly used method in the
tests with something o
83 matches
Mail list logo
