[GitHub] [pulsar-helm-chart] lhotari opened a new pull request #202: Increase default initialDelaySeconds for Zookeeper probes to workaround ZOOKEEPER-3988
lhotari opened a new pull request #202: URL: https://github.com/apache/pulsar-helm-chart/pull/202 ### Motivation - When TLS is enabled for Zookeeper, NettyServerCnxnFactory will be used. It contains the issue https://github.com/apache/pulsar/issues/11070 / https://issues.apache.org/jira/browse/ZOOKEEPER-3988 ### Modification - as a workaround, increase the initialDelaySeconds from 10 to 20 to reduce the likelyhood of ZOOKEEPER-3988 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] lhotari commented on pull request #190: Bump to Pulsar 2.8.2
lhotari commented on pull request #190: URL: https://github.com/apache/pulsar-helm-chart/pull/190#issuecomment-1014250323 I have created #202 as a workaround for the Zookeeper issue (when TLS is enabled). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] lhotari opened a new pull request #203: Change default podManagementPolicy to Parallel for Zookeeper
lhotari opened a new pull request #203: URL: https://github.com/apache/pulsar-helm-chart/pull/203 ### Motivation - For speeding up startup, podManagementPolicy should be set to `Parallel` for Zookeeper ### Modifications - set `zookeeper.podManagementPolicy` to `Parallel` in the default `values.yaml` file. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] lhotari commented on pull request #202: Increase default initialDelaySeconds for Zookeeper probes to workaround ZOOKEEPER-3988
lhotari commented on pull request #202: URL: https://github.com/apache/pulsar-helm-chart/pull/202#issuecomment-1014259746 There's also #203 to speed up startup by using `podManagementPolicy=Parallel` for Zookeeper. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] tuteng commented on a change in pull request #190: Feature support oauth2 for node client
tuteng commented on a change in pull request #190:
URL: https://github.com/apache/pulsar-client-node/pull/190#discussion_r785740743
##
File path: package-lock.json
##
@@ -1,9204 +1,8 @@
{
"name": "pulsar-client",
"version": "1.6.0-rc.0",
- "lockfileVersion": 2,
Review comment:
Thanks, revert this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-client-node] tuteng merged pull request #190: Feature support oauth2 for node client
tuteng merged pull request #190: URL: https://github.com/apache/pulsar-client-node/pull/190 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [VOTE] PIP-135: Include MetadataStore backend for Etcd
+1 PengHui Li 于2022年1月17日周一 14:01写道： > +1 (binding) > > Regards, > Penghui > > On Sat, Jan 15, 2022 at 9:24 PM Joe F wrote: > > > +1 (binding) > > > > On Sat, Jan 15, 2022 at 4:46 AM Enrico Olivelli > > wrote: > > > > > Il Sab 15 Gen 2022, 09:10 tamer Abdlatif ha > > > scritto: > > > > > > > Will that affect the existing ZK metadata in a pulsar instance , When > > we > > > > upgrade from a lower version to 2.10 ? In other words, > > > > Do we need a metadate migration to switch from ZK to Etcd ? > > > > > > > > > > There is no need to migrate. > > > > > > Most probably the first release will bring this feature as non > production > > > ready and it will take some time to stabilise > > > > > > Enrico > > > > > > > > > > > > > > > > > Thanks > > > > Tamer > > > > > > > > > > > > > > > > On Fri, 14 Jan 2022, 22:52 Matteo Merli, > > wrote: > > > > > > > > > https://github.com/apache/pulsar/issues/13717 > > > > > > > > > > - > > > > > > > > > > ## Motivation > > > > > > > > > > Since all the pieces that composed the proposal in PIP-45 were > > finally > > > > > merged > > > > > and are currently ready for 2.10 release, it is now possible to add > > > other > > > > > metadata backends that can be used to support a BookKeeper + Pulsar > > > > > cluster. > > > > > > > > > > One of the popular systems that is most commonly used as an > > alternative > > > > of > > > > > ZooKeeper is Etcd, thus it makes sense to have this as the first > > > > > non-zookeeper > > > > > implementation. > > > > > > > > > > ## Goal > > > > > > > > > > Provide an Etcd implementation for the `MetadataStore` API. This > will > > > > allow > > > > > users to deploy Pulsar clusters using Etcd service for the metadata > > and > > > > it > > > > > will > > > > > not require the presence of ZooKeeper. > > > > > > > > > > > > > > > ## Implementation > > > > > > > > > > * Use the existing JEtcd Java client library for Etcd > > > > > * Extends the `AbstractBatchedMetadataStore` class, in order to > > reuse > > > > the > > > > >transparent batching logic that will be shared with the > ZooKeeper > > > > >implementation. > > > > > > > > > > Work in progress: https://github.com/apache/pulsar/pull/13225 > > > > > > > > > > -- > > > > > Matteo Merli > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] PIP-135: Include MetadataStore backend for Etcd
+1, Thanks for your work! - lan.liang On 01/17/2022 20:50，ZhangJian He wrote： +1 PengHui Li 于2022年1月17日周一 14:01写道： +1 (binding) Regards, Penghui On Sat, Jan 15, 2022 at 9:24 PM Joe F wrote: +1 (binding) On Sat, Jan 15, 2022 at 4:46 AM Enrico Olivelli wrote: Il Sab 15 Gen 2022, 09:10 tamer Abdlatif ha scritto: Will that affect the existing ZK metadata in a pulsar instance , When we upgrade from a lower version to 2.10 ? In other words, Do we need a metadate migration to switch from ZK to Etcd ? There is no need to migrate. Most probably the first release will bring this feature as non production ready and it will take some time to stabilise Enrico Thanks Tamer On Fri, 14 Jan 2022, 22:52 Matteo Merli, wrote: https://github.com/apache/pulsar/issues/13717 - ## Motivation Since all the pieces that composed the proposal in PIP-45 were finally merged and are currently ready for 2.10 release, it is now possible to add other metadata backends that can be used to support a BookKeeper + Pulsar cluster. One of the popular systems that is most commonly used as an alternative of ZooKeeper is Etcd, thus it makes sense to have this as the first non-zookeeper implementation. ## Goal Provide an Etcd implementation for the `MetadataStore` API. This will allow users to deploy Pulsar clusters using Etcd service for the metadata and it will not require the presence of ZooKeeper. ## Implementation * Use the existing JEtcd Java client library for Etcd * Extends the `AbstractBatchedMetadataStore` class, in order to reuse the transparent batching logic that will be shared with the ZooKeeper implementation. Work in progress: https://github.com/apache/pulsar/pull/13225 -- Matteo Merli
Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ
Thanks for creating this proposal Zike Yang. I have two ideas about it. 1. Instead of modifying the current protocol, we can use producer metadata to carry the init subscription 2. Please add auth for subscription creation when create topic by producer, otherwise, it will be easily attacked. Thanks, Hang Matteo Merli 于2022年1月12日周三 15:13写道： > > > If we want to hold that the DLQ is not a normal topic, then I can see > > why we would have a DLQ specific feature here. > > I think that, good or bad, the impression that users have that the DLQ > is not a "normal" topic comes from 2 factors: > 1. The experience with traditional messaging systems JMS and others > where the DLQ are handled in slightly different ways, compared to > other topics > 2. The name "DLQ" which in a way it's implying a "queue"... which can > be implemented on topic, using a subscription..
[GitHub] [pulsar-helm-chart] lhotari opened a new pull request #204: Upgrade cert-manager to 1.5.4
lhotari opened a new pull request #204: URL: https://github.com/apache/pulsar-helm-chart/pull/204 ### Motivation - Keep cert-manager version updated ### Modifications - upgrade cert-manager to 1.5.4 version - use apiVersion `cert-manager.io/v1` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] lhotari commented on pull request #204: Upgrade cert-manager to 1.5.4
lhotari commented on pull request #204: URL: https://github.com/apache/pulsar-helm-chart/pull/204#issuecomment-1014618466 upgrades up to 1.5.x are backwards compatible with cert-manager < 1.0. After 1.6.x, there's a breaking change explained in https://cert-manager.io/docs/installation/upgrading/upgrading-1.5-1.6/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] romainbrancourt commented on issue #189: Fail to build `pulsar-client` on M1
romainbrancourt commented on issue #189: URL: https://github.com/apache/pulsar-client-node/issues/189#issuecomment-1014732347 @csauvage Run this before running npm install, this may fix your problem :D ``` export CPLUS_INCLUDE_PATH="$CPLUS_INCLUDE_PATH:$(brew --prefix)/include" export LIBRARY_PATH="$LIBRARY_PATH:$(brew --prefix)/lib" ``` or ``` export CPLUS_INCLUDE_PATH="$(brew --prefix)/include" export LIBRARY_PATH="$(brew --prefix)/lib" ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[ANNOUNCE] New PMC Member - Lari Hotari
Hi - The Apache Pulsar Project Management Committee (PMC) has invited Lari Hotari (https://github.com/lhotari) as a member of the PMC and we are pleased to announce that he has accepted. He is very active in the community on GitHub and on the pulsar slack channel. Welcome Lari to the Apache Pulsar PMC Best Regards, Dave Fisher on behalf of the Pulsar PMC
Re: [ANNOUNCE] New PMC Member - Lari Hotari
Congratulations! Enrico Il Lun 17 Gen 2022, 22:03 frank ha scritto: > Congrats Lari - very well deserved > > Frank > > > > > > > On Mon, 17 Jan 2022 15:50:37 -0500 w...@apache.org > wrote > > Hi - > > The Apache Pulsar Project Management Committee (PMC) has invited Lari > Hotari > (https://github.com/lhotari) as a member of the PMC and we are pleased to > announce that he has accepted. > > He is very active in the community on GitHub and on the pulsar slack > channel. > > Welcome Lari to the Apache Pulsar PMC > > Best Regards, > Dave Fisher on behalf of the Pulsar PMC > > >
Re: [ANNOUNCE] New PMC Member - Lari Hotari
Congrats Lari - very well deservedFrank On Mon, 17 Jan 2022 15:50:37 -0500 w...@apache.org wrote Hi - The Apache Pulsar Project Management Committee (PMC) has invited Lari Hotari (https://github.com/lhotari) as a member of the PMC and we are pleased to announce that he has accepted. He is very active in the community on GitHub and on the pulsar slack channel. Welcome Lari to the Apache Pulsar PMC Best Regards, Dave Fisher on behalf of the Pulsar PMC
Re: [ANNOUNCE] New PMC Member - Lari Hotari
Congrats!! Well done, Lari. Definitely well deserved. Devin Bost (Sent from mobile) From: Enrico Olivelli Sent: Monday, January 17, 2022 3:07:25 PM To: pulsar-users Cc: dev Subject: Re: [ANNOUNCE] New PMC Member - Lari Hotari Congratulations! Enrico Il Lun 17 Gen 2022, 22:03 frank ha scritto: > Congrats Lari - very well deserved > > Frank > > > > > > > On Mon, 17 Jan 2022 15:50:37 -0500 w...@apache.org > wrote > > Hi - > > The Apache Pulsar Project Management Committee (PMC) has invited Lari > Hotari > (https://github.com/lhotari) as a member of the PMC and we are pleased to > announce that he has accepted. > > He is very active in the community on GitHub and on the pulsar slack > channel. > > Welcome Lari to the Apache Pulsar PMC > > Best Regards, > Dave Fisher on behalf of the Pulsar PMC > > >
Re: [DISCUSSION] PIP-133 Pulsar Functions Add API For Accessing Other Function States
Thanks for the feedback. I see your concerns.
I've been thinking about some ways to mitigate these concerns without expanding
the scope of this too much. First, I think it could be a good idea to limit
state access to just functions within the same namespace. This will at least
avoid any issues that might arise with different namespaces having different
state storage implementations.
Another thing we could consider is making states read-only by other functions.
This allows us to clearly define the owner of the data and avoid unexpected
issues with multiple functions trying to change the same state. It would limit
some potentially desirable functionality such as 2 functions being able to
increment the same counter, but keeping the data ownership clearly defined may
be more important for now.
Another option to think about could be adding a way to differentiate
public/private states or defining what other functions are allowed to access
certain states would ease security concerns. It would require some more
development time, though, since it would complicate the current implementation
a bit. We'd have to address how and where state access is defined.
Another option we could consider could be having a separate public state store
that all functions in a namespace have access to. It would be simple to
implement and would at least separate a function's private states from states
it wants other functions to have access to. Data ownership is a bit messy with
the public states for this solution, but it would at least provide a method of
sharing data that needs to be shared.
Let me know if you have any thoughts on any of these changes
From: Enrico Olivelli
Sent: Tuesday, January 11, 2022 1:45 PM
To: Dev
Subject: Re: [DISCUSSION] PIP-133 Pulsar Functions Add API For Accessing Other
Function States
Thank you for posting your PIP !
I am sharing some of Neng's concerns.
We should define clearly how security works.
Also, currently the function defines some "namespace" for the state
storage, and we recently added support for custom state storage
implementation. With this change each function will possibly access
other state storage namespaces (think about using a Database per each
tenant).
We should also state something about guarantees while accessing
multiple storages and/or about transactional (atomic?) access
Enrico
Il giorno mar 11 gen 2022 alle ore 21:38 Neng Lu ha scritto:
>
> Before we advance further, we first need to get on the same page of the
> pros and cons of allowing this feature.
>
> If functions can access (especially the write access) other functions'
> state, the data ownership will be a mess, isolation is broken and data
> security might be compromised.
>
>
>
>
>
> On Wed, Jan 5, 2022 at 3:45 PM Ethan Merrill
> wrote:
>
> > Original PIP:
> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fissues%2F13633&data=04%7C01%7Cethan.merrill%40legrand.us%7Cedd443d57f1e41c8588408d9d543508e%7C199686b5bef4496087867a6b1888fee3%7C1%7C0%7C637775307351827417%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DbCae%2FULTgUiV3pIrUQbOtzvPlilATc%2Bcn50I1eg0Iw%3D&reserved=0
> >
> > Pasted below for quoting convenience.
> >
> > -
> >
> > ## Motivation
> >
> > Certain uses of Pulsar functions could benefit from the ability to access
> > the states of other functions. Currently functions can only access their
> > own states, and so sharing information between functions requires other
> > solutions such as writing to a separate database.
> >
> > ## Goal
> >
> > The goal is to enable the ability for a function to access another
> > function's state. Given another function's tenant, namespace, and name, any
> > function should be able to access the other function's state for read and
> > write purposes. This PIP is not concerned with expanding the capabilities
> > of function states, It only deals with expanding access to function states.
> >
> > ## API Changes
> >
> > The Pulsar function API would be modified to have the function context
> > implement the following interface for accessing function states using a
> > tenant, namespace, and name.
> >
> > ```
> > public interface SharedContext {
> > /**
> > * Update the state value for the key.
> > *
> > * @param key name of the key
> > * @param value state value of the key
> > * @param tenant the state tenant name
> > * @param ns the state namespace name
> > * @param name the state store name
> > */
> > void putState(String key, ByteBuffer value, String tenant, String ns,
> > String name);
> >
> > /**
> > * Update the state value for the key, but don't wait for the
> > operation to be completed
> > *
> > * @param key name of the key
> > * @param value state value of the key
> > * @param tenant the state tenant name
> > * @pa
Re: [ANNOUNCE] New PMC Member - Lari Hotari
Congratulations! Devin Bost 于2022年1月18日周二 07:07写道： > Congrats!! Well done, Lari. Definitely well deserved. > > Devin Bost > (Sent from mobile) > > > From: Enrico Olivelli > Sent: Monday, January 17, 2022 3:07:25 PM > To: pulsar-users > Cc: dev > Subject: Re: [ANNOUNCE] New PMC Member - Lari Hotari > > Congratulations! > > > > Enrico > > Il Lun 17 Gen 2022, 22:03 frank ha scritto: > > > Congrats Lari - very well deserved > > > > Frank > > > > > > > > > > > > > > On Mon, 17 Jan 2022 15:50:37 -0500 w...@apache.org > > wrote > > > > Hi - > > > > The Apache Pulsar Project Management Committee (PMC) has invited Lari > > Hotari > > (https://github.com/lhotari) as a member of the PMC and we are pleased > to > > announce that he has accepted. > > > > He is very active in the community on GitHub and on the pulsar slack > > channel. > > > > Welcome Lari to the Apache Pulsar PMC > > > > Best Regards, > > Dave Fisher on behalf of the Pulsar PMC > > > > > > >
Re: [ANNOUNCE] New PMC Member - Lari Hotari
I am so glad to hear this news , lots of critical issues were solved by him and really help us a lot Thomas Aloys Zhang 於 2022年1月18日 週二 上午10:25寫道： > Congratulations! > > Devin Bost 于2022年1月18日周二 07:07写道： > >> Congrats!! Well done, Lari. Definitely well deserved. >> >> Devin Bost >> (Sent from mobile) >> >> >> From: Enrico Olivelli >> Sent: Monday, January 17, 2022 3:07:25 PM >> To: pulsar-users >> Cc: dev >> Subject: Re: [ANNOUNCE] New PMC Member - Lari Hotari >> >> Congratulations! >> >> >> >> Enrico >> >> Il Lun 17 Gen 2022, 22:03 frank ha scritto: >> >> > Congrats Lari - very well deserved >> > >> > Frank >> > >> > >> > >> > >> > >> > >> > On Mon, 17 Jan 2022 15:50:37 -0500 w...@apache.org> > >> > wrote >> > >> > Hi - >> > >> > The Apache Pulsar Project Management Committee (PMC) has invited Lari >> > Hotari >> > (https://github.com/lhotari) as a member of the PMC and we are pleased >> to >> > announce that he has accepted. >> > >> > He is very active in the community on GitHub and on the pulsar slack >> > channel. >> > >> > Welcome Lari to the Apache Pulsar PMC >> > >> > Best Regards, >> > Dave Fisher on behalf of the Pulsar PMC >> > >> > >> > >> >
Re: [VOTE] PIP-132: Include message header size when check maxMessageSize of non-batch message on the client side.
+1
On 2022/01/12 01:57:59 Haiting Jiang wrote:
> This is the voting thread for PIP-132. It will stay open for at least 48
> hours.
>
> https://github.com/apache/pulsar/issues/13591
>
> Pasted below for quoting convenience.
>
>
>
> ## Motivation
>
> Currently, Pulsar client (Java) only checks payload size for max message size
> validation.
>
> Client throws TimeoutException if we produce a message with too many
> properties, see [1].
> But the root cause is that is trigged TooLongFrameException in broker server.
>
> In this PIP, I propose to include message header size when check
> maxMessageSize of non-batch
> messages, this brings the following benefits.
> 1. Clients can throw InvalidMessageException immediately if properties takes
> too much storage space.
> 2. This will make the behaviour consistent with topic level max message size
> check in broker.
> 3. Strictly limit the entry size less than maxMessageSize, avoid sending
> message to bookkeeper failed.
>
> ## Goal
>
> Include message header size when check maxMessageSize for non-batch message
> on the client side.
>
> ## Implementation
>
> ```
> // Add a size check in
> org.apache.pulsar.client.impl.ProducerImpl#processOpSendMsg
> if (op.msg != null // for non-batch messages only
> && op.getMessageHeaderAndPayloadSize() > ClientCnx.getMaxMessageSize()) {
> // finish send op with InvalidMessageException
> releaseSemaphoreForSendOp(op);
> op.sendComplete(new PulsarClientException(new InvalidMessageException,
> op.sequenceId));
> }
>
>
> //
> org.apache.pulsar.client.impl.ProducerImpl.OpSendMsg#getMessageHeaderAndPayloadSize
>
> public int getMessageHeaderAndPayloadSize() {
> ByteBuf cmdHeader = cmd.getFirst();
> cmdHeader.markReaderIndex();
> int totalSize = cmdHeader.readInt();
> int cmdSize = cmdHeader.readInt();
> int msgHeadersAndPayloadSize = totalSize - cmdSize - 4;
> cmdHeader.resetReaderIndex();
> return msgHeadersAndPayloadSize;
> }
> ```
>
> ## Reject Alternatives
> Add a new property like "maxPropertiesSize" or "maxHeaderSize" in broker.conf
> and pass it to
> client like maxMessageSize. But the implementation is much more complex, and
> don't have the
> benefit 2 and 3 mentioned in Motivation.
>
> ## Compatibility Issue
> As a matter of fact, this PIP narrows down the sendable range. Previously,
> when maxMessageSize
> is 1KB, it's ok to send message with 1KB properties and 1KB payload. But with
> this PIP, the
> sending will fail with InvalidMessageException.
>
> One conservative way is to add a boolean config "includeHeaderInSizeCheck" to
> enable this
> feature. But I think it's OK to enable this directly as it's more reasonable,
> and I don't see good
> migration plan if we add a config for this.
>
> [1] https://github.com/apache/pulsar/issues/13560
>
> Thanks,
> Haiting Jiang
>
Re: [VOTE] PIP 131: Resolve produce chunk messages failed when topic level maxMessageSize is set
+1
On 2021/12/29 02:29:21 Haiting Jiang wrote:
> This is the voting thread for PIP-131. It will stay open for at least 48h.
>
> https://github.com/apache/pulsar/issues/13544
>
> The discussion thread is
> https://lists.apache.org/thread/c63d9s73j9x1m3dkqr3r38gyp8s7cwzf
>
> ## Motivation
>
> Currently, chunk messages producing fails if topic level maxMessageSize is
> set [1]. The root cause of this issue is because chunk message is using
> broker level maxMessageSize as chunk size. And topic level maxMessageSize is
> always <= broker level maxMessageSize. So once it is set, the on-going chunk
> message producing fails.
>
> ## Goal
>
> Resolve topic level maxMessageSize compatibility issue with chunking messages.
>
> ## Implementation
>
> Current best solution would be just skipping topic level maxMessageSize check
> in org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.
> Topic level maxMessageSize is introduced in [2], for the purpose of "easier
> to plan resource quotas for client allocation". And IMO this change will not
> bring further complex into this.
>
> ## Reject Alternatives
>
> Add a client side topic level maxMessageSize and keep it synced with broker.
>
> Required changes:
> - [client] Add a new field
> org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this
> client side topic level maxMessageSize.
> - [PulsarApi.proto] Add a new field maxMessageSize in the
> CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
> - [PulsarApi.proto] Add a new Command like
> CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> Further more, some other data consistency issues need be handled very
> carefully when maxMessageSize is updated.
> This alternative is complex but can also solve other topic level
> maxMessageSize issue [3] when batching is enabled (non-batching case is
> solved with PR [4]).
>
> [1] https://github.com/apache/pulsar/issues/13360
> [2] https://github.com/apache/pulsar/pull/8732
> [3] https://github.com/apache/pulsar/issues/12958
> [4] https://github.com/apache/pulsar/pull/13147
>
> Thanks,
> Haiting Jiang
>
Re: [ANNOUNCE] New PMC Member - Lari Hotari
Congratulations! BR, Haiting Jiang On 2022/01/18 02:26:32 陳智弘 wrote: > I am so glad to hear this news , lots of critical issues were solved by him > and really help us a lot > > Thomas > > Aloys Zhang 於 2022年1月18日 週二 上午10:25寫道： > > > Congratulations! > > > > Devin Bost 于2022年1月18日周二 07:07写道： > > > >> Congrats!! Well done, Lari. Definitely well deserved. > >> > >> Devin Bost > >> (Sent from mobile) > >> > >> > >> From: Enrico Olivelli > >> Sent: Monday, January 17, 2022 3:07:25 PM > >> To: pulsar-users > >> Cc: dev > >> Subject: Re: [ANNOUNCE] New PMC Member - Lari Hotari > >> > >> Congratulations! > >> > >> > >> > >> Enrico > >> > >> Il Lun 17 Gen 2022, 22:03 frank ha scritto: > >> > >> > Congrats Lari - very well deserved > >> > > >> > Frank > >> > > >> > > >> > > >> > > >> > > >> > > >> > On Mon, 17 Jan 2022 15:50:37 -0500 w...@apache.org >> > > >> > wrote > >> > > >> > Hi - > >> > > >> > The Apache Pulsar Project Management Committee (PMC) has invited Lari > >> > Hotari > >> > (https://github.com/lhotari) as a member of the PMC and we are pleased > >> to > >> > announce that he has accepted. > >> > > >> > He is very active in the community on GitHub and on the pulsar slack > >> > channel. > >> > > >> > Welcome Lari to the Apache Pulsar PMC > >> > > >> > Best Regards, > >> > Dave Fisher on behalf of the Pulsar PMC > >> > > >> > > >> > > >> > > >
Re: [ANNOUNCE] New PMC Member - Lari Hotari
Congratulations. Very well deserved. Lari is an inspiration! Regards, Shivji Kumar Jha http://www.shivjijha.com/ +91 8884075512 On Tue, 18 Jan 2022 at 08:20, Haiting Jiang wrote: > Congratulations! > > BR, > Haiting Jiang > > On 2022/01/18 02:26:32 陳智弘 wrote: > > I am so glad to hear this news , lots of critical issues were solved by > him > > and really help us a lot > > > > Thomas > > > > Aloys Zhang 於 2022年1月18日 週二 上午10:25寫道： > > > > > Congratulations! > > > > > > Devin Bost 于2022年1月18日周二 07:07写道： > > > > > >> Congrats!! Well done, Lari. Definitely well deserved. > > >> > > >> Devin Bost > > >> (Sent from mobile) > > >> > > >> > > >> From: Enrico Olivelli > > >> Sent: Monday, January 17, 2022 3:07:25 PM > > >> To: pulsar-users > > >> Cc: dev > > >> Subject: Re: [ANNOUNCE] New PMC Member - Lari Hotari > > >> > > >> Congratulations! > > >> > > >> > > >> > > >> Enrico > > >> > > >> Il Lun 17 Gen 2022, 22:03 frank ha scritto: > > >> > > >> > Congrats Lari - very well deserved > > >> > > > >> > Frank > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > On Mon, 17 Jan 2022 15:50:37 -0500 w...@apache.org< > w...@apache.org > > >> > > > >> > wrote > > >> > > > >> > Hi - > > >> > > > >> > The Apache Pulsar Project Management Committee (PMC) has invited > Lari > > >> > Hotari > > >> > (https://github.com/lhotari) as a member of the PMC and we are > pleased > > >> to > > >> > announce that he has accepted. > > >> > > > >> > He is very active in the community on GitHub and on the pulsar slack > > >> > channel. > > >> > > > >> > Welcome Lari to the Apache Pulsar PMC > > >> > > > >> > Best Regards, > > >> > Dave Fisher on behalf of the Pulsar PMC > > >> > > > >> > > > >> > > > >> > > > > > >
[GitHub] [pulsar-helm-chart] lhotari merged pull request #201: Add custom labels to all k8s objects in chart
lhotari merged pull request #201: URL: https://github.com/apache/pulsar-helm-chart/pull/201 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] lhotari commented on pull request #61: anti-affinity rules refactor and default image tags
lhotari commented on pull request #61: URL: https://github.com/apache/pulsar-helm-chart/pull/61#issuecomment-1015120620 @EladDolev good work! > use publishNotReadyAddresses correctly, expose it as a parameter, and remove deprecated alpha annotation this change would be valuable. Please send it as a separate PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] lhotari merged pull request #165: useV2WireProtocol for bookkeeper autorecovery
lhotari merged pull request #165: URL: https://github.com/apache/pulsar-helm-chart/pull/165 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-adapters] aditiwari01 commented on a change in pull request #31: [Issue #29] [pulsar-spark] Adding SparkPulsarReliableReceiver
aditiwari01 commented on a change in pull request #31:
URL: https://github.com/apache/pulsar-adapters/pull/31#discussion_r786475056
##
File path:
pulsar-spark/src/main/scala/org/apache/spark/streaming/pulsar/SparkStreamingReliablePulsarReceiver.scala
##
@@ -0,0 +1,181 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.streaming.pulsar
+
+import com.google.common.util.concurrent.RateLimiter
+import org.apache.pulsar.client.api._
+import org.apache.pulsar.client.impl.PulsarClientImpl
+import org.apache.spark.SparkConf
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.receiver.Receiver
+import org.slf4j.LoggerFactory
+
+import scala.collection.JavaConversions._
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ListBuffer
+
+/**
+ * Custom spark receiver to pull messages from Pubsub topic and push into
reliable store.
+ * If backpressure is enabled,the message ingestion rate for this receiver
will be managed by Spark.
+ *
+ * Following spark configurations can be used to control rates and block size
+ * spark.streaming.backpressure.enabled
+ * spark.streaming.backpressure.initialRate
+ * spark.streaming.receiver.maxRate
+ * spark.streaming.blockQueueSize: Controlling block size
+ * spark.streaming.backpressure.pid.minRate
+ *
+ * See Spark streaming configurations doc
+ * https://spark.apache.org/docs/latest/configuration.html#spark-streaming
+ *
+ * @param storageLevel Storage level to be used
+ * @param serviceUrlService URL to the Pubsub Cluster
+ * @param pulsarConfMap of PulsarConf containing keys from
`org.apache.pulsar.client.impl.conf.ConsumerConfigurationData`
+ * @param authenticationAuthentication object for authenticating
consumer to pulsar. Use `AuthenticationDisabled` is authentication is disabled.
+ * @param sparkConf Spark configs
+ * @param maxPollSize Max number of records to read by consumer
in one batch
+ * @param consumerName Consumer name to be used by receiver
+ * @param autoAcknowledge Acknowledge pubsub message or not
+ * @param rateMultiplierFactor Rate multiplier factor in case you want
to have more rate than what PIDRateEstimator suggests. > 1.0 is useful in case
+ * of spark dynamic allocation to utilise
extra resources. Keep it 1.0 if spark dynamic allocation is disabled.
+ */
+
+class SparkStreamingReliablePulsarReceiver(override val storageLevel:
StorageLevel,
+ val serviceUrl: String,
+ val pulsarConf: Map[String, Any],
+ val authentication: Authentication,
+ val sparkConf: SparkConf,
+ val maxPollSize: Int,
+ val consumerName: String,
+ val autoAcknowledge: Boolean = true,
+ val rateMultiplierFactor: Double =
1.0) extends Receiver[SparkPulsarMessage](storageLevel) {
+
+ private val maxRateLimit: Long =
sparkConf.getLong("spark.streaming.receiver.maxRate", Long.MaxValue)
+ private val blockSize: Int =
sparkConf.getInt("spark.streaming.blockQueueSize", maxPollSize)
+ private val blockIntervalMs: Long =
sparkConf.getTimeAsMs("spark.streaming.blockInterval", "200ms")
+
+ lazy val rateLimiter: RateLimiter = RateLimiter.create(math.min(
+sparkConf.getLong("spark.streaming.backpressure.initialRate",
maxRateLimit),
+maxRateLimit
+ ).toDouble)
+
+ val buffer: ListBuffer[SparkPulsarMessage] = new
ListBuffer[SparkPulsarMessage]()
+
+ private var pulsarClient: PulsarClient = null
+ private var consumer: Consumer[Array[Byte]] = null
+ private var consumerThread: Thread = null
+
+ var latestStorePushTime: Long = System.currentTimeMillis()
+
+ override def onStart(): Unit = {
+try {
+ pulsarClient =
PulsarClient.builder.serviceUrl(serviceUrl).authentication(authentication).build
+
