Re: [ANNOUNCE] New PMC Member - Lari Hotari
Congrats Lari ! Le lun. 17 janv. 2022 à 21:50, Dave Fisher a écrit : > Hi - > > The Apache Pulsar Project Management Committee (PMC) has invited Lari > Hotari > (https://github.com/lhotari) as a member of the PMC and we are pleased to > announce that he has accepted. > > He is very active in the community on GitHub and on the pulsar slack > channel. > > Welcome Lari to the Apache Pulsar PMC > > Best Regards, > Dave Fisher on behalf of the Pulsar PMC
Re: [ANNOUNCE] New PMC Member - Lari Hotari
Congratulations Lari. 🥂 On Tue, Jan 18, 2022, 4:13 PM Christophe Bornet wrote: > Congrats Lari ! > > Le lun. 17 janv. 2022 à 21:50, Dave Fisher a écrit : > > > Hi - > > > > The Apache Pulsar Project Management Committee (PMC) has invited Lari > > Hotari > > (https://github.com/lhotari) as a member of the PMC and we are pleased > to > > announce that he has accepted. > > > > He is very active in the community on GitHub and on the pulsar slack > > channel. > > > > Welcome Lari to the Apache Pulsar PMC > > > > Best Regards, > > Dave Fisher on behalf of the Pulsar PMC >
Re: [ANNOUNCE] New PMC Member - Lari Hotari
Congratulations. Penghui On Tue, Jan 18, 2022 at 6:54 PM sourav agrawal wrote: > Congratulations Lari. 🥂 > > On Tue, Jan 18, 2022, 4:13 PM Christophe Bornet > wrote: > > > Congrats Lari ! > > > > Le lun. 17 janv. 2022 à 21:50, Dave Fisher a écrit : > > > > > Hi - > > > > > > The Apache Pulsar Project Management Committee (PMC) has invited Lari > > > Hotari > > > (https://github.com/lhotari) as a member of the PMC and we are pleased > > to > > > announce that he has accepted. > > > > > > He is very active in the community on GitHub and on the pulsar slack > > > channel. > > > > > > Welcome Lari to the Apache Pulsar PMC > > > > > > Best Regards, > > > Dave Fisher on behalf of the Pulsar PMC > > >
Re: [VOTE] PIP 131: Resolve produce chunk messages failed when topic level maxMessageSize is set
Close the VOTE with 3 (+1) + 2 bindings and 0 (-1) bindings
Thanks,
Penghui
On Tue, Jan 18, 2022 at 10:30 AM Lin Lin wrote:
> +1
>
> On 2021/12/29 02:29:21 Haiting Jiang wrote:
> > This is the voting thread for PIP-131. It will stay open for at least
> 48h.
> >
> > https://github.com/apache/pulsar/issues/13544
> >
> > The discussion thread is
> https://lists.apache.org/thread/c63d9s73j9x1m3dkqr3r38gyp8s7cwzf
> >
> > ## Motivation
> >
> > Currently, chunk messages producing fails if topic level maxMessageSize
> is set [1]. The root cause of this issue is because chunk message is using
> broker level maxMessageSize as chunk size. And topic level maxMessageSize
> is always <= broker level maxMessageSize. So once it is set, the on-going
> chunk message producing fails.
> >
> > ## Goal
> >
> > Resolve topic level maxMessageSize compatibility issue with chunking
> messages.
> >
> > ## Implementation
> >
> > Current best solution would be just skipping topic level maxMessageSize
> check in
> org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.
> > Topic level maxMessageSize is introduced in [2], for the purpose of
> "easier to plan resource quotas for client allocation". And IMO this change
> will not bring further complex into this.
> >
> > ## Reject Alternatives
> >
> > Add a client side topic level maxMessageSize and keep it synced with
> broker.
> >
> > Required changes:
> > - [client] Add a new field
> org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this
> client side topic level maxMessageSize.
> > - [PulsarApi.proto] Add a new field maxMessageSize in the
> CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
> > - [PulsarApi.proto] Add a new Command like
> CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> > Further more, some other data consistency issues need be handled very
> carefully when maxMessageSize is updated.
> > This alternative is complex but can also solve other topic level
> maxMessageSize issue [3] when batching is enabled (non-batching case is
> solved with PR [4]).
> >
> > [1] https://github.com/apache/pulsar/issues/13360
> > [2] https://github.com/apache/pulsar/pull/8732
> > [3] https://github.com/apache/pulsar/issues/12958
> > [4] https://github.com/apache/pulsar/pull/13147
> >
> > Thanks,
> > Haiting Jiang
> >
>
Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ
+1 for adding the DLQ_init_sub to producer metadata so that we don't need to introduce a new field in CommandProducer, and the new field looks a little confusing Thanks, Penghui On Mon, Jan 17, 2022 at 10:19 PM Hang Chen wrote: > Thanks for creating this proposal Zike Yang. I have two ideas about it. > 1. Instead of modifying the current protocol, we can use producer > metadata to carry the init subscription > 2. Please add auth for subscription creation when create topic by > producer, otherwise, it will be easily attacked. > > Thanks, > Hang > > Matteo Merli 于2022年1月12日周三 15:13写道： > > > > > If we want to hold that the DLQ is not a normal topic, then I can see > > > why we would have a DLQ specific feature here. > > > > I think that, good or bad, the impression that users have that the DLQ > > is not a "normal" topic comes from 2 factors: > > 1. The experience with traditional messaging systems JMS and others > > where the DLQ are handled in slightly different ways, compared to > > other topics > > 2. The name "DLQ" which in a way it's implying a "queue"... which can > > be implemented on topic, using a subscription.. >
[GitHub] [pulsar-helm-chart] lhotari merged pull request #203: Change default podManagementPolicy to Parallel for Zookeeper
lhotari merged pull request #203: URL: https://github.com/apache/pulsar-helm-chart/pull/203 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] lhotari merged pull request #202: Increase default initialDelaySeconds for Zookeeper probes to workaround ZOOKEEPER-3988
lhotari merged pull request #202: URL: https://github.com/apache/pulsar-helm-chart/pull/202 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [DISCUSSION] PIP-133 Pulsar Functions Add API For Accessing Other Function States
I have concerns about security in this case and potential consistency
issues. We will need to define and implement some sort of ACLs system
first on top of state for this to make sense.
On Mon, Jan 17, 2022 at 5:52 PM Ethan Merrill
wrote:
> Thanks for the feedback. I see your concerns.
>
> I've been thinking about some ways to mitigate these concerns without
> expanding the scope of this too much. First, I think it could be a good
> idea to limit state access to just functions within the same namespace.
> This will at least avoid any issues that might arise with different
> namespaces having different state storage implementations.
>
> Another thing we could consider is making states read-only by other
> functions. This allows us to clearly define the owner of the data and avoid
> unexpected issues with multiple functions trying to change the same state.
> It would limit some potentially desirable functionality such as 2 functions
> being able to increment the same counter, but keeping the data ownership
> clearly defined may be more important for now.
>
> Another option to think about could be adding a way to differentiate
> public/private states or defining what other functions are allowed to
> access certain states would ease security concerns. It would require some
> more development time, though, since it would complicate the current
> implementation a bit. We'd have to address how and where state access is
> defined.
>
> Another option we could consider could be having a separate public state
> store that all functions in a namespace have access to. It would be simple
> to implement and would at least separate a function's private states from
> states it wants other functions to have access to. Data ownership is a bit
> messy with the public states for this solution, but it would at least
> provide a method of sharing data that needs to be shared.
>
> Let me know if you have any thoughts on any of these changes
>
>
> From: Enrico Olivelli
> Sent: Tuesday, January 11, 2022 1:45 PM
> To: Dev
> Subject: Re: [DISCUSSION] PIP-133 Pulsar Functions Add API For Accessing
> Other Function States
>
> Thank you for posting your PIP !
>
> I am sharing some of Neng's concerns.
> We should define clearly how security works.
>
> Also, currently the function defines some "namespace" for the state
> storage, and we recently added support for custom state storage
> implementation. With this change each function will possibly access
> other state storage namespaces (think about using a Database per each
> tenant).
>
> We should also state something about guarantees while accessing
> multiple storages and/or about transactional (atomic?) access
>
>
> Enrico
>
> Il giorno mar 11 gen 2022 alle ore 21:38 Neng Lu ha
> scritto:
> >
> > Before we advance further, we first need to get on the same page of the
> > pros and cons of allowing this feature.
> >
> > If functions can access (especially the write access) other functions'
> > state, the data ownership will be a mess, isolation is broken and data
> > security might be compromised.
> >
> >
> >
> >
> >
> > On Wed, Jan 5, 2022 at 3:45 PM Ethan Merrill
> > wrote:
> >
> > > Original PIP:
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fissues%2F13633&data=04%7C01%7Cethan.merrill%40legrand.us%7Cedd443d57f1e41c8588408d9d543508e%7C199686b5bef4496087867a6b1888fee3%7C1%7C0%7C637775307351827417%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DbCae%2FULTgUiV3pIrUQbOtzvPlilATc%2Bcn50I1eg0Iw%3D&reserved=0
> > >
> > > Pasted below for quoting convenience.
> > >
> > > -
> > >
> > > ## Motivation
> > >
> > > Certain uses of Pulsar functions could benefit from the ability to
> access
> > > the states of other functions. Currently functions can only access
> their
> > > own states, and so sharing information between functions requires other
> > > solutions such as writing to a separate database.
> > >
> > > ## Goal
> > >
> > > The goal is to enable the ability for a function to access another
> > > function's state. Given another function's tenant, namespace, and
> name, any
> > > function should be able to access the other function's state for read
> and
> > > write purposes. This PIP is not concerned with expanding the
> capabilities
> > > of function states, It only deals with expanding access to function
> states.
> > >
> > > ## API Changes
> > >
> > > The Pulsar function API would be modified to have the function context
> > > implement the following interface for accessing function states using a
> > > tenant, namespace, and name.
> > >
> > > ```
> > > public interface SharedContext {
> > > /**
> > > * Update the state value for the key.
> > > *
> > > * @param key name of the key
> > > * @param value state value of the key
> > > * @param tenant the state tenant name
> > > * @param ns the state namesp
[GitHub] [pulsar-helm-chart] pellicano opened a new pull request #205: Tiered Storage config
pellicano opened a new pull request #205: URL: https://github.com/apache/pulsar-helm-chart/pull/205 Fixes #72 ### Motivation Comply with Tiered Storage configuration. https://pulsar.apache.org/docs/en/concepts-tiered-storage/ https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/ ### Modifications Add a modified Helm Chart version from https://github.com/kafkaesque-io/pulsar-helm-chart. ### Verifying this change - [ ] Make sure that the change passes the CI checks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] ckdarby commented on pull request #205: Tiered Storage config
ckdarby commented on pull request #205: URL: https://github.com/apache/pulsar-helm-chart/pull/205#issuecomment-1015852264 Hey @michaeljmarshall! @lhotari & @cdbartholomew mentioned that we should ping you when we made this contribution :+1: Let us know if there is anything missing or what we can do to move this forward! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [VOTE] PIP-132: Include message header size when check maxMessageSize of non-batch message on the client side.
Thanks for your participation.
Close this vote with 3 (+1) bindings and 6 (+1) non-bindings, 0 (-1).
Thanks,
Haiting
On 2022/01/12 01:57:59 Haiting Jiang wrote:
> This is the voting thread for PIP-132. It will stay open for at least 48
> hours.
>
> https://github.com/apache/pulsar/issues/13591
>
> Pasted below for quoting convenience.
>
>
>
> ## Motivation
>
> Currently, Pulsar client (Java) only checks payload size for max message size
> validation.
>
> Client throws TimeoutException if we produce a message with too many
> properties, see [1].
> But the root cause is that is trigged TooLongFrameException in broker server.
>
> In this PIP, I propose to include message header size when check
> maxMessageSize of non-batch
> messages, this brings the following benefits.
> 1. Clients can throw InvalidMessageException immediately if properties takes
> too much storage space.
> 2. This will make the behaviour consistent with topic level max message size
> check in broker.
> 3. Strictly limit the entry size less than maxMessageSize, avoid sending
> message to bookkeeper failed.
>
> ## Goal
>
> Include message header size when check maxMessageSize for non-batch message
> on the client side.
>
> ## Implementation
>
> ```
> // Add a size check in
> org.apache.pulsar.client.impl.ProducerImpl#processOpSendMsg
> if (op.msg != null // for non-batch messages only
> && op.getMessageHeaderAndPayloadSize() > ClientCnx.getMaxMessageSize()) {
> // finish send op with InvalidMessageException
> releaseSemaphoreForSendOp(op);
> op.sendComplete(new PulsarClientException(new InvalidMessageException,
> op.sequenceId));
> }
>
>
> //
> org.apache.pulsar.client.impl.ProducerImpl.OpSendMsg#getMessageHeaderAndPayloadSize
>
> public int getMessageHeaderAndPayloadSize() {
> ByteBuf cmdHeader = cmd.getFirst();
> cmdHeader.markReaderIndex();
> int totalSize = cmdHeader.readInt();
> int cmdSize = cmdHeader.readInt();
> int msgHeadersAndPayloadSize = totalSize - cmdSize - 4;
> cmdHeader.resetReaderIndex();
> return msgHeadersAndPayloadSize;
> }
> ```
>
> ## Reject Alternatives
> Add a new property like "maxPropertiesSize" or "maxHeaderSize" in broker.conf
> and pass it to
> client like maxMessageSize. But the implementation is much more complex, and
> don't have the
> benefit 2 and 3 mentioned in Motivation.
>
> ## Compatibility Issue
> As a matter of fact, this PIP narrows down the sendable range. Previously,
> when maxMessageSize
> is 1KB, it's ok to send message with 1KB properties and 1KB payload. But with
> this PIP, the
> sending will fail with InvalidMessageException.
>
> One conservative way is to add a boolean config "includeHeaderInSizeCheck" to
> enable this
> feature. But I think it's OK to enable this directly as it's more reasonable,
> and I don't see good
> migration plan if we add a config for this.
>
> [1] https://github.com/apache/pulsar/issues/13560
>
> Thanks,
> Haiting Jiang
>
Re: [ANNOUNCE] New PMC Member - Lari Hotari
Congratulations! Thank you for the many ways you have helped build our Pulsar community! - Michael On Tue, Jan 18, 2022 at 6:21 AM PengHui Li wrote: > > Congratulations. > > Penghui > > On Tue, Jan 18, 2022 at 6:54 PM sourav agrawal > wrote: > > > Congratulations Lari. 🥂 > > > > On Tue, Jan 18, 2022, 4:13 PM Christophe Bornet > > wrote: > > > > > Congrats Lari ! > > > > > > Le lun. 17 janv. 2022 à 21:50, Dave Fisher a écrit : > > > > > > > Hi - > > > > > > > > The Apache Pulsar Project Management Committee (PMC) has invited Lari > > > > Hotari > > > > (https://github.com/lhotari) as a member of the PMC and we are pleased > > > to > > > > announce that he has accepted. > > > > > > > > He is very active in the community on GitHub and on the pulsar slack > > > > channel. > > > > > > > > Welcome Lari to the Apache Pulsar PMC > > > > > > > > Best Regards, > > > > Dave Fisher on behalf of the Pulsar PMC > > > > >
[GitHub] [pulsar-helm-chart] michaeljmarshall commented on a change in pull request #205: Tiered Storage config
michaeljmarshall commented on a change in pull request #205:
URL: https://github.com/apache/pulsar-helm-chart/pull/205#discussion_r787352821
##
File path: charts/pulsar/templates/broker-statefulset.yaml
##
@@ -279,6 +279,13 @@ spec:
path: broker/token
{{- end}}
{{- end}}
+ {{- if .Values.storageOffload.driver }}
+ {{- if eq .Values.storageOffload.driver "google-cloud-storage" }}
+ - name: gcp-service-account
Review comment:
Should we also add this volume to the container's `volumeMounts`?
##
File path: charts/pulsar/templates/broker-configmap.yaml
##
@@ -43,6 +43,80 @@ data:
zooKeeperSessionTimeoutMillis: "3"
statusFilePath: "{{ template "pulsar.home" . }}/status"
+ # Tiered storage settings
+ {{- if .Values.storageOffload.driver }}
+ {{- if eq .Values.storageOffload.driver "aws-s3" }}
+ managedLedgerOffloadDriver: "{{ .Values.storageOffload.driver }}"
+ s3ManagedLedgerOffloadBucket: "{{ .Values.storageOffload.bucket }}"
+ s3ManagedLedgerOffloadRegion: "{{ .Values.storageOffload.region }}"
+ AWS_ACCESS_KEY_ID: "{{ .Values.storageOffload.accessKey }}"
+ AWS_SECRET_ACCESS_KEY: "{{ .Values.storageOffload.accessSecret }}"
+ {{- if
.Values.storageOffload.managedLedgerOffloadAutoTriggerSizeThresholdBytes }}
+ # Pre 2.6.0 value
+ managedLedgerOffloadAutoTriggerSizeThresholdBytes: "{{
.Values.storageOffload.managedLedgerOffloadAutoTriggerSizeThresholdBytes }}"
Review comment:
Given that the Pulsar community just made 2.6 EOL, I don't think we need
to support it explicitly here. Note also that users can still make the chart
work for <=2.6 by adding values here: `.Values.broker.configData`.
##
File path: charts/pulsar/values.yaml
##
@@ -405,6 +405,69 @@ zookeeper:
usePolicy: true
maxUnavailable: 1
+## Tiered Storage
+##
+storageOffload: {}
+ ## General
+ ## ===
+ # bucket:
+ # region:
+ # maxBlockSizeInBytes: "6400"
+ # readBufferSizeInBytes: "100"
+ ## The following are default values for the cluster. They can be changed
+ ## on each namespace.
+ # managedLedgerOffloadDeletionLagMs: "1440"
+ # managedLedgerOffloadAutoTriggerSizeThresholdBytes: "-1" # disabled
+
+ # For AWS S3
+ # ==
+ # You must create an IAM account with access to the bucket and
+ # generate keys for that account.
+ #
+ # driver: aws-s3
+ # accessKey:
+ # accessSecret: # pragma: allowlist secret
+
+ # For S3 Compatible
+ # =
+ # Need to create access and secret key for S3 compatible service
+ #
+ # driver: aws-s3
+ # accessKey:
+ # accessSecret: # pragma: allowlist secret
+ # serviceEndpoint: host:port
+
+ # For Aure Blob
+ # =
+ # Need to create an Azure storage account and a blob containter (bucket)
+ # To retrieve key, see
https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal#code-try-1
+ #
+ # driver: azureblob
+ # storageAccount:
+ # storageAccountKey:
+
+
+ ## For Google Cloud Storage
+ ##
+ ## You must create a service account that has access to the objects in GCP
buckets
+ ## and upload its key as a JSON file to a secret.
+ ##
+ ## 1. Go to https://console.cloud.google.com/iam-admin/serviceaccounts
+ ## 2. Select your project.
+ ## 3. Create a new service account.
+ ## 4. Give the service account permission to access the bucket. For example,
+ ##the "Storage Object Admin" role.
+ ## 5. Create a key for the service account and save it as a JSON file.
+ ## 6. Save the JSON file in a secret:
+ ## kubectl create secret generic pulsar-gcp-sa-secret \
+ ## --from-file=kafkaesque-223201-f12856532197.json \
Review comment:
I think we should make this example name generic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-helm-chart] michaeljmarshall commented on a change in pull request #205: Tiered Storage config
michaeljmarshall commented on a change in pull request #205:
URL: https://github.com/apache/pulsar-helm-chart/pull/205#discussion_r787353261
##
File path: charts/pulsar/values.yaml
##
@@ -405,6 +405,69 @@ zookeeper:
usePolicy: true
maxUnavailable: 1
+## Tiered Storage
+##
+storageOffload: {}
+ ## General
+ ## ===
+ # bucket:
+ # region:
+ # maxBlockSizeInBytes: "6400"
+ # readBufferSizeInBytes: "100"
+ ## The following are default values for the cluster. They can be changed
+ ## on each namespace.
+ # managedLedgerOffloadDeletionLagMs: "1440"
+ # managedLedgerOffloadAutoTriggerSizeThresholdBytes: "-1" # disabled
+
+ # For AWS S3
+ # ==
+ # You must create an IAM account with access to the bucket and
+ # generate keys for that account.
+ #
+ # driver: aws-s3
+ # accessKey:
+ # accessSecret: # pragma: allowlist secret
+
+ # For S3 Compatible
+ # =
+ # Need to create access and secret key for S3 compatible service
+ #
+ # driver: aws-s3
+ # accessKey:
+ # accessSecret: # pragma: allowlist secret
+ # serviceEndpoint: host:port
+
+ # For Aure Blob
+ # =
+ # Need to create an Azure storage account and a blob containter (bucket)
+ # To retrieve key, see
https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal#code-try-1
+ #
+ # driver: azureblob
+ # storageAccount:
+ # storageAccountKey:
+
+
+ ## For Google Cloud Storage
+ ##
+ ## You must create a service account that has access to the objects in GCP
buckets
+ ## and upload its key as a JSON file to a secret.
+ ##
+ ## 1. Go to https://console.cloud.google.com/iam-admin/serviceaccounts
+ ## 2. Select your project.
+ ## 3. Create a new service account.
+ ## 4. Give the service account permission to access the bucket. For example,
+ ##the "Storage Object Admin" role.
+ ## 5. Create a key for the service account and save it as a JSON file.
+ ## 6. Save the JSON file in a secret:
+ ## kubectl create secret generic pulsar-gcp-sa-secret \
+ ## --from-file=kafkaesque-223201-f12856532197.json \
Review comment:
I think we should make this example file name generic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
