[GitHub] [pulsar] dineshkothuru2020 created a discussion: Namespace isolation policy :: No broker was available to own persistent

[GitBox]

GitHub user dineshkothuru2020 created a discussion: Namespace isolation policy 
:: No broker was available to own persistent

Appreciate your help to resolve this issues: 

Thanks in Advance :-)

Created an single plusar cluster setup with four broker services running on 
four different servers, my intend is to point the applicaiton to the broker 
running on same server, following are the steps has been followed

Note : Customize ports are used for zookeeper,bookie and broker instead of 
default ports

1) Create zookeeper setup on 3 servers
2) Initilize the cluster setup with zookeeper,configruation (used zookeeper ip 
and port), webservice and broker url pointed to one of the server where broker 
service is running
3) create the tenants and namespaces,
4) created an namespaces isolation policy, assigned the primary and secondary 
broker url

try to consume topic on the namespace using the pulsar-client tool for which 
ns-islation policy is created  and attached, following error is thrown in 
console,

./pulsar  initialize-cluster-metadata --cluster cluster1 --zookeeper  
server1.com:8041  --configuration-store server1.com:8041  --web-service-url  
http://server1.com:8008 --broker-service-url pulsar://server1.com:8050

./pulsar-daemon start broker
./pulsar-admin tenants create testtenant
./pulsar-admin namespaces create testtenant/ns1

./pulsar-admin ns-isolation-policy set --auto-failover-policy-params 
min_limit=1,usage_threshold=100,isPrimary=true --auto-failover-policy-type 
min_available --namespaces "testtenant/ns1"  --primary 
"server1.com:8008,server2.com:8008" --secondary 
"server3.com:8008,server4.com:8008" cluster1 ns-cluster1

Error

07:04:19.842 [pulsar-client-io-1-1] INFO  
org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xbd927667, L:/:59344 
- R:server/x:8050]] Connected to server
07:04:19.853 [pulsar-client-io-1-1] ERROR 
org.apache.pulsar.client.impl.ClientCnx - [id: 0xbd927667, L:/x:59344 - 
R:server1/x:8050] Close connection because received internal-server error 
No broker was available to own persistent://testtenant/ns1/11
07:04:19.854 [pulsar-client-io-1-1] WARN  
org.apache.pulsar.client.impl.BinaryProtoLookupService - 
[persistent://testtenant/ns1/11] failed to send lookup request : 
org.apache.pulsar.client.api.PulsarClientException$LookupException: No broker 
was available to own persistent://testtenant/ns1/11
07:04:19.854 [pulsar-client-io-1-1] WARN  
org.apache.pulsar.client.impl.BinaryProtoLookupService - 
[persistent://testtenant/ns1/11] lookup failed : 
org.apache.pulsar.client.api.PulsarClientException$LookupException: No broker 
was available to own persistent://testtenant/ns1/11
java.util.concurrent.CompletionException: 
org.apache.pulsar.client.api.PulsarClientException$LookupException: No broker 
was available to own persistent://testtenant/ns1/11
at 
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
 ~[?:1.8.0_252]
at 
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
 ~[?:1.8.0_252]
at 
java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:661) 
~[?:1.8.0_252]
at 
java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646)
 ~[?:1.8.0_252]
at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) 
[?:1.8.0_252]
at 
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
 [?:1.8.0_252]
at 
org.apache.pulsar.client.impl.ClientCnx.handleLookupResponse(ClientCnx.java:474)
 [org.apache.pulsar-pulsar-client-original-2.6.0.jar:2.6.0]
at 
org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:131)
 [org.apache.pulsar-pulsar-common-2.6.0.jar:2.6.0]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
 [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
 [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
 [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at 
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
 [io.netty-netty-codec-4.1.48.Final.jar:4.1.48.Final]
at 
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
 [io.netty-netty-codec-4.1.48.Final.jar:4.1.48.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
 [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
 [io.netty-netty-transport-4.1.48.Final.jar:4.1.48

[GitHub] [pulsar] hpvd added a comment to the discussion: Arguments for "why pulsar is secure?"

[GitBox]

GitHub user hpvd added a comment to the discussion: Arguments for "why pulsar 
is secure?"

this is how the summary from link above looks like:
![2022-10-13_14h16_24](https://user-images.githubusercontent.com/5681880/195593724-b137ecc2-dc7f-43e6-a3ed-cfa348692b2c.png)



GitHub link: 
https://github.com/apache/pulsar/discussions/18754#discussioncomment-4321235


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] hpvd added a comment to the discussion: Arguments for "why pulsar is secure?"

[GitBox]

GitHub user hpvd added a comment to the discussion: Arguments for "why pulsar 
is secure?"

some details:
![2022-10-13_14h18_45](https://user-images.githubusercontent.com/5681880/195594313-518801c3-1fbd-42b5-bb7e-b1b4a9f62af6.png)


GitHub link: 
https://github.com/apache/pulsar/discussions/18754#discussioncomment-4321236


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] hpvd added a comment to the discussion: Arguments for "why pulsar is secure?"

[GitBox]

GitHub user hpvd added a comment to the discussion: Arguments for "why pulsar 
is secure?"

would be very interesting to see, how an updated helm chart with
- latest pulsar image (2.9.3 -> 2.10.1 or even 2.11)
  https://github.com/apache/pulsar
- latest prometheus image (2.17.2 -> 2.39.1)
  https://github.com/prometheus/prometheus
- latest streamnative/pulsar grafana image (0.0.16 -> no new release but in 
repository are some commits with updates)
  https://github.com/streamnative/apache-pulsar-grafana-dashboard
 
would/will perform **in the same scan.**
=> is the update already the big part of the solution?

GitHub link: 
https://github.com/apache/pulsar/discussions/18754#discussioncomment-4321240


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] eolivelli added a comment to the discussion: Arguments for "why pulsar is secure?"

[GitBox]

GitHub user eolivelli added a comment to the discussion: Arguments for "why 
pulsar is secure?"

@hpvd  thanks for opening this issue. I think that your point is really valid.

I think that it is appropriate to start this discussion on the mailing lists 
(dev@pulsar.apache.org), as that's the place we run broad discussions.


Regarding this issue:
this issue is more about the "Apache Pulsar Helm Chart", that is on 
https://github.com/apache/pulsar-helm-chart 
**maybe** it is better to post your findings there.

Regarding the core Pulsar package we try to keep all the dependencies as 
up-to-date as possible and to cut releases as soon as problems are discovered 
and reported to the PMC.
Unfortunately the Pulsar Helm Chart is not very active and this is way the 
default versions, even for the Pulsar docker image are pretty out-dated and we 
should upgrade



GitHub link: 
https://github.com/apache/pulsar/discussions/18754#discussioncomment-4321238


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] hpvd added a comment to the discussion: Arguments for "why pulsar is secure?"

[GitBox]

GitHub user hpvd added a comment to the discussion: Arguments for "why pulsar 
is secure?"

before writing this issue, I had a conversation with Apache Security team,
that it is no problem to contribute this as normal issue:

> we'd warmly welcome   if you'd contribute this finding as a normal bug
> report and/or patch to the project. Since outdated dependencies
> without further analysis are quite public, there is no need to use this
> private reporting mechanism in that case.

GitHub link: 
https://github.com/apache/pulsar/discussions/18754#discussioncomment-4321237


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] hpvd added a comment to the discussion: Arguments for "why pulsar is secure?"

[GitBox]

GitHub user hpvd added a comment to the discussion: Arguments for "why pulsar 
is secure?"

@eolivelli thanks for your comment.

Just opened an issue also in the repository of pulsars helm chart:
https://github.com/apache/pulsar-helm-chart/issues/294

GitHub link: 
https://github.com/apache/pulsar/discussions/18754#discussioncomment-4321239


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] hpvd added a comment to the discussion: Arguments for "why pulsar is secure?"

[GitBox]

GitHub user hpvd added a comment to the discussion: Arguments for "why pulsar 
is secure?"

just as background info, the security scanner used by artifacthub, providing 
results shown above
is trivy, **so all the finding should be pretty valid**.

For details, see:
https://artifacthub.io/docs/topics/security_report/

and trivy
https://github.com/aquasecurity/trivy

GitHub link: 
https://github.com/apache/pulsar/discussions/18754#discussioncomment-4321241


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] hpvd added a comment to the discussion: Arguments for "why pulsar is secure?"

[GitBox]

GitHub user hpvd added a comment to the discussion: Arguments for "why pulsar 
is secure?"

there is also an easy to use **github action for scanning with trivy**
-  the complete repository,
-  pull requests,
- docker container
- IaC
- etc.
 
=> Maybe this is interesting to integrate this directly into the CI pipeline...
See Readme of https://github.com/aquasecurity/trivy-action

GitHub link: 
https://github.com/apache/pulsar/discussions/18754#discussioncomment-4321242


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] hpvd added a comment to the discussion: Arguments for "why pulsar is secure?"

[GitBox]

GitHub user hpvd added a comment to the discussion: Arguments for "why pulsar 
is secure?"

number of vulnerabilities is still growing:

Today images in Pulsars helm chart crosses the 1k:
**1005 vulnerabilities (683 fixable) have been detected in this package's 
images**.
see https://artifacthub.io/packages/helm/apache/pulsar?modal=security-report

Good to know there was hard work done during last week and there will be soon a 
major update
with updated pulsar (2.10.2) and a new monitoring stack
https://github.com/apache/pulsar-helm-chart/issues/326

Details and voting: 
https://lists.apache.org/thread/b02r3605jvmom23p1ccjbzsfwrq5tkjv

GitHub link: 
https://github.com/apache/pulsar/discussions/18754#discussioncomment-4321243


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] hpvd added a comment to the discussion: Arguments for "why pulsar is secure?"

[GitBox]

GitHub user hpvd added a comment to the discussion: Arguments for "why pulsar 
is secure?"

Latest security analysis show
- a stunning step in reducing the number of included vulnerabilities (minus 
85%!)
  - v2.9.4 with pulsar 2.9.3 **1024 vulnerabilities (698 fixable) have been 
detected in this package's images.**
https://artifacthub.io/packages/helm/apache/pulsar/2.9.4?modal=security-report
  - v3.0.0 with pulsar 2.10.2 **136 vulnerabilities (79 fixable) have been 
detected in this package's images.**
https://artifacthub.io/packages/helm/apache/pulsar?modal=security-report

GitHub link: 
https://github.com/apache/pulsar/discussions/18754#discussioncomment-4321245


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] hpvd added a comment to the discussion: Arguments for "why pulsar is secure?"

[GitBox]

GitHub user hpvd added a comment to the discussion: Arguments for "why pulsar 
is secure?"

- on the other hand
  - the number of fixable vulnerabilities with a severity of CRITICAL has risen 
from 1 to 4 (plus 300%!)
 
![2022-11-04_10h05_52](https://user-images.githubusercontent.com/5681880/199942132-2574e544-bf50-4355-89a7-e8e7a2b32a9c.png)

![2022-11-04_10h07_06](https://user-images.githubusercontent.com/5681880/199942169-ad0a3b71-e77f-4035-b2d2-8743e14b64ee.png)





GitHub link: 
https://github.com/apache/pulsar/discussions/18754#discussioncomment-4321246


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] hpvd added a comment to the discussion: Arguments for "why pulsar is secure?"

[GitBox]

GitHub user hpvd added a comment to the discussion: Arguments for "why pulsar 
is secure?"

-
   - very old fixable and already reported vulnerabilities (up to 9 years old) 
are still included in pulsar image:
   
![2022-11-04_09h21_54](https://user-images.githubusercontent.com/5681880/199937604-17826ab6-0d4d-469d-a1cd-7df6019138bb.png)


edit: just opened a separate issue for this: 
https://github.com/apache/pulsar/issues/18338

GitHub link: 
https://github.com/apache/pulsar/discussions/18754#discussioncomment-4321247


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] hpvd added a comment to the discussion: Arguments for "why pulsar is secure?"

[GitBox]

GitHub user hpvd added a comment to the discussion: Arguments for "why pulsar 
is secure?"

new Helm chart v 3.0 on the base of Pulsar 2.10.2 was just released:
https://github.com/apache/pulsar-helm-chart/releases/tag/pulsar-3.0.0


GitHub link: 
https://github.com/apache/pulsar/discussions/18754#discussioncomment-4321244


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] github-actions[bot] added a comment to the discussion: Arguments for "why pulsar is secure?"

[GitBox]

GitHub user github-actions[bot] added a comment to the discussion: Arguments 
for "why pulsar is secure?"

The issue had no activity for 30 days, mark with Stale label.

GitHub link: 
https://github.com/apache/pulsar/discussions/18754#discussioncomment-4321248


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] sfescape created a discussion: websocket feature parity

[GitBox]

GitHub user sfescape created a discussion: websocket feature parity

Is there a roadmap that discusses websocket features? It looks like websockets 
is pretty far behind in feature parity.
I'm considering moving from kafka to pulsar but topics regex consumer is an 
important feature for me.


GitHub link: https://github.com/apache/pulsar/discussions/18756


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] hpvd added a comment to the discussion: websocket feature parity

[GitBox]

GitHub user hpvd added a comment to the discussion: websocket feature parity

for latest state see: 
https://github.com/apache/pulsar/wiki/Client-Features-Matrix

GitHub link: 
https://github.com/apache/pulsar/discussions/18756#discussioncomment-4321273


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] hpvd added a comment to the discussion: websocket feature parity

[GitBox]

GitHub user hpvd added a comment to the discussion: websocket feature parity

imho websocket is for pulsar something like "the window to the web world". It 
enables usage of pulsar for fields like web-tracking, live chat and many more...
If one see the potential of this community, one should give websocket bugs and 
feature-requests a fair priority in the next releases
While writing this there are roughly 25 issues open regarding websocket topics.
https://github.com/apache/pulsar/issues?q=is%3Aissue+is%3Aopen+websocket+
3 includes open pull request which may be easy to close.

Of course recourses are limited and not all fields have the same importance for 
adoption of pulsar.
But at least the points
- docs e.g. https://github.com/apache/pulsar/issues/8610, 
https://github.com/apache/pulsar/issues/8688
- security e.g. https://github.com/apache/pulsar/issues/3493
- performance e.g. https://github.com/apache/pulsar/issues/8584, 
https://github.com/apache/pulsar/issues/5979

are important for everyone thinking about using pulsar with websocket..





GitHub link: 
https://github.com/apache/pulsar/discussions/18756#discussioncomment-4321274


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] hpvd added a comment to the discussion: websocket feature parity

[GitBox]

GitHub user hpvd added a comment to the discussion: websocket feature parity

feature parity would indeed be pretty awesome and would imho help adoption 
further

GitHub link: 
https://github.com/apache/pulsar/discussions/18756#discussioncomment-4321272


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] jiazhai created a discussion: Provide documentation for python end-to-end messagecrypto

[GitBox]

GitHub user jiazhai created a discussion: Provide documentation for python 
end-to-end messagecrypto

in issue #6834, user report there is no doc for configure and use feature 
end-to-end messagecrypto in python.
We need to confirm whether this feature is supported in python, and add needed 
doc. 

GitHub link: https://github.com/apache/pulsar/discussions/18757


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] Huanli-Meng added a comment to the discussion: Provide documentation for python end-to-end messagecrypto

[GitBox]

GitHub user Huanli-Meng added a comment to the discussion: Provide 
documentation for python end-to-end messagecrypto

@jiazhai , whether end-to-end messagecrypto is supported on Python?

GitHub link: 
https://github.com/apache/pulsar/discussions/18757#discussioncomment-4321320


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] BewareMyPower added a comment to the discussion: Provide documentation for python end-to-end messagecrypto

[GitBox]

GitHub user BewareMyPower added a comment to the discussion: Provide 
documentation for python end-to-end messagecrypto

Here is a test that shows how to use e2e encryption: 
https://github.com/apache/pulsar-client-python/blob/9182dd7ab69266ad0fb630a129a3ef478c1f8ab0/tests/pulsar_test.py#L369

GitHub link: 
https://github.com/apache/pulsar/discussions/18757#discussioncomment-4321323


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] tisonkun added a comment to the discussion: Provide documentation for python end-to-end messagecrypto

[GitBox]

GitHub user tisonkun added a comment to the discussion: Provide documentation 
for python end-to-end messagecrypto

cc @BewareMyPower @shibd @RobertIndie what's the current status of this feature?

GitHub link: 
https://github.com/apache/pulsar/discussions/18757#discussioncomment-4321322


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] jiazhai edited a discussion: How to do end-to-end messagecrypto with Python client

[GitBox]

GitHub user jiazhai edited a discussion: How to do end-to-end messagecrypto 
with Python client

in issue #6834, user report there is no doc for configure and use feature 
end-to-end messagecrypto in python.
We need to confirm whether this feature is supported in python, and add needed 
doc. 

GitHub link: https://github.com/apache/pulsar/discussions/18757


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] devdattakulkarni added a comment to the discussion: Provide documentation for python end-to-end messagecrypto

[GitBox]

GitHub user devdattakulkarni added a comment to the discussion: Provide 
documentation for python end-to-end messagecrypto

@jiazhai We are also interested in this. Do you know if this is already 
supported and just the documentation is missing, or if this feature is yet to 
be supported in the Python client?

GitHub link: 
https://github.com/apache/pulsar/discussions/18757#discussioncomment-4321321


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] tisonkun added a comment to the discussion: Provide documentation for python end-to-end messagecrypto

[GitBox]

GitHub user tisonkun added a comment to the discussion: Provide documentation 
for python end-to-end messagecrypto

@BewareMyPower Thank you! I'll convert this thread to a Q&A and select the 
answer.

Further improvement to add the example to 
https://pulsar.apache.org/docs/client-libraries-python or elsewhere is possible.

GitHub link: 
https://github.com/apache/pulsar/discussions/18757#discussioncomment-4321324


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] beyondyinjl2 added a comment to the discussion: Is there any development documentation for the authentication and authorization plug-in, and what is the address?

[GitBox]

GitHub user beyondyinjl2 added a comment to the discussion: Is there any 
development documentation for the authentication and authorization plug-in, and 
what is the address?

I have read the document you mentioned, but I don’t know the role of each 
interface, the loading process, and the flow of data.

GitHub link: 
https://github.com/apache/pulsar/discussions/18758#discussioncomment-4321368


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] codelipenghui added a comment to the discussion: Is there any development documentation for the authentication and authorization plug-in, and what is the address?

[GitBox]

GitHub user codelipenghui added a comment to the discussion: Is there any 
development documentation for the authentication and authorization plug-in, and 
what is the address?

The issue had no activity for 30 days, mark with Stale label.

GitHub link: 
https://github.com/apache/pulsar/discussions/18758#discussioncomment-4321369


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] beyondyinjl2 created a discussion: Is there any development documentation for the authentication and authorization plug-in, and what is the address?

[GitBox]

GitHub user beyondyinjl2 created a discussion: Is there any development 
documentation for the authentication and authorization plug-in, and what is the 
address?

Is there any development documentation for the authentication and authorization 
plug-in, and what is the address?

GitHub link: https://github.com/apache/pulsar/discussions/18758


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] Anonymitaet added a comment to the discussion: Is there any development documentation for the authentication and authorization plug-in, and what is the address?

[GitBox]

GitHub user Anonymitaet added a comment to the discussion: Is there any 
development documentation for the authentication and authorization plug-in, and 
what is the address?

Hi @beyondyinjl2 thanks for your question. Before raising an issue, please make 
sure you have investigated it thoroughly. When you opening a doc issue, please 
share your findings/references, so that others know your concerns clearly and 
know how to help. Is 
[this](https://pulsar.apache.org/docs/en/next/security-extending/#client-authentication-plugin)
 you looking for?

GitHub link: 
https://github.com/apache/pulsar/discussions/18758#discussioncomment-4321367


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] github-actions[bot] added a comment to the discussion: Wishing list: optimize blog page

[GitBox]

GitHub user github-actions[bot] added a comment to the discussion: Wishing 
list: optimize blog page

The issue had no activity for 30 days, mark with Stale label.

GitHub link: 
https://github.com/apache/pulsar/discussions/18759#discussioncomment-4321585


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[DISCUSS] Remove restful producer component

[mattisonchao]

Hello, everyone.

I'd like to start the discussion about `Remove restful producer component`. The 
Github repository path is here[1].

As discussed before[2], moving this feature to another project is better. Also, 
we didn't provide the consumer part in the pulsar repo. I think it's a good 
chance to remove it after 2.11 is released.

Best,
Mattison

[1] 
https://github.com/apache/pulsar/tree/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/rest
[2] https://lists.apache.org/thread/fl2rbb6sxlzwgkt7ybx4jxfkfnlb27z1

[GitHub] [pulsar] Jennifer88huang-zz created a discussion: How to run Pulsar Function in K8S

[GitBox]

GitHub user Jennifer88huang-zz created a discussion: How to run Pulsar Function 
in K8S

**Describe the bug**
My Pulsar function using State Storage (written in Python ) is working 
perfectly in Docker Container, but the same code is not working on Kubernetes. 
Can u please help me on that? Is there any configuration changes required to 
run it in Kubernetes?

Issue from rajesh thakur

GitHub link: https://github.com/apache/pulsar/discussions/18761


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] Jennifer88huang-zz added a comment to the discussion: How to run Pulsar Function in K8S

[GitBox]

GitHub user Jennifer88huang-zz added a comment to the discussion: How to run 
Pulsar Function in K8S

@nlu90 are you familiar with K8S? Could you please give a hand on it? Thank you 
very much.

GitHub link: 
https://github.com/apache/pulsar/discussions/18761#discussioncomment-4321837


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] codelipenghui added a comment to the discussion: How to run Pulsar Function in K8S

[GitBox]

GitHub user codelipenghui added a comment to the discussion: How to run Pulsar 
Function in K8S

The issue had no activity for 30 days, mark with Stale label.

GitHub link: 
https://github.com/apache/pulsar/discussions/18761#discussioncomment-4321838


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] KannarFr created a discussion: Pulsar SQL: support user defined indexes

[GitBox]

GitHub user KannarFr created a discussion: Pulsar SQL: support user defined 
indexes

**Is your feature request related to a problem? Please describe.**
Currently, there is no index used to query topic using presto.  
`__publish_time__` can be considered as index because of ledger storage way but 
it's not a real one.

**Describe the solution you'd like**
AvroSchema used to insert to topic should comes with a indexes definition. 
Since then, we should be able to have managedledger for indexes referencing 
classical managedledgers or messageid? And then configure pulsar presto impl to 
use user defined indexes from schema. (This is a suggestion to initialize the 
discussion, as @jerrypeng and I discussed it's a large discussion to have).

**Describe alternatives you've considered**
There are probably multiples ways to do it, feel free to suggest your pov.

**Additional context**
Reduce the query runtime.


GitHub link: https://github.com/apache/pulsar/discussions/18763


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] KannarFr added a comment to the discussion: Pulsar SQL: support user defined indexes

[GitBox]

GitHub user KannarFr added a comment to the discussion: Pulsar SQL: support 
user defined indexes

And how can we handle it on offloaded parts?

GitHub link: 
https://github.com/apache/pulsar/discussions/18763#discussioncomment-4321904


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] sijie added a comment to the discussion: Pulsar SQL: support user defined indexes

[GitBox]

GitHub user sijie added a comment to the discussion: Pulsar SQL: support user 
defined indexes

@KannarFr 

The indexes can be built in a background process using the approach that was 
used for compaction. The "compacted" ledger is essentially an "index" to the 
original data.

The index maintains some forms of mapping between "keys" to the "offsets" to 
the original data. The "offset" is essentially the message-id which is 
referencing a ledger and an entry id. It doesn't matter if a ledger is in the 
bookkeeper or already offloaded to the tiered storage.

GitHub link: 
https://github.com/apache/pulsar/discussions/18763#discussioncomment-4321905


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] KannarFr added a comment to the discussion: Pulsar SQL: support user defined indexes

[GitBox]

GitHub user KannarFr added a comment to the discussion: Pulsar SQL: support 
user defined indexes

Adding the index definition into schema definition but maybe it is not the best 
to do. I'm asking your opinion.

GitHub link: 
https://github.com/apache/pulsar/discussions/18763#discussioncomment-4321908


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] sijie added a comment to the discussion: Pulsar SQL: support user defined indexes

[GitBox]

GitHub user sijie added a comment to the discussion: Pulsar SQL: support user 
defined indexes

Are you talking about adding the index definition into schema definition? Or 
using Avro schema specification for describing the indexes?

GitHub link: 
https://github.com/apache/pulsar/discussions/18763#discussioncomment-4321907


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] KannarFr added a comment to the discussion: Pulsar SQL: support user defined indexes

[GitBox]

GitHub user KannarFr added a comment to the discussion: Pulsar SQL: support 
user defined indexes

@sijie 
Ok, about the indexes definition, what do you think about the definition 
approach using avroschema to define indexes?

GitHub link: 
https://github.com/apache/pulsar/discussions/18763#discussioncomment-4321906


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] sijie added a comment to the discussion: Pulsar SQL: support user defined indexes

[GitBox]

GitHub user sijie added a comment to the discussion: Pulsar SQL: support user 
defined indexes

I don't think it is a good idea to add an index definition to the schema 
definition. The schema definition defines the structure of the original data. 
The index definition depends on the schema definition but it is different from 
the original data. So the index definition should be associated with the 
storage that is used for storing the index data. For example, if we are using 
another managed ledger for storing the index, then the index definition should 
be the schema definition of the managed ledger. Does that make sense?

GitHub link: 
https://github.com/apache/pulsar/discussions/18763#discussioncomment-4321909


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] pointearth added a comment to the discussion: Pulsar SQL: support user defined indexes

[GitBox]

GitHub user pointearth added a comment to the discussion: Pulsar SQL: support 
user defined indexes

I think user-defined indexes are very important for Pulsar SQL, it could be 
able to be the real way that we can use pulsar as a database. I think it will 
make pulsar more popular.
And I agree to define user-defined indexes individually. we can extend 
"pulsar-admin topic" to manage indexes, to create, read, update, delete, 
reIndex them.
Can we discuss more and push it forward?

GitHub link: 
https://github.com/apache/pulsar/discussions/18763#discussioncomment-4321910


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] KannarFr added a comment to the discussion: Pulsar SQL: support user defined indexes

[GitBox]

GitHub user KannarFr added a comment to the discussion: Pulsar SQL: support 
user defined indexes

@pointearth @sijie How do you imagine the index composition regarding ledgers? 
As the first implementation, regarding pub/sub system, timestamp index-based 
would be a good start. I think we should first be able to auto-create an index 
per topic like:

```java
Map[Date, LedgerId]
```
Or maybe

```java
Map[Date, IndexItem]

IndexItem(PreviousLedgerId, LedgerId, NextLedgerId)
```

WDYT? Maybe we should directly point to message and not ledger.

GitHub link: 
https://github.com/apache/pulsar/discussions/18763#discussioncomment-4321911


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] KannarFr added a comment to the discussion: Pulsar SQL: support user defined indexes

[GitBox]

GitHub user KannarFr added a comment to the discussion: Pulsar SQL: support 
user defined indexes

And where/how we store this index? 

GitHub link: 
https://github.com/apache/pulsar/discussions/18763#discussioncomment-4321912


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] golden-yang added a comment to the discussion: Pulsar SQL: support user defined indexes

[GitBox]

GitHub user golden-yang added a comment to the discussion: Pulsar SQL: support 
user defined indexes

Is there any progress on this issue? 
Being able to support indexes in Pulsar Sql will be a very meaningful feature.

One way is to support it **natively**, and the other way I think it can be 
achieved through **tiered storage**. For example, combined with the data lake, 
with the help of Apache Hudi and so on.

I saw some articles about the combination of hudi and pulsar, is there any 
progress?
@sijie  

GitHub link: 
https://github.com/apache/pulsar/discussions/18763#discussioncomment-4321915


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] pointearth added a comment to the discussion: Pulsar SQL: support user defined indexes

[GitBox]

GitHub user pointearth added a comment to the discussion: Pulsar SQL: support 
user defined indexes

Can we store it in bookKeeper? and it may be able to start when the presto 
enhancement switching is open.

GitHub link: 
https://github.com/apache/pulsar/discussions/18763#discussioncomment-4321914


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] pointearth added a comment to the discussion: Pulsar SQL: support user defined indexes

[GitBox]

GitHub user pointearth added a comment to the discussion: Pulsar SQL: support 
user defined indexes

I know querying with timestamp is very fast, because data in bookKeeper save 
timestamp as key.
My suggestion is to create a non-clustered index, based on the key in 
bookKeeper, for example:
If we want to create an index on the field name, We can create indexItem as 
map[name, List<__publish_time__>]

Can you supply some describe the source code around this? and then we can 
discuss it again.

GitHub link: 
https://github.com/apache/pulsar/discussions/18763#discussioncomment-4321913


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] Hugues514 created a discussion: Pulsar SQL should support readCompact on topics

[GitBox]

GitHub user Hugues514 created a discussion: Pulsar SQL should support 
readCompact on topics

So far, when a Pulsar topic is compacted, the Pulsar SQL query will read all 
the records, not considering the compacted option

It could be interesting to be able to query such a topic using a readCompact 
option so that the reader will skip all the 'compacted records'.


GitHub link: https://github.com/apache/pulsar/discussions/18764


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] longtengz added a comment to the discussion: Pulsar SQL should support readCompact on topics

[GitBox]

GitHub user longtengz added a comment to the discussion: Pulsar SQL should 
support readCompact on topics

Does this also imply pulsar sql cannot process empty payload messages for the 
time being? Because I'm experiencing #13127 

GitHub link: 
https://github.com/apache/pulsar/discussions/18764#discussioncomment-4321922


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] jiazhai added a comment to the discussion: Pulsar SQL should support readCompact on topics

[GitBox]

GitHub user jiazhai added a comment to the discussion: Pulsar SQL should 
support readCompact on topics

We need a special handle in `PulsarSplitManager.getSplitsForTopic` to split 
based also on `compactionHorizon`(contained in `CompactedTopicImpl`); so that 
for data before compactionHorizon, we could read from compacted topic, while 
data after compactionHorizon, read from non-compacted topic.

GitHub link: 
https://github.com/apache/pulsar/discussions/18764#discussioncomment-4321919


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] pointearth added a comment to the discussion: Pulsar SQL should support readCompact on topics

[GitBox]

GitHub user pointearth added a comment to the discussion: Pulsar SQL should 
support readCompact on topics

Can you guys open this ticket and finish it or move it to a suitable 
appointment? appreciate!
[Add readCompact option #7461](https://github.com/apache/pulsar/pull/7461 )

GitHub link: 
https://github.com/apache/pulsar/discussions/18764#discussioncomment-4321921


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] jiazhai added a comment to the discussion: Pulsar SQL should support readCompact on topics

[GitBox]

GitHub user jiazhai added a comment to the discussion: Pulsar SQL should 
support readCompact on topics

more info: 
https://github.com/apache/pulsar/wiki/PIP-19:-Pulsar-SQL
https://github.com/apache/pulsar/wiki/PIP-14:-Topic-compaction


GitHub link: 
https://github.com/apache/pulsar/discussions/18764#discussioncomment-4321920


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] tisonkun added a comment to the discussion: Pulsar SQL: support user defined indexes

[GitBox]

GitHub user tisonkun added a comment to the discussion: Pulsar SQL: support 
user defined indexes

Moved to general discussions since there's a wishing list item and not 
actionable.

GitHub link: 
https://github.com/apache/pulsar/discussions/18763#discussioncomment-4321928


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] KannarFr edited a discussion: Wishing list: Pulsar SQL support user defined indexes

[GitBox]

GitHub user KannarFr edited a discussion: Wishing list: Pulsar SQL support user 
defined indexes

**Is your feature request related to a problem? Please describe.**
Currently, there is no index used to query topic using presto.  
`__publish_time__` can be considered as index because of ledger storage way but 
it's not a real one.

**Describe the solution you'd like**
AvroSchema used to insert to topic should comes with a indexes definition. 
Since then, we should be able to have managedledger for indexes referencing 
classical managedledgers or messageid? And then configure pulsar presto impl to 
use user defined indexes from schema. (This is a suggestion to initialize the 
discussion, as @jerrypeng and I discussed it's a large discussion to have).

**Describe alternatives you've considered**
There are probably multiples ways to do it, feel free to suggest your pov.

**Additional context**
Reduce the query runtime.


GitHub link: https://github.com/apache/pulsar/discussions/18763


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] Hugues514 edited a discussion: Wishing list: Pulsar SQL should support readCompact on topics

[GitBox]

GitHub user Hugues514 edited a discussion: Wishing list: Pulsar SQL should 
support readCompact on topics

So far, when a Pulsar topic is compacted, the Pulsar SQL query will read all 
the records, not considering the compacted option

It could be interesting to be able to query such a topic using a readCompact 
option so that the reader will skip all the 'compacted records'.


GitHub link: https://github.com/apache/pulsar/discussions/18764


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] tisonkun added a comment to the discussion: Wishing list: Pulsar SQL should support readCompact on topics

[GitBox]

GitHub user tisonkun added a comment to the discussion: Wishing list: Pulsar 
SQL should support readCompact on topics

Moved to general discussions since there's a wishing list item and no one works 
on it now.

GitHub link: 
https://github.com/apache/pulsar/discussions/18764#discussioncomment-4321936


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] ivankelly created a discussion: Proxy should autodetect whether to use TLS to connect brokers

[GitBox]

GitHub user ivankelly created a discussion: Proxy should autodetect whether to 
use TLS to connect brokers

Currently you need to set `tlsEnabledWithBroker` in the proxy to use TLS to 
connect to brokers. However, if the URL is https:// or pulsar+ssl:// it should 
be obvious that we should use TLS. Likewise, when the url is http:// or 
pulsar:// it should be obvious that we should not use TLS.

GitHub link: https://github.com/apache/pulsar/discussions/18765


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] waxzce created a discussion: Use graalVM native image to build client and client admin

[GitBox]

GitHub user waxzce created a discussion: Use graalVM native image to build 
client and client admin

Using the pulsar-admin command can be very frustrating because of the startup 
time. That's one point were graalVM native image is interesting: build a static 
"binary".

Various problem to compile to native-image can occur, using reflection and 
dynamic classloader are often the problems providers. From my point of view, 
auth plugins are the main problem, but apart of these, do you see any points to 
look at?


GitHub link: https://github.com/apache/pulsar/discussions/18766


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] sijie added a comment to the discussion: Use graalVM native image to build client and client admin

[GitBox]

GitHub user sijie added a comment to the discussion: Use graalVM native image 
to build client and client admin

alternatively, you can now use 
[pulsarctl](https://github.com/streamnative/pulsarctl). it is a static linked 
CLI written in golang.

GitHub link: 
https://github.com/apache/pulsar/discussions/18766#discussioncomment-4322271


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] aahmed-se added a comment to the discussion: Use graalVM native image to build client and client admin

[GitBox]

GitHub user aahmed-se added a comment to the discussion: Use graalVM native 
image to build client and client admin

I don't know how stable graalVM is last I checked it's still is an RC candidate 
we might also need to look at other http client libs that support aot code 
generation.

GitHub link: 
https://github.com/apache/pulsar/discussions/18766#discussioncomment-4322270


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] fantapsody added a comment to the discussion: Use graalVM native image to build client and client admin

[GitBox]

GitHub user fantapsody added a comment to the discussion: Use graalVM native 
image to build client and client admin

Pulsar itself should also be built with AOT compilers to reduce the time to 
start the process. Sometimes it takes up to a minute to start the broker with 
function worker enabled.

GitHub link: 
https://github.com/apache/pulsar/discussions/18766#discussioncomment-4322272


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] harissecic added a comment to the discussion: Use graalVM native image to build client and client admin

[GitBox]

GitHub user harissecic added a comment to the discussion: Use graalVM native 
image to build client and client admin

> Broker will require many changes and unlikely it can be made to work with 
> graalvm anytime soon. Client should not an issue it has a dependency on netty 
> , but netty authors should have already provided a guide to to use it with 
> graal and it should work just fine.

Shaded client which is the one I have to use in this case as netty needs to be 
supported independently of the Pulsar Client so I really can't due to other 
limitations extract it. I can only research again what netty methods are used 
with reflection and try to list them with org.apache.pulsar... prefix. On the 
other hand looking at the source code I'm not sure how safe calls to Schema 
instantiation would be which heavily rely on reflection. In any case I've made 
own Jackson schema to support passing in ObjectMapper from framework while 
using shaded library. In many cases this library tends to give me warnings and 
such running on new Java versions like 14. This is why it would be nice if 
possible to have less of those reflection and unsafe calls. On the other hand I 
guess main issue is in fact in netty in my case I just have to list all of 
those calls with apache pulsar extension as package name. An example from my 
point of view is 
```java
public static Schema newBytesSchema() {
return catchExceptions(
() -> (Schema) 
newClassInstance("org.apache.pulsar.client.impl.schema.BytesSchema")
.newInstance());
}

public static Schema newStringSchema() {
return catchExceptions(
() -> (Schema) 
newClassInstance("org.apache.pulsar.client.impl.schema.StringSchema")
.newInstance());
}
```

In my case it's replaced by simple `new StringSchema()` calls instead of 
`Schema STRING = DefaultImplementation.newStringSchema();`. I get that 
most of reflective calls that it uses are "safe" around graal but not really 
liked.
In any case thanks for the response and I'll try to figure out how to make this 
specific use case work because I expect it's not really that applicable (having 
to use shaded but having to list netty calls for graal yet again having to pass 
in your own jackson mapper).


GitHub link: 
https://github.com/apache/pulsar/discussions/18766#discussioncomment-4322276


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] harissecic added a comment to the discussion: Use graalVM native image to build client and client admin

[GitBox]

GitHub user harissecic added a comment to the discussion: Use graalVM native 
image to build client and client admin

I'm experiencing similar issues trying to make Micronaut integration for Pulsar 
Java client is really hard mainly due to reflection and Micronaut now demands 
that it can build for GraalVM native image. 
If anyone has advice how to make this library run (which reflection calls to 
add) please let me know. I support @fantapsody as Java client library is a bit 
too heavy now to run due to many in-between calls  which are mainly just using 
reflection to call other classes. This would maybe increase compile time but 
it's much more beneficial in the end to not have it or have as less as posible.

GitHub link: 
https://github.com/apache/pulsar/discussions/18766#discussioncomment-4322273


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] aahmed-se added a comment to the discussion: Use graalVM native image to build client and client admin

[GitBox]

GitHub user aahmed-se added a comment to the discussion: Use graalVM native 
image to build client and client admin

Broker will require many changes and unlikely it can be made to work with 
graalvm anytime soon. Client should not an issue it has a dependency on netty , 
but netty authors should have already provided a guide to to use it with graal 
and it should work just fine.

GitHub link: 
https://github.com/apache/pulsar/discussions/18766#discussioncomment-4322274


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

Re: [DISCUSS] Remove restful producer component

[PengHui Li]

+1

And we should also remove the documentation of this part.

Penghui

> On Dec 6, 2022, at 18:36, mattisonc...@gmail.com wrote:
> 
> 
> Hello, everyone.
> 
> I'd like to start the discussion about `Remove restful producer component`. 
> The Github repository path is here[1].
> 
> As discussed before[2], moving this feature to another project is better. 
> Also, we didn't provide the consumer part in the pulsar repo. I think it's a 
> good chance to remove it after 2.11 is released.
> 
> Best,
> Mattison
> 
> [1] 
> https://github.com/apache/pulsar/tree/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/rest
> [2] https://lists.apache.org/thread/fl2rbb6sxlzwgkt7ybx4jxfkfnlb27z1
> 
> 

Re: [DISCUSS] Remove restful producer component

[Enrico Olivelli]

+1
I agree that such kind of APIs can be on side projects


Enrico

Il giorno mar 6 dic 2022 alle ore 14:20 PengHui Li
 ha scritto:
>
> +1
>
> And we should also remove the documentation of this part.
>
> Penghui
>
> > On Dec 6, 2022, at 18:36, mattisonc...@gmail.com wrote:
> >
> >
> > Hello, everyone.
> >
> > I'd like to start the discussion about `Remove restful producer component`. 
> > The Github repository path is here[1].
> >
> > As discussed before[2], moving this feature to another project is better. 
> > Also, we didn't provide the consumer part in the pulsar repo. I think it's 
> > a good chance to remove it after 2.11 is released.
> >
> > Best,
> > Mattison
> >
> > [1] 
> > https://github.com/apache/pulsar/tree/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/rest
> > [2] https://lists.apache.org/thread/fl2rbb6sxlzwgkt7ybx4jxfkfnlb27z1
> >
> >
>

Re: [VOTE] Pulsar Release 2.11.0 Candidate-2

[Enrico Olivelli]

+1 (binding)
- verified checksums, digests
- built from sources
- tested the docker image built from sources and the docker image
provided as convenience with the VOTE
- run successfully the Jakarta JMS 2.0 TCK against those two docker images

My problem with the JMS TCK is that the behaviour of "delete topic"
with "force=true" changed, and now if the topic does not exist we get
a 404 error, in Pulsar 2.10.x the response code was 200
I think that this behaviour change is acceptable in a major release,
and that actually the new behaviour is more consistent.

More details here:
https://github.com/datastax/pulsar-jms/pull/101

Thank you for driving the release

Enrico

Il giorno lun 5 dic 2022 alle ore 16:21 Enrico Olivelli
 ha scritto:
>
> FYI I am investigating some errors I see while running the Jakarta JMS
> TCK against a Pulsar 2.11.0rc2 server with the provided docker images.
>
> I still can't understand the problem, so I will keep you posted.
>
> Please hold on
>
> Enrico
>
> Il giorno dom 4 dic 2022 alle ore 19:13 Christophe Bornet
>  ha scritto:
> >
> > +1 (non-binding)
> >
> > - Start docker image standalone
> > - Test HTTP Sink
> > - Test Transformation Function
> >
> > Best regards
> >
> > Christophe
> >
> > Le lun. 28 nov. 2022 à 13:57, guo jiwei  a écrit :
> >
> > > This is the second release candidate for Apache Pulsar, version 2.11.0.
> > >
> > > This release contains 1574 commits by 64 contributors.
> > > https://github.com/apache/pulsar/compare/v2.10.2...v2.11.0-candidate-2
> > >
> > > CI for this release candidate
> > > https://github.com/Technoboy-/pulsar/pull/14
> > >
> > > *** Please download, test and vote on this release. This vote will stay
> > > open
> > > for at least 72 hours ***
> > >
> > > Note that we are voting upon the source (tag), binaries are provided for
> > > convenience.
> > >
> > > Source and binary files:
> > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.11.0-candidate-2
> > >
> > > SHA-512 checksums:
> > >
> > >
> > > 199eb731bad0635fbc0388adf090e8c60ce189634e6b1692a12cfdf43f15347371d79c42d040b91dd7b2d0302061ad48bff850638e3bb48540ab24908792dd5e
> > >
> > >  ./apache-pulsar-2.11.0-bin.tar.gz
> > >
> > >
> > > 035d5a656cdd4c2430ae4f7c4fef952ad05e432219a64474fcfaa097fcc7ad6cd76fd82b77e91d667e0ec8495c35fc08a5bb25a54058ccdfaf4072a22b94f6d7
> > >
> > >  ./apache-pulsar-2.11.0-src.tar.gz
> > >
> > > Maven staging repo:
> > > https://repository.apache.org/content/repositories/orgapachepulsar-1192/
> > > 
> > >
> > > The tag to be voted upon:
> > > v2.11.0-candidate-2 (c0a823a242a834dac35b9a6fcd6a2064a0e4bfb5)
> > > https://github.com/apache/pulsar/releases/tag/v2.11.0-candidate-2
> > >
> > > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > >
> > > Docker images:
> > >
> > > https://hub.docker.com/layers/mattison/pulsar-all/2.11.0-rc2/images/sha256-1b6b04673d62ccefe6fd6eed27a001ef8a9607668fcc5fb79459fed73f07e9e4
> > >
> > > https://hub.docker.com/layers/mattison/pulsar/2.11.0-rc2/images/sha256-21a4afa5a83ed5780c2d4986a3e13bc9d43142614f610434d5d304c1d61a65eb
> > >
> > > Please download the source package, and follow the
> > > release-candidate-validation doc to build
> > > and run the Pulsar standalone service.
> > > https://pulsar.apache.org/contribute/validate-release-candidate
> > >
> > > Since the metadata store is changed from ZK to RocksDB, the verification 
> > > of
> > > the `stateful functions` needs to set the parameter "export
> > > PULSAR_STANDALONE_USE_ZOOKEEPER=1"
> > >
> > >
> > > Regards
> > > Jiwei Guo (Tboy)
> > >

[GitHub] [pulsar] flowchartsman added a comment to the discussion: Golang | Pulsar Functions Runtime Death Exception

[GitBox]

GitHub user flowchartsman added a comment to the discussion: Golang | Pulsar 
Functions Runtime Death Exception

This is unfortunately a pretty generic response. Debugging pulsar function 
crashes in go can be kind of a pain, especially if they happen before 
`pf.Start` is called, since there's no logging at that point. Usually you can 
find the more specific cause in the function worker or broker logs, depending 
on how you're running the function.

GitHub link: 
https://github.com/apache/pulsar/discussions/18649#discussioncomment-4323323


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] flowchartsman edited a comment on the discussion: Golang | Pulsar Functions Runtime Death Exception

[GitBox]

GitHub user flowchartsman edited a comment on the discussion: Golang | Pulsar 
Functions Runtime Death Exception

This is unfortunately a pretty generic response. Debugging pulsar function 
crashes in go can be kind of a pain, especially if they happen before 
`pf.Start` is called, since there's no logging at that point. Usually you can 
find the more specific cause in the function worker or broker logs, depending 
on how you're running the function. If you can, `pulsar-admin localrun` can 
potentially provide some insight.

GitHub link: 
https://github.com/apache/pulsar/discussions/18649#discussioncomment-4323323


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] divya-nsurely added a comment to the discussion: Golang | Pulsar Functions Runtime Death Exception

[GitBox]

GitHub user divya-nsurely added a comment to the discussion: Golang | Pulsar 
Functions Runtime Death Exception

Thanks for your response.

The Create API returns a successful response. However, the function stops 
running due to this Death Exception error.

When I try to restart the function, I receive a 204 response. In spite of this, 
when I try to get the status of the function, it gives me the below error.

standalone_1  | 2022-12-03T11:50:35,902+ 
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO  
org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout 
while reading L765 E3 from bookie: 127.0.0.1:3181
standalone_1  | 2022-12-03T11:50:35,902+ 
[BookKeeperClientWorker-OrderedExecutor-0-0] ERROR 
org.apache.bookkeeper.client.PendingReadOp - Read of ledger entry failed: L765 
E3-E3, Sent to [127.0.0.1:3181], Heard from [] : bitset = {}, Error = 'Bookie 
operation timeout'. First unread entry is (-1, rc = null)
standalone_1  | 2022-12-03T11:50:36,057+ [main] ERROR 
org.apache.pulsar.functions.worker.FunctionActioner - t09/n09/FOURTHFUNC Error 
starting function
standalone_1  | org.apache.distributedlog.exceptions.BKTransmitException: 
Encountered issue on reading entry 0 @ log segment [LogSegmentId:765, 
firstTxId:1024, lastTxId:9223372036854775807, version:VERSION_V5_SEQUENCE_ID, 
completionTime:1670066952044, recordCount:42692, regionId:0, status:0, 
logSegmentSequenceNumber:1, lastEntryId:171, lastSlotId:0, inprogress:false, 
minActiveDLSN:DLSN{logSegmentSequenceNo=1, entryId=0, slotId=0}, 
startSequenceId:0] : -23
standalone_1  | at 
org.apache.distributedlog.impl.logsegment.BKLogSegmentEntryReader.readEntriesFromReadAheadCache(BKLogSegmentEntryReader.java:814)
 ~[org.apache.distributedlog-distributedlog-core-4.14.5.jar:4.14.5]
standalone_1  | at 
org.apache.distributedlog.impl.logsegment.BKLogSegmentEntryReader.safeRun(BKLogSegmentEntryReader.java:733)
 ~[org.apache.distributedlog-distributedlog-core-4.14.5.jar:4.14.5]
standalone_1  | at 
org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) 
~[org.apache.bookkeeper-bookkeeper-common-4.14.5.jar:4.14.5]
standalone_1  | at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
standalone_1  | at 
java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
standalone_1  | at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
 ~[?:?]
standalone_1  | at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
~[?:?]
standalone_1  | at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
~[?:?]
standalone_1  | at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
standalone_1  | at java.lang.Thread.run(Thread.java:829) ~[?:?]


It would be great if you can take a look at this.

GitHub link: 
https://github.com/apache/pulsar/discussions/18649#discussioncomment-4323482


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] flowchartsman edited a discussion: Go Function SDK Feature Parity

[GitBox]

GitHub user flowchartsman edited a discussion: Go Function SDK Feature Parity

The Pulsar functions for Go SDK needs some love. It is lacking in some key 
features and has a couple of rough edges. Broadly, I think the SDK has been 
held back a little a little bit from trying to be too much like the Java 
version, which has resulted in a couple awkward abstractions such as the 
runtime function deflection dispatch and 
[NewOuputMessage()](https://github.com/apache/pulsar/pull/8327).

This discussion serves to track interest and hopefully chart a way forward.

- [ ] State support
  - [Issue](https://github.com/apache/pulsar/issues/5138)
  - [Another issue](https://github.com/apache/pulsar/issues/10560)
  - [Abandoned Bookeeper PR](https://github.com/apache/bookkeeper/pull/2750)
- [ ] Better multi-topic dispatch
  - [Issue](https://github.com/apache/pulsar/issues/9511)

GitHub link: https://github.com/apache/pulsar/discussions/18768


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] flowchartsman edited a discussion: Go Function SDK Feature Parity

[GitBox]

GitHub user flowchartsman edited a discussion: Go Function SDK Feature Parity

The Pulsar functions for Go SDK needs some love. It is lacking in some key 
features and has a couple of rough edges. Broadly, I think the SDK has been 
held back a little a little bit from trying to be too much like the Java 
version, which has resulted in a couple awkward abstractions such as the 
runtime function deflection dispatch and 
[NewOuputMessage()](https://github.com/apache/pulsar/pull/8327).

This discussion serves to track interest and hopefully chart a way forward.

- [ ] State support
  - [Issue](https://github.com/apache/pulsar/issues/5138)
  - [Another issue](https://github.com/apache/pulsar/issues/10560)
  - [Abandoned Bookeeper PR](https://github.com/apache/bookkeeper/pull/2750)
- [ ] Better multi-topic dispatch
  - https://github.com/apache/pulsar/issues/9512
  - https://github.com/apache/pulsar/issues/9511

GitHub link: https://github.com/apache/pulsar/discussions/18768


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] flowchartsman edited a discussion: Go Function SDK Feature Parity

[GitBox]

GitHub user flowchartsman edited a discussion: Go Function SDK Feature Parity

The Pulsar functions for Go SDK needs some love. It is lacking in some key 
features and has a couple of rough edges. Broadly, I think the SDK has been 
held back a little a little bit from trying to be too much like the Java 
version, which has resulted in a couple awkward abstractions such as the 
runtime function deflection dispatch and 
[NewOuputMessage()](https://github.com/apache/pulsar/pull/8327).

This discussion serves to track interest and hopefully chart a way forward.

- [ ] State support
  - https://github.com/apache/pulsar/issues/5138
  - https://github.com/apache/pulsar/issues/10560
  - https://github.com/apache/bookkeeper/pull/2750 - Abandoned?
- [ ] Better multi-topic dispatch
  - https://github.com/apache/pulsar/issues/9512
  - https://github.com/apache/pulsar/issues/9511

GitHub link: https://github.com/apache/pulsar/discussions/18768


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] flowchartsman edited a discussion: Go Function SDK Feature Parity

[GitBox]

GitHub user flowchartsman edited a discussion: Go Function SDK Feature Parity

The Pulsar functions for Go SDK needs some love. It is lacking in some key 
features and has a couple of rough edges. Broadly, I think the SDK has been 
held back a little a little bit from trying to be too much like the Java 
version, which has resulted in a couple awkward abstractions such as the 
runtime function deflection dispatch and 
[NewOuputMessage()](https://github.com/apache/pulsar/pull/8327).

This discussion serves to track interest and hopefully chart a way forward.

- [ ] State support
  - https://github.com/apache/pulsar/issues/5138
  - https://github.com/apache/pulsar/issues/10560
  - https://github.com/apache/bookkeeper/pull/2750 - Abandoned?
  - 
[https://pkg.go.dev/github.com/phemmer/go-pulsar-state/pulsar?utm_source=godoc] 
Reference implementation with 
[usage](https://github.com/phemmer/go-pulsar-state/blob/master/pulsar/examples/wordcount/wordcount.go)
- [ ] Better multi-topic dispatch
  - https://github.com/apache/pulsar/issues/9512
  - https://github.com/apache/pulsar/issues/9511

GitHub link: https://github.com/apache/pulsar/discussions/18768


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] flowchartsman edited a discussion: Go Function SDK Feature Parity

[GitBox]

GitHub user flowchartsman edited a discussion: Go Function SDK Feature Parity

The Pulsar functions for Go SDK needs some love. It is lacking in some key 
features and has a couple of rough edges. Broadly, I think the SDK has been 
held back a little a little bit from trying to be too much like the Java 
version, which has resulted in a couple awkward abstractions such as the 
runtime function deflection dispatch and 
[NewOuputMessage()](https://github.com/apache/pulsar/pull/8327).

This discussion serves to track interest and hopefully chart a way forward.

- [ ] State support
  - https://github.com/apache/pulsar/issues/5138
  - https://github.com/apache/pulsar/issues/10560
  - https://github.com/apache/bookkeeper/pull/2750 - Abandoned?
  - 
[https://pkg.go.dev/github.com/phemmer/go-pulsar-state/pulsar?utm_source=godoc](Reference
 implementation) with 
[usage](https://github.com/phemmer/go-pulsar-state/blob/master/pulsar/examples/wordcount/wordcount.go)
- [ ] Better multi-topic dispatch
  - https://github.com/apache/pulsar/issues/9512
  - https://github.com/apache/pulsar/issues/9511

GitHub link: https://github.com/apache/pulsar/discussions/18768


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] flowchartsman edited a discussion: Go Function SDK Feature Parity

[GitBox]

GitHub user flowchartsman edited a discussion: Go Function SDK Feature Parity

The Pulsar functions for Go SDK needs some love. It is lacking in some key 
features and has a couple of rough edges. Broadly, I think the SDK has been 
held back a little a little bit from trying to be too much like the Java 
version, which has resulted in a couple awkward abstractions such as the 
runtime function deflection dispatch and 
[NewOuputMessage()](https://github.com/apache/pulsar/pull/8327).

This discussion serves to track interest and hopefully chart a way forward.

- [ ] State support
  - https://github.com/apache/pulsar/issues/5138
  - https://github.com/apache/pulsar/issues/10560
  - https://github.com/apache/bookkeeper/pull/2750 - Abandoned?
  - [Reference 
implementation](https://pkg.go.dev/github.com/phemmer/go-pulsar-state/pulsar?utm_source=godoc)
 with 
[usage](https://github.com/phemmer/go-pulsar-state/blob/master/pulsar/examples/wordcount/wordcount.go)
- [ ] Better multi-topic dispatch
  - https://github.com/apache/pulsar/issues/9512
  - https://github.com/apache/pulsar/issues/9511

GitHub link: https://github.com/apache/pulsar/discussions/18768


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] woodsmur created a discussion: multiple topic sink to multiple elasticsearch index

[GitBox]

GitHub user woodsmur created a discussion: multiple topic sink to multiple 
elasticsearch index

Hi,
Is there a way to send data from multiple topic to different elasticsearch 
indices? For example, if I have `t-2020.12.01` and `t-2020.12.02`, how could I 
consume these two topics and create two corresponding indices `t-2020.12.01` 
and `t-2020.12.02`?

GitHub link: https://github.com/apache/pulsar/discussions/18770


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] hpvd added a comment to the discussion: multiple topic sink to multiple elasticsearch index

[GitBox]

GitHub user hpvd added a comment to the discussion: multiple topic sink to 
multiple elasticsearch index

yes this would be helpful in many cases of bigger installations.
At this time doc is only talking about "singular" 
https://pulsar.apache.org/docs/en/io-elasticsearch/

Problem and possible result looks somehow similar to 
https://github.com/apache/pulsar/issues/4547




GitHub link: 
https://github.com/apache/pulsar/discussions/18770#discussioncomment-4324182


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] zymap added a comment to the discussion: [Doc] "negatively acknowledged cumulatively" ambiguous?

[GitBox]

GitHub user zymap added a comment to the discussion: [Doc] "negatively 
acknowledged cumulatively" ambiguous?

Do you want this? 
https://pulsar.apache.org/docs/concepts-messaging#acknowledgement

GitHub link: 
https://github.com/apache/pulsar/discussions/18771#discussioncomment-4324377


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] startjava added a comment to the discussion: [Doc] "negatively acknowledged cumulatively" ambiguous?

[GitBox]

GitHub user startjava added a comment to the discussion: [Doc] "negatively 
acknowledged cumulatively" ambiguous?

> Do you want this? 
> https://pulsar.apache.org/docs/concepts-messaging#acknowledgement

YES，i see the doc too .

GitHub link: 
https://github.com/apache/pulsar/discussions/18771#discussioncomment-4324378


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] startjava added a comment to the discussion: [Doc] "negatively acknowledged cumulatively" ambiguous?

[GitBox]

GitHub user startjava added a comment to the discussion: [Doc] "negatively 
acknowledged cumulatively" ambiguous?

@momo-jun 

question1：
why 
（1）individually (in Shared and Key_Shared subscription types)
（2）cumulatively (Exclusive and Failover subscription types).

why bottom is error：
（1）cumulatively  (in Shared and Key_Shared subscription types)
（2）individually  (Exclusive and Failover subscription types).

question2：
but not have about "negatively acknowledged cumulatively" api，only have 
"negatively acknowledged individually" api(Consumer.negativeAcknowledeg)

i have 2 question 
thank you jun !



GitHub link: 
https://github.com/apache/pulsar/discussions/18771#discussioncomment-4324381


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] startjava added a comment to the discussion: [Doc] "negatively acknowledged cumulatively" ambiguous?

[GitBox]

GitHub user startjava added a comment to the discussion: [Doc] "negatively 
acknowledged cumulatively" ambiguous?

@momo-jun please see the question 

GitHub link: 
https://github.com/apache/pulsar/discussions/18771#discussioncomment-4324379


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] momo-jun added a comment to the discussion: [Doc] "negatively acknowledged cumulatively" ambiguous?

[GitBox]

GitHub user momo-jun added a comment to the discussion: [Doc] "negatively 
acknowledged cumulatively" ambiguous?

> Messages are negatively acknowledged individually or cumulatively, depending 
> on the consumption subscription type.
> * In Exclusive and Failover subscription types, consumers only negatively 
> acknowledge the last message they receive.
> * In Shared and Key_Shared subscription types, consumers can negatively 
> acknowledge messages individually.

I don't see the ambiguity in this paragraph. It means the negative 
acknowledgment of messages can be performed either individually (in Shared and 
Key_Shared subscription types) or cumulatively (Exclusive and Failover 
subscription types).

Did this answer your question? @startjava 

GitHub link: 
https://github.com/apache/pulsar/discussions/18771#discussioncomment-4324380


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] momo-jun added a comment to the discussion: [Doc] "negatively acknowledged cumulatively" ambiguous?

[GitBox]

GitHub user momo-jun added a comment to the discussion: [Doc] "negatively 
acknowledged cumulatively" ambiguous?

> but i don't know what mean "negatively acknowledged cumulatively"。

You can understand it as "Cumulative NACK" - the negative acknowledgement is 
performed cumulatively, that is, consumers only negatively acknowledge the last 
message they receive.


GitHub link: 
https://github.com/apache/pulsar/discussions/18771#discussioncomment-4324384


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] labuladong added a comment to the discussion: [Doc] "negatively acknowledged cumulatively" ambiguous?

[GitBox]

GitHub user labuladong added a comment to the discussion: [Doc] "negatively 
acknowledged cumulatively" ambiguous?

You can try `negativeAcknowledge` in `Failover/Exclusive` subscription types. 
All the messages the consumer didn't acknowledge will be redelivered, this 
behavior could be negatively acknowledged cumulatively. But there do have some 
discussions about the redelivery behavior on `Failover/Exclusive` subscription 
types, see https://lists.apache.org/thread/vdwhg5wrpjc8x41oqwxrsdhx7dmyzt3j

GitHub link: 
https://github.com/apache/pulsar/discussions/18771#discussioncomment-4324382


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] startjava added a comment to the discussion: [Doc] "negatively acknowledged cumulatively" ambiguous?

[GitBox]

GitHub user startjava added a comment to the discussion: [Doc] "negatively 
acknowledged cumulatively" ambiguous?

Actually I want to know what situation belongs to "the acknowledged 
cumulatively"?
thank you !

GitHub link: 
https://github.com/apache/pulsar/discussions/18771#discussioncomment-4324376


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] startjava added a comment to the discussion: [Doc] "negatively acknowledged cumulatively" ambiguous?

[GitBox]

GitHub user startjava added a comment to the discussion: [Doc] "negatively 
acknowledged cumulatively" ambiguous?

but i don't know what mean "negatively acknowledged cumulatively"。



GitHub link: 
https://github.com/apache/pulsar/discussions/18771#discussioncomment-4324383


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] tisonkun added a comment to the discussion: [Doc] "negatively acknowledged cumulatively" ambiguous?

[GitBox]

GitHub user tisonkun added a comment to the discussion: [Doc] "negatively 
acknowledged cumulatively" ambiguous?

This is about knowledge sharing. Not an issue to resolve.

GitHub link: 
https://github.com/apache/pulsar/discussions/18771#discussioncomment-4324394


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] flowchartsman added a comment to the discussion: Go Function SDK Feature Parity

[GitBox]

GitHub user flowchartsman added a comment to the discussion: Go Function SDK 
Feature Parity

Please comment with any other issues you feel like you need to track, and they 
can be added to the master list above.

GitHub link: 
https://github.com/apache/pulsar/discussions/18768#discussioncomment-4325648


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] codelipenghui created a discussion: Dataset by pulsar functions

[GitBox]

GitHub user codelipenghui created a discussion: Dataset by pulsar functions

When use pulsar functions, The ending strategy of consume messages is useful in 
dataset scenes.

Such as:

1. Ending by message produce time.
2. Ending by message id.
3. Ending by user(user support the ending strategy).

Is this feature suitable for pulsar function? 
The reason for this feature is i think pulsar function can support offline not 
only stream.


GitHub link: https://github.com/apache/pulsar/discussions/18773


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] srkukarni added a comment to the discussion: Dataset by pulsar functions

[GitBox]

GitHub user srkukarni added a comment to the discussion: Dataset by pulsar 
functions

This feature indeed make sense. Is it possible for you to to volunteer to add 
this functionality?

GitHub link: 
https://github.com/apache/pulsar/discussions/18773#discussioncomment-4327675


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] flowchartsman edited a discussion: Go Function SDK Feature Parity

[GitBox]

GitHub user flowchartsman edited a discussion: Go Function SDK Feature Parity

The Pulsar functions for Go SDK needs some love. It is lacking in some key 
features and has a couple of rough edges. Broadly, I think the SDK has been 
held back a little bit from trying to be too much like the Java version, which 
has resulted in a couple awkward abstractions such as the runtime function 
deflection dispatch and 
[NewOuputMessage()](https://github.com/apache/pulsar/pull/8327).

This discussion serves to track interest and hopefully chart a way forward.

- [ ] State support
  - https://github.com/apache/pulsar/issues/5138
  - https://github.com/apache/pulsar/issues/10560
  - https://github.com/apache/bookkeeper/pull/2750 - Abandoned?
  - [Reference 
implementation](https://pkg.go.dev/github.com/phemmer/go-pulsar-state/pulsar?utm_source=godoc)
 with 
[usage](https://github.com/phemmer/go-pulsar-state/blob/master/pulsar/examples/wordcount/wordcount.go)
- [ ] Better multi-topic dispatch
  - https://github.com/apache/pulsar/issues/9512
  - https://github.com/apache/pulsar/issues/9511

GitHub link: https://github.com/apache/pulsar/discussions/18768


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] flowchartsman edited a discussion: Go Function SDK Feature Parity

[GitBox]

GitHub user flowchartsman edited a discussion: Go Function SDK Feature Parity

The Pulsar functions for Go SDK needs some love. It is lacking in some key 
features and has a couple of rough edges. Broadly, I think the SDK has been 
held back a little bit from trying to be too much like the Java version, which 
has resulted in a couple awkward abstractions such as the runtime function 
deflection dispatch and 
[NewOuputMessage()](https://github.com/apache/pulsar/pull/8327).

This discussion serves to track interest and hopefully chart a way forward.

- [ ] State support
  - https://github.com/apache/pulsar/issues/5138
  - https://github.com/apache/pulsar/issues/10560
  - https://github.com/apache/bookkeeper/pull/2750 - Abandoned?
  - [Reference 
implementation](https://pkg.go.dev/github.com/phemmer/go-pulsar-state/pulsar?utm_source=godoc)
 with 
[usage](https://github.com/phemmer/go-pulsar-state/blob/master/pulsar/examples/wordcount/wordcount.go)
- [ ] Better multi-topic dispatch
  - https://github.com/apache/pulsar/issues/9512
  - https://github.com/apache/pulsar/issues/9511
- [ ] Secrets Provider
  - https://github.com/apache/pulsar/issues/8425 (stalled/abandoned)
- [ ] Pre-function init
- [ ] Depencency-free logging
- [ ] Type-safe function signatures

GitHub link: https://github.com/apache/pulsar/discussions/18768


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] flowchartsman added a comment to the discussion: Go Function SDK Feature Parity

[GitBox]

GitHub user flowchartsman added a comment to the discussion: Go Function SDK 
Feature Parity

### Pre-function init

Most things available from the function context such as tenant, namespace, etc 
should be available without needing the function context. This exists for user 
config, and should exist for secrets at the very least.

GitHub link: 
https://github.com/apache/pulsar/discussions/18768#discussioncomment-4328428


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] flowchartsman added a comment to the discussion: Go Function SDK Feature Parity

[GitBox]

GitHub user flowchartsman added a comment to the discussion: Go Function SDK 
Feature Parity

###  Depencency-free logging
A [new standard, leveling logger](https://pkg.go.dev/golang.org/x/exp/slog) 
will likely be landing in the standard library in the next few versions. 
Switching to this will provide a more idiomatic logging interface and remove 
the dependency on logrus.

GitHub link: 
https://github.com/apache/pulsar/discussions/18768#discussioncomment-4328486


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] flowchartsman edited a comment on the discussion: Go Function SDK Feature Parity

[GitBox]

GitHub user flowchartsman edited a comment on the discussion: Go Function SDK 
Feature Parity

### Pre-function init

Most things available from the function context such as tenant, namespace, etc 
should be available without needing the function context. This exists for user 
config, and should exist for secrets at the very least.

Logging should also be available as soon as possible, instead of after the 
function is started.

GitHub link: 
https://github.com/apache/pulsar/discussions/18768#discussioncomment-4328428


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] Anonymitaet added a comment to the discussion: [Website + Doc] generate connector configuration info automatically

[GitBox]

GitHub user Anonymitaet added a comment to the discussion: [Website + Doc] 
generate connector configuration info automatically

Why close this issue? What are the criteria for keeping an issue open or 
closed? 

GitHub link: 
https://github.com/apache/pulsar/discussions/18776#discussioncomment-4328571


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] github-actions[bot] added a comment to the discussion: [Website + Doc] generate connector configuration info automatically

[GitBox]

GitHub user github-actions[bot] added a comment to the discussion: [Website + 
Doc] generate connector configuration info automatically

The issue had no activity for 30 days, mark with Stale label.

GitHub link: 
https://github.com/apache/pulsar/discussions/18776#discussioncomment-4328568


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] tisonkun added a comment to the discussion: [Website + Doc] generate connector configuration info automatically

[GitBox]

GitHub user tisonkun added a comment to the discussion: [Website + Doc] 
generate connector configuration info automatically

Closed as stale and no progress.

GitHub link: 
https://github.com/apache/pulsar/discussions/18776#discussioncomment-4328570


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org