[GitHub] [pulsar-adapters] aditiwari01 opened a new pull request #31: [Issue #29] [pulsar-spark] Adding SparkPulsarReliableReceiver
aditiwari01 opened a new pull request #31: URL: https://github.com/apache/pulsar-adapters/pull/31 Fixes #29 ### Motivation Current pulsar-spark adapter uses spark_streaming_2.10 while scala dependency is 2.11. Apart from this current receiver does not take care about reliability, rate limit and. backpressure. Added a new receiver with all these considerations. ### Modifications Includes: 1. Updating spark_streaming_2.10 to spark_streaming_2.11. 2. Includes rate limit/ backpressure logic in receiver. 3. Batch read from pulsar instead of record by record. 4. Making receiver reliable using batch store call. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-adapters] aditiwari01 commented on issue #29: Pulsar - Spark adapter for scala 2.11
aditiwari01 commented on issue #29: URL: https://github.com/apache/pulsar-adapters/issues/29#issuecomment-1001188765 @eolivelli I have raised a PR with a new receiver. I'm yet to add examples/documentation for the same, but would appreciate if you can go through it and provide the reeview. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-adapters] aditiwari01 edited a comment on issue #29: Pulsar - Spark adapter for scala 2.11
aditiwari01 edited a comment on issue #29: URL: https://github.com/apache/pulsar-adapters/issues/29#issuecomment-1001188765 @eolivelli I have raised a PR with a new receiver. I'm yet to add examples/documentation for the same, but would appreciate if you can go through it and provide the review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-adapters] aditiwari01 commented on issue #29: Pulsar - Spark adapter for scala 2.11
aditiwari01 commented on issue #29: URL: https://github.com/apache/pulsar-adapters/issues/29#issuecomment-1001189132 Also can you approve running workflows so I can check if there are any issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-adapters] aditiwari01 commented on issue #29: Pulsar - Spark adapter for scala 2.11
aditiwari01 commented on issue #29: URL: https://github.com/apache/pulsar-adapters/issues/29#issuecomment-1001223370 @eolivelli Need a small help. In local I've been building with skip License. Flow is failing because of `Some files do not have the expected license header` How can I add license to the new files that I have added? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [DISCUSS] PIP-124: Pulsar Client Shared State API
+1, Great feature. > It is not a goal to implement a Pulsar backed Database system But this does look like a database system. I believe there would be some limitations here, either in availability, consistency, performance, storage cost, etc. So I think we can provide more information about disadvantages to guide user's choice. On 2021/12/24 09:53:37 Enrico Olivelli wrote: > Hello everyone, > I want to start a discussion about PIP-124 Pulsar Client Shared State API > > This is the PIP document > https://github.com/apache/pulsar/issues/13490 > > This is a demo implementation (a proof-of-concept): > https://github.com/eolivelli/pulsar-shared-state-manager > > Please take a look and share your thoughts > > I believe that this will unlock the potential of the Exclusive > Producer and it will also make easier the life of many developers who > are using Pulsar and need some API to share configuration, metadata, > or any simple key-value data structure without adding a Database or > other components to their library, Pulsar IO connector or Pulsar > Protocol Handler. > > Thanks > Enrico >
Re: [ANNOUNCE] Apache Pulsar 2.9.1 released
Great work!. Penghui On Fri, Dec 24, 2021 at 10:04 PM mattison chao wrote: > great work! > > On Fri, 24 Dec 2021 at 19:23, 陳智弘 wrote: > > > I am really appreciate those contributors and volunteers to make this > > release happen! > > > > Dianjin Wang 於 2021年12月24日 週五 16:43 寫道： > > > > > Great! Thanks to all the contributors that made this release happen. > > > > > > Best, > > > Dianjin Wang > > > > > > > > > On Fri, Dec 24, 2021 at 3:18 PM Enrico Olivelli > > > wrote: > > > > > > > The Apache Pulsar team is proud to announce Apache Pulsar version > > 2.9.1. > > > > > > > > Pulsar is a highly scalable, low latency messaging platform running > on > > > > commodity hardware. It provides simple pub-sub semantics over topics, > > > > guaranteed at-least-once delivery of messages, automatic cursor > > > management > > > > for > > > > subscribers, and cross-datacenter replication. > > > > > > > > For Pulsar release details and downloads, visit: > > > > > > > > https://pulsar.apache.org/download > > > > > > > > Release Notes are at: > > > > http://pulsar.apache.org/release-notes > > > > > > > > We would like to thank the contributors that made the release > possible. > > > > > > > > Regards, > > > > The Pulsar Team > > > > > > > > > >
[DISCUSS] Apache Pulsar 2.10.0 release
Hi, everyone I hope you’ve all been doing well. I would like to start an email thread to discuss features that we planned for 2.10.0. According to the time-based release plan https://github.com/apache/pulsar/wiki/PIP-47%3A-Time-Based-Release-Plan, we should release 2.10.0 at the end of December 2021, since we have reached the end of December, I would like to target the 2.10.0 to the end of January 2022 There are some powerful features and enhancements in 2.10.0 such as - PIP 84: Message redelivery epoch - PIP 104: Add new consumer type: TableView - PIP 106: Negative acknowledgment backoff - PIP 110: Topic customized metadata support - PIP 117: Change Pulsar standalone defaults - PIP 118: Do not restart brokers when ZooKeeper session expires - PIP 119: Enable consistent hashing by default on KeyShared dispatcher - PIP 120: Enable client memory limit by default - PIP 121: Pulsar cluster level auto failover - PIP 123: Pulsar metadata CLI tool - Metadata service batch operations - RocksDB metadata service backend - Etcd metadata service backend - Ack timeout redelivery backoff policy - Global topic policies Most of them have been completed, some work in progress we need to try to complete within 2 weeks. This can give me a 2 week buffer period to prepare for release and complete the release vote. For the unfinished parts, we can move them to 2.11.0. Some proposals are just being discussed, so I do not list them because I'm not sure if we can complete them in two weeks. You can find all the change lists from https://github.com/apache/pulsar/pulls?q=milestone%3A2.10.0+-label%3Arelease%2F2.9.1 There are more than 500 commits. If I missed something or you have any suggestions please let me know. Regards, Penghui
Re: [DISCUSS] Apache Pulsar 2.10.0 release
PengHui, Il giorno lun 27 dic 2021 alle ore 05:47 PengHui Li ha scritto: > > Hi, everyone > > I hope you’ve all been doing well. I would like to start an email thread to > discuss features that we planned for 2.10.0. > According to the time-based release plan > https://github.com/apache/pulsar/wiki/PIP-47%3A-Time-Based-Release-Plan, > we should release 2.10.0 at the end of December 2021, since we have reached > the end of December, > I would like to target the 2.10.0 to the end of January 2022 makes sense Enrico > > There are some powerful features and enhancements in 2.10.0 such as > > - PIP 84: Message redelivery epoch > - PIP 104: Add new consumer type: TableView > - PIP 106: Negative acknowledgment backoff > - PIP 110: Topic customized metadata support > - PIP 117: Change Pulsar standalone defaults > - PIP 118: Do not restart brokers when ZooKeeper session expires > - PIP 119: Enable consistent hashing by default on KeyShared dispatcher > - PIP 120: Enable client memory limit by default > - PIP 121: Pulsar cluster level auto failover > - PIP 123: Pulsar metadata CLI tool > - Metadata service batch operations > - RocksDB metadata service backend > - Etcd metadata service backend > - Ack timeout redelivery backoff policy > - Global topic policies > > Most of them have been completed, some work in progress we need to try to > complete within 2 weeks. > This can give me a 2 week buffer period to prepare for release and complete > the release vote. > For the unfinished parts, we can move them to 2.11.0. > > Some proposals are just being discussed, so I do not list them because I'm > not sure if we can complete them in two weeks. > > You can find all the change lists from > https://github.com/apache/pulsar/pulls?q=milestone%3A2.10.0+-label%3Arelease%2F2.9.1 > There are more than 500 commits. > > If I missed something or you have any suggestions please let me know. > > Regards, > Penghui
