Re: [DISCUSS] SPIP: Structured Streaming - Arbitrary State API v2

+1 on the newer APIs. I believe these APIs provide a much powerful mechanism for the user to perform arbitrary state management in Structured Streaming queries. Thanks Bhuwan Sahni On Mon, Jan 8, 2024 at 10:07 AM L. C. Hsieh wrote: > +1 > > I left some comments in the SPIP doc and go

Re: [VOTE] SPIP: Structured Streaming - Arbitrary State API v2

+1. This is a good addition. <http://www.databricks.com> *Bhuwan Sahni* Staff Software Engineer bhuwan.sa...@databricks.com 500 108th Ave. NE Bellevue, WA 98004 USA On Wed, Jan 10, 2024 at 9:00 AM Burak Yavuz wrote: > +1. Excited to see more stateful workloads with Structured

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

TIJCJaVCLHGgHDJOCmJ11L9pU6yO7k0wc9hCg$> >>>>>>>> .) >>>>>>>> >>>>>>>> Is anyone interested in mentoring and reviewing this effort? >>>>>>>> >>>>>>>> Also can the repository admin/owner re-open the PR? ( I guess >>>>>>>> people only with admin access to the repository can do that). >>>>>>>> >>>>>>>> Thank you, >>>>>>>> >>>>>>>> Pavan >>>>>>>> >>>>>>>> On Tue, Feb 20, 2024 at 2:08 PM Krystal Mitchell >>>>>>>> wrote: >>>>>>>> >>>>>>>>> +1 >>>>>>>>> >>>>>>>>> On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote: >>>>>>>>> > Thanks for proposing and voting for the feature Mich. >>>>>>>>> > >>>>>>>>> > adding some references to the thread. >>>>>>>>> > >>>>>>>>> >- Jira ticket - SPARK-24815 >>>>>>>>> ><https://issues.apache.org/jira/browse/SPARK-24815> >>>>>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r64f0rbw$> >>>>>>>>> >- Design Doc >>>>>>>>> >< >>>>>>>>> https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing> >>>>>>>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r44a1rO8$> >>>>>>>>> > >>>>>>>>> >- discussion thread >>>>>>>>> >< >>>>>>>>> https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo> >>>>>>>>> <https://urldefense.com/v3/__https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rkLpTOYM$> >>>>>>>>> >- PR with initial implementation - >>>>>>>>> >https://github.com/apache/spark/pull/42352 >>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rZAZFOls$> >>>>>>>>> > >>>>>>>>> > Please vote with: >>>>>>>>> > >>>>>>>>> > [ ] +1: Accept the proposal and start with the development. >>>>>>>>> > [ ] +0 >>>>>>>>> > [ ] -1: I don’t think this is a good idea because … >>>>>>>>> > >>>>>>>>> > Thank you, >>>>>>>>> > >>>>>>>>> > Pavan >>>>>>>>> > >>>>>>>>> > On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh >>>>>>>> > >>>>>>>>> > wrote: >>>>>>>>> > >>>>>>>>> > > >>>>>>>>> > > +1 for me (non binding) >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > *Disclaimer:* Use it at your own risk. Any and all >>>>>>>>> responsibility for any >>>>>>>>> > > loss, damage or destruction of data or any other property >>>>>>>>> which may arise >>>>>>>>> > > from relying on this email's technical content is explicitly >>>>>>>>> disclaimed. >>>>>>>>> > > The author will in no case be liable for any monetary damages >>>>>>>>> arising from >>>>>>>>> > > such loss, damage or destruction. >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > >>>>>>>>> >>>>>>>> -- <http://www.databricks.com> *Bhuwan Sahni* Staff Software Engineer bhuwan.sa...@databricks.com 500 108th Ave. NE Bellevue, WA 98004 USA

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Thanks a lot for creating the risk table Pavan. My apologies. I was tied up with high priority items for the last couple weeks and could not respond. I will review the PR by tomorrow's end, and get back to you. Appreciate your patience. Thanks Bhuwan Sahni On Sun, Mar 17, 2024 at 4:42 PM

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Pavan, I looked at the PR, and the changes look simple and contained. It would be useful to add dynamic resource allocation to Spark Structured Streaming. Jungtaek. Would you be able to shepherd this change? On Tue, Mar 19, 2024 at 10:38 AM Bhuwan Sahni wrote: > Thanks a lot for creat