Re: Enhanced Console Sink for Structured Streaming

Agree, the default behavior does not need to change. Neil, how about separating it into two sections: - Actual rows in the sink (same as current output) - Followed by metadata data

Re: [EXTERNAL] Re: Spark Kafka Rack Aware Consumer

Overall the proposal to make this an option for Kafka source SGTM. You can address the doc review and can send PR (in parallel or after the review). Note that currently executors cache client connection to Kafka and reuse the connection and buffered records for next micro-batch. Your proposal would

Re: [VOTE] SPIP: Structured Streaming - Arbitrary State API v2

+1. This is a major improvement to the state API. Raghu. On Tue, Jan 9, 2024 at 1:42 AM Mich Talebzadeh wrote: > +1 for me as well > > > Mich Talebzadeh, > Dad | Technologist | Solutions Architect | Engineer > London > United Kingdom > > >view my Linkedin profile >

Re: [DISCUSS] SPIP: State Data Source - Reader

+1 overall and a big +1 to keeping offline state-rebalancing as a primary use case. Raghu. On Mon, Oct 16, 2023 at 11:25 AM Bartosz Konieczny wrote: > Thank you, Jungtaek, for your answers! It's clear now. > > +1 for me. It seems like a prerequisite for further ops-related > improvements for th

Re: Watermark on late data only

I like some way to expose watermarks to the user. It does affect the processing of the records, so it is relevant for the users. `current_watermark()` is a good option. The implementation of this might be engine specific. But it is a very relevant concept for authors of streaming pipelines. Ideally

Re: Spark 3.5 Branch Cut

Thanks Yuanjian for accepting these for warmfix. Raghu. On Mon, Jul 17, 2023 at 1:04 PM Yuanjian Li wrote: > Hi, all > > FYI, I cut branch-3.5 as https://github.com/apache/spark/tree/branch-3.5 > > Here is the complete list of exception merge requests received before the > cut: > >- > >

Re: [Reminder] Spark 3.5 Branch Cut

Thank you. We plan to get remaining major pieces for Streaming Spark Connect (Epic SPARK-42938 ). I would like to request a warmfix exception for the following tweaks and improvements over the next two weeks (all in the same epic). - SPARK-4294

Re: Time for Spark v3.5.0 release

We have a bunch of work in progress for Spark Connect trying to meet the branch cut deadline. Moving to 17th is certainly welcome. Is it feasible to extend it by a couple of more days? Alternatively, we could have a relaxed warmfix process for Spark Connect code for a week or two since it does no