Re: Enhanced Console Sink for Structured Streaming

2024-02-08 Thread Jerry Peng
I am generally a +1 on this as we can use this information in our docs to demonstrate certains concepts to potential users. I am in agreement with other reviewers that we should keep the existing default behavior of the console sink. This new style of output should be enabled behind a flag. As f

Re: [DISCUSS] Deprecate DStream in 3.4

2023-01-13 Thread Jerry Peng
+1 in general for marking the DStreams API as deprecated Jungtaek, can you please provide / elaborate on the concrete actions you intend on taking for the depreciation process? Best, Jerry On Thu, Jan 12, 2023 at 11:16 PM L. C. Hsieh wrote: > +1 > > On Thu, Jan 12, 2023 at 10:39 PM Jungtaek L

Re: [VOTE][RESULT][SPIP] Asynchronous Offset Management in Structured Streaming

2022-12-05 Thread Jerry Peng
Thanks Jungtaek for shepherding this effort! On Sun, Dec 4, 2022 at 6:25 PM Jungtaek Lim wrote: > The vote passes with 7 +1s (5 binding +1s). > Thanks to all who reviews the SPIP doc and votes! > > (* = binding) > +1: > - Jungtaek Lim > - Xingbo Jiang > - Mridul Muralidharan (*) > - Hyukjin Kwon

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

2022-11-28 Thread Jerry Peng
d move it to deprecated ? >> >> I am much more comfortable not supporting this SPIP for CM if it was >> deprecated. >> >> Thoughts ? >> >> Regards, >> Mridul >> >> >> >> >> On Wed, Nov 23, 2022 at 1:16 AM Jerry Pen

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

2022-11-22 Thread Jerry Peng
> ps. Eventually we could probably discuss retiring continuous mode if the > new approach gets accepted and eventually considered as a stable one after > several minor releases. That's just me. > > On Wed, Nov 23, 2022 at 5:16 AM Jerry Peng > wrote: > >> Hi all, >>

[DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

2022-11-22 Thread Jerry Peng
Hi all, I would like to start the discussion for a SPIP, Asynchronous Offset Management in Structured Streaming. The high level summary of the SPIP is that currently in Structured Streaming we perform a couple of offset management operations for progress tracking purposes synchronously on the cri

Re: [VOTE] Release Spark 3.3.0 (RC5)

2022-06-08 Thread Jerry Peng
I agree with Jungtaek, -1 from me because of the issue of Kafka source throwing an error with an incorrect error message that was introduced recently. This may mislead users and cause unnecessary confusion. On Wed, Jun 8, 2022 at 12:04 AM Jungtaek Lim wrote: > Apologize for late participation.