Re: [DISCUSS] Deprecate DStream in 3.4

2023-01-12 Thread L. C. Hsieh
+1 On Thu, Jan 12, 2023 at 10:39 PM Jungtaek Lim wrote: > > Yes, exactly. I'm sorry to bring confusion - should have clarified action > items on the proposal. > > On Fri, Jan 13, 2023 at 3:31 PM Dongjoon Hyun wrote: >> >> Then, could you elaborate `the proposed code change` specifically? >> May

Re: [DISCUSS] Deprecate DStream in 3.4

2023-01-12 Thread Jungtaek Lim
Yes, exactly. I'm sorry to bring confusion - should have clarified action items on the proposal. On Fri, Jan 13, 2023 at 3:31 PM Dongjoon Hyun wrote: > Then, could you elaborate `the proposed code change` specifically? > Maybe, usual deprecation warning logs and annotation on the API? > > > On T

Re: [DISCUSS] Deprecate DStream in 3.4

2023-01-12 Thread Dongjoon Hyun
Then, could you elaborate `the proposed code change` specifically? Maybe, usual deprecation warning logs and annotation on the API? On Thu, Jan 12, 2023 at 10:05 PM Jungtaek Lim wrote: > Maybe I need to clarify - my proposal is "explicitly" deprecating it, > which incurs code change for sure. G

Re: [DISCUSS] Deprecate DStream in 3.4

2023-01-12 Thread Jungtaek Lim
There might be possible terminology differences, so let me elaborate the action item from the proposal explicitly: - Add "deprecation" annotation to the user-facing public API in streaming directory (DStream) - Write a release note to explicitly mention the deprecation. (Maybe promote again that t

Re: [DISCUSS] Deprecate DStream in 3.4

2023-01-12 Thread Jungtaek Lim
Maybe I need to clarify - my proposal is "explicitly" deprecating it, which incurs code change for sure. Guidance on the Spark website is done already as I mentioned - we updated the DStream doc page to mention that DStream is a "legacy" project and users should move to SS. I don't feel this is suf

Re: [DISCUSS] Deprecate DStream in 3.4

2023-01-12 Thread Reynold Xin
+1 On Thu, Jan 12, 2023 at 9:46 PM, Dongjoon Hyun < dongjoon.h...@gmail.com > wrote: > > +1 for the proposal (guiding only without any code change). > > > Thanks, > Dongjoon. > > On Thu, Jan 12, 2023 at 9:33 PM Shixiong Zhu < zsxwing@ gmail. com ( > zsxw...@gmail.com ) > wrote: > > >> +1 >

Re: [DISCUSS] Deprecate DStream in 3.4

2023-01-12 Thread Dongjoon Hyun
+1 for the proposal (guiding only without any code change). Thanks, Dongjoon. On Thu, Jan 12, 2023 at 9:33 PM Shixiong Zhu wrote: > +1 > > > On Thu, Jan 12, 2023 at 5:08 PM Tathagata Das > wrote: > >> +1 >> >> On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon wrote: >> >>> +1 >>> >>> On Fri, 13 Ja

Re: [DISCUSS] Deprecate DStream in 3.4

2023-01-12 Thread Shixiong Zhu
+1 On Thu, Jan 12, 2023 at 5:08 PM Tathagata Das wrote: > +1 > > On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon wrote: > >> +1 >> >> On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim >> wrote: >> >>> bump for more visibility. >>> >>> On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim < >>> kabhwan.opensou.

Re: [DISCUSS] Deprecate DStream in 3.4

2023-01-12 Thread Tathagata Das
+1 On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon wrote: > +1 > > On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim > wrote: > >> bump for more visibility. >> >> On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> Hi dev, >>> >>> I'd like to propose the depreca

Re: [DISCUSS] Deprecate DStream in 3.4

2023-01-12 Thread Hyukjin Kwon
+1 On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim wrote: > bump for more visibility. > > On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim < > kabhwan.opensou...@gmail.com> wrote: > >> Hi dev, >> >> I'd like to propose the deprecation of DStream in Spark 3.4, in favor of >> promoting Structured Streamin

Re: [DISCUSS] Deprecate DStream in 3.4

2023-01-12 Thread Jungtaek Lim
bump for more visibility. On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim wrote: > Hi dev, > > I'd like to propose the deprecation of DStream in Spark 3.4, in favor of > promoting Structured Streaming. > (Sorry for the late proposal, if we don't make the change in 3.4, we will > have to wait for a