BTW the largest change to SS right now is probably the entire data source API v2 effort, which aims to unify streaming and batch from data source perspective, and provide a reliable, expressive source/sink API.
On Mon, Jan 14, 2019 at 5:34 PM, Reynold Xin < r...@databricks.com > wrote: > > There are a few things to keep in mind: > > > > 1. Structured Streaming isn't an independent project. It actually (by > design) depends on all the rest of Spark SQL, and virtually all > improvements to Spark SQL benefit Structured Streaming. > > > > 2. The project as far as I can tell is relatively mature for core ETL and > incremental processing purpose. I interact with a lot of users using it > everyday. We can always expand the use cases and add more, but that also > adds maintenance burden. In any case, it'd be good to get some activity > here. > > > > > > > > > On Mon, Jan 14, 2019 at 5:11 PM, Nicholas Chammas < nicholas. chammas@ gmail. > com ( nicholas.cham...@gmail.com ) > wrote: > >> As an observer, this thread is interesting and concerning. Is there an >> emerging consensus that Structured Streaming is somehow not relevant >> anymore? Or is it just that folks consider it "complete enough"? >> >> >> Structured Streaming was billed as the replacement to DStreams. If >> committers, generally speaking, have lost interest in Structured >> Streaming, does that mean the Apache Spark project is somehow no longer >> aiming to provide a "first-class" solution to the problem of stream >> processing? >> >> On Mon, Jan 14, 2019 at 3:43 PM Jungtaek Lim < kabhwan@ gmail. com ( >> kabh...@gmail.com ) > wrote: >> >> >>> Cody, I guess I already addressed your comments in the PR (#22138). The >>> approach was changed to address your concern, and after that Gabor helped >>> to review the PR. Please take a look again when you have time to get into. >>> >>> >>> >>> 2019년 1월 15일 (화) 오전 1:01, Cody Koeninger < cody@ koeninger. org ( >>> c...@koeninger.org ) >님이 작성: >>> >>> >>>> I feel like I've already said my piece on >>>> https:/ / github. com/ apache/ spark/ pull/ 22138 ( >>>> https://github.com/apache/spark/pull/22138 ) let me know if you have >>>> more questions. >>>> >>>> As for SS in general, I don't have a production SS deployment, so I'm >>>> less comfortable with reviewing large changes to it. But if no other >>>> committers are working on it... >>>> >>>> On Sun, Jan 13, 2019 at 5:19 PM Sean Owen < srowen@ gmail. com ( >>>> sro...@gmail.com ) > wrote: >>>> > >>>> > Yes you're preaching to the choir here. SS does seem somewhat >>>> > abandoned by those that have worked on it. I have also been at times >>>> > frustrated that some areas fall into this pattern. >>>> > >>>> > There isn't a way to make people work on it, and I personally am not >>>> > interested in it nor have a background in SS. >>>> > >>>> > I did leave some comments on your PR and will see if we can get >>>> > comfortable with merging it, as I presume you are pretty knowledgeable >>>> > about the change. >>>> > >>>> > On Sun, Jan 13, 2019 at 4:55 PM Jungtaek Lim < kabhwan@ gmail. com ( >>>> kabh...@gmail.com ) > wrote: >>>> > > >>>> > > Sean, this is actually a fail-back on pinging committers. I know who >>>> can review and merge in SS area, and pinged to them, didn't work. Even >>>> there's a PR which approach was encouraged by committer and reviewed the >>>> first phase, and no review. >>>> > > >>>> > > That's not the first time I have faced the situation, and I used the >>>> fail-back approach at that time. (You can see there was no response even >>>> in the mail thread.) Not sure which approach worked. >>>> > > https:/ / lists. apache. org/ thread. html/ >>>> > > c61f32249949b1ff1b265c1a7148c2ea7eda08891e3016fb24008561@ >>>> %3Cdev. spark. apache. org%3E ( >>>> https://lists.apache.org/thread.html/c61f32249949b1ff1b265c1a7148c2ea7eda08891e3016fb24008561@%3Cdev.spark.apache.org%3E >>>> ) >>>> > > >>>> > > I've observed that only (critical) bugfixes are being reviewed and >>>> merged in time for SS area. For other stuffs like new features and >>>> improvements, both discussions and PRs were pretty less popular from >>>> committers: though there was even participation/approve from non-committer >>>> community. I don't think SS is the thing to be turned into maintenance. >>>> > > >>>> > > I guess PMC members should try to resolve such situation, as it will >>>> (slowly and quietly) make some issues like contributors leaving, module >>>> stopped growing up, etc.. The problem will grow up like a snowball: >>>> getting bigger and bigger. I don't mind if there's no interest on both >>>> contributors and committers for such module, but SS is not. Maybe either >>>> other committers who weren't familiar with should try to get familiar and >>>> cover the area, or the area needs more committers. >>>> > > >>>> > > -Jungtaek Lim (HeartSaVioR) >>>> > > >>>> > > 2019년 1월 13일 (일) 오후 11:37, Sean Owen < srowen@ gmail. com ( >>>> sro...@gmail.com ) >님이 작성: >>>> > >> >>>> > >> Jungtaek, the best strategy is to find who wrote the code you are >>>> > >> modifying (use Github history or git blame) and ping them directly on >>>> >>>> > >> the PR. I don't know this code well myself. >>>> > >> It also helps if you can address why the functionality is important, >>>> > >> and describe compatibility implications. >>>> > >> >>>> > >> Most PRs are not merged, note. Not commenting on this particular one, >>>> >>>> > >> but it's not a 'bug' if it's not being merged. >>>> > >> >>>> > >> On Sun, Jan 13, 2019 at 12:29 AM Jungtaek Lim < kabhwan@ gmail. com ( >>>> kabh...@gmail.com ) > wrote: >>>> > >> > >>>> > >> > I'm sorry but let me remind this, as non-SS PRs are being reviewed >>>> accordingly, whereas many of SS PRs (regardless of who create) are still >>>> not reviewed and merged in time. >>>> > >> > >>>> > >> > 2019년 1월 3일 (목) 오전 7:57, Jungtaek Lim < kabhwan@ gmail. com ( >>>> kabh...@gmail.com ) >님이 작성: >>>> > >> >> >>>> > >> >> Spark devs, happy new year! >>>> > >> >> >>>> > >> >> I would like to remind this kindly, since there was actually no >>>> review after initiating the thread. >>>> > >> >> >>>> > >> >> Thanks, >>>> > >> >> Jungtaek Lim (HeartSaVioR) >>>> > >> >> >>>> > >> >> 2018년 12월 12일 (수) 오후 11:12, Vaclav Kosar < admin@ vaclavkosar. com >>>> ( ad...@vaclavkosar.com ) >님이 작성: >>>> > >> >>> >>>> > >> >>> I am also waiting for any finalization of my PR [3]. I seems that >>>> SS PRs are not being reviewed much these days. >>>> > >> >>> >>>> > >> >>> [3] https:/ / github. com/ apache/ spark/ pull/ 21919 ( >>>> https://github.com/apache/spark/pull/21919 ) >>>> > >> >>> >>>> > >> >>> >>>> > >> >>> On 12. 12. 18 14:37, Dongjin Lee wrote: >>>> > >> >>> >>>> > >> >>> If it is possible, could you review my PR on Kafka's header >>>> functionality[^1] also? It was added in Kafka 0.11.0.0 but still not >>>> supported in Spark. >>>> > >> >>> >>>> > >> >>> Thanks, >>>> > >> >>> Dongjin >>>> > >> >>> >>>> > >> >>> [^1]: https:/ / github. com/ apache/ spark/ pull/ 22282 ( >>>> https://github.com/apache/spark/pull/22282 ) >>>> > >> >>> [^2]: https:/ / issues. apache. org/ jira/ browse/ KAFKA-4208 ( >>>> https://issues.apache.org/jira/browse/KAFKA-4208 ) >>>> > >> >>> >>>> > >> >>> On Wed, Dec 12, 2018 at 6:43 PM Jungtaek Lim < kabhwan@ gmail. com >>>> ( kabh...@gmail.com ) > wrote: >>>> > >> >>>> >>>> > >> >>>> Hi devs, >>>> > >> >>>> >>>> > >> >>>> Would I kindly ask for reviewing on PRs for Structured >>>> Streaming? I have 5 open pull requests on SS side [1] (earliest PR was >>>> opened around 4 months so far), and there looks like couple of PR for >>>> others [2] which looks good to be reviewed, too. >>>> > >> >>>> >>>> > >> >>>> Thanks in advance, >>>> > >> >>>> Jungtaek Lim (HeartSaVioR) >>>> > >> >>>> >>>> > >> >>>> 1. https:/ / github. com/ apache/ spark/ >>>> > >> >>>> pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+author%3AHeartSaVioR+%5BSS%5D >>>> ( >>>> https://github.com/apache/spark/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+author%3AHeartSaVioR+%5BSS%5D >>>> ) >>>> > >> >>>> 2. https:/ / github. com/ apache/ spark/ >>>> > >> >>>> pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+%5BSS%5D+ >>>> ( >>>> https://github.com/apache/spark/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+%5BSS%5D+ >>>> ) >>>> > >> >>>> >>>> > >> >>> >>>> > >> >>> >>>> > >> >>> -- >>>> > >> >>> Dongjin Lee >>>> > >> >>> >>>> > >> >>> A hitchhiker in the mathematical world. >>>> > >> >>> >>>> > >> >>> github: github. com/ dongjinleekr ( http://github.com/dongjinleekr >>>> ) >>>> > >> >>> linkedin: kr. linkedin. com/ in/ dongjinleekr ( >>>> http://kr.linkedin.com/in/dongjinleekr ) >>>> > >> >>> speakerdeck: speakerdeck. com/ dongjin ( >>>> http://speakerdeck.com/dongjin ) >>>> > >>>> > --------------------------------------------------------------------- >>>> > To unsubscribe e-mail: dev-unsubscribe@ spark. apache. org ( >>>> dev-unsubscr...@spark.apache.org ) >>>> > >>> >>> >>> >> >> > >
