If you're suggesting making it un-Experimental, probably yes, as it is de facto not going to change much I expect. If you're saying remove it, probably not? I don't see that it's anywhere near deprecated, and not sure it's unmaintained - obviously tests etc still have to keep passing.
On Mon, Sep 14, 2020 at 11:34 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > > Hi devs, > > It was Spark 2.3 in Feb 2018 which introduced continuous mode in Structured > Streaming as "experimental". > > Now we are here at 2.5 years after its release - I feel it would be a good > time to evaluate the mode, whether the mode has been widely used or not, and > the mode has been making progress, as the mode is "experimental". > > At least from the surface I don't see any active effort for continuous mode > around the community - the last major effort was stateful operation which was > incomplete and I removed that. There were some couples of bug reports as well > as fixes more than a year ago and almost nothing has been handled. (A trivial > bugfix PR has been merged recently but that's all.) The new features > introduced to the Structured Streaming (at least observable metrics, SS UI) > don't apply to continuous mode, and no one made "support continuous mode" as > a hard requirement on passing review in these PRs. > > I have no idea how many companies are using the mode in production (please > add the voice if someone has statistics about this) but I don't see any bug > reports recently, and see only a few questions in SO, which makes me think > about cost on maintenance. > > I know there's a mood to avoid discontinue support as possible, but it sounds > weird to keep something as "unmaintained", especially it's still > "experimental" and main authors are no more active enough to promise > maintenance/improvement on the module. Thoughts? > > Thanks, > Jungtaek Lim (HeartSaVioR) --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
