Is anyone seriously thinking about alternatives to microbatches? On Wed, Oct 19, 2016 at 2:45 PM, Michael Armbrust <mich...@databricks.com> wrote: > Anything that is actively being designed should be in JIRA, and it seems > like you found most of it. In general, release windows can be found on the > wiki. > > 2.1 has a lot of stability fixes as well as the kafka support you mentioned. > It may also include some of the following. > > The items I'd like to start thinking about next are: > - Evicting state from the store based on event time watermarks > - Sessionization (grouping together related events by key / eventTime) > - Improvements to the query planner (remove some of the restrictions on > what queries can be run). > > This is roughly in order based on what I've been hearing users hit the most. > Would love more feedback on what is blocking real use cases. > > On Tue, Oct 18, 2016 at 1:51 AM, Ofir Manor <ofir.ma...@equalum.io> wrote: >> >> Hi, >> I hope it is the right forum. >> I am looking for some information of what to expect from >> StructuredStreaming in its next releases to help me choose when / where to >> start using it more seriously (or where to invest in workarounds and where >> to wait). I couldn't find a good place where such planning discussed for 2.1 >> (like, for example ML and SPARK-15581). >> I'm aware of the 2.0 documented limits >> (http://spark.apache.org/docs/2.0.1/structured-streaming-programming-guide.html#unsupported-operations), >> like no support for multiple aggregations levels, joins are strictly to a >> static dataset (no SCD or stream-stream) etc, limited sources / sinks (like >> no sink for interactive queries) etc etc >> I'm also aware of some changes that have landed in master, like the new >> Kafka 0.10 source (and its on-going improvements) in SPARK-15406, the >> metrics in SPARK-17731, and some improvements for the file source. >> If I remember correctly, the discussion on Spark release cadence concluded >> with a preference to a four-month cycles, with likely code freeze pretty >> soon (end of October). So I believe the scope for 2.1 should likely quite >> clear to some, and that 2.2 planning should likely be starting about now. >> Any visibility / sharing will be highly appreciated! >> thanks in advance, >> >> Ofir Manor >> >> Co-Founder & CTO | Equalum >> >> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io > >
--------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
