[discuss] SPIP: Continuous Processing Mode for Structured Streaming

2017-10-23 Thread Reynold Xin
Please take a look at the attached PDF for the SPIP: Continuous Processing Mode for Structured Streaming https://issues.apache.org/jira/browse/SPARK-20928 It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also

[Spark SQL] experimental join strategy

2017-10-23 Thread Jonathan Gray
Hi, I'm trying to implement a custom join strategy using the experimental extraStrategies extension point. I have created a very simple implementation (see the below gist) but currently it is failing and I don't understand why. https://gist.github.com/jongray/a413b5401c4edd47b7d8f559e5b2f79b I