Re: Spark structured streaming watermarks on nested attributes

2019-05-06 Thread Joe Ammann
Hi Yuanjian On 5/7/19 4:55 AM, Yuanjian Li wrote: > Hi Joe > > I think you met this issue: https://issues.apache.org/jira/browse/SPARK-27340 > You can check the description in Jira and PR. We also met this in our > production env and fixed by the providing PR. > > The PR is still in review. cc

Re: Spark structured streaming watermarks on nested attributes

2019-05-06 Thread Yuanjian Li
Hi Joe I think you met this issue: https://issues.apache.org/jira/browse/SPARK-27340 You can check the description in Jira and PR. We also met this in our production env and fixed by the providing PR. The PR is still in review. cc Langchang Zhu(zhuliangch...@baidu.com), who's the author for the f

Re: Spark structured streaming watermarks on nested attributes

2019-05-06 Thread Joe Ammann
On 5/6/19 6:23 PM, Pat Ferrel wrote: > Streams have no end until watermarked or closed. Joins need bounded datasets, > et voila. Something tells me you should consider the streaming nature of your > data and whether your joins need to use increments/snippets of infinite > streams or to re-join t

Re: Spark structured streaming watermarks on nested attributes

2019-05-06 Thread Pat Ferrel
Streams have no end until watermarked or closed. Joins need bounded datasets, et voila. Something tells me you should consider the streaming nature of your data and whether your joins need to use increments/snippets of infinite streams or to re-join the entire contents of the streams accumulated at