Re: Spark structured streaming generate output path runtime

2018-06-04 Thread Swapnil Chougule
Thanks Jayesh. It worked for me. ~Swapnil On Fri, 1 Jun 2018, 7:10 pm Lalwani, Jayesh, wrote: > This will not work the way you have implemented it. The code that you have > here will be called only once before the streaming query is started. Once > the streaming query starts, this code is not c

Re: Spark structured streaming generate output path runtime

2018-06-01 Thread Lalwani, Jayesh
This will not work the way you have implemented it. The code that you have here will be called only once before the streaming query is started. Once the streaming query starts, this code is not called What I would do is 1. Implement a udf that calculates flourtimestamp 2. Add a column in