Hi TD,
Thanks for that. The only reason I ask is I don't see any alternative
solution to solve the problem below using raw sql.
How to select the max row for every group in spark structured streaming
2.3.0 without using order by since it requires complete mode or
mapGroupWithState?
*Input:*
id | amount | my_timestamp
-------------------------------------------
1 | 5 | 2018-04-01T01:00:00.000Z
1 | 10 | 2018-04-01T01:10:00.000Z
2 | 20 | 2018-04-01T01:20:00.000Z
2 | 30 | 2018-04-01T01:25:00.000Z
2 | 40 | 2018-04-01T01:30:00.000Z
*Expected Output:*
id | amount | my_timestamp
-------------------------------------------
1 | 10 | 2018-04-01T01:10:00.000Z
2 | 40 | 2018-04-01T01:30:00.000Z
Looking for a streaming solution using either raw sql like
sparkSession.sql("sql
query") or similar to raw sql but not something like mapGroupWithState
On Mon, Apr 16, 2018 at 8:32 PM, Tathagata Das <tathagata.das1...@gmail.com>
wrote:
> Unfortunately no. Honestly it does not make sense as for type-aware
> operations like map, mapGroups, etc., you have to provide an actual JVM
> function. That does not fit in with the SQL language structure.
>
> On Mon, Apr 16, 2018 at 7:34 PM, kant kodali <kanth...@gmail.com> wrote:
>
>> Hi All,
>>
>> can we use mapGroupsWithState in raw SQL? or is it in the roadmap?
>>
>> Thanks!
>>
>>
>>
>
