Spark JIRA SPARK-3655 to add a
transformation like this to core Spark.”
From: Ofir Manor
Date: Thursday, November 17, 2016 at 9:57 AM
To: "assaf.mendelson"
Cc: dev
Subject: Re: structured streaming and window functions
I agree with you, I think that once we will have sessionization
ou do it with structured streaming?
>
> As this is currently not supported, is there a plan on how to support it
> in the future?
>
> Assaf.
>
>
>
> *From:* Herman van Hövell tot Westerflier-2 [via Apache Spark Developers
> List] [mailto:[hidden email]
> <
]
[mailto:ml-node+s1001551n19935...@n3.nabble.com]
Sent: Thursday, November 17, 2016 5:13 PM
To: Mendelson, Assaf
Subject: Re: structured streaming and window functions
Assaf, I think what you are describing is actually sessionizing, by user, where
a session is ended by a successful login event.
On
rman van Hövell tot Westerflier-2 [via Apache Spark Developers
> List] [mailto:ml-node+[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=19934&i=0>]
> *Sent:* Thursday, November 17, 2016 1:27 PM
> *To:* Mendelson, Assaf
> *Subject:* Re: structured streaming
supported, is there a plan on how to support it in the
future?
Assaf.
From: Herman van Hövell tot Westerflier-2 [via Apache Spark Developers List]
[mailto:ml-node+s1001551n19933...@n3.nabble.com]
Sent: Thursday, November 17, 2016 1:27 PM
To: Mendelson, Assaf
Subject: Re: structured streaming and
What kind of window functions are we talking about? Structured streaming
only supports time window aggregates, not the more general sql window
function (sum(x) over (partition by ... order by ...)) aggregates.
The basic idea is that you use incremental aggregation and store the
aggregation buffer
