> .
>
> Best,
> Anastasios
>
> On Mon, Dec 18, 2017 at 10:38 AM, Julien CHAMP
> wrote:
>
>> I've been looking for several solutions but I can't find something
>> efficient to compute many window function efficiently ( optimized
>> computation or eff
I've been looking for several solutions but I can't find something
efficient to compute many window function efficiently ( optimized
computation or efficient parallelism )
Am I the only one interested by this ?
Regards,
Julien
Le ven. 15 déc. 2017 à 21:34, Julien CHAMP a écrit :
May be I should consider something like impala ?
Le ven. 15 déc. 2017 à 11:32, Julien CHAMP a écrit :
> Hi Spark Community members !
>
> I want to do several ( from 1 to 10) aggregate functions using window
> functions on something like 100 columns.
>
> Instead of doing severa
l").over(tw))
Is not really efficient :/
It seems that it iterates on the whole column for each aggregation ? Am I
right ?
Is there a way to compute all the required operations on a columns with a
single pass ?
Event better, to compute all the required operations on ALL columns with a
si
hebaane
a écrit :
> Hi Julien,
>
> Could you give more details about the problems you faced?
> Here is a working example with Spark dataframe and Spark SQL:
> https://gist.github.com/radcheb/d16042d8bb3815d3dd42030ecedc43cf
>
>
> Cheers,
> Radhwane Chebaane
>
>
FROM df a CROSS JOIN df b
> ON b.timestamp >= a.timestamp - 20L and b.timestamp <= a.timestamp
> ) c
> GROUP BY c.id, c.timestamp, c.value ORDER BY c.timestamp
>
>
> This must be also possible also on Spark Streaming however don't expect high
> performance.
>
(
https://issues.apache.org/jira/browse/SPARK-19451 ) when working with Long
values !!! So I can't use this
So my question is ( of course ) how can I resolve my problem ?
If I use spark streaming I will face the same issue ?
I'll be glad to discuss this problem with you, feel free t
(
https://issues.apache.org/jira/browse/SPARK-19451 ) when working with Long
values !!! So I can't use this
So my question is ( of course ) how can I resolve my problem ?
If I use spark streaming I will face the same issue ?
I'll be glad to discuss this problem with you, feel free to an
