GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/3703
[SPARK-1442][SQL][WIP] Initial window function implementation (refactored
from #2953)
This WIP PR is refactored from PR #2953. Please refer to the original PR
description for features implemented and not implemented in this PR.
The original PR was a huge one, commenting on each issue could be very time
consuming. After offline discussions with @guowei2, I decided to work on a
refactoring branch to fix most minor issues first and then start discussion
based on this refactored version.
Major issues left in this PR are:
1. Window spec is added to aggregation functions with a `var`, which breaks
query plan immutability.
2. When used with window specs, common aggregation functions like `COUNT`,
`SUM`, `AVG` etc are not translated into Hive aggregation functions rather than
Spark SQL builtin implementations.
3. Execution code (`execution.WindowFunction`) can be further simplified.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark window-refact
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3703.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3703
----
commit 9897413564ff27f0a311cc2cef6322422f3807ab
Author: guowei2 <[email protected]>
Date: 2014-10-24T09:55:47Z
window function
commit 7d7a703d5e7bf37e00d074cb8c04e2150f8fbeb4
Author: guowei2 <[email protected]>
Date: 2014-10-27T05:29:35Z
window function
commit 1999e07a23c18808738e4e3b14b64c1db108eda2
Author: guowei2 <[email protected]>
Date: 2014-10-27T06:03:17Z
window function
commit 76bfd4b8b1137426b0dbcde5d56cefb0c98cfab5
Author: guowei2 <[email protected]>
Date: 2014-10-27T14:16:22Z
window function
commit 88c5789d9f6989d0fedcbdd129de097152e2d8eb
Author: guowei2 <[email protected]>
Date: 2014-10-28T04:00:57Z
window function
commit 828199a48c619d03b4ec524dbdfe9c043baa5e14
Author: guowei2 <[email protected]>
Date: 2014-10-29T07:49:01Z
fix problems after rebase
commit 03bd77d5533f76484d7589e0296283b58f2d0688
Author: guowei2 <[email protected]>
Date: 2014-10-30T10:12:42Z
change test suite and golden files
commit d06baeba2dc859f860c8fd43c292275837b3e0e6
Author: guowei2 <[email protected]>
Date: 2014-11-05T03:01:33Z
add constant objectinspector support for udafs, such as last_value(col,
false)
commit 173016c08770fd2aa6ee15c3f194c2282bd46e68
Author: guowei2 <[email protected]>
Date: 2014-11-26T06:58:47Z
fix window function to support multi-different window partitions
commit ab21933e64b3ee7afdcbb622bec935a34fe0785c
Author: guowei2 <[email protected]>
Date: 2014-11-27T08:40:26Z
fix DecimalType bug after rebase
commit 66ef7a6d449f6ec1e644d2e73118d8be1cb56cde
Author: guowei2 <[email protected]>
Date: 2014-11-28T09:51:16Z
fix bug about attribute reference
commit dc87d8d08c33644e61f6355ed07baf720b0e9ef9
Author: Cheng Lian <[email protected]>
Date: 2014-12-04T06:34:32Z
WIP: refactoring window functions support
commit 2da61753590fe00ecf46219f387d70d48c6dd32a
Author: Cheng Lian <[email protected]>
Date: 2014-12-15T06:42:57Z
Removed trailing spaces from query string in HiveWindowFunctionSuite
commit 922a8b9bfe0278577378c3cd9fc13cb9998b6e0f
Author: Cheng Lian <[email protected]>
Date: 2014-12-15T17:18:42Z
Fixes COUNT with window spec
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
