[
https://issues.apache.org/jira/browse/FLINK-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995971#comment-15995971
]
ASF GitHub Bot commented on FLINK-6428:
---------------------------------------
GitHub user sunjincheng121 opened a pull request:
https://github.com/apache/flink/pull/3817
[FLINK-6428][table] Add support DISTINCT in dataStream SQL
With the changes of this PR. `DISTINCT` keyword can be supported `SELECT
Clause` of dataStream SQL.
In standard database there are two situations can using `DISTINCT` keyword.
* in `SELECT Clause`, e.g.: `SELECT DISTINCT name FROM table`
* in `AGG Clause`, e.g.: `COUNT([ALL|DISTINCT]
expression)`,`SUM([ALL|DISTINCT] expression)`, etc.
In this JIRA. we talk about `SELECT Clause`. With the growing elements, the
limitations tend to be back-end storage(flink state). In theory, external
storage is infinitely large (user can control and expect), this point of view,
the infinite STREAM of the DISTINCT can be supported.In addition, external
storage, for example: RocksDB, the user can set the TTL according to the actual
amount of business data to ensure that external storage is working properly.
- [x] General
- The pull request references the related JIRA issue
("[FLINK-6428][table] Add support DISTINCT in dataStream SQL")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sunjincheng121/flink FLINK-6428-PR
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3817.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3817
----
commit f1aaffd765f05f8b51801ae1bb66426d99480970
Author: sunjincheng121 <sunjincheng...@gmail.com>
Date: 2017-05-03T03:47:09Z
[FLINK-6428][table] Add support DISTINCT in dataStream SQL
----
> Add support DISTINCT in dataStream SQL
> --------------------------------------
>
> Key: FLINK-6428
> URL: https://issues.apache.org/jira/browse/FLINK-6428
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
> Reporter: sunjincheng
> Assignee: sunjincheng
>
> Add support DISTINCT in dataStream SQL as follow:
> DATA:
> {code}
> (name, age)
> (kevin, 28),
> (sunny, 6),
> (jack, 6)
> {code}
> SQL:
> {code}
> SELECT DISTINCT age FROM MyTable"
> {code}
> RESULTS:
> {code}
> 28, 6
> {code}
> To DataStream:
> {code}
> inputDS
> .keyBy() // KeyBy on all fields
> .flatMap() // Eliminate duplicate data
> {code}
> [~fhueske] do we need this feature?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
