Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

Sorry, I thought that StreamTableSourceScanRule always fires but it is only used in StreamTableEnvironment so everything is fine. I think you can open PR if you like. Timo Am 30/08/16 um 05:25 schrieb Jark Wu: It seems that we have done that (?). The BatchTableEnvironment.registerTableSourc

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

It seems that we have done that (?). The BatchTableEnvironment.registerTableSource(name, tableSource) only accept a BatchTableSource. In contrast, the StreamTableEnvironment.registerTableSource(name, tableSource) only accept a StreamTableSource. So that, if a TableSource implements both batch

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

At first glance, I thought we are losing the possibility to distingish between choosing a batch or streaming table if a TableSource implements both. Because currently you are using a StreamTableSource as default if a TableSource implements both types. I think it would be better to determine bat

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

Hi Timo, Yes, you are right. The STREAM keyword is invalid now. If there is a STREAM keyword in the query, the parser will throw "can’t convert table xxx to stream" exception. Because we register the table as a regular table not streamable. - Jark Wu > 在 2016年8月29日，下午8:13，Timo Walther 写道： >

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

Hi Jark, your code looks good and it also simplifies many parts. So the STREAM keyword is not optional but invalid now, right? What happens if there is keyword in the query? Timo Am 29/08/16 um 05:40 schrieb Jark Wu: Hi Fabian, Timo, I have created a prototype for removing STREAM keyword

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

Hi Fabian, Timo, I have created a prototype for removing STREAM keyword and using batch sql parser for stream jobs. This is the working brach: https://github.com/wuchong/flink/tree/remove-stream Looking forward to your feedback. - Jark

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

Starting with a prototype would be great, Jark. We had some trouble with Calcite's StreamableTable interface anyways. A few things can be simplified if we do not declare our tables as streamable. I would try to implement DataStreamTable (and all related classes and methods) equivalent to DataSetTab

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

Hi Fabian, You are right, the main thing we need to change for removing STREAM keyword is the table registration. If you would like, I can do a prototype. Hi Timo, I’m glad to contribute our work back to Flink. I will look into it and create JIRAs next days. - Jark Wu > 在 2016年8月24日，上午12:

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

Hi Jark, We can think about removing the STREAM keyword or not. In principle, Calcite should allow the same windowing syntax on streaming and static tables (this is one of the main goals of Calcite). The Table API can also distinguish stream and batch without the STREAM keyword by looking at the E

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

Hi Jark, your design document looks very promising. It would be great if you could contribute parts of your implementation back. E.g. UDTFs, UDAFs or even your CROSS APPLY operator. We don't need a FLIP for every little new feature. So feel free to create a Jira issue, discuss it a little bit

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

Hi Fabian, Timo, Sorry for the late response. Regarding Calcite’s StreamSQL syntax, what I concern is only the STREAM keyword and no agg-without-window. Which makes different syntax for streaming and static tables. I don’t think Flink should have a custom SQL syntax, but it’s better to have a

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

On Tue, Aug 23, 2016 at 9:47 AM, Fabian Hueske wrote: > I drafted an API proposal a few months ago [2] and could convert this into > a FLIP to discuss the API and break it down into subtasks. > > What do you think? Sounds very reasonable :) +1

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

Hi, I did a bit of prototyping yesterday to check to what extend Calcite supports window operations on streams if we would implement them for the Table API. For the Table API we do not go through Calcite's SQL parser and validator, but generate the logical plan (tree of RelNodes) ourselves mostly

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

Hi Jark, thanks for starting this discussion. Actually, I think we are rather "blocked" on the internal handling of streaming windows in Calcite than the SQL parser. IMO, it should be possible to exchange or modify the parser if we want that. Regarding Calcite's StreamSQL syntax: Except for the S

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

Hi Jark, sorry that I didn't wrote back earlier. I wanted to talk to Fabian first about this. In general, according to Calcite's plans, even SQL queries containing the "STREAM" keyword are regular standard SQL. In theory we could omit the "STREAM" keyword as long as it is guaranteed that the

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

Hi, I personally would like it a lot if the SQL queries for batch and stream programs looked the same. With the decision to move the Table API on top of Calcite and also use the Calcite SQL parser Flink is somewhat tied to Calcite so I don't know whether we can add our own window constructs and tea

[DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

Hi, Currently, Flink use Calcite for SQL parsing. So we use the StreamSQL grammer proposed by Calcite[1] which we have to use the `STREAM` keyword in SQL. For example, `SELECT * FROM Orders` is a regular standard SQL and will be translated to a batch job. If you want to statement a stream job,