Re: [PROPOSAL]: a new feature branch for SQL DSL

Thanks guys for your interest and discussion. A quick update that, 1. the feature branch is ready [1], 2. a new component created in JIRA as well [2], 3. a list of tasks in my sight [3], Any contribute, feedback are very welcome !! [1]. https://github.com/apache/beam/tree/DSL_SQL [2]. https:/

Re: [PROPOSAL]: a new feature branch for SQL DSL

It's more about how State API can be introduced in SQL, the snapshot of state converts stream to a table which is very helpful. SQL keyword INSERT INTO may be an option to do that but I've no confidence so far. On Fri, Apr 14, 2017 at 3:03 PM, Tyler Akidau wrote: > Tarush: I don't think it depe

Re: [PROPOSAL]: a new feature branch for SQL DSL

Tarush: I don't think it depends upon the time frame (although you may be interested in only a specific timeframe materialized within the table). Stream to table conversion is purely a byproduct of grouping a stream. I have a doc I'm getting some initial reviews on currently that I hope to send out

Re: [PROPOSAL]: a new feature branch for SQL DSL

Expose streaming snapshot via STATE is attractive in Beam model, but doubt it's the right way in SQL. IMO,there's 'INSERT INTO' to persistent streaming output. On Wed, Apr 12, 2017 at 8:37 PM, tarush grover wrote: > Hi Tyler, > > Transforming stream into a table will also depend on the time fra

Re: [PROPOSAL]: a new feature branch for SQL DSL

Hi Tyler, Transforming stream into a table will also depend on the time frame in the stream or what windows we choose for the stream. Regards, Tarush On Tue, 11 Apr 2017 at 11:29 PM, Tyler Akidau wrote: > Hi 陈竞, > > I'm doubtful there will be an explicit equivalent of the State API in SQL, >

Re: [PROPOSAL]: a new feature branch for SQL DSL

Hi 陈竞, I'm doubtful there will be an explicit equivalent of the State API in SQL, at least not in the SQL portion of the DSL itself (it might make sense to expose one within UDFs). The State API is an imperative interface for accessing an underlying persistent state table, whereas SQL operates mor

Re: [PROPOSAL]: a new feature branch for SQL DSL

Gonna merge tonight. Regards JB On 04/11/2017 07:09 PM, Mingmin Xu wrote: It's not, you can use the feature branch https://github.com/apache/beam/tree/DSL_SQL after https://github.com/apache/beam/pull/2479 is merged, stay tuned. On Tue, Apr 11, 2017 at 10:00 AM, tarush grover wrote: Wanted

Re: [PROPOSAL]: a new feature branch for SQL DSL

Thanks Mingmin. Regards, Tarush On Tue, 11 Apr 2017 at 10:39 PM, Mingmin Xu wrote: > It's not, you can use the feature branch > https://github.com/apache/beam/tree/DSL_SQL after > https://github.com/apache/beam/pull/2479 is merged, stay tuned. > > On Tue, Apr 11, 2017 at 10:00 AM, tarush grover

Re: [PROPOSAL]: a new feature branch for SQL DSL

It's not, you can use the feature branch https://github.com/apache/beam/tree/DSL_SQL after https://github.com/apache/beam/pull/2479 is merged, stay tuned. On Tue, Apr 11, 2017 at 10:00 AM, tarush grover wrote: > Wanted to know active branch for the beam SQL feature, is it beam - 301 > where the

Re: [PROPOSAL]: a new feature branch for SQL DSL

Wanted to know active branch for the beam SQL feature, is it beam - 301 where the development for this feature would happen? Regards, Tarush On Tue, 11 Apr 2017 at 10:49 AM, 陈竞 wrote: > i just want to know what the SQL State API equivalent is for SQL, since > beam has already support stateful p

Re: [PROPOSAL]: a new feature branch for SQL DSL

i just want to know what the SQL State API equivalent is for SQL, since beam has already support stateful processing using state DoFn 2017-04-11 2:12 GMT+08:00 Tyler Akidau : > 陈竞, what are you specifically curious about regarding state? Are you > wanting to know what the SQL State API equivalent

Re: [PROPOSAL]: a new feature branch for SQL DSL

陈竞, what are you specifically curious about regarding state? Are you wanting to know what the SQL State API equivalent is for SQL? Or are you asking an operational question about where the state for a given SQL pipeline will live? -Tyler On Sun, Apr 9, 2017 at 12:39 PM Mingmin Xu wrote: > Than

Re: [PROPOSAL]: a new feature branch for SQL DSL

Thanks @JB, will come out the initial PR soon. On Sun, Apr 9, 2017 at 12:28 PM, Jean-Baptiste Onofré wrote: > As discussed, I created the DSL_SQL branch with the skeleton. Mingmin is > rebasing on this branch to submit the PR. > > Regards > JB > > > On 04/09/2017 08:02 PM, Mingmin Xu wrote: > >>

Re: [PROPOSAL]: a new feature branch for SQL DSL

As discussed, I created the DSL_SQL branch with the skeleton. Mingmin is rebasing on this branch to submit the PR. Regards JB On 04/09/2017 08:02 PM, Mingmin Xu wrote: State is not touched yet, welcome to add it. On Sun, Apr 9, 2017 at 2:40 AM, 陈竞 wrote: how will this sql support state bot

Re: [PROPOSAL]: a new feature branch for SQL DSL

State is not touched yet, welcome to add it. On Sun, Apr 9, 2017 at 2:40 AM, 陈竞 wrote: > how will this sql support state both in streaming and batch mode > > 2017-04-07 4:54 GMT+08:00 Mingmin Xu : > > > @Tyler, there's no big change in the previous design doc, I added some > > details in chapter

Re: [PROPOSAL]: a new feature branch for SQL DSL

how will this sql support state both in streaming and batch mode 2017-04-07 4:54 GMT+08:00 Mingmin Xu : > @Tyler, there's no big change in the previous design doc, I added some > details in chapter 'Part 2. DML( [INSERT] SELECT )' , describing steps to > process a query, feel free to leave a comm

Re: [PROPOSAL]: a new feature branch for SQL DSL

@Tyler, there's no big change in the previous design doc, I added some details in chapter 'Part 2. DML( [INSERT] SELECT )' , describing steps to process a query, feel free to leave a comment. Come through your doc of 'EMIT', it's awesome from my perspective. I've some tests on GroupBy with default

Re: [PROPOSAL]: a new feature branch for SQL DSL

I'm very excited by this development as well, thanks for continuing to push this forward, Mingmin. :-) I noticed you'd made some changes to your design doc . Is it ready for another review? How reflective is it c

Re: [PROPOSAL]: a new feature branch for SQL DSL

Very cool! I'm really excited about this integration. On Thu, Apr 6, 2017 at 9:39 AM, Jean-Baptiste Onofré wrote: > Hi, > > Mingmin and I prepared a new branch to have the SQL DSL in dsls/sql > location. > > Any help is welcome ! > > Thanks, > Regards > JB > > > On 04/06/2017 06:36 PM, Mingmin X

Re: [PROPOSAL]: a new feature branch for SQL DSL

Hi, Mingmin and I prepared a new branch to have the SQL DSL in dsls/sql location. Any help is welcome ! Thanks, Regards JB On 04/06/2017 06:36 PM, Mingmin Xu wrote: @Tarush, you're very welcome to join the effort. On Thu, Apr 6, 2017 at 7:22 AM, tarush grover wrote: Hi, Can I be also par

Re: [PROPOSAL]: a new feature branch for SQL DSL

@Tarush, you're very welcome to join the effort. On Thu, Apr 6, 2017 at 7:22 AM, tarush grover wrote: > Hi, > > Can I be also part of this feature development. > > Regards, > Tarush Grover > > On Thu, Apr 6, 2017 at 3:17 AM, Ted Yu wrote: > > > I compiled BEAM-301 branch with calcite 1.12 - pas

Re: [PROPOSAL]: a new feature branch for SQL DSL

Hi, Can I be also part of this feature development. Regards, Tarush Grover On Thu, Apr 6, 2017 at 3:17 AM, Ted Yu wrote: > I compiled BEAM-301 branch with calcite 1.12 - passed. > > Julian tries to not break existing things, but he will if there's a reason > to do so :-) > > On Wed, Apr 5, 201

Re: [PROPOSAL]: a new feature branch for SQL DSL

I compiled BEAM-301 branch with calcite 1.12 - passed. Julian tries to not break existing things, but he will if there's a reason to do so :-) On Wed, Apr 5, 2017 at 2:36 PM, Mingmin Xu wrote: > @Ted, thanks for the note. I intend to stick with one version, Beam 0.6.0 > and Calcite 1.11 so far,

Re: [PROPOSAL]: a new feature branch for SQL DSL

@Ted, thanks for the note. I intend to stick with one version, Beam 0.6.0 and Calcite 1.11 so far, unless impacted by API change. Before it's merged back to master, will upgrade to the latest version. On Wed, Apr 5, 2017 at 2:14 PM, Ted Yu wrote: > Working in feature branch is good - you may wan

Re: [PROPOSAL]: a new feature branch for SQL DSL

Working in feature branch is good - you may want to periodically sync up with master. I noticed that you are using 1.11.0 of calcite. 1.12 is out, FYI On Wed, Apr 5, 2017 at 2:05 PM, Mingmin Xu wrote: > Hi all, > > I'm working on https://issues.apache.org/jira/browse/BEAM-301(Add a Beam > SQL D

Re: [PROPOSAL]: a new feature branch for SQL DSL

That will be awesome! On Wed, Apr 5, 2017, 2:05 PM Mingmin Xu wrote: > Hi all, > > I'm working on https://issues.apache.org/jira/browse/BEAM-301(Add a Beam > SQL DSL). The skeleton is already in > https://github.com/XuMingmin/beam/tree/BEAM-301, using Java SDK in the > back-end. The goal is to p

[PROPOSAL]: a new feature branch for SQL DSL

Hi all, I'm working on https://issues.apache.org/jira/browse/BEAM-301(Add a Beam SQL DSL). The skeleton is already in https://github.com/XuMingmin/beam/tree/BEAM-301, using Java SDK in the back-end. The goal is to provide a SQL interface over Beam, based on Calcite, including: 1). a translator to