Hi Timo, Thanks for putting together the proposal! I really love the idea to combining solution for historic and recent data and left some suggestions on that part.
Regarding the table type, e.g. for kafka streams, I agree with @hequn's idea that it should be pretty much inferable from the SQL context. I think there might be some questions need to be addressed when unifying the definition, for example: - Should a Kafka table used in "INSERT INTO" statement be used again in "FROM" statement, and vise versa ? - How to enforce checks in combo-table use cases ? - Can user change the way a table is used (e.g. source/sink) in interactive env such as sql-client ? Thanks, Rong On Thu, Oct 4, 2018 at 7:31 AM Hequn Cheng <chenghe...@gmail.com> wrote: > Hi, > > Thanks a lot for the proposal. I like the idea to unify table definitions. > I think we can drop the table type since the type can be derived from the > sql, i.e, a table be inserted can only be a sink table. > > I left some minor suggestions in the document, mainly include: > - Maybe we also need to allow define properties for tables. > - Support specify Computed Columns in a table > - Support define keys for sources. > > Best, Hequn > > > On Thu, Oct 4, 2018 at 4:09 PM Shuyi Chen <suez1...@gmail.com> wrote: > > > Thanks a lot for the proposal, Timo. I left a few comments. Also, it > seems > > the example in the doc does not have the table type (source, sink and > both) > > property anymore. Are you suggesting drop it? I think the table type > > properties is still useful as it can restrict a certain connector to be > > only source/sink, for example, we usually want a Kafka topic to be either > > read-only or write-only, but not both. > > > > Shuyi > > > > On Mon, Oct 1, 2018 at 1:53 AM Timo Walther <twal...@apache.org> wrote: > > > > > Hi everyone, > > > > > > as some of you might have noticed, in the last two releases we aimed to > > > unify SQL connectors and make them more modular. The first connectors > > > and formats have been implemented and are usable via the SQL Client and > > > Java/Scala/SQL APIs. > > > > > > However, after writing more connectors/example programs and talking to > > > users, there are still a couple of improvements that should be applied > > > to unified SQL connector API. > > > > > > I wrote a design document [1] that discusses limitations that I have > > > observed and consideres feedback that I have collected over the last > > > months. I don't know whether we will implement all of these > > > improvements, but it would be great to get feedback for a satisfactory > > > API and for future priorization. > > > > > > The general goal should be to connect to external systems as convenient > > > and type-safe as possible. Any feedback is highly appreciated. > > > > > > Thanks, > > > > > > Timo > > > > > > [1] > > > > > > > > > https://docs.google.com/document/d/1Yaxp1UJUFW-peGLt8EIidwKIZEWrrA-pznWLuvaH39Y/edit?usp=sharing > > > > > > > > > > -- > > "So you have to trust that the dots will somehow connect in your future." > > >
