Hi everyone,
thanks for the feedback that we got so far. I will update the document
in the next couple of hours such that we can continue with the discussion.
Regarding the table type: Actually I just didn't mention it in the
document, because the table type is a SQL Client/External catalog
interface specific property that is evaluated before the unified
connector API (depending on the table type a source and/or sink is
discovered). I agree with Shuyi's comments that it should be possible to
restrict read/write access. The general goal should be that properties
defined in the design document apply to both sources and sinks, i.e., no
special source-only or sink-only properties.
@Rong: Currently, a user can not change the way how a table is used in
the interactive shell. Tables defined in an environment file are
immutable. This will be possible using a SQL DDL in the future.
Regards,
Timo
Am 05.10.18 um 07:30 schrieb Shuyi Chen:
In the case of normal Flink job, I agree we can infer the table type from
the queries. However, for SQL client, the query is adhoc and not known
beforehand. In such case, we might want to enforce the table open mode at
startup time, so users won't accidentally write to a Kafka topic that is
supposed to be written only by some producer. What do you guys think?
Shuyi
On Thu, Oct 4, 2018 at 7:31 AM Hequn Cheng <chenghe...@gmail.com> wrote:
Hi,
Thanks a lot for the proposal. I like the idea to unify table definitions.
I think we can drop the table type since the type can be derived from the
sql, i.e, a table be inserted can only be a sink table.
I left some minor suggestions in the document, mainly include:
- Maybe we also need to allow define properties for tables.
- Support specify Computed Columns in a table
- Support define keys for sources.
Best, Hequn
On Thu, Oct 4, 2018 at 4:09 PM Shuyi Chen <suez1...@gmail.com> wrote:
Thanks a lot for the proposal, Timo. I left a few comments. Also, it
seems
the example in the doc does not have the table type (source, sink and
both)
property anymore. Are you suggesting drop it? I think the table type
properties is still useful as it can restrict a certain connector to be
only source/sink, for example, we usually want a Kafka topic to be either
read-only or write-only, but not both.
Shuyi
On Mon, Oct 1, 2018 at 1:53 AM Timo Walther <twal...@apache.org> wrote:
Hi everyone,
as some of you might have noticed, in the last two releases we aimed to
unify SQL connectors and make them more modular. The first connectors
and formats have been implemented and are usable via the SQL Client and
Java/Scala/SQL APIs.
However, after writing more connectors/example programs and talking to
users, there are still a couple of improvements that should be applied
to unified SQL connector API.
I wrote a design document [1] that discusses limitations that I have
observed and consideres feedback that I have collected over the last
months. I don't know whether we will implement all of these
improvements, but it would be great to get feedback for a satisfactory
API and for future priorization.
The general goal should be to connect to external systems as convenient
and type-safe as possible. Any feedback is highly appreciated.
Thanks,
Timo
[1]
https://docs.google.com/document/d/1Yaxp1UJUFW-peGLt8EIidwKIZEWrrA-pznWLuvaH39Y/edit?usp=sharing
--
"So you have to trust that the dots will somehow connect in your future."
