Hi Danny, Regarding to `supportedHintOptions()` interface, I suggest to use the inverted version, `unsupportedHintOptions()`. Because I think the disallowed list is much smaller. In addition, it's hard to list all the properties under `connector.properties.*`. But we know `connector.properties.bootstrap.servers` and `connector.properties.zookeeper.connect` are the only security options.
Best, Jark On Thu, 26 Mar 2020 at 16:47, Kurt Young <ykt...@gmail.com> wrote: > Hi Danny, > > Thanks for the updates. I have 2 comments regarding to latest document: > > 1) I think we also need `*supportedHintOptions*` for > `*TableFormatFactory*` > 2) IMO "dynamic-table-options.enabled" should belong to ` > *OptimizerConfigOptions*` > > Best, > Kurt > > > On Thu, Mar 26, 2020 at 4:40 PM Timo Walther <twal...@apache.org> wrote: > > > Thanks for the update Danny. +1 for this proposal. > > > > Regards, > > Timo > > > > On 26.03.20 04:51, Danny Chan wrote: > > > Thanks everyone who engaged in this discussion ~ > > > > > > Our goal is "Supports Dynamic Table Options for Flink SQL". After an > > > offline discussion with Kurt, Timo and Dawid, we have made the final > > > conclusion, here is the summary: > > > > > > > > > - Use comment style syntax to specify the dynamic table options: > "/*+ > > > *OPTIONS*(k1='v1', k2='v2') */" > > > - Have constraint on the options keys: the options that may bring > in > > > security problems should not be allowed, i.e. Kafka connector > > zookeeper > > > endpoint URL and topic name > > > - Use white-list to control the allowed options for each connector, > > > which is more safe for future extention > > > - We allow to enable/disable this feature globally > > > - Implement based on the current code base first, and when FLIP-95 > is > > > checked in, implement this feature based on new interface > > > > > > Any suggestions are appreciated ~ > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL > > > > > > Best, > > > Danny Chan > > > > > > Jark Wu <imj...@gmail.com> 于2020年3月18日周三 下午10:38写道： > > > > > >> Hi everyone, > > >> > > >> Sorry, but I'm not sure about the `supportedHintOptions`. I'm afraid > it > > >> doesn't solve the problems but increases some development and learning > > >> burdens. > > >> > > >> # increase development and learning burden > > >> > > >> According to the discussion so far, we want to support overriding a > > subset > > >> of options in hints which doesn't affect semantics. > > >> With the `supportedHintOptions`, it's up to the connector developers > to > > >> decide which options will not affect semantics, and to be hint > options. > > >> However, the question is how to distinguish whether an option will > > *affect > > >> semantics*? What happens if an option will affect semantics but > > provided as > > >> hint options? > > >> From my point of view, it's not easy to distinguish. For example, the > > >> "format.ignore-parse-error" can be a very useful dynamic option but > that > > >> will affect semantic, because the result is different (null vs > > exception). > > >> Another example, the "connector.lookup.cache.*" options are also very > > >> useful to tune jobs, however, it will also affect the job results. I > can > > >> come up many more useful options but may affect semantics. > > >> > > >> I can see that the community will under endless discussion around "can > > this > > >> option to be a hint option?", "wether this option will affect > > semantics?". > > >> You can also find that we already have different opinions on > > >> "ignore-parse-error". Those discussion is a waste of time! That's not > > what > > >> users want! > > >> The problem is user need this, this, this options and HOW to expose > > them? > > >> We should focus on that. > > >> > > >> Then there could be two endings in the future: > > >> 1) compromise on the usability, we drop the rule that hints don't > affect > > >> semantics, allow all the useful options in the hints list. > > >> 2) stick on the rule, users will find this is a stumbling feature > which > > >> doesn't solve their problems. > > >> And they will be surprised why this option can't be set, but the > > other > > >> could. *semantic* is hard to be understood by users. > > >> > > >> # doesn't solve the problems > > >> > > >> I think the purpose of this FLIP is to allow users to quickly override > > some > > >> connectors' properties to tune their jobs. > > >> However, `supportedHintOptions` is off track. It only allows a subset > > >> options and for the users it's not *clear* which subset is allowed. > > >> > > >> Besides, I'm not sure `supportedHintOptions` can work well for all > > cases. > > >> How could you support kafka properties (`connector.properties.*`) as > > hint > > >> options? Some kafka properties may affect semantics > (bootstrap.servers), > > >> some may not (max.poll.records). Besides, I think it's not possible to > > list > > >> all the possible kafka properties [1]. > > >> > > >> In summary, IMO, `supportedHintOptions` > > >> (1) it increase the complexity to develop a connector > > >> (2) it confuses users which options can be used in hint, which are > not, > > >> they have to check the docs again and again. > > >> (3) it doesn't solve the problems which we want to solve by this FLIP. > > >> > > >> I think we should avoid introducing some partial solutions. Otherwise, > > we > > >> will be stuck in a loop that introduce new API -> deprecate API -> > > >> introduce new API.... > > >> > > >> I personally in favor of an explicit WITH syntax after the table as a > > part > > >> of the query which is mentioned by Kurt before, e.g. SELECT * from T > > >> WITH('key' = 'value') . > > >> It allows users to dynamically set options which can affect semantics. > > It > > >> will be very flexible to solve users' problems so far. > > >> > > >> Best, > > >> Jark > > >> > > >> [1]: https://kafka.apache.org/documentation/#consumerconfigs > > >> > > >> On Wed, 18 Mar 2020 at 21:44, Danny Chan <yuzhao....@gmail.com> > wrote: > > >> > > >>> My POC is here for the hints options merge [1]. > > >>> > > >>> Personally, I have no strong objections for splitting hints with the > > >>> CatalogTable, the only cons is a more complex implementation but the > > >>> concept is more clear, and I have updated the WIKI. > > >>> > > >>> I think it would be nice if we can support the format “ignore-parse > > >> error” > > >>> option key, the CSV source already has a key [2] and we can use that > in > > >> the > > >>> supportedHIntOptions, for the common CSV and JSON formats, we cal > also > > >> give > > >>> a support. This is the only kind of key in formats that “do not > change > > >> the > > >>> semantics” (somehow), what do you think about this ~ > > >>> > > >>> [1] > > >>> > > >> > > > https://github.com/danny0405/flink/commit/5d925fa16c3c553423c4b7d93001521b8e6e6bee#diff-6e569a6dd124fd2091c18e2790fb49c5 > > >>> [2] > > >>> > > >> > > > https://github.com/apache/flink/blob/b83060dff6d403b6994b6646b3f29a374f599530/flink-table/flink-table-api-java-bridge/src/main/java/org/apache/flink/table/sources/CsvTableSourceFactoryBase.java#L92 > > >>> > > >>> Best, > > >>> Danny Chan > > >>> 在 2020年3月18日 +0800 PM9:10，Timo Walther <twal...@apache.org>，写道： > > >>>> Hi everyone, > > >>>> > > >>>> +1 to Kurt's suggestion. Let's just have it in source and sink > > >> factories > > >>>> for now. We can still move this method up in the future. Currently, > I > > >>>> don't see a need for catalogs or formats. Because how would you > target > > >> a > > >>>> format in the query? > > >>>> > > >>>> @Danny: Can you send a link to your PoC? I'm very skeptical about > > >>>> creating a new CatalogTable in planner. Actually CatalogTable should > > be > > >>>> immutable between Catalog and Factory. Because a catalog can return > > its > > >>>> own factory and fully control the instantiation. Depending on the > > >>>> implementation, that means it can be possible that the catalog has > > >>>> encoded more information in a concrete subclass implementing the > > >>>> interface. I vote for separating the concerns of catalog information > > >> and > > >>>> hints in the factory explicitly. > > >>>> > > >>>> Regards, > > >>>> Timo > > >>>> > > >>>> > > >>>> On 18.03.20 05:41, Jingsong Li wrote: > > >>>>> Hi, > > >>>>> > > >>>>> I am thinking we can provide hints to *table* related instances. > > >>>>> - TableFormatFactory: of cause we need hints support, there are > many > > >>> format > > >>>>> options in DDL too. > > >>>>> - catalog and module: I don't know, maybe in future we can provide > > >> some > > >>>>> hints for them. > > >>>>> > > >>>>> Best, > > >>>>> Jingsong Lee > > >>>>> > > >>>>> On Wed, Mar 18, 2020 at 12:28 PM Danny Chan <yuzhao....@gmail.com> > > >>> wrote: > > >>>>> > > >>>>>> Yes, I think we should move the `supportedHintOptions` from > > >>> TableFactory > > >>>>>> to TableSourceFactory, and we also need to add the interface to > > >>>>>> TableSinkFactory though because sink target table may also have > > >> hints > > >>>>>> attached. > > >>>>>> > > >>>>>> Best, > > >>>>>> Danny Chan > > >>>>>> 在 2020年3月18日 +0800 AM11:08，Kurt Young <ykt...@gmail.com>，写道： > > >>>>>>> Have one question for adding `supportedHintOptions` method to > > >>>>>>> `TableFactory`. It seems > > >>>>>>> `TableFactory` is a base factory interface for all *table module* > > >>> related > > >>>>>>> instances, such as > > >>>>>>> catalog, module, format and so on. It's not created only for > > >>> *table*. Is > > >>>>>> it > > >>>>>>> possible to move it > > >>>>>>> to `TableSourceFactory`? > > >>>>>>> > > >>>>>>> Best, > > >>>>>>> Kurt > > >>>>>>> > > >>>>>>> > > >>>>>>> On Wed, Mar 18, 2020 at 10:59 AM Danny Chan < > > >> yuzhao....@gmail.com> > > >>>>>> wrote: > > >>>>>>> > > >>>>>>>> Thanks Timo ~ > > >>>>>>>> > > >>>>>>>> For the naming itself, I also think the PROPERTIES is not that > > >>>>>> concise, so > > >>>>>>>> +1 for OPTIONS (I had thought about that, but there are many > > >>> codes in > > >>>>>>>> current Flink called it properties, i.e. the > > >>> DescriptorProperties, > > >>>>>>>> #getSupportedProperties), let’s use OPTIONS if this is our new > > >>>>>> preference. > > >>>>>>>> > > >>>>>>>> +1 to `Set<ConfigOption> supportedHintOptions()` because the > > >>>>>> ConfigOption > > >>>>>>>> can take more info. AFAIK, Spark also call their table options > > >>> instead > > >>>>>> of > > >>>>>>>> properties. [1] > > >>>>>>>> > > >>>>>>>> In my local POC, I did create a new CatalogTable, and it works > > >>> for > > >>>>>> current > > >>>>>>>> connectors well, all the DDL tables would finally yield a > > >>> CatalogTable > > >>>>>>>> instance and we can apply the options to that(in the > > >>> CatalogSourceTable > > >>>>>>>> when we generating the TableSource), the pros is that we do not > > >>> need to > > >>>>>>>> modify the codes of connectors itself. If we split the options > > >>> from > > >>>>>>>> CatalogTable, we may need to add some additional logic in each > > >>>>>> connector > > >>>>>>>> factories in order to merge these properties (and the logic are > > >>> almost > > >>>>>> the > > >>>>>>>> same), what do you think about this? > > >>>>>>>> > > >>>>>>>> [1] > > >>>>>>>> > > >>>>>> > > >>> > > >> > > > https://docs.databricks.com/spark/latest/spark-sql/language-manual/create-table.html > > >>>>>>>> > > >>>>>>>> Best, > > >>>>>>>> Danny Chan > > >>>>>>>> 在 2020年3月17日 +0800 PM10:10，Timo Walther <twal...@apache.org > > >>> ，写道： > > >>>>>>>>> Hi Danny, > > >>>>>>>>> > > >>>>>>>>> thanks for updating the FLIP. I think your current design is > > >>>>>> sufficient > > >>>>>>>>> to separate hints from result-related properties. > > >>>>>>>>> > > >>>>>>>>> One remark to the naming itself: I would vote for calling the > > >>> hints > > >>>>>>>>> around table scan `OPTIONS('k'='v')`. We used the term > > >>> "properties" > > >>>>>> in > > >>>>>>>>> the past but since we want to unify the Flink configuration > > >>>>>> experience, > > >>>>>>>>> we should use consistent naming and classes around > > >>> `ConfigOptions`. > > >>>>>>>>> > > >>>>>>>>> It would be nice to use `Set<ConfigOption> > > >>> supportedHintOptions();` > > >>>>>> to > > >>>>>>>>> start using config options instead of pure string properties. > > >>> This > > >>>>>> will > > >>>>>>>>> also allow us to generate documentation in the future around > > >>>>>> supported > > >>>>>>>>> data types, ranges, etc. for options. At some point we would > > >>> also > > >>>>>> like > > >>>>>>>>> to drop `DescriptorProperties` class. "Options" is also used > > >>> in the > > >>>>>>>>> documentation [1] and in the SQL/MED standard [2]. > > >>>>>>>>> > > >>>>>>>>> Furthermore, I would still vote for separating CatalogTable > > >>> and hint > > >>>>>>>>> options. Otherwise the planner would need to create a new > > >>>>>> CatalogTable > > >>>>>>>>> instance which might not always be easy. We should offer them > > >>> via: > > >>>>>>>>> > > >>>>>>>>> > > >>> org.apache.flink.table.factories.TableSourceFactory.Context#getHints: > > >>>>>>>>> ReadableConfig > > >>>>>>>>> > > >>>>>>>>> What do you think? > > >>>>>>>>> > > >>>>>>>>> Regards, > > >>>>>>>>> Timo > > >>>>>>>>> > > >>>>>>>>> [1] > > >>>>>>>>> > > >>>>>>>> > > >>>>>> > > >>> > > >> > > > https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/create.html#create-table > > >>>>>>>>> [2] https://wiki.postgresql.org/wiki/SQL/MED > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> On 12.03.20 15:06, Stephan Ewen wrote: > > >>>>>>>>>> @Danny sounds good. > > >>>>>>>>>> > > >>>>>>>>>> Maybe it is worth listing all the classes of problems that > > >>> you > > >>>>>> want to > > >>>>>>>>>> address and then look at each class and see if hints are a > > >>> good > > >>>>>> default > > >>>>>>>>>> solution or a good optional way of simplifying things? > > >>>>>>>>>> The discussion has grown a lot and it is starting to be > > >> hard > > >>> to > > >>>>>>>> distinguish > > >>>>>>>>>> the parts where everyone agrees from the parts were there > > >> are > > >>>>>> concerns. > > >>>>>>>>>> > > >>>>>>>>>> On Thu, Mar 12, 2020 at 2:31 PM Danny Chan < > > >>> danny0...@apache.org> > > >>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> Thanks Stephan ~ > > >>>>>>>>>>> > > >>>>>>>>>>> We can remove the support for properties that may change > > >>> the > > >>>>>>>> semantics of > > >>>>>>>>>>> query if you think that is a trouble. > > >>>>>>>>>>> > > >>>>>>>>>>> How about we support the /*+ properties() */ hint only > > >> for > > >>> those > > >>>>>>>> optimize > > >>>>>>>>>>> parameters, such as the fetch size of source or something > > >>> like > > >>>>>> that, > > >>>>>>>> does > > >>>>>>>>>>> that make sense? > > >>>>>>>>>>> > > >>>>>>>>>>> Stephan Ewen <se...@apache.org>于2020年3月12日 周四下午7:45写道： > > >>>>>>>>>>> > > >>>>>>>>>>>> I think Bowen has actually put it very well. > > >>>>>>>>>>>> > > >>>>>>>>>>>> (1) Hints that change semantics looks like trouble > > >>> waiting to > > >>>>>>>> happen. For > > >>>>>>>>>>>> example Kafka offset handling should be in filters. The > > >>> Kafka > > >>>>>>>> source > > >>>>>>>>>>> should > > >>>>>>>>>>>> support predicate pushdown. > > >>>>>>>>>>>> > > >>>>>>>>>>>> (2) Hints should not be a workaround for current > > >>> shortcomings. > > >>>>>> A > > >>>>>>>> lot of > > >>>>>>>>>>> the > > >>>>>>>>>>>> suggested above sounds exactly like that. Working > > >> around > > >>>>>>>> catalog/DDL > > >>>>>>>>>>>> shortcomings, missing exposure of metadata (offsets), > > >>> missing > > >>>>>>>> predicate > > >>>>>>>>>>>> pushdown in Kafka. Abusing a feature like hints now as > > >> a > > >>> quick > > >>>>>> fix > > >>>>>>>> for > > >>>>>>>>>>>> these issues, rather than fixing the root causes, will > > >>> much > > >>>>>> likely > > >>>>>>>> bite > > >>>>>>>>>>> us > > >>>>>>>>>>>> back badly in the future. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Best, > > >>>>>>>>>>>> Stephan > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:43 AM Kurt Young < > > >>> ykt...@gmail.com> > > >>>>>>>> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>>> It seems this FLIP's name is somewhat misleading. > > >> From > > >>> my > > >>>>>>>>>>> understanding, > > >>>>>>>>>>>>> this FLIP is trying to > > >>>>>>>>>>>>> address the dynamic parameter issue, and table hints > > >>> is the > > >>>>>> way > > >>>>>>>> we wan > > >>>>>>>>>>> to > > >>>>>>>>>>>>> choose. I think we should > > >>>>>>>>>>>>> be focus on "what's the right way to solve dynamic > > >>> property" > > >>>>>>>> instead of > > >>>>>>>>>>>>> discussing "whether table > > >>>>>>>>>>>>> hints can affect query semantics". > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> For now, there are two proposed ways to achieve > > >> dynamic > > >>>>>> property: > > >>>>>>>>>>>>> 1. FLIP-110: create temporary table xx like xx with > > >>> (xxx) > > >>>>>>>>>>>>> 2. use custom "from t with (xxx)" syntax > > >>>>>>>>>>>>> 3. "Borrow" the table hints to have a special > > >>> PROPERTIES > > >>>>>> hint. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> The first one didn't break anything, but the only > > >>> problem i > > >>>>>> see > > >>>>>>>> is a > > >>>>>>>>>>>> little > > >>>>>>>>>>>>> more verbose than the table hint > > >>>>>>>>>>>>> approach. I can imagine when someone using SQL CLI to > > >>> have a > > >>>>>> sql > > >>>>>>>>>>>>> experience, it's quite often that > > >>>>>>>>>>>>> he will modify the table property, some use cases i > > >> can > > >>>>>> think of: > > >>>>>>>>>>>>> 1. the source contains some corrupted data, i want to > > >>> turn > > >>>>>> on the > > >>>>>>>>>>>>> "ignore-error" flag for certain formats. > > >>>>>>>>>>>>> 2. I have a kafka table and want to see some sample > > >>> data > > >>>>>> from the > > >>>>>>>>>>>>> beginning, so i change the offset > > >>>>>>>>>>>>> to "earliest", and then I want to observe the latest > > >>> data > > >>>>>> which > > >>>>>>>> keeps > > >>>>>>>>>>>>> coming in. I would write another query > > >>>>>>>>>>>>> to select from the latest table. > > >>>>>>>>>>>>> 3. I want to my jdbc sink flush data more eagerly > > >> then > > >>> i can > > >>>>>>>> observe > > >>>>>>>>>>> the > > >>>>>>>>>>>>> data from database side. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Most of such use cases are quite ad-hoc. If every > > >> time > > >>> I > > >>>>>> want to > > >>>>>>>> have a > > >>>>>>>>>>>>> different experience, i need to create > > >>>>>>>>>>>>> a temporary table and then also modify my query, it > > >>> doesn't > > >>>>>> feel > > >>>>>>>>>>> smooth. > > >>>>>>>>>>>>> Embed such dynamic property into > > >>>>>>>>>>>>> query would have better user experience. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Both 2 & 3 can make this happen. The cons of #2 is > > >>> breaking > > >>>>>> SQL > > >>>>>>>>>>>> compliant, > > >>>>>>>>>>>>> and for #3, it only breaks some > > >>>>>>>>>>>>> unwritten rules, but we can have an explanation on > > >>> that. And > > >>>>>> I > > >>>>>>>> really > > >>>>>>>>>>>> doubt > > >>>>>>>>>>>>> whether user would complain about > > >>>>>>>>>>>>> this when they actually have flexible and good > > >>> experience > > >>>>>> using > > >>>>>>>> this. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> My tendency would be #3 > #1 > #2, what do you think? > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Best, > > >>>>>>>>>>>>> Kurt > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 1:11 PM Danny Chan < > > >>>>>> yuzhao....@gmail.com > > >>>>>>>>> > > >>>>>>>>>>> wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> Thanks Aljoscha ~ > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> I agree for most of the query hints, they are > > >>> optional as > > >>>>>> an > > >>>>>>>>>>> optimizer > > >>>>>>>>>>>>>> instruction, especially for the traditional RDBMS. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> But, just like BenChao said, Flink as a computation > > >>> engine > > >>>>>> has > > >>>>>>>> many > > >>>>>>>>>>>>>> different kind of data sources, thus, dynamic > > >>> parameters > > >>>>>> like > > >>>>>>>>>>>>> start_offest > > >>>>>>>>>>>>>> can only bind to each table scope, we can not set a > > >>> session > > >>>>>>>> config > > >>>>>>>>>>> like > > >>>>>>>>>>>>>> KSQL because they are all about Kafka: > > >>>>>>>>>>>>>>> SET ‘auto.offset.reset’=‘earliest’; > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Thus the most flexible way to set up these dynamic > > >>> params > > >>>>>> is > > >>>>>>>> to bind > > >>>>>>>>>>> to > > >>>>>>>>>>>>>> the table scope in the query when we want to > > >> override > > >>>>>>>> something, so > > >>>>>>>>>>> we > > >>>>>>>>>>>>> have > > >>>>>>>>>>>>>> these solutions above (with pros and cons from my > > >>> side): > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> • 1. Select * from t(offset=123) (from Timo) > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Pros: > > >>>>>>>>>>>>>> - Easy to add > > >>>>>>>>>>>>>> - Parameters are part of the main query > > >>>>>>>>>>>>>> Cons: > > >>>>>>>>>>>>>> - Not SQL compliant > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> • 2. Select * from t /*+ PROPERTIES(offset=123) */ > > >>> (from > > >>>>>> me) > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Pros: > > >>>>>>>>>>>>>> - Easy to add > > >>>>>>>>>>>>>> - SQL compliant because it is nested in the > > >> comments > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Cons: > > >>>>>>>>>>>>>> - Parameters are not part of the main query > > >>>>>>>>>>>>>> - Cryptic syntax for new users > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> The biggest problem for hints way may be the “if > > >>> hints > > >>>>>> must be > > >>>>>>>>>>>> optional”, > > >>>>>>>>>>>>>> actually we have though about 1 for a while but > > >>> aborted > > >>>>>>>> because it > > >>>>>>>>>>>> breaks > > >>>>>>>>>>>>>> the SQL standard too much. And we replace it with > > >> 2, > > >>>>>> because > > >>>>>>>> the > > >>>>>>>>>>> hints > > >>>>>>>>>>>>>> syntax do not break SQL standard(nested in > > >> comments). > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> What if we have the special /*+ PROPERTIES */ hint > > >>> that > > >>>>>> allows > > >>>>>>>>>>> override > > >>>>>>>>>>>>>> some properties of table dynamically, it does not > > >>> break > > >>>>>>>> anything, at > > >>>>>>>>>>>>> lease > > >>>>>>>>>>>>>> for current Flink use cases. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Planner hints are optional just because they are > > >>> naturally > > >>>>>>>> enforcers > > >>>>>>>>>>> of > > >>>>>>>>>>>>>> the planner, most of them aim to instruct the > > >>> optimizer, > > >>>>>> but, > > >>>>>>>> the > > >>>>>>>>>>> table > > >>>>>>>>>>>>>> hints is a little different, table hints can > > >> specify > > >>> the > > >>>>>> table > > >>>>>>>> meta > > >>>>>>>>>>>> like > > >>>>>>>>>>>>>> index column, and it is very convenient to specify > > >>> table > > >>>>>>>> properties. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Or shall we not call /*+ PROPERTIES(offset=123) */ > > >>> table > > >>>>>> hint, > > >>>>>>>> we > > >>>>>>>>>>> can > > >>>>>>>>>>>>>> call it table dynamic parameters. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>> Danny Chan > > >>>>>>>>>>>>>> 在 2020年3月11日 +0800 PM9:20，Aljoscha Krettek < > > >>>>>>>> aljos...@apache.org>，写道： > > >>>>>>>>>>>>>>> Hi, > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> I don't understand this discussion. Hints, as I > > >>>>>> understand > > >>>>>>>> them, > > >>>>>>>>>>>> should > > >>>>>>>>>>>>>>> work like this: > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> - hints are *optional* advice for the optimizer > > >> to > > >>> try > > >>>>>> and > > >>>>>>>> help it > > >>>>>>>>>>> to > > >>>>>>>>>>>>>>> find a good execution strategy > > >>>>>>>>>>>>>>> - hints should not change query semantics, i.e. > > >>> they > > >>>>>> should > > >>>>>>>> not > > >>>>>>>>>>>> change > > >>>>>>>>>>>>>>> connector properties executing a query with > > >> taking > > >>> into > > >>>>>>>> account the > > >>>>>>>>>>>>>>> hints *must* produce the same result as executing > > >>> the > > >>>>>> query > > >>>>>>>> without > > >>>>>>>>>>>>>>> taking into account the hints > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> From these simple requirements you can derive a > > >>> solution > > >>>>>>>> that makes > > >>>>>>>>>>>>>>> sense. I don't have a strong preference for the > > >>> syntax > > >>>>>> but we > > >>>>>>>>>>> should > > >>>>>>>>>>>>>>> strive to be in line with prior work. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>> Aljoscha > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On 11.03.20 11:53, Danny Chan wrote: > > >>>>>>>>>>>>>>>> Thanks Timo for summarize the 3 options ~ > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> I agree with Kurt that option2 is too > > >>> complicated to > > >>>>>> use > > >>>>>>>> because: > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> • As a Kafka topic consumer, the user must > > >>> define both > > >>>>>> the > > >>>>>>>>>>> virtual > > >>>>>>>>>>>>>> column for start offset and he must apply a special > > >>> filter > > >>>>>>>> predicate > > >>>>>>>>>>>>> after > > >>>>>>>>>>>>>> each query > > >>>>>>>>>>>>>>>> • And for the internal implementation, the > > >>> metadata > > >>>>>> column > > >>>>>>>> push > > >>>>>>>>>>>> down > > >>>>>>>>>>>>>> is another hard topic, each kind of message queue > > >>> may have > > >>>>>> its > > >>>>>>>> offset > > >>>>>>>>>>>>>> attribute, we need to consider the expression type > > >>> for > > >>>>>>>> different > > >>>>>>>>>>> kind; > > >>>>>>>>>>>>> the > > >>>>>>>>>>>>>> source also need to recognize the constant column > > >> as > > >>> a > > >>>>>> config > > >>>>>>>>>>>>> option(which > > >>>>>>>>>>>>>> is weird because usually what we pushed down is a > > >>> table > > >>>>>> column) > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> For option 1 and option3, I think there is no > > >>>>>> difference, > > >>>>>>>> option1 > > >>>>>>>>>>>> is > > >>>>>>>>>>>>>> also a hint syntax which is introduced in Sybase > > >> and > > >>>>>>>> referenced then > > >>>>>>>>>>>>>> deprecated by MS-SQL in 199X years because of the > > >>>>>>>> ambitiousness. > > >>>>>>>>>>>>> Personally > > >>>>>>>>>>>>>> I prefer /*+ */ style table hint than WITH keyword > > >>> for > > >>>>>> these > > >>>>>>>> reasons: > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> • We do not break the standard SQL, the hints > > >> are > > >>>>>> nested > > >>>>>>>> in SQL > > >>>>>>>>>>>>>> comments > > >>>>>>>>>>>>>>>> • We do not need to introduce additional WITH > > >>> keyword > > >>>>>>>> which may > > >>>>>>>>>>>>> appear > > >>>>>>>>>>>>>> in a query if we use that because a table can be > > >>>>>> referenced in > > >>>>>>>> all > > >>>>>>>>>>>> kinds > > >>>>>>>>>>>>> of > > >>>>>>>>>>>>>> SQL contexts: INSERT/DELETE/FROM/JOIN …. That would > > >>> make > > >>>>>> our > > >>>>>>>> sql > > >>>>>>>>>>> query > > >>>>>>>>>>>>>> break too much of the SQL from standard > > >>>>>>>>>>>>>>>> • We would have uniform syntax for hints as > > >> query > > >>>>>> hint, one > > >>>>>>>>>>> syntax > > >>>>>>>>>>>>>> fits all and more easy to use > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> And here is the reason why we choose a uniform > > >>> Oracle > > >>>>>>>> style query > > >>>>>>>>>>>>>> hint syntax which is addressed by Julian Hyde when > > >> we > > >>>>>> design > > >>>>>>>> the > > >>>>>>>>>>> syntax > > >>>>>>>>>>>>>> from the Calcite community: > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> I don’t much like the MSSQL-style syntax for > > >>> table > > >>>>>> hints. > > >>>>>>>> It > > >>>>>>>>>>> adds a > > >>>>>>>>>>>>>> new use of the WITH keyword that is unrelated to > > >> the > > >>> use of > > >>>>>>>> WITH for > > >>>>>>>>>>>>>> common-table expressions. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> A historical note. Microsoft SQL Server > > >>> inherited its > > >>>>>> hint > > >>>>>>>> syntax > > >>>>>>>>>>>>> from > > >>>>>>>>>>>>>> Sybase a very long time ago. (See “Transact SQL > > >>>>>>>> Programming”[1], page > > >>>>>>>>>>>>> 632, > > >>>>>>>>>>>>>> “Optimizer hints”. The book was written in 1999, > > >> and > > >>> covers > > >>>>>>>> Microsoft > > >>>>>>>>>>>> SQL > > >>>>>>>>>>>>>> Server 6.5 / 7.0 and Sybase Adaptive Server 11.5, > > >>> but the > > >>>>>>>> syntax very > > >>>>>>>>>>>>>> likely predates Sybase 4.3, from which Microsoft > > >> SQL > > >>>>>> Server was > > >>>>>>>>>>> forked > > >>>>>>>>>>>> in > > >>>>>>>>>>>>>> 1993.) > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Microsoft later added the WITH keyword to make > > >>> it less > > >>>>>>>> ambiguous, > > >>>>>>>>>>>> and > > >>>>>>>>>>>>>> has now deprecated the syntax that does not use > > >> WITH. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> They are forced to keep the syntax for > > >> backwards > > >>>>>>>> compatibility > > >>>>>>>>>>> but > > >>>>>>>>>>>>>> that doesn’t mean that we should shoulder their > > >>> burden. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> I think formatted comments are the right > > >>> container for > > >>>>>>>> hints > > >>>>>>>>>>>> because > > >>>>>>>>>>>>>> it allows us to change the hint syntax without > > >>> changing > > >>>>>> the SQL > > >>>>>>>>>>> parser, > > >>>>>>>>>>>>> and > > >>>>>>>>>>>>>> makes clear that we are at liberty to ignore hints > > >>>>>> entirely. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Julian > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> [1] https://www.amazon.com/s?k=9781565924017 < > > >>>>>>>>>>>>>> https://www.amazon.com/s?k=9781565924017> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>>> Danny Chan > > >>>>>>>>>>>>>>>> 在 2020年3月11日 +0800 PM4:03，Timo Walther < > > >>>>>> twal...@apache.org > > >>>>>>>>> ，写道： > > >>>>>>>>>>>>>>>>> Hi Danny, > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> it is true that our DDL is not standard > > >>> compliant by > > >>>>>>>> using the > > >>>>>>>>>>>> WITH > > >>>>>>>>>>>>>>>>> clause. Nevertheless, we aim for not > > >> diverging > > >>> too > > >>>>>> much > > >>>>>>>> and the > > >>>>>>>>>>>>> LIKE > > >>>>>>>>>>>>>>>>> clause is an example of that. It will solve > > >>> things > > >>>>>> like > > >>>>>>>>>>>> overwriting > > >>>>>>>>>>>>>>>>> WATERMARKs, add additional/modifying > > >>> properties and > > >>>>>>>> inherit > > >>>>>>>>>>>> schema. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Bowen is right that Flink's DDL is mixing 3 > > >>> types > > >>>>>>>> definition > > >>>>>>>>>>>>>> together. > > >>>>>>>>>>>>>>>>> We are not the first ones that try to solve > > >>> this. > > >>>>>> There > > >>>>>>>> is also > > >>>>>>>>>>>> the > > >>>>>>>>>>>>>> SQL > > >>>>>>>>>>>>>>>>> MED standard [1] that tried to tackle this > > >>> problem. I > > >>>>>>>> think it > > >>>>>>>>>>>> was > > >>>>>>>>>>>>>> not > > >>>>>>>>>>>>>>>>> considered when designing the current DDL. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Currently, I see 3 options for handling Kafka > > >>>>>> offsets. I > > >>>>>>>> will > > >>>>>>>>>>>> give > > >>>>>>>>>>>>>> some > > >>>>>>>>>>>>>>>>> examples and look forward to feedback here: > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> *Option 1* Runtime and semantic parms as part > > >>> of the > > >>>>>>>> query > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> `SELECT * FROM MyTable('offset'=123)` > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Pros: > > >>>>>>>>>>>>>>>>> - Easy to add > > >>>>>>>>>>>>>>>>> - Parameters are part of the main query > > >>>>>>>>>>>>>>>>> - No complicated hinting syntax > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Cons: > > >>>>>>>>>>>>>>>>> - Not SQL compliant > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> *Option 2* Use metadata in query > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> `CREATE TABLE MyTable (id INT, offset AS > > >>>>>>>>>>>>> SYSTEM_METADATA('offset'))` > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> `SELECT * FROM MyTable WHERE offset > > > >> TIMESTAMP > > >>>>>>>> '2012-12-12 > > >>>>>>>>>>>>>> 12:34:22'` > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Pros: > > >>>>>>>>>>>>>>>>> - SQL compliant in the query > > >>>>>>>>>>>>>>>>> - Access of metadata in the DDL which is > > >>> required > > >>>>>> anyway > > >>>>>>>>>>>>>>>>> - Regular pushdown rules apply > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Cons: > > >>>>>>>>>>>>>>>>> - Users need to add an additional comlumn in > > >>> the DDL > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> *Option 3*: Use hints for properties > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> ` > > >>>>>>>>>>>>>>>>> SELECT * > > >>>>>>>>>>>>>>>>> FROM MyTable /*+ PROPERTIES('offset'=123) */ > > >>>>>>>>>>>>>>>>> ` > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Pros: > > >>>>>>>>>>>>>>>>> - Easy to add > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Cons: > > >>>>>>>>>>>>>>>>> - Parameters are not part of the main query > > >>>>>>>>>>>>>>>>> - Cryptic syntax for new users > > >>>>>>>>>>>>>>>>> - Not standard compliant. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> If we go with this option, I would suggest to > > >>> make it > > >>>>>>>> available > > >>>>>>>>>>>> in > > >>>>>>>>>>>>> a > > >>>>>>>>>>>>>>>>> separate map and don't mix it with statically > > >>> defined > > >>>>>>>>>>> properties. > > >>>>>>>>>>>>>> Such > > >>>>>>>>>>>>>>>>> that the factory can decide which properties > > >>> have the > > >>>>>>>> right to > > >>>>>>>>>>> be > > >>>>>>>>>>>>>>>>> overwritten by the hints: > > >>>>>>>>>>>>>>>>> TableSourceFactory.Context.getQueryHints(): > > >>>>>>>> ReadableConfig > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Regards, > > >>>>>>>>>>>>>>>>> Timo > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> [1] https://en.wikipedia.org/wiki/SQL/MED > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Currently I see 3 options as a > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> On 11.03.20 07:21, Danny Chan wrote: > > >>>>>>>>>>>>>>>>>> Thanks Bowen ~ > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> I agree we should somehow categorize our > > >>> connector > > >>>>>>>>>>> parameters. > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> For type1, I’m already preparing a solution > > >>> like > > >>>>>> the > > >>>>>>>>>>> Confluent > > >>>>>>>>>>>>>> schema registry + Avro schema inference thing, so > > >>> this may > > >>>>>> not > > >>>>>>>> be a > > >>>>>>>>>>>>> problem > > >>>>>>>>>>>>>> in the near future. > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> For type3, I have some questions: > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> "SELECT * FROM mykafka WHERE offset > > > >> 12pm > > >>>>>> yesterday” > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Where does the offset column come from, a > > >>> virtual > > >>>>>>>> column from > > >>>>>>>>>>>> the > > >>>>>>>>>>>>>> table schema, you said that > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> They change > > >>>>>>>>>>>>>>>>>> almost every time a query starts and have > > >>> nothing > > >>>>>> to > > >>>>>>>> do with > > >>>>>>>>>>>>>> metadata, thus > > >>>>>>>>>>>>>>>>>> should not be part of table definition/DDL > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> But why you can reference it in the query, > > >>> I’m > > >>>>>>>> confused for > > >>>>>>>>>>>> that, > > >>>>>>>>>>>>>> can you elaborate a little ? > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>>>>> Danny Chan > > >>>>>>>>>>>>>>>>>> 在 2020年3月11日 +0800 PM12:52，Bowen Li < > > >>>>>>>> bowenl...@gmail.com > > >>>>>>>>>>>> ，写道： > > >>>>>>>>>>>>>>>>>>> Thanks Danny for kicking off the effort > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> The root cause of too much manual work is > > >>> Flink > > >>>>>> DDL > > >>>>>>>> has > > >>>>>>>>>>>> mixed 3 > > >>>>>>>>>>>>>> types of > > >>>>>>>>>>>>>>>>>>> params together and doesn't handle each > > >> of > > >>> them > > >>>>>> very > > >>>>>>>> well. > > >>>>>>>>>>>>> Below > > >>>>>>>>>>>>>> are how I > > >>>>>>>>>>>>>>>>>>> categorize them and corresponding > > >>> solutions in my > > >>>>>>>> mind: > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> - type 1: Metadata of external data, like > > >>>>>> external > > >>>>>>>>>>>>> endpoint/url, > > >>>>>>>>>>>>>>>>>>> username/pwd, schemas, formats. > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> Such metadata are mostly already > > >>> accessible in > > >>>>>>>> external > > >>>>>>>>>>>> system > > >>>>>>>>>>>>>> as long as > > >>>>>>>>>>>>>>>>>>> endpoints and credentials are provided. > > >>> Flink can > > >>>>>>>> get it > > >>>>>>>>>>> thru > > >>>>>>>>>>>>>> catalogs, but > > >>>>>>>>>>>>>>>>>>> we haven't had many catalogs yet and thus > > >>> Flink > > >>>>>> just > > >>>>>>>> hasn't > > >>>>>>>>>>>>> been > > >>>>>>>>>>>>>> able to > > >>>>>>>>>>>>>>>>>>> leverage that. So the solution should be > > >>> building > > >>>>>>>> more > > >>>>>>>>>>>>> catalogs. > > >>>>>>>>>>>>>> Such > > >>>>>>>>>>>>>>>>>>> params should be part of a Flink table > > >>>>>>>> DDL/definition, and > > >>>>>>>>>>>> not > > >>>>>>>>>>>>>> overridable > > >>>>>>>>>>>>>>>>>>> in any means. > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> - type 2: Runtime params, like jdbc > > >>> connector's > > >>>>>>>> fetch size, > > >>>>>>>>>>>>>> elasticsearch > > >>>>>>>>>>>>>>>>>>> connector's bulk flush size. > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> Such params don't affect query results, > > >> but > > >>>>>> affect > > >>>>>>>> how > > >>>>>>>>>>>> results > > >>>>>>>>>>>>>> are produced > > >>>>>>>>>>>>>>>>>>> (eg. fast or slow, aka performance) - > > >> they > > >>> are > > >>>>>>>> essentially > > >>>>>>>>>>>>>> execution and > > >>>>>>>>>>>>>>>>>>> implementation details. They change often > > >>> in > > >>>>>>>> exploration or > > >>>>>>>>>>>>>> development > > >>>>>>>>>>>>>>>>>>> stages, but not quite frequently in > > >>> well-defined > > >>>>>>>>>>> long-running > > >>>>>>>>>>>>>> pipelines. > > >>>>>>>>>>>>>>>>>>> They should always have default values > > >> and > > >>> can be > > >>>>>>>> missing > > >>>>>>>>>>> in > > >>>>>>>>>>>>>> query. They > > >>>>>>>>>>>>>>>>>>> can be part of a table DDL/definition, > > >> but > > >>> should > > >>>>>>>> also be > > >>>>>>>>>>>>>> replaceable in a > > >>>>>>>>>>>>>>>>>>> query - *this is what table "hints" in > > >>> FLIP-113 > > >>>>>>>> should > > >>>>>>>>>>>> cover*. > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> - type 3: Semantic params, like kafka > > >>> connector's > > >>>>>>>> start > > >>>>>>>>>>>> offset. > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> Such params affect query results - the > > >>> semantics. > > >>>>>>>> They'd > > >>>>>>>>>>>> better > > >>>>>>>>>>>>>> be as > > >>>>>>>>>>>>>>>>>>> filter conditions in WHERE clause that > > >> can > > >>> be > > >>>>>> pushed > > >>>>>>>> down. > > >>>>>>>>>>>> They > > >>>>>>>>>>>>>> change > > >>>>>>>>>>>>>>>>>>> almost every time a query starts and have > > >>>>>> nothing to > > >>>>>>>> do > > >>>>>>>>>>> with > > >>>>>>>>>>>>>> metadata, thus > > >>>>>>>>>>>>>>>>>>> should not be part of table > > >>> definition/DDL, nor > > >>>>>> be > > >>>>>>>>>>> persisted > > >>>>>>>>>>>> in > > >>>>>>>>>>>>>> catalogs. > > >>>>>>>>>>>>>>>>>>> If they will, users should create views > > >> to > > >>> keep > > >>>>>> such > > >>>>>>>> params > > >>>>>>>>>>>>>> around (note > > >>>>>>>>>>>>>>>>>>> this is different from variable > > >>> substitution). > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> Take Flink-Kafka as an example. Once we > > >>> get these > > >>>>>>>> params > > >>>>>>>>>>>> right, > > >>>>>>>>>>>>>> here're the > > >>>>>>>>>>>>>>>>>>> steps users need to do to develop and run > > >>> a Flink > > >>>>>>>> job: > > >>>>>>>>>>>>>>>>>>> - configure a Flink > > >>> ConfluentSchemaRegistry with > > >>>>>> url, > > >>>>>>>>>>>> username, > > >>>>>>>>>>>>>> and password > > >>>>>>>>>>>>>>>>>>> - run "SELECT * FROM mykafka WHERE offset > > >>>> 12pm > > >>>>>>>> yesterday" > > >>>>>>>>>>>>>> (simplified > > >>>>>>>>>>>>>>>>>>> timestamp) in SQL CLI, Flink > > >> automatically > > >>>>>> retrieves > > >>>>>>>> all > > >>>>>>>>>>>>>> metadata of > > >>>>>>>>>>>>>>>>>>> schema, file format, etc and start the > > >> job > > >>>>>>>>>>>>>>>>>>> - users want to make the job read Kafka > > >>> topic > > >>>>>>>> faster, so it > > >>>>>>>>>>>>> goes > > >>>>>>>>>>>>>> as "SELECT > > >>>>>>>>>>>>>>>>>>> * FROM mykafka /* faster_read_key=value*/ > > >>> WHERE > > >>>>>>>> offset > > > >>>>>>>>>>> 12pm > > >>>>>>>>>>>>>> yesterday" > > >>>>>>>>>>>>>>>>>>> - done and satisfied, users submit it to > > >>>>>> production > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> Regarding "CREATE TABLE t LIKE with > > >> (k1=v1, > > >>>>>> k2=v2), > > >>>>>>>> I think > > >>>>>>>>>>>>> it's > > >>>>>>>>>>>>>> a > > >>>>>>>>>>>>>>>>>>> nice-to-have feature, but not a > > >>> strategically > > >>>>>>>> critical, > > >>>>>>>>>>>>>> long-term solution, > > >>>>>>>>>>>>>>>>>>> because > > >>>>>>>>>>>>>>>>>>> 1) It may seem promising at the current > > >>> stage to > > >>>>>>>> solve the > > >>>>>>>>>>>>>>>>>>> too-much-manual-work problem, but that's > > >>> only > > >>>>>>>> because Flink > > >>>>>>>>>>>>>> hasn't > > >>>>>>>>>>>>>>>>>>> leveraged catalogs well and handled the 3 > > >>> types > > >>>>>> of > > >>>>>>>> params > > >>>>>>>>>>>> above > > >>>>>>>>>>>>>> properly. > > >>>>>>>>>>>>>>>>>>> Once we get the params types right, the > > >>> LIKE > > >>>>>> syntax > > >>>>>>>> won't > > >>>>>>>>>>> be > > >>>>>>>>>>>>> that > > >>>>>>>>>>>>>>>>>>> important, and will be just an easier way > > >>> to > > >>>>>> create > > >>>>>>>> tables > > >>>>>>>>>>>>>> without retyping > > >>>>>>>>>>>>>>>>>>> long fields like username and pwd. > > >>>>>>>>>>>>>>>>>>> 2) Note that only some rare type of > > >>> catalog can > > >>>>>>>> store k-v > > >>>>>>>>>>>>>> property pair, so > > >>>>>>>>>>>>>>>>>>> table created this way often cannot be > > >>>>>> persisted. In > > >>>>>>>> the > > >>>>>>>>>>>>>> foreseeable > > >>>>>>>>>>>>>>>>>>> future, such catalog will only be > > >>> HiveCatalog, > > >>>>>> and > > >>>>>>>> not > > >>>>>>>>>>>> everyone > > >>>>>>>>>>>>>> has a Hive > > >>>>>>>>>>>>>>>>>>> metastore. To be honest, without > > >>> persistence, > > >>>>>>>> recreating > > >>>>>>>>>>>> tables > > >>>>>>>>>>>>>> every time > > >>>>>>>>>>>>>>>>>>> this way is still a lot of keyboard > > >> typing. > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> Cheers, > > >>>>>>>>>>>>>>>>>>> Bowen > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> On Tue, Mar 10, 2020 at 8:07 PM Kurt > > >> Young > > >>> < > > >>>>>>>>>>> ykt...@gmail.com > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> If a specific connector want to have > > >> such > > >>>>>>>> parameter and > > >>>>>>>>>>>> read > > >>>>>>>>>>>>>> if out of > > >>>>>>>>>>>>>>>>>>>> configuration, then that's fine. > > >>>>>>>>>>>>>>>>>>>> If we are talking about a configuration > > >>> for all > > >>>>>>>> kinds of > > >>>>>>>>>>>>>> sources, I would > > >>>>>>>>>>>>>>>>>>>> be super careful about that. > > >>>>>>>>>>>>>>>>>>>> It's true it can solve maybe 80% cases, > > >>> but it > > >>>>>>>> will also > > >>>>>>>>>>>> make > > >>>>>>>>>>>>>> the left 20% > > >>>>>>>>>>>>>>>>>>>> feels weird. > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>>>>>>> Kurt > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> On Wed, Mar 11, 2020 at 11:00 AM Jark > > >> Wu > > >>> < > > >>>>>>>>>>> imj...@gmail.com > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> Hi Kurt, > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> #3 Regarding to global offset: > > >>>>>>>>>>>>>>>>>>>>> I'm not saying to use the global > > >>>>>> configuration to > > >>>>>>>>>>>> override > > >>>>>>>>>>>>>> connector > > >>>>>>>>>>>>>>>>>>>>> properties by the planner. > > >>>>>>>>>>>>>>>>>>>>> But the connector should take this > > >>>>>> configuration > > >>>>>>>> and > > >>>>>>>>>>>>>> translate into their > > >>>>>>>>>>>>>>>>>>>>> client API. > > >>>>>>>>>>>>>>>>>>>>> AFAIK, almost all the message queues > > >>> support > > >>>>>>>> eariliest > > >>>>>>>>>>>> and > > >>>>>>>>>>>>>> latest and a > > >>>>>>>>>>>>>>>>>>>>> timestamp value as start point. > > >>>>>>>>>>>>>>>>>>>>> So we can support 3 options for this > > >>>>>>>> configuration: > > >>>>>>>>>>>>>> "eariliest", "latest" > > >>>>>>>>>>>>>>>>>>>>> and a timestamp string value. > > >>>>>>>>>>>>>>>>>>>>> Of course, this can't solve 100% > > >>> cases, but I > > >>>>>>>> guess can > > >>>>>>>>>>>>>> sovle 80% or 90% > > >>>>>>>>>>>>>>>>>>>>> cases. > > >>>>>>>>>>>>>>>>>>>>> And the remaining cases can be > > >>> resolved by > > >>>>>> LIKE > > >>>>>>>> syntax > > >>>>>>>>>>>>> which > > >>>>>>>>>>>>>> I guess is > > >>>>>>>>>>>>>>>>>>>> not > > >>>>>>>>>>>>>>>>>>>>> very common cases. > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>>>>>>>> Jark > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> On Wed, 11 Mar 2020 at 10:33, Kurt > > >>> Young < > > >>>>>>>>>>>> ykt...@gmail.com > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> Good to have such lovely > > >>> discussions. I > > >>>>>> also > > >>>>>>>> want to > > >>>>>>>>>>>>> share > > >>>>>>>>>>>>>> some of my > > >>>>>>>>>>>>>>>>>>>>>> opinions. > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> #1 Regarding to error handling: I > > >>> also > > >>>>>> think > > >>>>>>>> ignore > > >>>>>>>>>>>>>> invalid hints would > > >>>>>>>>>>>>>>>>>>>>> be > > >>>>>>>>>>>>>>>>>>>>>> dangerous, maybe > > >>>>>>>>>>>>>>>>>>>>>> the simplest solution is just throw > > >>> an > > >>>>>>>> exception. > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> #2 Regarding to property > > >>> replacement: I > > >>>>>> don't > > >>>>>>>> think > > >>>>>>>>>>> we > > >>>>>>>>>>>>>> should > > >>>>>>>>>>>>>>>>>>>> constraint > > >>>>>>>>>>>>>>>>>>>>>> ourself to > > >>>>>>>>>>>>>>>>>>>>>> the meaning of the word "hint", and > > >>>>>> forbidden > > >>>>>>>> it > > >>>>>>>>>>>>> modifying > > >>>>>>>>>>>>>> any > > >>>>>>>>>>>>>>>>>>>> properties > > >>>>>>>>>>>>>>>>>>>>>> which can effect > > >>>>>>>>>>>>>>>>>>>>>> query results. IMO `PROPERTIES` is > > >>> one of > > >>>>>> the > > >>>>>>>> table > > >>>>>>>>>>>>> hints, > > >>>>>>>>>>>>>> and a > > >>>>>>>>>>>>>>>>>>>> powerful > > >>>>>>>>>>>>>>>>>>>>>> one. It can > > >>>>>>>>>>>>>>>>>>>>>> modify properties located in DDL's > > >>> WITH > > >>>>>> block. > > >>>>>>>> But I > > >>>>>>>>>>>> also > > >>>>>>>>>>>>>> see the harm > > >>>>>>>>>>>>>>>>>>>>> that > > >>>>>>>>>>>>>>>>>>>>>> if we make it > > >>>>>>>>>>>>>>>>>>>>>> too flexible like change the kafka > > >>> topic > > >>>>>> name > > >>>>>>>> with a > > >>>>>>>>>>>>> hint. > > >>>>>>>>>>>>>> Such use > > >>>>>>>>>>>>>>>>>>>> case > > >>>>>>>>>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>>>>>>>>> not common and > > >>>>>>>>>>>>>>>>>>>>>> sounds very dangerous to me. I > > >> would > > >>>>>> propose > > >>>>>>>> we have > > >>>>>>>>>>> a > > >>>>>>>>>>>>> map > > >>>>>>>>>>>>>> of hintable > > >>>>>>>>>>>>>>>>>>>>>> properties for each > > >>>>>>>>>>>>>>>>>>>>>> connector, and should validate all > > >>> passed > > >>>>>> in > > >>>>>>>>>>> properties > > >>>>>>>>>>>>>> are actually > > >>>>>>>>>>>>>>>>>>>>>> hintable. And combining with > > >>>>>>>>>>>>>>>>>>>>>> #1 error handling, we can throw an > > >>>>>> exception > > >>>>>>>> once > > >>>>>>>>>>>>> received > > >>>>>>>>>>>>>> invalid > > >>>>>>>>>>>>>>>>>>>>>> property. > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> #3 Regarding to global offset: I'm > > >>> not sure > > >>>>>>>> it's > > >>>>>>>>>>>>> feasible. > > >>>>>>>>>>>>>> Different > > >>>>>>>>>>>>>>>>>>>>>> connectors will have totally > > >>>>>>>>>>>>>>>>>>>>>> different properties to represent > > >>> offset, > > >>>>>> some > > >>>>>>>> might > > >>>>>>>>>>> be > > >>>>>>>>>>>>>> timestamps, > > >>>>>>>>>>>>>>>>>>>> some > > >>>>>>>>>>>>>>>>>>>>>> might be string literals > > >>>>>>>>>>>>>>>>>>>>>> like "earliest", and others might > > >> be > > >>> just > > >>>>>>>> integers. > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>>>>>>>>> Kurt > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> On Tue, Mar 10, 2020 at 11:46 PM > > >>> Jark Wu < > > >>>>>>>>>>>>> imj...@gmail.com> > > >>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> Hi everyone, > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> I want to jump in the discussion > > >>> about > > >>>>>> the > > >>>>>>>> "dynamic > > >>>>>>>>>>>>>> start offset" > > >>>>>>>>>>>>>>>>>>>>>> problem. > > >>>>>>>>>>>>>>>>>>>>>>> First of all, I share the same > > >>> concern > > >>>>>> with > > >>>>>>>> Timo > > >>>>>>>>>>> and > > >>>>>>>>>>>>>> Fabian, that the > > >>>>>>>>>>>>>>>>>>>>>>> "start offset" affects the query > > >>>>>> semantics, > > >>>>>>>> i.e. > > >>>>>>>>>>> the > > >>>>>>>>>>>>>> query result. > > >>>>>>>>>>>>>>>>>>>>>>> But "hints" is just used for > > >>> optimization > > >>>>>>>> which > > >>>>>>>>>>>> should > > >>>>>>>>>>>>>> affect the > > >>>>>>>>>>>>>>>>>>>>> result? > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> I think the "dynamic start > > >> offset" > > >>> is an > > >>>>>> very > > >>>>>>>>>>>> important > > >>>>>>>>>>>>>> usability > > >>>>>>>>>>>>>>>>>>>>> problem > > >>>>>>>>>>>>>>>>>>>>>>> which will be faced by many > > >>> streaming > > >>>>>>>> platforms. > > >>>>>>>>>>>>>>>>>>>>>>> I also agree "CREATE TEMPORARY > > >>> TABLE Temp > > >>>>>>>> (LIKE t) > > >>>>>>>>>>>> WITH > > >>>>>>>>>>>>>>>>>>>>>>> > > >>> ('connector.startup-timestamp-millis' = > > >>>>>>>>>>>>>> '1578538374471')" is verbose, > > >>>>>>>>>>>>>>>>>>>>>> what > > >>>>>>>>>>>>>>>>>>>>>>> if we have 10 tables to join? > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> However, what I want to propose > > >>> (should > > >>>>>> be > > >>>>>>>> another > > >>>>>>>>>>>>>> thread) is a > > >>>>>>>>>>>>>>>>>>>> global > > >>>>>>>>>>>>>>>>>>>>>>> configuration to reset start > > >>> offsets of > > >>>>>> all > > >>>>>>>> the > > >>>>>>>>>>>> source > > >>>>>>>>>>>>>> connectors > > >>>>>>>>>>>>>>>>>>>>>>> in the query session, e.g. > > >>>>>>>>>>>>> "table.sources.start-offset". > > >>>>>>>>>>>>>> This is > > >>>>>>>>>>>>>>>>>>>>> possible > > >>>>>>>>>>>>>>>>>>>>>>> now because > > >>> `TableSourceFactory.Context` > > >>>>>> has > > >>>>>>>>>>>>>> `getConfiguration` > > >>>>>>>>>>>>>>>>>>>>>>> method to get the session > > >>> configuration, > > >>>>>> and > > >>>>>>>> use it > > >>>>>>>>>>>> to > > >>>>>>>>>>>>>> create an > > >>>>>>>>>>>>>>>>>>>>> adapted > > >>>>>>>>>>>>>>>>>>>>>>> TableSource. > > >>>>>>>>>>>>>>>>>>>>>>> Then we can also expose to SQL > > >> CLI > > >>> via > > >>>>>> SET > > >>>>>>>> command, > > >>>>>>>>>>>>> e.g. > > >>>>>>>>>>>>>> `SET > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>> 'table.sources.start-offset'='earliest';`, > > >>>>>>>> which is > > >>>>>>>>>>>>>> pretty simple and > > >>>>>>>>>>>>>>>>>>>>>>> straightforward. > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> This is very similar to KSQL's > > >> `SET > > >>>>>>>>>>>>>> 'auto.offset.reset'='earliest'` > > >>>>>>>>>>>>>>>>>>>>> which > > >>>>>>>>>>>>>>>>>>>>>>> is very helpful IMO. > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>>>>>>>>>> Jark > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, 10 Mar 2020 at 22:29, > > >> Timo > > >>>>>> Walther < > > >>>>>>>>>>>>>> twal...@apache.org> > > >>>>>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> Hi Danny, > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> compared to the hints, FLIP-110 > > >>> is > > >>>>>> fully > > >>>>>>>>>>> compliant > > >>>>>>>>>>>> to > > >>>>>>>>>>>>>> the SQL > > >>>>>>>>>>>>>>>>>>>>> standard. > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> I don't think that `CREATE > > >>> TEMPORARY > > >>>>>> TABLE > > >>>>>>>> Temp > > >>>>>>>>>>>> (LIKE > > >>>>>>>>>>>>>> t) WITH > > >>>>>>>>>>>>>>>>>>>> (k=v)` > > >>>>>>>>>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>>>>>>>>>>> too verbose or awkward for the > > >>> power of > > >>>>>>>> basically > > >>>>>>>>>>>>>> changing the > > >>>>>>>>>>>>>>>>>>>> entire > > >>>>>>>>>>>>>>>>>>>>>>>> connector. Usually, this > > >>> statement > > >>>>>> would > > >>>>>>>> just > > >>>>>>>>>>>> precede > > >>>>>>>>>>>>>> the query in > > >>>>>>>>>>>>>>>>>>>> a > > >>>>>>>>>>>>>>>>>>>>>>>> multiline file. So it can be > > >>> change > > >>>>>>>> "in-place" > > >>>>>>>>>>> like > > >>>>>>>>>>>>>> the hints you > > >>>>>>>>>>>>>>>>>>>>>>> proposed. > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> Many companies have a > > >>> well-defined set > > >>>>>> of > > >>>>>>>> tables > > >>>>>>>>>>>> that > > >>>>>>>>>>>>>> should be > > >>>>>>>>>>>>>>>>>>>> used. > > >>>>>>>>>>>>>>>>>>>>>> It > > >>>>>>>>>>>>>>>>>>>>>>>> would be dangerous if users can > > >>> change > > >>>>>> the > > >>>>>>>> path > > >>>>>>>>>>> or > > >>>>>>>>>>>>>> topic in a hint. > > >>>>>>>>>>>>>>>>>>>>> The > > >>>>>>>>>>>>>>>>>>>>>>>> catalog/catalog manager should > > >>> be the > > >>>>>>>> entity that > > >>>>>>>>>>>>>> controls which > > >>>>>>>>>>>>>>>>>>>>> tables > > >>>>>>>>>>>>>>>>>>>>>>>> exist and how they can be > > >>> accessed. > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>> what’s the problem there if > > >> we > > >>> user > > >>>>>> the > > >>>>>>>> table > > >>>>>>>>>>>> hints > > >>>>>>>>>>>>>> to support > > >>>>>>>>>>>>>>>>>>>>>> “start > > >>>>>>>>>>>>>>>>>>>>>>>> offset”? > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> IMHO it violates the meaning of > > >>> a hint. > > >>>>>>>> According > > >>>>>>>>>>>> to > > >>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>> dictionary, > > >>>>>>>>>>>>>>>>>>>>> a > > >>>>>>>>>>>>>>>>>>>>>>>> hint is "a statement that > > >>> expresses > > >>>>>>>> indirectly > > >>>>>>>>>>> what > > >>>>>>>>>>>>>> one prefers not > > >>>>>>>>>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>>>>>>>>> say explicitly". But offsets > > >> are > > >>> a > > >>>>>>>> property that > > >>>>>>>>>>>> are > > >>>>>>>>>>>>>> very explicit. > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> If we go with the hint > > >> approach, > > >>> it > > >>>>>> should > > >>>>>>>> be > > >>>>>>>>>>>>>> expressible in the > > >>>>>>>>>>>>>>>>>>>>>>>> TableSourceFactory which > > >>> properties are > > >>>>>>>> supported > > >>>>>>>>>>>> for > > >>>>>>>>>>>>>> hinting. Or > > >>>>>>>>>>>>>>>>>>>> do > > >>>>>>>>>>>>>>>>>>>>>> you > > >>>>>>>>>>>>>>>>>>>>>>>> plan to offer those hints in a > > >>> separate > > >>>>>>>>>>> Map<String, > > >>>>>>>>>>>>>> String> that > > >>>>>>>>>>>>>>>>>>>>> cannot > > >>>>>>>>>>>>>>>>>>>>>>>> overwrite existing properties? > > >> I > > >>> think > > >>>>>>>> this would > > >>>>>>>>>>>> be > > >>>>>>>>>>>>> a > > >>>>>>>>>>>>>> different > > >>>>>>>>>>>>>>>>>>>>>> story... > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> Regards, > > >>>>>>>>>>>>>>>>>>>>>>>> Timo > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> On 10.03.20 10:34, Danny Chan > > >>> wrote: > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks Timo ~ > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>> Personally I would say that > > >>> offset > > > >>>>>> 0 > > >>>>>>>> and > > >>>>>>>>>>> start > > >>>>>>>>>>>>>> offset = 10 does > > >>>>>>>>>>>>>>>>>>>>> not > > >>>>>>>>>>>>>>>>>>>>>>>> have the same semantic, so from > > >>> the SQL > > >>>>>>>> aspect, > > >>>>>>>>>>> we > > >>>>>>>>>>>>> can > > >>>>>>>>>>>>>> not > > >>>>>>>>>>>>>>>>>>>> implement > > >>>>>>>>>>>>>>>>>>>>> a > > >>>>>>>>>>>>>>>>>>>>>>>> “starting offset” hint for > > >> query > > >>> with > > >>>>>> such > > >>>>>>>> a > > >>>>>>>>>>>> syntax. > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>> And the CREATE TABLE LIKE > > >>> syntax is a > > >>>>>>>> DDL which > > >>>>>>>>>>>> is > > >>>>>>>>>>>>>> just verbose > > >>>>>>>>>>>>>>>>>>>> for > > >>>>>>>>>>>>>>>>>>>>>>>> defining such dynamic > > >> parameters > > >>> even > > >>>>>> if > > >>>>>>>> it could > > >>>>>>>>>>>> do > > >>>>>>>>>>>>>> that, shall we > > >>>>>>>>>>>>>>>>>>>>>> force > > >>>>>>>>>>>>>>>>>>>>>>>> users to define a temporal > > >> table > > >>> for > > >>>>>> each > > >>>>>>>> query > > >>>>>>>>>>>> with > > >>>>>>>>>>>>>> dynamic > > >>>>>>>>>>>>>>>>>>>> params, > > >>>>>>>>>>>>>>>>>>>>> I > > >>>>>>>>>>>>>>>>>>>>>>>> would say it’s an awkward > > >>> solution. > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>> "Hints should give "hints" > > >> but > > >>> not > > >>>>>>>> affect the > > >>>>>>>>>>>>> actual > > >>>>>>>>>>>>>> produced > > >>>>>>>>>>>>>>>>>>>>>> result.” > > >>>>>>>>>>>>>>>>>>>>>>>> You mentioned that multiple > > >>> times and > > >>>>>>>> could we > > >>>>>>>>>>>> give a > > >>>>>>>>>>>>>> reason, > > >>>>>>>>>>>>>>>>>>>> what’s > > >>>>>>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>>>>>> problem there if we user the > > >>> table > > >>>>>> hints to > > >>>>>>>>>>> support > > >>>>>>>>>>>>>> “start offset” > > >>>>>>>>>>>>>>>>>>>> ? > > >>>>>>>>>>>>>>>>>>>>>> From > > >>>>>>>>>>>>>>>>>>>>>>>> my side I saw some benefits for > > >>> that: > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>> • It’s very convent to set up > > >>> these > > >>>>>>>> parameters, > > >>>>>>>>>>>> the > > >>>>>>>>>>>>>> syntax is > > >>>>>>>>>>>>>>>>>>>> very > > >>>>>>>>>>>>>>>>>>>>>> much > > >>>>>>>>>>>>>>>>>>>>>>>> like the DDL definition > > >>>>>>>>>>>>>>>>>>>>>>>>> • It’s scope is very clear, > > >>> right on > > >>>>>> the > > >>>>>>>> table > > >>>>>>>>>>> it > > >>>>>>>>>>>>>> attathed > > >>>>>>>>>>>>>>>>>>>>>>>>> • It does not affect the > > >> table > > >>>>>> schema, > > >>>>>>>> which > > >>>>>>>>>>>> means > > >>>>>>>>>>>>>> in order to > > >>>>>>>>>>>>>>>>>>>>>> specify > > >>>>>>>>>>>>>>>>>>>>>>>> the offset, there is no need to > > >>> define > > >>>>>> an > > >>>>>>>> offset > > >>>>>>>>>>>>>> column which is > > >>>>>>>>>>>>>>>>>>>>> weird > > >>>>>>>>>>>>>>>>>>>>>>>> actually, offset should never > > >> be > > >>> a > > >>>>>> column, > > >>>>>>>> it’s > > >>>>>>>>>>>> more > > >>>>>>>>>>>>>> like a > > >>>>>>>>>>>>>>>>>>>> metadata > > >>>>>>>>>>>>>>>>>>>>>> or a > > >>>>>>>>>>>>>>>>>>>>>>>> start option. > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>> So in total, FLIP-110 uses > > >> the > > >>> offset > > >>>>>>>> more > > >>>>>>>>>>> like a > > >>>>>>>>>>>>>> Hive partition > > >>>>>>>>>>>>>>>>>>>>>> prune, > > >>>>>>>>>>>>>>>>>>>>>>>> we can do that if we have an > > >>> offset > > >>>>>>>> column, but > > >>>>>>>>>>>> most > > >>>>>>>>>>>>>> of the case we > > >>>>>>>>>>>>>>>>>>>>> do > > >>>>>>>>>>>>>>>>>>>>>>> not > > >>>>>>>>>>>>>>>>>>>>>>>> define that, so there is > > >>> actually no > > >>>>>>>> conflict or > > >>>>>>>>>>>>>> overlap. > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>>>>>>>>>>>> Danny Chan > > >>>>>>>>>>>>>>>>>>>>>>>>> 在 2020年3月10日 +0800 > > >> PM4:28，Timo > > >>>>>> Walther < > > >>>>>>>>>>>>>> twal...@apache.org>，写道： > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Danny, > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't FLIP-110[1] solve > > >>> most > > >>>>>> of the > > >>>>>>>>>>>> problems > > >>>>>>>>>>>>>> we have around > > >>>>>>>>>>>>>>>>>>>>>>> defining > > >>>>>>>>>>>>>>>>>>>>>>>>>> table properties more > > >>> dynamically > > >>>>>>>> without > > >>>>>>>>>>>> manual > > >>>>>>>>>>>>>> schema work? > > >>>>>>>>>>>>>>>>>>>> Also > > >>>>>>>>>>>>>>>>>>>>>>>>>> offset definition is easier > > >>> with > > >>>>>> such a > > >>>>>>>>>>> syntax. > > >>>>>>>>>>>>>> They must not be > > >>>>>>>>>>>>>>>>>>>>>>> defined > > >>>>>>>>>>>>>>>>>>>>>>>>>> in catalog but could be > > >>> temporary > > >>>>>>>> tables that > > >>>>>>>>>>>>>> extend from the > > >>>>>>>>>>>>>>>>>>>>>> original > > >>>>>>>>>>>>>>>>>>>>>>>>>> table. > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> In general, we should aim > > >> to > > >>> keep > > >>>>>> the > > >>>>>>>> syntax > > >>>>>>>>>>>>>> concise and don't > > >>>>>>>>>>>>>>>>>>>>>> provide > > >>>>>>>>>>>>>>>>>>>>>>>>>> too many ways of doing the > > >>> same > > >>>>>> thing. > > >>>>>>>> Hints > > >>>>>>>>>>>>>> should give "hints" > > >>>>>>>>>>>>>>>>>>>>> but > > >>>>>>>>>>>>>>>>>>>>>>> not > > >>>>>>>>>>>>>>>>>>>>>>>>>> affect the actual produced > > >>> result. > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Some connector properties > > >>> might > > >>>>>> also > > >>>>>>>> change > > >>>>>>>>>>> the > > >>>>>>>>>>>>>> plan or schema > > >>>>>>>>>>>>>>>>>>>> in > > >>>>>>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>>>>>>>> future. E.g. they might > > >> also > > >>> define > > >>>>>>>> whether a > > >>>>>>>>>>>>>> table source > > >>>>>>>>>>>>>>>>>>>>> supports > > >>>>>>>>>>>>>>>>>>>>>>>>>> certain push-downs (e.g. > > >>> predicate > > >>>>>>>>>>> push-down). > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Dawid is currently working > > >> a > > >>> draft > > >>>>>>>> that might > > >>>>>>>>>>>>>> makes it possible > > >>>>>>>>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>>>>>>>>>>> expose a Kafka offset via > > >> the > > >>>>>> schema > > >>>>>>>> such > > >>>>>>>>>>> that > > >>>>>>>>>>>>>> `SELECT * FROM > > >>>>>>>>>>>>>>>>>>>>> Topic > > >>>>>>>>>>>>>>>>>>>>>>>>>> WHERE offset > 10` would > > >>> become > > >>>>>>>> possible and > > >>>>>>>>>>>>> could > > >>>>>>>>>>>>>> be pushed > > >>>>>>>>>>>>>>>>>>>> down. > > >>>>>>>>>>>>>>>>>>>>>> But > > >>>>>>>>>>>>>>>>>>>>>>>>>> this is of course, not > > >>> planned > > >>>>>>>> initially. > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Regards, > > >>>>>>>>>>>>>>>>>>>>>>>>>> Timo > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [1] > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>> > > >>>>>> > > >>> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-110%3A+Support+LIKE+clause+in+CREATE+TABLE > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> On 10.03.20 08:34, Danny > > >> Chan > > >>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Wenlong ~ > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> For PROPERTIES Hint Error > > >>>>>> handling > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Actually we have no way > > >> to > > >>>>>> figure out > > >>>>>>>>>>>> whether a > > >>>>>>>>>>>>>> error prone > > >>>>>>>>>>>>>>>>>>>> hint > > >>>>>>>>>>>>>>>>>>>>>> is a > > >>>>>>>>>>>>>>>>>>>>>>>> PROPERTIES hint, for example, > > >> if > > >>> use > > >>>>>>>> writes a > > >>>>>>>>>>> hint > > >>>>>>>>>>>>> like > > >>>>>>>>>>>>>>>>>>>> ‘PROPERTIAS’, > > >>>>>>>>>>>>>>>>>>>>>> we > > >>>>>>>>>>>>>>>>>>>>>>> do > > >>>>>>>>>>>>>>>>>>>>>>>> not know if this hint is a > > >>> PROPERTIES > > >>>>>>>> hint, what > > >>>>>>>>>>> we > > >>>>>>>>>>>>>> know is that > > >>>>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>>>> hint > > >>>>>>>>>>>>>>>>>>>>>>>> name was not registered in our > > >>> Flink. > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> If the user writes the > > >>> hint name > > >>>>>>>> correctly > > >>>>>>>>>>>>> (i.e. > > >>>>>>>>>>>>>> PROPERTIES), > > >>>>>>>>>>>>>>>>>>>> we > > >>>>>>>>>>>>>>>>>>>>>> did > > >>>>>>>>>>>>>>>>>>>>>>>> can enforce the validation of > > >>> the hint > > >>>>>>>> options > > >>>>>>>>>>>> though > > >>>>>>>>>>>>>> the pluggable > > >>>>>>>>>>>>>>>>>>>>>>>> HintOptionChecker. > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> For PROPERTIES Hint > > >> Option > > >>> Format > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> For a key value style > > >> hint > > >>>>>> option, > > >>>>>>>> the key > > >>>>>>>>>>>> can > > >>>>>>>>>>>>>> be either a > > >>>>>>>>>>>>>>>>>>>> simple > > >>>>>>>>>>>>>>>>>>>>>>>> identifier or a string literal, > > >>> which > > >>>>>>>> means that > > >>>>>>>>>>>> it’s > > >>>>>>>>>>>>>> compatible > > >>>>>>>>>>>>>>>>>>>> with > > >>>>>>>>>>>>>>>>>>>>>> our > > >>>>>>>>>>>>>>>>>>>>>>>> DDL syntax. We support simple > > >>>>>> identifier > > >>>>>>>> because > > >>>>>>>>>>>> many > > >>>>>>>>>>>>>> other hints > > >>>>>>>>>>>>>>>>>>>> do > > >>>>>>>>>>>>>>>>>>>>>> not > > >>>>>>>>>>>>>>>>>>>>>>>> have the component complex keys > > >>> like > > >>>>>> the > > >>>>>>>> table > > >>>>>>>>>>>>>> properties, and we > > >>>>>>>>>>>>>>>>>>>>> want > > >>>>>>>>>>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>>>>>>>>> unify the parse block. > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Danny Chan > > >>>>>>>>>>>>>>>>>>>>>>>>>>> 在 2020年3月10日 +0800 > > >>>>>>>> PM3:19，wenlong.lwl < > > >>>>>>>>>>>>>> wenlong88....@gmail.com > > >>>>>>>>>>>>>>>>>>>>>>> ，写道： > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Danny, thanks for > > >> the > > >>>>>> proposal. > > >>>>>>>> +1 for > > >>>>>>>>>>>>>> adding table hints, > > >>>>>>>>>>>>>>>>>>>> it > > >>>>>>>>>>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>>>>>>>>>>> really > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a necessary feature for > > >>> flink > > >>>>>> sql > > >>>>>>>> to > > >>>>>>>>>>>>> integrate > > >>>>>>>>>>>>>> with a catalog. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> For error handling, I > > >>> think it > > >>>>>>>> would be > > >>>>>>>>>>>> more > > >>>>>>>>>>>>>> natural to throw > > >>>>>>>>>>>>>>>>>>>> an > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exception when error > > >>> table hint > > >>>>>>>> provided, > > >>>>>>>>>>>>>> because the > > >>>>>>>>>>>>>>>>>>>> properties > > >>>>>>>>>>>>>>>>>>>>>> in > > >>>>>>>>>>>>>>>>>>>>>>>> hint > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> will be merged and used > > >>> to find > > >>>>>>>> the table > > >>>>>>>>>>>>>> factory which would > > >>>>>>>>>>>>>>>>>>>>>> cause > > >>>>>>>>>>>>>>>>>>>>>>> an > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exception when error > > >>> properties > > >>>>>>>> provided, > > >>>>>>>>>>>>>> right? On the other > > >>>>>>>>>>>>>>>>>>>>>> hand, > > >>>>>>>>>>>>>>>>>>>>>>>> unlike > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> other hints which just > > >>> affect > > >>>>>> the > > >>>>>>>> way to > > >>>>>>>>>>>>>> execute the query, > > >>>>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>>>>>> property > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> table hint actually > > >>> affects the > > >>>>>>>> result of > > >>>>>>>>>>>> the > > >>>>>>>>>>>>>> query, we should > > >>>>>>>>>>>>>>>>>>>>>> never > > >>>>>>>>>>>>>>>>>>>>>>>> ignore > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the given property > > >> hints. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> For the format of > > >>> property > > >>>>>> hints, > > >>>>>>>>>>>> currently, > > >>>>>>>>>>>>>> in sql client, we > > >>>>>>>>>>>>>>>>>>>>>>> accept > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> properties in format of > > >>> string > > >>>>>>>> only in > > >>>>>>>>>>> DDL: > > >>>>>>>>>>>>>>>>>>>>>>> 'connector.type'='kafka', > > >>>>>>>>>>>>>>>>>>>>>>>> I > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> think the format of > > >>> properties > > >>>>>> in > > >>>>>>>> hint > > >>>>>>>>>>>> should > > >>>>>>>>>>>>>> be the same as > > >>>>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>>>>>> format we > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> defined in ddl. What do > > >>> you > > >>>>>> think? > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Bests, > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Wenlong Lyu > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, 10 Mar 2020 at > > >>> 14:22, > > >>>>>>>> Danny Chan > > >>>>>>>>>>> < > > >>>>>>>>>>>>>>>>>>>> yuzhao....@gmail.com> > > >>>>>>>>>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> To Weike: About the > > >>> Error > > >>>>>> Handing > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> To be consistent with > > >>> other > > >>>>>> SQL > > >>>>>>>>>>> vendors, > > >>>>>>>>>>>>> the > > >>>>>>>>>>>>>> default is to > > >>>>>>>>>>>>>>>>>>>> log > > >>>>>>>>>>>>>>>>>>>>>>>> warnings > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and if there is any > > >>> error > > >>>>>>>> (invalid hint > > >>>>>>>>>>>>> name > > >>>>>>>>>>>>>> or options), the > > >>>>>>>>>>>>>>>>>>>>>> hint > > >>>>>>>>>>>>>>>>>>>>>>>> is just > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ignored. I have > > >> already > > >>>>>>>> addressed in > > >>>>>>>>>>> the > > >>>>>>>>>>>>>> wiki. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> To Timo: About the > > >>> PROPERTIES > > >>>>>>>> Table > > >>>>>>>>>>> Hint > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> • The properties > > >> hints > > >>> is > > >>>>>> also > > >>>>>>>>>>> optional, > > >>>>>>>>>>>>>> user can pass in an > > >>>>>>>>>>>>>>>>>>>>>> option > > >>>>>>>>>>>>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> override the table > > >>> properties > > >>>>>>>> but this > > >>>>>>>>>>>> does > > >>>>>>>>>>>>>> not mean it is > > >>>>>>>>>>>>>>>>>>>>>>> required. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> • They should not > > >>> include > > >>>>>>>> semantics: > > >>>>>>>>>>> does > > >>>>>>>>>>>>>> the properties > > >>>>>>>>>>>>>>>>>>>> belong > > >>>>>>>>>>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> semantic ? I don't > > >>> think so, > > >>>>>> the > > >>>>>>>> plan > > >>>>>>>>>>>> does > > >>>>>>>>>>>>>> not change right ? > > >>>>>>>>>>>>>>>>>>>>> The > > >>>>>>>>>>>>>>>>>>>>>>>> result > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> set may be affected, > > >>> but > > >>>>>> there > > >>>>>>>> are > > >>>>>>>>>>>> already > > >>>>>>>>>>>>>> some hints do so, > > >>>>>>>>>>>>>>>>>>>>> for > > >>>>>>>>>>>>>>>>>>>>>>>> example, > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MS-SQL MAXRECURSION > > >> and > > >>>>>> SNAPSHOT > > >>>>>>>> hint > > >>>>>>>>>>> [1] > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> • `SELECT * FROM > > >> t(k=v, > > >>>>>> k=v)`: > > >>>>>>>> this > > >>>>>>>>>>>> grammar > > >>>>>>>>>>>>>> breaks the SQL > > >>>>>>>>>>>>>>>>>>>>>> standard > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> compared to the hints > > >>>>>> way(which > > >>>>>>>> is > > >>>>>>>>>>>> included > > >>>>>>>>>>>>>> in comments) > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> • I actually didn't > > >>> found any > > >>>>>>>> vendors > > >>>>>>>>>>> to > > >>>>>>>>>>>>>> support such > > >>>>>>>>>>>>>>>>>>>> grammar, > > >>>>>>>>>>>>>>>>>>>>>> and > > >>>>>>>>>>>>>>>>>>>>>>>> there > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> is no way to override > > >>> table > > >>>>>> level > > >>>>>>>>>>>>> properties > > >>>>>>>>>>>>>> dynamically. For > > >>>>>>>>>>>>>>>>>>>>>>> normal > > >>>>>>>>>>>>>>>>>>>>>>>> RDBMS, > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think there are no > > >>> requests > > >>>>>>>> for such > > >>>>>>>>>>>>>> dynamic parameters > > >>>>>>>>>>>>>>>>>>>>> because > > >>>>>>>>>>>>>>>>>>>>>>>> all the > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> table have the same > > >>> storage > > >>>>>> and > > >>>>>>>>>>>> computation > > >>>>>>>>>>>>>> and they are > > >>>>>>>>>>>>>>>>>>>> almost > > >>>>>>>>>>>>>>>>>>>>>> all > > >>>>>>>>>>>>>>>>>>>>>>>> batch > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> tables. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> • While Flink as a > > >>>>>> computation > > >>>>>>>> engine > > >>>>>>>>>>> has > > >>>>>>>>>>>>>> many connectors, > > >>>>>>>>>>>>>>>>>>>>>>>> especially for > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> some message queue > > >> like > > >>>>>> Kafka, > > >>>>>>>> we would > > >>>>>>>>>>>>> have > > >>>>>>>>>>>>>> a start_offset > > >>>>>>>>>>>>>>>>>>>>> which > > >>>>>>>>>>>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different each time > > >> we > > >>> start > > >>>>>> the > > >>>>>>>> query, > > >>>>>>>>>>>>> such > > >>>>>>>>>>>>>> parameters can > > >>>>>>>>>>>>>>>>>>>> not > > >>>>>>>>>>>>>>>>>>>>>> be > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> persisted to catalog, > > >>> because > > >>>>>>>> it’s not > > >>>>>>>>>>>>>> static, this is > > >>>>>>>>>>>>>>>>>>>> actually > > >>>>>>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> background we propose > > >>> the > > >>>>>> table > > >>>>>>>> hints > > >>>>>>>>>>> to > > >>>>>>>>>>>>>> indicate such > > >>>>>>>>>>>>>>>>>>>>> properties > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> dynamically. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> To Jark and Jinsong: > > >> I > > >>> have > > >>>>>>>> removed the > > >>>>>>>>>>>>>> query hints part and > > >>>>>>>>>>>>>>>>>>>>>> change > > >>>>>>>>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> title. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>> > > >>>>>> > > >>> > > >> > > > https://docs.microsoft.com/en-us/sql/t-sql/queries/hints-transact-sql-query?view=sql-server-ver15 > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Danny Chan > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 在 2020年3月9日 +0800 > > >>> PM5:46，Timo > > >>>>>>>> Walther < > > >>>>>>>>>>>>>> twal...@apache.org > > >>>>>>>>>>>>>>>>>>>>> ，写道： > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Danny, > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the > > >>> proposal. I > > >>>>>>>> agree with > > >>>>>>>>>>>>> Jark > > >>>>>>>>>>>>>> and Jingsong. > > >>>>>>>>>>>>>>>>>>>>> Planner > > >>>>>>>>>>>>>>>>>>>>>>>> hints > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and table hints are > > >>>>>> orthogonal > > >>>>>>>> topics > > >>>>>>>>>>>>> that > > >>>>>>>>>>>>>> should be > > >>>>>>>>>>>>>>>>>>>> discussed > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> separately. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I share Jingsong's > > >>> opinion > > >>>>>>>> that we > > >>>>>>>>>>>> should > > >>>>>>>>>>>>>> not use planner > > >>>>>>>>>>>>>>>>>>>>> hints > > >>>>>>>>>>>>>>>>>>>>>>> for > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> passing connector > > >>>>>> properties. > > >>>>>>>> Planner > > >>>>>>>>>>>>>> hints should be > > >>>>>>>>>>>>>>>>>>>> optional > > >>>>>>>>>>>>>>>>>>>>>> at > > >>>>>>>>>>>>>>>>>>>>>>>> any > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time. They should > > >> not > > >>>>>> include > > >>>>>>>>>>> semantics > > >>>>>>>>>>>>>> but only affect > > >>>>>>>>>>>>>>>>>>>>>> execution > > >>>>>>>>>>>>>>>>>>>>>>>> time. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Connector > > >> properties > > >>> are an > > >>>>>>>> important > > >>>>>>>>>>>>> part > > >>>>>>>>>>>>>> of the query > > >>>>>>>>>>>>>>>>>>>>> itself. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Have you thought > > >>> about > > >>>>>> options > > >>>>>>>> such > > >>>>>>>>>>> as > > >>>>>>>>>>>>>> `SELECT * FROM t(k=v, > > >>>>>>>>>>>>>>>>>>>>>>> k=v)`? > > >>>>>>>>>>>>>>>>>>>>>>>> How > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are other vendors > > >>> deal with > > >>>>>>>> this > > >>>>>>>>>>>> problem? > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regards, > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Timo > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 09.03.20 10:37, > > >>>>>> Jingsong Li > > >>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Danny, +1 for > > >>> table > > >>>>>> hints, > > >>>>>>>>>>> thanks > > >>>>>>>>>>>>> for > > >>>>>>>>>>>>>> driving. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I took a look to > > >>> FLIP, > > >>>>>> most > > >>>>>>>> of > > >>>>>>>>>>>> content > > >>>>>>>>>>>>>> are talking about > > >>>>>>>>>>>>>>>>>>>>> query > > >>>>>>>>>>>>>>>>>>>>>>>> hints. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It is > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hard to > > >> discussion > > >>> and > > >>>>>>>> voting. So > > >>>>>>>>>>> +1 > > >>>>>>>>>>>> to > > >>>>>>>>>>>>>> split it as Jark > > >>>>>>>>>>>>>>>>>>>>> said. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Another thing is > > >>>>>>>> configuration that > > >>>>>>>>>>>>>> suitable to config with > > >>>>>>>>>>>>>>>>>>>>>> table > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> hints: > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "connector.path" > > >>> and > > >>>>>>>>>>>> "connector.topic", > > >>>>>>>>>>>>>> Are they really > > >>>>>>>>>>>>>>>>>>>>>> suitable > > >>>>>>>>>>>>>>>>>>>>>>>> for > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> table > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hints? Looks > > >> weird > > >>> to me. > > >>>>>>>> Because I > > >>>>>>>>>>>>>> think these properties > > >>>>>>>>>>>>>>>>>>>>> are > > >>>>>>>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> core of > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jingsong Lee > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 9, > > >>> 2020 at > > >>>>>> 5:30 > > >>>>>>>> PM Jark > > >>>>>>>>>>>> Wu > > >>>>>>>>>>>>> < > > >>>>>>>>>>>>>> imj...@gmail.com> > > >>>>>>>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Danny > > >> for > > >>>>>> starting > > >>>>>>>> the > > >>>>>>>>>>>>>> discussion. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 for this > > >>> feature. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we just > > >> focus > > >>> on the > > >>>>>>>> table > > >>>>>>>>>>> hints > > >>>>>>>>>>>>>> not the query hints in > > >>>>>>>>>>>>>>>>>>>>>> this > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> release, > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> could you split > > >>> the > > >>>>>> FLIP > > >>>>>>>> into two > > >>>>>>>>>>>>>> FLIPs? > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Because it's > > >>> hard to > > >>>>>> vote > > >>>>>>>> on > > >>>>>>>>>>>> partial > > >>>>>>>>>>>>>> part of a FLIP. You > > >>>>>>>>>>>>>>>>>>>> can > > >>>>>>>>>>>>>>>>>>>>>>> keep > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the table > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hints proposal > > >> in > > >>>>>> FLIP-113 > > >>>>>>>> and > > >>>>>>>>>>> move > > >>>>>>>>>>>>>> query hints into > > >>>>>>>>>>>>>>>>>>>> another > > >>>>>>>>>>>>>>>>>>>>>>> FLIP. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So that we can > > >>> focuse > > >>>>>> on > > >>>>>>>> the > > >>>>>>>>>>> table > > >>>>>>>>>>>>>> hints in the FLIP. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jark > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 9 Mar > > >>> 2020 at > > >>>>>>>> 17:14, > > >>>>>>>>>>> DONG, > > >>>>>>>>>>>>>> Weike < > > >>>>>>>>>>>>>>>>>>>>>>> kyled...@connect.hku.hk > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Danny, > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is a > > >> nice > > >>>>>> feature, > > >>>>>>>> +1. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> One thing I > > >> am > > >>>>>>>> interested in > > >>>>>>>>>>> but > > >>>>>>>>>>>>> not > > >>>>>>>>>>>>>> mentioned in the > > >>>>>>>>>>>>>>>>>>>>>> proposal > > >>>>>>>>>>>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> error > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> handling, as > > >>> it is > > >>>>>> quite > > >>>>>>>> common > > >>>>>>>>>>>> for > > >>>>>>>>>>>>>> users to write > > >>>>>>>>>>>>>>>>>>>>>>> inappropriate > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> hints in > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> SQL code, if > > >>> illegal > > >>>>>> or > > >>>>>>>> "bad" > > >>>>>>>>>>>> hints > > >>>>>>>>>>>>>> are given, would the > > >>>>>>>>>>>>>>>>>>>>>> system > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> simply > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ignore them > > >> or > > >>> throw > > >>>>>>>>>>> exceptions? > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks : ) > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Weike > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Mar > > >> 9, > > >>> 2020 > > >>>>>> at > > >>>>>>>> 5:02 PM > > >>>>>>>>>>>>> Danny > > >>>>>>>>>>>>>> Chan < > > >>>>>>>>>>>>>>>>>>>>>>> yuzhao....@gmail.com> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Note: > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we only > > >> plan > > >>> to > > >>>>>>>> support table > > >>>>>>>>>>>>>> hints in Flink release > > >>>>>>>>>>>>>>>>>>>> 1.11, > > >>>>>>>>>>>>>>>>>>>>>> so > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> please > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> focus > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mainly on > > >>> the table > > >>>>>>>> hints > > >>>>>>>>>>> part > > >>>>>>>>>>>>> and > > >>>>>>>>>>>>>> just ignore the > > >>>>>>>>>>>>>>>>>>>> planner > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> hints, sorry > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that > > >> mistake > > >>> ~ > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Danny Chan > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 在 2020年3月9日 > > >>> +0800 > > >>>>>>>>>>> PM4:36，Danny > > >>>>>>>>>>>>>> Chan < > > >>>>>>>>>>>>>>>>>>>> yuzhao....@gmail.com > > >>>>>>>>>>>>>>>>>>>>>>>> ，写道： > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, > > >>> fellows ~ > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would > > >>> like to > > >>>>>>>> propose the > > >>>>>>>>>>>>>> supports for SQL hints for > > >>>>>>>>>>>>>>>>>>>>> our > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Flink SQL. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We would > > >>> support > > >>>>>>>> hints > > >>>>>>>>>>> syntax > > >>>>>>>>>>>>> as > > >>>>>>>>>>>>>> following: > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> select > > >> /*+ > > >>>>>>>> NO_HASH_JOIN, > > >>>>>>>>>>>>>> RESOURCE(mem='128mb', > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> parallelism='24') */ > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> emp /*+ > > >>>>>> INDEX(idx1, > > >>>>>>>> idx2) > > >>>>>>>>>>> */ > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> join > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dept /*+ > > >>>>>>>>>>> PROPERTIES(k1='v1', > > >>>>>>>>>>>>>> k2='v2') */ > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> emp.deptno > > >>> = > > >>>>>>>> dept.deptno > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Basically > > >>> we > > >>>>>> would > > >>>>>>>> support > > >>>>>>>>>>>> both > > >>>>>>>>>>>>>> query hints(after the > > >>>>>>>>>>>>>>>>>>>>>> SELECT > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> keyword) > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and table > > >>>>>> hints(after > > >>>>>>>> the > > >>>>>>>>>>>>>> referenced table name), for > > >>>>>>>>>>>>>>>>>>>>> 1.11, > > >>>>>>>>>>>>>>>>>>>>>> we > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> plan to > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support > > >>> table hints > > >>>>>>>> with a > > >>>>>>>>>>> hint > > >>>>>>>>>>>>>> probably named > > >>>>>>>>>>>>>>>>>>>> PROPERTIES: > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> table_name > > >>> /*+ > > >>>>>>>>>>>>>> PROPERTIES(k1='v1', k2='v2') *+/ > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I am > > >>> looking > > >>>>>> forward > > >>>>>>>> to > > >>>>>>>>>>> your > > >>>>>>>>>>>>>> comments. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> You can > > >>> access > > >>>>>> the > > >>>>>>>> FLIP > > >>>>>>>>>>> here: > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>> > > >>>>>> > > >>> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+SQL+and+Planner+Hints > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Danny > > >> Chan > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>> > > >>>>> > > >>>>> > > >>>> > > >>> > > >> > > > > > > > >
