[VOTE] FLIP-188 Introduce Built-in Dynamic Table Storage

2021-11-09 Thread Jingsong Li
Hi everyone, Thanks for all the feedback so far. Based on the discussion[1] we seem to have consensus, so I would like to start a vote on FLIP-188 for which the FLIP has now also been updated[2]. The vote will last for at least 72 hours (Nov 13th 3:00 GMT) unless there is an objection or insuffic

Re: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage

2021-11-09 Thread Jingsong Li
Hi all, I have started the voting thread [1]. Please cast your vote there or ask additional questions here. [1] https://lists.apache.org/thread/v3fzx0p6n2jogn86sptzr30kr3yw37sq Best, Jingsong On Mon, Nov 1, 2021 at 5:41 PM Jingsong Li wrote: > > Hi Till, > > Thanks for your sugges

Re: [VOTE] FLIP-188 Introduce Built-in Dynamic Table Storage

2021-11-09 Thread Jingsong Li
Best, > Kurt > > > On Wed, Nov 10, 2021 at 10:52 AM Jingsong Li wrote: > > > Hi everyone, > > > > Thanks for all the feedback so far. Based on the discussion[1] we seem > > to have consensus, so I would like to start a vote on FLIP-188 for > > which the

Re: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage

2021-11-10 Thread Jingsong Li
is in place, so that the Pulsar dev > community may hack on a prototype implementation? > > I believe this is important for maintaining the perception that Flink > doesn't unduly favor Kafka. > > -Eron > > On Tue, Nov 9, 2021 at 6:53 PM Jingsong Li wrote: > > > Hi

Re: [DISCUSS] Update Policy for old releases

2021-11-11 Thread Jingsong Li
Thanks Piotr for starting this discussion. LTS looks better. +1 But I think we should pay attention to how to choose a long-term supported version. Before version planning, we need to carefully consider whether features should be placed in this version or in the next long-term supported version.

Re: [ANNOUNCE] New Apache Flink Committer - Yangze Guo

2021-11-11 Thread Jingsong Li
Congrats & well deserved! Best, Jingsong On Fri, Nov 12, 2021 at 10:28 AM Zakelly Lan wrote: > > Congrats & well deserved, Yangze Guo! > > Best, > Zakelly > > On Fri, Nov 12, 2021 at 10:11 AM Xintong Song wrote: > > > Hi everyone, > > > > On behalf of the PMC, I'm very happy to announce Yangze

Re: [ANNOUNCE] New Apache Flink Committer - Leonard Xu

2021-11-11 Thread Jingsong Li
Congrats & well deserved, Leonard! Thank you for your exciting contributions to Flink! Best, Jingsong On Fri, Nov 12, 2021 at 12:13 PM Jark Wu wrote: > > Hi everyone, > > On behalf of the PMC, I'm very happy to announce Leonard Xu as a new Flink > committer. > > Leonard has been a very active c

Re: [VOTE] FLIP-188 Introduce Built-in Dynamic Table Storage

2021-11-13 Thread Jingsong Li
;> again. > >>> If *the built-in storage could provide same consumer API as the commonly > >>> used message queues*, data copying may be avoided. So the built-in > >> dynamic > >>> table storage may be promoted faster in the streaming data warehouse &

Re: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage

2021-11-14 Thread Jingsong Li
Hi, Stephan and Timo, thanks very much for your replies. I try to reply to you one by one, split into multiple emails. First, according to my understanding, from the perspective of database (mysql) and storage, think about what streaming and batch are: - Batch is the snapshot, it contains the full

Re: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage

2021-11-15 Thread Jingsong Li
Hi Stephan, ### about PK: > (6) Different consistency guarantees with PK and without PK > (7) Eventual Consistency Mode vs. Faster Checkpoints Let's clarify one thing first: For sink with PK, Flink SQL cannot generate complete changes (without UPDATE_BEFORE). - For d

Re: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage

2021-11-15 Thread Jingsong Li
Hi Timo, > It would be great if we can add not only `Receive any type of changelog` but > also `Receive any type of datatype`. Nice, I think we can. > Please clarify whether the compact DDL is a synchronous or asynchrounous > operation in the API? So far all DDL was synchrounous. And only DML

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

2021-11-17 Thread Jingsong Li
Congratulations, Jing! Well deserved! On Wed, Nov 17, 2021 at 3:00 PM Lincoln Lee wrote: > > Congratulations, Jing! > > Best, > Lincoln Lee > > > Jing Zhang 于2021年11月17日周三 上午10:24写道: > > > Thanks to everyone. It's my honor to work in community with you all. > > > > Best, > > Jing Zhang > > > > Z

Re: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage

2021-11-19 Thread Jingsong Li
with PK would actually not be > correct, it would still contain some duplicates. > But that means we cannot expose a correct changelog to users (external > subscribers) in all cases? > > The PK behavior seems like something that needs some general improvement in > the SQL engine. &

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

2022-06-14 Thread Jingsong Li
Thanks everyone. It's great to be with you in the Flink community! Best, Jingsong On Wed, Jun 15, 2022 at 2:11 PM Yun Gao wrote: > > Congratulations, Jingsong! > > Best, > Yun Gao > > > -- > From:Jing Zhang > Send Time:2022 Jun. 1

Re: [ANNOUNCE] New Apache Flink Committers: Qingsheng Ren, Shengkai Fang

2022-06-20 Thread Jingsong Li
Congratulations Qingsheng and Shengkai. Well deserved. Best, Jingsong On Mon, Jun 20, 2022 at 3:57 PM Martijn Visser wrote: > > Congratulations to both of you, this is very much deserved! > > Op ma 20 jun. 2022 om 09:53 schreef Jark Wu : > > > Hi everyone, > > > > On behalf of the PMC, I'm very

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

2022-06-21 Thread Jingsong Li
song, > >>>> > >>>> Thanks for your comments! > >>>> > >>>>> AllCache definition is not flexible, for example, PartialCache can use > >>>> any custom storage, while the AllCache can not, AllCache can also be > >>

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

2022-06-22 Thread Jingsong Li
uce another layer and expose too much since this FLIP is already a huge > one with bunch of classes and interfaces. > > Best, > Qingsheng > > > On Jun 22, 2022, at 11:16, Jingsong Li wrote: > > > > Thanks Qingsheng and all. > > > > I like this design. >

Re: Re: [VOTE] FLIP-221: Abstraction for lookup source and metric

2022-06-23 Thread Jingsong Li
+1 Best, Jingsong On Fri, Jun 24, 2022 at 9:30 AM zst...@163.com wrote: > > Thanks Qingsheng for driving this. > > > > > +1 (non-binding) > > > > > Best regards, > > Yuan > > > > > At 2022-06-23 15:44:55, "Martijn Visser" wrote: > >Great work on the FLIP. > > > >+1 (binding) > > > >Op do 23 jun

Re: [VOTE] FLIP-240: Introduce "ANALYZE TABLE" Syntax

2022-06-27 Thread Jingsong Li
+1 Best, Jingsong On Fri, Jun 24, 2022 at 11:08 PM godfrey he wrote: > > Hi everyone, > > Thanks for all the feedback so far. Based on the discussion[1] we seem > to have consensus, so I would like to start a vote on FLIP-240 for > which the FLIP has now also been updated[2]. > > The vote will l

Re: [VOTE] Release 1.15.1, release candidate #1

2022-06-30 Thread Jingsong Li
Hi David, Thanks for creating this RC. -1 We found an incompatible modification in 1.15.0 [1] I think we should fix it. [1] https://issues.apache.org/jira/browse/FLINK-28322 Best, Jingsong On Tue, Jun 28, 2022 at 8:45 PM Robert Metzger wrote: > > +1 (binding) > > - staging repo contents look

Re: [VOTE] Release 1.15.1, release candidate #1

2022-07-03 Thread Jingsong Li
dannycran...@apache.org>: > > > Hello all, > > > > -1 > > > > The Kinesis Data Streams consumer does not work with Stop With Savepoint > > [1]. We are planning to have a fix ready to merge tomorrow and would > > appreciate getting this in 1.15.1. > &

Re: [VOTE] Release 1.15.1, release candidate #1

2022-07-03 Thread Jingsong Li
k > 1.15.2. Still I stand to my point, that we shouldn't abort the current > release. > > --- > > +1 (binding) > > * checked release blog post > * verified signatures and checksums > * all artifacts present > > Best, > > Konstantin > > >

Re: [DISCUSS] FLIP-248: Introduce dynamic partition pruning

2022-07-11 Thread Jingsong Li
Thanks Godfrey for driving. I like this FLIP. We can restrict this capability to more than just partitions. Here are some inputs from Lake Storage. The format of the splits generated by Lake Storage is roughly as follows: Split { Path filePath; Statistics[] fieldStats; } Stats contain the

Re: [DISCUSS] FLIP-247 Bulk fetch of table and column statistics for given partitions

2022-07-14 Thread Jingsong Li
Thanks for starting this discussion. Have we considered introducing a listPartitionWithStats() in Catalog? Best, Jingsong On Fri, Jul 15, 2022 at 10:08 AM Jark Wu wrote: > > Hi Jing, > > Thanks for starting this discussion. The bulk fetch is a great improvement > for the optimizer. > The FLIP l

Re: [DISCUSS] FLIP-247 Bulk fetch of table and column statistics for given partitions

2022-07-19 Thread Jingsong Li
nt, that > calculates statistics, uses the "list" naming convention. > > Best regards, > Jing > > On Fri, Jul 15, 2022 at 5:38 AM Jingsong Li wrote: > > > Thanks for starting this discussion. > > > > Have we considered introducing a listPartitionWithS

[ANNOUNCE] Table Store release-0.2 branch cut

2022-07-19 Thread Jingsong Li
Hi Flink devs! The version on the main branch has been updated to 0.3-SNAPSHOT. The release-0.2 branch has been forked from main: https://github.com/apache/flink-table-store/tree/release-0.2 Documentation: https://nightlies.apache.org/flink/flink-table-store-docs-release-0.2/ The most important

Re: [DISCUSS] FLIP-247 Bulk fetch of table and column statistics for given partitions

2022-07-20 Thread Jingsong Li
see my answers and further questions below. > > Best regards, > Jing > > On Wed, Jul 20, 2022 at 4:28 AM Jingsong Li wrote: > > > Hi Jing, > > > > I understand that the statistics for partitions are currently only > > used by Hive, so we can look at the Hive im

Re: [VOTE] FLIP-247 Bulk fetch of table and column statistics for given partitions

2022-07-26 Thread Jingsong Li
+1 Best, Jingsong On Tue, Jul 26, 2022 at 10:11 AM godfrey he wrote: > > +1 > > Best, > Godfrey > > Jark Wu 于2022年7月25日周一 17:23写道: > > > > +1 (binding) > > > > Best, > > Jark > > > > On Mon, 25 Jul 2022 at 15:10, Jing Ge wrote: > > > > > Hi all, > > > > > > Many thanks for all your feedback. B

Re: [DISCUSS] FLIP-248: Introduce dynamic partition pruning

2022-07-26 Thread Jingsong Li
runing > > > > > [2] FLIP-245: > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-245%3A+Source+Supports+Speculative+Execution+For+Batch+Job > > > > > > > > > > > > > > > Jark Wu 于2022年7月12日周二 13:16写道: > > > > > > > > > > > I agree with Jingsong. DPP

Re: [VOTE] FLIP-248: Introduce dynamic partition pruning

2022-07-27 Thread Jingsong Li
+1 On Wed, Jul 27, 2022 at 3:30 PM Jark Wu wrote: > > +1 (binding) > > Best, > Jark > > On Wed, 27 Jul 2022 at 13:34, Yun Gao wrote: > > > +1 (binding) > > > > Thanks for proposing the FLIP! > > > > Best, > > Yun Gao > > > > > > --

Re: [Phishing Risk] [External] [DISCUSS] FLIP-255 Introduce pre-aggregated merge to Table Store

2022-08-01 Thread Jingsong Li
Thanks Nathan for starting this discussion. This [1] is a very good requirement to build the materialized view on Flink Table Store. ## Aggregate Functions For 'aggregate-function' = '{sum_field1:sum,max_field2:max}'. Do you refer to any other systems? For us at Flink, a viable approach is someth

[VOTE] Apache Flink Table Store 0.2.0, release candidate #1

2022-08-02 Thread Jingsong Li
Hi everyone, Please review and vote on the release candidate #1 for the version 0.2.0 of Apache Flink Table Store, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) **Release Overview** As an overview, the release consists of the follo

Re: [Phishing Risk] [External] [DISCUSS] FLIP-255 Introduce pre-aggregated merge to Table Store

2022-08-03 Thread Jingsong Li
Hi Nathan, Thanks for the reply. ## Aggregate functions I think `'fields.sum_field1.function'='sum'` is a safe one, I am more worried about using complex json, which will affect the sql parsing. ## Supported functions I think it is better to be consistent with Flink Table [1] and PostgreSQL [2

Re: [VOTE] Apache Flink Table Store 0.2.0, release candidate #1

2022-08-14 Thread Jingsong Li
-1 https://issues.apache.org/jira/browse/FLINK-28903 This jira causes hive to not work properly with ssl on, which is a relatively common configuration. Best, Jingsong On Wed, Aug 3, 2022 at 2:14 PM Jingsong Li wrote: > > Hi everyone, > > Please review and vote on the release cand

[VOTE] Apache Flink Table Store 0.2.0, release candidate #2

2022-08-17 Thread Jingsong Li
Hi everyone, Please review and vote on the release candidate #2 for the version 0.2.0 of Apache Flink Table Store, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) **Release Overview** As an overview, the release consists of the follo

[VOTE] Apache Flink Table Store 0.2.0, release candidate #3

2022-08-24 Thread Jingsong Li
Hi everyone, Please review and vote on the release candidate #3 for the version 0.2.0 of Apache Flink Table Store, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) **Release Overview** As an overview, the release consists of the follo

Re: [VOTE] Apache Flink Table Store 0.2.0, release candidate #3

2022-08-28 Thread Jingsong Li
nt > > > > and listed the check items as follows: > > > > > > > > * Compiled the sources and built the source distribution - PASSED > > > > * Ran through Quick Start Guide - PASSED > > > > * Checked Spark 2.3.4&3.3.0 reade

Re: [VOTE] Apache Flink Table Store 0.2.0, release candidate #3

2022-08-28 Thread Jingsong Li
Thanks everyone, We have 3 binding and 2 non-binding now. This voting was closed. Best, Jingsong On Mon, Aug 29, 2022 at 10:01 AM Jingsong Li wrote: > > +1 (binding) > > - check the source code > - check the signatures and sums > - run the quick start > > Best, >

[RESULT][VOTE] Apache Flink Table Store 0.2.0, release candidate #3

2022-08-28 Thread Jingsong Li
I'm happy to announce that we have unanimously approved this release. There are 5 approving votes, 3 of which are binding: * Nicholas Jiang (non-binding) * Yu Li (binding) * Jark Wu (binding) * Jane Chan (non-binding) * Jingsong Lee (binding) There are no disapproving votes. Thank you for verify

[ANNOUNCE] Apache Flink Table Store 0.2.0 released

2022-08-28 Thread Jingsong Li
The Apache Flink community is very happy to announce the release of Apache Flink Table Store 0.2.0. Apache Flink Table Store is a unified storage to build dynamic tables for both streaming and batch processing in Flink, supporting high-speed data ingestion and timely data query. Please check out

Re: [DISCUSS] Releasing Flink 1.14.6

2022-09-06 Thread Jingsong Li
+1 for 1.14.6 Thanks Xingbo for driving. Best, Jingsong On Tue, Sep 6, 2022 at 4:42 PM Xingbo Huang wrote: > > Hi all, > > I would like to start discussing releasing Flink 1.14.6. > > It has already been almost three months since we released 1.14.5. There are > currently 35 tickets[1] and 33 co

Re: [VOTE] FLIP-258 Guarantee binary compatibility for Public/-Evolving APIs between patch releases​

2022-09-06 Thread Jingsong Li
+1 On Tue, Sep 6, 2022 at 7:11 PM Yu Li wrote: > > +1 > > Thanks for the efforts, Chesnay > > Best Regards, > Yu > > > On Tue, 6 Sept 2022 at 18:17, Martijn Visser > wrote: > > > +1 (binding) > > > > Op di 6 sep. 2022 om 11:59 schreef Xingbo Huang : > > > > > Thanks Chesnay for driving this, > >

Re: [VOTE] Release 1.14.6, release candidate #1

2022-09-08 Thread Jingsong Li
-1 shasum -a 512 flink-1.14.6-bin-scala_2.11.tgz 33d55347e93369164b166a54cd1625d617f3df92f5f788d87778c36bb7b2b7bc64d565d87f25c4fcb5f3c70ffa96fc89f9a774143459a6d810667534db1cd695 flink-1.14.6-bin-scala_2.11.tgz cat flink-1.14.6-bin-scala_2.11.tgz.sha512 a2b1cac32778272c1741befa79832056d07977f9b71

Re: [ANNOUNCE] New Apache Flink PMC Member - Martijn Visser

2022-09-12 Thread Jingsong Li
Congrats, Martijn! Best, Jingsong On Tue, Sep 13, 2022 at 10:48 AM Yang Wang wrote: > > Congrats, Martijn! > > Best, > Yang > > Lijie Wang 于2022年9月13日周二 10:10写道: > > > Congratulations, Martijn! > > > > Best, > > Lijie > > > > yuxia 于2022年9月13日周二 09:52写道: > > > > > Congrats, Martijn! > > > > >

Re: [DISCUSS] FLIP-260: Expose Finish Method For UserDefinedFunction

2022-09-18 Thread Jingsong Li
+1 to add `finish()` method to `TableFunction` only. Can we use `endInput` just like `BoundedOneInput`? Best, Jingsong On Fri, Sep 16, 2022 at 11:54 PM Lincoln Lee wrote: > > Hi Dawid, Piotr, >Agree with you that add finish() method to `TableFunction` only. Other > `UserDefinedFunction`s (`

Re: [VOTE] FLIP-254 Redis Streams connector

2022-09-19 Thread Jingsong Li
+1 (binding) Best, Jingsong On Mon, Sep 19, 2022 at 2:20 PM Yu Li wrote: > > +1 (binding) > > Thanks for the clarification Martijn. I'd suggest considering adding a note > in the FLIP document (since the proposed repository name for this connector > is "flink-connector-redis") or changing the re

Re: [VOTE] Release 1.14.6, release candidate #2

2022-09-19 Thread Jingsong Li
+1 (binding) - Check if checksums and GPG files match the corresponding release files - Verify that the source archives do not contains any binaries - Build the source with Maven - Check that POM files point to the same version - Started session cluster and SQL-Client Best, Jingsong On Thu, Sep

Re: [VOTE] FLIP-260: Expose Finish Method For TableFunction

2022-09-22 Thread Jingsong Li
+1 (binding) Best, Jingsong On Thu, Sep 22, 2022 at 9:45 PM Timo Walther wrote: > > +1 (binding) > > Please make also sure to update ExpressionReducer during implementation. > This is not mentioned in the FLIP yet. > > Thanks, > Timo > > On 22.09.22 10:41, Piotr Nowojski wrote: > > +1 (binding)

[VOTE] Apache Flink Table Store 0.2.1, release candidate #1

2022-09-26 Thread Jingsong Li
Hi everyone, Please review and vote on the release candidate #1 for the version 0.2.1 of Apache Flink Table Store, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) **Release Overview** As an overview, the release consists of the follo

Re: [VOTE] Apache Flink Table Store 0.2.1, release candidate #1

2022-09-28 Thread Jingsong Li
-1 https://issues.apache.org/jira/browse/FLINK-29345 brings orc bugs. The orc writer has memory management and needs to be initialized in the writing thread to avoid thread safety conflicts caused by memory management. Best, Jingsong On Tue, Sep 27, 2022 at 11:00 AM Jingsong Li wrote: > &

Re: [ANNOUNCE] Apache Flink 1.14.6 released

2022-09-28 Thread Jingsong Li
Thanks Xingbo for releasing it. Best, Jingsong On Wed, Sep 28, 2022 at 10:52 AM Xingbo Huang wrote: > > The Apache Flink community is very happy to announce the release of Apache > Flink 1.14.6, which is the fifth bugfix release for the Apache Flink 1.14 > series. > > Apache Flink® is an open-

[VOTE] Apache Flink Table Store 0.2.1, release candidate #2

2022-10-07 Thread Jingsong Li
Hi everyone, Please review and vote on the release candidate #2 for the version 0.2.1 of Apache Flink Table Store, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) **Release Overview** As an overview, the release consists of the follo

Re: [Discuss]- Donate Iceberg Flink Connector

2022-10-09 Thread Jingsong Li
Thanks Abid for driving. +1 for this. Can you open the permissions for https://docs.google.com/document/d/1WC8xkPiVdwtsKL2VSPAUgzm9EjrPs8ZRjEtcwv93ISI/edit?usp=sharing ? Best, Jingsong On Mon, Oct 10, 2022 at 9:22 AM Abid Mohammed wrote: > > Hi, > > I would like to start a discussion about con

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

2022-10-09 Thread Jingsong Li
Thanks for driving, Qingsheng. +1 for reverting sink metric name. We often forget that metric is also one of the important APIs. +1 for releasing 1.15.3 to fix this. Best, Jingsong On Sun, Oct 9, 2022 at 11:35 PM Becket Qin wrote: > > Thanks for raising the discussion, Qingsheng, > > +1 on re

Re: Edit permissions to Flink wiki

2022-10-09 Thread Jingsong Li
Hi Abid, I just added the permissions for Abid Mohammed (abmo-x). Best, Jingsong On Mon, Oct 10, 2022 at 9:57 AM Abid Mohammed wrote: > > Hi, > > Can someone please grant me edit permissions so that I can create a new FLIP. > > Thanks > Abid

Re: [VOTE] Apache Flink Table Store 0.2.1, release candidate #2

2022-10-12 Thread Jingsong Li
ck start: *OK* > > * Verified with both flink 1.14.5 and 1.15.1 > > - Checked the website updates: *OK* > > * Minor: left some comments in the PR, please check > > > > Thanks for driving this release, Jingsong! > > > > Best Regards, > > Yu > >

Re: [VOTE] Apache Flink Table Store 0.2.1, release candidate #2

2022-10-12 Thread Jingsong Li
+1 (binding) - Checked sums and signatures: *OK* - Checked the jars in the staging repo: *OK* - Checked source distribution doesn't include binaries: *OK* - Maven clean install from source: *OK* Best, Jingsong On Wed, Oct 12, 2022 at 6:10 PM Jingsong Li wrote: > > Thanks Yu, >

Re: [VOTE] Remove HCatalog connector

2022-10-12 Thread Jingsong Li
+1 Thanks for driving. Best, Jingsong On Thu, Oct 13, 2022 at 3:46 AM Martijn Visser wrote: > > Hi everyone, > > Since no comments were made, I'm opening a vote to remove the HCatalog > connector [1] > > The voting period will be open at least 72hrs. > > Best regards, > > Martijn > > [1] > http

[RESULT][VOTE] Apache Flink Table Store 0.2.1, release candidate #2

2022-10-12 Thread Jingsong Li
I'm happy to announce that we have unanimously approved this release. There are 3 approving votes, 3 of which are binding: * Yu Li (binding) * Jark Wu (binding) * Jingsong Lee (binding) There are no disapproving votes. Thank you for verifying the release candidate. I will now proceed to finalize

Re: [DISCUSS] FLIP-273: Improve Catalog API to Support ALTER TABLE syntax

2022-11-21 Thread Jingsong Li
Thanks Shengkai for the proposal! This should be very useful! > In some cases, users might need to modify the nested column. Our proposal is > able to express the idea by introducing a new class named. My concern is that the connector needs to be implemented multiple times. Can we define classe

Re: [DISCUSS] FLIP-273: Improve Catalog API to Support ALTER TABLE syntax

2022-11-23 Thread Jingsong Li
Thanks for the update! Looks good to me. Best, Jingsong On Thu, Nov 24, 2022 at 3:07 PM Shengkai Fang wrote: > > Hi, Jingsong. > > Thanks for your feedback! > > > Can we define classes at once so that the connector can be fully > implemented, but we will not pass changes of the nested column. >

Re: [ANNOUNCE] New Apache Flink PMC Members - Godfrey He, Xingbo Huang

2022-11-24 Thread Jingsong Li
Congrats, Godfrey and Xingbo! Best, Jingsong On Thu, Nov 24, 2022 at 4:03 PM Sergey Nuyanzin wrote: > > Congrats to both! > > On Thu, Nov 24, 2022 at 8:59 AM Yun Gao > wrote: > > > Congratulations Godfrey and Xingbo! > > Best, > > Yun Gao > >

Re: [DISCUSS] Hierarchies in ConfigOption

2020-04-29 Thread Jingsong Li
Thanks Timo for staring the discussion. I am +1 for "format: 'json'". Take a look to Dawid's yaml case: connector: 'filesystem' path: '...' format: 'json' format: option1: '...' option2: '...' option3: '...' Is this work? According to my understanding, 'format' key is the attribute o

Re: [DISCUSS] Hierarchies in ConfigOption

2020-04-29 Thread Jingsong Li
n' value.format.option: option1: '...' option2: '...' Best, Jingsong Lee On Thu, Apr 30, 2020 at 10:16 AM Jingsong Li wrote: > Thanks Timo for staring the discussion. > > I am +1 for "format: 'json'". > Take a look to Dawid's y

Re: [DISCUSS] Introduce TableFactory for StatefulSequenceSource

2020-05-05 Thread Jingsong Li
a generating table source to > an > > > ecosystem project promoted on flink-packages.org. I think this has a > lot > > > of > > > potential (e.g. in combination with Java Faker [1]), but it would > > probably > > > be better served in a small separately maintained re

Re: [DISCUSS]Refactor flink-jdbc connector structure

2020-05-05 Thread Jingsong Li
Thanks Leonard for starting this discussion. +1 from my side. > JDBCOutputFormat Already deprecated, so let it in old package. > JDBCInputFormat Not only input format, but also ParameterValuesProvider, looks good to me for keeping them in old package. You can wait until the next time you refacto

Re: [DISCUSS] Hierarchies in ConfigOption

2020-05-06 Thread Jingsong Li
Hi, +1 to: format = parquet parquet.compression = ... parquet.block.size = ... parquet.page.size = ... For the formats like parquet and orc, Not just Flink itself, but this way also let Flink keys compatible with the property keys of Hadoop / Hive / Spark. And like Jark said, this way works for

Re: UpsertStreamTableSink vs OverwritableTableSink

2020-05-06 Thread Jingsong Li
Hi, Insert overwrite comes from Batch SQL in Hive. It means overwriting whole table/partition instead of overwriting per key. So if "insert overwrite kudu_table", should delete whole table in kudu first, and then insert new data to the table in kudu. The same semantics should be used in streaming

Re: [DISCUSS] Releasing "fat" and "slim" Flink distributions

2020-05-12 Thread Jingsong Li
t; > >>>>>> 2. I think there is also the problem that the matrix of possible > >>>>>> combinations that can be useful is already big. Do we want to have a > >>>>>> distribution for: > >>>>>> > >>>>>&g

Re: Reading parquet from S3

2020-05-13 Thread Jingsong Li
Hi Kirill, For DataSet API, yes, we have "ParquetRowInputFormat", "ParquetPojoInputFormat" and "ParquetMapInputFormat" [1]. For file system parquet files in table: - Before 1.11, you can register a ParquetTableSource to batch table environment with legacy planner. - In 1.11, you can create file s

Re: [DISCUSS] Stability guarantees for @PublicEvolving classes

2020-05-14 Thread Jingsong Li
Thanks Till for starting this discussion. +1 for enabling the japicmp-maven-plugin for @PublicEvolving for bug fix releases. Bug fix should just be user imperceptible bug fix. Should not affect API and binary compatibility. And even PublicEvolving api change for "y" release, we should expose it i

Re: [DISCUSS] Exact feature freeze date

2020-05-15 Thread Jingsong Li
+1 for Monday morning. Best, Jingsong Lee On Fri, May 15, 2020 at 8:45 PM Till Rohrmann wrote: > +1 from my side extend the feature freeze until Monday morning. > > Cheers, > Till > > On Fri, May 15, 2020 at 2:04 PM Robert Metzger > wrote: > > > I'm okay, but I would suggest to agree on a time

Re: [VOTE] Guarantee that @PublicEvolving classes are API and binary compatible across bug fix releases (x.y.u and x.y.v)

2020-05-15 Thread Jingsong Li
+1 (non-binding) Best, Jingsong Lee On Fri, May 15, 2020 at 9:26 PM Thomas Weise wrote: > +1 > > > On Fri, May 15, 2020 at 6:15 AM Till Rohrmann > wrote: > > > Dear community, > > > > with reference to the dev ML thread about guaranteeing API and binary > > compatibility for @PublicEvolving cl

Re: [DISCUSS] FLIP-84 Feedback Summary

2020-05-17 Thread Jingsong Li
+1 Best, Jingsong Lee On Sun, May 17, 2020 at 3:42 PM Kurt Young wrote: > +1 from my side. > > Best, > Kurt > > > On Sun, May 17, 2020 at 12:41 PM godfrey he wrote: > > > hi everyone, > > > > I would like to bring up another topic about the return value of > > TableResult#collect() method. > >

Re: [DISCUSS] Semantics of our JIRA fields

2020-05-25 Thread Jingsong Li
+1 for the proposal. It makes me clearer. Best, Jingsong Lee On Mon, May 25, 2020 at 2:51 PM Zhijiang wrote: > Thanks for launching this discussion and giving so detailed infos, > Robert! +1 on my side for the proposal. > > For "Affects Version", I previously thought it was only for the alread

Re: [DISCUSS] Releasing "fat" and "slim" Flink distributions

2020-06-04 Thread Jingsong Li
Jingsong Li wrote: > Thanks for your discussion. > > Sorry to start discussing another thing: > > The biggest problem I see is the variety of problems caused by users' lack > of format dependency. > As Aljoscha said, these three formats are very small and no third party

Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song

2020-06-04 Thread Jingsong Li
Congratulations Xintong, well deserved! Best, Jingsong Lee On Fri, Jun 5, 2020 at 2:00 PM Yang Wang wrote: > Congratulations, Xintong. > Well deserved. > > Best, > Yang > > Weihua Hu 于2020年6月5日周五 下午1:57写道: > > > Congratulations! > > > > Best > > Weihua Hu > > > > > 2020年6月5日 13:50,Xingbo Huang

Re: [DISCUSS] Releasing "fat" and "slim" Flink distributions

2020-06-07 Thread Jingsong Li
年6月5日周五 下午2:31写道: > > > > > >> +1, at least, we should keep an out of the box SQL-CLI, it’s very poor > > >> experience to add such required format jars for SQL users. > > >> > > >> Best, > > >> Danny Chan > > >> 在 20

Re: [ANNOUNCE] New Flink Committer: Benchao Li

2020-06-08 Thread Jingsong Li
Congratulations, Benchao. Well deserved! Best, Jingsong Lee On Tue, Jun 9, 2020 at 2:13 PM Forward Xu wrote: > Congratulations, Benchao > > Best, > Forward > > Jark Wu 于2020年6月9日周二 下午2:10写道: > > > Hi everyone, > > > > On behalf of the PMC, I'm very happy to announce Benchao Li as a new > Apach

Re: [DISCUSS] Update our EditorConfig file

2020-06-10 Thread Jingsong Li
+1 looks more friendly to Flink newbies. Best, Jingsong Lee On Wed, Jun 10, 2020 at 8:38 PM Aljoscha Krettek wrote: > Hi, > > is anyone actually using our .editorconfig file? IntelliJ has a plugin > for this that is actually quite powerful. > > I managed to write a .editorconfig file that I qui

Re: [ANNOUNCE] Yu Li is now part of the Flink PMC

2020-06-16 Thread Jingsong Li
Congratulations Yu, well deserved! Best, Jingsong On Wed, Jun 17, 2020 at 1:42 PM Yuan Mei wrote: > Congrats, Yu! > > GXGX & well deserved!! > > Best Regards, > > Yuan > > On Wed, Jun 17, 2020 at 9:15 AM jincheng sun > wrote: > >> Hi all, >> >> On behalf of the Flink PMC, I'm happy to announce

Re: [ANNOUNCE] Apache Flink 1.11.0, release candidate #2

2020-06-17 Thread Jingsong Li
Hi Piotr and Zhijiang, I merged three commits in 1 hour ago: one blocker: 0dafcf8792220dbe5b77544261f726a566b054f5 two critical issues: 1830c1c47b8a985ec328a7332e92d21433c0a4df 80fa0f5c5b8600f4b386487f267bde80b882bd07 I have synced with Zhijiang, they were merged after RC2 branch cutting. So RC2 d

[DISCUSS] Sql-client lack support for new features

2020-06-18 Thread Jingsong Li
Hi all, I'd like to start a discussion for the new features lacking support of Sql-client. I've seen the new DDL syntax that SQL client lacks support for many times. For every new DDL syntax, we need to add support in sql-client. Add a corresponding SqlCommand in sql-client, otherwise this DDL is

Re: [DISCUSS] [FLINK-16824] Creating Temporal Table Function via DDL

2020-06-27 Thread Jingsong Li
Thanks for your discussion. Looks like the problem is supporting the versioned temporal table for the changelog source. I want to share more of my thoughts: When I think about changelog sources, I treat it as a view like: "CREATE VIEW changelog_table AS SELECT ... FROM origin_table GROUP BY ..."

Re: [DISCUSS] Sql-client lack support for new features

2020-07-05 Thread Jingsong Li
, 18 Jun 2020 at 15:40, Rui Li wrote: > > > > > Thanks Jingsong for bringing up the discussion. Perhaps we can do both > #1 > > > and #2? I like the idea that SQL Client should just forward SQL > > statements > > > to TableEnvironment. IIUC, with Tabl

Re: [VOTE] Release 1.11.0, release candidate #4

2020-07-05 Thread Jingsong Li
+1 (non-binding) - verified signature and checksum - build from source - checked webui and log sanity - played with filesystem and new connectors - played with Hive connector Best, Jingsonga On Mon, Jul 6, 2020 at 9:50 AM Xintong Song wrote: > +1 (non-binding) > > - verified signature and chec

Re: [ANNOUNCE] New PMC member: Piotr Nowojski

2020-07-06 Thread Jingsong Li
Congratulations Piotr! Well deserved! Best, Jingsong On Tue, Jul 7, 2020 at 9:37 AM Xintong Song wrote: > Congratulations Piotr~! > > Thank you~ > > Xintong Song > > > > On Tue, Jul 7, 2020 at 9:35 AM Zhu Zhu wrote: > > > Congratulations! > > > > > > Leonard Xu 于2020年7月7日周二 上午9:16写道: > > > >

Re: [ANNOUNCE] Apache Flink 1.11.0 released

2020-07-07 Thread Jingsong Li
Congratulations! Thanks Zhijiang and Piotr as release managers, and thanks everyone. Best, Jingsong On Wed, Jul 8, 2020 at 10:51 AM chaojianok wrote: > Congratulations! > > Very happy to make some contributions to Flink! > > > > > > At 2020-07-07 22:06:05, "Zhijiang" wrote: > > The Apache Fli

Re: [DISCUSS] Releasing Flink 1.11.1 soon?

2020-07-09 Thread Jingsong Li
FLINK-18461 is really a blocker for the CDC feature. So +1 for releasing Flink 1.11.1 soon. Best, Jingsong On Thu, Jul 9, 2020 at 5:34 PM jincheng sun wrote: > Thanks for bring up this discussion Jark. > +1, looking forward the first bugfix version of Flink 1.11. > > Best, > Jincheng > > Dian

Re: [DISCUSS] FLIP-129: Refactor Descriptor API to register connector in Table API

2020-07-16 Thread Jingsong Li
Thanks for the discussion. Descriptor lacks the watermark and the computed column is too long. 1) +1 for just `column(...)` 2) +1 for being consistent with Table API, the Java Table API should be Expression DSL. We don't need pure string support, users should just use DDL instead. I think this i

Re: [ANNOUNCE] Apache Flink 1.11.1 released

2020-07-22 Thread Jingsong Li
Thanks for being the release manager for the 1.11.1 release, Dian. Best, Jingsong On Thu, Jul 23, 2020 at 10:12 AM Zhijiang wrote: > Thanks for being the release manager and the efficient work, Dian! > > Best, > Zhijiang > > -- > F

Re: [DISCUSS] FLIP-129: Refactor Descriptor API to register connector in Table API

2020-07-24 Thread Jingsong Li
gt; `TableSourceTable#tableIdentifier` which is used to calculate the digest. > > This requires that the registered table must have an unique identifier. > The > > old `TableSourceQueryOperation` will also generate the identifier > according > > to the hashcode of the TableSourc

Re: [VOTE] FLIP-129: Refactor Descriptor API to register connector in Table API

2020-07-28 Thread Jingsong Li
+1 Thanks Jark for driving this. On Mon, Jul 27, 2020 at 4:14 PM Leonard Xu wrote: > Thanks Jark > > +1(non-binding) > > Best > Leonard > > 在 2020年7月27日,15:32,jincheng sun 写道: > > > > +1(binding) > > > > Best, > > Jincheng > > > > > > Jark Wu 于2020年7月27日周一 上午10:01写道: > > > >> +1 (binding) > >

Re: [DISCUSS] Align the semantic of returning null from InputFormat#nextRecord

2020-07-29 Thread Jingsong Li
Hi Benchao, My understanding is that #1 treats null as the end of input. This means we should try our best to avoid returning null before the end of input in the implementation of `InputFormat`. Even if we have to return null, we have to return it at the end of input. I think this is the most safe

[DISCUSS] Introduce SupportsParallelismReport and SupportsStatisticsReport for Hive and Filesystem

2020-07-29 Thread Jingsong Li
Hi all, ## SupportsParallelismReport Now that FLIP-95 [1] is ready, only Hive and Filesystem are still using the old interfaces. We are considering migrating to the new interface. However, one problem is that in the old interface implementation, connectors infer parallelism by itself instead of

Re: [DISCUSS] Align the semantic of returning null from InputFormat#nextRecord

2020-07-29 Thread Jingsong Li
o it's > common that we encounters dirty > data and want to skip it like in DeserializationSchema. > That's why I prefer #2. Anyway, If we are gonna to deprecate InputFormat in > the near future, I think there > is no need to change this in the community. > > > Jings

Re: [DISCUSS] Introduce SupportsParallelismReport and SupportsStatisticsReport for Hive and Filesystem

2020-07-30 Thread Jingsong Li
n is what the oldStats parameter is used for? > > > > 3) Internal or Public. I don't think we should mark them as internal. > They > > are currently only used by internal > > connectors doesn't mean this interface should be internal. I can imagine > &

Re: [DISCUSS] Introduce SupportsParallelismReport and SupportsStatisticsReport for Hive and Filesystem

2020-08-04 Thread Jingsong Li
lism setting for the source operator, or do you want to let the > source operator set the parallelism? > > Best, > Kurt > > > On Fri, Jul 31, 2020 at 1:43 PM Jingsong Li > wrote: > > > Hi, thanks for your responses. > > > > To Benchao: > > > >

[DISCUSS] Introduce partitioning strategies to Table/SQL

2020-08-24 Thread Jingsong Li
Hi all, ## Motivation FLIP-63 [1] introduced initial support for PARTITIONED BY clause to an extent that let us support Hive's partitioning. But this partition definition is completely specific to Hive/File systems, with the continuous development of the system, there are new requirements: - FLI

Re: [DISCUSS] Whether catalog table factory should be used for temporary tables

2020-08-25 Thread Jingsong Li
Hi Dawid, But the temporary table does not belong to Catalog, actually Catalog doesn't know the existence of the temporary table. Let the table factory of catalog to create source/sink sounds a little sudden. If we want to make temporary tables belong to Catalog, I think we need to involve catalo

<    1   2   3   4   5   6   7   >