[jira] [Created] (FLINK-19728) YARN tests failed with "RM uses DefaultResourceCalculator which used only memory as resource-type but invalid resource-types specified"

Dian Fu created FLINK-19728: --- Summary: YARN tests failed with "RM uses DefaultResourceCalculator which used only memory as resource-type but invalid resource-types specified" Key: FLINK-19728 URL: https://issues.apache.

[jira] [Created] (FLINK-19727) Implement ParallelismProvider for sink in blink planner

Jingsong Lee created FLINK-19727: Summary: Implement ParallelismProvider for sink in blink planner Key: FLINK-19727 URL: https://issues.apache.org/jira/browse/FLINK-19727 Project: Flink Issue

[jira] [Created] (FLINK-19726) Implement new providers for blink planner

Jingsong Lee created FLINK-19726: Summary: Implement new providers for blink planner Key: FLINK-19726 URL: https://issues.apache.org/jira/browse/FLINK-19726 Project: Flink Issue Type: Sub-tas

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

Hi Konstantin, Thanks for your reply. > It uses the "kafka" connector and does not specify a primary key. The dimensional table `users` is a ktable connector and we can specify the pk on the KTable. > Will it possible to use a "ktable" as a dimensional table in FLIP-132 Yes. We can specify the w

[jira] [Created] (FLINK-19725) DataStreamTests.test_key_by_on_connect_stream is failed on Azure

Jark Wu created FLINK-19725: --- Summary: DataStreamTests.test_key_by_on_connect_stream is failed on Azure Key: FLINK-19725 URL: https://issues.apache.org/jira/browse/FLINK-19725 Project: Flink Issue

[jira] [Created] (FLINK-19724) tableEnv.toRetractStream(table, PoJo.class) when

HunterHunter created FLINK-19724: Summary: tableEnv.toRetractStream(table, PoJo.class) when Key: FLINK-19724 URL: https://issues.apache.org/jira/browse/FLINK-19724 Project: Flink Issue Type:

[jira] [Created] (FLINK-19723) The retry mechanism of jdbc connector has the risk of data duplication

shizhengchao created FLINK-19723: Summary: The retry mechanism of jdbc connector has the risk of data duplication Key: FLINK-19723 URL: https://issues.apache.org/jira/browse/FLINK-19723 Project: Flink

Re: [VOTE] FLIP-135: Approximate Task-Local Recovery

Thanks for driving this effort, Yuan. +1 (binding) on my side. Best, Zhijiang -- From:Piotr Nowojski Send Time:2020年10月19日(星期一) 21:02 To:dev Subject:Re: [VOTE] FLIP-135: Approximate Task-Local Recovery Hey, I carry over my +1

[jira] [Created] (FLINK-19721) Speed up the frequency of checks in RpcGatewayRetriever

Dan Hill created FLINK-19721: Summary: Speed up the frequency of checks in RpcGatewayRetriever Key: FLINK-19721 URL: https://issues.apache.org/jira/browse/FLINK-19721 Project: Flink Issue Type: I

[jira] [Created] (FLINK-19722) Pushdown Watermark to SourceProvider (new Source)

Jingsong Lee created FLINK-19722: Summary: Pushdown Watermark to SourceProvider (new Source) Key: FLINK-19722 URL: https://issues.apache.org/jira/browse/FLINK-19722 Project: Flink Issue Type:

[jira] [Created] (FLINK-19720) Introduce DataStream and Source Providers

Jingsong Lee created FLINK-19720: Summary: Introduce DataStream and Source Providers Key: FLINK-19720 URL: https://issues.apache.org/jira/browse/FLINK-19720 Project: Flink Issue Type: Sub-tas

[jira] [Created] (FLINK-19719) FLIP-146: Improve new TableSource and TableSink interfaces

Jingsong Lee created FLINK-19719: Summary: FLIP-146: Improve new TableSource and TableSink interfaces Key: FLINK-19719 URL: https://issues.apache.org/jira/browse/FLINK-19719 Project: Flink Is

Re: Re: [VOTE] FLIP-146: Improve new TableSource and TableSink interfaces

Hi all, The voting time for FLIP-146 has passed. I'm closing the vote now. There were 6 +1 votes, 3 of which are binding: - Danny - Admin - Jark (binding) - Leonard - Jingsong (binding) - Godfrey (binding) - Shouwei - Hailong There were no disapproving votes. Thus, FLIP-146 has been accepted.

[jira] [Created] (FLINK-19718) HiveTableSourceITCase.testStreamPartitionRead is not stable on Azure

Jark Wu created FLINK-19718: --- Summary: HiveTableSourceITCase.testStreamPartitionRead is not stable on Azure Key: FLINK-19718 URL: https://issues.apache.org/jira/browse/FLINK-19718 Project: Flink I

[jira] [Created] (FLINK-19717) SourceReaderBase.pollNext may return END_OF_INPUT if SplitReader.fetch throws

Kezhu Wang created FLINK-19717: -- Summary: SourceReaderBase.pollNext may return END_OF_INPUT if SplitReader.fetch throws Key: FLINK-19717 URL: https://issues.apache.org/jira/browse/FLINK-19717 Project: Fl

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

Hi Shengkai, Thank you for driving this effort. I believe this a very important feature for many users who use Kafka and Flink SQL together. A few questions and thoughts: * Is your example "Use KTable as a reference/dimension table" correct? It uses the "kafka" connector and does not specify a pr

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

Hi Danny, First of all, we didn't introduce any concepts from KSQL (e.g. Stream vs Table notion). This new connector will produce a changelog stream, so it's still a dynamic table and doesn't conflict with Flink core concepts. The "ktable" is just a connector name, we can also call it "compacted-

[jira] [Created] (FLINK-19716) [Kinesis][EFO] Unable to use Assume Role with EFO record publisher

Danny Cranmer created FLINK-19716: - Summary: [Kinesis][EFO] Unable to use Assume Role with EFO record publisher Key: FLINK-19716 URL: https://issues.apache.org/jira/browse/FLINK-19716 Project: Flink

Re: [VOTE] FLIP-135: Approximate Task-Local Recovery

Hey, I carry over my +1 (binding) from the discussion thread. Best, Piotrek pon., 19 paź 2020 o 14:56 Yuan Mei napisał(a): > Hey, > > I would like to start a voting thread for FLIP-135 [1], for approximate > task local recovery. The proposal has been discussed in [2]. > > The vote will be open

Re:Re: [VOTE] FLIP-146: Improve new TableSource and TableSink interfaces

+1 This feature sounds great to me. Allowing users to customize parallelism or infer parallelism for source or sink is indeed an urgent need. For example, In my inner flink jobs (1w+), we need to provide the ability for users to customize parallelism, and the parallelism for Kafka always is t

[VOTE] FLIP-135: Approximate Task-Local Recovery

Hey, I would like to start a voting thread for FLIP-135 [1], for approximate task local recovery. The proposal has been discussed in [2]. The vote will be open till Oct. 23rd (72h, excluding weekends) unless there is an objection or not enough votes. [1] https://cwiki.apache.org/confluence/displ

[jira] [Created] (FLINK-19715) Optimize re-assignment of excess resources

Chesnay Schepler created FLINK-19715: Summary: Optimize re-assignment of excess resources Key: FLINK-19715 URL: https://issues.apache.org/jira/browse/FLINK-19715 Project: Flink Issue Type

[jira] [Created] (FLINK-19714) Assign different logical pipelined regions of a DataSet job to different slot sharing group

Zhu Zhu created FLINK-19714: --- Summary: Assign different logical pipelined regions of a DataSet job to different slot sharing group Key: FLINK-19714 URL: https://issues.apache.org/jira/browse/FLINK-19714 Pro

[jira] [Created] (FLINK-19713) Update `JobGraphUtils` type to class

xiaozilong created FLINK-19713: -- Summary: Update `JobGraphUtils` type to class Key: FLINK-19713 URL: https://issues.apache.org/jira/browse/FLINK-19713 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-19712) Pipelined region failover should skip restarting tasks in CREATED

Zhu Zhu created FLINK-19712: --- Summary: Pipelined region failover should skip restarting tasks in CREATED Key: FLINK-19712 URL: https://issues.apache.org/jira/browse/FLINK-19712 Project: Flink Issu

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

Hi Jingsong, As the FLIP describes, "KTable connector produces a changelog stream, where each data record represents an update or delete event.". Therefore, a ktable source is an unbounded stream source. Selecting a ktable source is similar to selecting a kafka source with debezium-json format tha

Re: [DISCUSS] FLIP-135: Approximate Task-Local Recovery

Hi, I was involved in some off-line discussions and some early stages of the discussions on this topic. In recent weeks Yuan has also prepared a good looking prototype and draft changes that look good to me, so I would be +1 for accepting this FLIP. I guess as there were no objections here, I wou

Re: [DISCUSS] Release 1.12 Feature Freeze

Per FLIP-145, we have many runtime operators to implement and bridge it with the planner. Best, Danny Chan 在 2020年10月19日 +0800 PM6:59，Robert Metzger ，写道： > Thank you for your responses so far. > > @Kurt, Jingsong, Danny: Which JIRAs/FLIPs are going to benefit from the > extension? > > On Mon, Oct

Re: [DISCUSS] Release 1.12 Feature Freeze

Thank you for your responses so far. @Kurt, Jingsong, Danny: Which JIRAs/FLIPs are going to benefit from the extension? On Mon, Oct 19, 2020 at 12:45 PM Aljoscha Krettek wrote: > @Robert Your (and Dian's) suggestions sound good to me! I like keeping > to master frozen for a while since it will

[jira] [Created] (FLINK-19711) Parsing of invalid CSV fails even with ignoreParseErrors is enabled

Roman Khachatryan created FLINK-19711: - Summary: Parsing of invalid CSV fails even with ignoreParseErrors is enabled Key: FLINK-19711 URL: https://issues.apache.org/jira/browse/FLINK-19711 Project

Re: [DISCUSS] Release 1.12 Feature Freeze

@Robert Your (and Dian's) suggestions sound good to me! I like keeping to master frozen for a while since it will prevent a lot of duplicate merging efforts. Regarding the date: I'm fine with the proposed date but I can also see that extending it to the end of the week could be helpful. Aljo

Re: [DISCUSS] Enforce common opinionated coding style using Spotless

I don't like checkstyle because it cannot be easily applied from the commandline. I'm happy to learn otherwise, though. And I'd also be very happy about alternative suggestions that can do that. Right now, I think Spotless is the most straightforward tool for that. Also I don't care so much ab

Re: [DISCUSS] Release 1.12 Feature Freeze

+1 for Kurt suggestion, there are many features for SQL yet, 2 more days are valuable. Best, Danny Chan 在 2020年10月19日 +0800 PM6:22，Jingsong Li ，写道： > Hi Robert, > > Thanks for your detailed explanation. > > At present, we are preparing or participating in Flink forward, so +1 for > appropriate ex

Re: [DISCUSS] Release 1.12 Feature Freeze

Hi Robert, Thanks for your detailed explanation. At present, we are preparing or participating in Flink forward, so +1 for appropriate extension of deadline. Best, Jingsong On Mon, Oct 19, 2020 at 5:36 PM Kurt Young wrote: > Can we change the freeze date to October 30th (Friday next week)? It

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

The concept seems conflicts with the Flink abstraction “dynamic table”, in Flink we see both “stream” and “table” as a dynamic table, I think we should make clear first how to express stream and table specific features on one “dynamic table”, it is more natural for KSQL because KSQL takes stream

[jira] [Created] (FLINK-19710) Avoid performance regression introduced by thread-local keyword of FRocksDB

Yun Tang created FLINK-19710: Summary: Avoid performance regression introduced by thread-local keyword of FRocksDB Key: FLINK-19710 URL: https://issues.apache.org/jira/browse/FLINK-19710 Project: Flink

Re: [DISCUSS] Release 1.12 Feature Freeze

Can we change the freeze date to October 30th (Friday next week)? It would be helpful for us if we have 2 more days. Best, Kurt On Mon, Oct 19, 2020 at 5:00 PM Robert Metzger wrote: > Hi all, > > Dian and I would like to discuss a few items regarding the upcoming Flink > 1.12 feature freeze: >

Re: [DISCUSS] FLIP-148: Introduce Sort-Merge Based Blocking Shuffle to Flink

Thanks for launching the discussion and the respective FLIP, Yingjie! In general, I am +1 for this proposal since sort-merge ability has already been taken widely in other batch-based project, like MR, Spark, etc. And it indeed has some performance benefits in some scenarios as mentioned in FLIP

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

Thanks Shengkai for your proposal. +1 for this feature. > Future Work: Support bounded KTable source I don't think it should be a future work, I think it is one of the important concepts of this FLIP. We need to understand it now. Intuitively, a ktable in my opinion is a bounded table rather th

Re: [REMINDER] Please only merge pull requests when CI is green and help keep our CI stable

Thanks for the reminder Robert. I think you are completely right. Since we are getting closer to the feature freeze and therefore also the testing period, having green builds and only merging PRs where we are sure that CI passes will decrease the likelihood of introducing new bugs and, hence, it wi

[REMINDER] Please only merge pull requests when CI is green and help keep our CI stable

Hi folks, As we are approaching the final stages of the Flink 1.12 release cycle, I would like to remind everybody to *only merge pull requests when the CI system gives green light*! I know that the CI system sometimes shows "FAILED" due to CI system instabilities, but please carefully check those

[DISCUSS] Release 1.12 Feature Freeze

Hi all, Dian and I would like to discuss a few items regarding the upcoming Flink 1.12 feature freeze: *A) Exact feature freeze day* So far, we've always said "end of October " for the freeze. We propose (end of day CEST) October 28t

[DISCUSS] FLIP-149: Introduce the KTable Connector

Hi, devs. Jark and I want to start a new FLIP to introduce the KTable connector. The KTable is a shortcut of "Kafka Table", it also has the same semantics with the KTable notion in Kafka Stream. FLIP-149: https://cwiki.apache.org/confluence/display/FLINK/FLIP-149%3A+Introduce+the+KTable+Connector

[jira] [Created] (FLINK-19709) Support KeyedSortedMapState in DataStream API

Sean Z created FLINK-19709: -- Summary: Support KeyedSortedMapState in DataStream API Key: FLINK-19709 URL: https://issues.apache.org/jira/browse/FLINK-19709 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-19708) Support KeyedSortedMapState in DataStream API

Sean Z created FLINK-19708: -- Summary: Support KeyedSortedMapState in DataStream API Key: FLINK-19708 URL: https://issues.apache.org/jira/browse/FLINK-19708 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-19707) Refactor table streaming file sink

Jingsong Lee created FLINK-19707: Summary: Refactor table streaming file sink Key: FLINK-19707 URL: https://issues.apache.org/jira/browse/FLINK-19707 Project: Flink Issue Type: Test

[jira] [Created] (FLINK-19706) Introduce `Repeated Partition Commit Check` in `org.apache.flink.table.filesystem.PartitionCommitPolicy`

Lsw_aka_laplace created FLINK-19706: --- Summary: Introduce `Repeated Partition Commit Check` in `org.apache.flink.table.filesystem.PartitionCommitPolicy` Key: FLINK-19706 URL: https://issues.apache.org/jira/brows