Re: [VOTE] NEW FLIP-102: Add More Metrics to TaskManager

+1 Thanks for the work. Best, Jark On Fri, 23 Oct 2020 at 10:13, Xintong Song wrote: > Thanks Yadong, Mattias and Lining for reviving this FLIP. > > I've seen so many users confused by the current webui page of task manager > metrics. This FLIP should definitely help them understand the memory

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

Hi, all. It seems we have reached a consensus on the FLIP. If no one has other objections, I would like to start the vote for FLIP-149. Best, Shengkai Jingsong Li 于2020年10月23日周五 下午2:25写道： > Thanks for explanation, > > I am OK for `upsert`. Yes, Its concept has been accepted by many systems. > >

[jira] [Created] (FLINK-19778) Failed job reinitiated with wrong checkpoint after a ZK reconnection

Paul Lin created FLINK-19778: Summary: Failed job reinitiated with wrong checkpoint after a ZK reconnection Key: FLINK-19778 URL: https://issues.apache.org/jira/browse/FLINK-19778 Project: Flink

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

Thanks for explanation, I am OK for `upsert`. Yes, Its concept has been accepted by many systems. Best, Jingsong On Fri, Oct 23, 2020 at 12:38 PM Jark Wu wrote: > Hi Timo, > > I have some concerns about `kafka-cdc`, > 1) cdc is an abbreviation of Change Data Capture which is commonly used for

Re: [DISCUSS] Release 1.12 Feature Freeze

+1 for November 2nd Thanks, Zhu Xintong Song 于2020年10月20日周二 下午11:26写道： > +1 for Nov. 2nd. > > Thank you~ > > Xintong Song > > > > On Tue, Oct 20, 2020 at 8:56 PM Leonard Xu wrote: > > > +1 for **Sunday night**. > > > > > > Best > > Leonard > > > > > 在 2020年10月20日，19:03，Jingsong Li 写道： > > > >

Re: [DISCUSS] FLIP-148: Introduce Sort-Merge Based Blocking Shuffle to Flink

Hi devs, This discussion thread has been opened for over a week. If there is no other concerns, I'd like to open a voting thread soon. Best, Yingjie Yingjie Cao 于2020年10月23日周五 上午11:56写道： > Hi Zhijiang, > > Thanks for your reply and suggestions. > > 1. For > taskmanager.network.sort-merge-block

[jira] [Created] (FLINK-19777) java.lang.NullPointerException

frank wang created FLINK-19777: -- Summary: java.lang.NullPointerException Key: FLINK-19777 URL: https://issues.apache.org/jira/browse/FLINK-19777 Project: Flink Issue Type: Bug Environme

Resuming Savepoint issue with upgraded Flink version 1.11.2

Hi, We are trying to save checkpoints for one of the flink job running in Flink version 1.9 and tried to resume the same flink job in Flink version 1.11.2. We are getting the below error when trying to restore the saved checkpoint in the newer flink version. Can Cannot map checkpoint/savepoint

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

Hi Timo, I have some concerns about `kafka-cdc`, 1) cdc is an abbreviation of Change Data Capture which is commonly used for databases, not for message queues. 2) usually, cdc produces full content of changelog, including UPDATE_BEFORE, however "upsert kafka" doesn't 3) `kafka-cdc` sounds like a n

Re: [DISCUSS] FLIP-148: Introduce Sort-Merge Based Blocking Shuffle to Flink

Hi Zhijiang, Thanks for your reply and suggestions. 1. For taskmanager.network.sort-merge-blocking-shuffle.max-files-per-partition, we decide to append all data produced by one result partition to one file, so this option will be removed. 2. For taskmanager.network.sort-merge-blocking-shuffle.bu

[jira] [Created] (FLINK-19776) cancel with savePoint throw Unrecognized field "status"

tonychan created FLINK-19776: Summary: cancel with savePoint throw Unrecognized field "status" Key: FLINK-19776 URL: https://issues.apache.org/jira/browse/FLINK-19776 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

The `kafka-cdc` looks good to me. We can even give options to indicate whether to turn on compact, because compact is just an optimization? - ktable let me think about KSQL. - kafka-compacted it is not just compacted, more than that, it still has the ability of CDC - upsert-kafka , upsert is back,

Re: [VOTE] NEW FLIP-102: Add More Metrics to TaskManager

Thanks Yadong, Mattias and Lining for reviving this FLIP. I've seen so many users confused by the current webui page of task manager metrics. This FLIP should definitely help them understand the memory footprints and tune the configurations for task managers. The design part of this proposal look

[jira] [Created] (FLINK-19775) SystemProcessingTimeServiceTest.testImmediateShutdown is instable

Dian Fu created FLINK-19775: --- Summary: SystemProcessingTimeServiceTest.testImmediateShutdown is instable Key: FLINK-19775 URL: https://issues.apache.org/jira/browse/FLINK-19775 Project: Flink Issu

Re: NullPointerException when trying to read null array in Postgres using JDBC Connector

Looking at this, it’s a simple enough fix. My question would just be around a unit test that tests this particular bug. It doesn’t look like there is anything that directly tests the Postgres row converter. There is a test that utilizes Derby, but it looks like only Postgres supports arrays as f

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

Hi Jark, I would be fine with `connector=upsert-kafka`. Another idea would be to align the name to other available Flink connectors [1]: `connector=kafka-cdc`. Regards, Timo [1] https://github.com/ververica/flink-cdc-connectors On 22.10.20 17:17, Jark Wu wrote: Another name is "connector=u

[jira] [Created] (FLINK-19774) Introduce sub partition view version for approximate Failover

Yuan Mei created FLINK-19774: Summary: Introduce sub partition view version for approximate Failover Key: FLINK-19774 URL: https://issues.apache.org/jira/browse/FLINK-19774 Project: Flink Issue

[jira] [Created] (FLINK-19773) Exponential backoff restart strategy

Levi Ramsey created FLINK-19773: --- Summary: Exponential backoff restart strategy Key: FLINK-19773 URL: https://issues.apache.org/jira/browse/FLINK-19773 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

Another name is "connector=upsert-kafka', I think this can solve Timo's concern on the "compacted" word. Materialize also uses "ENVELOPE UPSERT" [1] keyword to identify such kafka sources. I think "upsert" is a well-known terminology widely used in many systems and matches the behavior of how we

Re: Un-ignored Parsing Exceptions in the CsvFormat

Hey Roman, Sorry to miss this -- thanks for the confirmation and making the ticket. I'm happy to propose a fix if someone is able to assign the ticket to me. Best, Austin On Mon, Oct 19, 2020 at 6:56 AM Khachatryan Roman < khachatryan.ro...@gmail.com> wrote: > Hey Austin, > > I think you are ri

[jira] [Created] (FLINK-19772) Add utility for retrying sync operations with exponential timeout

Chesnay Schepler created FLINK-19772: Summary: Add utility for retrying sync operations with exponential timeout Key: FLINK-19772 URL: https://issues.apache.org/jira/browse/FLINK-19772 Project: Fl

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

Good validation messages can't solve the broken user experience, especially that such update mode option will implicitly make half of current kafka options invalid or doesn't make sense. Best, Kurt On Thu, Oct 22, 2020 at 10:31 PM Jark Wu wrote: > Hi Timo, Seth, > > The default value "insertin

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

Hi Timo, Seth, The default value "inserting" of "mode" might be not suitable, because "debezium-json" emits changelog messages which include updates. On Thu, 22 Oct 2020 at 22:10, Seth Wiesman wrote: > +1 for supporting upsert results into Kafka. > > I have no comments on the implementation det

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

+1 for supporting upsert results into Kafka. I have no comments on the implementation details. As far as configuration goes, I tend to favor Timo's option where we add a "mode" property to the existing Kafka table with default value "inserting". If the mode is set to "updating" then the validatio

Re: NullPointerException when trying to read null array in Postgres using JDBC Connector

Danny, Thanks! I have created a new JIRA issue [1]. I’ll look into how hard it is to get a patch and unit test myself, although I may need a hand on the process of making a change to both the master branch and a release branch if it is desired to get a fix into 1.11. Regards, Dylan Forciea [1

[jira] [Created] (FLINK-19771) NullPointerException when accessing null array from postgres in JDBC Connector

Dylan Forciea created FLINK-19771: - Summary: NullPointerException when accessing null array from postgres in JDBC Connector Key: FLINK-19771 URL: https://issues.apache.org/jira/browse/FLINK-19771 Proj

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

Hi Jark, "calling it "kafka-compacted" can even remind users to enable log compaction" But sometimes users like to store a lineage of changes in their topics. Indepent of any ktable/kstream interpretation. I let the majority decide on this topic to not further block this effort. But we mig

Re: Need help in creating Flink Streaming s3 Job for multiple path reader one by one

The existing ContinuousFileMonitoringFunction and ContinuousFileReaderOperator already take care of that. Unless you aren't re-implementing them from scratch you shouldn't have to do anything. On 10/22/2020 1:47 PM, Satyaa Dixit wrote: Hi Chesnay, Thanks for your support.It helped a lot. I nee

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

Hi Timo, Thanks for your opinions. 1) Implementation We will have an stateful operator to generate INSERT and UPDATE_BEFORE. This operator is keyby-ed (primary key as the shuffle key) after the source operator. The implementation of this operator is very similar to the existing `DeduplicateKeepLa

[jira] [Created] (FLINK-19770) mvn clean verify - testConfigurePythonExecution failing

Juha Mynttinen created FLINK-19770: -- Summary: mvn clean verify - testConfigurePythonExecution failing Key: FLINK-19770 URL: https://issues.apache.org/jira/browse/FLINK-19770 Project: Flink I

[jira] [Created] (FLINK-19769) Reuse StreamRecord in SourceOutputWithWatermarks#collect

Caizhi Weng created FLINK-19769: --- Summary: Reuse StreamRecord in SourceOutputWithWatermarks#collect Key: FLINK-19769 URL: https://issues.apache.org/jira/browse/FLINK-19769 Project: Flink Issue

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

Hi Shengkai, Hi Jark, thanks for this great proposal. It is time to finally connect the changelog processor with a compacted Kafka topic. "The operator will produce INSERT rows, or additionally generate UPDATE_BEFORE rows for the previous image, or produce DELETE rows with all columns filled

Re: Need help in creating Flink Streaming s3 Job for multiple path reader one by one

Hi Chesnay, Thanks for your support.It helped a lot. I need one more help on how to do checkpointing as part of the s3 reader source in case if some failure happens due to OutOfMemoryError exception or it could be any other failure, and want to recover the data from last reader splitted offset duri

[VOTE] NEW FLIP-102: Add More Metrics to TaskManager

Hi all I want to start a new vote for FLIP-102, which proposes to add more metrics to the task manager in web UI. The new FLIP-102 was revisited and adapted following the old ML discussion

Re: [VOTE] FLIP-102: Add More Metrics to TaskManager

Hi all there have been lots of discussions since the vote started and many suggestions have been made Matthias and I had updated the FLIP-102 following the suggestions and discussions I want to cancel

[jira] [Created] (FLINK-19768) The shell "./yarn-session.sh " not use log4j-session.properties , it use log4j.properties

YUJIANBO created FLINK-19768: Summary: The shell "./yarn-session.sh " not use log4j-session.properties , it use log4j.properties Key: FLINK-19768 URL: https://issues.apache.org/jira/browse/FLINK-19768 Pr

[jira] [Created] (FLINK-19767) Add AbstractSlotPoolFactory

Chesnay Schepler created FLINK-19767: Summary: Add AbstractSlotPoolFactory Key: FLINK-19767 URL: https://issues.apache.org/jira/browse/FLINK-19767 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-19766) Introduce File streaming compaction operators

Jingsong Lee created FLINK-19766: Summary: Introduce File streaming compaction operators Key: FLINK-19766 URL: https://issues.apache.org/jira/browse/FLINK-19766 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-19765) flink SqlUseCatalog.getCatalogName is not unified with SqlCreateCatalog and SqlDropCatalog

jackylau created FLINK-19765: Summary: flink SqlUseCatalog.getCatalogName is not unified with SqlCreateCatalog and SqlDropCatalog Key: FLINK-19765 URL: https://issues.apache.org/jira/browse/FLINK-19765 Pr

[jira] [Created] (FLINK-19764) Add More Metrics to TaskManager in Web UI

Yadong Xie created FLINK-19764: -- Summary: Add More Metrics to TaskManager in Web UI Key: FLINK-19764 URL: https://issues.apache.org/jira/browse/FLINK-19764 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-19763) Missing test MetricUtilsTest.testNonHeapMetricUsageNotStatic

Matthias created FLINK-19763: Summary: Missing test MetricUtilsTest.testNonHeapMetricUsageNotStatic Key: FLINK-19763 URL: https://issues.apache.org/jira/browse/FLINK-19763 Project: Flink Issue T

[jira] [Created] (FLINK-19762) Selecting Job-ID in web UI covers more than the ID

Matthias created FLINK-19762: Summary: Selecting Job-ID in web UI covers more than the ID Key: FLINK-19762 URL: https://issues.apache.org/jira/browse/FLINK-19762 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-19761) Add lookup method for registered ShuffleDescriptor in ShuffleMaster

Xuannan Su created FLINK-19761: -- Summary: Add lookup method for registered ShuffleDescriptor in ShuffleMaster Key: FLINK-19761 URL: https://issues.apache.org/jira/browse/FLINK-19761 Project: Flink