Unable to resolve the operator from the metafile using State Processor API

2021-02-10 Thread Partha Mishra
Hi, I have specified operator uuid for all of my operator in Flink job and triggered a savepoint and `_metafile` got generated. When I use state processor APIs to read the saved snapshot and try to read the mapping, I am not able to resolve which operator id map to which operator in my code. 1

[jira] [Created] (FLINK-21360) Add resourceTimeout configuration

2021-02-10 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-21360: -- Summary: Add resourceTimeout configuration Key: FLINK-21360 URL: https://issues.apache.org/jira/browse/FLINK-21360 Project: Flink Issue Type: Sub-task

Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

2021-02-10 Thread Yun Tang
Congratulations, Roman! Today is also the beginning of Chinese Spring Festival holiday, at which we Chinese celebrate across the world for the next lunar new year, and also very happy to have you on board! Best Yun Tang From: Roman Khachatryan Sent: Thursday, F

Re: [Announce] Hugo migration complete

2021-02-10 Thread Robert Metzger
I'm taking care of the buildbot scripts. Thanks a lot for the effort! On Wed, Feb 10, 2021 at 7:23 PM Chesnay Schepler wrote: > Looks like the buildbot scripts have not been updated yet: > > https://ci.apache.org/builders/flink-docs-master/builds/2134/steps/Build%20docs/logs/stdio > > On 2/10/2

Re: First development steps, broken build workarounds?

2021-02-10 Thread Chesnay Schepler
Sure, but for WSL in particular we'd ideally we have way to detect that we're even running within it. I'm not sure whether there is a sane solution for that that works across various distros. On 2/11/2021 1:14 AM, bytesandwich wrote: Hi Chesnay and 郁蓝 Thank you for your help. I think Chesnay'

Re: First development steps, broken build workarounds?

2021-02-10 Thread bytesandwich
Hi Chesnay and 郁蓝 Thank you for your help. I think Chesnay's feedback makes sense that some tests are bound to fail under WSL. Maybe if more people encounter this we can make a marked test suite? Thanks again, Jack -- Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

[jira] [Created] (FLINK-21359) CompatibilityResult issues with Flink 1.9.0

2021-02-10 Thread Siva (Jira)
Siva created FLINK-21359: Summary: CompatibilityResult issues with Flink 1.9.0 Key: FLINK-21359 URL: https://issues.apache.org/jira/browse/FLINK-21359 Project: Flink Issue Type: Bug Environm

Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

2021-02-10 Thread Roman Khachatryan
Many thanks to all of you! Regards, Roman On Wed, Feb 10, 2021 at 7:12 PM Matthias Pohl wrote: > Congratulations, Roman! :-) > > On Wed, Feb 10, 2021 at 3:23 PM Kezhu Wang wrote: > >> Congratulations! >> >> Best, >> Kezhu Wang >> >> >> On February 10, 2021 at 21:53:52, Dawid Wysakowicz ( >> d

Re: Source of Cloudera 3 Hadoop Flink shaded uber jars?

2021-02-10 Thread Adam Roberts
Awesome, thank you everyone for the prompt replies! That does clear things up for me. I'll go back to those asking for it to confirm why the particular library is to be used instead (and if there is a good reason I shall continue my search). Cheers! On Wed, 10 Feb 2021, 19:25 Gyula Fóra, wrote:

Re: Source of Cloudera 3 Hadoop Flink shaded uber jars?

2021-02-10 Thread Gyula Fóra
Hi Adam! Cloudera has published these artifacts before but they are no longer used similarly to the Flink project. You should set the HADOOP_CLASSPATH to the hadoop libs instead of using the shaded jars. Cheers Gyula On Wed, Feb 10, 2021 at 7:27 PM Chesnay Schepler wrote: > I can't tell you w

Re: Source of Cloudera 3 Hadoop Flink shaded uber jars?

2021-02-10 Thread Chesnay Schepler
I can't tell you where it comes from, just that the Flink project had nothing to do with it. FYI; Hadoop 3 has been supported by Flink since 1.11 . We do no longer publish shaded-hadoop jars though because it is a maintenance nightmare. On 2/10/2021 6:54 PM, Adam Roberts wrote: Hey everyone,

Re: [Announce] Hugo migration complete

2021-02-10 Thread Chesnay Schepler
Looks like the buildbot scripts have not been updated yet: https://ci.apache.org/builders/flink-docs-master/builds/2134/steps/Build%20docs/logs/stdio On 2/10/2021 7:16 PM, Matthias Pohl wrote: Thanks again, Seth. Good job! :+1: On Wed, Feb 10, 2021 at 5:00 PM Seth Wiesman wrote: Piotr, docs

Re: [Announce] Hugo migration complete

2021-02-10 Thread Matthias Pohl
Thanks again, Seth. Good job! :+1: On Wed, Feb 10, 2021 at 5:00 PM Seth Wiesman wrote: > Piotr, > > docs/content is where Jekyll stores its build artifacts. In > release-1.11/release-1.12 it is hidden via .gitignore so its content is not > removed when switching to master from those branches. Yo

Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

2021-02-10 Thread Matthias Pohl
Congratulations, Roman! :-) On Wed, Feb 10, 2021 at 3:23 PM Kezhu Wang wrote: > Congratulations! > > Best, > Kezhu Wang > > > On February 10, 2021 at 21:53:52, Dawid Wysakowicz (dwysakow...@apache.org > ) > wrote: > > Congratulations Roman! Glad to have you on board! > > Best, > > Dawid > > On 1

Re: [DISCUSS] Releasing Apache Flink 1.12.2

2021-02-10 Thread Matthias Pohl
I created FLINK-21358 [1] (and a PR [2] for it) which we might want to include in 1.12.2 as well. Best, Matthias [1] https://issues.apache.org/jira/browse/FLINK-21358 [2] https://github.com/apache/flink/pull/14920 On Tue, Feb 9, 2021 at 1:19 PM Yuan Mei wrote: > Thanks very much for your repli

Source of Cloudera 3 Hadoop Flink shaded uber jars?

2021-02-10 Thread Adam Roberts
Hey everyone, I realise this could be a long shot (but perhaps not given the list of committers and perhaps some are from Cloudera!) - I'm wondering if anyone knows the source location and build instructions for the jars at https://repository.cloudera.com/artifactory/libs-release-local/org/apache/f

[jira] [Created] (FLINK-21358) Missing snapshot version compatibility for 1.12

2021-02-10 Thread Matthias (Jira)
Matthias created FLINK-21358: Summary: Missing snapshot version compatibility for 1.12 Key: FLINK-21358 URL: https://issues.apache.org/jira/browse/FLINK-21358 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-21357) Add periodic materialization

2021-02-10 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-21357: - Summary: Add periodic materialization Key: FLINK-21357 URL: https://issues.apache.org/jira/browse/FLINK-21357 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-21356) Implement incremental checkpointing and recovery using state changelog

2021-02-10 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-21356: - Summary: Implement incremental checkpointing and recovery using state changelog Key: FLINK-21356 URL: https://issues.apache.org/jira/browse/FLINK-21356 Proj

[jira] [Created] (FLINK-21355) Send changes to the state changelog (still proxy everything)

2021-02-10 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-21355: - Summary: Send changes to the state changelog (still proxy everything) Key: FLINK-21355 URL: https://issues.apache.org/jira/browse/FLINK-21355 Project: Flink

[jira] [Created] (FLINK-21354) Add ChangelogStateBackend (proxy-everything)

2021-02-10 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-21354: - Summary: Add ChangelogStateBackend (proxy-everything) Key: FLINK-21354 URL: https://issues.apache.org/jira/browse/FLINK-21354 Project: Flink Issue

[jira] [Created] (FLINK-21353) Add FS-based StateChangelog implementation

2021-02-10 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-21353: - Summary: Add FS-based StateChangelog implementation Key: FLINK-21353 URL: https://issues.apache.org/jira/browse/FLINK-21353 Project: Flink Issue Ty

[jira] [Created] (FLINK-21352) FLIP-158: Generalized incremental checkpoints

2021-02-10 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-21352: - Summary: FLIP-158: Generalized incremental checkpoints Key: FLINK-21352 URL: https://issues.apache.org/jira/browse/FLINK-21352 Project: Flink Issue

Re: [Announce] Hugo migration complete

2021-02-10 Thread Seth Wiesman
Piotr, docs/content is where Jekyll stores its build artifacts. In release-1.11/release-1.12 it is hidden via .gitignore so its content is not removed when switching to master from those branches. You only need to remember to delete this folder when you are switching from release-1.11/12 to master

Re: [Announce] Hugo migration complete

2021-02-10 Thread Piotr Nowojski
Hi Seth, To clarify if I understand it correctly. Does it mean that now every time we are switching between master/release-1.11/release-1.12 branches we will need to remember to remove docs/content? Best, Piotrek śr., 10 lut 2021 o 16:08 Seth Wiesman napisał(a): > One thing at a time :) > > I'

[jira] [Created] (FLINK-21351) Incremental checkpoint data would be lost once a non-stop savepoint completed

2021-02-10 Thread Yun Tang (Jira)
Yun Tang created FLINK-21351: Summary: Incremental checkpoint data would be lost once a non-stop savepoint completed Key: FLINK-21351 URL: https://issues.apache.org/jira/browse/FLINK-21351 Project: Flink

Re: [Announce] Hugo migration complete

2021-02-10 Thread Seth Wiesman
One thing at a time :) I'm creating tickets now for the ones on the PR but please create others as they come up. On Wed, Feb 10, 2021 at 9:02 AM Chesnay Schepler wrote: > Thank you Seth for the work you put in. This is a really nice improvement. > (When do we migrate the Flink website? :P ) > >

Re: [Announce] Hugo migration complete

2021-02-10 Thread Chesnay Schepler
Thank you Seth for the work you put in. This is a really nice improvement. (When do we migrate the Flink website? :P ) Regarding the follow-ups raised in the PR, will you go over them and create tickets, or...? On 2/10/2021 3:40 PM, Seth Wiesman wrote: Hi Everyone, I'm happy to say the migra

[jira] [Created] (FLINK-21350) ParquetInputFormat incorrectly interprets timestamps encoded in microseconds as timestamps encoded in milliseconds

2021-02-10 Thread Jeffrey Charles (Jira)
Jeffrey Charles created FLINK-21350: --- Summary: ParquetInputFormat incorrectly interprets timestamps encoded in microseconds as timestamps encoded in milliseconds Key: FLINK-21350 URL: https://issues.apache.org/j

[Announce] Hugo migration complete

2021-02-10 Thread Seth Wiesman
Hi Everyone, I'm happy to say the migration of Flinks documentation to hugo is complete! There will likely be some continued work over the next weeks to clean up the css but the documentation freeze is over. Please carefully read docs/README for details about how to build and work with our new do

Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

2021-02-10 Thread Kezhu Wang
Congratulations! Best, Kezhu Wang On February 10, 2021 at 21:53:52, Dawid Wysakowicz (dwysakow...@apache.org) wrote: Congratulations Roman! Glad to have you on board! Best, Dawid On 10/02/2021 14:44, Igal Shilman wrote: > Welcome Roman! > Top-notch stuff! :) > > All the best, > Igal. > > On

[jira] [Created] (FLINK-21349) 'Quickstarts Scala nightly end-to-end test' fails with errors in logs

2021-02-10 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-21349: Summary: 'Quickstarts Scala nightly end-to-end test' fails with errors in logs Key: FLINK-21349 URL: https://issues.apache.org/jira/browse/FLINK-21349 Project

Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

2021-02-10 Thread Dawid Wysakowicz
Congratulations Roman! Glad to have you on board! Best, Dawid On 10/02/2021 14:44, Igal Shilman wrote: > Welcome Roman! > Top-notch stuff! :) > > All the best, > Igal. > > On Wed, Feb 10, 2021 at 2:15 PM Kostas Kloudas wrote: > >> Congrats Roman! >> >> Kostas >> >> On Wed, Feb 10, 2021 at 2:08

Re: Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

2021-02-10 Thread Igal Shilman
Welcome Roman! Top-notch stuff! :) All the best, Igal. On Wed, Feb 10, 2021 at 2:15 PM Kostas Kloudas wrote: > Congrats Roman! > > Kostas > > On Wed, Feb 10, 2021 at 2:08 PM Arvid Heise wrote: > > > > Congrats! Well deserved. > > > > On Wed, Feb 10, 2021 at 1:54 PM Yun Gao > > wrote: > > > >

[ANNOUNCE] Randomized tests

2021-02-10 Thread Arvid Heise
Dear devs, I just merged FLINK-19520, which adds randomization to all tests that use the MiniCluster(WithResource). The overall goal is to provide a faster and better coverage for future changes in the network stack in particular in conjunction with checkpointing. We implemented it in a way to fu

[jira] [Created] (FLINK-21348) Add tests for StateWithExecutionGraph

2021-02-10 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-21348: -- Summary: Add tests for StateWithExecutionGraph Key: FLINK-21348 URL: https://issues.apache.org/jira/browse/FLINK-21348 Project: Flink Issue Type: Sub-tas

Re: Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

2021-02-10 Thread Kostas Kloudas
Congrats Roman! Kostas On Wed, Feb 10, 2021 at 2:08 PM Arvid Heise wrote: > > Congrats! Well deserved. > > On Wed, Feb 10, 2021 at 1:54 PM Yun Gao > wrote: > > > Congratulations Roman! > > > > Best, > > Yun > > > > > > --Original Mail -- > > Sender:Till Rohrmann

Re: Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

2021-02-10 Thread Arvid Heise
Congrats! Well deserved. On Wed, Feb 10, 2021 at 1:54 PM Yun Gao wrote: > Congratulations Roman! > > Best, > Yun > > > --Original Mail -- > Sender:Till Rohrmann > Send Date:Wed Feb 10 20:53:21 2021 > Recipients:dev > CC:Khachatryan Roman , Roman Khachatryan < >

Re: Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

2021-02-10 Thread Yun Gao
Congratulations Roman! Best, Yun --Original Mail -- Sender:Till Rohrmann Send Date:Wed Feb 10 20:53:21 2021 Recipients:dev CC:Khachatryan Roman , Roman Khachatryan Subject:Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer Congratulations R

Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

2021-02-10 Thread Till Rohrmann
Congratulations Roman :-) Cheers, Till On Wed, Feb 10, 2021 at 1:01 PM Konstantin Knauf wrote: > Congratulations Roman! > > On Wed, Feb 10, 2021 at 11:29 AM Piotr Nowojski > wrote: > > > Hi everyone, > > > > I'm very happy to announce that Roman Khachatryan has accepted the > > invitation to >

Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

2021-02-10 Thread Konstantin Knauf
Congratulations Roman! On Wed, Feb 10, 2021 at 11:29 AM Piotr Nowojski wrote: > Hi everyone, > > I'm very happy to announce that Roman Khachatryan has accepted the > invitation to > become a Flink committer. > > Roman has been recently active in the runtime parts of the Flink. He is one > of the

[jira] [Created] (FLINK-21347) Extract interface out of ExecutionGraph for better testability

2021-02-10 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-21347: -- Summary: Extract interface out of ExecutionGraph for better testability Key: FLINK-21347 URL: https://issues.apache.org/jira/browse/FLINK-21347 Project: Flink

Re: [DISCUSS] FLIP-164: Improve Schema Handling in Catalogs

2021-02-10 Thread Timo Walther
Hi Jark, I don't think many users use WatermarkSpec. UniqueConstraint could cause some confusion but this mostly affects catalog or connector implementers. After deprecating the old APIs it should be obvious when an outdated interface is used. I'm fine with using a different name, do we have

[ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

2021-02-10 Thread Piotr Nowojski
Hi everyone, I'm very happy to announce that Roman Khachatryan has accepted the invitation to become a Flink committer. Roman has been recently active in the runtime parts of the Flink. He is one of the main developers behind FLIP-76 Unaligned Checkpoints, FLIP-151 Incremental Heap/FS State Backe

Re: [VOTE] FLIP-158: Generalized incremental checkpoints

2021-02-10 Thread Roman Khachatryan
+1 from myself (binding). As there were no vetoes in more than 72hrs and three binding +1 the FLIP is now accepted. Thanks! Regards, Roman On Wed, Feb 3, 2021 at 8:35 AM Arvid Heise wrote: > FLIP looks good to me. +1 > > On Wed, Feb 3, 2021 at 8:00 AM Piotr Nowojski > wrote: > > > I'm carry