Question about MapState size

2022-01-21 Thread Abdul Rahman
Hello, I have a streaming application that has an operator based on the KeyedCoProcessFunction. The operator has a MapState object. I store some data in this operator with a fixed ttl. I would like to monitor the size/count of this state over time since its related to some operational metrics we

Re: Examples / Documentation for Flink ML 2

2022-01-21 Thread Dong Lin
Hey Bonino, Sounds great. Since we have not set up the website for Flink ML yet, how about we create PRs for https://github.com/apache/flink-ml and put those Markdown files under flink-ml/docs? Best Regards, Dong On Sat, Jan 22, 2022 at 12:25 AM Bonino Dario wrote: > Hi Dong, > > We assembled

Re: Flink 1.14.2 - Log4j2 -Dlog4j.configurationFile is ignored and falls back to default /opt/flink/conf/log4j-console.properties

2022-01-21 Thread Tamir Sagi
Yes, Thank you! I will handle that. Best, Tamir From: Yang Wang Sent: Friday, January 21, 2022 5:11 AM To: Tamir Sagi Cc: user@flink.apache.org Subject: Re: Flink 1.14.2 - Log4j2 -Dlog4j.configurationFile is ignored and falls back to default /opt/flink/conf/l

Re: Examples / Documentation for Flink ML 2

2022-01-21 Thread Bonino Dario
Hi Dong, We assembled a first, very small, Markdown document providing a jump-start description using a kMeans example. I could already share it with you to check if we are pointing in the right direction. I had a look at the Flink contribution guidelines, however the flink-ml project is  som

Re: Unhandled exception in flink 1.14.2

2022-01-21 Thread John Smith
Ok I see. So I guess in 1.4.3, one was fixed but broke something else? Because in 1.4.2 it feels like it worked after a refresh but with 1.4.3 it always fails looks like. On Fri., Jan. 21, 2022, 3:29 a.m. Chesnay Schepler, wrote: > What you are seeing in 1.14.3 is a separate bug, that behaves

Window function - flush on job stop

2022-01-21 Thread Lars Skjærven
We're doing a stream.keyBy().window().aggregate() to aggregate customer feedback into sessions. Every now and then we have to update the job, e.g. change the key, so that we can't easlily continue from the previous state. Cancelling the job (without restarting from last savepoint) will result in l

Flink 1.14 metrics : taskmanager host name missing

2022-01-21 Thread Mayur Gubrele
Hello, We recently upgraded our Flink cluster to 1.14 and noticed that all the taskmanager metrics we receive in our Prometheus data source get host IPs instead of hostnames, which was the case earlier before we moved to 1.14. I see on the flink dashboard as well under taskmanager details, host I

[DISCUSS] Deprecate/remove Twitter connector

2022-01-21 Thread Martijn Visser
Hi everyone, I would like to discuss deprecating Flinks' Twitter connector [1]. This was one of the first connectors that was added to Flink, which could be used to access the tweets from Twitter. Given the evolution of Flink over Twitter, I don't think that: * Users are still using this connecto

Re: [DISCUSS] Future of Per-Job Mode

2022-01-21 Thread Konstantin Knauf
Thanks Thomas & Biao for your feedback. Any additional opinions on how we should proceed with per job-mode? As you might have guessed, I am leaning towards proposing to deprecate per-job mode. On Thu, Jan 13, 2022 at 5:11 PM Thomas Weise wrote: > Regarding session mode: > > ## Session Mode > *

Re: Unhandled exception in flink 1.14.2

2022-01-21 Thread Chesnay Schepler
What you are seeing in 1.14.3 is a separate bug, that behaves very similarly. https://issues.apache.org/jira/browse/FLINK-25732 On 20/01/2022 22:11, John Smith wrote: As per another recent thread. This is still an issue. On Wed, 19 Jan 2022 at 06:36, Chesnay Schepler wrote: This is a ser

Re: Issue with Flink UI for Flink 1.14.0

2022-01-21 Thread Chesnay Schepler
While FLINK-24550 was indeed fixed unfortunately a similar bug was also introduced (https://issues.apache.org/jira/browse/FLINK-25732). On 20/01/2022 21:18, Peter Westermann wrote: Just tried this again with Flink 1.14.3 since https://issues.apache.org/jira/browse/FLINK-24550 is listed as fix