Yes. But the filter us usually a very light weight operation.
From: Mingliang Liu
Sent: Monday, June 3, 2024 7:16 PM
To: Colletta, Edward
Cc: mejri houssem ; user@flink.apache.org
Subject: Re: Implementing Multiple sink
NOTICE: This email is from an external sender - do not click on links or
I usually just reuse the stream, sending it to through different filters and
adding different sinks to the filtered streams.
Something like
streamWithMultipleConditions.filter(FilterForCondition1)
.addSink(SinkforCondtiton1);
streamWithMu
ds:[session],groupByType:bySession,aggregationKey:'3'}
{session:3,account:1,value:3200,groupByFields:[account],groupByType:byAccount,aggregationKey:'2'}
From: Colletta, Edward
Sent: Tuesday, January 25, 2022 1:29 PM
To: M Singh ; Caizhi Weng ;
User-Flink
Subject: RE: Apache Fin
, January 25, 2022 1:12 PM
To: Caizhi Weng ; User-Flink ;
Colletta, Edward
Subject: Re: Apache Fink - Adding/Removing KeyedStreams at run time without
stopping the application
NOTICE: This email is from an external sender - do not click on links or
attachments unless you recognize the sender and
A general pattern for dynamically adding new aggregations could be something
like this
BroadcastStream broadcastStream =
aggregationInstructions
.broadcast(broadcastStateDescriptor);
DataStream
streamReadyToAggregate = dataToAggregate
.connect(broadcast
If have jar files built using flink version 11.2 in dependencies, and I upgrade
my cluster to 11.6, is it safe to run the existing jars on the upgraded cluster
or should I rebuild all jobs against 11.6?
Thanks,
Eddie Colletta
Thanks, will try that.
From: Chesnay Schepler
Sent: Friday, August 20, 2021 8:06 AM
To: Colletta, Edward ; user@flink.apache.org
Subject: Re: failures during job start
NOTICE: This email is from an external sender - do not click on links or
attachments unless you recognize the sender and know
To: Colletta, Edward ; user@flink.apache.org
Subject: Re: failures during job start
NOTICE: This email is from an external sender - do not click on links or
attachments unless you recognize the sender and know the content is safe.
This exception means that a task was deployed, but the task that
Any help with this would be appreciated. Is it possible that this is a
data/application issue or a flink config/resource issue?
Using flink 11.2, java 11, session cluster, 5 nodes 32 cores each node.
I have an issue where starting a job takes a long time, and sometimes fails
with PartitionNot
is I'm not aware of any issue related to the
upgrading of the ZK version that may cause the leadership loss.
Thank you~
Xintong Song
On Sun, Jan 31, 2021 at 4:14 AM Colletta, Edward
mailto:edward.colle...@fmr.com>> wrote:
“but I'm not aware of any similar issue reported since
Thank you.
-Original Message-
From: Tzu-Li (Gordon) Tai
Sent: Monday, March 15, 2021 3:05 AM
To: user@flink.apache.org
Subject: Re: uniqueness of name when constructing a StateDescriptor
NOTICE: This email is from an external sender - do not click on links or
attachments unless you
The documentation for ValueStateDescriptor documents the name parameter as -
"name - The (unique) name for the state."
What is the scope of the uniqueness? Unique within an RichFunction instance?
Unique withing job? Unique within a session cluster?
I ask because I have several jobs that use a K
Using FsStateBackend.
I was under the impression that ValueState.value will serialize an object which
is stored in the local state backend, copy the serialized object and
deserializes it. Likewise update() would do the same steps copying the object
back to local state backend.And as a cons
You can use a tumbling processing time window with an offset of 13 hours + your
time zone offset.
https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/stream/operators/windows.html#tumbling-windows
https://ci.apache.org/projects/flink/flink-docs-release-1.12/api/java/org/apache/flink/
“but I'm not aware of any similar issue reported since the upgrading”
For the record, we experienced this same error on Flink 1.11.2 this past week.
From: Xintong Song
Sent: Friday, January 29, 2021 7:34 PM
To: user
Subject: Re: Flink 1.11 job hit error "Job leader lost leadership" or
"Resource
Using flink 11.2 on java 11, session cluster with 16 jobs running on aws ecs
instances. Cluster has 3 JMs and 3 TMs, separate zookeeper cluster has 3 nodes.
One of our taskmanagers crashed today with what seems to be rooted in a
zookeeper timeout. We are wondering if there is any tuning that
...@aliyun.com>>
Sent: Monday, December 21, 2020 11:28 AM
To: Colletta, Edward mailto:edward.colle...@fmr.com>>;
user@flink.apache.org<mailto:user@flink.apache.org>
Subject: Re: RE: checkpointing seems to be throttled.
This email is from an external source - exercise caution regarding link
Doh! Yeah, we set the state backend in code and I read the flink-conf.yaml
file and use the high-availability storage dir.
From: Yun Gao
Sent: Monday, December 21, 2020 11:28 AM
To: Colletta, Edward ; user@flink.apache.org
Subject: Re: RE: checkpointing seems to be throttled.
This email is
Thanks for the quick response.
We are using FsStateBackend, and I did see checkpoint files and directories in
the EFS mounted directory.
We do monitor backpressure through rest api periodically and we do not see any.
From: Yun Gao
Sent: Monday, December 21, 2020 10:40 AM
To: Colletta, Edward
Using session cluster with three taskmanagers, cluster.evenly-spread-out-slots
is set to true. 13 jobs running. Average parallelism of each job is 4.
Flink version 1.11.2, Java 11.
Running on AWS EC2 instances with EFS for high-availability.storageDir.
We are seeing very high checkpoint times
Thanks you for the response. We do see the heap actually shrink after
starting new jobs.
From: Matthias Pohl
Sent: Thursday, November 5, 2020 8:20 AM
To: Colletta, Edward
Cc: user@flink.apache.org
Subject: Re: a couple of memory questions
This email is from an external source - exercise
Using Flink 1.9.2 with FsStateBackend, Session cluster.
1. Does heap state get cleaned up when a job is cancelled?
We have jobs that we run on a daily basis. We start each morning and cancel
each evening. We noticed that the process size does not seem to shrink. We
are looking at the res
, I did that just for the test. For my prod code, going forward, I am
following flink’s rules for POJO types, adding static to any inner class, and
checking for any POJO warnings in the logs.
From: Arvid Heise
Sent: Sunday, October 11, 2020 3:46 PM
To: Colletta, Edward
Cc: Dawid Wysakowicz
: Dawid Wysakowicz mailto:dwysakow...@apache.org>>
Sent: Thursday, October 8, 2020 6:26 AM
To: Colletta, Edward mailto:edward.colle...@fmr.com>>;
user@flink.apache.org<mailto:user@flink.apache.org>
Subject: Re: state access causing segmentation fault
Hi,
It should be absolutely fine
Using Flink 1.9.2, Java, FsStateBackend. Running Session cluster on EC2
instances.
I have a KeyedProcessFunction that is causing a segmentation fault, crashing
the flink task manager. The seems to be caused by using 3 State variables in
the operator. The crash happens consistently after some
Using Flink 1.9.2, Java, FsStateBackend.
I was getting com.esotericsoftware.kryo.KryoException:
java.lang.NullPointerException on a value() operation on a ValueState variable
in a KeyedProcessFunction.
The object stored in state contained 2 PriorityQueue fields and the error
message indicated t
26 matches
Mail list logo
