Thanks Caizhi, this was very helpful.
// ah
From: Caizhi Weng
Sent: Thursday, August 26, 2021 10:41 PM
To: Hailu, Andreas [Engineering]
Cc: user@flink.apache.org
Subject: Re: 1.9 to 1.11 Managed Memory Migration Questions
Hi!
I've read the first mail again and discover that the direct memory
I guess the best option is to attach a debugger and set a breakpoint at the
NotSerializableException. There definitively has to be a non-serializable
component in that FlinkKafkaConsumer and it can only come from the
DeserializationSchema or Properties.
Maybe the consumer internally caches some val
> Is there a scenario where a task manager could fail but the number of
registered task managers metric reported by the job manager is not updated?
The "common" case would be where you have configured a /really/ large
heartbeat timeout, such that Flink does not notice that the TaskExecutor
has
Hi,
A couple of months ago we observed a scenario in our Flink deployment where the
‘numRegisteredTaskManagers’ job manager metric reported the presence of 3 task
managers, despite the fact that only 2 task managers were active at the time,
because one of the task managers had crashed. We obs
There's no inner classes, and none of the fields
of DebeziumAvroRegistryDeserializationSchema have an Avro schema, even when
expanded, including KafkaClusterConfig. KafkaClusterConfig is just composed
of Strings and Booleans.
DebeziumAvroRegistryDeserializationSchema has a field that initializes a
Thanks for clarifying that, Amit. Rolling updates with JobManagers and
TaskManagers coming from different Flink versions in the same Flink cluster
is not supported.
@Yang Wang Do you have any recommendations you
could share in this regard?
Best,
Matthias
On Fri, Aug 27, 2021 at 2:44 PM Amit Bha
Hi Matthias,
What you mention is a little tricky. When we create a new cluster it will
have its own volume (PVC) so sending savepoint/checkpoint data from volume
(PVC) of the older cluster to the newer cluster is a manual task. Also not
sure if savepoint/checkpoint data needs to be copied to the
hello i am deploying a flink application cluster with kubernetes HA mode, but i
am facing this recurrent problem and i didn't know how to solve it.
Any help would be appreciated.
this of the jobManager:
{"@timestamp":"2021-08-27T14:19:42.447+02:00","@version":"1","message":"Exception
occurr
The upgrade approach mentioned in my previous answer should also work in
the context of k8s and pods: Creating a Flink cluster having the newer
version should be done before migrating the job using a savepoint. But
maybe, I misunderstand your question. Do you have something in mind where
you upgrad
Hi Hemant,
One possible reason is that another Kafka consumer is using the same consumer
group id as the one in FlinkKafkaConsumer. You can try to use another group.id
in FlinkKafkaConsumer to validate this.
If it’s not group id’s problem, there are some Kafka consumer metrics [1] that
might b
Hi Matthias,
Thanks for the information but this upgrade is looking like on native
(physical/virtual) deployment.
I want to understand the upgrade strategies on kubernetes deployments where
Flink is running in pods. If you could help in that area it would be great.
Regards,
Amit Bhatia
On Thu, A
11 matches
Mail list logo
