Hey folks,
After digging into this a bit it does seem like Broadcast State would fit the
bill for this scenario and keeping the downstream operators up-to-date as
messages arrived in my Kafka topic.
My question is - is there a pattern for pre-populating the state initially? In
my case, I need
Hi Alexey,
I don't think the new reactive mode makes any changes to the
checkpoint/savepoint mechanism, at least not at the moment.
However, you might want to take a look at the unaligned checkpoint [1]. The
unaligned checkpoint is designed to be tolerant with back pressure. AFAIK,
this can work
Hi Alexey,
We also have the same problem running on Yarn using Flink 1.9.0.
JM log shows this:
We are also looking for a way to troubleshoot this problem.
Best regards.
Smile
Alexey Trenikhun wrote
> Hello,
>
> I periodically see in JM log (Flink 12.2):
>
> {"ts":"2021-05-15T21:10:36.325Z",
JM log shows this:
INFO org.apache.flink.yarn.YarnResourceManager - The
heartbeat of JobManager with id 41e3ef1f248d24ddefdccd1887947106 timed out.
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Hi Alexey & Smile,
JM & RM are located in the same process, thus it's unlikely a network
issue. Such timeouts are usually caused by one of the two endpoints not
responding timely.
Some common causes:
- The process is under severe GC pressure. You can check the GC logs for
the pressure.
- Insuffic
Thanks for your help Timo and Fabian,
Got it working with Timo’s suggestion.
On Fri, May 14, 2021 at 6:14 AM Fabian Paul
wrote:
> Hi John,
>
> Can you maybe share more code about how you build the DataStrean?
> It would also be good to know against which Flink version you are testing.
> I just
>
Hi Alexey,
Flink supports rescaling from a normal checkpoint if you are not changing
your application too much. So if normal checkpointing works, you can just
use that for rescaling by using Retained Checkpoints and supply the path on
resume at the place where you supplied the savepoint path befor
Hi Rion,
I think FLIP-150[1] should be able to solve this scenario.
Since FLIP-150 is still under discussion, for now a temporary method come
to me might be
1. Write a first job to read the kafka and update the broadcast state of some
operator. The job
would keep the source alive after all the
You check if there is a configuration option listed here:
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/
If it is, you can add it to config/flink-config.yaml.
Maybe others have other pointers.
Otherwise you will need to use Table API instead of SQL Client.
Hi Xintong,
Does reactive mode need checkpoint for re-scheduling ?
Thanks,
Alexey
From: Xintong Song
Sent: Sunday, May 16, 2021 7:30:15 PM
To: Flink User Mail List
Subject: Re: reactive mode and back pressure
Hi Alexey,
I don't think the new reactive mode make
10 matches
Mail list logo
