I restart flink job via savepoint. command as following:
cancel command:
/home/datadev/flink-1.12.2/flink-1.12.2/bin/flink cancel \
-yid application_1625497885855_698371 \
-s
hdfs://ztcluster/flink_realtime_warehouse/checkpoint/UserClickLogAll/savepoint
\
59cf6ccc83aa163bd1e0cd3304dfe06a
print s
Hi all, my flink job consume kafka topic A, and write to kafka topic B.
When i restart my flink job via savepoint, topic B have some duplicate
message. Any one can help me how to solve this problem? Thanks!
My Versions:
Flink 1.12.4
Kafka 2.0.1
Java 1.8
Core code:
env.enableCheckpointing(30);
Hi Leonard,
I am using flink 1.11.2 and using debezium-json to read CDC data generated
by debezium.
For each table, I convert the Kafka dynamic table to a retract stream and
finally that stream is converted to DataStream. Here's the sample
function
private DataStream getDataStream(String sql) {
1. I use the filesystem as the state backend, and the state should be in memory.
2. The mini-batch function is disabled.
3. Does mini-batch reduce memory usage? I found that the memory usage of the
overwindows grew fast and the JVM FunllGC was frequent. Tenured Gen occupies a
large amount of me
AFAIK, the ClusterClient should not be exposed through the public API.
Would you like to explain your use case and why you need to get the
web UI programmatically?
Best,
Yangze Guo
On Fri, Jul 30, 2021 at 9:54 PM Hailu, Andreas [Engineering]
wrote:
>
> Hello Yangze, thanks for responding.
>
> I'
Hi, Ayush
Thanks for the detailed description.
Before analyze the issue, I have two questions that which Flink and Flink CDC
version are you using? Is Flink CDC used in SQL or DataStream ?
That’s helpful if you can post you Flink CDC connector parameters.
Best,
Leonard
> 在 2021年7月29日，18:57，
Hi!
As the state grows the processing speed will slow down a bit. Which state
backend are you using? Is mini batch enabled[1]?
[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/config/#table-exec-mini-batch-enabled
Wanghui (HiCampus) 于2021年7月30日周五 下午3:59写道：
> Hi :
>
> W
