RE: Access to CheckpointStatsCounts

Thanks From: Robert Metzger [mailto:rmetz...@apache.org] Sent: Donnerstag, 5. Dezember 2019 10:55 To: Tan, Min Cc: vino yang; user Subject: [External] Re: Access to CheckpointStatsCounts Hey Min, If checking for empty map states works for you, this could be an option, yes. Alternatively, check

RE: Access to CheckpointStatsCounts

Dear Robert, Thank you very much for sending your reply. What we try to achieve is that 1) In a normal situation, checkpoints or save points are preserved, an application restarts from one of these paths (with configurations are kept in Map states). 2) Sometimes, e.g. during a vers

RE: Access to CheckpointStatsCounts

Many thanks for sending your reply. It is not for monitoring but for configuration. For a job starting from an empty status, we like to load the fresh configurations. For a job recovering from a checkpoint, we like to rely on the checkpoint. Regards, Min From: vino yang [mailto:yanghua1...@gm

Access to CheckpointStatsCounts

Hi, Just wonder how to access the CheckpointStatsCoutns from the main method of a job? We need to detect if a job recovers from a checkpoint or starts from an empty status. Regards, Min E-mails can involve SUBSTANTIAL RISKS, e.g. lack of confidentiality, potential manipulation of contents a

Savepoints and checkpoints

Hi, Are Flink savepoints and checkpoitns still vlaid after some data entity changes e.g. Kafka topic name changes? I expect the answer is "No"? Similarly, are Flink savepoints and checkpoitns still valid after some job graph changes e.g. one stateful operator splits into two? I expect the answer

RE: Does operator uid() have to be unique across all jobs?

Thank you for your reply. Any tool enables us to inspect (list) statically all the "uid"ed operators or all the operators? for a jar? Also addSink and addSource are not on the operator list https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/ But they both have an

RE: Does operator uid() have to be unique across all jobs?

Thanks for your reply. Our sources and sinks are connected to Kafka, therefore they are statful. We did not set uid on them but only name(). The log says Caused by: java.lang.IllegalStateException: Failed to rollback to checkpoint/savepoint file:/var/flink/data-remote/savepoint-00-dae01410

RE: Does operator uid() have to be unique across all jobs?

Thank you very much for your helpful response. Our new production release complains about the an uid mismatch (we use exactly once checkpoints). I hope I understand your correctly: map and print are certainly stateless, therefore no uid is required. What about addSink and addSoure? Do they need

RE: Does operator uid() have to be unique across all jobs?

Hi, I have some simple questions on the uid as well. 1) Do we add a uid for every operator e.g. print(), addSink and addSource? 2) For chained operators, do we need to uids for each operator? Or just the last operator? e.g. .map().uid("some-id").print().uid("print-id"); Rega

Job recovery from a checkpoint

Hi, We can get a job recovery from a save point nicely after a restart of our flink cluster using bin/flink run -s :savepointPath [:runArgs] The previous job states are recovered after this reload. I expect I do something similar to recover a flink from a checkpoint location after a restart of o

RE: Recovery from job manager crash using check points

Thanks for the helpful reply. One more question, does this zookeeper or HA requirement apply for a savepoint? Can I bounce a single jobmanager cluster and rerun my flink job from its previous states with a save point directory? e.g. ./bin/flink run myJob.jar -s savepointDirectory Regards, Min

Recovery from job manager crash using check points

Hi, I can use check points to recover Flink states when a task manger crashes. I can not use check points to recover Flink states when a job manger crashes. Do I need to set up zookeepers to keep the states when a job manager crashes? Regards Min E-mails can involve SUBSTANTIAL RISKS, e.g. l

Flink end to end intergration test

Hi, I am new to Flink, at least to the testing part. We need an end to end integration test for a flink job. Where can I find documentation for this? I am envisaging a test similar to that: 1) Start a local job instance in an IDE or maven test 2) Fire event jsons to the data source

RE: Re: Default Kafka producers pool size for FlinkKafkaProducer.Semantic.EXACTLY_ONCE

Many thanks for your replies. After I increased MinPauseBetweenCheckpoints and moved to a memory backend for checkpoint. It has disappeared. Thank you both again for your help. Regards, Min From: Piotr Nowojski [mailto:pi...@ververica.com] Sent: Donnerstag, 11. April 2019 15:01 To: Fabian Hue

RE: Re: Lantency caised Flink Checkpoint on EXACTLY_ONCE mode

Many thanks for your quick reply. 1) My implementation has no commits. All commits are done in FlinkKafkaProducer class I envisage. KeyedSerializationSchemaWrapper keyedSerializationSchemaWrapper = new KeyedSerializationSchemaWrapper(new SimpleStringSchema()); new FlinkKafkaProducer("t

Default Kafka producers pool size for FlinkKafkaProducer.Semantic.EXACTLY_ONCE

Hi, I keep getting exceptions "org.apache.flink.streaming.connectors.kafka.FlinkKafkaException: Too many ongoing snapshots. Increase kafka producers pool size or decrease number of concurrent checkpoints." I understand that DEFAULT_KAFKA_PRODUCERS_POOL_SIZE is 5 and need to increase this size

Lantency caised Flink Checkpoint on EXACTLY_ONCE mode

Hi, I have a simple data pipeline of a Kafka source, a flink map operator and a Kafka sink. I have a quick question about latency caused by the checkpoint on the exactly once mode. Due to the changes are committed and visible on a checkpoint completion, so the latency could be as long

Default Kafka producers pool size for FlinkKafkaProducer.Semantic.EXACTLY_ONCE

Hi, I keep getting exceptions "org.apache.flink.streaming.connectors.kafka.FlinkKafkaException: Too many ongoing snapshots. Increase kafka producers pool size or decrease number of concurrent checkpoints." I think that DEFAULT_KAFKA_PRODUCERS_POOL_SIZE is 5 and need to increase this size. W

Flink to MQ connector with checkpoint support for exctly once semantics

Hi, Our Flink jobs need to read messages from IBM MQ and write messages into IBM. Just wonder if there are already some MQ connectors with two phase committee sink Function or Checkpoint Listener and checkpoint function implemented to support the exactly once semantics. Many thanks in advance.

Externalised checkpoint keeps ValueState after a crash of a Flink cluster

Hi, I have a question about to keep a ValueState after a Flink 1.7.2 cluster is crashed. My Flink job is simple 1) read dummy events (an event only has a string Id) from a Kafka source. 2) do a count on input events and save it as a ValueState 3) setup an externalized checkpoint running every

Flink 1.7.0 HA based on zookeepers

Hi, I have a simple HA setting with Flink 1.7.0: Node1 (active master, active slave) Node2 (standby master, active slave) Step 1, start-cluster.sh from Node1, no problem Step 2, manually kill the active master on Node1, no problem and the standby master become active Step 3, bin/jobmanager.sh st

Zookeeper shared by Flink and Kafka

Hi, I am new to Flink. I have a question: Can a zookeeper cluster be shared by a flink cluster and a kafka cluster? Regards, Min Check out our new brand campaign: www.ubs.com/together E-mails can involve SUBSTANTIAL RISKS, e.g. lack of confidentiality, potential manipulation of contents and/o

RE: Re: Use event time

Many thanks for sending your email. Does this mean that the event time only impacts on the event selection for a time window? Without use of a time window, the event time has no impact on the order of any records/events? Is my understanding correct? Thank you very much for your help. Regards

Use event time

Hi, I am new to Flink. I have the following small code to use the event time. I did not get the result expected, i.e. it print out events in the order of event time. Did I miss something here? Regards, Min --Event time-- public static void main(String[] args)

A question on the Flink "rolling" FoldFunction

Hi, I am new to Flink. I have a question on this "rolling" fold function. If its parallelism is large than one, does the "rolling" order remains the same? i.e. it is always keep the "1-2-3-4-5" on an increasing sequence. Regards, Min ---