Re: Using Flink's Kubernetes API inside Java

Hi Roman, In the operator I mentioned I see logic like the one here: https://github.com/wangyang0918/flink-native-k8s-operator/blob/a60a9826d4bcdaa4f23cf296d95954b9f6f328c3/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkApplicationController.java#L169 For instance, DefaultClu

Re: Understanding recovering from savepoint / checkpoint with additional operators when chaining

Hi, Just to clarify, you are recovering from a savepoint, not a retained checkpoint, right? And how are you setting the operator IDs? You mentioned that with allowNonRestoredState set to false recovery fails. Does it succeed with this flag set to true? Answering your questions: Each operator stat

Re: Using Flink's Kubernetes API inside Java

Hi Alexis, Have you looked at flink-on-k8s-operator [1]? It seems to have the functionality you need: https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/blob/0310df76d6e2128cd5d2bc51fae4e842d370c463/controllers/flinkcluster_reconciler.go#L569 I couldn't find many Flink-specific classes

Re: Flink State Processor API Example - Java

Hi Sandeep, Could you provide the error stack trace and Flink version you are using? Regards, Roman On Fri, Jul 2, 2021 at 6:42 PM Sandeep khanzode wrote: > > Hi Guowei, > > I followed the document, but somehow, I am unable to get a working Java > example for Avro state. > > So, I tried to sim

Understanding recovering from savepoint / checkpoint with additional operators when chaining

Hello Flink, I'm trying to understand the state recovery mechanism when there are extra stateless operators. I'm using flink-sql, and I tested a 'select `number_col` from source' query, where the stream graph looks like: `source (stateful with fixed uid) -> [several stateless operators transla

Re: Flink State Processor API Example - Java

Hi Guowei, I followed the document, but somehow, I am unable to get a working Java example for Avro state. So, I tried to simply use the Java SpecificRecords created by Avro Maven Plugin and inject. Now, that works correctly, but I use Avro 1.7.7 since it is the last version that I saw which d

Re: Memory Usage - Total Memory Usage on UI and Metric

When we run Flink jobs in EMR (typically batch, though) we disable the pmem (permanent memory) and vmem (virtual memory) checks. This was initially done for much older versions of Flink (1.6???), where the memory model wasn’t so well documented or understood by us. But I think the pmem check mi

Memory Usage - Total Memory Usage on UI and Metric

Hi, I am running a streaming job (Flink 1.9) on EMR on yarn. Flink web UI or metrics reported from prometheus shows total memory usage within specified task manager memory - 3GB. Metrics shows below numbers(in MB) - Heap - 577 Non Heap - 241 DirectMemoryUsed - 852 Non-heap does rise gradually, s

Re: owasp-dependency-check is flagging flink 1.13 for scala 2.12.7

Its unlikely to be relevant for you since the vulnerability only affects the scaladocs, i.e., documentation. On 7/2/2021 2:10 PM, Debraj Manna wrote: Hi, I was running owasp-dependency-check in a java application based on flink-1.13.0 (scala

Re: Job Recovery Time on TM Lost

Could you share the full logs with us for the second experiment, Lu? I cannot tell from the top of my head why it should take 30s unless you have configured a restart delay of 30s. Let's discuss FLINK-23216 on the JIRA ticket, Gen. I've now implemented FLINK-23209 [1] but it somehow has the probl

owasp-dependency-check is flagging flink 1.13 for scala 2.12.7

Hi, I was running owasp-dependency-check in a java application based on flink-1.13.0 (scala 2.12). scala 2.12.7 was getting flagged for this

Re: Job Recovery Time on TM Lost

Thanks for sharing, Till and Yang. @Lu Sorry but I don't know how to explain the new test with the log. Let's wait for others' reply. @Till It would be nice if JIRAs could be fixed. Thanks again for proposing them. In addition, I was tracking an issue that RM keeps allocating and freeing slots a