How To subscribe a Kinesis Stream using enhance fanout?

Hello Flink Community! I'm currently coding on a project relying on AWS Kinesis. With the provided connector (flink-connector-kinesis_2.11;1.10.0), I can consume the message. But as the main stream is used among several other teams, I was required to use the enhance fanout of Kinesis.

Re: How To subscribe a Kinesis Stream using enhance fanout?

://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html > > On Wed, May 13, 2020 at 1:44 PM Xiaolong Wang > wrote: > >> Hello Flink Community! >> >> I'm currently coding on a project relying on AWS Kinesis. With the >> provided co

Flink failed to resume from checkpoint stored on S3

Deare community, One of my Flink job failed yesterday, and when I tried to resume from the latest checkpoint, following exceptions happen: ``` Log Type: jobmanager.err Log Upload Time: Wed Jul 22 09:04:24 + 2020 Log Length: 506 SLF4J: Class path contains multiple SLF4J bindings. SLF4J:

How to mount PVC volumes using Flink Native Kubernetes ?

Hi, I'm facing a tough question. I want to start a Flink Native Kubernetes job with each of the task manager pod mounted with an aws-ebs PVC. The first thought is to use the pod-template file to do this, but it soon went to a dead end. Since the pod-template on each of the task manager pod i

Flink on Native K8s jobs turn in to `SUSPENDED` status unexpectedly.

Hello, Recently our Flink jobs on Native K8s encountered failing in the `SUSPENDED` status and got restarted for no reason. Flink version: 1.13.2 Logs: ``` 2022-05-11 05:01:41 2022-05-10 21:01:41,771 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 17921

SQL-gateway Failed to Run

Hi, I've tested the Flink SQL-gateway to run some simple Hive queries but met some exceptions. Environment Description: Run on : Kubernetes Deployment Mode: Session Mode (created by a flink-kubernetes-operator) Steps to run: 1. Apply a `flinkdeployment` of flink session cluster to flink operator

How to resume a job from checkpoint with the SQL gateway.

Hi, I'm currently working on providing a SQL gateway to submit both streaming and batch queries. My question is, if a streaming SQL is submitted and then the jobmanager crashes, is it possible to resume the streaming SQL from the latest checkpoint with the SQL gateway ?

Re: How to resume a job from checkpoint with the SQL gateway.

able/sqlclient/#terminating-a-job > > Best, > Shammon FY > > > On Tue, Jul 18, 2023 at 9:56 AM Xiaolong Wang > wrote: > >> Hi, Shammon, >> >> I know that the job manager can auto-recover via HA configurations, but >> what if I want to upgrade the running F

[Bug-report]Flink-operator 1.6.0 repo does not exist yet

Hi, I noticed that the newest documentation of the flink-operator has pointed to v1.6.0, yet when using the `helm repo add flink-operator-repo https://downloads.apache.org/flink/flink-kubernetes-operator-1.6.0/` command to install, it turns out that the given URL does not exist. I suppose that 1.

Flink K8S operator does not support IPv6

Hi, I was testing flink-kubernetes-operator in an IPv6 cluster and found out the below issues: *Caused by: javax.net.ssl.SSLPeerUnverifiedException: Hostname > fd70:e66a:970d::1 not verified:* > > *certificate: sha256/EmX0EhNn75iJO353Pi+1rClwZyVLe55HN3l5goaneKQ=* > > *DN: CN=kube-apiserve

Re: Flink K8S operator does not support IPv6

Ok, thank you. On Tue, Aug 8, 2023 at 11:22 AM Peter Huang wrote: > We will handle it asap. Please check the status of this jira > https://issues.apache.org/jira/browse/FLINK-32777 > > On Mon, Aug 7, 2023 at 8:08 PM Xiaolong Wang > wrote: > >> Hi, >> >> I w

Re: Flink K8S operator does not support IPv6

FYI, adding environment variables of ` KUBERNETES_DISABLE_HOSTNAME_VERIFICATION=true` works for me. This env variable needs to be added to both the Flink operator and the Flink job definition. On Tue, Aug 8, 2023 at 12:03 PM Xiaolong Wang wrote: > Ok, thank you. > > On Tue, Aug 8, 2

Flink Kubernetes operator keeps reporting REST client timeout.

Hi, Recently I upgraded the flink-kubernetes-operator from 1.4.0 to 1.6.1 to use Flink 1.18. After that, the operator kept reporting the following exception: 2023-11-21 03:26:50,505 o.a.f.k.o.r.d.AbstractFlinkResourceReconciler [INFO > ][sn-push/sn-push-decision-maker-log-s3-hive-prd] Resource fu

Re: Flink Kubernetes operator keeps reporting REST client timeout.

Seems the operator didn't get restarted automatically after the configmap is changed. After a roll-out restart, the exception disappeared. Never mind this issue. Thanks. On Tue, Nov 21, 2023 at 11:31 AM Xiaolong Wang wrote: > Hi, > > Recently I upgraded the flink-kubernetes-opera

[flink-k8s-connector] In-place scaling up often takes several times till it succeeds.

Hi, I'm playing with a Flink 1.18 demo with the auto-scaler and the adaptive scheduler. The operator can correctly collect data and order the job to scale up, but it'll take the job several times to reach the required parallelism. E.g. The original parallelism for each vertex is something like b

Found an issue when using Flink 1.19's AsyncScalarFunction

Hi, I found a ClassNotFound exception when using Flink 1.19's AsyncScalarFunction. Stack trace: Caused by: java.lang.ClassNotFoundException: > org.apache.commons.text.StringSubstitutor > > at java.net.URLClassLoader.findClass(Unknown Source) ~[?:?] > > at java.lang.ClassLoader.loadClass(Unknown

Why this Flink batch SQL immediately produces records ?

Hi, team, I'm running a Flink SQL via Flink SQL gateway in the version of 1.20. The SQL reads from Hive and writes into Kafka but needs to join with a sub-query that queries out a problematic uuid and filter it out, it looks like this: INSERT INTO > kafka_sink > SELECT /*+ BROADCAST(t1) */ >

Re: Why this Flink batch SQL immediately produces records ?

Or Flink will send a delete record to Kafka once the sub-query is done ?