Relation between Two Phase Commit and Kafka's transaction aware producer

Hi team, I just have a bit of confusion where Two Phase Commit and Kafka's transaction aware producer using transaction.id and enable.autocommit plays together what I understand of Flink checkpoint (correct me if I'm wrong) is that it saves the transaction ID as well as the consumer's commit offse

Re: Native Kubernetes annotation parsing problem

I think it will be more generic question of how I inject IAM roles in Native Kubernetes pods I'm using Kubeiam and seems the namespace annotation doesn't work On Mon, Feb 8, 2021 at 2:30 PM Kevin Kwon wrote: > Hi team, I'm using Native Kubernetes

Native Kubernetes annotation parsing problem

Hi team, I'm using Native Kubernetes annotation config *kubernetes.jobmanager.annotations* and I'm facing some problem with parsing. I use annotation *iam.amazonaws.com/role:'arn:aws:iam:::role/XX/ '* but seems no

Re: Using ClusterIP with KubernetesHAServicesFactory

appear. > However, you still have > a limitation. You could not use "flink cancel/list/savepoint" to interact > with the Flink cluster. Because > the network is not reachable. But you could do it via rest API if you have > configured the ingress. > > > [1]. https://i

Re: Using ClusterIP with KubernetesHAServicesFactory

Ok it seems that this check is ran by the K8S CLI which in my case runs in a CICD cluster If this check should happen, I'd like to override this value with the ingress address Is there a way I can override the rest address that the K8S CLI taps on? On Fri, Jan 15, 2021 at 7:55 PM Kevin

Using ClusterIP with KubernetesHAServicesFactory

Hi team, I have some concerns using ClusterIP with Kubernetes Native Deployment with KubernetesHAServiceFactory for High Availability It seems that the KubernetesHAServicesFactory taps on the Service of the the Flink K8S Native Cluster to access the JobManager's availability, although I have some

Questions regarding DDL and savepoints

I have a question regarding DDLs if they are considered operators and can be savepointed For example CREATE TABLE mytable ( id BIGINT, data STRING WATERMARK(...) ) with ( connector = 'kafka' ) If I create the table like above, save&exit and resume application, will the application start

Duplication error on Kafka Connector Libraries

Hi community, I'm testing out 1.12-SNAPSHOT in master branch I built my application with library 'flink-connector-kafka' but when I start the app, I get Caused by: org.apache.kafka.common.KafkaException: class org.apache.kafka.common.serialization.ByteArrayDeserializer is not an instance of org.a

Is there a way we can specify operator ID for DDLs?

For SQLs, I know that the operator ID assignment is not possible now since the query optimizer may not be backward compatible in each release But are DDLs also affected by this? for example, CREATE TABLE mytable ( id BIGINT, data STRING ) with ( connector = 'kafka' ... id = 'mytable'

Re: Hi I'm having problems with self-signed certificiate trust with Native K8S

K8S resources? On Fri, Nov 20, 2020 at 9:23 PM Kevin Kwon wrote: > Hi I am using MinIO as a S3 mock backend for Native K8S > > Everything seems to be fine except that it cannot connect to S3 since > self-signed certificates' trusted store are not cloned in Deployment > resources >

Hi I'm having problems with self-signed certificiate trust with Native K8S

Hi I am using MinIO as a S3 mock backend for Native K8S Everything seems to be fine except that it cannot connect to S3 since self-signed certificates' trusted store are not cloned in Deployment resources Below is in order, how I add the trusted keystore by using keytools and how I run my app wit

Hi all I'm having trouble with spinning up native Kubernetes cluster

Hi guys, I'm trying out the native K8s cluster and having trouble with SSL I think. I use *k3d* as my local cluster for experiment here's how I launch my cluster k3d cluster create docker run \ -u flink:flink \ -v /Users/user/.kube:/opt/flink/.kube \ --network host \ --entry-point /bin/bash \

Re: I have some interesting result with my test code

Long): Long = { order.getTimestamp } }) On Tue, Nov 3, 2020 at 12:01 AM Kevin Kwon wrote: > Hi guys, I've been recently experimenting with end-to-end testing > environment with Kafka and Flink (1.11) > > I've setup an infrastructure with Docker Compo

I have some interesting result with my test code

Hi guys, I've been recently experimenting with end-to-end testing environment with Kafka and Flink (1.11) I've setup an infrastructure with Docker Compose composed of single Kafka broker / Flink (1.11) / MinIO for checkpoint saves Here's the test scenario 1. Send 1000 messages with manual timest

Some questions regarding operator IDs

Hi team I'm subscribing 2 topics from Kafka Consumer, joining them and publishing back to a new topic via KafkaProducer (with Exactly Once semantic) As it's highly recommended to set uid for each operator, I'm curious how this works. For example, val topicASource = env .addSource(topicAConsume