Re: Not able to connect with Elasticsearch using PyFlink and connector-jar

I think you should use [1] or [2] instead. See [3] for more details. PS: This question is more fit for the user mailing list. Regards, Dian [1] https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-elasticsearch6_2.11/1.14.4/flink-sql-connector-elasticsearch6_2.11-1.14.4.jar

Re: BigQuery connector debugging

Thanks Robert! All working now. Turns out an incorrect log4j2 configuration was slurping up a warning about converting org.apache.avro.util.Utf8 to the internal flink data type. I've drawn some inspiration from flink-avro and added a converter that converts and re-orders the Avro encoded BigQuery

Re: The flink-kubernetes-operator vs third party flink operators

Thanks for the interest on the flink-kubernetes-operator project. I believe you could leave a comment in the ticket FLINK-27049. If the reporter has not start working on this ticket, then you could be assigned. Best, Yang Hao t Chang 于2022年4月6日周三 06:30写道： > Hi Gyula > > > > Thanks for the reply

RE: The flink-kubernetes-operator vs third party flink operators

Hi Gyula Thanks for the reply. I look forward to making contributions. I am assuming if an issue, for example this one, has no assignee, th

Re: Question About Histograms

Hello Prasanna, Thanks for your response, Could you elaborate on what you meant by "overriding the Prometheus Histogram class provided "? if possible with any samples? Regards, Anil On Tue, Apr 5, 2022 at 1:11 AM Prasanna kumar wrote: > Anil, > > Flink Histograms are actually summaries .. You n

Re: Submit jobs via Rest API and deploy Flink on a running Kubernetes cluster (Native way)

Hi Burcu, Nope. Submitting a job via Flink REST API requires setting up the JM first, and for application mode, there's no JM set up for you in advance. If you do want to implement similar things, you essentially re-implement the Flink client for application mode, which is of no point. Best, Z

Why first op after union cannot be chained?

Dear all, I was recently investigating why the chaining behavior of a Flink SQL job containing union ops is a bit surprising. The SQL, simplified to the extreme, is as below: CREATE TABLE datagen_source (word VARCHAR) WITH ('connector' = 'datagen', 'rows-per-second' = '5'); CREATE TA

RE: Submit jobs via Rest API and deploy Flink on a running Kubernetes cluster (Native way)

Thank you for your quick reply Zhanghao. I will update my design according to your comments. Best regards, Burcu From: Zhanghao Chen [mailto:zhanghao.c...@outlook.com] Sent: Tuesday, April 5, 2022 4:16 PM To: Burcu Gul POLAT EGRI ; user@flink.apache.org Subject: [*Newsletter*] Re: Submit jobs vi

Re: BIGDECIMAL data handling

Hi Martijn, I am using flink version 1.11.0. The flink application code snippet is like: [image: image.png] The Error I am receiving while providing BIGDECIMAL as datatype is : [image: image.png] Can I use unregistered structured custom data types in DDLs like Create Table? Thanks and Regards,

Re: BIGDECIMAL data handling

Hi Anitha, Looking at Bigquery's documentation, they're aliasing it as a BIGDECIMAL [1]. According to Flink's documentation, you can create an unregistered structured type as an user-defined data type [2]. You're mentioning that you've failed to implement this, but what is the issue that you're ru

BIGDECIMAL data handling

Hi, I am doing a developed a Fink connector for GCP BigQuery. When we are reading BIGNUMERIC data from BigQuery table, we didn't find any matching data types in Flink. Also failed to implement user defined data type for BIGNUMERIC. Please let me know if there is any way to handle this. Thanks

Re: Submit jobs via Rest API and deploy Flink on a running Kubernetes cluster (Native way)

FYI, I've just created a new JIRA ticket on extending /jars/:jarid/run API to support setting Flink configs: https://issues.apache.org/jira/browse/FLINK-27060. Best, Zhanghao Chen From: Burcu Gul POLAT EGRI Sent: Tuesday, April 5, 2022 18:20 To: Zhanghao Chen ;

Re: Submit jobs via Rest API and deploy Flink on a running Kubernetes cluster (Native way)

Hi Burcu, Thanks for elaborating more details! 1. [Burcu] Do you mean a running Flink container which is started as jobmanager is enough for REST server? Is this execution like a session mode? Because as far as I understand from Flink documentation samples, after executing jobs in applicati

Re: WatermarkStrategy for IngestionTime

Hi, IngestionTime is usually used when the records don't have a proper event time associated with it, but the job has a long topology, and users want to persist the (time)order of events as they arrive in the system. In that sense, you can use the regular event time watermark strategies also for i

Re: Flink metric

Hi, multiple records are in the system at the same time, because Flink is buffering records in various components, for efficiency reasons. That's why you see that an individual record might have a latency of ~100ms, while Flink is processing many more messages. On Tue, Apr 5, 2022 at 12:54 PM lo

Re: BigQuery connector debugging

Hi Matt, At first glance your code looks fine. I guess you'll need to follow the codepaths more with the debugger. Have you made sure that "reachedEnd()" returns false? On Tue, Apr 5, 2022 at 9:42 AM Matthew Brown wrote: > Hi all, > > I'm attempting to build a Table API connector for BigQuery

Flink metric

Hi,I'm trying to work with monitoring a job on Flink. Just a simple process: a data source from kafka joining with a data stream table (load from database).I've already enabled the latency tracking like the instruction from:  https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/met

RE: Submit jobs via Rest API and deploy Flink on a running Kubernetes cluster (Native way)

Thank you Zhanghao, I have some questions according to your comments. My comments are inlined… Best regards, Burcu From: Zhanghao Chen [mailto:zhanghao.c...@outlook.com] Sent: Tuesday, April 5, 2022 10:33 AM To: Burcu Gul POLAT EGRI ; user@flink.apache.org Subject: [*Newsletter*] Re: Submit job

FlinkKafkaProducer - Avro - Schema Registry

Hi guys, I’m working on a solution where I ingest Kafka Records and I need to sink them to another topic using Avro and Schema Registry. The problem I’m facing, is that I can’t find a suitable configuration that actually works for me. I’m going to explain. 1. I have a KafkaSource that cons

BigQuery connector debugging

Hi all, I'm attempting to build a Table API connector for BigQuery using the BigQuery Storage API ( https://cloud.google.com/bigquery/docs/reference/storage). I've got a base structure built out at https://github.com/mnbbrown/bigquery-connector There's a couple of things I have to do yet like cor

Re: Submit jobs via Rest API and deploy Flink on a running Kubernetes cluster (Native way)

Hi Burcu, 1. No matter which deployment mode you are using, to submit a Job via Flink REST API, you'll need to firstly set the Flink REST server up, and that essentially means set up the Flink JM ahead of time. 1. For submitting job via Flink REST API, you could refer to https://nightli