Re: Broadcasting control messages to a sink

Hey Piotr, Thanks for your help! The main thing I was missing was the .broadcast partition operation on a stream (searching for “broadcasting” obviously brought up the broadcast state pattern). This coupled with my misunderstanding of an error in my code as being an error in Flink code resulted

PyFlink:

Dear list, I’m having my first go at using Flink and quickly stumbled over a problem I find no easy way around. I hope you may help me. I try to read an avro encoded kafka topic. Doing so, I do get a NoClassDefFoundError. Org.apache.avro.SchemaBuilder could not be found, but this should be inc

Re: Flink Kubernetes / Helm

We use the Ververica Platform and have built an operator for it here[1] :) and use Helm with it as well. Best, Austin [1]: https://github.com/fintechstudios/ververica-platform-k8s-operator On Fri, Oct 16, 2020 at 3:12 PM Dan Hill wrote: > What libraries do people use for running Flink on Kube

Flink Kubernetes / Helm

What libraries do people use for running Flink on Kubernetes? Some links I've found: - Flink official documentation - Ververica documentation

Un-ignored Parsing Exceptions in the CsvFormat

Hey all, I'm ingesting CSV files with Flink 1.10.2 using SQL and the CSV Format[1]. Even with the `ignoreParseErrors()` set, the job fails when it encounters some types of malformed rows. The root cause is indeed a `ParseException`, so I'm wondering if there's anything more I need to do to ignore

ZooKeeper connection SUSPENDING

Hi, Related to https://mail-archives.apache.org/mod_mbox/flink-dev/201709.mbox/%3CCA+faj9yvPyzmmLoEWAMPgXDP6kx+0oed1Z5k4s3K9sgiCFyb=w...@mail.gmail.com%3E and https://issues.apache.org/jira/browse/FLINK-10052, I was wondering if there's a way to prevent Flink instances from failing while doing

Re: Issue with Flink - unrelated executeSql causes other executeSqls to fail.

Hi Dan, I think it is a good idea to use an exponential backoff strategy in the RpcGatewayRetriever. So from my side you can open an issue and a PR for fixing it. Cheers, Till On Fri, Oct 16, 2020 at 7:24 PM Dan Hill wrote: > To be clear, I'd be fine coding this. > > On Fri, Oct 16, 2020 at 9:

Re: Effect of renaming a primary key

>Yes, I think Flink still can infer the primary key from renamed pk column, the state and checkpoint does not effect by the rename action. Thanks for the info, that's good news. >but It’s unusual the data from Kafka is unique as I know I'm using Debezium CDC so the PK is unique per row update. T

Error in my streamtableExample(java style),Thanks for your help

I want to write an StreamTableExample(java style) of https://github.com/apache/flink/blob/master/flink-examples/flink-examples-table/src/main/scala/org/apache/flink/table/examples/scala/basics/StreamTableExample.scala My code is: https://paste.ubuntu.com/p/WcBb7Rr6gb/ Could you tell me why my c

Re: [DISCUSS] Remove flink-connector-filesystem module.

@Seth: Earlier in this discussion it was said that the BucketingSink would not be usable in 1.12 . On 10/16/2020 4:25 PM, Seth Wiesman wrote: +1 It has been deprecated for some time and the StreamingFileSink has stabalized with a large number of formats and features. Plus, the bucketing sink o

Re: [DISCUSS] Remove flink-connector-filesystem module.

+1 It has been deprecated for some time and the StreamingFileSink has stabalized with a large number of formats and features. Plus, the bucketing sink only implements a small number of stable interfaces[1]. I would expect users to continue to use the bucketing sink from the 1.11 release with futur

Re: TaskManager takes abnormally long time to register with JobManager on Kubernetes for Flink 1.11.0

Done, you are assigned now Weike. Cheers, Till On Fri, Oct 16, 2020 at 1:33 PM DONG, Weike wrote: > Hi Till, > > Thank you for the kind reminder, and I have created a JIRA ticket for this > issue https://issues.apache.org/jira/browse/FLINK-19677 > > Could you please assign it to me? I will try

Re: In 1.11.2/flinkSql/batch, tableEnv.getConfig.setNullCheck(false) seems to break group by-s

Hi Jon, I would not recommend to use the configuration parameter. It is not deprecated yet but can be considered legacy code from before we reworked the type system. Regards, Timo On 16.10.20 13:23, Kurt Young wrote: Yes, I think this is a bug, feel free to open a jira and a pull request.

Re: TaskManager takes abnormally long time to register with JobManager on Kubernetes for Flink 1.11.0

Hi Till, Thank you for the kind reminder, and I have created a JIRA ticket for this issue https://issues.apache.org/jira/browse/FLINK-19677 Could you please assign it to me? I will try to submit a PR this weekend to fix this : ) Sincerely, Weike On Fri, Oct 16, 2020 at 5:54 PM Till Rohrmann wr

Re: In 1.11.2/flinkSql/batch, tableEnv.getConfig.setNullCheck(false) seems to break group by-s

Yes, I think this is a bug, feel free to open a jira and a pull request. Best, Kurt On Fri, Oct 16, 2020 at 4:13 PM Jon Alberdi wrote: > Hello to all, > > on flink-1.11.2 the program written at > https://gist.github.com/yetanotherion/d007fa113d97411226eaea4f20cd4c2d > > creates unexpected sta

Re: TaskManager takes abnormally long time to register with JobManager on Kubernetes for Flink 1.11.0

Great, thanks a lot Weike. I think the first step would be to open a JIRA issue, get assigned and then start on fixing it and opening a PR. Cheers, Till On Fri, Oct 16, 2020 at 10:02 AM DONG, Weike wrote: > Hi all, > > Thanks for all the replies, and I agree with Yang, as we have found that > f

Password usage in ssl configuration

Hello, I have a query regarding the ssl configuration in flink. In flink with ssl enabled, flink-conf.yaml configuration file will contain the cleartext passwords for keystore and truststore files. Suppose if any attacker gains access to this configuration file, using these passwords keystore a

In 1.11.2/flinkSql/batch, tableEnv.getConfig.setNullCheck(false) seems to break group by-s

Hello to all, on flink-1.11.2 the program written at https://gist.github.com/yetanotherion/d007fa113d97411226eaea4f20cd4c2d creates unexpected stack traces when the line “// triggerBug…” Is uncommented (some lines of the stack trace are written in the gist). (It correctly outputs “ +

Re: TaskManager takes abnormally long time to register with JobManager on Kubernetes for Flink 1.11.0

Hi all, Thanks for all the replies, and I agree with Yang, as we have found that for a pod without a service (like TaskManager pod), the reverse DNS lookup would always fail, so this lookup is not necessary for the Kubernetes environment. I am glad to help fix this issue to make Flink better : )

Re: Effect of renaming a primary key

Hi, Rex > , won't that therefore change the key used to reference the state in RockDB > and in a checkpoint for the associated table? > How might this effect state storage and checkpointing? > Will the pk that was renamed remain a key for state or is some other > mechanism used to form a key?