Re: Help installing apache-flink and numpy to run flink python examples

Hi Kenneth, In flink 1.15, pyflink only guarantees support for python 3.6,3.7 and 3.8[1]. In release-1.16, pyflink will provide support for python 3.9[2]. Go back to your installation error. In flink 1.15, the version range of numpy that pyflink depends on is numpy>=1.14.3,<1.20. So when you exec

Re: How to parse protobuf data from kafka by pyflink

It still doesn't support custom Deserializer developed with Python language. However, you could create a Java Deserializer and use it in PyFlink. Regards, Dian On Thu, Jun 2, 2022 at 10:25 AM laizhic...@hongkingsystem.cn < laizhic...@hongkingsystem.cn> wrote: > > Hi Everyone: >I used protobu

Re: Cannot cast GoogleHadoopFileSystem to hadoop.fs.FileSystem to list file in Flink 1.15

On Thu, Jun 02, 2022 at 11:17:19AM +0800, Qingsheng Ren wrote: > Hi ChangZhuo, > > I assume it’s a classloading issue but I can’t track down to the root cause > in code. Would you mind sharing the entire exception stack and some JM/TM > logs related to file system? The following is exception lo

Re: Cannot cast GoogleHadoopFileSystem to hadoop.fs.FileSystem to list file in Flink 1.15

Hi ChangZhuo, I assume it’s a classloading issue but I can’t track down to the root cause in code. Would you mind sharing the entire exception stack and some JM/TM logs related to file system? Best regards, Qingsheng > On Jun 2, 2022, at 09:08, ChangZhuo Chen (陳昌倬) wrote: > > Hi, > > We u

Re: Flink Kubernetes Operator v1.0 ETA

If everything goes well, I will close the VOTE for RC4 on Friday night, which should run for more than 48 hours. And then finalize the release. Best, Yang Gyula Fóra 于2022年6月1日周三 23:30写道： > Hi Jeesmon! > > We are currently working through the release process. We are now in the > middle of votin

Re:Help installing apache-flink and numpy to run flink python examples

Hi, Kenneth, have you tried the setup file[1] in the Flink-Python? It maybe can work. [1] https://github.com/apache/flink/blob/master/flink-python/setup.py At 2022-06-02 03:53:36, "Kenneth Shine" wrote: I am using MacOS Monterey 12.3.1 I have download flink source and examples from GitHub

How to parse protobuf data from kafka by pyflink

Hi Everyone:    I used protobuf in kafka,in java code ,I could define custom  Deserializer,but how colud I do this on pyflink?

Cannot cast GoogleHadoopFileSystem to hadoop.fs.FileSystem to list file in Flink 1.15

Hi, We use GCS as storage, and have the following functions to list files in GCS path for Flink batch mode to buidl states: def listPath(p: String): Seq[String] = { val path = new Path(p) val fs = path.getFileSystem(new Configuration()) fs.listStatus(path) match { case null =

Re: [External] Re: Exception when running Java UDF with Blink table planner

Hi Shengkai, In order to reproduce the issue, the input argument type must be `Object[]`. Also DPScalarFunction is a typo and should be ScalarFunction. Are you able to observe the error if you try with the changed input type: public static class ListToString extends ScalarFunction { public St

Re: Can we resume a job from a savepoint from Java api?

Hi Qing, There are also some useful utilities in the Sink/Source testing suites that you could either extract or adjust for your purposes [1] [1] https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-connector-test-utils/src/main/java/org/apache/flink/connector/testframe/tests

Re: Is there an HA solution to run flink job with multiple source

Hi Bariša, The way I see it is you either - need data from all sources because you are doing some conjoint processing. In that case stopping the pipeline is usually the right thing to do. - the streams consumed from multiple servers are not combined and hence could be processed in independent Flin

Re: Is there an HA solution to run flink job with multiple source

Hi Bariša, Could you share the reason why your data processing pipeline should keep running when one kafka source is down? It seems like any one among the multiple kafka sources is optional for the data processing logic, because any kafka source could be the one that is down. Best regards, Jing

Help installing apache-flink and numpy to run flink python examples

I am using MacOS Monterey 12.3.1 I have download flink source and examples from GitHub I have installed python 3.10.4 from https://www.python.org/downloads/macos/ In flink/libexec/examples/python/datastream, I run /usr/local/bin/python3 word_count.py But get error Traceback (most recent c

Re:Is there an HA solution to run flink job with multiple source

I think you can try to use a custom source to do that although the one of the kafka sources is down the operator is also running(just do nothing). The only trouble is that you need to manage the checkpoint and something else yourself. But the good news is that you can copy the implementation of

Re: Flink Kubernetes Operator v1.0 ETA

Hi Jeesmon! We are currently working through the release process. We are now in the middle of voting for RC4 (we have identified and fixed a number of blocker issues in earlier RCs). We are hopeful that this RC will be successful in which case you will have a release by the end of the week. If we

Flink Kubernetes Operator v1.0 ETA

Hi there, Is there an ETA on v1.0 release of operator? We are prototyping with a CI build from release-1.0 branch but would like to know the approximate ETA of official 1.0 release so that we can plan accordingly. Thanks, Jeesmon

Is there an HA solution to run flink job with multiple source

Hi, we are running a flink job with multiple kafka sources connected to different kafka servers. The problem we are facing is when one of the kafka's is down, the flink job starts restarting. Is there anyway for flink to pause processing of the kafka which is down, and yet continue processing from

Re: Flink task stuck - MapPartition WAITING on java.util.concurrent.CompletableFuture$Signaller

Hi Jan, I have not checked the harness log. I have now checked it *Apache Beam worker log) and found this, but currently not sure what it means: 2022/06/01 13:34:40 Python exited: 2022/06/01 13:34:41 Python exited: Exception in thread read_grpc_client_inputs: Traceback (most recent call last):

RE: Can we resume a job from a savepoint from Java api?

Thanks both, that’s perfect! From: Jing Ge Sent: 01 June 2022 14:29 To: yuxia Cc: Qing Lim ; User Subject: Re: Can we resume a job from a savepoint from Java api? Hi, yuxia has already pointed out the correct direction. The exact line for using the savepoint path to resume the job from a sav

Re: Can we resume a job from a savepoint from Java api?

Hi, yuxia has already pointed out the correct direction. The exact line for using the savepoint path to resume the job from a savepoint is at line 1326 [1] [1] https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-tests/src/test/java/org/apache/flink/test/checkpointi

Re: Can we resume a job from a savepoint from Java api?

Hope the unit test SavepointITCase#testCanRestoreWithModifiedStatelessOperators[1] in Flink repo can help you. [1] [ https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/checkpointing/SavepointITCase.java#L1228 | https://github.com/apache/flink/blob/ma

Can we resume a job from a savepoint from Java api?

Hi, is it possible to resume a job from a savepoint in Java code? I wish to test failure recovery in my test code, I am thinking to simulate failure recovery by saving state to a save point and the recover from it, is this possible with local MiniCluster setup? Kind regards This e-mail and any

Re: Flink task stuck - MapPartition WAITING on java.util.concurrent.CompletableFuture$Signaller

Hi Gorjan, +user@beam The trace you posted is just waiting for a bundle to finish in the SDK harness. I would suspect there is a problem in the logs of the harness. Did you look for possible errors there?  Jan On 5/31/22 13:54, Gorjan Todorovski wrote: Hi, I

Re: Flink DataStream and remote Stateful Functions interoperability

Stateful Functions are in another repository. Take a look at device and phone packages. Deployment breaks into 2 parts: 1. Deploying your dataflow job (just a Flink cluster with your job running) 2. Deploying a service that implements remote functions as an ordinary service Best, Tymur Yarosh On

Re: multiple pipeline deployment using flink k8s operator

The current application mode has the limitation that only one job could be submitted when HA enabled[1]. So a feasible solution is to use the session mode[2], it will be supported in the coming release-1.0.0. However, I am afraid it still could not satisfy your requirement "2 task managers (one pe