AW: How to add Flink a Flink connector to stateful functions

Hello Igal, Thanks for replying in detail and also so quickly. It’ll take me some time to try it out, thank you! Best, Christian -- Dr. Christian Krudewig Corporate Development – Data Analytics Deutsche Post DHL Headquarters Charles-de-Gaulle-Str. 20 53113 Bonn Germany Ph

Re: rpc invocation exceeds the maximum akka framesize

Is the akka framesize a function of broadcast size? From: Deshpande, Omkar Sent: Monday, September 27, 2021 6:31 PM To: user@flink.apache.org Cc: Benenson, Mikhail ; Hwang, Nick ; Canchi, Srivathsan Subject: rpc invocation exceeds the maximum akka framesize Hell

[no subject]

unsubscribe

Re:

Hi Violeta, in order to unsubscribe emails from Flink user mail list, send an email to user-unsubscr...@flink.apache.org. For more information, please have a look at https://flink.apache.org/community.html#mailing-lists. Best Ingo On Tue, Sep 28, 2021 at 10:50 AM Violeta Milanović < violeta.mi

Re: rpc invocation exceeds the maximum akka framesize

Hi, There could be many reasons for exceeding akka framesize, for example 1. "inlined" state that is stored in checkpoint .metadata file (rather than "data" files - see [1]) 2. broadcast state as you mentioned (though only the metadata is sent unless the data fits the above limits) 3. too many sta

Re: Unable to connect to Mesos on mesos-appmaster.sh start

Hi, No additional ports need to be open as far as I know. Probably, $HOSTNAME is substituted for something not resolvable on TMs? Please also make sure that the following gets executed before mesos-appmaster.sh: export HADOOP_CLASSPATH=$(hadoop classpath) export MESOS_NATIVE_JAVA_LIBRARY=/path/t

Event is taking a lot of time between the operators

Hi All, I am new to Flink. While developing a Flink application We observed that our message is taking around 10 seconds between the two Async operators. Below are the details. * Flink Flow: Kinesis Source -> Process -> Async1 -> Async2 -> Process -> Kinesis Sink * Environment: Amazon

Re: How to enable customize logging library based on SLF4J for Flink deployment in Kubernetes

Could you clarify whether this internal framework uses a custom slfj4/log4j2 version, or is it just using what Flink comes with? Did you only add the configmap and put a single jar into lib, or did you make other changes in Flink? Can you remove just the configmap, start the cluster, and prov

Re: How to add Flink a Flink connector to stateful functions

I just want to add that the StateFun documentation does cover using custom Flink connectors[1]. [1] https://nightlies.apache.org/flink/flink-statefun-docs-release-3.1/docs/modules/io/flink-connectors/#flink-connectors On Tue, Sep 28, 2021 at 2:52 AM Christian Krudewig (Corporate Development) < c

Re: Event is taking a lot of time between the operators

Hi, With Flink 1.8.0 I'm not sure how reliable the backpressure status is in the WebUI when it comes to the Async operators. If I remember correctly until around Flink 1.10 (+/- 2 version) backpressure monitoring was checking for thread dumps stuck in requesting Flink's network memory buffers. If

Re: Unable to connect to Mesos on mesos-appmaster.sh start

Hi Javier, I don't see anything that's configured in the wrong way based on the jobmanager logs you've provided. Have you been able to deploy other applications to this Mesos cluster? Do the Mesos master logs reveal anything? The variable resolution on the TaskManager side is a valid concern shared

Re: Questions about keyed streams

Hi! I'm just getting back to this. Questions: 1. Across operators, does the same key group ids get mapped to the same task managers? E.g. if an item is in key group 1 of operator A and that runs on taskmanager-0, will key group 1 of operator B also run on taskmanager-0? 2. Are there any internal

Re: Unable to connect to Mesos on mesos-appmaster.sh start

Thanks, Roman! Looking at the log, seems that the TaskManager can resolve $HOSTNAME to its own hostname (07a6b681ee0f), as seen in these lines: 2021-09-27 22:02:41.067 [main] INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Djobmanager.rpc.address=*07a6b681ee0f* 2021-09-27 22:02:43

Re: Unable to connect to Mesos on mesos-appmaster.sh start

Thanks, Matthias! There are lots of apps deployed to the Mesos cluster, the task manager itself is deployed to Mesos via Marathon. In the Mesos log I can see the Job manager agent starting, but no error messages related to it. As you say, TaskManagers don't even have the chance to get confused ab

Flink run different jars

Hi, Is that possible to run a flink app without a job? What I am trying to do is I build multiple jars. And switch jar to run different jobs. I am not sure if flink supports this mode. I saw rest API can upload jar, cancel job and run a jar. Right now I can upload a jar to flink. But when I cancel

Flink Demo with Redpanda

Hey all, I thought this might be of interest to the community. Basically, we took Jark Wu's demo ( https://flink.apache.org/2020/07/28/flink-sql-demo-building-e2e-streaming-application.html) and replaced Kafka with Redpanda: https://vectorized.io/blog/redpanda-flink-docker/ Regards, P

Re: Unable to connect to Mesos on mesos-appmaster.sh start

Another update: Looking more carefully in my appmaster log, I see the following 2021-09-29 01:15:39.680 [flink-akka.actor.default-dispatcher-3] INFO o.a.f.m.runtime.clusterframework.MesosResourceManagerDriver - Registering as new framework. 2021-09-29 01:15:39.680 [flink-akka.actor.default-disp

Re: Flink run different jars

Hi, Qihua IIUC, what you want might be a standalone cluster[1] or session cluster[2][3]. [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/ [2] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-provide

HA setup Flink version flink:1.13.2-scala_2.12-java11

I am following this link for setting up  HA configuration ZooKeeper HA Services | | | | ZooKeeper HA Services ZooKeeper HA Services # Flink’s ZooKeeper HA services use ZooKeeper for high availability services. Flink lever... | | | zookeeper version which I am using is 3.4.10     high

Re: Flink run different jars

Hi Yangze, Thanks a lot for your reply. References are very helpful! Another quick question. Reference 1 can start a standalone cluster (session Mode). That cluster has a jobManager. I can submit job to run. How about taskManger? Do I need to manually start multiple taskManagers? Is there a comple

RE: Event is taking a lot of time between the operators

Hi @Piotr Nowojski, Thank you for replying back. Yes, first async is taking between 1300-1500 milliseconds but that is called on a CompletableFuture.supplyAsync and the Async Capacity is set to 1000. Async Code Structure: Inside asyncInvoke we are calling Completab

Re: Event is taking a lot of time between the operators

Hi Sanket, I have a similar use case. how are you measuring the time for Async1` function to return the result and external api call On Wed, Sep 29, 2021 at 10:47 AM Sanket Agrawal wrote: > Hi @Piotr Nowojski , > > > > Thank you for replying back. Yes, first async is taking between 1300-1500 >

Re: Flink run different jars

You need to edit the conf/workers. Example of the config[1] and the process[2]. [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/#starting-and-stopping-a-cluster [2] https://ci.apache.org/projects/flink/flink-docs-master/docs/deplo

RE: Event is taking a lot of time between the operators

Hi Ragini, For measuring time in an async we have put a logger as the first and the last statement in asyncInvoke and for measuring time between the asyncs we are simply subtracting the message2's start time and message1's end time. Also, we are using 1 as the parallelism. Please let me know i

RE: Upgrading from 1.11.3 -> 1.13.1 - random jobs stays in "CREATED"state, then fails with Slot request bulk is not fulfillable!

Hi Tobias, If your number of pipelines equals number of Flink job then this is exactly what you should observe: It takes one slot per Flink job and parallelism, hence for parallelism 1 you would have to provide at least 40 slots. … independent of Flink version … for Beam on Flink I’m not sure,

Re: HA setup Flink version flink:1.13.2-scala_2.12-java11

Thanks    I got this working On Wednesday, September 29, 2021, 12:12:17 AM EDT, Dhiru wrote: I am following this link for setting up  HA configuration ZooKeeper HA Services | | | | ZooKeeper HA Services ZooKeeper HA Services # Flink’s ZooKeeper HA services use ZooKeeper for high