Re: Flink sql task failure recovery does not work.

Hi I think you could configure some restart strategy[1] likes restart-strategy: fixed-delay [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/execution/task_failure_recovery/#fixed-delay-restart-strategy Best, Guowei On Thu, May 13, 2021 at 12:02 PM 1095193...@qq.com <109

Re: Need Clarity about Checkpoint for Flink-1.12.2

Hi Sudhansu, I think you do not need to set the config in flink-conf. Best, Guowei On Thu, May 13, 2021 at 1:06 PM sudhansu jena wrote: > Hi Team, > > We have recently enabled Check Pointing in our flink job using > FSStateBackend pointing to S3 bucket. > > Below is the sample code for enabling

Re: docker based taskmanager can't connect to job/resource manager

Hi, Here is the link[1]. [image: image.png] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/#starting-a-session-cluster-on-docker Best, Guowei On Thu, May 13, 2021 at 1:53 PM guenterh.lists wrote: > Hi Guowei, > > thanks for your re

Re: docker based taskmanager can't connect to job/resource manager

Hi Guowei, thanks for your reply! This information was still missing. The presenter mentioned the documentation but I hadn't found it. So your link to the specific place is valuable too. Günter On 13.05.21 06:09, Guowei Ma wrote: Hi, I do not try it. But from the documentation[1] it seems t

Need Clarity about Checkpoint for Flink-1.12.2

Hi Team, We have recently enabled Check Pointing in our flink job using FSStateBackend pointing to S3 bucket. Below is the sample code for enabling check pointing though app code and we are using flink version - 1.12.2 . env.setStateBackend(new FsStateBackend("s3://flinkcheckpointing/job-name/",

Re: Setup of Scala/Flink project using Bazel

That would be awesome Austin, thanks again for your help on that. In the meantime, I also filled an issue in the `rules_scala` repo: https://github.com/bazelbuild/rules_scala/issues/1268. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: docker based taskmanager can't connect to job/resource manager

Hi, I do not try it. But from the documentation[1] it seems that you might need add the "jobmanager.rpc.address: jobmanager" to the FLINK_PROPERTIES before creating a network. [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/ Best, Gu

Flink sql task failure recovery does not work.

Hi team, Following Task Failure Recovery document https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/execution/task_failure_recovery/ , I have enabled state.checkpoints.dir parameter in flink-conf.yaml. state.checkpoints.dir: hdfs://172.16.1.192:9000/flink-checkpoin

How does JobManager terminate dangling task manager

Hi, Trying to understand how JobManager. kills TaskManager that didn't respond for heartbeat after a certain time. For example: If a network connection b/w JobManager and TaskManager is lost for some reasons, the JobManager will bring up another Taskmanager post hearbeat timeout. In such a case,

Re: Root Exception can not be shown on Web UI in Flink 1.13.0

Hi, it seems to be related to FLINK-22276. Thus, I'd involve Matthias to take a look. @Matthias My gut feeling is that not all execution who has failureInfo has been deployed? Best, Yangze Guo On Wed, May 12, 2021 at 10:12 PM Gary Wu wrote: > > Hi, > > We have upgraded our Flink applications to

Re: Regarding Stateful Functions

Hey Jessy, I'm not a Statefun expert but, hopefully, I can point you in the right direction for some of your questions. I'll also cc Gordan, who helps to maintain Statefun. *1. Is the stateful function a good candidate for a system(as above) that > should process incoming requests at the rate of

Re: Apache Flink - A question about Tables API and SQL interfaces

Thanks Austin for your helpful references. I did take a look at [2]/[3] - but did not find anything relevant on searching for string 'late' (for allowed lateness etc) or side output.  So from my understanding the late events will be dropped if I am using Table API or SQL and the only option is

Re: Apache Flink - A question about Tables API and SQL interfaces

Hi Mans, I don't believe there are explicit triggers/evictors/timers in the Table API/ SQL, as that is abstracted away from the lower-level DataStream API. If you need to get into the fine-grained details, Flink 1.13 has made some good improvements in going from the Table API to the DataStream API

Re: Setup of Scala/Flink project using Bazel

I know @Aaron Levin is using `rules_scala` for building Flink apps, perhaps he can help us out here (and hope he doesn't mind the ping). On Wed, May 12, 2021 at 4:13 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Yikes, I see what you mean. I also can not get `neverlink` or addin

Re: Setup of Scala/Flink project using Bazel

Yikes, I see what you mean. I also can not get `neverlink` or adding the org.scala.lang artifacts to the deploy_env to remove them from the uber jar. I'm not super familiar with sbt/ scala, but do you know how exactly the assembly `includeScala` works? Is it just a flag that is passed to scalac?

docker based taskmanager can't connect to job/resource manager

Hi, I'm trying to start a mini cluster following the explanations given in a flink forward presentation [1] Starting a jobmanager task is possible FLINK_PROPERTIES="jobmanager.memory.process.size: 2048m parallelism.default: 4 " docker network create flink-network docker run  \ --rm   \ --nam

Re: Job initialization and scheduling takes a very long time (15+ minutes)

Solved this. After trying multiple things i just decided to delete the massive number of old leader configMaps from my k8s cluster (200+) After that the time to initialize + schedule is down to a couple of seconds -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com

Re: Setup of Scala/Flink project using Bazel

Hi Austin, Yep, removing Flink dependencies is working well as you pointed out. The problem now is that I would also need to remove the scala library...by inspecting the jar you will see a lot of scala-related classes. If you take a look at the end of the build.sbt file, I have ``` // exclude Sc

Re: Setup of Scala/Flink project using Bazel

Hi Salva, I think you're almost there. Confusion is definitely not helped by the ADDONS/ PROVIDED_ADDONS thingy – I think I tried to get too fancy with that in the linked thread. I think the only thing you have to do differently is to adjust the target you are building/ deploying – instead of `//

Apache Flink - A question about Tables API and SQL interfaces

Hey Folks: I have the following questions regarding Table API/SQL in streaming mode: 1. Is there is a notion triggers/evictors/timers when using Table API or SQL interfaces ?2. Is there anything like side outputs and ability to define allowed lateness when dealing with the Table API or SQL interf

Re: TimeoutException in Flink 1.11 stop command

Thanks. I tried this command and it worked. > flink stop -p s3a://path_to_savepoint/savepoints > 5f9241d336ea2c652a84f79ac3158597 -yid application_1620673166934_0001 I will look at the "client.timeout" also to figure out what actually happened. Thanks. On Tue, May 11, 2021 at 3:04 AM Chesnay

Regarding Stateful Functions

Hi all, I have gone through the stateful function's documentation and required some expert advice or clarification regarding the following points. *Note: My data processing flow is as follows,* *ingress(10k/s)--> First transformation based on certain static rules --> second transformation bas

[no subject]

Re: failure notice

‫בתאריך יום ד׳, 12 במאי 2021 ב-18:08 מאת <‪mailer-dae...@apache.org‬‏>:‬ > Hi. This is the qmail-send program at apache.org. > I'm afraid I wasn't able to deliver your message to the following > addresses. > This is a permanent error; I've given up. Sorry it didn't work out. > > : > Must be sent f

Re: Setup of Scala/Flink project using Bazel

Hi Austin, I followed your instructions and gave `rules_jvm_external` a try. Overall, I think I advanced a bit, but I'm not quite there yet. I have followed the link [1] given by Matthias, making the necessary changes to my repo: https://github.com/salvalcantara/bazel-flink-scala In particular,

Root Exception can not be shown on Web UI in Flink 1.13.0

Hi, We have upgraded our Flink applications to 1.13.0 but we found that Root Exception can not be shown on Web UI with an internal server error message. After opening browser development console and trace the message, we found that there is a exception in jobmanager: *2021-05-12 13:30:45,589 ERRO

Re: After upgrade from 1.11.2 to 1.13.0 parameter taskmanager.numberOfTaskSlots set to 1.

The contents of FLINK_PROPERTIES are piped as-is into the flink configuration, and thus require the same format as the configuration. On 5/12/2021 2:36 PM, Alex Drobinsky wrote: Thanks a lot ! I used TASK_MANAGER_NUMBER_OF_TASK_SLOTS in my docker-compose.yml, it works perfectly :) In which fo

Re: After upgrade from 1.11.2 to 1.13.0 parameter taskmanager.numberOfTaskSlots set to 1.

Thanks a lot ! I used TASK_MANAGER_NUMBER_OF_TASK_SLOTS in my docker-compose.yml, it works perfectly :) In which format I could provide parameters via FLINK_PROPERTIES ? I'm thinking of abandoning the idea to copy flink-conf in Dockerfile. Is it limited to a specific set of parameters or generic ?

Re: After upgrade from 1.11.2 to 1.13.0 parameter taskmanager.numberOfTaskSlots set to 1.

You could also configure the number of slots via the TASK_MANAGER_NUMBER_OF_TASK_SLOTS environment variable. On 5/12/2021 2:19 PM, Chesnay Schepler wrote: I believe this is due to FLINK-21037; we did not consider the possibility of users mounting the configuration directly, and instead assumed

Re: After upgrade from 1.11.2 to 1.13.0 parameter taskmanager.numberOfTaskSlots set to 1.

I believe this is due to FLINK-21037; we did not consider the possibility of users mounting the configuration directly, and instead assumed that modifications to the config always go through the FLINK_PROPERTIES environment variable. That would also be the workaround for your issue. On 5/12/2

After upgrade from 1.11.2 to 1.13.0 parameter taskmanager.numberOfTaskSlots set to 1.

Dear flink community, First I need provide some minimum information about my deployment scenario: I'm running application inside of Flink docker, below original Dockerfile: --- FROM flink:1.13.

Re: Possible way to avoid unnecessary serialization calls.

Hi Alex, I cannot reproduce the issue. Do you mind checking if it is not an issue on your side? P.S. It would be nice if you could reply to the ML as well. That way other people can benefit from the answers. Moreover there will be more people who could help answering your question. Best, Dawid

Re: Flink 1.13.0: check flamegraph on WebUI shows 404 not found

Add rest.flamegraph.enabled: true to conf/flink-conf.yaml solves my problem. Thanks! Best, Yik San On Wed, May 12, 2021 at 5:00 PM Chesnay Schepler wrote: > Did you enable the flamegraph fea

Re: [EXTERNAL] Re: sideOutputLateData not propagating late reports once window expires

Ah, sorry for the compile issue. I wasn't able to reproduce the issue; conceptually your code looks fine. Can you provide us with a self-contained reproducer for the issue? For reference, here's the test I used, that you can maybe adjust as necessary to replicate your use-case: @Test public

Re: Could not resolve ResourceManager address in native kubernetes

It seems that the TaskManager pod could not resolve the JobManager address "franz-01.default", which is constructed in "k8s-service-name.namespace". I think you need to check whether the coreDNS is running normally in your K8s cluster. You could start a busybox pod on the same node with TaskManager

Re: Flink 1.13.0: check flamegraph on WebUI shows 404 not found

Did you enable the flamegraph feature in the configuration? On 5/12/2021 10:51 AM, Yik San Chan wrote: Hi community, Flink 1.13.0 releases flamegraph. However, when I run Flink 1.13.0 locally, and try to check the flamegraph of an operator that is running, I got this "404 not found" error. Th

Flink 1.13.0: check flamegraph on WebUI shows 404 not found

Hi community, Flink 1.13.0 releases flamegraph. However, when I run Flink 1.13.0 locally, and try to check the flamegraph of an operator that is running, I got this "404 not found" error. This is the request and response. Request: curl ' http://localhost:8081/jobs/9f4ba3d39de6b2d9de0ea77ba8f8431a

Re: Unused Checkpointed folder in S3

Hi Sudhansu, the usual flow is that if you cancel a job, you have 2 choices: - Either you want to later resume the job, then you use stop-with-savepoint and resume from savepoint. That would allow you to upgrade the job or the Flink version. However, taking a final savepoint takes some time. - Or