Re: Parallelism of state processor jobs

lockingQueue whose size is hard > coded as 16 in the Flink sources. Am I missing something crucial for tuning > such jobs? > > Regards, > Alexis. > > On Sat, 6 Jul 2024, 03:29 Junrui Lee, wrote: > >> Hi Alexis, >> >> For the SavepointWriter, I've brief

Re: Parallelism of state processor jobs

Hi Alexis, For the SavepointWriter, I've briefly looked over the code and the write operation is enforced as non-parallel. Best, Junrui Alexis Sarda-Espinosa 于2024年7月6日周六 01:27写道： > Hi Gabor, > > Thanks for the quick response. What about SavepointWriter? In my case I'm > actually writing a job

Re: Exception: Coordinator of operator xxxx does not exist or the job vertex this operator belongs to is not initialized.

o Geng > > 发送自 Outlook for iOS <https://aka.ms/o0ukef> > -- > *发件人:* Junrui Lee > *发送时间:* Sunday, June 16, 2024 12:49:10 PM > *收件人:* Corin > *抄送:* user@flink.apache.org > *主题:* Re: Exception: Coordinator of operator does not exist or t

Re: Exception: Coordinator of operator xxxx does not exist or the job vertex this operator belongs to is not initialized.

Hi, This exception is common in batch jobs and is caused by the collect sink attempting to fetch data from the corresponding operator coordinator on the JM based on the operator ID. However, due to the sequential scheduling of batch jobs, if a job vertex has not been initialized yet, the correspon

RE: Does Application mode support multiple submissions in HA mode?

Currently, Application mode does not support multiple job submissions when HA is enabled. You can check the official documentation for this statement:https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode

Re: Slack Community Invite?

Hi Alexandre, You can try this link: https://join.slack.com/t/apache-flink/shared_invite/zt-2jn2dlgoi-P4oQBWRJT4I_3HY8ZbLxdg Best, Junrui Alexandre Lemaire 于2024年5月28日周二 01:18写道： > Hello! > > Does the Slack community still exist? The link on the site is expired. > > Thank you! > Alex > > >

Re: How to start a flink job on a long running yarn cluster from a checkpoint (with arguments)

Hi Sachin, Yes, that's correct. To resume from a savepoint, use the command bin/flink run -s . You can find more details in the Flink documentation on [1]. Additionally, information on how to trigger a savepoint can be found in the section for triggering savepoints [2]. [1] https://nightlies.ap

Re: flink version stable

Hi, The latest stable version of FLINK is 1.19.0 > > Fokou Toukam, Thierry > 于2024年3月29日周五 16:25写道： > >> Hi, just want to know which version of flink is stable? >> >> *Thierry FOKOU *| * IT M.A.Sc Student* >> >> Département de génie logiciel et TI >> >> École de technologie sup

Re: Batch Job with Adaptive Batch Scheduler failing with JobInitializationException: Could not start the JobMaster

Hi Dipak, Regarding question 1, I noticed from the logs that the method createBatchExecutionEnvironment from Beam is being used in your job. IIUC, this method utilizes Flink's DataSet API. If indeed the DataSet API is being used, the configuration option execution.batch-shuffle-mode will not take

Re: Question around manually setting Flink jobId

Hi Allison, The PIPELINE_FIXED_JOB_ID configuration option is not intended for public use. IIUC, the only way to manually specify the jobId is submitting a job through the JAR RUN REST API, where you can provide the jobId in the request body ( https://nightlies.apache.org/flink/flink-docs-master/d

Re: Flink SQL query using a UDTAGG

Hi Pouria, Table aggregate functions are not currently supported in SQL, they have been introduced in the Table API as per FLIP-29: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739. Best, Junrui Pouria Pirzadeh 于2024年3月13日周三 02:06写道： > Hi, > I am using the SQL api on F

Re: Using User-defined Table Aggregates Functions in SQL

Hi Jad, You can refer to the CREATE FUNCTION section ( https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-function) and the Table Aggregate Functions section ( https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/udfs/#table-aggregate-fu

Re: I used the yarn per job mode to submit tasks, which will end in 4 minutes

Hello, The issue you're encountering is related to a new heartbeat mechanism between the client and job in Flink-1.17. If the job does not receive any heartbeats from the client within a specific timeout, it will cancel itself to avoid hanging indefinitely. To address this, you have two options:

Re: Batch mode execution

Hello Irakli, The error is due to the fact that the Adaptive Scheduler doesn’t support batch jobs, as detailed in the Flink documentation[1]. When operating in reactive mode, Flink automatically decides the type of scheduler to use. For batch execution, the default scheduler is AdaptiveBatchSchedu

Re: Redis as a State Backend

Hi Chirag, Indeed, the possibility of using Redis as a state backend for Flink has been considered in the past. You can find a detailed discussion about this topic in the JIRA issue FLINK-3035[1] as well as in the comments section of this PR[2]. The outcome of these discussions was that Redis is

Re: 退订

Please send email to user-unsubscr...@flink.apache.org and user-zh-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org and user...@flink.apache.org, you can refer [1][2] for more details. Best, Junrui [1] https://flink.apache.org/zh/community/#%e9%82%ae%e4%

Re: Flink caching mechanism

Hi Вова In Flink, there is no built-in mechanism for caching SQL query results; every query execution is independent, and results are not stored for future queries. The StateBackend's role is to maintain operational states within jobs, such as aggregations or windowing, which is critical for ensur

Re: Configuration not propagating unless explicitly initializing FileSystem for Azure File System

Hi, Yuval The issue you're encountering arises because Flink's FileSystem is designed to operate with cluster-level configuration. And this configuration is sourced from the flink-conf.yaml file. Consequently, when the FileSystem is initialized, it isn't able to access the configuration objects th

Re: Java 8 support in Flink 1.18

Chandna 于2023年12月18日周一 18:14写道： > Thanks a lot for your reply. > > > > Is there any expected timelines when the Java 8 support will be removed ? > > > > Thanks !! > > > > *From:* Junrui Lee > *Sent:* 18 December 2023 14:37 > *To:* Praveen Chandna >

Re: Java 8 support in Flink 1.18

Hello, Praveen Java 8 is still supported in the latest Flink 1.18 release despite its deprecation. You can continue using Java 8 for now. However, deprecation serves as a warning that in future releases, Java 8 may no longer be supported. It is recommended that you begin planning your migration to

Re: Cast Exception

Hello Tauseef, The issue you're encountering is due to the fact that the Properties class in Java stores both keys and values as Strings. When you are trying to cast the value directly to Double, it throws a ClassCastException because values from the properties file are loaded as String and cannot

Re: 退订

Hi Jintao, Please send an email to user-unsubscr...@flink.apache.org to unsubscribe the user mailing list. Jintao Ma 于2023年11月27日周一 09:24写道： > 退订 >

Re: [DISCUSS] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

more than 30 methods > offered flat by the RuntimeContext. I am not sure if this could help users > find the right method in the context of execution config better than > before. > > I might miss something and look forward to your thoughts. Thanks! > > Best regards, >

Re: [DISCUSS] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

clean up the codebase. > > > +1 (non-binding). > > > Best, > Wencong > > > > > > > > > > > > > > > > > > > > At 2023-11-15 16:51:15, "Junrui Lee" wrote: > >Hi all, > > > >I'd like to start a disc

Re: [DISCUSS] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

u Zhu wrote: > >> Thanks Junrui for creating the FLIP and kicking off this discussion. >> >> Exposing a mutable ExecutionConfig which is even shared by multiple >> operators is truly a defect which can result in weird results. >> >> +1 >> >> Thanks, >&

[DISCUSS] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

Hi all, I'd like to start a discussion of FLIP-391: Deprecate RuntimeContext#getExecutionConfig[1]. Currently, the FLINK RuntimeContext is important for connecting user functions to the underlying runtime details. It provides users with necessary runtime information during job execution. However,

Re: Elasticsearch source

Hi Tauseef, AFAIK, Flink does not support ElasticSearch as a source connector. It only supports ElasticSearch as a sink connector. Best regards, Junrui Tauseef Janvekar 于2023年11月15日周三 14:36写道： > Dear Team, > > We were looking for some elasticsearch source connector for flink and I > could not

Re: unsubscribe

Hi, Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org. Ralph Matthias Debusmann 于2023年11月15日周三 07:29写道： > >

Re: FLINK CONNECTOR 1.18 and Kafka 2.7

Hi, The externalized Flink Kafka connector v3.0.1 uses Kafka 3.2.3, see https://github.com/apache/flink-connector-kafka/blob/ea4fac3966c84f4cae8b80d70873254f03b1c333/pom.xml#L53 And you can download it from here: https://flink.apache.org/downloads/#apache-flink-kafka-connector-301 Best, Junrui

Re: Disable flink old checkpoint clean

Hi Yang, You can try configuring "execution.checkpointing.externalized-checkpoint-retention: RETAIN_ON_CANCELLATION"[1] and increasing the value of "state.checkpoints.num-retained"[2] to retain more checkpoints. Here are the official documentation links for more details: [1] https://nightlies.

Re: Error in /jars/upload curl request

Hi, Tauseef Based on the screenshot you provided, it appears that you have not included the '@' prefix before the file path in your curl command. This prefix is necessary to indicate to curl that the specified argument should be treated as a file to be uploaded. Please add the '@' prefix before t

Re: Queryable state feature in latest version!!

Hi, Puneet Thank you for reaching out. In the latest release of Flink (version 1.18), we have marked Queryable State as @Deprecated and removed the related content from the stable documentation. This means that Queryable State is no longer actively supported or recommended for use. More details c

Re: Flink Job Failed With Kafka Exception

Hi Madan, Do you mean you want to restart only the failed tasks, rather than restarting the entire pipeline region? As far as I know, currently Flink does not support task-level restart, but requires restarting the pipeline region. Best, Junrui Madan D via user 于2023年10月11日周三 12:37写道： > Hello

Re: Instructions / steps regarding contributing : flink-training

Hello Milind, Thank you for your interest in contributing to flink-training! To add or modify unit tests, please start by creating a JIRA ticket at the FLINK JIRA page: https://issues.apache.org/jira/browse/FLINK, and set the "Component/s" to "Documentation / Training / Exercises" for your ticket.

Re: How to tell if job is being restarted in log?

Hi John, If you want to know more details about why your job is restarting, you can search for the keyword "to FAILED" in the JobManager logs. These log entries will show you the timing of each restart and the associated exception information. Additionally, you can check the exception page to find

Re: Inquiry about ActiveResourceManager and StandaloneResourceManager in Flink

Hi, Steven Regarding the term "active" in ActiveResourceManager, it signifies that it is responsible for actively creating and managing TaskManager instances based on requirements. It dynamically adjusts resources in the cluster, starting new TaskManager instances when needed. On the other hand,