[jira] [Created] (FLINK-36622) Remove the dependency of StateBenchmark on RocksDBKeyedStateBackend APIs.

2024-10-29 Thread Han Yin (Jira)
Han Yin created FLINK-36622: --- Summary: Remove the dependency of StateBenchmark on RocksDBKeyedStateBackend APIs. Key: FLINK-36622 URL: https://issues.apache.org/jira/browse/FLINK-36622 Project: Flink

[jira] [Created] (FLINK-36621) Build failure: StatefulSink not found

2024-10-29 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-36621: -- Summary: Build failure: StatefulSink not found Key: FLINK-36621 URL: https://issues.apache.org/jira/browse/FLINK-36621 Project: Flink Issue Type: Improve

[jira] [Created] (FLINK-36625) Add helper classes for Lineage integration in connectors

2024-10-29 Thread Zhenqiu Huang (Jira)
Zhenqiu Huang created FLINK-36625: - Summary: Add helper classes for Lineage integration in connectors Key: FLINK-36625 URL: https://issues.apache.org/jira/browse/FLINK-36625 Project: Flink Is

Micro batching with flink

2024-10-29 Thread Anil Dasari
Hello team, I apologize for reaching out on the dev mailing list. I'm working on implementing micro-batching with near real-time processing. I've seen similar questions in the Flink Slack channel and user mailing list, but there hasn't been much discussion or feedback. Here are the options I've

[jira] [Created] (FLINK-36627) Failure to process a CSV file in Flink due to a character encoding mismatch: the file is in ISO-8859 and the application expects UTF-8.

2024-10-29 Thread Hector Miuler Malpica Gallegos (Jira)
Hector Miuler Malpica Gallegos created FLINK-36627: -- Summary: Failure to process a CSV file in Flink due to a character encoding mismatch: the file is in ISO-8859 and the application expects UTF-8. Key: FLINK-3662

kafka-connector random WakeupException after enabling watermark alignment

2024-10-29 Thread Alberto Lago
Hello, in Flink 19, AWS managed Flink flink-connector-kafka:3.3.0-1.19 After i enable watermark alignment at KafkaSource, It starts throwing uncaught WakeupException. It happens: * On every checkpoint unless i disable offset committing: setProperty(KafkaSourceOptions.COMMIT_OFFSETS_ON_CHECKPOI

[jira] [Created] (FLINK-36624) Log JobID in SourceCoordinator

2024-10-29 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-36624: -- Summary: Log JobID in SourceCoordinator Key: FLINK-36624 URL: https://issues.apache.org/jira/browse/FLINK-36624 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Elasticsearch v3.1 release

2024-10-29 Thread Ahmed Hamdy
Thanks Danny for the input, I figured out most of the steps require committer permissions, I will reach out offline for the next steps so thanks in advance for the assistance also. Best Regards Ahmed Hamdy On Mon, 14 Oct 2024 at 17:23, Danny Cranmer wrote: > Hey Ahmed, > > +1 for the release a

[ANNOUNCE] Apache Flink Kubernetes Operator 1.10.0 released

2024-10-29 Thread Őrhidi Mátyás
The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.10.0 The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. Please check out the release blog post for

Re: [DISCUSS] Release HBase connector with partial JDK support

2024-10-29 Thread Márton Balassi
+1 for Ferenc as the release manager and the content. On Mon, Oct 28, 2024 at 5:03 PM Ferenc Csaky wrote: > Hi, > > Based on this discussion I would like to suggest to move on with > the originally planned release with the HBase connector 4.0, that > will support 1.18, and 1.19. > > I volunteer

[jira] [Created] (FLINK-36623) Improve logging in DefaultStateTransitionManager

2024-10-29 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-36623: - Summary: Improve logging in DefaultStateTransitionManager Key: FLINK-36623 URL: https://issues.apache.org/jira/browse/FLINK-36623 Project: Flink Is

[jira] [Created] (FLINK-36626) Flink SQL JOINs behavior change from Flink 1.15 to Flink 1.18+

2024-10-29 Thread Eduardo Breijo (Jira)
Eduardo Breijo created FLINK-36626: -- Summary: Flink SQL JOINs behavior change from Flink 1.15 to Flink 1.18+ Key: FLINK-36626 URL: https://issues.apache.org/jira/browse/FLINK-36626 Project: Flink

[jira] [Created] (FLINK-36620) Add support for the flink-home parameter to be set in both “--flink-home $FLINK_HOME” and “--flink-home=$FLINK_HOME” formats

2024-10-29 Thread zjjiang (Jira)
zjjiang created FLINK-36620: --- Summary: Add support for the flink-home parameter to be set in both “--flink-home $FLINK_HOME” and “--flink-home=$FLINK_HOME” formats Key: FLINK-36620 URL: https://issues.apache.org/jira/br

[DISCUSS] Optimize the processing logic of TopNFunction to avoid the expiration of downstream operator states when state TTL is set

2024-10-29 Thread 李阳
Hello devs, I would like to initiate a discussion about the Flink TopNFunction. In our experience, we encountered the following issue while implementing the Top3Function. During this process, if no input has a timestamp smaller than the RowData currently held by the Top3Function, that operator wi

Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-10-29 Thread Ron Liu
I also have some questions: 1. Whether all SQL commands such as DDL & DML & SELECT are supported. 2. How to determine JobID and return JobID & ClusterId from the application cluster 3. How to dynamically download the JAR specified by the user when submitting the sql script, and whether it is possi

[DISCUSS][FLINK-36547] Add option to retain `RowKind` sematics for cdc formats

2024-10-29 Thread Yubin Li
Hi everyone, As official docs said, `RowKind` semantics have been changed after encode: -U -> -D, +D -> +I. In fact, we also have a demand to make it consistent in many scenarios, such as those that require different processing of -U/-D and +U/+I. I have taken advantage of the difference between

Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-10-29 Thread Ron Liu
Hi, Shengkai Thanks for initializing this FLIP, supports application mode for SQL Gateway is a great job. The FLIP design looks good to me. I've read the FLIP-316 which mentions supporting deploying SQL job to application clusters for interactive or non-interactive gateway mode. But I noticed t

[jira] [Created] (FLINK-36628) OpenTelemetryTestBase.eventuallyConsumeJson failed on AZP

2024-10-29 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-36628: -- Summary: OpenTelemetryTestBase.eventuallyConsumeJson failed on AZP Key: FLINK-36628 URL: https://issues.apache.org/jira/browse/FLINK-36628 Project: Flink Issue T

Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-10-29 Thread Feng Jin
Hi Shengkai, Thank you for the timely updates and replies: 1. I have been studying your POC code, introducing OperationExecutor in SqlRunner is a great idea, which can maximize compatibility with SqlClient. Especially some SQL statements cannot be executed directly in the Table Environmen

Re: Micro batching with flink

2024-10-29 Thread Venkatakrishnan Sowrirajan
Can you share more details on what do you mean by micro-batching? Can you explain with an example to understand it better? Thanks Venkat On Tue, Oct 29, 2024, 1:22 PM Anil Dasari wrote: > Hello team, > I apologize for reaching out on the dev mailing list. I'm working on > implementing micro-bat

Re: Micro batching with flink

2024-10-29 Thread Anil Dasari
Hi Venkat,Thanks for the reply.  Microbatching is a data processing technique where small batches of data are collected and processed together at regular intervals.However, I'm aiming to avoid traditional micro-batch processing by tagging records within a time window as a batch, allowing for ne