[jira] [Created] (FLINK-36980) Build a new introductory practice "Streaming ELT from MySQL to Kafka"

sukang created FLINK-36980: -- Summary: Build a new introductory practice "Streaming ELT from MySQL to Kafka" Key: FLINK-36980 URL: https://issues.apache.org/jira/browse/FLINK-36980 Proj

[jira] [Created] (FLINK-36466) Change default value of execution.runtime-mode from STREAMING to AUTOMATIC

Wenjun Ruan created FLINK-36466: --- Summary: Change default value of execution.runtime-mode from STREAMING to AUTOMATIC Key: FLINK-36466 URL: https://issues.apache.org/jira/browse/FLINK-36466 Project

[jira] [Created] (FLINK-36346) Remove deprecated API in flink-streaming-java module

Weijie Guo created FLINK-36346: -- Summary: Remove deprecated API in flink-streaming-java module Key: FLINK-36346 URL: https://issues.apache.org/jira/browse/FLINK-36346 Project: Flink Issue Type

[jira] [Created] (FLINK-36327) Remove the dependencies of the flink-scala and flink-streaming-scala modules from the table module.

xuyang created FLINK-36327: -- Summary: Remove the dependencies of the flink-scala and flink-streaming-scala modules from the table module. Key: FLINK-36327 URL: https://issues.apache.org/jira/browse/FLINK-36327

[jira] [Created] (FLINK-36280) Snapshot scan mode do not support batch execution mode, and not as expected in mode streaming

HunterXHunter created FLINK-36280: - Summary: Snapshot scan mode do not support batch execution mode, and not as expected in mode streaming Key: FLINK-36280 URL: https://issues.apache.org/jira/browse/FLINK-36280

[jira] [Created] (FLINK-35636) Streaming File Sink s3 end-to-end test did not finish after 900 seconds

Weijie Guo created FLINK-35636: -- Summary: Streaming File Sink s3 end-to-end test did not finish after 900 seconds Key: FLINK-35636 URL: https://issues.apache.org/jira/browse/FLINK-35636 Project: Flink

[jira] [Created] (FLINK-35284) Streaming File Sink end-to-end test times out

Ryan Skraba created FLINK-35284: --- Summary: Streaming File Sink end-to-end test times out Key: FLINK-35284 URL: https://issues.apache.org/jira/browse/FLINK-35284 Project: Flink Issue Type: Bug

Re: Bug report for reading Hive table as streaming source.

gt;> Best regards, >> Yuxia >> >> - 原始邮件 - >> 发件人: "Xiaolong Wang" >> 收件人: "dev" >> 发送时间: 星期四, 2024年 3 月 28日 下午 5:11:20 >> 主题: Re: Bug report for reading Hive table as streaming source. >> >> I think it worth mentioni

Re: Bug report for reading Hive table as streaming source.

> 主题: Re: Bug report for reading Hive table as streaming source. > > I think it worth mentioning in the documentation of Hive read that it > cannot read a table that has more than 32,767 partitions. > > On Thu, Mar 28, 2024 at 5:10 PM Xiaolong Wang > > wrote: > >

[jira] [Created] (FLINK-35042) Streaming File Sink s3 end-to-end test failed as TM lost

Weijie Guo created FLINK-35042: -- Summary: Streaming File Sink s3 end-to-end test failed as TM lost Key: FLINK-35042 URL: https://issues.apache.org/jira/browse/FLINK-35042 Project: Flink Issue

Re: Bug report for reading Hive table as streaming source.

Thanks for reporting. Could you please help create a jira about it? Best regards, Yuxia - 原始邮件 - 发件人: "Xiaolong Wang" 收件人: "dev" 发送时间: 星期四, 2024年 3 月 28日 下午 5:11:20 主题: Re: Bug report for reading Hive table as streaming source. I think it worth mentioning in the d

Re: Bug report for reading Hive table as streaming source.

24 at 5:00 PM Xiaolong Wang > wrote: > >> Hi, >> >> I found a weird bug when reading a Hive table as a streaming source. >> >> In summary, if the first partition is not time related, then the Hive >> table cannot be read as a streaming source. >>

Re: Bug report for reading Hive table as streaming source.

partition. On Tue, Mar 26, 2024 at 5:00 PM Xiaolong Wang wrote: > Hi, > > I found a weird bug when reading a Hive table as a streaming source. > > In summary, if the first partition is not time related, then the Hive > table cannot be read as a streaming source. > > e.g. >

Bug report for reading Hive table as streaming source.

Hi, I found a weird bug when reading a Hive table as a streaming source. In summary, if the first partition is not time related, then the Hive table cannot be read as a streaming source. e.g. I've a Hive table in the definition of ``` CREATE TABLE article ( id BIGINT, edition STRIN

[jira] [Created] (FLINK-34873) [Bug] After starting Streaming ELT from MySQL to StarRocks using Flink CDC 3.0, the newly created tables are not being synchronized.

Flink CDC Issue Import created FLINK-34873: -- Summary: [Bug] After starting Streaming ELT from MySQL to StarRocks using Flink CDC 3.0, the newly created tables are not being synchronized. Key: FLINK-34873

[jira] [Created] (FLINK-34792) [Bug] Incremental source never switches to streaming mode with disabled checkpointing and parallelism > 1

Flink CDC Issue Import created FLINK-34792: -- Summary: [Bug] Incremental source never switches to streaming mode with disabled checkpointing and parallelism > 1 Key: FLINK-34792 URL: ht

[jira] [Created] (FLINK-34694) Delete num of associations for streaming outer join

Roman Boyko created FLINK-34694: --- Summary: Delete num of associations for streaming outer join Key: FLINK-34694 URL: https://issues.apache.org/jira/browse/FLINK-34694 Project: Flink Issue Type

[jira] [Created] (FLINK-34665) Add streaming rule for union to Rand and it convert to StreamExecDeduplicate finally

Jacky Lau created FLINK-34665: - Summary: Add streaming rule for union to Rand and it convert to StreamExecDeduplicate finally Key: FLINK-34665 URL: https://issues.apache.org/jira/browse/FLINK-34665

[jira] [Created] (FLINK-34657) Implement Lineage Graph for streaming API use cases

Zhenqiu Huang created FLINK-34657: - Summary: Implement Lineage Graph for streaming API use cases Key: FLINK-34657 URL: https://issues.apache.org/jira/browse/FLINK-34657 Project: Flink Issue

[jira] [Created] (FLINK-34581) streaming code throws java.lang.reflect.InaccessibleObjectException

Henning Schmiedehausen created FLINK-34581: -- Summary: streaming code throws java.lang.reflect.InaccessibleObjectException Key: FLINK-34581 URL: https://issues.apache.org/jira/browse/FLINK-34581

[jira] [Created] (FLINK-34569) 'Streaming File Sink s3 end-to-end test' failed

Matthias Pohl created FLINK-34569: - Summary: 'Streaming File Sink s3 end-to-end test' failed Key: FLINK-34569 URL: https://issues.apache.org/jira/browse/FLINK-34569 Project: Flink

[jira] [Created] (FLINK-34316) Reduce instantiation of ScanRuntimeProvider for in streaming mode

Timo Walther created FLINK-34316: Summary: Reduce instantiation of ScanRuntimeProvider for in streaming mode Key: FLINK-34316 URL: https://issues.apache.org/jira/browse/FLINK-34316 Project: Flink

Broadcast hint support in Streaming SQL

different Join strategies for Join and there is no choice of join strategies in the stream. The join hints listed in the current flip should be ignored (maybe can be warned) in streaming mode. When in the future the stream mode has the choice of join strategies, I think that's a good time t

[jira] [Created] (FLINK-34238) In streaming mode, redundant exchange nodes can be optimally deleted in some cases

xuyang created FLINK-34238: -- Summary: In streaming mode, redundant exchange nodes can be optimally deleted in some cases Key: FLINK-34238 URL: https://issues.apache.org/jira/browse/FLINK-34238 Project

[jira] [Created] (FLINK-33967) Remove/Rename log4j2-test.properties in flink-streaming-java's test bundle

Koala Lam created FLINK-33967: - Summary: Remove/Rename log4j2-test.properties in flink-streaming-java's test bundle Key: FLINK-33967 URL: https://issues.apache.org/jira/browse/FLINK-33967 Project:

Re: [DISCUSSION] Consider Flink operator having a way to monitor the status of bounded streaming jobs after they finish or error?

only available in Flink 1.15 and above. >>> >>> Cheers, >>> Gyula >>> >>> On Thu, Dec 7, 2023 at 9:15 AM richard.su >> wrote: >>> >>>> Hi, Community, I had found out this issue, but I'm not sure this issue >>>

Re: [DISCUSSION] Consider Flink operator having a way to monitor the status of bounded streaming jobs after they finish or error?

t; > > Cheers, > > Gyula > > > > On Thu, Dec 7, 2023 at 9:15 AM richard.su > wrote: > > > >> Hi, Community, I had found out this issue, but I'm not sure this issue > >> have any solution. I have tried flink operator 1.6, which this

Re: [DISCUSSION] Consider Flink operator having a way to monitor the status of bounded streaming jobs after they finish or error?

erator 1.6, which this issue is >> still exist. >> >> If not, I think this could create a jira issue to following. >> >> When we create a bounded streaming jobs which will finally to become >> Finished status, after this job's status from

Re: [DISCUSSION] Consider Flink operator having a way to monitor the status of bounded streaming jobs after they finish or error?

erator 1.6, which this issue is >> still exist. >> >> If not, I think this could create a jira issue to following. >> >> When we create a bounded streaming jobs which will finally to become >> Finished status, after this job's status from

Re: [DISCUSSION] Consider Flink operator having a way to monitor the status of bounded streaming jobs after they finish or error?

; When we create a bounded streaming jobs which will finally to become > Finished status, after this job's status from Running to Finished, flink > will shut down kubernetes cluster, at code of flink-kubernetes package, > class KubernetesResourceManagerDriver's method deregisterA

[DISCUSSION] Consider Flink operator having a way to monitor the status of bounded streaming jobs after they finish or error?

Hi, Community, I had found out this issue, but I'm not sure this issue have any solution. I have tried flink operator 1.6, which this issue is still exist. If not, I think this could create a jira issue to following. When we create a bounded streaming jobs which will finally to become Fin

Re: Streaming data from AWS S3 Source

Hi Neelabh You can use FileSystem Connector . for DataStream [1] for TableAPI [2] . And you need to put necessary dependency to your flink environment. [3] For Flink SQL setup, you can reference `sql getting started module`[4] [1]. https://nightlies.apache.org/flink/flink-docs-master/docs/c

Streaming data from AWS S3 Source

Hey Team, I want to stream data from AWS S3 source as they are being generated by an event to Apache Flink Stream for a data transformation job. I found out about FileSystem SQL Connector but need some ref

[jira] [Created] (FLINK-33606) Support streaming mode in Flink JDBC driver

Sangram Reddy created FLINK-33606: - Summary: Support streaming mode in Flink JDBC driver Key: FLINK-33606 URL: https://issues.apache.org/jira/browse/FLINK-33606 Project: Flink Issue Type

[jira] [Created] (FLINK-33536) Flink Table API CSV streaming sink throws "IOException: Stream closed"

Prabhu Joseph created FLINK-33536: - Summary: Flink Table API CSV streaming sink throws "IOException: Stream closed" Key: FLINK-33536 URL: https://issues.apache.org/jira/browse/FLINK-33536

[jira] [Created] (FLINK-32821) Streaming examples failed to execute due to error in packaging

Zhanghao Chen created FLINK-32821: - Summary: Streaming examples failed to execute due to error in packaging Key: FLINK-32821 URL: https://issues.apache.org/jira/browse/FLINK-32821 Project: Flink

[jira] [Created] (FLINK-32795) Release Testing: Verify FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

Qingsheng Ren created FLINK-32795: - Summary: Release Testing: Verify FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode Key: FLINK-32795 URL: https

[jira] [Created] (FLINK-32702) streaming examples could not run

Weihua Hu created FLINK-32702: - Summary: streaming examples could not run Key: FLINK-32702 URL: https://issues.apache.org/jira/browse/FLINK-32702 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-32344) MongoDB connector support unbounded streaming read via ChangeStream feature

Jiabao Sun created FLINK-32344: -- Summary: MongoDB connector support unbounded streaming read via ChangeStream feature Key: FLINK-32344 URL: https://issues.apache.org/jira/browse/FLINK-32344 Project

[jira] [Created] (FLINK-31728) Remove Scala API dependencies from batch/streaming examples

Chesnay Schepler created FLINK-31728: Summary: Remove Scala API dependencies from batch/streaming examples Key: FLINK-31728 URL: https://issues.apache.org/jira/browse/FLINK-31728 Project: Flink

[jira] [Created] (FLINK-31429) Support CTAS(create table as) streaming job with schema changes in table store

Caizhi Weng created FLINK-31429: --- Summary: Support CTAS(create table as) streaming job with schema changes in table store Key: FLINK-31429 URL: https://issues.apache.org/jira/browse/FLINK-31429 Project

[jira] [Created] (FLINK-31376) CSVReader for streaming does not support splittable

ramkrishna.s.vasudevan created FLINK-31376: -- Summary: CSVReader for streaming does not support splittable Key: FLINK-31376 URL: https://issues.apache.org/jira/browse/FLINK-31376 Project

[jira] [Created] (FLINK-31311) Supports Bounded Watermark streaming read

Jingsong Lee created FLINK-31311: Summary: Supports Bounded Watermark streaming read Key: FLINK-31311 URL: https://issues.apache.org/jira/browse/FLINK-31311 Project: Flink Issue Type

[jira] [Created] (FLINK-31201) Provides option to sort partition for full stage in streaming read

Jingsong Lee created FLINK-31201: Summary: Provides option to sort partition for full stage in streaming read Key: FLINK-31201 URL: https://issues.apache.org/jira/browse/FLINK-31201 Project: Flink

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

ons when jobs are submitted concurrently. But I'm > sorry that I may not have explained the relationship between it and > sink/source clearly. Generally speaking, the interactive between Flink and > MetaService is as: > 1. When the Client submits a flink job (streaming&batc

[jira] [Created] (FLINK-31072) Introduce streaming-read-atomic to ensure UB and UA cannot be split

Jingsong Lee created FLINK-31072: Summary: Introduce streaming-read-atomic to ensure UB and UA cannot be split Key: FLINK-31072 URL: https://issues.apache.org/jira/browse/FLINK-31072 Project: Flink

[jira] [Created] (FLINK-31037) Table Store supports streaming reading a whole snapshot in one checkpoint

Jingsong Lee created FLINK-31037: Summary: Table Store supports streaming reading a whole snapshot in one checkpoint Key: FLINK-31037 URL: https://issues.apache.org/jira/browse/FLINK-31037 Project

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

I'm sorry that I may not have explained the relationship between it and sink/source clearly. Generally speaking, the interactive between Flink and MetaService is as: 1. When the Client submits a flink job (streaming&batch), it interacts with MetaService through Catalog in CatalogManager, inc

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

This way, latency would be behaving very much like it currently does in Flink. For example if we have a following streaming SQL: INSERT INTO alerts_with_user SELECT * FROM alerts a, users u WHERE a.user_id = u.id If there is some lag in the users table, alerts would be still generated. Downstream a

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

mechanism > instead > >> of restricting it to TableStore+ETL graphs? For example, it seems to me > to > >> be possible and valuable to define instead the contract that > sources/sinks > >> need to implement in order to participate in globally consistent > s

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

s >> need to implement in order to participate in globally consistent snapshots. > > A general consistency mechanism is cool! In my mind, the overall > `consistency system` consists of three components: Streaming & Batch ETL, > Streaming & Batch Storage and MetaService.

[jira] [Created] (FLINK-30867) Support streaming read changes from overwrite

yuzelin created FLINK-30867: --- Summary: Support streaming read changes from overwrite Key: FLINK-30867 URL: https://issues.apache.org/jira/browse/FLINK-30867 Project: Flink Issue Type: New Feature

Re: [VOTE] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

; > > > > > > > Best, > > > > Jark > > > > > > > > > 2023年1月9日 15:46，Jing Ge 写道： > > > > > > > > > > Hi, > > > > > > > > > > I'd like to start a vote on FLIP-279 Unified

[jira] [Created] (FLINK-30841) Incorrect calc merge in streaming

lincoln lee created FLINK-30841: --- Summary: Incorrect calc merge in streaming Key: FLINK-30841 URL: https://issues.apache.org/jira/browse/FLINK-30841 Project: Flink Issue Type: Bug

Re: [DISCUSS] streaming shuffle to improve data clustering and tame small files problem

: > Hi, > > We had a proposal to add a streaming shuffling stage in the Flink Iceberg > sink to to improve data clustering and tame the small files problem [1]. > > Here are a couple of common use cases. > * Event time partitioned table where we can get small files problem due

[DISCUSS] streaming shuffle to improve data clustering and tame small files problem

Hi, We had a proposal to add a streaming shuffling stage in the Flink Iceberg sink to to improve data clustering and tame the small files problem [1]. Here are a couple of common use cases. * Event time partitioned table where we can get small files problem due to skewed and long-tail

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

dated FLIP. Looking forward to your feedback, THX [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-276%3A+Data+Consistency+of+Streaming+and+Batch+ETL+in+Flink+and+Table+Store#FLIP276:DataConsistencyofStreamingandBatchETLinFlinkandTableStore-DataConsistencyType Best, Shammon On Sat, J

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

uable to define instead the contract that sources/sinks need to implement in order to participate in globally consistent snapshots. A general consistency mechanism is cool! In my mind, the overall `consistency system` consists of three components: Streaming & Batch ETL, Streaming & Batch Storage

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

greement on an important point: we finally need `Timestamp Barrier > > Mechanism` to support data consistency. But according to our (@jinsong > lee > > and I) opinions, the total design and implementation based on 'Timestamp > > Barrier' will be too complex, and it's

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

Timestamp Barrier > Mechanism` to support data consistency. But according to our (@jinsong lee > and I) opinions, the total design and implementation based on 'Timestamp > Barrier' will be too complex, and it's also too big in one FLIP. > > So we‘d like to use FLIP-

[ANNOUNCE] Share your Streaming Stories with us at Current 2023

Do you have a great data streaming story to share? We want to hear from you! Speaking at Current 2023 is a great way to connect with hundreds of your peers, become more involved in the data streaming community, and have a public platform for you to share your story of the future of streaming and

[RESULT][VOTE] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

Hi all, I am happy to announce that FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode [1] has been accepted. There are 3 binding votes and 1 non-binding vote [2]: - Jark Wu (binding) - Shengkai Fang (binding) - Shammon FY (non-binding

Re: [VOTE] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

023年1月12日周四 19:22写道： > > > > > +1 (binding) > > > Thank you for driving this effort. > > > > > > Best, > > > Jark > > > > > > > 2023年1月9日 15:46，Jing Ge 写道： > > > > > > > > Hi, > > > > > > &

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

FLIP. So we‘d like to use FLIP-276[1] as an overview design of data consistency in Flink Streaming and Batch ETL based on `Timestamp Barrier`. @jinsong and I hope that we can reach an agreement on the overall design in FLINK-276 first, and then on the basic of FLIP-276 we can create other FLIPs with

Re: [VOTE] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

t; > > > > 2023年1月9日 15:46，Jing Ge 写道： > > > > > > Hi, > > > > > > I'd like to start a vote on FLIP-279 Unified the max display column > width > > > for SqlClient and Table APi in both Streaming and Batch execMode. The > > > disc

Re: [VOTE] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

splay column width > > for SqlClient and Table APi in both Streaming and Batch execMode. The > > discussion can be found at [1]. > > > > The vote will last for at least 72 hours (Jan 12th at 9:00 GMT) unless > > there is an objection or insufficient votes. > > >

Re: [VOTE] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

+1 (binding) Thank you for driving this effort. Best, Jark > 2023年1月9日 15:46，Jing Ge 写道： > > Hi, > > I'd like to start a vote on FLIP-279 Unified the max display column width > for SqlClient and Table APi in both Streaming and Batch execMode. The > discussion can

[jira] [Created] (FLINK-30605) 'Streaming File Sink end-to-end test' fails with UnsupportedOperationException

Matthias Pohl created FLINK-30605: - Summary: 'Streaming File Sink end-to-end test' fails with UnsupportedOperationException Key: FLINK-30605 URL: https://issues.apache.org/jira/browse/F

[VOTE] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

Hi, I'd like to start a vote on FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode. The discussion can be found at [1]. The vote will last for at least 72 hours (Jan 12th at 9:00 GMT) unless there is an objection or insufficient

Re: [DISCUSS] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

t this proposal. I will start a voting >> thread. Thanks! >> >> Best regards, >> Jing >> >> On Fri, Dec 16, 2022 at 11:09 AM Jing Ge wrote: >> >> > Hi Dev, >> > >> > I'd like to start a discussion about FLIP-279: Unified the

Re: [DISCUSS] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

e max display > > column width for SqlClient and Table APi in both Streaming and Batch > > execMode[1]. > > > > On one side, currently, there is only one ConfigOption > > > SqlClientOptions.DISPLAY_MAX_COLUMN_WIDTH(sql-client.display.max-column-width') > > coul

Re: [DISCUSS] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

scussion about FLIP-279: Unified the max display > column width for SqlClient and Table APi in both Streaming and Batch > execMode[1]. > > On one side, currently, there is only one ConfigOption > SqlClientOptions.DISPLAY_MAX_COLUMN_WIDTH(sql-client.display.max-column-width') >

Re: Streaming queries in FTS using Kafka log

ink-table-store/blob/7e0d55ff3dc9fd48455b17d9a439647b0554d020/flink-table-store-connector/src/main/java/org/apache/flink/table/store/connector/source/TableStoreSource.java#L136-L141 Best, Alex On Fri, Dec 16, 2022 at 5:28 PM Alexander Sorokoumov < asorokou...@confluent.io> wrote: > Hello community, >

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

lerance mechanism from the subject of the data consistency. > > > > > > Best, > > > Piotrek > > > > > > czw., 15 gru 2022 o 13:01 Shammon FY napisał(a): > > > > > > > Hi Piotr, > > > > > > > > It's kind of amazin

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

t; > Best, > > Piotrek > > > > czw., 15 gru 2022 o 13:01 Shammon FY napisał(a): > > > > > Hi Piotr, > > > > > > It's kind of amazing about the image, it's a simple example and I have > to > > > put it in a document >

Streaming queries in FTS using Kafka log

Hello community, I want to ask about streaming queries with Flink Table Store. After reading the documentation on Streaming Queries [1], I was under the impression that only tables with LogStore-over-TableStore and No Changelog Producer need the normalization step since the Kafka log has the

[DISCUSS] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

Hi Dev, I'd like to start a discussion about FLIP-279: Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode[1]. On one side, currently, there is only one ConfigOption SqlClientOptions.DISPLAY_MAX_COLUMN_WIDTH(sql-client.display.max-column-

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

amazing about the image, it's a simple example and I have to > > put it in a document > > > > > https://bytedance.feishu.cn/docx/FC6zdq0eqoWxHXxli71cOxe9nEe?from=from_copylink > > :) > > > > > Does it have to be combining watermarks and checkpoint barri

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

and from "Watermark" is the > "Consistency Semantics", > > 1) Only "Align Data" can reach data consistency when performing queries on > upstream and downstream tables. I gave an example of "Global Count Tables" > in our previous discussion. We n

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

downstream tables. I gave an example of "Global Count Tables" in our previous discussion. We need a "Align Event" in the streaming processing, it's the most basic. 2) Only "Timestamp" can provide complete consistency semantics. You gave some good examples about &

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

runtime at least) like two completely different mechanisms. Not one leading to the other. Best, Piotrek śr., 14 gru 2022 o 15:19 Shammon FY napisał(a): > Hi Piotr, > > Thanks for your valuable input which makes me consider the core point of > data consistency in deep. I'd like to define

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

Hi Piotr, Thanks for your valuable input which makes me consider the core point of data consistency in deep. I'd like to define the data consistency on the whole streaming & batch processing as follows and I hope that we can have an agreement on it: BOutput = Fn(BInput), BInput is

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

> table, this will not bring too much complexity to the system. We only need > to support different strategies when calculating table versions for query. > So we give this decision to users, who can use "consistency.type" to set > different consistency in "Catalog". We can

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

rs, who can use "consistency.type" to set different consistency in "Catalog". We can continue to refine this later. For example, dynamic parameters support different consistency requirements for each query 2) MetaService module Many Flink streaming jobs use application mode, and

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

ROM Table1 GROUP BY word > > 2. ETL2 from Table2 to Table3: INSERT INTO Table3 SELECT cnt, count(*) FROM > Table2 GROUP BY cnt > > ETL1 has 2 subtasks to read multiple buckets from Table1, where subtask1 > reads streaming data as [a, b, c, a, d, a, b, c, d ...] and subtask2 reads

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

ts from Table1, where subtask1 reads streaming data as [a, b, c, a, d, a, b, c, d ...] and subtask2 reads streaming data as [a, c, d, q, a, v, c, d ...]. 1. Unbounded streaming data is divided into multiple sets according to some semantic requirements. The most extreme may be one set for each data. As

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

aligned checkpoints in your current proposal. 2. I don't understand why this is a problem? Just store in the "sink table" what's the watermark (T1), and downstream jobs should process the data with that "watermark" anyway. Record "V13" should be treated

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

nges that need to be made to Flink in order to > support this? > > Best regards, > > Martijn > > > On Thu, Dec 1, 2022 at 4:49 AM Shammon FY wrote: > > > Hi devs: > > > > I'd like to start a discussion about FLIP-276: Data Consistency of > > Streaming

FW: Why streaming over agg not supporting ranged upper boundary?

From: Renjie Liu Date: Tuesday, November 1, 2022 at 15:13 To: u...@flink.apache.org Subject: Why streaming over agg not supporting ranged upper boundary? Hi, flink community: Per the doc here https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/table/sql/queries/over-agg

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

22 at 4:49 AM Shammon FY wrote: > Hi devs: > > I'd like to start a discussion about FLIP-276: Data Consistency of > Streaming and Batch ETL in Flink and Table Store[1]. In the whole data > stream processing, there are consistency problems such as how to manage the > dependenc

[DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

Hi devs: I'd like to start a discussion about FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store[1]. In the whole data stream processing, there are consistency problems such as how to manage the dependencies of multiple jobs and tables, how to define and handl

[jira] [Created] (FLINK-30006) Cannot remove columns that are incorrectly considered constants from an Aggregate In Streaming

lincoln lee created FLINK-30006: --- Summary: Cannot remove columns that are incorrectly considered constants from an Aggregate In Streaming Key: FLINK-30006 URL: https://issues.apache.org/jira/browse/FLINK-30006

[jira] [Created] (FLINK-29968) Update streaming query document for Table Store to include full compaction changelog producer

Caizhi Weng created FLINK-29968: --- Summary: Update streaming query document for Table Store to include full compaction changelog producer Key: FLINK-29968 URL: https://issues.apache.org/jira/browse/FLINK-29968

[jira] [Created] (FLINK-29874) Streaming File Sink end-to-end test failed on azure

Leonard Xu created FLINK-29874: -- Summary: Streaming File Sink end-to-end test failed on azure Key: FLINK-29874 URL: https://issues.apache.org/jira/browse/FLINK-29874 Project: Flink Issue Type

[jira] [Created] (FLINK-29838) Hive streaming sink for partitioned table should contain metastore by default

luoyuxia created FLINK-29838: Summary: Hive streaming sink for partitioned table should contain metastore by default Key: FLINK-29838 URL: https://issues.apache.org/jira/browse/FLINK-29838 Project: Flink

[jira] [Created] (FLINK-29678) Data may loss when sink bounded stream into filesystem with auto compact enabled in streaming mode

luoyuxia created FLINK-29678: Summary: Data may loss when sink bounded stream into filesystem with auto compact enabled in streaming mode Key: FLINK-29678 URL: https://issues.apache.org/jira/browse/FLINK-29678

[jira] [Created] (FLINK-29226) Throw exception for streaming insert overwrite

Jingsong Lee created FLINK-29226: Summary: Throw exception for streaming insert overwrite Key: FLINK-29226 URL: https://issues.apache.org/jira/browse/FLINK-29226 Project: Flink Issue Type

[jira] [Created] (FLINK-29179) Translate "Temporal Table Function" page of "Streaming Concepts" into Chinese

ChunJi created FLINK-29179: -- Summary: Translate "Temporal Table Function" page of "Streaming Concepts" into Chinese Key: FLINK-29179 URL: https://issues.apache.org/jira/browse/FLINK-29179

[jira] [Created] (FLINK-29070) Provide a option to force the removal of the normalize node when streaming read

Jingsong Lee created FLINK-29070: Summary: Provide a option to force the removal of the normalize node when streaming read Key: FLINK-29070 URL: https://issues.apache.org/jira/browse/FLINK-29070

[jira] [Created] (FLINK-28728) Support to set the end offset for streaming connector

tanjialiang created FLINK-28728: --- Summary: Support to set the end offset for streaming connector Key: FLINK-28728 URL: https://issues.apache.org/jira/browse/FLINK-28728 Project: Flink Issue

  1   2   3   4   5   6   7   8   9   10   >