Re: [ANNOUNCE] Apache Flink CDC 3.3.0 released

Thanks Hang for the great work and all contributors who involved! Best, Leonard > 2025年1月21日 17:48，Hang Ruan 写道： > > The Apache Flink community is very happy to announce the release of > Apache Flink CDC 3.3.0. > > Apache Flink CDC is a distributed data integration tool for real time > data an

Re: Flink CDC blog

I’ve added the two blogposts to Flink CDC Github wiki page[1], FYI Best, Leonard [1]https://github.com/apache/flink-cdc/wiki/Flink-CDC-Blog > On Dec 12, 2024, at 2:00 PM, Leonard Xu wrote: > > Thanks Robin and Andrew for your sharing, and these information encourage us > a lot. &

Re: Flink CDC blog

Thanks Robin and Andrew for your sharing, and these information encourage us a lot. I’d like to add your blogpost to Flink CDC wiki at least, and I will start a discussion in dev mailing about whether to add blogpost directory in Flink CDC website later. Best, Leonard > On Dec 12, 2024, at 3

Re: Cdc on HA postgresql

Thanks Gunnar for sharing the information, looking forward to your blogpost. Best, Leonard > On Nov 26, 2024, at 3:13 PM, Gunnar Morling > wrote: > > Hey all, > > Postgres 17 supports failover slots, i.e. a replication slot can be created > on the primary instance and its state will be sync

Re: Request to joining Apache Flink community on Slack

the slack community ? > thanks in advance. > > Regards, > Anil > > On Wed, Oct 9, 2024 at 8:29 PM Leonard Xu <mailto:xbjt...@gmail.com>> wrote: > Welcome Ken, I’ve sent the invitation to your email. > > > Best, > Leonard > > >> 2024年

Re: Flink SQL CDC connector do nothing to the setting "debizum.*" when create the source table

> 2024年10月10日 上午4:34，Yaroslav Tkachenko 写道： > > Hi Ken, > > Snapshotting is implemented differently in Flink CDC, it doesn't re-use > Debezium's implementation. So you can override some Debezium properties using > "debezium.", but not "debezium.snapshot.". Yeah, Yaroslav is right that Flink

Re: Request to joining Apache Flink community on Slack

Welcome Ken, I’ve sent the invitation to your email. Best, Leonard > 2024年10月10日 上午3:52，Ken CHUAN YU 写道： > > Hi there > > I like to join the Apache Flink community on Slack, my mail address for slack > is: ken.h...@vestiairecollective.com >

Re: [ANNOUNCE] Apache Flink CDC 3.2.0 released

Nice! Thanks to @Qingsheng and all contributors involved for making this happen! Best, Leonard > 2024年9月5日 下午4:12，Qingsheng Ren 写道： > > The Apache Flink community is very happy to announce the release of Apache > Flink CDC 3.2.0. > > Apache Flink CDC is a distributed data integration to

Re: [ANNOUNCE] Apache Flink CDC 3.1.1 released

Congratulations! Thanks Qingsheng for the release work and all contributors involved. Best, Leonard > 2024年6月18日 下午11:50，Qingsheng Ren 写道： > > The Apache Flink community is very happy to announce the release of Apache > Flink CDC 3.1.1. > > Apache Flink CDC is a distributed data integration

Re: Flink kinesis connector 4.3.0 release estimated date

Hey, Vararu The kinesis connector 4.3.0 release is under vote phase and we hope to finalize the release work in this week if everything goes well. Best, Leonard > 2024年5月22日 下午11:51，Vararu, Vadim 写道： > > Hi guys, > > Any idea when the 4.3.0 kinesis connector is estimated to be released? >

Re: [ANNOUNCE] Apache Flink CDC 3.1.0 released

Congratulations ! Thanks Qingsheng for the great work and all contributors involved !! Best, Leonard > 2024年5月17日 下午5:32，Qingsheng Ren 写道： > > The Apache Flink community is very happy to announce the release of > Apache Flink CDC 3.1.0. > > Apache Flink CDC is a distributed data integration

[ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Hi devs and users, We are thrilled to announce that the donation of Flink CDC as a sub-project of Apache Flink has completed. We invite you to explore the new resources available: - GitHub Repository: https://github.com/apache/flink-cdc - Flink CDC Documentation: https://nightlies.apache.org/f

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congratulations, thanks release managers and all involved for the great work! Best, Leonard > 2024年3月18日 下午4:32，Jingsong Li 写道： > > Congratulations! > > On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> wrote: >> >> Congratulations, thanks for the great work! >> >> Best, >> Rui

Re: [ANNOUNCE] Apache Flink 1.18.1 released

all contributors of the Apache Flink community who > made this release possible! Special thanks to @Qingsheng Ren @Leonard Xu > @Xintong Song @Matthias Pohl @Martijn Visser for the support during this > release. > > A Jira task series based on the Flink release wiki has been created for

Re: [ANNOUNCE] Apache Flink 1.17.2 released

Thanks Yun for driving the release. Thanks a lot to everyone that has contributed with bug fixes and other improvements! Best, Leonard > 2023年11月29日 下午1:05，Yun Tang 写道： > > The Apache Flink community is very happy to announce the release of Apache > Flink 1.17.2, which is the second bugfi

Re: dependency error with latest Kafka connector

> built a fat uber jar from quickstart with Flink 1.18.0 for > flink-streaming-java and flink-clients, and flink-connector-kafka version > 3.0.1-1.18 > then submitted to local Flink cluster 1.18.0. Things worked as expected and > the job ran fine. Hey，@Gordan I guess things may work as expecte

Re: dependency error with latest Kafka connector

Hi, Gurnterh It seems a bug for me that 3.0.1-1.18 flink Kafka connector use flink 1.17 dependency which lead to your issue. I guess we need propose a new release for Kafka connector for fix this issue. CC: Gordan, Danny, Martijn Best, Leonard > 2023年11月14日 下午6:53，Alexey Novakov via user

Re: [ANNOUNCE] Apache Flink 1.18.0 released

Congratulations, Well done! Best, Leonard On Fri, Oct 27, 2023 at 12:23 AM Lincoln Lee wrote: > Thanks for the great work! Congrats all! > > Best, > Lincoln Lee > > > Jing Ge 于2023年10月27日周五 00:16写道： > > > The Apache Flink community is very happy to announce the release of > Apache > > Flink 1.

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

Congrats to all ! It will be helpful to promote Apache Flink if we can add a page to our website like others[2]. I’ve created an issue to improve this. Best, Leonard [1] https://issues.apache.org/jira/browse/FLINK-32555 [2] https://spark.apache.org/news/sigmod-system-award.html

Re: Unsubscribe

Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org, and you can refer [1][2] for more details. 请发送任意内容的邮件到 user-unsubscr...@flink.apache.org 地址来取消订阅来自 user@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Leonard [1] https://

[ANNOUNCE] Apache flink-connector-pulsar v3.0.1 released

The Apache Flink community is very happy to announce the release of Apache flink-connector-pulsar v3.0.1. This release is compatible with Flink 1.16.x series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data stream

Re: 取消订阅

可以发送任意内容的邮件到 user-unsubscr...@flink.apache.org 取消订阅来自 user@flink.apache.org 邮件列表的邮件，其他邮件列表的订阅和退订管理也类似，邮件列表的订阅管理，可以参考[1] 祝好， Leonard Xu [1] https://flink.apache.org/community.html#how-to-subscribe-to-a-mailing-list > On Apr 18, 2023, at 2:53 PM, wangw...@sina.cn wrote: > > 取消订阅

Re: Issue with the flink version 1.10.1

Hi, Kiran To be honest, both 1.10 and 1.9 are pretty old version, it’s hard to fix and release a bugfix version for 1.10.1 even the community help troubleshoot your issue. So, could you try latest versions like Flink 1.16.1 or 1.17.0? Best, Leonard > On Mar 27, 2023, at 8:28 PM, Kiran Kumar K

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

Congratulations! Best, Leonard > On Mar 27, 2023, at 5:23 PM, Yu Li wrote: > > Dear Flinkers, > > As you may have noticed, we are pleased to announce that Flink Table Store > has joined the Apache Incubator as a separate project called Apache > Paimon(incubating) [1] [2] [3]. The new projec

[ANNOUNCE] Apache Flink 1.17.0 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.17.0, which is the first release for the Apache Flink 1.17 series. Apache Flink® is an open-source unified stream and batch data processing framework for distributed, high-performing, always-available, and accurat

Re: IntervalJoin invisibly becomes a regular Join - why?

> > CREATE TEMPORARY VIEW filteredResults AS > SELECT * from suspiciousOrders WHERE small_ts > large_ts; Looks like after added the condition, the final expanded query should not match the condition[1] of an interval join that leads to the planner recognize it as an interval join. It’s not

Re: is there any detrimental side-effect if i set the max parallelismas 32768

> > 退订 请发送任意邮件到 user-unsubscr...@flink.apache.org 取消 订阅来自 user@flink.apache.org 邮件列表的邮件，发送到 user@flink.apache.org 是不会取消订阅的。 > 发自我的iPhone > > > -- Original -- > From: Tony Wei > Date: Tue,Mar 14,2023 1:11 PM > To: David Anderson > Cc: Hangxiang Yu , user >

[SUMMARY] Flink 1.17 Release Sync 3/7/2023

Hi devs and users, I'd like to share some highlights from Flink 1.17 release sync on 3/7/2023. 1.17 Blockers: - Currently, there is one blocker issue (FLINK-31351[1]) that needs to be resolved before we can create a votable RC1. Our contributors are working hard to fix it as soon as possible

Re: 退订

Please send an email to user-unsubscr...@flink.apache.org to unsubscribe > On Mar 3, 2023, at 8:42 AM, zhangjunjie wrote: > > 退订 > >

Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

Thanks Yanfei for driving the release ! ! Best, Leonard > On Jan 31, 2023, at 3:43 PM, Yun Tang wrote: > > Thanks Yuanfei for driving the frocksdb release! > > Best > Yun Tang > From: Yuan Mei > Sent: Tuesday, January 31, 2023 15:09 > To: Jing Ge > Cc: Yanfei Lei ; d...@flink.apache.org >

[SUMMARY] Flink 1.17 Release Sync 12/13/2022

Hi devs and users, I’d like to share the highlights about the 1.17 release sync on 12/13/2022. - Release tracking page: - 1.17 development is moving forward [1], we have 5 weeks remaining - @committers Please continuously update the the progress in the 1.17 page - Externalized Connect

Re: Weird Flink SQL error

Do not trust the line number from sql parser exception, you should use ROW in your DDL when you declare a composite row type, try the following: CREATE TABLE test_content_metrics ( dt STRING NOT NULL, `body` ROW< `platform_id` BIGINT, `content_id` STRING > ) PARTITIONED BY (

[SUMMARY] Flink 1.17 Release Sync 11/15/2022

Hi devs and users, I’d like to share some highlights about the 1.17 release sync on 11/15/2022. - Release tracking page: - The community has collected some great features on the 1.17 page[1] - @committers Please continuously update the page in the coming week - JIRA account apply

Re: [ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

Thanks Chesnay and Martijn for the great work! I believe the flink-connector-shared-utils[1] you built will help Flink connector developers a lot. Best, Leonard [1] https://github.com/apache/flink-connector-shared-utils > 2022年11月10日 下午9:53，Martijn Visser 写道： > > Really happy with the firs

Re: UDFs classloading changes in 1.16

Thanks Alexander for reporting this issue, Could you open a jira ticket as well? CC: Shengkai, please take a look this ticket, looks like an incompatibility change. Best, Leonard > 2022年11月4日 下午6:15，Alexander Fedulov 写道： > > Hi everyone, > > 1.16 introduced quite a lot of changes with resp

Re: OutOfMemoryError (java heap space) on small, local test

Hi, Matt I’ve checked your job is pretty simple, I've CC Xingbo who is a PyFlink expert to help take a quick look. Best， Leonard > 2022年10月31日 上午11:47，Matt Fysh 写道： > > Hi there, > > I am running a local test with: > * source = env.from_collection > * sink = datastream.execute_and_collect

Re: Performing left join between two streams

Hi, Lalwani Flink does not support outer join on two data streams now[1], you can use the DataStream API ds1.coGroup(ds2) as a workaround. Flink SQL support outer joins well, you can also try SQL way[2]. Best, Leonard [1] https://issues.apache.org/jira/browse/FLINK-4187 [2] https://nightlie

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

Thanks Qingsheng for starting this thread. +1 on reverting sink metric name and releasing 1.15.3 to fix this inconsistent behavior. Best, Leonard > 2022年10月10日 下午3:06，Jark Wu 写道： > > Thanks for discovering this problem, Qingsheng! > > I'm also +1 for reverting the breaking changes. >

Re: accuracy validation of streaming pipeline

Hi, vtygoss > I'm working on migrating from full-data-pipeline(with spark) to > incremental-data-pipeline(with flink cdc), and i met a problem about accuracy > validation between pipeline based flink and spark. Glad to hear that ! > For bounded data, it's simple to validate the two result se

Re: table.local-time-zone not working

ng property as follows: > > Configuration tconf = tenv.getConfig().getConfiguration(); > tconf.setString("table.local-time-zone", "America/Los_Angeles"); > > tenv.getConfig().setLocalTimeZone(ZoneId.of("America/Los_Angeles")); &g

Re: table.local-time-zone not working

Hi, Lalwani Could you share how you set this property and your Flink version? Best, Leonard > 2022年5月10日 下午3:01，Surendra Lalwani 写道： > > Hi Team, > > I have tried using this property > table.local-time-zone > > But it seems like it is not making any impact, on calling functions like > PRO

Re: Flink-SQL returning duplicate rows for some records

Hi Joost Could you share your flink version and the two records in debezium-json format which produced by two MS SQL UPDATE statement ? Best, Leonard > 2022年5月2日 下午9:59，Joost Molenaar 写道： > > Hello all, > > I'm trying to use Flink-SQL to monitor a Kafka topic that's populated by > Debezium,

Re: [ANNOUNCE] Apache Flink 1.1.4.4 released

Thanks a lot for being our release manager Konstantin and everyone who involved! Best, Leonard > 2022年3月15日 下午9:34，Martijn Visser 写道： > > Thank you Konstantin and everyone who contributed!

Re: [Table API] [JDBC CDC] Caching Configuration from MySql instance needed in multiple Flink Jobs

Hello, Dan > 2022年2月21日 下午9:11，Dan Serb 写道： > 1.Have a processor that uses Flink JDBC CDC Connector over the table that > stores the information I need. (This is implemented currently - working) You mean you’ve implemented a Flink JDBC Connector？ Maybe the Flink CDC Connectors[1] would help yo

Re: CDC using Query

Hello, mohan > 1. Does flink have any support to track any missed source Jdbc CDC records ? Flink CDC Connector provides Exactly once semantics which means they won’t miss records. Tips: The Flink JDBC Connector only Scan the database once which can not continuously read CDC stream. > 2. What

Re: [ANNOUNCE] Apache Flink 1.14.2 / 1.13.5 / 1.12.7 / 1.11.6 released

I guess this is related to publishers everywhere are updating their artifacts in response to the log4shell vulnerability[1]. All we can do and need to do is wait. ☕️ Best, Leonard [1] https://issues.sonatype.org/browse/OSSRH-76300 > 2021年12月17

Re: flink hang : es_rejected_execution_exception导致的flink 假死问题

Hi, ren I think the root cause is you didn’t set proper FailureHandler for ElasticSearch connector, the `RetryRejectedExecutionFailureHandler` can resolve your issue, you can see ElasticSearch connector docs[1] for more information. You can also set 'connector.failure-handler to 'retry-rejected

Re: window join in flink sql

Tips: The documentation in https://ci.apache.org/projects/flink is no longer updated any more, The new documentation site is https://nightlies.apache.org/flink/ , please use the new one. Best, Leonard > 在 2021年10月

Re: database as stream source issue

Hi, Qihua JDBC connector support Postgres dailect, but it is implemented as a bounded source which means it only captures the snapshot data(the existed records) and then finished its work, the new adding transaction log records (as known as MySql bin-log) won’t be captured. You should receive a

Re: [ANNOUNCE] Apache Flink 1.13.3 released

Thanks to Chesnay & Martijn and everyone who made this release happen. > 在 2021年10月21日，20:08，Martijn Visser 写道： > > Thank you Chesnay, Leonard and all contributors! > > On Thu, 21 Oct 2021 at 13:40, Jingsong Li > wrote: > Thanks, Chesnay & Martijn > > 1.13.3 r

Re: [DISCUSS] Creating an external connector repository

Hi, all I understand very well that the maintainers of the community want to move the connector to an external system. Indeed, the development and maintenance of the connector requires a lot of energy, and these do not involve the Flink core framework, which can reduce the maintenance pressure

Re: Can't access Debezium metadata fields in Kafka table

Hi, Harshvardhan The format debezium-avro-confluent doesn’t support read metadata yet[1], the supported formats including debezium-json, canal-json and maxwell-json, you can try the supported formats. Best, Leonard [1] https://issues.apache.org/jira/browse/FLINK-20454

Re: Streaming SQL support for redis streaming connector

Hi, Osada Just want to offer some material here.The flink-cdc-connectors project [1] maybe also help you, we supports the document db MongoDB[2] recently. Best, Leonard [1] https://github.com/ververica/flink-cdc-connectors [2] https://verve

[ANNOUNCE] Flink mailing lists archive service has migrated to Apache Archive service

Hi, all The mailing list archive service Nabble Archive was broken at the end of June, the Flink community has migrated the mailing lists archives[1] to Apache Archive service by commit[2], you can refer [3] to know more mailing lists archives of Flink. Apache Archive service is maintained by

Re: 退订

Hi, Please send email to dev-unsubscr...@flink.apache.org if you want to unsubscribe the mail from d...@flink.apache.org . Please send email to user-unsubscr...@flink.apache.org

Re: 【Announce】Zeppelin 0.10.0 is released, Flink on Zeppelin Improved

Thanks Jeff for the great work ! Best, Leonard > 在 2021年8月25日，22:48，Jeff Zhang 写道： > > Hi Flink users, > > We (Zeppelin community) are very excited to announce Zeppelin 0.10.0 is > officially released. In this version, we made several improvements on Flink > interpreter. Here's the main fe

Re: Flink CDC job getting failed due to G1 old gc

e) > > .tableSchema(FlinkSchemaUtil.toSchema(FlinkSchemaUtil.convert(icebergTable.schema( > .tableLoader(tableLoader) > .equalityFieldColumns(tableConfig.getEqualityColumns()) > .build(); > > Please let me know if you need some other information too > > > On

Re: Flink CDC job getting failed due to G1 old gc

Hi, Ayush Thanks for the detailed description. Before analyze the issue, I have two questions that which Flink and Flink CDC version are you using? Is Flink CDC used in SQL or DataStream ? That’s helpful if you can post you Flink CDC connector parameters. Best, Leonard > 在 2021年7月29日，18:57，

Re: How to convert local to iso 8601 time in flink sql?

Hi, Unfortunately Flink SQL doesn’t support TIMESTAMP WITH TIME ZONE type yet[1], maybe the you can try write an UDF to convert the timestamp '2021-06-29 09:00:00’ field to String(the string representation like '2021-06-29T09:00:00+08:00’). And could you share your scenario about using TIMESTA

Re: Flink TPC-DS 3TB BenchMark result is not good.

Hi, vtygoss Thanks for the detail report, a quick reply as I wrote the org.apache.flink.table.tpcds.TpcdsTestProgram in community, I guess you missed table statistics information. The table statistics information used in the TPC-DS e2e tests is constant for 1GB verification data set, I wrote

Re: unsubscribe

You should send an email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mails from user@flink.apache.org Best, Leonard > 在 2021年6月21日，15:33，steven chen 写道： > > unsubscribe > > >

Re: Flink sql case when problem

Hi, houying It looks like a bug when code generate the operator code, which Flink version are you using? Could you help create an JIRA ticket? Best, Leonard > 在 2021年6月17日，19:48，纳兰清风 写道： > > Hello, > > When I am using case when statement in flink sql, I got an error as > follow: > >

Re: confused about `TO_TIMESTAMP` document description

Hi，Tony > I found this code snippet [2] might be related to `TO_TIMESTAMP` udf, and > seems like it won't set use any timezone configuration, so maybe the document > might be wrong. Your analysis is right，the document is wrong, we should correct it. Would you like to create an jira ticket and

Re: How to unsubscribe?

Hi, Morgan Just send an email with any content to user-unsubscr...@flink.apache.org will unsubscribe the mail from Flink user mail list. And also send an email to with any content to dev-unsubscr...@flink.apache.org

Re: [VOTE] Release 1.13.1, release candidate #1

+1 (non-binding) - verified signatures and hashes - built from source code with scala 2.11 succeeded - started a cluster, WebUI was accessible, ran a window word count job, no suspicious log output - ran some SQL jobs in SQL Client, the queries result is expected - the web PR looks good Best, L

Re: Table name for table created fromDataStream

> 在 2021年5月8日，08:00，tbud 写道： > > Hi Leonard, > Yes that would be one solution. But why is it necessary to create a > temporaryView from already created table ? The name “Table” is quite misleading here, the table API object Table actually represents a relational query (e.g. Table table = m

Re: FlieSystem Connector's Success File Was Submitted Late

Hi, forideal I also encountered this problem and opened an issue[1], you can have a look. Best, Leonard [1] https://issues.apache.org/jira/browse/FLINK-22472 > 在 2021年5月7日，20:31，forideal 写道： > > I found the reason： > >Late data processing: The record will be written into its partition w

Re: Watermark time zone issue

Hi, forideal It’s not because the time zone issue, the watermark value is timestamp in UTC mills, you should convert it to UTC timestamp and then compare with your data. Best, Leonard > 在 2021年5月7日，18:28，forideal 写道： > > Hi My friends: > My watermark added 8 more hours to the timestamp d

Re: Table name for table created fromDataStream

Hi, tbud You can register the Table API object as a temporary view and then run query on it: tableEnv.createTemporaryView(“MyTable”， eventsTable); tableEnv.executeSql(“SELECT * FROM MyTable“).print(); Best, Leonard > 在 2021年5月7日，03:17，tbud 写道： > > Does anybody know how to set the name for th

Re: [ANNOUNCE] Apache Flink 1.13.0 released

Thanks Dawid & Guowei for the great work, thanks everyone involved. Best, Leonard > 在 2021年5月5日，17:12，Theo Diefenthal 写道： > > Thanks for managing the release. +1. I like the focus on improving operations > with this version. > > Von: "Matthias Pohl" > An: "Etienne Chauchot" > CC: "dev" ,

Re: Watermarks in Event Time Temporal Join

Thanks for your example, Maciej I can explain more about the design. > Let's have events. > S1, id1, v1, 1 > S2, id1, v2, 1 > > Nothing is happening as none of the streams have reached the watermark. > Now let's add > S2, id2, v2, 101 > This should trigger join for id1 because we have all the kno

Re: Watermarks in Event Time Temporal Join

Hello, Maciej > I agree the watermark should pass on versioned table side, because > this is the only way to know which version of record should be used. > But if we mimics behaviour of interval join then main stream watermark > could be skipped. IIRC, rowtime interval join requires the watermark

Re: [ANNOUNCE] Apache Flink 1.12.2 released

Well done! Thanks to Roman and Yuan, and everyone who contributed to the release. Best, Leonard > 在 2021年3月9日，11:39，Zhu Zhu 写道： > > Thanks Roman and Yuan for being the release managers! Thanks everyone who has > made this release possible! > > Cheers, > Zhu > > Piotr Nowojski mailto:pnowoj.

Re: [Flink-SQL] FLOORing OR CEILing a DATE or TIMESTAMP to WEEK uses Thursdays as week start

as suggested in one of the > Calcite discussions. > > The changes proposed by FLIP-126 definitely look good. I'll check its details > further. > > Best Regards, > > On Thu, 4 Mar 2021 at 04:18, Leonard Xu <mailto:xbjt...@gmail.com>> wrote: > Hi

Re: [DISCUSS] Deprecation and removal of the legacy SQL planner

+1 for the roadmap. Thanks Timo for driving this. Best, Leonard > 在 2021年3月4日，20:40，Timo Walther 写道： > > Last call for feedback on this topic. > > It seems everyone agrees to finally complete FLIP-32. Since FLIP-32 has been > accepted for a very long time, I think we don't need another votin

Re: Processing-time temporal join is not supported yet.

mars 2021 à 05:31, Leonard Xu <mailto:xbjt...@gmail.com>> a écrit : > Hi, Eric > >> what will be the best workaround to enrich stream of data from a kafka >> topics with statical data based on id? > Currently you can put your statical data in Hive/JDBC/HBase which su

Re: Processing-time temporal join is not supported yet.

the s3 files that can be used to enrich your stream data. Best, Leonard > > > Le sam. 27 févr. 2021 à 05:15, Leonard Xu <mailto:xbjt...@gmail.com>> a écrit : > Hi, Eric > > Firstly FileSystemTableSource doe not implement LookupTableSource which means > we canno

Re: [Flink-SQL] FLOORing OR CEILing a DATE or TIMESTAMP to WEEK uses Thursdays as week start

the patch, what we need is only to adapt it in Flink code, I’d like to make this as a part of FLIP-162 and fix it soon. Thanks Julian and Timo for the reminder. Best, Leonard > 在 2021年3月4日，12:14，Leonard Xu 写道： > > Hi, Jaffe > > Flink follows old version calcite’s behaviour wh

Re: [Flink-SQL] FLOORing OR CEILing a DATE or TIMESTAMP to WEEK uses Thursdays as week start

Hi, Jaffe Flink follows old version calcite’s behaviour which lead to the wrong behavior. snd Julian is right that calcite has corrected FLOOR and CEIL functions in CALCITE-3412, Flink has upgraded calcite to 1.26 version which contains the patch, what we need is only to adapt it in Flink code

Re: Processing-time temporal join is not supported yet.

Hi, Eric Firstly FileSystemTableSource doe not implement LookupTableSource which means we cannot directly lookup a Filesystem table. In FLINK-19830, we plan to support Processing-time temporal join any table/views by lookup the data in join operator state which scanned from the filesystem tabl

[DISCUSS] FLIP-162: Consistent Flink SQL time function behavior

Hello, everyone I want to start the discussion of FLIP-162: Consistent Flink SQL time function behavior[1]. We’ve some initial discussion of several problematic functions in dev mail list[2], and I think it's the right time to resolve them by a FLIP. Currently some time function behaviors

Re: [DISCUSS] Correct time-related function behavior in Flink SQL

f use the default Flink SQL,  the window time range of the >> statistics is incorrect, then the statistical results will naturally be >> incorrect. To zhisheng, sorry to hear that this problem influenced your production jobs, Could you share your SQL pattern? we can have more inp

Re: Pyflink Join with versioned view / table

Hi, Torben Happy to hear you address your problem, the first option can resolve the situation that partial partitions of the Kafka topic did not receive data, but if all partitions didn’t receive data, the watermark won’t be pushed forward, and the temporal join won’t be triggered. Otherwise,

Re: Flink sql interval join problem

Hi, >I only saw inner interval join on the official website, I don't see > outer interval join on the offical website, Is there an example of an OUTER > INTERVAL JOIN? >official website > link://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/sql/queries.html#join

Re: Pushing Down Filters

Hi, Shekhar > 1. Optimizer does not use both - ProjectableTableSource and > FilterableTableSource - in a single query even if the source implements both > interfaces. Each interface works correctly if implemented independently. I didn’t see your custom source implementation, but I think the tw

Re: Pushing Down Filters

Hi, Shekhar > 1. Optimizer does not use both - ProjectableTableSource and > FilterableTableSource - in a single query even if the source implements both > interfaces. Each interface works correctly if implemented independently. I didn’t your custom source implementation, but I think the two int

Re: Pyflink Join with versioned view / table

Hi, Torben > When implementing the join I get only updates when the right table changes The event-time temporal join versioned table is triggered watermark which calculated by both left and right table’s watermark, so you get only updated when the right table changes(which is the slower one in

Re: Publishing a table to Kafka

Hi, Rai > What are my options to still write to Kafka? I don't mind introducing > another boolean/etc field in the Kafka output records containing the > row kind or similar info. The recommended way is use `upset-kafka`[1] connector which you can write insert/update/retract message to a compac

Re: table rowtime timezome problem

Hi，Jiazhi > When DataStream is converted to table, eventTime is converted to > rowTime. Rowtime is 8 hours slow. How to solve this problem? The reason is that the only data type that used to define an event time in Table/SQL is TIMESTAMP(3), and TIMESTAMP type isn’t related to your loca

Re: Flink SQL continuous join checkpointing

Hi, Taras > But checkpoint data size for join task is permanently increasing despite the > watermarks on the tables and "Low watermark" mark in UI. > As far as I understand outdated records from both tables must be dropped from > checkpoint after 2 hours, but looks like it holds all job state s

Re: upsert-kafka to do temporal table joins

Hi， guoliubin Please ignore my previous answer, I mixed your question with an another one, I post the right temporal join sql syntax here. SELECT [column_list] FROM table1 [AS ] [LEFT] JOIN table2 FOR SYSTEM_TIME AS OF table1.{ proctime | rowtime } [AS ] ON table1.column-name1 = table2.column-na

Re: upsert-kafka to do temporal table joins

Hi， guoliubin Sorry for the late reply, I think the example in release note has a minor typo error which missed the ‘AS’ keyword. SELECT o.order_id, o.order_time, o.amount * r.currency_rate AS amount, r.currency FROM orders AS o, latest_rates FOR SYSTEM_TIME AS OF o.order_time

Re: [ANNOUNCE] Apache Flink 1.12.0 released

Thanks Dian and Robert for the great work as release manager ! And thanks everyone who makes the release possible ! Best, Leonard > 在 2020年12月10日，20:17，Robert Metzger 写道： > > The Apache Flink community is very happy to announce the release of Apache > Flink 1.12.0, which is the latest maj

Re: how to register TableAggregateFunction?

Hi, appleyuchi Sorry for the late reply, but could you describe you problem more or post your exception stack? The doc you posted has contained the section to define and register function. And I suggest you post your entire code in email directly that can reproduce the problem, thus the com

Re: Join a datastream with tables stored in Hive

Hi, Krzysztof > * I have a high pace stream of events coming in Kafka. > * I have some dimension tables stored in Hive. These tables are changed > daily. I can keep a snapshot for each day. For this use case, Flink supports temporal join the latest hive partition as temporal table now, you

Re: Join a datastream with tables stored in Hive

Hi, Maciej > > I didn't find a SQL solution to this problem. > Now Flink provides the SQL solution, you can see the doc[1], the Flink-1.12 document link that posted by Chesnay should have updated but not..., I’ll check the document of 1.12. Best, Leonard [1] https://ci.apache.org/projects/f

Re: FlinkSQL kafka->dedup->kafka

function does not implements correctly, I create an issue[1] to fix this. Best, Leonard [1] https://issues.apache.org/jira/browse/FLINK-20405 > On Fri, 27 Nov 2020 at 03:28, Leonard Xu <mailto:xbjt...@gmail.com>> wrote: > Hi, Laurent > >> Basically, I need to dedup

Re: FlinkSQL kafka->dedup->kafka

Hi, Laurent > Basically, I need to deduplicate, but only keeping in the deduplication state > the latest value of the changed column to compare with. While here it seems > to keep all previous values… You can use ` ORDER BY proctime() DESC` in the deduplication query, it will keep last row,

Re: Setting JDBC connector options using JdbcCatalog

Hi, Dylan > Is there a way to specify the sql hints using the Table API? Unfortunately the sql hints only supported in SQL, there’s no plan to support it in Table API AFAK, I’d suggest use SQL if possible, currently SQL has better feature support than Table API. Best, Leonard

Re: Question about Flink-SQL

Hi, Roc > Does Flink-SQL support fetching Mysql meta information automaticly in the > latest version, ? If not, could the you adding this feature ? You can obtain the latest meta information(table schema) by using Flink JdbcCalatag[1], only PostgresCatalog is implemented, user can implemented

  1   2   >