Re: [ANNOUNCE] Apache Flink 1.18.1 released

Hi folks, The bug has been fixed and PR at docker-library/official-images has been merged. The official images are available now. Best regards, Jing On Mon, Jan 22, 2024 at 11:39 AM Jing Ge wrote: > Hi folks, > > I am still working on the official images because of the issu

Re: [ANNOUNCE] Apache Flink 1.18.1 released

Hi folks, I am still working on the official images because of the issue https://issues.apache.org/jira/browse/FLINK-34165. Images under apache/flink are available. Best regards, Jing On Sun, Jan 21, 2024 at 11:06 PM Jing Ge wrote: > Thanks Leonard for the feedback! Also thanks @Jark

Re: [ANNOUNCE] Apache Flink 1.18.1 released

t; Thanks all who involved this release! > > Best, > Leonard > > > 2024年1月20日 上午12:01，Jing Ge 写道： > > > > The Apache Flink community is very happy to announce the release of > Apache > > Flink 1.18.1, which is the first bugfix release for the Apache Flink 1

[ANNOUNCE] Apache Flink 1.18.1 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.18.1, which is the first bugfix release for the Apache Flink 1.18 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming a

Re: [DISCUSS] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

any further thoughts. > > Best regards, > Junrui > > Jing Ge 于2023年11月20日周一 05:46写道： > >> Hi Junrui, >> >> Thanks for bringing this to our attention. First of all, it makes sense >> to deprecate RuntimeContext#getExecutionConfig. >> >> Afaic, t

Re: [DISCUSS] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

Hi Junrui, Thanks for bringing this to our attention. First of all, it makes sense to deprecate RuntimeContext#getExecutionConfig. Afaic, this is an issue of how we design API with clean concepts/aspects. There are two issues mentioned in the FLIP: 1. short of user-facing abstraction - we just e

[ANNOUNCE] Apache Flink 1.18.0 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.18.0, which is the first release for the Apache Flink 1.18 series. Apache Flink® is an open-source unified stream and batch data processing framework for distributed, high-performing, always-available, and accurate

Re: Flink 1.17.2 planned?

also be patched by us). > > > > Kind regards, > > Christian > > > > *Von: *Jing Ge via user > *Datum: *Dienstag, 22. August 2023 um 11:40 > *An: *liu ron > *Cc: *user@flink.apache.org > *Betreff: *Re: Flink 1.17.2 planned? > > This email has reached Mapp via

Fwd: [Discussion] Slack Channel

Hi devs, Thanks Giannis for your suggestion. It seems that the last email wasn't sent to the dev ML. It is also an interesting topic for devs and user-zh. Best regards, Jing -- Forwarded message - From: Giannis Polyzos Date: Tue, Aug 22, 2023 at 11:11 AM Subject: [Discussion] Sl

Re: Flink 1.17.2 planned?

Hi Christian, Thanks for reaching out. Liked Ron pointed out that the community is focusing on the 1.18 release. If you are facing urgent issues, would you like to volunteer as the release manager of 1.17.2 and drive the release? Theoretically, everyone could be the release manager of a bugs fix r

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

Congratulations! Best regards, Jing On Mon, Jul 3, 2023 at 3:21 PM yuxia wrote: > Congratulations! > > Best regards, > Yuxia > > -- > *发件人: *"Pushpa Ramakrishnan" > *收件人: *"Xintong Song" > *抄送: *"dev" , "User" > *发送时间: *星期一, 2023年 7 月 03日 下午 8:36:30 > *主题: *Re: [

Call for Presentations: Flink Forward Seattle 2023

ward to receiving your submission and welcoming you as a speaker at the Flink Forward Conference. Thank you for your time and consideration. Best regards, -- Jing Ge | Head of Engineering j...@ververica.com <https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forw

Re: apache-flink java question

Hi Chris, not yet and we are working on it[1]. best regards, Jing [1] https://issues.apache.org/jira/browse/FLINK-15736?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=17697544#comment-17697544 On Thu, Jun 1, 2023 at 11:40 PM Joseph, Chris S wrote: > H

Re: [SUMMARY] Flink 1.18 Release Sync 05/30/2023

Thanks Qingsheng for driving it! @Devs As you might already be aware of, there are many externalizations and new releases of Flink connectors. Once a connector has been externalized successfully, i.e. the related module has been removed in the Flink repo, we will not set a priority higher than maj

Re: [ANNOUNCE] Apache Flink 1.16.2 released

gt; Best regards, > > Weijie > > > [1] > https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release > > [2] https://github.com/pypa/twine/issues/997 > > > Jing Ge 于2023年5月27日周六 00:15写道： > > > Hi Weijie, > > > > Thanks again for your effort

Re: [ANNOUNCE] Apache Flink 1.16.2 released

Hi Weijie, Thanks again for your effort. I was wondering if there were any obstacles you had to overcome while releasing 1.16.2 and 1.17.1 that could lead us to any improvement wrt the release process and management? Best regards, Jing On Fri, May 26, 2023 at 4:41 PM Martijn Visser wrote: > Th

Re: [ANNOUNCE] Apache Flink 1.17.1 released

Hi Weijie, That is earlier than I expected! Thank you so much for your effort! Best regards, Jing On Fri, May 26, 2023 at 4:44 PM Martijn Visser wrote: > Same here as with Flink 1.16.2, thank you Weijie and those who helped with > testing! > > On Fri, May 26, 2023 at 1:08 PM weijie guo > wrot

Re: Maven plugin to detect issues early on

cc user ML to get more attention, since the plugin will be used by Flink application developers. Best regards, Jing On Mon, May 22, 2023 at 3:32 PM Jing Ge wrote: > Hi Emre, > > Thanks for clarifying it. Afaiac, it is a quite interesting proposal, > especially for Flink job develo

Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)

? Best regards, Jing On Thu, Apr 27, 2023 at 8:18 AM Tamir Sagi wrote: > More details about the JDK bug here > https://bugs.openjdk.org/browse/JDK-8277529 > > Related Jira ticket > https://issues.apache.org/jira/browse/FLINK-24998 > > ------ > *From:* J

Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)

Thanks Chesnay for working on this. Would you like to share more info about the JDK bug? Best regards, Jing On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler wrote: > As it turns out Kryo isn't a blocker; we ran into a JDK bug. > > On 31/03/2023 08:57, Chesnay Schepler wrote: > > > https://gith

[SUMMARY] Flink 1.18 Release Sync 4/4/2023

Dear devs and users, Today was the kickoff meeting for Flink 1.18 release cycle. I'd like to share the info synced in the meeting. Meeting schedule: Zoom will be used with a 40 mins limit for each meeting. That should be fine for now. We will check it again if we have an issue with the time limi

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

Congrats! Best regards, Jing On Mon, Mar 27, 2023 at 2:32 PM Leonard Xu wrote: > Congratulations! > > > Best, > Leonard > > On Mar 27, 2023, at 5:23 PM, Yu Li wrote: > > Dear Flinkers, > > > > As you may have noticed, we are pleased to announce that Flink Table Store > has joined the Apache I

Re: [ANNOUNCE] Apache Flink 1.17.0 released

Excellent work! Congratulations! Appreciate the hard work and contributions of everyone in the Apache Flink community who helped make this release possible. Looking forward to those new features. Cheers! Best regards, Jing On Thu, Mar 23, 2023 at 10:24 AM Leonard Xu wrote: > The Apache Flink co

Re: KafkaSink handling message size produce errors

, 2023 at 8:17 AM Shammon FY wrote: > >> Hi jing, >> >> It sounds good to me, we can add an option for it >> >> Best, >> Shammon >> >> >> On Fri, Feb 17, 2023 at 3:13 PM Jing Ge wrote: >> >>> Hi, >>> >>> It makes se

Re: KafkaSink handling message size produce errors

Hi, It makes sense to offer this feature of catching and ignoring exp with config on/off, when we put ourselves in users' shoes. WDYT? I will create a ticket if most of you consider it as a good feature to help users. Best regards, Jing On Fri, Feb 17, 2023 at 6:01 AM Shammon FY wrote: > Hi Ha

Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

Hi Yanfei, Thanks for your effort. Looking forward to checking it. Best regards, Jing On Mon, Jan 30, 2023 at 1:42 PM Yanfei Lei wrote: > It is very happy to announce the release of FRocksDB 6.20.3-ververica-2.0. > > Compiled files for Linux x86, Linux arm, Linux ppc64le, MacOS x86, > MacOS ar

Re: Reading Parquet file with array of structs cause error

> >> >> Do you know, if this problem will be addressed in FLINK-28867 >> <https://issues.apache.org/jira/browse/FLINK-28867> or it deserve a >> separate Jira? >> >> >> >> >> >> *From: *Jing Ge >> *Date: *Tues

Re: Reading Parquet file with array of structs cause error

Hi Michael, Currently, ParquetColumnarRowInputFormat does not support schemas with nested columns. If your parquet file stores Avro records. You might want to try e.g. Avro Generic record[1]. [1] https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/formats/parquet

Re: [blog article] Howto migrate a real-life batch pipeline from the DataSet API to the DataStream API

Hi Etienne, Nice blog! Thanks for sharing! Best regards, Jing On Wed, Nov 9, 2022 at 5:49 PM Etienne Chauchot wrote: > Hi Yun Gao, > > FYI I just updated the article after your review: > https://echauchot.blogspot.com/2022/11/flink-howto-migrate-real-life-batch.html > > Best > > Etienne > Le

Re: [ANNOUNCE] Apache Flink 1.16.0 released

Congrats! On Fri, Oct 28, 2022 at 1:22 PM 任庆盛 wrote: > Congratulations and a big thanks to Chesnay, Martijn, Godfrey and Xingbo > for the awesome work for 1.16! > > Best regards, > Qingsheng Ren > > > On Oct 28, 2022, at 14:46, Xingbo Huang wrote: > > > > The Apache Flink community is very hap

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

o make IO. This needs time to design, discuss, adjust > and vote, but considering this is blocking 1.16, maybe it’s better to > rescue the compatibility for now, and leave the huge reconstruction to > future versions (maybe 2.0). > > Best, > Qingsheng > > On Wed, Oct 12, 2

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

Hi Qingsheng, Just want to make sure we are on the same page. Are you suggesting switching the naming between "numXXXSend" and "numXXXOut" or reverting all the changes we did with FLINK-26126 and FLINK-26492? For the naming switch, please pay attention that the behaviour has been changed since we

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

Hi Martijn, Thanks for bringing this up. It is generally a great idea, so +1. Since both scala extension projects mentioned in the FLIP are still very young and I don't think they will attract more scala developers as Flink could just because they are external projects. It will be a big issue for

Re: DataStream and DataStreamSource

Hi, Welcome to the Flink community! A DataStreamSource is a DataStream. It is normally used as the starting point of a DataStream. All related methods in StreamExecutionEnvironment that create a DataStream return actually a DataStreamSource, because it is where a DataStream starts. Commonly, yo

Re: Flink upgrade path

Hi, I would recommend you to check the release notes of 1.14[1] and 1.15[2]. If your Flink jobs are using Flink features that have big improvements in these two releases, it would be better to upgrade step by step without skipping 1.14.x. In general, depending on how complicated your jobs are, it

Re: [ANNOUNCE] Apache Flink 1.15.2 released

Thanks Danny for your effort! Best regards, Jing On Wed, Aug 24, 2022 at 11:43 PM Danny Cranmer wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.15.2, which is the second bugfix release for the Apache Flink 1.15 > series. > > Apache Flink® is an open

Re: Using RocksDBStateBackend and SSD to store states, application runs slower..

Hi, using FLASH_SSD_OPTIMIZED already sets the number of threads to 4. This optimization can improve the source throughput and reduce the delayed wrate rate. If this optimization didn't fix the back pressure, could you share more information about your job? Could you check the metric of the back

Re: Unit test have Error "could not find implicit value for evidence parameter"

Hi, you don't have to do that. Next time you can try "Invalidate Caches..." under the File menu in Intellij Idea. Best regards, Jing On Wed, Jul 13, 2022 at 7:21 PM Min Tu via user wrote: > Thanks a lot !! I have removed the .idea folder and the unit test works. > > On Mon, Jul 11, 2022 at 2:4

Re: 退订/unsubscribe

: bastien dine Cc: Martijn Visser , Jing Ge , user 请问这个邮件咋退订？ Replied Message >From bastien dine Date 06/15/2022 17:50 To Martijn Visser Cc Jing Ge , user Subject Re: New KafkaSource API : Change in default behavior regarding starting offset Hello Martijn, Thanks for the link

Re: Metrics for FileSource

cs of these > components. We would ideally like to measure these: > 1. Number of rows emitted by the source per unit time > 2. Time taken by the enumerator to discover the splits > 3. Total splits discovered > > > Regards, > Meghajit > > > On Fri, Jun 10, 2022 at 10:0

Re: Flink config driven tool ?

Hi, Just like Shengkai mentioned. I would strongly suggest trying SQL for ETL dag. If you find anything that SQL does not work for you, please share your requirements with us. We might check if it makes sense to build new features in Flink to support them. Best regards, Jing On Wed, Jun 15, 202

Re: New KafkaSource API : Change in default behavior regarding starting offset

Hi Bastien, Thanks for asking. I didn't find any call of setStartFromGroupOffsets() within Flink in the master branch. Could you please point out the code that committed offset is used as default? W.r.t. the new KafkaSource, if OffsetsInitializer.committedOffsets() is used, an exception will be t

Re: Apache Flink - Reading data from Scylla DB

Hi, Please be aware that SourceFunction will be deprecated soon[1]. It is recommended to build a new source connector based on the new Source API design by FLIP-27[2]. You might take the Kafka connector as the reference implementation. Best regards, Jing [1] https://lists.apache.org/thread/d6cwq

Re: Metrics for FileSource

Hi meghajit, I think it makes sense to extend the current metrics. Could you list all metrics you need? Thanks! Best regards, Jing On Fri, Jun 10, 2022 at 5:06 PM Lijie Wang wrote: > Hi Meghajit, > > As far as I know, currently, the FileSource does not have the metrics you > need. You can imp

Re: [External] Re: Source vs SourceFunction and testing

Hi Carlos, You might want to join the discussion about FLIP-238[1] to share your thoughts with us. Thanks! Best regards, Jing [1] https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt On Thu, Jun 9, 2022 at 2:13 PM Sanabria, Carlos < carlos.sanab...@accenture.com> wrote: > Thanks f

Re: SourceFunction

Hi Alexey, There is a thread[1] discussing this issue right now. It would be great if you could share some thoughts about your experience. Thanks! Best regards, Jing [1]https://lists.apache.org/thread/d6cwqw9b3105wcpdkwq7rr4s7x4ywqr9 On Wed, Jun 8, 2022 at 4:42 PM Alexey Trenikhun wrote: > He

Re: Add me to slack

Hi Xiao, Just done, please check. Thanks! Best regards, Jing On Mon, Jun 6, 2022 at 3:59 AM Xiao Ma wrote: > Hi Jing, > > Could you please add me to the slack channel also? > > Thank you. > > > Best, > Mark Ma > > On Sun, Jun 5, 2022 at 9:57 PM Jing Ge wro

Re: Flink source Code Explanation

Hi Sri, Flink is very well documented. You can find it under e.g. https://nightlies.apache.org/flink/flink-docs-master/ Best regards, Jing On Mon, Jun 6, 2022 at 3:39 AM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Hi Flink Community, > > I want to go through flink source cod

Re: Add me to slack

Hi Raghunadh, Just did, please check your email. Thanks! Best regards, Jing On Mon, Jun 6, 2022 at 3:51 AM Raghunadh Nittala wrote: > Team, Kindly add me to the slack channel. > > Best Regards. >

Re: Need help to join Apache Flink community on.Slack

mail.com> wrote: > >> Hi Jing, >> >> Please add me kali.tumm...@gmail.com. >> >> Thanks >> Sri >> >> On Sat, Jun 4, 2022 at 4:47 PM Jing Ge wrote: >> >>> Hi Santhosh, >>> >>> just invited you. Please check your ema

Re: Need help to join Apache Flink community on.Slack

>> >> Please add me kali.tumm...@gmail.com. >> >> Thanks >> Sri >> >> On Sat, Jun 4, 2022 at 4:47 PM Jing Ge wrote: >> >>> Hi Santhosh, >>> >>> just invited you. Please check your email. Looking forward to knowing >>&g

Re: Need help to join Apache Flink community on.Slack

Hi Sri, I have invited you, please check. Thanks! Best regards, Jing On Sun, Jun 5, 2022 at 6:02 PM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Hi Jing, > > Please add me kali.tumm...@gmail.com. > > Thanks > Sri > > On Sat, Jun 4, 2022 at 4

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

Amazing! Thanks Yang for driving this! Thanks all for your effort! Best regards, Jing On Sun, Jun 5, 2022 at 11:30 AM tison wrote: > Congrats! Thank you all for making this release happen. > > Best, > tison. > > > rui fan <1996fan...@gmail.com> 于2022年6月5日周日 17:19写道： > >> Thanks Yang for driving

Re: slack invite link

Hi, Invites have been sent. Please check your emails. Thanks! Regards, JIng On Sun, Jun 5, 2022 at 12:38 PM Jay Ghiya wrote: > Request community to share invite link for me at ghiya6...@gmail.com and > jay.gh...@ge.com >

Re: Need help to join Apache Flink community on.Slack

Hi Santhosh, just invited you. Please check your email. Looking forward to knowing your story! Thanks! To anyone else who wants to join, please send an email to user@flink.apache.org, you might have a better chance to get the invite. Thanks. Regards, Jing On Sat, Jun 4, 2022 at 10:37 PM santhos

Re: [ANNOUNCE] Welcome to join the Apache Flink community on Slack

PM Yuan Mei wrote: > Thanks, Xintong and Jark the great effort driving this, and everyone for > making this possible. > > I've also Twittered this announcement on our Apache Flink Twitter account. > > Best > > Yuan > > > > On Fri, Jun 3, 2022 at 12:54 AM J

Re: Flink/Scala contract positions ?

Hi, Currently, the Flink Scala API is not in a good shape. Would you like to start from there? Best regards, Jing On Fri, Jun 3, 2022 at 4:29 PM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Hi Folks, > > Is anyone hiring for Flink or Scala Akka contract corp to corp positions

Re: flink-ml algorithms

Hi, It seems like an evaluation with a small dataset. In this case, would you like to share your data sample and code? In addition, have you tried KMeans with the same dataset and got inconsistent results too? Best regards, Jing On Fri, Jun 3, 2022 at 4:29 AM Natia Chachkhiani < natia.chachkhia.

Re: [ANNOUNCE] Welcome to join the Apache Flink community on Slack

Thanks everyone for your effort! Best regards, Jing On Thu, Jun 2, 2022 at 4:17 PM Martijn Visser wrote: > Thanks everyone for joining! It's good to see so many have joined in such > a short time already. I've just refreshed the link which you can always > find on the project website [1] > > Be

Re: Is there an HA solution to run flink job with multiple source

Hi Bariša, Could you share the reason why your data processing pipeline should keep running when one kafka source is down? It seems like any one among the multiple kafka sources is optional for the data processing logic, because any kafka source could be the one that is down. Best regards, Jing

Re: Can we resume a job from a savepoint from Java api?

Hi, yuxia has already pointed out the correct direction. The exact line for using the savepoint path to resume the job from a savepoint is at line 1326 [1] [1] https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-tests/src/test/java/org/apache/flink/test/checkpointi

Re: Status of File Sink Common (flink-file-sink-common)

Hi, Afaik, there are still a lot of unit tests depending on it. I don't think we can drop it before dropping all of these unit tests. Best regards, Jing On Tue, May 31, 2022 at 8:10 AM Yun Gao wrote: > Hi Jun, > > I think the release notes should only include the issues that cause changes > vi

Re: GlobalCommitter in Flink's two-phase commit

Hi, 1. What are the general usage scenarios of GlobalCommitter? - GlobalCommitter is used for creating and committing an aggregated committable. It is part of a 2-phase-commit protocol. One use case is the compaction of small files. 2. Why should GlobalCommitter be removed in the new version of t

Re: Missing metrics in Flink v 1.15.0 rc-0

Hi, Flink 1.15 has developed a new feature to support different sink pre- and post-topologies[1]. New metrics e.g. NumRecordsSend has been developed to measure records sent to the external system. Metrics like "Bytes Sent" and "Records Sent" measure records sent to the next task. So, in your cas

Re: Flink SQL 1.12 How to implement query Hbase table on secondary index

HI WuKong, Afaiu, technically, you are not using Hbase secondary index(coprocessor). What you are trying to do is to store the synced dim table in elasticsearch and query from there to get the rowkeys and then use the rowkeys to get dim table rows from Hbase. In this way, a (full) table scan in Hb

Re: [DISCUSS] Flink's supported APIs and Hive query syntax

Hi, Thanks Martijn for driving this discussion. Your concerns are very rational. We should do our best to keep the Flink development on the right track. I would suggest discussing it in a vision/goal oriented way. Since Flink has a clear vision of unified batch and stream processing, supporting b

Re: How to proper hashCode() for keys.

Hi John, your getKey() implementation shows that it is not deterministic, since calling it with the same click instance multiple times will return different keys. For example a call at 12:01:59.950 and a call at 12:02:00.050 with the same click instance will return two different keys: 2022-04-07T

Re: [DISCUSS] Deprecate/remove Twitter connector

Thanks @Martijn for driving this! +1 for deprecating and removing it. All the concerns mentioned previously are valid. It is good to know that the upcoming connector template/archetype will help the user for the kickoff. Beyond that, speaking of using a real connector as a sample, since Flink is he

Re: RowType for complex types in Parquet File

Hi Meghajit, like the exception described, parquet schema with nested columns is not supported currently. It is on our todo list with high priority. Best regards Jing On Fri, Jan 7, 2022 at 6:12 AM Meghajit Mazumdar < meghajit.mazum...@gojek.com> wrote: > Hello, > > Flink documentation mentions

Re: Converting parquet MessageType to flink RowType

500, > false, > true); > > Regards, > Meghajit > > On Thu, Jan 6, 2022 at 3:43 PM Jing Ge wrote: > >> Hi Meghajit, >> >> thanks for asking. If you took a look at the source cod

Re: Converting parquet MessageType to flink RowType

Hi Meghajit, thanks for asking. If you took a look at the source code https://github.com/apache/flink/blob/9bbadb9b105b233b7565af120020ebd8dce69a4f/flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/ParquetVectorizedInputFormat.java#L174, you should see Parquet MessageType

Re: [DISCUSS] Drop Gelly

Hi, thanks Martijn for bringing it up for discussion. I think we could make the discussion a little bit clearer by splitting it into two questions: 1. should Flink drop Gelly? 2. should Flink drop the graph computing? The answer of the first question could be yes, since there have been no change