[jira] [Created] (FLINK-29914) KafkaTableITCase.testKafkaSourceSink fails

Matthias Pohl created FLINK-29914: - Summary: KafkaTableITCase.testKafkaSourceSink fails Key: FLINK-29914 URL: https://issues.apache.org/jira/browse/FLINK-29914 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-29915) -netty-tcnative-static not built on CI

Chesnay Schepler created FLINK-29915: Summary: -netty-tcnative-static not built on CI Key: FLINK-29915 URL: https://issues.apache.org/jira/browse/FLINK-29915 Project: Flink Issue Type: Bu

Re: [DISCUSS] FLIP-271: Autoscaling

@Guyla, Thanks for the explanation and the follow up actions. That sounds good to me. Thanks, JunRui Lee Yanfei Lei 于2022年11月7日周一 12:20写道： > Hi Max, > > Thanks for the proposal. This proposal makes Flink better adapted to > cloud-native applications! > > After reading the FLIP, I'm curious abo

[jira] [Created] (FLINK-29916) Levels in Table Store may mistakenly ignore level 0 files when two files have the same sequence number

Caizhi Weng created FLINK-29916: --- Summary: Levels in Table Store may mistakenly ignore level 0 files when two files have the same sequence number Key: FLINK-29916 URL: https://issues.apache.org/jira/browse/FLINK-299

[jira] [Created] (FLINK-29917) In standalone mode can't view flink dashboard page in default

Hang HOU created FLINK-29917: Summary: In standalone mode can't view flink dashboard page in default Key: FLINK-29917 URL: https://issues.apache.org/jira/browse/FLINK-29917 Project: Flink Issue

Re: [DISCUSS] Flink release retro

Thanks for all the insights on how it was done in the past and the pro's and con's of the different approaches. ...also being reminded on how we did it for 1.14 and the positive feedback we got from it was helpful. I like the idea of an async survey. I also think that utilizing the mailing list fo

[jira] [Created] (FLINK-29918) Generalized delegation token support

Gabor Somogyi created FLINK-29918: - Summary: Generalized delegation token support Key: FLINK-29918 URL: https://issues.apache.org/jira/browse/FLINK-29918 Project: Flink Issue Type: New Featur

Re: [DISCUSS] Repeatable cleanup of checkpoint data

This is a nice FLIP. I particular like how much background it provides on the issue; something that other FLIPs could certainly benefit from... I went over the FLIP and had a chat with Matthias about it. Somewhat unrelated to the FLIP we found a flaw in the current cleanup mechanism of failed

Re: [VOTE] Release Apache Flink Elasticsearch connector 3.0.0, rc2

I remembered now; the test failure you ran into is because this particular fails if you don't have logging enabled. See FLINK-292789. On 03/11/2022 21:25, Danny Cranmer wrote: Hello Chesnay, -1 (binding) I do not approve the release due to the source release [1] containing binaries, the full

Re: [VOTE] Release Apache Flink Elasticsearch connector 3.0.0, rc2

+1 (binding) - correct Flink version in source release - no -SNAPSHOT version in source release - source release compiles and tests are passing   - (mvn verify -Dlog4j.configurationFile=file://$(pwd)/tools/ci/log4j.properties) - clean source release - tag exists and points to correct commit -

[DISCUSS] FLIP-272: Generalized delegation token support

Hi team, Delegation token framework is going to be finished soon (added in FLIP-211 ). Previously there were concerns that the current implementation is bound to Hadoop an

Re: [VOTE] Release Apache Flink Elasticsearch connector 3.0.0, rc2

+1 (binding) - Source release does not contain binaries - Source release builds and tests pass (when using CI log4j config) - Verified signature of source archive - Source code tag exists - pom versions look correct in the source release - README is ok - The staging repository [1] contains expecte

Re: [DISCUSS] FLIP-272: Generalized delegation token support

Hi Gabor, Thanks for driving this effort! A few thoughts on the topic: - Could you please add a few examples of the delegation token providers we expected to be added in the near future? Ideally these providers could be configured independently from each other. However the configuration defaults

[blog article] Howto migrate a real-life batch pipeline from the DataSet API to the DataStream API

Hi everyone, In case some of you are interested, I just posted a blog article about migrating a real-life batch pipeline from the DataSet API to the DataStream API: https://echauchot.blogspot.com/2022/11/flink-howto-migrate-real-life-batch.html Best Etienne

Questions about Flink Table Store

I’m Alexander from Confluent. I am new to Flink and its community. I would like to contribute to the Flink Table Store, but am missing certain details. Can someone please clarify the points mentioned below to me? - Given that there is always a single writer to a stream, in what situations ca

Re: [DISCUSS] FLIP-272: Generalized delegation token support

Hi Matyas, Thanks for your comments, answered inline. G On Mon, Nov 7, 2022 at 2:58 PM Őrhidi Mátyás wrote: > Hi Gabor, > > Thanks for driving this effort! A few thoughts on the topic: > - Could you please add a few examples of the delegation token providers we > expected to be added in the n

[jira] [Created] (FLINK-29919) Support operator leader election

Gyula Fora created FLINK-29919: -- Summary: Support operator leader election Key: FLINK-29919 URL: https://issues.apache.org/jira/browse/FLINK-29919 Project: Flink Issue Type: New Feature

Re: [DISCUSS] FLIP-271: Autoscaling

Thanks for the explanation Gyula. Please see my reply inline. BTW, has the proposed solution been deployed and evaluated with any production workload? If yes, I am wondering if you could share the experience, e.g. what is the likelihood of having regression and improvement respectively after enabl

Re: [DISCUSS] FLIP-271: Autoscaling

@Dong: Looking at the busyTime metrics in the TaskOMetricGroup it seems that busy time is actually defined as "not idle or (soft) backpressured" . So I think it would give us the correct reading based on what you said about the Kafka sink. In any case we have to test this and if something is not

Re: [DISCUSS] FLIP-271: Autoscaling

Thanks for all the interest here and for the great remarks! Gyula already did a great job addressing the questions here. Let me try to add additional context: @Biao Geng: >1. For source parallelisms, if the user configure a much larger value than >normal, there should be very little pending rec

Re: ASF Slack

There is a way to work around the invite issue. For example, the Beam project has a direct invite link which sends you to the #beam channel: https://app.slack.com/client/T4S1WH2J3/C9H0YNP3P I'm not 100% sure whether this link actually works. I've take it from: https://beam.apache.org/community/join

Re: [DISCUSS] FLIP-272: Generalized delegation token support

Hi Matyas, In the meantime I was thinking about the per provider re-obtain feature and here are my thoughts related that: * I think it's a good feature in general but as mentioned I would add it in a separate FLIP * In case of Hadoop providers it just wouldn't work (HBase doesn't have end timestam

Re: [DISCUSS] FLIP-272: Generalized delegation token support

Oh gosh, copied wrong config keys so fixed my last mail with green. On Mon, Nov 7, 2022 at 6:07 PM Gabor Somogyi wrote: > Hi Matyas, > > In the meantime I was thinking about the per provider re-obtain feature > and here are my thoughts related that: > * I think it's a good feature in general but

Re: ASF Slack

+1 to leveraging the larger ASF Community/Resources Slack Channel rather than an independant one ... ASSUMING ANYONE CAN JOIN [ so that needs to be verified ]. On Mon, Nov 7, 2022 at 9:05 AM Maximilian Michels wrote: > There is a way to work around the invite issue. For example, the Beam > proje

[jira] [Created] (FLINK-29920) Minor reformat Kafka connector documentation

Mingliang Liu created FLINK-29920: - Summary: Minor reformat Kafka connector documentation Key: FLINK-29920 URL: https://issues.apache.org/jira/browse/FLINK-29920 Project: Flink Issue Type: Bu

[jira] [Created] (FLINK-29921) Can't deserialize Avro type if schema contains a field with name `schema`.

Bill G created FLINK-29921: -- Summary: Can't deserialize Avro type if schema contains a field with name `schema`. Key: FLINK-29921 URL: https://issues.apache.org/jira/browse/FLINK-29921 Project: Flink

Re: [blog article] Howto migrate a real-life batch pipeline from the DataSet API to the DataStream API

Wow, cool! Thanks for your work. It'll be definitely helpful for the users that want to migrate their batch job from DataSet API to DataStream API. Best regards, Yuxia - 原始邮件 - 发件人: "Etienne Chauchot" 收件人: "dev" , "User" 发送时间: 星期一, 2022年 11 月 07日 下午 10:29:54 主题: [blog article] Howto m

Re: [VOTE] Release Apache Flink Elasticsearch connector 3.0.0, rc2

+1 (non-binding) - checked signatures - checked tag - checked LICENSE/NOTICE files - no binaries in source - checked versions On Mon, Nov 7, 2022 at 2:32 PM Danny Cranmer wrote: > +1 (binding) > > - Source release does not contain binaries > - Source release builds and tests pass (when using

Re: [blog article] Howto migrate a real-life batch pipeline from the DataSet API to the DataStream API

Thanks for your post, It looks very good to me, also maybe for developers, Best, Liudalong yuxia 于2022年11月8日周二 09:11写道： > Wow, cool! Thanks for your work. > It'll be definitely helpful for the users that want to migrate their batch > job from DataSet API to DataStream API. > > Best regards, >

Re: [DISCUSS] FLIP-269: Properly Handling the Processing Timers on Job Termination

Hi Yun, Thanks for the proposal. It appears that the issues mentioned in the motivation section are all related to using Windows on the DataStream API, where the user's code typically does not have anything to do with Timer. I am wondering if the following alternative solution might be more user-

Re: Re: [DISCUSS] FLIP-269: Properly Handling the Processing Timers on Job Termination

Hi Maximilian, Thanks for the discussion! It seems there are still other kinds of scenarios that could not be flushed, like scenarios like "emit record X if record Y hasn't arrived within 30 seconds after record Z" or "fails the job if the external system does not response in 30 seconds", these

Re: [DISCUSS] FLIP-271: Autoscaling

Thanks for the fruitful discussion and I am really excited to see that the auto-scaling really happens for Flink Kubernetes operator. It will be a very important step to make the long-running Flink job more smoothly. I just have some immature ideas and want to share them here. # Resource Reserv

[jira] [Created] (FLINK-29922) Table store hive catalog support create external table

konwu created FLINK-29922: - Summary: Table store hive catalog support create external table Key: FLINK-29922 URL: https://issues.apache.org/jira/browse/FLINK-29922 Project: Flink Issue Type: Improvem

[jira] [Created] (FLINK-29923) Hybrid Shuffle may face deadlock when running a task need to execute big size data

AlexHu created FLINK-29923: -- Summary: Hybrid Shuffle may face deadlock when running a task need to execute big size data Key: FLINK-29923 URL: https://issues.apache.org/jira/browse/FLINK-29923 Project: Flink

[jira] [Created] (FLINK-29924) Update official document

Hang HOU created FLINK-29924: Summary: Update official document Key: FLINK-29924 URL: https://issues.apache.org/jira/browse/FLINK-29924 Project: Flink Issue Type: Improvement Components

FeatHub - A feature store that uses Apache Flink for real-time feature ETL

Hi everyone, In case some of you might be interested, we (developers at Alibaba Flink team) recently developed and open-sourced FeatHub , a feature store that uses Apache Flink for feature engineering. The github page https://github.com/alibaba/feathub provides

Re: [blog article] Howto migrate a real-life batch pipeline from the DataSet API to the DataStream API

Hi Etienne, Very thanks for the article! Flink is currently indeed keeping increasing the ability of unified batch / stream processing with the same api, and its a great pleasure that more and more users are trying this functionality. But I also have some questions regarding some details. First IM

[jira] [Created] (FLINK-29925) table ui of configure value is strange

jiadong.lu created FLINK-29925: -- Summary: table ui of configure value is strange Key: FLINK-29925 URL: https://issues.apache.org/jira/browse/FLINK-29925 Project: Flink Issue Type: Improvement