Re: Issues about removed topics with KafkaSource

2023-11-01 Thread Emily Li via user
Hey Martijn Thanks for the clarification. Now it makes sense. I saw this feature FLIP-246 is still a WIP and there's no release date yet, and it actually contains quite some changes in it. We noticed there's a WIP PR for this change, just wondering if there's any plan in releasing this feature?

Fwd: flink-kubernetes-operator cannot handle SPECCHANGE for 100+ FlinkDeployments concurrently

2023-11-01 Thread Tony Chen
Hi Flink Community, I am currently running flink-kubernetes-operator 1.6-patched ( https://github.com/apache/flink-kubernetes-operator/commit/3f0dc2ee5534084bc162e6deaded36e93bb5e384), and I have 3 flink-kubernetes-operator pods running. Recently, I deployed around 110 new FlinkDeployments, and I

Re: Monitoring File Processing Progress in Flink Jobs

2023-11-01 Thread Feng Jin
Hi arjun Flink will save the currently processed file and its corresponding offset in Flink state [1]. You may need to use the Flink state process API[1] to access it. However, I don't think this is a good approach. I suggest adding relevant metrics to the FileSystem connector to report the curre

Re: Issues about removed topics with KafkaSource

2023-11-01 Thread Martijn Visser
Hi, That's by design: you can't dynamically add and remove topics from an existing Flink job that is being restarted from a snapshot. The feature you're looking for is being planned as part of FLIP-246 [1] Best regards, Martijn [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageI

Re: [DISCUSS][FLINK-33240] Document deprecated options as well

2023-11-01 Thread Zhanghao Chen
Hi Samrat and Ruan, Thanks for the suggestion. I'm actually in favor of adding the deprecated options in the same section as the non-deprecated ones. This would make user search for descriptions of the replacement options more easily. It would be a different story for options deprecated because

Re: [DISCUSS][FLINK-33240] Document deprecated options as well

2023-11-01 Thread Zhanghao Chen
Hi Alexander, I haven't done a complete analysis yet. But through simple code search, roughly 35 options would be added with this change. Also note that some old options defined in a ConfigConstant class won's be added here as flink-doc won't discover these constant-based options. Best, Zhangh

Re: Re: Query on Flink SQL create DDL primary key for nested field

2023-11-01 Thread elakiya udhayanan
Hi Xuyang, Thanks again for giving me some insights on how to use the Datastream API for my use case, I will explore it and experiment with it. I wanted to use the value inside the row datatype as a primary key because, I might get multiple records for the same id and when I try to make a join wi

Re:Re: Query on Flink SQL create DDL primary key for nested field

2023-11-01 Thread Xuyang
Hi, Elakiya, I think you can get what you need here[1] with many examples briging DataStream api and Table API. There may be some redundancy, and I'm not sure this is a best way to resolve the question. First, use the StreamTableEnvironment to execute your original ddl without pk. Second, us

Re: [DISCUSS][FLINK-33240] Document deprecated options as well

2023-11-01 Thread Hang Ruan
Thanks for the proposal. +1 from my side and +1 for putting them to a separate section. Best, Hang Samrat Deb 于2023年11月1日周三 15:32写道: > Thanks for the proposal , > +1 for adding deprecated identifier > > [Thought] Can we have seperate section / page for deprecated configs ? Wdut > ? > > > Bests

Re: Query on Flink SQL create DDL primary key for nested field

2023-11-01 Thread elakiya udhayanan
Hi Xuyang, Thank you for your response. Since, I have no access to create a ticket in the ASF jira I have requested for the access and once I get the access will raise a ticket for the same. Also, you have asked me to use Datastream API to extract the id and then use the TableAPI feature, since I