[jira] [Created] (FLINK-27862) FLIP-235: Hybrid Shuffle Mode

2022-05-31 Thread Guo Weijie (Jira)
Guo Weijie created FLINK-27862: -- Summary: FLIP-235: Hybrid Shuffle Mode Key: FLINK-27862 URL: https://issues.apache.org/jira/browse/FLINK-27862 Project: Flink Issue Type: New Feature C

[jira] [Created] (FLINK-27861) Introduce UserResourceManager to manage user defined resource

2022-05-31 Thread dalongliu (Jira)
dalongliu created FLINK-27861: - Summary: Introduce UserResourceManager to manage user defined resource Key: FLINK-27861 URL: https://issues.apache.org/jira/browse/FLINK-27861 Project: Flink Issu

RE: [DISCUSS] FLIP-233: Introduce HTTP Connector

2022-05-31 Thread Maciej Bryński
Hi Jeremy, We as GetinData already started some work related with HTTP connector. As a first step we created HTTP Source [1] described in blog post here [2]. During next 3 months we are planning to create HTTP Sink based on Flink 1.15 new Async Sink API. How can we help with making this sink “offi

Re: [DISCUSS] FLIP-234: Support Retryable Lookup Join To Solve Delayed Updates Issue In External Systems

2022-05-31 Thread Lincoln Lee
Hi Jingsong, Thanks for your feedback! Yes, the existing HBase connector use an option 'lookup.async' to control its lookup source implementations that exposed to the planner, however it's a private option for the HBase connector only, so it will not affect the common API. And as discussed in th

Re: [DISCUSS] FLIP-234: Support Retryable Lookup Join To Solve Delayed Updates Issue In External Systems

2022-05-31 Thread Jingsong Li
Hi Lincoln, The unified lookup hint is what I want. And I like 'async'='true|false' option. But there is a compatibility issue, as I remember if async is currently controlled by connector, and this may also require some API changes? We need to have a clear story for the connector combined with

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.0.0, release candidate #3

2022-05-31 Thread Gyula Fóra
I agree to create a new RC based on this find. And 48h voting time seems reasonable as all functional testing will carry over to this new candidate. Gyula On Wed, Jun 1, 2022 at 7:56 AM Yang Wang wrote: > Thanks all for your testing and patience. > > And sorry for I have to cancel this VOTE sin

[jira] [Created] (FLINK-27860) List the CSS/docs dependencies in the NOTICE

2022-05-31 Thread Yang Wang (Jira)
Yang Wang created FLINK-27860: - Summary: List the CSS/docs dependencies in the NOTICE Key: FLINK-27860 URL: https://issues.apache.org/jira/browse/FLINK-27860 Project: Flink Issue Type: Bug

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.0.0, release candidate #3

2022-05-31 Thread Yang Wang
Thanks all for your testing and patience. And sorry for I have to cancel this VOTE since @Márton Balassi found a license issue. We do not list the CSS/docs dependencies in the NOTICE file of source release[1]. I will create another release candidate today including this fix. Given that there's n

Re: [DISCUSS] FLIP-231: Introduce SupportStatisticReport to support reporting statistics from source connectors

2022-05-31 Thread godfrey he
Hi, Jing. Thanks for the suggestion, I have updated the doc and will continue to optimize the code in subsequent PR. Best, Godfrey Jing Ge 于2022年6月1日周三 04:41写道: > > Hi Godfrey, > > Thanks for clarifying it. I personally prefer the new change you suggested. > > Would you please help to understan

Re: [VOTE] FLIP-235: Hybrid Shuffle Mode

2022-05-31 Thread weijie guo
Hi all, The voting time for FLIP-235 has passed. I'm closing the vote now. There were 5 votes, 3 of which are binding: - Aitozi - Jing Ge - Xintong Song (binding) - Yangze Guo (binding) - Yingjie Cao (binding) There were no disapproving votes. Thus, FLIP-235 has been accepted. Thanks everyone

[jira] [Created] (FLINK-27859) Hive dialect should support create table that begins with digits

2022-05-31 Thread luoyuxia (Jira)
luoyuxia created FLINK-27859: Summary: Hive dialect should support create table that begins with digits Key: FLINK-27859 URL: https://issues.apache.org/jira/browse/FLINK-27859 Project: Flink Is

[jira] [Created] (FLINK-27858) Hive source support filter push down for parquet format

2022-05-31 Thread luoyuxia (Jira)
luoyuxia created FLINK-27858: Summary: Hive source support filter push down for parquet format Key: FLINK-27858 URL: https://issues.apache.org/jira/browse/FLINK-27858 Project: Flink Issue Type: S

[jira] [Created] (FLINK-27857) Hive source support filter push down for orc format

2022-05-31 Thread luoyuxia (Jira)
luoyuxia created FLINK-27857: Summary: Hive source support filter push down for orc format Key: FLINK-27857 URL: https://issues.apache.org/jira/browse/FLINK-27857 Project: Flink Issue Type: Sub-t

Re: About Native Deployment's Autoscaling implementation

2022-05-31 Thread Talat Uyarer
Hi Yang and Gyula, Yang, Could you give a little bit more information ? What prevents us from changing task managers' count ? I am aware of ActiveResourceManager of Flink. But Flink only calls resources when it initializes a cluster. If we set - jobmanager.scheduler: adaptive - cluster.dec

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.0.0, release candidate #3

2022-05-31 Thread Jim Busche
Hi Yang, Thank you for RC3 +1 (not-binding) * Podman builds look great with your .dockerignore file and “COPY . .” Thank you for the fix. * Twistlock security scans look clean of both your “ghcr.io/apache/flink-kubernetes-operator:52b50c2 image” as well as a locally built image from

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

2022-05-31 Thread Jing Ge
Hi Jark, Thanks for clarifying it. It would be fine. as long as we could provide the no-cache solution. I was just wondering if the client side cache could really help when HBase is used, since the data to look up should be huge. Depending how much data will be cached on the client side, the data

Re: [DISCUSS] FLIP-231: Introduce SupportStatisticReport to support reporting statistics from source connectors

2022-05-31 Thread Jing Ge
Hi Godfrey, Thanks for clarifying it. I personally prefer the new change you suggested. Would you please help to understand one more thing? The else if (filterPushDownSpec != null) branch is the only branch that doesn't have to check if the newTableStat has been calculated previously. The reason

[jira] [Created] (FLINK-27856) Adding pod template without spec crashes job manager

2022-05-31 Thread Jeesmon Jacob (Jira)
Jeesmon Jacob created FLINK-27856: - Summary: Adding pod template without spec crashes job manager Key: FLINK-27856 URL: https://issues.apache.org/jira/browse/FLINK-27856 Project: Flink Issue

[jira] [Created] (FLINK-27855) Job Manager fails to recover with S3 storage and HA enabled

2022-05-31 Thread Sushant (Jira)
Sushant created FLINK-27855: --- Summary: Job Manager fails to recover with S3 storage and HA enabled Key: FLINK-27855 URL: https://issues.apache.org/jira/browse/FLINK-27855 Project: Flink Issue Type

[jira] [Created] (FLINK-27854) Refactor FlinkImageBuilder and FlinkContainerBuilder

2022-05-31 Thread Alexander Fedulov (Jira)
Alexander Fedulov created FLINK-27854: - Summary: Refactor FlinkImageBuilder and FlinkContainerBuilder Key: FLINK-27854 URL: https://issues.apache.org/jira/browse/FLINK-27854 Project: Flink

[jira] [Created] (FLINK-27853) FLIP-229: Introduces Join Hint for Flink SQL Batch Job

2022-05-31 Thread xuyang (Jira)
xuyang created FLINK-27853: -- Summary: FLIP-229: Introduces Join Hint for Flink SQL Batch Job Key: FLINK-27853 URL: https://issues.apache.org/jira/browse/FLINK-27853 Project: Flink Issue Type: New Fe

Re: [DISCUSS] FLIP-231: Introduce SupportStatisticReport to support reporting statistics from source connectors

2022-05-31 Thread godfrey he
Hi Jark and Jing, +1 to use "report" instead of "collect". > // only filter push down (*the description means > partitionPushDownSpec == null but misses the case of > partitionPushDownSpec != null*) `if (partitionPushDownSpec != null && filterPushDownSpec == null)` this branch is only consider

[jira] [Created] (FLINK-27852) Contribute k8s operator to OperatorHub

2022-05-31 Thread Jira
Márton Balassi created FLINK-27852: -- Summary: Contribute k8s operator to OperatorHub Key: FLINK-27852 URL: https://issues.apache.org/jira/browse/FLINK-27852 Project: Flink Issue Type: Improv

Re: [VOTE] FLIP-235: Hybrid Shuffle Mode

2022-05-31 Thread Yingjie Cao
+1 Best, Yingjie Jing Ge 于2022年5月30日周一 05:02写道: > +1 (non-binding) > > Best Regards, > Jing > > On Sun, May 29, 2022 at 5:16 AM Aitozi wrote: > > > +1 (non-binding) > > > > Best, > > Aitozi > > > > Yangze Guo 于2022年5月27日周五 11:17写道: > > > > > +1 > > > > > > Best, > > > Yangze Guo > > > > > > O

Re: [DISCUSS] Bi-Weekly Flink Community Sync Meeting

2022-05-31 Thread Konstantin Knauf
Hi everyone, can you be more specific what you mean by "current topics in the Flink Community"? Shouldn't asynchronous communication be the default, and if that doesn't work, we consider a synchronous channel? Cheers, Konstantin Am Di., 31. Mai 2022 um 13:54 Uhr schrieb Jing Ge : > +1 > Sounds

Re: [DISCUSS] Bi-Weekly Flink Community Sync Meeting

2022-05-31 Thread Chesnay Schepler
I'm not necessarily against collecting ideas/topics around the community side of things and discussing them in some way, but Im very much not a friend of immediately starting out such an effort with synchronous meetings. IMO we should only setup such meetings after we have identified a clear nee

Re: [DISCUSS] Bi-Weekly Flink Community Sync Meeting

2022-05-31 Thread Jing Ge
+1 Sounds good! Thanks Robert! On Tue, May 31, 2022 at 1:46 PM Márton Balassi wrote: > Hi Robert, > > Thanks for the suggestion +1 from me. You already listed the topic of the > timezone on the wiki that I wanted to bring up. > > On Tue, May 31, 2022 at 9:38 AM Robert Metzger > wrote: > > > Hi

Re: [DISCUSS] Bi-Weekly Flink Community Sync Meeting

2022-05-31 Thread Márton Balassi
Hi Robert, Thanks for the suggestion +1 from me. You already listed the topic of the timezone on the wiki that I wanted to bring up. On Tue, May 31, 2022 at 9:38 AM Robert Metzger wrote: > Hi everyone, > > We currently have a bi-weekly release sync meeting on Google Meet every > Tuesday at 9am

Re: [DISCUSS] Initializing Apache Flink Slack

2022-05-31 Thread Robert Metzger
I renamed the channel. On Tue, May 31, 2022 at 10:54 AM Xintong Song wrote: > Ok, then let's try it out. > > Best, > > Xintong > > > > On Tue, May 31, 2022 at 4:24 PM Robert Metzger > wrote: > > > +1 to merging #contribution-helps with #dev. > > > > I'm pretty sure we'll have to revisit this on

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.0.0, release candidate #3

2022-05-31 Thread Yang Wang
+1 (binding) Successfully verified the following: - Verify that the checksums and GPG files - Verify that the source distributions do not contain any binaries - Build binary and image from release source - Verify the NOTICE and licenses in the docker image - Verify the helm chart values with corre

Re: [DISCUSS] Initializing Apache Flink Slack

2022-05-31 Thread Xintong Song
Ok, then let's try it out. Best, Xintong On Tue, May 31, 2022 at 4:24 PM Robert Metzger wrote: > +1 to merging #contribution-helps with #dev. > > I'm pretty sure we'll have to revisit this once we have a bit > more experience with running the slack community. > > On Tue, May 31, 2022 at 10:0

Re: [DISCUSS] Initializing Apache Flink Slack

2022-05-31 Thread Robert Metzger
+1 to merging #contribution-helps with #dev. I'm pretty sure we'll have to revisit this once we have a bit more experience with running the slack community. On Tue, May 31, 2022 at 10:07 AM Jark Wu wrote: > I'm fine with the #dev channel. > I remembered in the previous discussion that most peop

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.0.0, release candidate #3

2022-05-31 Thread Nicholas Jiang
Hi Yang! +1 (not-binding) Verified the following successfully: - Build from source, build container from source - Run the examples for application, session and session job deployments successfully and without any errors. Regards, Nicholas Jiang On 2022/05/31 06:26:02 Yang Wang wrote: > Hi eve

[jira] [Created] (FLINK-27851) Join can't access the pk from source table

2022-05-31 Thread xuyang (Jira)
xuyang created FLINK-27851: -- Summary: Join can't access the pk from source table Key: FLINK-27851 URL: https://issues.apache.org/jira/browse/FLINK-27851 Project: Flink Issue Type: Bug Comp

Re: [DISCUSS] Initializing Apache Flink Slack

2022-05-31 Thread Jark Wu
I'm fine with the #dev channel. I remembered in the previous discussion that most people are positive about the dev channel as long as the discussions are properly reflected back to the mailing lists. If we create the dev channel, maybe we can merge #contribution-helps into it as well? Just like,

Re: [DISCUSS] Initializing Apache Flink Slack

2022-05-31 Thread Gyula Fóra
I agree with Robert. I think discussing implementation ideas etc on the dev channel briefly before posting discussions on the ML can make the design discussions much more productive as the initial iterations can be often slow and cumbersome via email alone. I understand the general sentiment agai

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.0.0, release candidate #3

2022-05-31 Thread Gyula Fóra
Hi Yang! +1 (binding) Successfully verified the following: - Signatures, Hashes - No binaries in source release - Helm Repo works, Helm install works, docker image matches release commit tag - Build from source, build container from source - Submit example application, session and session job dep

Re: [DISCUSS] Initializing Apache Flink Slack

2022-05-31 Thread Robert Metzger
Thanks for your input. I share the concern that we'll potentially have too important discussions in Slack. However, I would still like to try it out. There are valid use-cases, such as requests for VOTEs on releases, or briefly floating an idea before opening a thread on dev@. If we find out that

[DISCUSS] Bi-Weekly Flink Community Sync Meeting

2022-05-31 Thread Robert Metzger
Hi everyone, We currently have a bi-weekly release sync meeting on Google Meet every Tuesday at 9am CEST / 3pm China Standard Time / 7am UTC. I would like to propose extending the purpose of the meeting to a general "Flink Community Sync" meeting, to discuss current topics in the Flink community.

[jira] [Created] (FLINK-27850) Hive dialect supports "add file xxx"

2022-05-31 Thread luoyuxia (Jira)
luoyuxia created FLINK-27850: Summary: Hive dialect supports "add file xxx" Key: FLINK-27850 URL: https://issues.apache.org/jira/browse/FLINK-27850 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Planning Flink 1.16 with the community

2022-05-31 Thread Robert Metzger
> > Is the feature freeze deadline on July 25 fixed or will it be adjusted > accordingly? Ideally we try not to push the deadline too much into the future. So I prefer to consider it fixed. On Mon, May 30, 2022 at 7:08 PM Jing Ge wrote: > Thanks Robert for the reminder. Thanks Martijn for shar