Re: [VOTE] FLIP-63: Rework table partition support

Just to clarify: FLIP wiki: https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support Discussion thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-63-Rework-table-partition-support-td32770.html Google Doc: https://docs.googl

[jira] [Created] (FLINK-14182) Make TimeUtils able to parse duration string with plural form labels

Zhu Zhu created FLINK-14182: --- Summary: Make TimeUtils able to parse duration string with plural form labels Key: FLINK-14182 URL: https://issues.apache.org/jira/browse/FLINK-14182 Project: Flink I

[jira] [Created] (FLINK-14183) Remove unnecessary scala Duration usages in flink-runtime

Zhu Zhu created FLINK-14183: --- Summary: Remove unnecessary scala Duration usages in flink-runtime Key: FLINK-14183 URL: https://issues.apache.org/jira/browse/FLINK-14183 Project: Flink Issue Type: S

Re: [COMMITTER] repo locked due to synchronization issues

Hi Bowen, I have also encountered this problem. I don't know how to fix this. But pushing to GitHub repo always works for me. Best, Jark On Tue, 24 Sep 2019 at 06:05, Bowen Li wrote: > Hi committers, > > Recently I've run a repo issue multiple times in different days. When I > tried to push a

Re: Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

What is the conclusion of this discussion? Are we gonna merge the old source/sink or do we believe that FLIP-27 will be completed in time so that we can merge the new source right away? Cheers, Till On Tue, Sep 24, 2019 at 3:43 AM 郭士榕 wrote: > > > unscribe > > > > > > > At 2019-09-24 08:53:25,

Re: [COMMITTER] repo locked due to synchronization issues

Pushing directly to Github also works for me without a problem. Cheers, Till On Tue, Sep 24, 2019 at 10:28 AM Jark Wu wrote: > Hi Bowen, > > I have also encountered this problem. I don't know how to fix this. > But pushing to GitHub repo always works for me. > > Best, > Jark > > On Tue, 24 Sep

Re: [COMMITTER] repo locked due to synchronization issues

Maybe it's a mix of pushing to the ASF repository and Github mirrors? I'm only pushing to the ASF repositories (although not that frequently anymore...). Cheers, Fabian Am Di., 24. Sept. 2019 um 10:50 Uhr schrieb Till Rohrmann < trohrm...@apache.org>: > Pushing directly to Github also works for

Re: [VOTE] FLIP-63: Rework table partition support

+1 from my side. Some implementation details could be revisited again during code reviewing. Best, Kurt On Tue, Sep 24, 2019 at 3:14 PM Jingsong Li wrote: > Just to clarify: > > FLIP wiki: > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support > > > D

Re: Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

Hi Till, My understanding is that we are going to hold back a little bit on the Pulsar source connector until FLIP-27 is ready. If we check in the old Pulsar source connector right away, we will have to maintain that old connector for quite a while (a year or more), which is something we don't wan

[jira] [Created] (FLINK-14184) Provide a stage listener API to be invoked per task manager

Stephen Connolly created FLINK-14184: Summary: Provide a stage listener API to be invoked per task manager Key: FLINK-14184 URL: https://issues.apache.org/jira/browse/FLINK-14184 Project: Flink

Re: [DISCUSS] Releasing Flink 1.9.1

+1 for 1.9.1 soon :) I would also like to include a fix to: FLINK-14145 - getLatestCheckpoint(true) returns wrong checkpoint It is already merged to master and just need to merge it to 1.9 if we all agree (https://github.com/apache/flink/pull/9756) Cheers, Gyula On Tue, Sep 24, 2019 at 8:23 AM

Re: [VOTE] FLIP-56: Dynamic Slot Allocation

Hi Xintong, Thanks for starting the vote. The proposed changes look good to me. +1 (binding) Best, Gary On Mon, Sep 23, 2019 at 4:18 PM Till Rohrmann wrote: > Thanks for updating the Flip. It looks good to me. > > +1 (binding) > > Cheers, > Till > > On Mon, Sep 23, 2019 at 4:12 PM Xintong Son

Re: [VOTE] FLIP-56: Dynamic Slot Allocation

If it's possible, I would suggest to add one sector in this doc to emphasize that current design has a prerequisite that each job should either has all its operators using unknown resource profile or all using specified amount of resource. This would make this document easier to understand. (I was

[jira] [Created] (FLINK-14185) AbstractServerTest failed on Travis

Yingjie Cao created FLINK-14185: --- Summary: AbstractServerTest failed on Travis Key: FLINK-14185 URL: https://issues.apache.org/jira/browse/FLINK-14185 Project: Flink Issue Type: Bug C

Re: Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

I agree that we can start from Pulsar sink and Pulsar Catalog. When we finish that, maybe FLIP-27 is ready then. Best, Jark On Tue, 24 Sep 2019 at 17:06, Becket Qin wrote: > Hi Till, > > My understanding is that we are going to hold back a little bit on the > Pulsar source connector until FLIP-

Re: [VOTE] FLIP-56: Dynamic Slot Allocation

Thanks for the votes, Gary and Kurt. @Kurt Sorry for the confusion. I've added a clarification in the section "Unknown Resource Requirement". And +1 (non-binding) from my side. Thank you~ Xintong Song On Tue, Sep 24, 2019 at 5:35 PM Kurt Young wrote: > If it's possible, I would suggest to

Have trouble on running flink

Hi Flink team, I am trying to submit flink job (version 1.8.2) with RocksDB backend to my own yarn cluster (hadoop version 2.6.0-cdh5.7.3), the job always failed after running for a few hours with the connection loss of some taskmanagers. Here

Re: [VOTE] FLIP-56: Dynamic Slot Allocation

Thanks all for the votes. So far, we have - 4 binding +1 votes (Till, Andrey, Gary and Kurt) - 1 un-binding +1 votes (Xintong) - No -1 votes There are more than 3 binding +1 votes and no -1 votes, and the voting time has past. According to the community bylaws, I'm glad to announce that

[jira] [Created] (FLINK-14186) Nightly e2e-misc-jdk11 fails complaining unrecognized VM option

Yu Li created FLINK-14186: - Summary: Nightly e2e-misc-jdk11 fails complaining unrecognized VM option Key: FLINK-14186 URL: https://issues.apache.org/jira/browse/FLINK-14186 Project: Flink Issue Type

[jira] [Created] (FLINK-14187) FLIP-56 Dynamic Slot Allocation

Xintong Song created FLINK-14187: Summary: FLIP-56 Dynamic Slot Allocation Key: FLINK-14187 URL: https://issues.apache.org/jira/browse/FLINK-14187 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-14189) Extend TaskExecutor to support dynamic slot allocation

Xintong Song created FLINK-14189: Summary: Extend TaskExecutor to support dynamic slot allocation Key: FLINK-14189 URL: https://issues.apache.org/jira/browse/FLINK-14189 Project: Flink Issue

[jira] [Created] (FLINK-14188) TaskExecutor derive and register with default slot resource profile

Xintong Song created FLINK-14188: Summary: TaskExecutor derive and register with default slot resource profile Key: FLINK-14188 URL: https://issues.apache.org/jira/browse/FLINK-14188 Project: Flink

[jira] [Created] (FLINK-14191) Extend SlotManager to support dynamic slot allocation on pending task executors

Xintong Song created FLINK-14191: Summary: Extend SlotManager to support dynamic slot allocation on pending task executors Key: FLINK-14191 URL: https://issues.apache.org/jira/browse/FLINK-14191 Proje

[jira] [Created] (FLINK-14190) Extend SlotManager to support dynamic slot allocation on registered TaskExecutors.

Xintong Song created FLINK-14190: Summary: Extend SlotManager to support dynamic slot allocation on registered TaskExecutors. Key: FLINK-14190 URL: https://issues.apache.org/jira/browse/FLINK-14190 Pr

[jira] [Created] (FLINK-14192) Enable the dynamic slot allocation feature.

Xintong Song created FLINK-14192: Summary: Enable the dynamic slot allocation feature. Key: FLINK-14192 URL: https://issues.apache.org/jira/browse/FLINK-14192 Project: Flink Issue Type: Sub-t

[jira] [Created] (FLINK-14193) Update RestAPI / Web UI

Xintong Song created FLINK-14193: Summary: Update RestAPI / Web UI Key: FLINK-14193 URL: https://issues.apache.org/jira/browse/FLINK-14193 Project: Flink Issue Type: Sub-task Repo

[jira] [Created] (FLINK-14195) Nightly connectors-jdk11 fails because of missing jaxb classes

Yu Li created FLINK-14195: - Summary: Nightly connectors-jdk11 fails because of missing jaxb classes Key: FLINK-14195 URL: https://issues.apache.org/jira/browse/FLINK-14195 Project: Flink Issue Type:

[jira] [Created] (FLINK-14194) Clean-up of legacy mode.

Xintong Song created FLINK-14194: Summary: Clean-up of legacy mode. Key: FLINK-14194 URL: https://issues.apache.org/jira/browse/FLINK-14194 Project: Flink Issue Type: Sub-task Rep

[jira] [Created] (FLINK-14196) Nightly e2e-sticky-hadoop2.8 fails due to NPE

Yu Li created FLINK-14196: - Summary: Nightly e2e-sticky-hadoop2.8 fails due to NPE Key: FLINK-14196 URL: https://issues.apache.org/jira/browse/FLINK-14196 Project: Flink Issue Type: Bug Com

Re: Have trouble on running flink

Hi Russell, I don't think `BackendBuildingException` is root cause. In your case, this exception appears when task is under cancelling. Have you ever checked the log of yarn node manager? There should be an exit code of container. Even more the container is probably killed by yarn node manager.

Re: [VOTE] FLIP-63: Rework table partition support

+1, Overall looks good. Best, Terry Wang > 在 2019年9月24日，下午5:02，Kurt Young 写道： > > +1 from my side. Some implementation details could be revisited > again during code reviewing. > > Best, > Kurt > > > On Tue, Sep 24, 2019 at 3:14 PM Jingsong Li wrote: > >> Just to clarify: >> >> FLIP wik

Re: [COMMITTER] repo locked due to synchronization issues

I met the same problem. Pushing to the GitHub repo directly works fine and it seems will resync the two repos. Best, Hequn On Tue, Sep 24, 2019 at 4:59 PM Fabian Hueske wrote: > Maybe it's a mix of pushing to the ASF repository and Github mirrors? > I'm only pushing to the ASF repositories (alt

[jira] [Created] (FLINK-14197) Increasing trend for state size of keyed stream using ProcessWindowFunction with ProcessingTimeSessionWindows

Oliver Kostera created FLINK-14197: -- Summary: Increasing trend for state size of keyed stream using ProcessWindowFunction with ProcessingTimeSessionWindows Key: FLINK-14197 URL: https://issues.apache.org/jira/bro

[jira] [Created] (FLINK-14198) Add type options to all flink python API doc

Wei Zhong created FLINK-14198: - Summary: Add type options to all flink python API doc Key: FLINK-14198 URL: https://issues.apache.org/jira/browse/FLINK-14198 Project: Flink Issue Type: Improvemen

Confluence permission for FLIP creation

Hi all, I'm proposing to support view in Flink SQL and I have a initial design doc which i want to convert to a FLIP. It would be great if anyone who can grant me the write permission to Confluence. My Confluence ID is: docete I will really appreciate if any of you can help me with this. *Best

[jira] [Created] (FLINK-14199) Only use dedicated/named classes for mailbox letters.

Arvid Heise created FLINK-14199: --- Summary: Only use dedicated/named classes for mailbox letters. Key: FLINK-14199 URL: https://issues.apache.org/jira/browse/FLINK-14199 Project: Flink Issue Typ

Re: [VOTE] FLIP-57: Rework FunctionCatalog

+1 Best, Terry Wang > 在 2019年9月24日，上午10:42，Kurt Young 写道： > > +1 > > Best, > Kurt > > > On Tue, Sep 24, 2019 at 2:30 AM Bowen Li wrote: > >> Hi all, >> >> I'd like to start a voting thread for FLIP-57 [1], which we've reached >> consensus in [2]. >> >> This voting will be open for mini

[DISCUSS] FLIP-73: Introducing Executors for job submission

Hi all, In the context of the discussion about introducing the Job Client API [1], there was a side-discussion about refactoring the way users submit jobs in Flink. There were many different interesting ideas on the topic and 3 design documents that were trying to tackle both the issue about code

[jira] [Created] (FLINK-14200) Temporal Table Function Joins do not work on Tables (only TableSources) on the query side

Benoît Paris created FLINK-14200: Summary: Temporal Table Function Joins do not work on Tables (only TableSources) on the query side Key: FLINK-14200 URL: https://issues.apache.org/jira/browse/FLINK-14200

Re: [DISCUSS] FLIP-73: Introducing Executors for job submission

Thanks for the proposal. These changes will make it significantly easier to programmatically use Flink in downstream frameworks. Should the new Executor execute method be defined as asynchronous? It could return a job handle to interact with the job and the legacy environments can still block to r

Re: [DISCUSS] FLIP-73: Introducing Executors for job submission

Hi Thomas, >Should the new Executor execute method be defined as asynchronous? It could >return a job handle to interact with the job and the legacy environments >can still block to retain their semantics. During our discussion there will be a method executeAsync(...): CompletableFuture where J

Re: [DISCUSS] Releasing Flink 1.9.1

FLINK-14010 has been merged. Cheers, Till On Tue, Sep 24, 2019 at 11:14 AM Gyula Fóra wrote: > +1 for 1.9.1 soon :) > > I would also like to include a fix to: > FLINK-14145 - getLatestCheckpoint(true) returns wrong checkpoint > > It is already merged to master and just need to merge it to 1.9 i

Re: [VOTE] FLIP-63: Rework table partition support

+1. Thanks, Jingsong! Bowen On Tue, Sep 24, 2019 at 4:38 AM Terry Wang wrote: > +1, Overall looks good. > > Best, > Terry Wang > > > > > 在 2019年9月24日，下午5:02，Kurt Young 写道： > > > > +1 from my side. Some implementation details could be revisited > > again during code reviewing. > > > > Best, > >

Re: [COMMITTER] repo locked due to synchronization issues

Thanks everyone for sharing your practices! The problem seems to be gone as usual and I am able to push to ASF repo. Verified by ASF INFRA [1], it's indeed caused by the mixed push problem that Fabian said. Quote from INFRA, "This issue can sometimes occur when people commit conflicting branches a

Re: [SURVEY] How many people are using customized RestartStrategy(s)

Zhu Zhu, Sorry, I was using different terminology. yes, Flink meter is what I was talking about regarding "fullRestarts" for threshold based alerting. On Mon, Sep 23, 2019 at 7:46 PM Zhu Zhu wrote: > Steven, > > In my mind, Flink counter only stores its accumulated count and reports > that valu

Re: [DISCUSS] FLIP-73: Introducing Executors for job submission

Since Exceutor is a new interface, why is backward compatibility a concern? I could see that become an issue later when replacing Executor execute with executeAsync. Or are both targeted for 1.10? On Tue, Sep 24, 2019 at 10:24 AM Zili Chen wrote: > Hi Thomas, > > >Should the new Executor execu

Re: [VOTE] FLIP-57: Rework FunctionCatalog

+1. LGTM On Tue, Sep 24, 2019 at 6:09 AM Terry Wang wrote: > +1 > > Best, > Terry Wang > > > > > 在 2019年9月24日，上午10:42，Kurt Young 写道： > > > > +1 > > > > Best, > > Kurt > > > > > > On Tue, Sep 24, 2019 at 2:30 AM Bowen Li wrote: > > > >> Hi all, > >> > >> I'd like to start a voting thread for FL

Re: [VOTE] FLIP-63: Rework table partition support

+1. I understand that the FLIP probably covers more we can do in one release, but let's proritize and start with the basics first. On Tue, Sep 24, 2019 at 10:42 AM Bowen Li wrote: > +1. Thanks, Jingsong! > > Bowen > > On Tue, Sep 24, 2019 at 4:38 AM Terry Wang wrote: > > > +1, Overall looks goo

Re: [VOTE] FLIP-68: Extend Core Table System with Modular Plugins

+1, LGTM On Mon, Sep 23, 2019 at 10:26 AM Bowen Li wrote: > Hi all, > > I'd like to start a vote for FLIP-68 [1], since there's no more concern in > the discussion thread [2] > > The vote will be open for minimum 3 days till 5:30pm UTC, Sep 26. > > Thanks, > Bowen > > [1] > > https://cwiki.apach

Re: [VOTE] FLIP-57: Rework FunctionCatalog

Hi, I really like the flip and think it clarifies important aspects of the system. I have two, I hope small suggestions, which will not take much time to agree on. 1. Could we follow the MySQL approach in regards to the existence of cat/db for temporary functions? That means not to check it, so e

Re: [DISCUSS] FLIP 69 - Flink SQL DDL Enhancement

BTW, will there be a "CREATE/DROP CATALOG" DDL? Though it's not SQL standard, I can see it'll be useful and handy for our end users in many cases. On Mon, Sep 23, 2019 at 12:28 PM Bowen Li wrote: > Hi Terry, > > Thanks for driving the effort! I left some comments in the doc. > > AFAIU, the bigg

Re: [DISCUSS] FLIP-73: Introducing Executors for job submission

>Since Exceutor is a new interface, why is backward compatibility a concern? For backward compatibility, it is on (Stream)ExecutionEnvironment#execute. You're right that we don't stick to blocking to return a JobExecutionResult in Executor aspect but implementing env.execute with a unique Executo

Re: Confluence permission for FLIP creation

I have given you edit permission, could you please re-login and check it. :) Danny Chan 于2019年9月24日周二 上午11:31写道： > Hi all, > > As communicated in an email thread, I'm proposing to support computed > column in Flink SQL . I have a draft design doc that I'd like to convert it > to a FLIP. Thus, it

Re: Confluence permission for FLIP creation

I have given you edit permission, could you please re-login and check it. :) Zhenghua Gao 于2019年9月24日周二 下午8:13写道： > Hi all, > > I'm proposing to support view in Flink SQL and I have a initial design doc > which i want to convert to a FLIP. It would be great if anyone who can > grant me the writ

Re: [DISCUSS] FLIP 69 - Flink SQL DDL Enhancement

Thanks Bowen for your insightful comments, I’ll think twice and do corresponding improvement. After finished, I’ll update in this mailing thread again. Best, Terry Wang > 在 2019年9月25日，上午8:28，Bowen Li 写道： > > BTW, will there be a "CREATE/DROP CATALOG" DDL? > > Though it's not SQL standard, I

Re: [VOTE] FLIP-63: Rework table partition support

+1. Thanks Jingsong for the efforts. On Wed, Sep 25, 2019 at 4:50 AM Xuefu Z wrote: > +1. I understand that the FLIP probably covers more we can do in one > release, but let's proritize and start with the basics first. > > On Tue, Sep 24, 2019 at 10:42 AM Bowen Li wrote: > > > +1. Thanks, Jings

Confluence permission for FLIP creation

Hi all I'm proposing to improve the Flink user experience in the Web UI and I have created a draft proposal doc. Could anyone grant the permission to convert the doc to a FLIP in Confluence, my id is vthink...@gmail.com Best

Re: [SURVEY] How many people are using customized RestartStrategy(s)

Hi Steven, As a conclusion, since we will have a meter metric[1] for restarts, customized restart strategy is not needed in your case. Is that right? [1] https://issues.apache.org/jira/browse/FLINK-14164 Thanks, Zhu Zhu Steven Wu 于2019年9月25日周三 上午2:30写道： > Zhu Zhu, > > Sorry, I was using diffe

Re: Confluence permission for FLIP creation

Many Thanks for your help! *Best Regards,* *Zhenghua Gao* On Wed, Sep 25, 2019 at 9:33 AM jincheng sun wrote: > I have given you edit permission, could you please re-login and check it. > :) > > Zhenghua Gao 于2019年9月24日周二 下午8:13写道： > > > Hi all, > > > > I'm proposing to support view in Flink

Re: [VOTE] FLIP-57: Rework FunctionCatalog

Hi Dawid, Re 1): I agree making it easy for users to run experiments is important. However, I'm not sure allowing users to register temp functions in nonexistent catalog/db is the optimal way. It seems a bit hacky, and breaks the current contract between Flink and users that catalog/db must be val

[jira] [Created] (FLINK-14201) SQL supports redis sink connector

Zijie Lu created FLINK-14201: Summary: SQL supports redis sink connector Key: FLINK-14201 URL: https://issues.apache.org/jira/browse/FLINK-14201 Project: Flink Issue Type: Wish Componen

Re: [VOTE] FLIP-57: Rework FunctionCatalog

Sorry, I missed some parts of the solution. The complete alternative is the following, basically having separate APIs in FunctionLookup for ambiguous and precise function lookup since planner is able to tell which API to call with parsed queries, and have a unified result: ``` class FunctionLookup

FLINK-14108

Hi. I have create the jira https://issues.apache.org/jira/browse/FLINK-14108 for improve confluent schema registry integration. I have the code but would like to have some feedback if the community would like the contribution. Lasse Nedergaard