[jira] [Created] (FLINK-14703) Port the Kafka SQL related tests.
Zheng Hu created FLINK-14703: Summary: Port the Kafka SQL related tests. Key: FLINK-14703 URL: https://issues.apache.org/jira/browse/FLINK-14703 Project: Flink Issue Type: Sub-task Reporter: Zheng Hu Port the three BASH SQL Kafka test cases : 1. test_sql_client_kafka.sh 2. test_sql_client_kafka010.sh 3. test_sql_client_kafka011.sh The rough patch of Kafka SQL e2e tests is here: https://github.com/openinx/flink/commit/24b7a6e34d1e504bc7a411781b43c996644c0910 Once the framework pull request (https://github.com/apache/flink/pull/10106) get merged, will create the PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14704) Remove uesless SynchronousSavepointLatch
Yun Tang created FLINK-14704:
Summary: Remove uesless SynchronousSavepointLatch
Key: FLINK-14704
URL: https://issues.apache.org/jira/browse/FLINK-14704
Project: Flink
Issue Type: Improvement
Reporter: Yun Tang
Fix For: 1.11.0
After FLINK-12482, we have already let checkpoint trigger and run via the mail
box on task side, which means previous {{SynchronousSavepointLatch}} is no
longer useful.
We should remove {{SynchronousSavepointLatch}} and its test
{{SynchronousSavepointSyncLatchTest}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Flink Read thousands of files with batch
Hi Dominik, Would you check whether the JM GC status? One possible cause is that the large number of file metas inHadoopInputFormat is burdening the JM memory. `akka.ask.timeout` is the default RPC timeout, while some RPCs may override this timeout for their own purpose. e.g. the RPCs from web usually use `web.timeout`. Providing the detailed call stack of the AskTimeoutException may help to identify where this timeout happened. Thanks, Zhu Zhu Dominik Wosiński 于2019年11月11日周一 上午3:35写道： > Hey, > I have a very specific use case. I have a history of records stored as > Parquet in S3. I would like to read and process them with Flink. The issue > is that the number of files is quite large ( >100k). If I provide the full > list of files to HadoopInputFormat that I am using it will fail with > AskTimeoutException, which Is weird since I am using YARN and setting the > -yD akka.ask.timeout=600s, even thought according to the logs the setting > is processed properly, the job execution still with AskTimeoutException > after 10s, which seems weird to me. I have managed to go around this, by > grouping files and reading them in a loop, so that finally I have the > Seq[DataSet]. But if I try to union those datasets, then I will > receive the AskTimeoutException again. So my question is, what can be the > reason behind this exception being thrown and why is the setting ignored, > even if this is pared properly. > > I will be glad for any help. > > Best Regards, > Dom. >
Re: [ANNOUNCE] Jark Wu is now part of the Flink PMC
Congrats Jark! On Mon, Nov 11, 2019 at 2:49 PM Yadong Xie wrote: > Congratulations Jark! > > Danny Chan 于2019年11月11日周一 上午10:41写道： > > > Congrats Jark! Well deserved ! > > > > Best, > > Danny Chan > > 在 2019年11月11日 +0800 AM9:59，dev@flink.apache.org，写道： > > > > > > Congrats Jark! > > > -- Best regards! Rui Li
[jira] [Created] (FLINK-14705) Remove nullables argument of BatchTestBase.registerCollection in blink planner
Danny Chen created FLINK-14705: -- Summary: Remove nullables argument of BatchTestBase.registerCollection in blink planner Key: FLINK-14705 URL: https://issues.apache.org/jira/browse/FLINK-14705 Project: Flink Issue Type: Improvement Reporter: Danny Chen The register collection method use TypeInformation to register collections, the type information does not have nullability attribute, we expect to use the new DataType instead. So for these test cases, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14706) Remove legacy KvStateMessage
Yun Tang created FLINK-14706:
Summary: Remove legacy KvStateMessage
Key: FLINK-14706
URL: https://issues.apache.org/jira/browse/FLINK-14706
Project: Flink
Issue Type: Sub-task
Reporter: Yun Tang
Fix For: 1.10.0
All {{KvStateMessage}} are actor messages for {{InternalKvState}} lookup and
registration, and actor gateway has been removed in FLINK-12323, we should also
remove these legacy actor messages.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-14707) Refactor checkpoint related methods within Environment
Yun Tang created FLINK-14707:
Summary: Refactor checkpoint related methods within Environment
Key: FLINK-14707
URL: https://issues.apache.org/jira/browse/FLINK-14707
Project: Flink
Issue Type: Improvement
Reporter: Yun Tang
Fix For: 1.10.0
After FLINK-7720 fixed, the two methods of
{{Environment#acknowledgeCheckpoint}} have been actually useless, and their
usage have been overridden by {{TaskStateManager#reportTaskStateSnapshots}}.
More generally, we would consider {{Environment}} interface as getter to access
components instead of doing real actions. I prefer to remove
{{acknowledgeCheckpoint}} and {{declineCheckpoint}} methods or at least
deprecated them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] Stateful Functions - Contribution Details
Thanks, all for the discussion! About the name: - Like Igal mentioned, the name "Stateful Functions" and the abbreviation "statefun" underwent some iterations and testing with a small sample of developers from a few companies. If anyone has an amazing suggestion for another name, please share. Would be great to also test it with a small sample of developers from a few companies, just to make sure we have at least a bit of outside feedback. - fun vs. fn vs. func: I think these are more or less equivalent, there are examples of each one in some language. Working with the code over the last months, we found "statefun" to be somehow appealing. Maybe as a datapoint, Beam uses "DoFn" but pronounces it "doo-fun". So, why not go with "fun" directly? About mailing lists: - There are pros and cons for separating the mailing lists or not to do that. - Having the same mailing lists gives synergies around questions for operating the system. - Having the same mailing lists can create confusion. For example, statefun uses a simpler, more restrictive, easier to understand serialization scheme. Answers coming from serialization in Flink core can easily be confusing there. - Having the same mailing lists can be overwhelming for developers that are new and only interested in that particular angle. - Having a different dev mailing list makes only sense if we use a different Jira project, because FLINK-X issue creation is linked to the mailing list. => I think it is fine to start with the same mailing list and observe first. If we find it problematic, we can separate the mailing lists. About the repository name: - The project is still called "Stateful Functions", but it is a mouth full, so it would be nice to have something more concise for the repo name, hence the suggestion for "statefun". - @Chesnay - Are you concerned about the project name (Stateful Functions) or the abbreviation (statefun) ? Best, Stephan On Mon, Nov 11, 2019 at 6:21 AM tison wrote: > I second Chesnay's opinions, which I'd like to pick up is that I highly > recommend > reuse existing mailing lists. We can always build a separated list when the > specific > community grows, but it is hard to do it in the contract direction. > > I don't stick to the name but vote my coin to "statefun". Playing with > statefun will be > fun, I think :-) (Generally, Erlang uses "fun", Go uses "func" and Rust > uses "fn", I > don't find a strong reason that "func" is an objective better choice > > Best, > tison. > > > Xuefu Z 于2019年11月9日周六 上午4:16写道： > > > Regarding the package name, etc: > > > > statefun certainly sounds more interesting, but it's confusing in my > > opinion and doesn't reflect its true nature. A letter "c" at the end may > > helps as "func" is more used as a short for "function" in CS. > > > > Thanks, > > Xuefu > > > > On Fri, Nov 8, 2019 at 3:52 AM Igal Shilman wrote: > > > > > Hi Chesnay, > > > > > > The correct link for [1] is: > > > > > > > > > http://mail-archives.apache.org/mod_mbox/flink-dev/201911.mbox/%3CCANC1h_vicBWQSGws6Q%2BTXJXde0K%2BAMoVN4VqGU_Hykb1N7J8ng%40mail.gmail.com%3E > > > > > > 1) There is no relevant post, this is the name that is currently used > > both > > > for the website and internally. > > > The name is not the original name, and it evolved out of internal > > > discussions and a/b-testing with few early users, this name > > > was able to "position" the project at the correct place better than > > others. > > > If more people would feel unconvinced, or you would strongly oppose to > > it, > > > then we can create a separate discussion thread. > > > > > > 4) Ok, I will change the proposal to option (b). > > > > > > Kind regards, > > > Igal. > > > > > > On Thu, Nov 7, 2019 at 5:29 PM Chesnay Schepler > > > wrote: > > > > > > > [1] Does not directly link to the voting thread. > > > > > > > > 1) I skimmed all 3 threads about the stateful functions proposal and > > > > could not find a rational for the repository name, I'd appreciate a > > > > direct link to the relevant post. > > > > > > > > 2.1) +1 as we use o.a.f also for flink-shaded > > > > > > > > 3) +1 as it follows the existing package conventions for libraries. > > > > > > > > 4) b; I see no reason why we would isolate mailing lists when we > > haven't > > > > done so for the myriad of other components that are largely > independent > > > > from each other (like SQL). > > > > There are some practical issues here with having a separate dev ML, > for > > > > example where to send FLIPs or release threads and ensuring they > reach > > a > > > > large enough audience, which a dedicated ML would likely hinder. > > > > I'm currently also assuming that builds/commits also go to the > general > > > > flink MLs, making it even weirder if just dev were spliced out. > > > > > > > > 5) separate component, like "API / Statefun" > > > > > > > > Personally I'm not sold on the "statefun" name, has this been a > > > > discussion item in one o
[jira] [Created] (FLINK-14708) Introduce full restarts failover strategy for NG scheduler
Zhu Zhu created FLINK-14708: --- Summary: Introduce full restarts failover strategy for NG scheduler Key: FLINK-14708 URL: https://issues.apache.org/jira/browse/FLINK-14708 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Affects Versions: 1.10.0 Reporter: Zhu Zhu Fix For: 1.10.0 This strategy allows users to force all the tasks to be restarted if any task fails. It may help if it's non-sense or even worse if only part of the tasks can acquire resources and process data. In such cases, checkpoints cannot be completed and progress cannot be persisted. Supporting this strategy also enables the NG scheduler to be backward compatible for existing failover strategy configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14709) Allow outputting elements in close method of chained drivers.
David Moravek created FLINK-14709: - Summary: Allow outputting elements in close method of chained drivers. Key: FLINK-14709 URL: https://issues.apache.org/jira/browse/FLINK-14709 Project: Flink Issue Type: Improvement Components: Runtime / Task Affects Versions: 1.9.1, 1.8.1, 1.7.2 Reporter: David Moravek Currently, BatchTask and DataSourceTask only allow outputting elements in close method of "rich" operators, that they directly execute. Task workflow is as follows: 1) open "head" driver (calls "open" method on udf) 2) open chained drivers 3) run "head" driver 4) close "head" driver (calls "close" method on udf) 5) close output collector (no elements can be collected after this point) 6) close chained drivers In order to properly support outputs from close method, we want to switch 6) and 5). We also need to tweak implementation of Reduce / Combine chained drivers, because they dispose sorters in closeTask method (this should be done in the close method). This would bring huge performance improvement for Beam users, because we could properly implement bundling on batch (whole partition = single bundle). -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] FLIP-79: Flink Function DDL Support (1.10 Release Feature Only)
+1 (binding) On Sat, Nov 9, 2019 at 11:17 PM Kurt Young wrote: > +1 (binding) > > Best, > Kurt > > > On Sun, Nov 10, 2019 at 12:25 PM Peter Huang > wrote: > > > Hi Yu, > > > > Thanks for your reminder about the timeline of delivering the basic > > function DDL in release 1.10. > > As I replied to Xuefu, the "CREATE FUNCTION" and "DROP FUNCTION" can > > relatively easy achieve by revising the existing PR. > > Definitely, I probably need to start to work on a basic version of PR for > > "ALTER FUNCTION" and "SHOW FUNCTIONS". > > Please let me know if you have any suggestion to better align the > timeline > > of the ongoing catalog related efforts. > > > > Best Regards > > Peter Huang > > > > > > On Sat, Nov 9, 2019 at 7:26 PM Yu Li wrote: > > > > > Thanks for driving this Peter! > > > > > > I agree it would be great if we could include this feature in 1.10. > > > However, FWIW, since we are following the time-based release policy [1] > > and > > > 1.10 release is approaching its feature freeze (planned to be at the > end > > of > > > November) [2], I'm a little bit concerned about the schedule. > > > > > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases > > > [2] > > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-10-td32824.html > > > > > > Best Regards, > > > Yu > > > > > > > > > On Sat, 9 Nov 2019 at 04:12, Xuefu Z wrote: > > > > > > > Hi Peter, > > > > > > > > Thanks for driving this. I'm all-in for this. However, as I read the > > > latest > > > > FLIP, I have a couple of questions/comments: > > > > > > > > 1. It seems that "JVM" is proposed as a language type in parallel to > > > > python. I'm not sure that's very intuitive. JVM stands for "java > > virtual > > > > machine", so the language is really "JAVA", correct? I know "scala" > is > > > also > > > > a language which can be generate java byte code that JVM can execute. > > > > > > > > 2. In the flip, "SHOW FUNCTIONS" and "ALTER FUNCTION" are mentioned > > but I > > > > don't see there is any implementation plan, either in 1.10 or > beyond. I > > > > think we could have more clarification on those. > > > > > > > > Thanks, > > > > Xuefu > > > > > > > > On Fri, Nov 8, 2019 at 10:36 AM Bowen Li > wrote: > > > > > > > > > Peter and I went thru the details and defined scope/plan for 1.10 > > > offline > > > > > in the last few days. +1 (binding) from my side. > > > > > > > > > > On Fri, Nov 8, 2019 at 12:55 AM Terry Wang > > wrote: > > > > > > > > > > > Thanks Peter driving on this. LGTM for 1.10 release feature. > > > > > > > > > > > > +1 from my side. (non-binding) > > > > > > > > > > > > Best, > > > > > > Terry Wang > > > > > > > > > > > > > > > > > > > > > > > > > 2019年11月8日 13:20，Peter Huang 写道： > > > > > > > > > > > > > > Dear All, > > > > > > > > > > > > > > I would like to start the vote for 1.10 release features in > > FLIP-79 > > > > [1] > > > > > > > which is discussed and research consensus in the discussion > > thread > > > > [2]. > > > > > > For > > > > > > > the advanced feature, such as loading function from remote > > > resources, > > > > > > > support scala/python function, we will have the further > > discussion > > > > > after > > > > > > > release 1.10. > > > > > > > > > > > > > > The vote will be open for at least 72 hours. If the voting > > passes, > > > I > > > > > will > > > > > > > close it by 2019-11-10 14:00 UTC. > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-79+Flink+Function+DDL+Support > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Discussion-FLIP-79-Flink-Function-DDL-Support-td33965.html > > > > > > > > > > > > > > Best Regards > > > > > > > Peter Huang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Xuefu Zhang > > > > > > > > "In Honey We Trust!" > > > > > > > > > >
Re: [VOTE] FLIP-79: Flink Function DDL Support (1.10 Release Feature Only)
+1 (non-binding) On Mon, Nov 11, 2019 at 9:54 AM Shuyi Chen wrote: > +1 (binding) > > On Sat, Nov 9, 2019 at 11:17 PM Kurt Young wrote: > > > +1 (binding) > > > > Best, > > Kurt > > > > > > On Sun, Nov 10, 2019 at 12:25 PM Peter Huang > > > wrote: > > > > > Hi Yu, > > > > > > Thanks for your reminder about the timeline of delivering the basic > > > function DDL in release 1.10. > > > As I replied to Xuefu, the "CREATE FUNCTION" and "DROP FUNCTION" can > > > relatively easy achieve by revising the existing PR. > > > Definitely, I probably need to start to work on a basic version of PR > for > > > "ALTER FUNCTION" and "SHOW FUNCTIONS". > > > Please let me know if you have any suggestion to better align the > > timeline > > > of the ongoing catalog related efforts. > > > > > > Best Regards > > > Peter Huang > > > > > > > > > On Sat, Nov 9, 2019 at 7:26 PM Yu Li wrote: > > > > > > > Thanks for driving this Peter! > > > > > > > > I agree it would be great if we could include this feature in 1.10. > > > > However, FWIW, since we are following the time-based release policy > [1] > > > and > > > > 1.10 release is approaching its feature freeze (planned to be at the > > end > > > of > > > > November) [2], I'm a little bit concerned about the schedule. > > > > > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases > > > > [2] > > > > > > > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-10-td32824.html > > > > > > > > Best Regards, > > > > Yu > > > > > > > > > > > > On Sat, 9 Nov 2019 at 04:12, Xuefu Z wrote: > > > > > > > > > Hi Peter, > > > > > > > > > > Thanks for driving this. I'm all-in for this. However, as I read > the > > > > latest > > > > > FLIP, I have a couple of questions/comments: > > > > > > > > > > 1. It seems that "JVM" is proposed as a language type in parallel > to > > > > > python. I'm not sure that's very intuitive. JVM stands for "java > > > virtual > > > > > machine", so the language is really "JAVA", correct? I know "scala" > > is > > > > also > > > > > a language which can be generate java byte code that JVM can > execute. > > > > > > > > > > 2. In the flip, "SHOW FUNCTIONS" and "ALTER FUNCTION" are mentioned > > > but I > > > > > don't see there is any implementation plan, either in 1.10 or > > beyond. I > > > > > think we could have more clarification on those. > > > > > > > > > > Thanks, > > > > > Xuefu > > > > > > > > > > On Fri, Nov 8, 2019 at 10:36 AM Bowen Li > > wrote: > > > > > > > > > > > Peter and I went thru the details and defined scope/plan for 1.10 > > > > offline > > > > > > in the last few days. +1 (binding) from my side. > > > > > > > > > > > > On Fri, Nov 8, 2019 at 12:55 AM Terry Wang > > > wrote: > > > > > > > > > > > > > Thanks Peter driving on this. LGTM for 1.10 release feature. > > > > > > > > > > > > > > +1 from my side. (non-binding) > > > > > > > > > > > > > > Best, > > > > > > > Terry Wang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2019年11月8日 13:20，Peter Huang > 写道： > > > > > > > > > > > > > > > > Dear All, > > > > > > > > > > > > > > > > I would like to start the vote for 1.10 release features in > > > FLIP-79 > > > > > [1] > > > > > > > > which is discussed and research consensus in the discussion > > > thread > > > > > [2]. > > > > > > > For > > > > > > > > the advanced feature, such as loading function from remote > > > > resources, > > > > > > > > support scala/python function, we will have the further > > > discussion > > > > > > after > > > > > > > > release 1.10. > > > > > > > > > > > > > > > > The vote will be open for at least 72 hours. If the voting > > > passes, > > > > I > > > > > > will > > > > > > > > close it by 2019-11-10 14:00 UTC. > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-79+Flink+Function+DDL+Support > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Discussion-FLIP-79-Flink-Function-DDL-Support-td33965.html > > > > > > > > > > > > > > > > Best Regards > > > > > > > > Peter Huang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Xuefu Zhang > > > > > > > > > > "In Honey We Trust!" > > > > > > > > > > > > > > > -- Xuefu Zhang "In Honey We Trust!"
Re: [VOTE] FLIP-79: Flink Function DDL Support (1.10 Release Feature Only)
+1 (non-binding) Best, Vino Xuefu Z 于2019年11月12日周二 上午3:27写道： > +1 (non-binding) > > On Mon, Nov 11, 2019 at 9:54 AM Shuyi Chen wrote: > > > +1 (binding) > > > > On Sat, Nov 9, 2019 at 11:17 PM Kurt Young wrote: > > > > > +1 (binding) > > > > > > Best, > > > Kurt > > > > > > > > > On Sun, Nov 10, 2019 at 12:25 PM Peter Huang < > huangzhenqiu0...@gmail.com > > > > > > wrote: > > > > > > > Hi Yu, > > > > > > > > Thanks for your reminder about the timeline of delivering the basic > > > > function DDL in release 1.10. > > > > As I replied to Xuefu, the "CREATE FUNCTION" and "DROP FUNCTION" can > > > > relatively easy achieve by revising the existing PR. > > > > Definitely, I probably need to start to work on a basic version of PR > > for > > > > "ALTER FUNCTION" and "SHOW FUNCTIONS". > > > > Please let me know if you have any suggestion to better align the > > > timeline > > > > of the ongoing catalog related efforts. > > > > > > > > Best Regards > > > > Peter Huang > > > > > > > > > > > > On Sat, Nov 9, 2019 at 7:26 PM Yu Li wrote: > > > > > > > > > Thanks for driving this Peter! > > > > > > > > > > I agree it would be great if we could include this feature in 1.10. > > > > > However, FWIW, since we are following the time-based release policy > > [1] > > > > and > > > > > 1.10 release is approaching its feature freeze (planned to be at > the > > > end > > > > of > > > > > November) [2], I'm a little bit concerned about the schedule. > > > > > > > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases > > > > > [2] > > > > > > > > > > > > > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-10-td32824.html > > > > > > > > > > Best Regards, > > > > > Yu > > > > > > > > > > > > > > > On Sat, 9 Nov 2019 at 04:12, Xuefu Z wrote: > > > > > > > > > > > Hi Peter, > > > > > > > > > > > > Thanks for driving this. I'm all-in for this. However, as I read > > the > > > > > latest > > > > > > FLIP, I have a couple of questions/comments: > > > > > > > > > > > > 1. It seems that "JVM" is proposed as a language type in parallel > > to > > > > > > python. I'm not sure that's very intuitive. JVM stands for "java > > > > virtual > > > > > > machine", so the language is really "JAVA", correct? I know > "scala" > > > is > > > > > also > > > > > > a language which can be generate java byte code that JVM can > > execute. > > > > > > > > > > > > 2. In the flip, "SHOW FUNCTIONS" and "ALTER FUNCTION" are > mentioned > > > > but I > > > > > > don't see there is any implementation plan, either in 1.10 or > > > beyond. I > > > > > > think we could have more clarification on those. > > > > > > > > > > > > Thanks, > > > > > > Xuefu > > > > > > > > > > > > On Fri, Nov 8, 2019 at 10:36 AM Bowen Li > > > wrote: > > > > > > > > > > > > > Peter and I went thru the details and defined scope/plan for > 1.10 > > > > > offline > > > > > > > in the last few days. +1 (binding) from my side. > > > > > > > > > > > > > > On Fri, Nov 8, 2019 at 12:55 AM Terry Wang > > > > > wrote: > > > > > > > > > > > > > > > Thanks Peter driving on this. LGTM for 1.10 release feature. > > > > > > > > > > > > > > > > +1 from my side. (non-binding) > > > > > > > > > > > > > > > > Best, > > > > > > > > Terry Wang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2019年11月8日 13:20，Peter Huang > > 写道： > > > > > > > > > > > > > > > > > > Dear All, > > > > > > > > > > > > > > > > > > I would like to start the vote for 1.10 release features in > > > > FLIP-79 > > > > > > [1] > > > > > > > > > which is discussed and research consensus in the discussion > > > > thread > > > > > > [2]. > > > > > > > > For > > > > > > > > > the advanced feature, such as loading function from remote > > > > > resources, > > > > > > > > > support scala/python function, we will have the further > > > > discussion > > > > > > > after > > > > > > > > > release 1.10. > > > > > > > > > > > > > > > > > > The vote will be open for at least 72 hours. If the voting > > > > passes, > > > > > I > > > > > > > will > > > > > > > > > close it by 2019-11-10 14:00 UTC. > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-79+Flink+Function+DDL+Support > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Discussion-FLIP-79-Flink-Function-DDL-Support-td33965.html > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > > Peter Huang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Xuefu Zhang > > > > > > > > > > > > "In Honey We Trust!" > > > > > > > > > > > > > > > > > > > > > > > -- > Xuefu Zhang > > "In Honey We Trust!" >
Re: Flink Read thousands of files with batch
I suspect you suffer from Client submission failure which also throws AskTimeoutException. The related configure option are `akka.client.timeout` which you can increase. However, there was some cases you can resolve the problem by upgrading Java to latest minimum version 8u212 Best, tison. Zhu Zhu 于2019年11月11日周一 下午6:03写道： > Hi Dominik, > > Would you check whether the JM GC status? > One possible cause is that the large number of file metas > inHadoopInputFormat is burdening the JM memory. > > `akka.ask.timeout` is the default RPC timeout, while some RPCs may override > this timeout for their own purpose. e.g. the RPCs from web usually use > `web.timeout`. > Providing the detailed call stack of the AskTimeoutException may help to > identify where this timeout happened. > > Thanks, > Zhu Zhu > > Dominik Wosiński 于2019年11月11日周一 上午3:35写道： > > > Hey, > > I have a very specific use case. I have a history of records stored as > > Parquet in S3. I would like to read and process them with Flink. The > issue > > is that the number of files is quite large ( >100k). If I provide the > full > > list of files to HadoopInputFormat that I am using it will fail with > > AskTimeoutException, which Is weird since I am using YARN and setting the > > -yD akka.ask.timeout=600s, even thought according to the logs the setting > > is processed properly, the job execution still with AskTimeoutException > > after 10s, which seems weird to me. I have managed to go around this, by > > grouping files and reading them in a loop, so that finally I have the > > Seq[DataSet]. But if I try to union those datasets, then I will > > receive the AskTimeoutException again. So my question is, what can be the > > reason behind this exception being thrown and why is the setting ignored, > > even if this is pared properly. > > > > I will be glad for any help. > > > > Best Regards, > > Dom. > > >
Re: [VOTE] FLIP-79: Flink Function DDL Support (1.10 Release Feature Only)
+1 (non-binding) Best, Jingsong Lee On Tue, Nov 12, 2019 at 9:49 AM vino yang wrote: > +1 (non-binding) > > Best, > Vino > > Xuefu Z 于2019年11月12日周二 上午3:27写道： > > > +1 (non-binding) > > > > On Mon, Nov 11, 2019 at 9:54 AM Shuyi Chen wrote: > > > > > +1 (binding) > > > > > > On Sat, Nov 9, 2019 at 11:17 PM Kurt Young wrote: > > > > > > > +1 (binding) > > > > > > > > Best, > > > > Kurt > > > > > > > > > > > > On Sun, Nov 10, 2019 at 12:25 PM Peter Huang < > > huangzhenqiu0...@gmail.com > > > > > > > > wrote: > > > > > > > > > Hi Yu, > > > > > > > > > > Thanks for your reminder about the timeline of delivering the basic > > > > > function DDL in release 1.10. > > > > > As I replied to Xuefu, the "CREATE FUNCTION" and "DROP FUNCTION" > can > > > > > relatively easy achieve by revising the existing PR. > > > > > Definitely, I probably need to start to work on a basic version of > PR > > > for > > > > > "ALTER FUNCTION" and "SHOW FUNCTIONS". > > > > > Please let me know if you have any suggestion to better align the > > > > timeline > > > > > of the ongoing catalog related efforts. > > > > > > > > > > Best Regards > > > > > Peter Huang > > > > > > > > > > > > > > > On Sat, Nov 9, 2019 at 7:26 PM Yu Li wrote: > > > > > > > > > > > Thanks for driving this Peter! > > > > > > > > > > > > I agree it would be great if we could include this feature in > 1.10. > > > > > > However, FWIW, since we are following the time-based release > policy > > > [1] > > > > > and > > > > > > 1.10 release is approaching its feature freeze (planned to be at > > the > > > > end > > > > > of > > > > > > November) [2], I'm a little bit concerned about the schedule. > > > > > > > > > > > > [1] > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-10-td32824.html > > > > > > > > > > > > Best Regards, > > > > > > Yu > > > > > > > > > > > > > > > > > > On Sat, 9 Nov 2019 at 04:12, Xuefu Z wrote: > > > > > > > > > > > > > Hi Peter, > > > > > > > > > > > > > > Thanks for driving this. I'm all-in for this. However, as I > read > > > the > > > > > > latest > > > > > > > FLIP, I have a couple of questions/comments: > > > > > > > > > > > > > > 1. It seems that "JVM" is proposed as a language type in > parallel > > > to > > > > > > > python. I'm not sure that's very intuitive. JVM stands for > "java > > > > > virtual > > > > > > > machine", so the language is really "JAVA", correct? I know > > "scala" > > > > is > > > > > > also > > > > > > > a language which can be generate java byte code that JVM can > > > execute. > > > > > > > > > > > > > > 2. In the flip, "SHOW FUNCTIONS" and "ALTER FUNCTION" are > > mentioned > > > > > but I > > > > > > > don't see there is any implementation plan, either in 1.10 or > > > > beyond. I > > > > > > > think we could have more clarification on those. > > > > > > > > > > > > > > Thanks, > > > > > > > Xuefu > > > > > > > > > > > > > > On Fri, Nov 8, 2019 at 10:36 AM Bowen Li > > > > wrote: > > > > > > > > > > > > > > > Peter and I went thru the details and defined scope/plan for > > 1.10 > > > > > > offline > > > > > > > > in the last few days. +1 (binding) from my side. > > > > > > > > > > > > > > > > On Fri, Nov 8, 2019 at 12:55 AM Terry Wang < > zjuwa...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks Peter driving on this. LGTM for 1.10 release > feature. > > > > > > > > > > > > > > > > > > +1 from my side. (non-binding) > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Terry Wang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2019年11月8日 13:20，Peter Huang > > > > 写道： > > > > > > > > > > > > > > > > > > > > Dear All, > > > > > > > > > > > > > > > > > > > > I would like to start the vote for 1.10 release features > in > > > > > FLIP-79 > > > > > > > [1] > > > > > > > > > > which is discussed and research consensus in the > discussion > > > > > thread > > > > > > > [2]. > > > > > > > > > For > > > > > > > > > > the advanced feature, such as loading function from > remote > > > > > > resources, > > > > > > > > > > support scala/python function, we will have the further > > > > > discussion > > > > > > > > after > > > > > > > > > > release 1.10. > > > > > > > > > > > > > > > > > > > > The vote will be open for at least 72 hours. If the > voting > > > > > passes, > > > > > > I > > > > > > > > will > > > > > > > > > > close it by 2019-11-10 14:00 UTC. > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-79+Flink+Function+DDL+Support > > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-flink-mailing-list-archi
Re: [DISCUSS] PyFlink User-Defined Function Resource Management
Hi Jincheng, Thanks for the reply and also looking forward to the feedback from the community. Thanks, Dian > 在 2019年11月11日，下午2:34，jincheng sun 写道： > > Hi all, > > +1， Thanks for bring up this discussion Dian! > > The Resource Management is very important for PyFlink UDF. So, It's great > if anyone can add more comments or inputs in the design doc or feedback in > ML. :) > > Best, > Jincheng > > Dian Fu 于2019年11月5日周二 上午11:32写道： > >> Hi everyone, >> >> In FLIP-58[1] it will add the support of Python user-defined stateless >> function for Python Table API. It will launch a separate Python process for >> Python user-defined function execution. The resources used by the Python >> process should be managed properly by Flink’s resource management >> framework. FLIP-49[2] has proposed a unified memory management framework >> and PyFlink user-defined function resource management should be based on >> it. Jincheng, Hequn, Xintong, GuoWei and I discussed offline about this. I >> draft a design doc[3] and want to start a discussion about PyFlink >> user-defined function resource management. >> >> Welcome any comments on the design doc or giving us feedback on the ML >> directly. >> >> Regards, >> Dian >> >> [1] >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table >> [2] >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors >> [3] >> https://docs.google.com/document/d/1LQP8L66Thu2yVv6YRSfmF9EkkMnwhBHGjcTQ11GUmFc/edit#heading=h.4q4ggaftf78m
[jira] [Created] (FLINK-14710) Decide column nullability according to Hive constraints
Rui Li created FLINK-14710: -- Summary: Decide column nullability according to Hive constraints Key: FLINK-14710 URL: https://issues.apache.org/jira/browse/FLINK-14710 Project: Flink Issue Type: Improvement Components: Connectors / Hive Reporter: Rui Li -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Releasing Flink 1.8.3
Hi all, Thanks a lot for your kindly reply and feedbacks. @Jincheng Thank you very much for providing the release help. @Yang Wang Thanks a lot for your PR. I think it is an important fix! I left some comments in FLINK-13184 and we can discuss whether this is a blocker there. I have gone over all the issues of 1.8.3. Currently, only one critical issue: [Critical]: [FLINK-13184] Starting a TaskExecutor blocks the YarnResourceManager's main thread(@Yang Wang is working on it, reviewed by @Rongrong and @Tison) All issues of this release can also be tracked here: https://issues.apache.org/jira/projects/FLINK/versions/12346112 Ideally, we can kick off the release vote for the first RC early next week. If there are some other critical issues for 1.8.3, please let me know here to account for it for the 1.8.3 release. Best, Hequn On Mon, Nov 11, 2019 at 2:27 PM Danny Chan wrote: > Thanks Hequn to be in change of the release ! > > Best, > Danny Chan > 在 2019年11月9日 +0800 PM3:59，Hequn Cheng ，写道： > > Hi Jincheng, > > > > Thanks for kicking this discussion off! > > > > +1 to the 1.8.3 release as it would be nice to have these important fixes > > and also two > > months have passed since the last release. > > > > Besides, I wonder if I can be the release manager of 1.8.3 or work with > you > > together @Jincheng? > > It's always exciting to help the community as much as possible. > > > > Best, Hequn > > > > On Sat, Nov 9, 2019 at 12:34 PM Dian Fu wrote: > > > > > Hi Jincheng, > > > > > > Thanks a lot for bringing up this discussion. +1 for releasing 1.8.3. > > > > > > Regards, > > > Dian > > > > > > On Sat, Nov 9, 2019 at 12:11 PM jincheng sun > > > > wrote: > > > > > > > Hi Flink devs, > > > > > > > > It has been more than 2 months since the 1.8.2 released. So, What do > you > > > > think about releasing Flink 1.8.3 soon? > > > > > > > > We already have many important bug fixes in the release-1.8 branch > (29 > > > > resolved issues). > > > > > > > > Most notable fixes are: > > > > > > > > - FLINK-14010 Dispatcher & JobManagers don't give up leadership when > AM > > > is > > > > shut down > > > > - FLINK-14315 NPE with JobMaster.disconnectTaskManager > > > > - FLINK-12848 Method equals() in RowTypeInfo should consider > fieldsNames > > > > - FLINK-12342 Yarn Resource Manager Acquires Too Many Containers > > > > - FLINK-14589 Redundant slot requests with the same AllocationID > leads to > > > > inconsistent slot table > > > > > > > > Furthermore, the following critical issues is in progress, maybe we > can > > > > wait for it if it is not too much effort. > > > > > > > > - FLINK-13184 Starting a TaskExecutor blocks the > YarnResourceManager's > > > main > > > > thread > > > > > > > > Please let me know what you think? > > > > > > > > Best, > > > > Jincheng > > > > > > > >
[jira] [Created] (FLINK-14711) Create Alter Function and Show Functions DDL
Zhenqiu Huang created FLINK-14711: - Summary: Create Alter Function and Show Functions DDL Key: FLINK-14711 URL: https://issues.apache.org/jira/browse/FLINK-14711 Project: Flink Issue Type: Sub-task Components: Table SQL / API Affects Versions: 1.10.0 Reporter: Zhenqiu Huang It is a sibling ticket for https://issues.apache.org/jira/browse/FLINK-7151. Alter Function Statement ALTER [TEMPORARY|TEMPORARY SYSTEM] FUNCTION [IF EXISTS] [catalog_name.][db_name.] function_name AS identifier [LANGUAGE JAVA|SCALA|PYTHON]; Show Function Statement SHOW FUNCTIONS [catalog_name.][db_name] -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] FLIP-79: Flink Function DDL Support (1.10 Release Feature Only)
Thanks, everyone for joining the discussion and giving feedback!. The voting time for FLIP-79 has passed. I'm closing the vote now. There were seven +1 votes, 3 of which are binding: - Bowen Li (binding) - Kurt Yong (binding) - Shuyi Chen (binding) - Terry Wang (non-binding) - Xuefu Zhang (non-binding) - Vino Yang (non-binding) - Jingsong Lee (non-binding) There were no disapproving votes. Thus, FLIP-79 has been accepted. Now, we agreed on the function DDL syntax as listed on the FLIP and also the initial execution plan for release 1.10. Currently, I will mainly work on these two Jira tickets. 1) https://issues.apache.org/jira/browse/FLINK-7151 2) https://issues.apache.org/jira/browse/FLINK-14711 Best Regards Peter Huang Best Regards Peter Huang On Mon, Nov 11, 2019 at 7:42 PM Jingsong Li wrote: > +1 (non-binding) > > Best, > Jingsong Lee > > On Tue, Nov 12, 2019 at 9:49 AM vino yang wrote: > > > +1 (non-binding) > > > > Best, > > Vino > > > > Xuefu Z 于2019年11月12日周二 上午3:27写道： > > > > > +1 (non-binding) > > > > > > On Mon, Nov 11, 2019 at 9:54 AM Shuyi Chen wrote: > > > > > > > +1 (binding) > > > > > > > > On Sat, Nov 9, 2019 at 11:17 PM Kurt Young wrote: > > > > > > > > > +1 (binding) > > > > > > > > > > Best, > > > > > Kurt > > > > > > > > > > > > > > > On Sun, Nov 10, 2019 at 12:25 PM Peter Huang < > > > huangzhenqiu0...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > Hi Yu, > > > > > > > > > > > > Thanks for your reminder about the timeline of delivering the > basic > > > > > > function DDL in release 1.10. > > > > > > As I replied to Xuefu, the "CREATE FUNCTION" and "DROP FUNCTION" > > can > > > > > > relatively easy achieve by revising the existing PR. > > > > > > Definitely, I probably need to start to work on a basic version > of > > PR > > > > for > > > > > > "ALTER FUNCTION" and "SHOW FUNCTIONS". > > > > > > Please let me know if you have any suggestion to better align the > > > > > timeline > > > > > > of the ongoing catalog related efforts. > > > > > > > > > > > > Best Regards > > > > > > Peter Huang > > > > > > > > > > > > > > > > > > On Sat, Nov 9, 2019 at 7:26 PM Yu Li wrote: > > > > > > > > > > > > > Thanks for driving this Peter! > > > > > > > > > > > > > > I agree it would be great if we could include this feature in > > 1.10. > > > > > > > However, FWIW, since we are following the time-based release > > policy > > > > [1] > > > > > > and > > > > > > > 1.10 release is approaching its feature freeze (planned to be > at > > > the > > > > > end > > > > > > of > > > > > > > November) [2], I'm a little bit concerned about the schedule. > > > > > > > > > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-10-td32824.html > > > > > > > > > > > > > > Best Regards, > > > > > > > Yu > > > > > > > > > > > > > > > > > > > > > On Sat, 9 Nov 2019 at 04:12, Xuefu Z > wrote: > > > > > > > > > > > > > > > Hi Peter, > > > > > > > > > > > > > > > > Thanks for driving this. I'm all-in for this. However, as I > > read > > > > the > > > > > > > latest > > > > > > > > FLIP, I have a couple of questions/comments: > > > > > > > > > > > > > > > > 1. It seems that "JVM" is proposed as a language type in > > parallel > > > > to > > > > > > > > python. I'm not sure that's very intuitive. JVM stands for > > "java > > > > > > virtual > > > > > > > > machine", so the language is really "JAVA", correct? I know > > > "scala" > > > > > is > > > > > > > also > > > > > > > > a language which can be generate java byte code that JVM can > > > > execute. > > > > > > > > > > > > > > > > 2. In the flip, "SHOW FUNCTIONS" and "ALTER FUNCTION" are > > > mentioned > > > > > > but I > > > > > > > > don't see there is any implementation plan, either in 1.10 or > > > > > beyond. I > > > > > > > > think we could have more clarification on those. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Xuefu > > > > > > > > > > > > > > > > On Fri, Nov 8, 2019 at 10:36 AM Bowen Li < > bowenl...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > > > Peter and I went thru the details and defined scope/plan > for > > > 1.10 > > > > > > > offline > > > > > > > > > in the last few days. +1 (binding) from my side. > > > > > > > > > > > > > > > > > > On Fri, Nov 8, 2019 at 12:55 AM Terry Wang < > > zjuwa...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Thanks Peter driving on this. LGTM for 1.10 release > > feature. > > > > > > > > > > > > > > > > > > > > +1 from my side. (non-binding) > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Terry Wang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2019年11月8日 13:20，Peter Huang < > huangzhenqiu0...@gmail.com > > > > > > > 写道： > > > > > > > > > > > > > > > > >
Re: [VOTE] FLIP-79: Flink Function DDL Support (1.10 Release Feature Only)
Thanks Peter! I assigned the ticket to you, and I can help with reviewing and merging PRs of this FLIP. Ccing 1.10 release manager Yu that we are on track of FLIP-79. Cheers, Bowen On Mon, Nov 11, 2019 at 9:03 PM Peter Huang wrote: > Thanks, everyone for joining the discussion and giving feedback!. The > voting time for FLIP-79 has passed. I'm closing the vote now. > > There were seven +1 votes, 3 of which are binding: > - Bowen Li (binding) > - Kurt Yong (binding) > - Shuyi Chen (binding) > > - Terry Wang (non-binding) > - Xuefu Zhang (non-binding) > - Vino Yang (non-binding) > - Jingsong Lee (non-binding) > > There were no disapproving votes. Thus, FLIP-79 has been accepted. Now, we > agreed on the function DDL syntax as listed on the FLIP > and also the initial execution plan for release 1.10. Currently, I will > mainly work on these two Jira tickets. > > 1) https://issues.apache.org/jira/browse/FLINK-7151 > 2) https://issues.apache.org/jira/browse/FLINK-14711 > > > > Best Regards > Peter Huang > > > Best Regards > Peter Huang > > On Mon, Nov 11, 2019 at 7:42 PM Jingsong Li > wrote: > > > +1 (non-binding) > > > > Best, > > Jingsong Lee > > > > On Tue, Nov 12, 2019 at 9:49 AM vino yang wrote: > > > > > +1 (non-binding) > > > > > > Best, > > > Vino > > > > > > Xuefu Z 于2019年11月12日周二 上午3:27写道： > > > > > > > +1 (non-binding) > > > > > > > > On Mon, Nov 11, 2019 at 9:54 AM Shuyi Chen > wrote: > > > > > > > > > +1 (binding) > > > > > > > > > > On Sat, Nov 9, 2019 at 11:17 PM Kurt Young > wrote: > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > Best, > > > > > > Kurt > > > > > > > > > > > > > > > > > > On Sun, Nov 10, 2019 at 12:25 PM Peter Huang < > > > > huangzhenqiu0...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Yu, > > > > > > > > > > > > > > Thanks for your reminder about the timeline of delivering the > > basic > > > > > > > function DDL in release 1.10. > > > > > > > As I replied to Xuefu, the "CREATE FUNCTION" and "DROP > FUNCTION" > > > can > > > > > > > relatively easy achieve by revising the existing PR. > > > > > > > Definitely, I probably need to start to work on a basic version > > of > > > PR > > > > > for > > > > > > > "ALTER FUNCTION" and "SHOW FUNCTIONS". > > > > > > > Please let me know if you have any suggestion to better align > the > > > > > > timeline > > > > > > > of the ongoing catalog related efforts. > > > > > > > > > > > > > > Best Regards > > > > > > > Peter Huang > > > > > > > > > > > > > > > > > > > > > On Sat, Nov 9, 2019 at 7:26 PM Yu Li wrote: > > > > > > > > > > > > > > > Thanks for driving this Peter! > > > > > > > > > > > > > > > > I agree it would be great if we could include this feature in > > > 1.10. > > > > > > > > However, FWIW, since we are following the time-based release > > > policy > > > > > [1] > > > > > > > and > > > > > > > > 1.10 release is approaching its feature freeze (planned to be > > at > > > > the > > > > > > end > > > > > > > of > > > > > > > > November) [2], I'm a little bit concerned about the schedule. > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-10-td32824.html > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > Yu > > > > > > > > > > > > > > > > > > > > > > > > On Sat, 9 Nov 2019 at 04:12, Xuefu Z > > wrote: > > > > > > > > > > > > > > > > > Hi Peter, > > > > > > > > > > > > > > > > > > Thanks for driving this. I'm all-in for this. However, as I > > > read > > > > > the > > > > > > > > latest > > > > > > > > > FLIP, I have a couple of questions/comments: > > > > > > > > > > > > > > > > > > 1. It seems that "JVM" is proposed as a language type in > > > parallel > > > > > to > > > > > > > > > python. I'm not sure that's very intuitive. JVM stands for > > > "java > > > > > > > virtual > > > > > > > > > machine", so the language is really "JAVA", correct? I know > > > > "scala" > > > > > > is > > > > > > > > also > > > > > > > > > a language which can be generate java byte code that JVM > can > > > > > execute. > > > > > > > > > > > > > > > > > > 2. In the flip, "SHOW FUNCTIONS" and "ALTER FUNCTION" are > > > > mentioned > > > > > > > but I > > > > > > > > > don't see there is any implementation plan, either in 1.10 > or > > > > > > beyond. I > > > > > > > > > think we could have more clarification on those. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Xuefu > > > > > > > > > > > > > > > > > > On Fri, Nov 8, 2019 at 10:36 AM Bowen Li < > > bowenl...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Peter and I went thru the details and defined scope/plan > > for > > > > 1.10 > > > > > > > > offline > > > > > > > > > > in the last few days. +1 (binding) from my side. > > >
Re: [VOTE] FLIP-79: Flink Function DDL Support (1.10 Release Feature Only)
Thank you in advance. Once the PR is done from my side, I will @ you and yu for review. On Mon, Nov 11, 2019 at 9:15 PM Bowen Li wrote: > Thanks Peter! I assigned the ticket to you, and I can help with reviewing > and merging PRs of this FLIP. > > Ccing 1.10 release manager Yu that we are on track of FLIP-79. > > Cheers, > Bowen > > On Mon, Nov 11, 2019 at 9:03 PM Peter Huang > wrote: > > > Thanks, everyone for joining the discussion and giving feedback!. The > > voting time for FLIP-79 has passed. I'm closing the vote now. > > > > There were seven +1 votes, 3 of which are binding: > > - Bowen Li (binding) > > - Kurt Yong (binding) > > - Shuyi Chen (binding) > > > > - Terry Wang (non-binding) > > - Xuefu Zhang (non-binding) > > - Vino Yang (non-binding) > > - Jingsong Lee (non-binding) > > > > There were no disapproving votes. Thus, FLIP-79 has been accepted. Now, > we > > agreed on the function DDL syntax as listed on the FLIP > > and also the initial execution plan for release 1.10. Currently, I will > > mainly work on these two Jira tickets. > > > > 1) https://issues.apache.org/jira/browse/FLINK-7151 > > 2) https://issues.apache.org/jira/browse/FLINK-14711 > > > > > > > > Best Regards > > Peter Huang > > > > > > Best Regards > > Peter Huang > > > > On Mon, Nov 11, 2019 at 7:42 PM Jingsong Li > > wrote: > > > > > +1 (non-binding) > > > > > > Best, > > > Jingsong Lee > > > > > > On Tue, Nov 12, 2019 at 9:49 AM vino yang > wrote: > > > > > > > +1 (non-binding) > > > > > > > > Best, > > > > Vino > > > > > > > > Xuefu Z 于2019年11月12日周二 上午3:27写道： > > > > > > > > > +1 (non-binding) > > > > > > > > > > On Mon, Nov 11, 2019 at 9:54 AM Shuyi Chen > > wrote: > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > On Sat, Nov 9, 2019 at 11:17 PM Kurt Young > > wrote: > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > Best, > > > > > > > Kurt > > > > > > > > > > > > > > > > > > > > > On Sun, Nov 10, 2019 at 12:25 PM Peter Huang < > > > > > huangzhenqiu0...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Yu, > > > > > > > > > > > > > > > > Thanks for your reminder about the timeline of delivering the > > > basic > > > > > > > > function DDL in release 1.10. > > > > > > > > As I replied to Xuefu, the "CREATE FUNCTION" and "DROP > > FUNCTION" > > > > can > > > > > > > > relatively easy achieve by revising the existing PR. > > > > > > > > Definitely, I probably need to start to work on a basic > version > > > of > > > > PR > > > > > > for > > > > > > > > "ALTER FUNCTION" and "SHOW FUNCTIONS". > > > > > > > > Please let me know if you have any suggestion to better align > > the > > > > > > > timeline > > > > > > > > of the ongoing catalog related efforts. > > > > > > > > > > > > > > > > Best Regards > > > > > > > > Peter Huang > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Nov 9, 2019 at 7:26 PM Yu Li > wrote: > > > > > > > > > > > > > > > > > Thanks for driving this Peter! > > > > > > > > > > > > > > > > > > I agree it would be great if we could include this feature > in > > > > 1.10. > > > > > > > > > However, FWIW, since we are following the time-based > release > > > > policy > > > > > > [1] > > > > > > > > and > > > > > > > > > 1.10 release is approaching its feature freeze (planned to > be > > > at > > > > > the > > > > > > > end > > > > > > > > of > > > > > > > > > November) [2], I'm a little bit concerned about the > schedule. > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-10-td32824.html > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > Yu > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, 9 Nov 2019 at 04:12, Xuefu Z > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Peter, > > > > > > > > > > > > > > > > > > > > Thanks for driving this. I'm all-in for this. However, > as I > > > > read > > > > > > the > > > > > > > > > latest > > > > > > > > > > FLIP, I have a couple of questions/comments: > > > > > > > > > > > > > > > > > > > > 1. It seems that "JVM" is proposed as a language type in > > > > parallel > > > > > > to > > > > > > > > > > python. I'm not sure that's very intuitive. JVM stands > for > > > > "java > > > > > > > > virtual > > > > > > > > > > machine", so the language is really "JAVA", correct? I > know > > > > > "scala" > > > > > > > is > > > > > > > > > also > > > > > > > > > > a language which can be generate java byte code that JVM > > can > > > > > > execute. > > > > > > > > > > > > > > > > > > > > 2. In the flip, "SHOW FUNCTIONS" and "ALTER FUNCTION" are > > > > > mentioned > > > > > > > > but I > > > > > > > > > > don't see there is any implementation plan, either in > 1.10 > >
[jira] [Created] (FLINK-14712) Add NetWork metric for IOMetricsInfo
lining created FLINK-14712:
--
Summary: Add NetWork metric for IOMetricsInfo
Key: FLINK-14712
URL: https://issues.apache.org/jira/browse/FLINK-14712
Project: Flink
Issue Type: Bug
Components: Runtime / Metrics, Runtime / Network, Runtime / REST
Reporter: lining
Attachments: image-2019-11-12-14-30-16-130.png
h4. (1) The current monitor is heavy-weight.
* Backpressure monitoring works by repeatedly taking stack trace samples of
your running tasks.
h4. (2) It is difficult to find out which vertex is the source of
backpressure.
* User need to know current and upstream's network metric to judge current
whether is the source of backpressure. Now user has to record relevant
information.
h3. Proposed Changes
Update IOMetricsInfo add outPoolUsage, inputExclusiveBuffersUsage,
inputFloatingBuffersUsage:
public final class IOMetricsInfo {private final float outPoolUsage;
private final float inputExclusiveBuffersUsage;private final float
inputFloatingBuffersUsage;}
JobDetailsInfo.JobVertexDetailsInfo merge use Math.max.(ps: outPoolUsage is
from upstream)、
According to !image-2019-11-12-14-30-16-130.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-14713) Show Attempt History in Vertex SubTask
lining created FLINK-14713:
--
Summary: Show Attempt History in Vertex SubTask
Key: FLINK-14713
URL: https://issues.apache.org/jira/browse/FLINK-14713
Project: Flink
Issue Type: Improvement
Components: Runtime / REST, Runtime / Web Frontend
Reporter: lining
Flink jobs could recovery by failover, but the user couldn't see any
information about the jobs' failure. There isn't information about the failed
attempt.
h3. Proposed Changes
h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* return subtask all attempt and state
* get prior attempts according to
{code:java}
final AccessExecution execution = executionVertex.getCurrentExecutionAttempt();
final int currentAttemptNum = execution.getAttemptNumber();
if (currentAttemptNum > 0) {
for (int i = currentAttemptNum - 1; i >= 0; i--) {
final AccessExecution currentExecution =
executionVertex.getPriorExecutionAttempt(i);
if (currentExecution != null) {
allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
metricFetcher, jobID, jobVertexID));
}
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [VOTE] FLIP-79: Flink Function DDL Support (1.10 Release Feature Only)
Thanks Bowen and Peter! Great to know. Best Regards, Yu On Tue, 12 Nov 2019 at 13:38, Peter Huang wrote: > Thank you in advance. Once the PR is done from my side, I will @ you and > yu for review. > > On Mon, Nov 11, 2019 at 9:15 PM Bowen Li wrote: > >> Thanks Peter! I assigned the ticket to you, and I can help with reviewing >> and merging PRs of this FLIP. >> >> Ccing 1.10 release manager Yu that we are on track of FLIP-79. >> >> Cheers, >> Bowen >> >> On Mon, Nov 11, 2019 at 9:03 PM Peter Huang >> wrote: >> >> > Thanks, everyone for joining the discussion and giving feedback!. The >> > voting time for FLIP-79 has passed. I'm closing the vote now. >> > >> > There were seven +1 votes, 3 of which are binding: >> > - Bowen Li (binding) >> > - Kurt Yong (binding) >> > - Shuyi Chen (binding) >> > >> > - Terry Wang (non-binding) >> > - Xuefu Zhang (non-binding) >> > - Vino Yang (non-binding) >> > - Jingsong Lee (non-binding) >> > >> > There were no disapproving votes. Thus, FLIP-79 has been accepted. Now, >> we >> > agreed on the function DDL syntax as listed on the FLIP >> > and also the initial execution plan for release 1.10. Currently, I will >> > mainly work on these two Jira tickets. >> > >> > 1) https://issues.apache.org/jira/browse/FLINK-7151 >> > 2) https://issues.apache.org/jira/browse/FLINK-14711 >> > >> > >> > >> > Best Regards >> > Peter Huang >> > >> > >> > Best Regards >> > Peter Huang >> > >> > On Mon, Nov 11, 2019 at 7:42 PM Jingsong Li >> > wrote: >> > >> > > +1 (non-binding) >> > > >> > > Best, >> > > Jingsong Lee >> > > >> > > On Tue, Nov 12, 2019 at 9:49 AM vino yang >> wrote: >> > > >> > > > +1 (non-binding) >> > > > >> > > > Best, >> > > > Vino >> > > > >> > > > Xuefu Z 于2019年11月12日周二 上午3:27写道： >> > > > >> > > > > +1 (non-binding) >> > > > > >> > > > > On Mon, Nov 11, 2019 at 9:54 AM Shuyi Chen >> > wrote: >> > > > > >> > > > > > +1 (binding) >> > > > > > >> > > > > > On Sat, Nov 9, 2019 at 11:17 PM Kurt Young >> > wrote: >> > > > > > >> > > > > > > +1 (binding) >> > > > > > > >> > > > > > > Best, >> > > > > > > Kurt >> > > > > > > >> > > > > > > >> > > > > > > On Sun, Nov 10, 2019 at 12:25 PM Peter Huang < >> > > > > huangzhenqiu0...@gmail.com >> > > > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > > > Hi Yu, >> > > > > > > > >> > > > > > > > Thanks for your reminder about the timeline of delivering >> the >> > > basic >> > > > > > > > function DDL in release 1.10. >> > > > > > > > As I replied to Xuefu, the "CREATE FUNCTION" and "DROP >> > FUNCTION" >> > > > can >> > > > > > > > relatively easy achieve by revising the existing PR. >> > > > > > > > Definitely, I probably need to start to work on a basic >> version >> > > of >> > > > PR >> > > > > > for >> > > > > > > > "ALTER FUNCTION" and "SHOW FUNCTIONS". >> > > > > > > > Please let me know if you have any suggestion to better >> align >> > the >> > > > > > > timeline >> > > > > > > > of the ongoing catalog related efforts. >> > > > > > > > >> > > > > > > > Best Regards >> > > > > > > > Peter Huang >> > > > > > > > >> > > > > > > > >> > > > > > > > On Sat, Nov 9, 2019 at 7:26 PM Yu Li >> wrote: >> > > > > > > > >> > > > > > > > > Thanks for driving this Peter! >> > > > > > > > > >> > > > > > > > > I agree it would be great if we could include this >> feature in >> > > > 1.10. >> > > > > > > > > However, FWIW, since we are following the time-based >> release >> > > > policy >> > > > > > [1] >> > > > > > > > and >> > > > > > > > > 1.10 release is approaching its feature freeze (planned >> to be >> > > at >> > > > > the >> > > > > > > end >> > > > > > > > of >> > > > > > > > > November) [2], I'm a little bit concerned about the >> schedule. >> > > > > > > > > >> > > > > > > > > [1] >> > > > > > > > >> > > > > >> > https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases >> > > > > > > > > [2] >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-10-td32824.html >> > > > > > > > > >> > > > > > > > > Best Regards, >> > > > > > > > > Yu >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > On Sat, 9 Nov 2019 at 04:12, Xuefu Z >> > > wrote: >> > > > > > > > > >> > > > > > > > > > Hi Peter, >> > > > > > > > > > >> > > > > > > > > > Thanks for driving this. I'm all-in for this. However, >> as I >> > > > read >> > > > > > the >> > > > > > > > > latest >> > > > > > > > > > FLIP, I have a couple of questions/comments: >> > > > > > > > > > >> > > > > > > > > > 1. It seems that "JVM" is proposed as a language type in >> > > > parallel >> > > > > > to >> > > > > > > > > > python. I'm not sure that's very intuitive. JVM stands >> for >> > > > "java >> > > > > > > > virtual >> > > > > > > > > > machine", so the language is really "JAVA", correct? I >> know >> > > > > "scala" >> > > > > > > is >> > > > > > > > > also >> > > > > > > > > > a language whic
[jira] [Created] (FLINK-14714) BytesHashMap should not warn invoking stack when it want to spill
Jingsong Lee created FLINK-14714: Summary: BytesHashMap should not warn invoking stack when it want to spill Key: FLINK-14714 URL: https://issues.apache.org/jira/browse/FLINK-14714 Project: Flink Issue Type: Improvement Components: Table SQL / Runtime Reporter: Jingsong Lee Fix For: 1.10.0 Now BytesHashMap will warn invoking stack when it want to spill, this lead to a lot of messages in logs. We need remove the stack and just print warns. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14715) HiveModuleTest fails with Hive-3.1.1
Rui Li created FLINK-14715: -- Summary: HiveModuleTest fails with Hive-3.1.1 Key: FLINK-14715 URL: https://issues.apache.org/jira/browse/FLINK-14715 Project: Flink Issue Type: Test Components: Connectors / Hive Reporter: Rui Li [ERROR] Failures: [ERROR] HiveModuleTest.testNumberOfBuiltinFunctions:62 expected:<287> but was:<306> -- This message was sent by Atlassian Jira (v8.3.4#803005)
