[jira] [Created] (FLINK-17257) AbstractYarnClusterTest does not compile with Hadoop 2.10

[Xintong Song (Jira)]

Xintong Song created FLINK-17257:


 Summary: AbstractYarnClusterTest does not compile with Hadoop 2.10
 Key: FLINK-17257
 URL: https://issues.apache.org/jira/browse/FLINK-17257
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN, Tests
Affects Versions: 1.9.3, 1.10.1, 1.11.0
Reporter: Xintong Song
 Fix For: 1.11.0, 1.10.2, 1.9.4


In {{AbstractYarnClusterTest}}, we create {{ApplicationReport}} with the static 
method {{ApplicationReport.newInstance}}, which is annotated as private and 
unstable. This method is no longer compatible in Hadoop 2.10.

As a workaround, we can create {{ApplicationReport}} with its default 
constructor and set only the fields that we need.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17258) Enable unaligned checkpoints in tests by default

[Arvid Heise (Jira)]

Arvid Heise created FLINK-17258:
---

 Summary: Enable unaligned checkpoints in tests by default
 Key: FLINK-17258
 URL: https://issues.apache.org/jira/browse/FLINK-17258
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.11.0
Reporter: Arvid Heise
Assignee: Arvid Heise
 Fix For: 1.11.0


To harden unaligned checkpoints as quickly as possible, we want to enable it by 
default for all tests similarly how we did it for credit-based flow control 
network stack.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17259) Have scala 2.12 support

[Jira]

João Boto created FLINK-17259:
-

 Summary: Have scala 2.12 support
 Key: FLINK-17259
 URL: https://issues.apache.org/jira/browse/FLINK-17259
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Affects Versions: 2.0.0
Reporter: João Boto


In statefun-flink is defined the scala.binary.version as 2.11

this force to use this the use of scala 2.11

 

should be the default 2.12? or have the option to chose the scala version



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17260) StreamingKafkaITCase failure on Azure

[Roman Khachatryan (Jira)]

Roman Khachatryan created FLINK-17260:
-

 Summary: StreamingKafkaITCase failure on Azure
 Key: FLINK-17260
 URL: https://issues.apache.org/jira/browse/FLINK-17260
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka, Tests
Affects Versions: 1.11.0
Reporter: Roman Khachatryan


https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_apis/build/builds/7544/logs/165

 
{code:java}
2020-04-16T00:12:32.2848429Z [INFO] Running 
org.apache.flink.tests.util.kafka.StreamingKafkaITCase
2020-04-16T00:14:47.9100927Z [ERROR] Tests run: 3, Failures: 1, Errors: 0, 
Skipped: 0, Time elapsed: 135.621 s <<< FAILURE! - in 
org.apache.flink.tests.util.k afka.StreamingKafkaITCase
2020-04-16T00:14:47.9103036Z [ERROR] testKafka[0: 
kafka-version:0.10.2.0](org.apache.flink.tests.util.kafka.StreamingKafkaITCase) 
 Time elapsed: 46.222 s  <<<  FAILURE!
2020-04-16T00:14:47.9104033Z java.lang.AssertionError: 
expected:<[elephant,27,64213]> but was:<[]>
2020-04-16T00:14:47.9104638Zat org.junit.Assert.fail(Assert.java:88)
2020-04-16T00:14:47.9105148Zat 
org.junit.Assert.failNotEquals(Assert.java:834)
2020-04-16T00:14:47.9105701Zat 
org.junit.Assert.assertEquals(Assert.java:118)
2020-04-16T00:14:47.9106239Zat 
org.junit.Assert.assertEquals(Assert.java:144)
2020-04-16T00:14:47.9107177Zat 
org.apache.flink.tests.util.kafka.StreamingKafkaITCase.testKafka(StreamingKafkaITCase.java:162)
2020-04-16T00:14:47.9107845Zat 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-04-16T00:14:47.9108434Zat 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-04-16T00:14:47.9109318Zat 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-04-16T00:14:47.9109914Zat 
java.lang.reflect.Method.invoke(Method.java:498)
2020-04-16T00:14:47.9110434Zat 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-04-16T00:14:47.9110985Zat 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-04-16T00:14:47.9111548Zat 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-04-16T00:14:47.9112083Zat 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2020-04-16T00:14:47.9112629Zat 
org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
2020-04-16T00:14:47.9113145Zat 
org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
2020-04-16T00:14:47.9113637Zat 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2020-04-16T00:14:47.9114072Zat 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2020-04-16T00:14:47.9114490Zat 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2020-04-16T00:14:47.9115256Zat 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2020-04-16T00:14:47.9115791Zat 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2020-04-16T00:14:47.9116292Zat 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-04-16T00:14:47.9116736Zat 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-04-16T00:14:47.9117779Zat 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-04-16T00:14:47.9118274Zat 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-04-16T00:14:47.9118766Zat 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-04-16T00:14:47.9119204Zat 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-04-16T00:14:47.9119625Zat 
org.junit.runners.Suite.runChild(Suite.java:128)
2020-04-16T00:14:47.9120005Zat 
org.junit.runners.Suite.runChild(Suite.java:27)
2020-04-16T00:14:47.9120428Zat 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-04-16T00:14:47.9120876Zat 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-04-16T00:14:47.9121350Zat 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-04-16T00:14:47.9121805Zat 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-04-16T00:14:47.9122273Zat 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-04-16T00:14:47.9122729Zat 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-04-16T00:14:47.9123130Zat 
org.junit.runners.Suite.runChild(Suite.java:128)
...
2020-04-16T00:14:47.9132530Z
2020-04-16T00:14:47.9134982Z [INFO] Running 
org.apache.flink.tests.util.kafka.SQLClientKafkaITCase
2020-04-16T00:17:18.7332734Z [INFO] Tests run: 3, Failures: 0, Errors: 0, 
Skipped: 0, Time elapsed: 150.813 s - in 
org.apache.flink.tests.util.kafka.SQLClient KafkaITCase
2020-04-16T00:17:19.0840872Z [INFO]
2020-04-16T00:17:19.0841522Z [

[jira] [Created] (FLINK-17261) Statefun docs broken

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-17261:


 Summary: Statefun docs broken
 Key: FLINK-17261
 URL: https://issues.apache.org/jira/browse/FLINK-17261
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 2.0.0, 2.1.0
Reporter: Chesnay Schepler


https://ci.apache.org/builders/flink-statefun-docs-master/builds/56

{code}  Liquid Exception: Liquid syntax error (line 67): Unknown tag 'higlight' 
in deployment-and-operations/packaging.md
Liquid syntax error (line 67): Unknown tag 'higlight'
/home/buildslave/slave/flink-statefun-docs-master/build/docs/.rubydeps/ruby/2.6.0/gems/liquid-4.0.3/lib/liquid/document.rb:23:in
 `unknown_tag'{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17262) Statefun snapshot deployments broken

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-17262:


 Summary: Statefun snapshot deployments broken
 Key: FLINK-17262
 URL: https://issues.apache.org/jira/browse/FLINK-17262
 Project: Flink
  Issue Type: Bug
  Components: Release System
Affects Versions: 2.0.0, 2.1.0
Reporter: Chesnay Schepler


https://builds.apache.org/job/flink-statefun-snapshot-deployment-2.0/12/

{code}
Commit message: "[FLINK-17193] [python-k8s-example] Abort script on failure. 
Build SDK distribution if was not previously built"
 > git rev-list --no-walk d660668ff45312f7c3b10529b29b478efe220e57 # timeout=10
[flink-statefun-snapshot-deployment-2.0] $ /bin/bash 
/tmp/jenkins419419675999100733.sh
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed

  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100   341  100   3410 0914  0 --:--:-- --:--:-- --:--:--   914

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
./tools/snapshots/deploy_snapshot_jars.sh: line 47: mvn: command not found
{code}
Happened twice in a row.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Apache Flink PMC Member - Hequn Chen

[Zhu Zhu]

Congratulations Hequn!

Thanks,
Zhu Zhu

zoudan  于2020年4月20日周一 上午10:34写道：

> Congratulations Hequn
>
> Best,
> Dan Zou
>
>
> > 在 2020年4月20日，10:23，Terry Wang  写道：
> >
> > Congratulations Hequn !!!
> > Best,
> > Terry Wang
> >
> >
> >
> >> 2020年4月20日 10:20，Jingsong Li  写道：
> >>
> >> Congratulations Hequn!
> >>
> >> Best,
> >> Jingsong Lee
> >>
> >> On Mon, Apr 20, 2020 at 9:52 AM jincheng sun 
> >> wrote:
> >>
> >>> Congratulations and welcome on board Hequn!
> >>>
> >>> Best,
> >>> Jincheng
> >>>
> >>>
> >>>
> >>> Zhijiang  于2020年4月19日周日 下午10:47写道：
> >>>
>  Congratulations, Hequn!
> 
>  Best,
>  Zhijiang
> 
> 
>  --
>  From:Yun Gao 
>  Send Time:2020 Apr. 19 (Sun.) 21:53
>  To:dev 
>  Subject:Re: [ANNOUNCE] New Apache Flink PMC Member - Hequn Chen
> 
>   Congratulations Hequn!
> 
>   Best,
>    Yun
> 
> 
>  --
>  From:Hequn Cheng 
>  Send Time:2020 Apr. 18 (Sat.) 12:48
>  To:dev 
>  Subject:Re: [ANNOUNCE] New Apache Flink PMC Member - Hequn Chen
> 
>  Many thanks for your support. Thank you!
> 
>  Best,
>  Hequn
> 
>  On Sat, Apr 18, 2020 at 1:27 AM Jacky Bai 
> >>> wrote:
> 
> > Congratulations！Hequn Chen.I hope to make so many contributions to
> >>> Flink
> > like you.
> >
> > Best
> > Bai Xu
> >
> > Congxian Qiu  于2020年4月17日周五 下午10:47写道：
> >
> >> Congratulations, Hequn!
> >>
> >> Best,
> >> Congxian
> >>
> >>
> >> Yu Li  于2020年4月17日周五 下午9:36写道：
> >>
> >>> Congratulations, Hequn!
> >>>
> >>> Best Regards,
> >>> Yu
> >>>
> >>>
> >>> On Fri, 17 Apr 2020 at 21:22, Kurt Young  wrote:
> >>>
>  Congratulations Hequn!
> 
>  Best,
>  Kurt
> 
> 
>  On Fri, Apr 17, 2020 at 8:57 PM Till Rohrmann <
>  trohrm...@apache.org>
>  wrote:
> 
> > Congratulations Hequn!
> >
> > Cheers,
> > Till
> >
> > On Fri, Apr 17, 2020 at 2:49 PM Shuo Cheng  
> >> wrote:
> >
> >> Congratulations, Hequn
> >>
> >> Best,
> >> Shuo
> >>
> >> On 4/17/20, hufeih...@mails.ucas.ac.cn <
> > hufeih...@mails.ucas.ac.cn
> >>>
> > wrote:
> >>> Congratulations , Hequn
> >>>
> >>> Best wish
> >>>
> >>>
> >>> hufeih...@mails.ucas.ac.cn
> >>> Congratulations, Hequn!
> >>>
> >>> Paul Lam  于2020年4月17日周五 下午3:02写道：
> >>>
>  Congrats Hequn! Thanks a lot for your contribution to the
> >>> community!
> 
>  Best,
>  Paul Lam
> 
>  Dian Fu  于2020年4月17日周五 下午2:58写道：
> 
> > Congratulations, Hequn!
> >
> >> 在 2020年4月17日，下午2:36，Becket Qin 
>  写道：
> >>
> >> Hi all,
> >>
> >> I am glad to announce that Hequn Chen has joined the
>  Flink
> >>> PMC.
> >>
> >> Hequn has contributed to Flink for years. He has
> >>> worked
>  on
>  several
> >> components including Table / SQL，PyFlink and Flink ML
> >>> Pipeline.
>  Besides,
> >> Hequn is also very active in the community since the
> >>> beginning.
> >>
> >> Congratulations, Hequn! Looking forward to your future
> >> contributions.
> >>
> >> Thanks,
> >>
> >> Jiangjie (Becket) Qin
> >> (On behalf of the Apache Flink PMC)
> >
> >
> 
> >>>
> >>>
> >>> --
> >>> Best Regards
> >>>
> >>> Jeff Zhang
> >>>
> >>
> >
> 
> >>>
> >>
> >
> 
> 
> 
> >>>
> >>
> >>
> >> --
> >> Best, Jingsong Lee
>
>

[jira] [Created] (FLINK-17263) Remove RepeatFamilyOperandTypeChecker in blink planner and replace it with calcite's CompositeOperandTypeChecker

[Terry Wang (Jira)]

Terry Wang created FLINK-17263:
--

 Summary: Remove RepeatFamilyOperandTypeChecker in blink planner 
and replace it  with calcite's CompositeOperandTypeChecker
 Key: FLINK-17263
 URL: https://issues.apache.org/jira/browse/FLINK-17263
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Affects Versions: 1.11.0
Reporter: Terry Wang


Remove RepeatFamilyOperandTypeChecker in blink planner and replace it  with 
calcite's CompositeOperandTypeChecker.
It seems that what CompositeOperandTypeChecker can do is a super set of 
RepeatFamilyOperandTypeChecker. To keep code easy to read, it's better to do 
such refactor.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Apache Flink PMC Member - Hequn Chen

[Yuan Mei]

Congrats!

On Fri, Apr 17, 2020 at 2:44 PM Becket Qin  wrote:

> Hi all,
>
> I am glad to announce that Hequn Chen has joined the Flink PMC.
>
> Hequn has contributed to Flink for years. He has worked on several
> components including Table / SQL，PyFlink and Flink ML Pipeline. Besides,
> Hequn is also very active in the community since the beginning.
>
> Congratulations, Hequn! Looking forward to your future contributions.
>
> Thanks,
>
> Jiangjie (Becket) Qin
> (On behalf of the Apache Flink PMC)
>

[jira] [Created] (FLINK-17264) taskmanager.sh could not work on Mac

[Yang Wang (Jira)]

Yang Wang created FLINK-17264:
-

 Summary: taskmanager.sh could not work on Mac
 Key: FLINK-17264
 URL: https://issues.apache.org/jira/browse/FLINK-17264
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Scripts
Reporter: Yang Wang
 Fix For: 1.11.0


Start a taskmanager on Mac via {{taskmanager.sh}} will get the following error.
{code:java}
wangyang-pc:build-target danrtsey.wy$ ./bin/taskmanager.sh start-foreground 
[ERROR] Unexpected result ( 1 lines): BASH_JAVA_UTILS_EXEC_RESULT:-Xmx536870902 
-Xms536870902 -XX:MaxDirectMemorySize=268435458 -XX:MaxMetaspaceSize=268435456 
[ERROR] extractExecutionParams only accepts exactly one line as the input 
[ERROR] Could not get JVM parameters properly.
{code}
 

The root cause is FLINK-17023 introduce the following change and it could not 
work as expected.
{code:java}
local num_lines=$(echo "$execution_config" | wc -l)
{code}
On linux environment, the output is "1". However, on Mac, it is "      1". 
Maybe we need to add "tr -d '[:space:]' after wc".

 

cc [~TsReaper] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Creating a new repo to host Flink benchmarks

[Tzu-Li (Gordon) Tai]

Repository has been created:
https://github.com/apache/flink-benchmarks

On Sat, Apr 18, 2020 at 5:10 PM Tzu-Li (Gordon) Tai 
wrote:

> There doesn't seem to be any objections in not considering the
> flink-benchmarks migration as a significant change that requires a vote.
>
> If no-one objects in the next days, I'll assume lazy consensus and create
> the new repo the following Monday (April 20).
>
> On Tue, Apr 14, 2020 at 5:13 PM Tzu-Li (Gordon) Tai 
> wrote:
>
>> +1 on creating a new repo, and migrating the flink-benchmarks there.
>>
>> Regarding Chesnay's concern whether this requires a vote:
>> While the flink-benchmarks is indeed an existing external codebase, I'm
>> not entirely sure we do need a vote here.
>> The bylaws state that:
>> "Adoption of large existing external codebase. This refers to
>> contributions big enough that potentially change the shape and direction of
>> the project with massive restructuring and future maintenance commitment."
>>
>> The Flink benchmarks should not be a significant change to the shape /
>> direction of the project.
>> What do others think?
>>
>> On Tue, Apr 14, 2020 at 4:53 PM Chesnay Schepler 
>> wrote:
>>
>>> Isn't this technically the adoption of a new codebase?
>>>
>>> On 14/04/2020 10:06, Yun Tang wrote:
>>> > Hi Flink devs
>>> >
>>> > Thanks for the feedbacks! Overall, everyone who replied is positive
>>> about creating such a separate repo to host the flink benchmarks.
>>> >
>>> > Since such an action does not necessarily need a vote (as defined by
>>> the project bylaws), I'll proceed to ask PMC's help to create the repo
>>> under Apache project.
>>> >
>>> > Best
>>> > Yun Tang
>>> >
>>> > 
>>> > From: Yu Li 
>>> > Sent: Friday, April 10, 2020 21:03
>>> > To: dev 
>>> > Subject: Re: [DISCUSS] Creating a new repo to host Flink benchmarks
>>> >
>>> > +1 for the proposal, thanks for driving this Yun.
>>> >
>>> > Best Regards,
>>> > Yu
>>> >
>>> >
>>> > On Fri, 10 Apr 2020 at 11:25, Dian Fu  wrote:
>>> >
>>> >> +1 for this proposal.
>>> >>
>>> >>> 在 2020年4月10日，上午10:58，Zhijiang 
>>> 写道：
>>> >>>
>>> >>> +1 for the proposal.
>>> >>>
>>> >>> Best,
>>> >>> Zhijiang
>>> >>>
>>> >>>
>>> >>> --
>>> >>> From:Robert Metzger 
>>> >>> Send Time:2020 Apr. 10 (Fri.) 02:15
>>> >>> To:dev 
>>> >>> Subject:Re: [DISCUSS] Creating a new repo to host Flink benchmarks
>>> >>>
>>> >>> +1 on creating the repo.
>>> >>>
>>> >>>
>>> >>> On Thu, Apr 9, 2020 at 5:54 PM Till Rohrmann 
>>> >> wrote:
>>>  I think it is a good idea to make the benchmarks available to the
>>> >> community
>>>  via a repo under the Apache project and to make updating it part of
>>> the
>>>  release process. Hence +1 for the proposal.
>>> 
>>>  Cheers,
>>>  Till
>>> 
>>>  On Thu, Apr 9, 2020 at 4:01 PM Piotr Nowojski 
>>> >> wrote:
>>> > Hi Yun Tang,
>>> >
>>> > Thanks for proposing the idea. Since we can not include benchmarks
>>> in
>>> >> the
>>> > Flink repository what you are proposing is the second best option.
>>> >
>>> > +1 from my side for the proposal.
>>> >
>>> > I think benchmarks have proven their value to justify this.
>>> >
>>> > Piotrek
>>> >
>>> >> On 9 Apr 2020, at 08:56, Yun Tang  wrote:
>>> >>
>>> >> Hi Flink devs,
>>> >>
>>> >> As Flink develops rapidly with more and more features added, how
>>> to
>>> > ensure no performance regression existed has become more and more
>>> > important. And we would like to create a new repo under apache
>>> project
>>> >> to
>>> > host previous flink-benchmarks [1] repo, which is inspired when we
>>>  discuss
>>> > under FLINK-16850 [2]
>>> >> Some background context on flink-benchmarks, for those who are not
>>> > familiar with the project yet:
>>> >> - Current flink-benchmarks does not align with the Flink release,
>>> >> which
>>> > lead developers not easy to verify
>>> >>   performance at specific Flink version because current
>>>  flink-benchmarks
>>> > always depends on the latest interfaces.
>>> >> - Above problem could be solved well if we could ensure
>>>  flink-benchmarks
>>> > also create release branch when we
>>> >>   releasing Flink. However, current flink-benchmarks repo is
>>> hosted
>>> > under dataArtisans (the former name of
>>> >>   ververica) project, which is not involved in Flink release
>>> manual
>>>  [3].
>>> > We propose to promote this repo under
>>> >>   apache project so that release manager could have the right to
>>>  release
>>> > on flink-benchmarks.
>>> >> - The reason why we not involve flink-benchmarks into the
>>> apache/flink
>>> > repo is because it heavily depends on
>>> >>   JMH [4], which is under GPLv2 license.
>>> >>
>>> >> What do you think?
>>> >>
>>> >> Best,
>>> >> Yun Tang
>>> >>
>>> >

Re: [DISCUSS] Integration of training materials into Apache Flink

[Tzu-Li (Gordon) Tai]

The flink-training repository has been created:
https://github.com/apache/flink-training

On Mon, Apr 20, 2020 at 12:47 AM David Anderson  wrote:

> How about calling the component "Documentation / Training"?
>
>
> Sounds good. Thanks!
>
> *David Anderson* | Training Coordinator
>
> Follow us @VervericaData
> --
> Join Flink Forward - The Apache Flink Conference
> Stream Processing | Event Driven | Real Time
>
>
> On Sun, Apr 19, 2020 at 10:57 AM Robert Metzger 
> wrote:
>
>> @David: I'm happy to create a component in JIRA.
>>
>> How about calling the component "Documentation / Training"?
>>
>> On Sat, Apr 18, 2020 at 11:14 AM Tzu-Li (Gordon) Tai 
>> wrote:
>>
>>> I can help create the flink-training repository.
>>>
>>> Before I do that though, I'd like to wait a bit on [1] over the next few
>>> days.
>>>
>>> There's a similar case there, where we want to create a new repo to host
>>> existing codebases,
>>> but may not necessarily need a vote because the contribution is not a
>>> direction-shifting change to the project.
>>>
>>> If consensus passes there, I'll create the repo for flink-training as
>>> well.
>>>
>>> [1]
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Creating-a-new-repo-to-host-Flink-benchmarks-td40037.html
>>>
>>> On Sat, Apr 18, 2020 at 4:09 AM Seth Wiesman 
>>> wrote:
>>>
 I'm not able to create the repository. INFRA requests that new repos are
 created using https://selfserve.apache.org/ and it requires a PMC
 member.
 It looks like it should take someone 30 seconds.

 Seth

 On Fri, Apr 17, 2020 at 9:07 AM David Anderson 
 wrote:

 > @Robert, do you think we could create a new flink-training component
 in
 > Jira for tracking tickets related to the flink-training content?
 >
 > David
 >
 > On Thu, Apr 16, 2020 at 10:54 AM David Anderson 
 > wrote:
 >
 >> > I am happy to get the repo created for you.
 >>
 >> Thank you, @seth. I think we are also going to want a new
 flink-training
 >> component in Jira. Maybe you can help there too?
 >>
 >> > If we go with the documentation (vs flink.apache.org) do you
 think we
 >> > should remove any of the existing content? There is already a
 getting
 >> > started section with quickstarts and walkthroughs and a concepts
 >> section.
 >> > In particular, the concepts section today is not complete and
 almost
 >> every
 >> > page on master contains multiple TODOs.
 >>
 >> I'll look at this, and also coordinate with what Aljoscha is doing
 there.
 >> But yes, there is room for improvement in this part of the docs, so
 I'm
 >> expecting to be able to help with that.
 >>
 >> David
 >>
 >> On Wed, Apr 15, 2020 at 9:20 PM Seth Wiesman 
 wrote:
 >>
 >>> Hi David,
 >>>
 >>> I am happy to get the repo created for you.
 >>>
 >>> If we go with the documentation (vs flink.apache.org) do you think
 we
 >>> should remove any of the existing content? There is already a
 getting
 >>> started section with quickstarts and walkthroughs and a concepts
 section.
 >>> In particular, the concepts section today is not complete and almost
 >>> every
 >>> page on master contains multiple TODOs. I don't believe anyone is
 working
 >>> on these.  What do you think about replacing the current concepts
 section
 >>> with the training material? I just re-examined the training site
 and I
 >>> believe it covers the same material as concepts but better. Of
 course, we
 >>> would salvage anything worth keeping, like the glossary.
 >>>
 >>> Seth
 >>>
 >>> On Wed, Apr 15, 2020 at 2:02 PM David Anderson >>> >
 >>> wrote:
 >>>
 >>> > Thank you all for the very positive response to our proposal to
 >>> contribute
 >>> > the training materials that have been at training.ververica.com
 to the
 >>> > Apache Flink project. Now I’d like to begin the more detailed
 >>> discussion of
 >>> > how to go about this.
 >>> >
 >>> > In that earlier thread I mentioned that we were thinking of
 merging the
 >>> > markdown-based web pages into flink.apache.org, and to add the
 >>> exercises
 >>> > to
 >>> > flink-playgrounds. This was based on thinking that it would be
 >>> something of
 >>> > a maintenance headache to add the website content into the docs,
 where
 >>> it
 >>> > would have to be versioned.
 >>> >
 >>> > Since then, a better approach has been suggested:
 >>> >
 >>> > We already have quite a bit of “getting started” material in the
 docs:
 >>> Code
 >>> > Walkthroughs, Docker Playgrounds, Tutorials, and Examples. Having
 a
 >>> second
 >>> > location (namely flink.apache.org) where this kind of content
 could be
 >>> > found doesn’t seem ideal. So let

[jira] [Created] (FLINK-17265) uniform time and timeUnit use

[Jake.zhang (Jira)]

Jake.zhang created FLINK-17265:
--

 Summary: uniform time and timeUnit use
 Key: FLINK-17265
 URL: https://issues.apache.org/jira/browse/FLINK-17265
 Project: Flink
  Issue Type: Wish
  Components: API / Scala
Affects Versions: 1.10.0
Reporter: Jake.zhang


// set restart strategy
{code:java}
env.setRestartStrategy(RestartStrategies.failureRateRestart(3,
 org.apache.flink.api.common.time.Time.of(5, TimeUnit.MINUTES), 
org.apache.flink.api.common.time.Time.of(10, TimeUnit.SECONDS))){code}
// set window time 
{code:java}
// import org.apache.flink.streaming.api.windowing.time.Time
.timeWindow(Time.minutes(1)){code}
It is strongly recommanded to unify Time and TimeUnit

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17266) WorkerResourceSpec is not serializable

[Gary Yao (Jira)]

Gary Yao created FLINK-17266:


 Summary: WorkerResourceSpec is not serializable
 Key: FLINK-17266
 URL: https://issues.apache.org/jira/browse/FLINK-17266
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Mesos
Affects Versions: 1.11.0
Reporter: Gary Yao
 Fix For: 1.11.0


{{MesosResourceManager}} cannot acquire new resources due to 
{{WorkerResourceSpec}} not being serializable.

{code}
Caused by: java.lang.Exception: Could not open output stream for state backend
at 
org.apache.flink.runtime.zookeeper.filesystem.FileSystemStateStorageHelper.store(FileSystemStateStorageHelper.java:70)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.runtime.zookeeper.ZooKeeperStateHandleStore.addAndLock(ZooKeeperStateHandleStore.java:136)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.mesos.runtime.clusterframework.store.ZooKeeperMesosWorkerStore.putWorker(ZooKeeperMesosWorkerStore.java:216)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager.startNewWorker(MesosResourceManager.java:441)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
... 34 more
Caused by: java.io.NotSerializableException: 
org.apache.flink.runtime.resourcemanager.WorkerResourceSpec
at 
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) 
~[?:1.8.0_242]
at 
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) 
~[?:1.8.0_242]
at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) 
~[?:1.8.0_242]
at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) 
~[?:1.8.0_242]
at 
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) 
~[?:1.8.0_242]
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) 
~[?:1.8.0_242]
at 
org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:594)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.runtime.zookeeper.filesystem.FileSystemStateStorageHelper.store(FileSystemStateStorageHelper.java:62)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.runtime.zookeeper.ZooKeeperStateHandleStore.addAndLock(ZooKeeperStateHandleStore.java:136)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.mesos.runtime.clusterframework.store.ZooKeeperMesosWorkerStore.putWorker(ZooKeeperMesosWorkerStore.java:216)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager.startNewWorker(MesosResourceManager.java:441)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
... 34 more
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Apache Flink PMC Member - Hequn Chen

[wenlong.lwl]

Congratulations Hequn!

Bests,
Wenlong

On Mon, 20 Apr 2020 at 17:56, Yuan Mei  wrote:

> Congrats!
>
> On Fri, Apr 17, 2020 at 2:44 PM Becket Qin  wrote:
>
> > Hi all,
> >
> > I am glad to announce that Hequn Chen has joined the Flink PMC.
> >
> > Hequn has contributed to Flink for years. He has worked on several
> > components including Table / SQL，PyFlink and Flink ML Pipeline. Besides,
> > Hequn is also very active in the community since the beginning.
> >
> > Congratulations, Hequn! Looking forward to your future contributions.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> > (On behalf of the Apache Flink PMC)
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Hequn Chen

[Canbin Zheng]

Congratulations Hequn!

Regards,
Canbin Zheng

wenlong.lwl  于2020年4月20日周一 下午8:02写道：

> Congratulations Hequn!
>
> Bests,
> Wenlong
>
> On Mon, 20 Apr 2020 at 17:56, Yuan Mei  wrote:
>
> > Congrats!
> >
> > On Fri, Apr 17, 2020 at 2:44 PM Becket Qin  wrote:
> >
> > > Hi all,
> > >
> > > I am glad to announce that Hequn Chen has joined the Flink PMC.
> > >
> > > Hequn has contributed to Flink for years. He has worked on several
> > > components including Table / SQL，PyFlink and Flink ML Pipeline.
> Besides,
> > > Hequn is also very active in the community since the beginning.
> > >
> > > Congratulations, Hequn! Looking forward to your future contributions.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > > (On behalf of the Apache Flink PMC)
> > >
> >
>

[jira] [Created] (FLINK-17267) supports EXPLAIN statement in TableEnvironment#executeSql and Table#explain api

[godfrey he (Jira)]

godfrey he created FLINK-17267:
--

 Summary: supports EXPLAIN statement in TableEnvironment#executeSql 
and Table#explain api
 Key: FLINK-17267
 URL: https://issues.apache.org/jira/browse/FLINK-17267
 Project: Flink
  Issue Type: Sub-task
Reporter: godfrey he
 Fix For: 1.11.0


[FLINK-16366|https://issues.apache.org/jira/browse/FLINK-16366] has introduced 
executeSql method in TableEnvironment, but EXPLAIN statement is not supported 
because [FLINK-17126|https://issues.apache.org/jira/browse/FLINK-17126] is not 
finished. This issue aims to support EXPLAIN statement after FLINK-17126 
finished, and introduce Table#explain api at the same time.





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17268) SourceReaderTestBase.testAddSplitToExistingFetcher deadlocks on Travis

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-17268:
-

 Summary: SourceReaderTestBase.testAddSplitToExistingFetcher 
deadlocks on Travis
 Key: FLINK-17268
 URL: https://issues.apache.org/jira/browse/FLINK-17268
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Common
Affects Versions: 1.11.0
Reporter: Till Rohrmann
 Fix For: 1.11.0


The {{SourceReaderTestBase.testAddSplitToExistingFetcher}} deadlocks on Travis 
with the following stack trace:

{code}
Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
2020-04-20 11:40:52
Full thread dump OpenJDK 64-Bit Server VM (25.242-b08 mixed mode):

"Attach Listener" #16 daemon prio=9 os_prio=0 tid=0x7f640046d000 nid=0x67b3 
waiting on condition [0x]
   java.lang.Thread.State: RUNNABLE

"SourceFetcher" #15 prio=5 os_prio=0 tid=0x7f6400abb000 nid=0x647a waiting 
on condition [0x7f63e6c4f000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x8fba81e0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at 
java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
at 
java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:104)
at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:85)
at 
org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:42)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

"process reaper" #11 daemon prio=10 os_prio=0 tid=0x7f6400774000 nid=0x6469 
waiting on condition [0x7f63e718e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x80054278> (a 
java.util.concurrent.SynchronousQueue$TransferStack)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at 
java.util.concurrent.ThreadPoolExecutor$WorknputStream.readInt(DataInputStream.java:387)
at 
org.apache.maven.surefire.booter.MasterProcessCommand.decode(MasterProcessCommand.java:115)
at 
org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:391)
at java.lang.Thread.run(Thread.java:748)

"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x7f6400203000 nid=0x645a 
runnable [0x]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x7f640020 
nid=0x6458 waiting on condition [0x]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x7f64001fd800 
nid=0x6457 waiting on condition [0x]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x7f64001fb800 
nid=0x6456 runnable [0x]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 
tid=0x7f64001fa000 nid=0x6454 waiting on condition [0x]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x7f64001c9800 nid=0x6450 in 
Object.wait() [0x7f63e9032000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x80088160> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x80088160> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00

[jira] [Created] (FLINK-17269) Translate new Tutorials Overview

[David Anderson (Jira)]

David Anderson created FLINK-17269:
--

 Summary: Translate new Tutorials Overview
 Key: FLINK-17269
 URL: https://issues.apache.org/jira/browse/FLINK-17269
 Project: Flink
  Issue Type: Improvement
  Components: chinese-translation
Reporter: David Anderson


The training materials being added to the documentation need to be translated 
to Chinese. This ticket is for translating 

tutorials/index.zh.md
concepts/index.zh.md




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Integration of training materials into Apache Flink

[Robert Metzger]

The Jira component has been created!


On Mon, Apr 20, 2020 at 12:04 PM Tzu-Li (Gordon) Tai 
wrote:

> The flink-training repository has been created:
> https://github.com/apache/flink-training
>
> On Mon, Apr 20, 2020 at 12:47 AM David Anderson 
> wrote:
>
>> How about calling the component "Documentation / Training"?
>>
>>
>> Sounds good. Thanks!
>>
>> *David Anderson* | Training Coordinator
>>
>> Follow us @VervericaData
>> --
>> Join Flink Forward - The Apache Flink Conference
>> Stream Processing | Event Driven | Real Time
>>
>>
>> On Sun, Apr 19, 2020 at 10:57 AM Robert Metzger 
>> wrote:
>>
>>> @David: I'm happy to create a component in JIRA.
>>>
>>> How about calling the component "Documentation / Training"?
>>>
>>> On Sat, Apr 18, 2020 at 11:14 AM Tzu-Li (Gordon) Tai <
>>> tzuli...@apache.org> wrote:
>>>
 I can help create the flink-training repository.

 Before I do that though, I'd like to wait a bit on [1] over the next
 few days.

 There's a similar case there, where we want to create a new repo to
 host existing codebases,
 but may not necessarily need a vote because the contribution is not a
 direction-shifting change to the project.

 If consensus passes there, I'll create the repo for flink-training as
 well.

 [1]
 http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Creating-a-new-repo-to-host-Flink-benchmarks-td40037.html

 On Sat, Apr 18, 2020 at 4:09 AM Seth Wiesman 
 wrote:

> I'm not able to create the repository. INFRA requests that new repos
> are
> created using https://selfserve.apache.org/ and it requires a PMC
> member.
> It looks like it should take someone 30 seconds.
>
> Seth
>
> On Fri, Apr 17, 2020 at 9:07 AM David Anderson 
> wrote:
>
> > @Robert, do you think we could create a new flink-training component
> in
> > Jira for tracking tickets related to the flink-training content?
> >
> > David
> >
> > On Thu, Apr 16, 2020 at 10:54 AM David Anderson  >
> > wrote:
> >
> >> > I am happy to get the repo created for you.
> >>
> >> Thank you, @seth. I think we are also going to want a new
> flink-training
> >> component in Jira. Maybe you can help there too?
> >>
> >> > If we go with the documentation (vs flink.apache.org) do you
> think we
> >> > should remove any of the existing content? There is already a
> getting
> >> > started section with quickstarts and walkthroughs and a concepts
> >> section.
> >> > In particular, the concepts section today is not complete and
> almost
> >> every
> >> > page on master contains multiple TODOs.
> >>
> >> I'll look at this, and also coordinate with what Aljoscha is doing
> there.
> >> But yes, there is room for improvement in this part of the docs, so
> I'm
> >> expecting to be able to help with that.
> >>
> >> David
> >>
> >> On Wed, Apr 15, 2020 at 9:20 PM Seth Wiesman 
> wrote:
> >>
> >>> Hi David,
> >>>
> >>> I am happy to get the repo created for you.
> >>>
> >>> If we go with the documentation (vs flink.apache.org) do you
> think we
> >>> should remove any of the existing content? There is already a
> getting
> >>> started section with quickstarts and walkthroughs and a concepts
> section.
> >>> In particular, the concepts section today is not complete and
> almost
> >>> every
> >>> page on master contains multiple TODOs. I don't believe anyone is
> working
> >>> on these.  What do you think about replacing the current concepts
> section
> >>> with the training material? I just re-examined the training site
> and I
> >>> believe it covers the same material as concepts but better. Of
> course, we
> >>> would salvage anything worth keeping, like the glossary.
> >>>
> >>> Seth
> >>>
> >>> On Wed, Apr 15, 2020 at 2:02 PM David Anderson <
> da...@ververica.com>
> >>> wrote:
> >>>
> >>> > Thank you all for the very positive response to our proposal to
> >>> contribute
> >>> > the training materials that have been at training.ververica.com
> to the
> >>> > Apache Flink project. Now I’d like to begin the more detailed
> >>> discussion of
> >>> > how to go about this.
> >>> >
> >>> > In that earlier thread I mentioned that we were thinking of
> merging the
> >>> > markdown-based web pages into flink.apache.org, and to add the
> >>> exercises
> >>> > to
> >>> > flink-playgrounds. This was based on thinking that it would be
> >>> something of
> >>> > a maintenance headache to add the website content into the docs,
> where
> >>> it
> >>> > would have to be versioned.
> >>> >
> >>> > Since then, a better approach has been suggested:
> >>> >
> >>> > We alre

Re: [ANNOUNCE] New Apache Flink PMC Member - Hequn Chen

[Robert Metzger]

Congratulations!

On Mon, Apr 20, 2020 at 2:05 PM Canbin Zheng  wrote:

> Congratulations Hequn!
>
> Regards,
> Canbin Zheng
>
> wenlong.lwl  于2020年4月20日周一 下午8:02写道：
>
> > Congratulations Hequn!
> >
> > Bests,
> > Wenlong
> >
> > On Mon, 20 Apr 2020 at 17:56, Yuan Mei  wrote:
> >
> > > Congrats!
> > >
> > > On Fri, Apr 17, 2020 at 2:44 PM Becket Qin 
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I am glad to announce that Hequn Chen has joined the Flink PMC.
> > > >
> > > > Hequn has contributed to Flink for years. He has worked on several
> > > > components including Table / SQL，PyFlink and Flink ML Pipeline.
> > Besides,
> > > > Hequn is also very active in the community since the beginning.
> > > >
> > > > Congratulations, Hequn! Looking forward to your future contributions.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > > (On behalf of the Apache Flink PMC)
> > > >
> > >
> >
>

[jira] [Created] (FLINK-17270) param has space in config.sh

[Yu Wang (Jira)]

Yu Wang created FLINK-17270:
---

 Summary: param has space in config.sh
 Key: FLINK-17270
 URL: https://issues.apache.org/jira/browse/FLINK-17270
 Project: Flink
  Issue Type: Bug
Reporter: Yu Wang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink-web] MarkSfik commented on a change in pull request #328: Add blog post: "Memory Management improvements with Apache Flink 1.10"

[GitBox]

MarkSfik commented on a change in pull request #328:
URL: https://github.com/apache/flink-web/pull/328#discussion_r411336028



##
File path: _posts/2020-04-17-memory-management-improvements-flink-1.10.md
##
@@ -0,0 +1,87 @@
+---
+layout: post
+title: "Memory Management improvements with Apache Flink 1.10"
+date: 2020-04-17T12:00:00.000Z
+authors:
+- andrey:
+  name: "Andrey Zagrebin"
+categories: news
+excerpt: This post discusses the recent changes to the memory model of the 
task managers and configuration options for your Flink applications in Flink 
1.10.

Review comment:
   ```suggestion
   excerpt: This post discusses the recent changes to the memory model of the 
Task Managers and configuration options for your Flink applications in Flink 
1.10.
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-web] MarkSfik commented on a change in pull request #328: Add blog post: "Memory Management improvements with Apache Flink 1.10"

[GitBox]

MarkSfik commented on a change in pull request #328:
URL: https://github.com/apache/flink-web/pull/328#discussion_r411336241



##
File path: _posts/2020-04-17-memory-management-improvements-flink-1.10.md
##
@@ -0,0 +1,87 @@
+---
+layout: post
+title: "Memory Management improvements with Apache Flink 1.10"
+date: 2020-04-17T12:00:00.000Z
+authors:
+- andrey:
+  name: "Andrey Zagrebin"
+categories: news
+excerpt: This post discusses the recent changes to the memory model of the 
task managers and configuration options for your Flink applications in Flink 
1.10.
+---
+
+Apache Flink 1.10 comes with significant changes to the memory model of the 
task managers and configuration options for your Flink applications. These 
recently-introduced changes make Flink more adaptable to all kinds of 
deployment environments (e.g. Kubernetes, Yarn, Mesos), providing strict 
control over its memory consumption. In this post, we describe Flink’s memory 
model, as it stands in Flink 1.10, how to set up and manage memory consumption 
of your Flink applications and the recent changes the community implemented in 
the latest Apache Flink release. 

Review comment:
   ```suggestion
   Apache Flink 1.10 comes with significant changes to the memory model of the 
Task Managers and configuration options for your Flink applications. These 
recently-introduced changes make Flink more adaptable to all kinds of 
deployment environments (e.g. Kubernetes, Yarn, Mesos), providing strict 
control over its memory consumption. In this post, we describe Flink’s memory 
model, as it stands in Flink 1.10, how to set up and manage memory consumption 
of your Flink applications and the recent changes the community implemented in 
the latest Apache Flink release. 
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-web] MarkSfik commented on a change in pull request #328: Add blog post: "Memory Management improvements with Apache Flink 1.10"

[GitBox]

MarkSfik commented on a change in pull request #328:
URL: https://github.com/apache/flink-web/pull/328#discussion_r411336465



##
File path: _posts/2020-04-17-memory-management-improvements-flink-1.10.md
##
@@ -0,0 +1,87 @@
+---
+layout: post
+title: "Memory Management improvements with Apache Flink 1.10"
+date: 2020-04-17T12:00:00.000Z
+authors:
+- andrey:
+  name: "Andrey Zagrebin"
+categories: news
+excerpt: This post discusses the recent changes to the memory model of the 
task managers and configuration options for your Flink applications in Flink 
1.10.
+---
+
+Apache Flink 1.10 comes with significant changes to the memory model of the 
task managers and configuration options for your Flink applications. These 
recently-introduced changes make Flink more adaptable to all kinds of 
deployment environments (e.g. Kubernetes, Yarn, Mesos), providing strict 
control over its memory consumption. In this post, we describe Flink’s memory 
model, as it stands in Flink 1.10, how to set up and manage memory consumption 
of your Flink applications and the recent changes the community implemented in 
the latest Apache Flink release. 
+
+## Introduction to Flink’s memory model
+
+Having a clear understanding of Apache Flink’s memory model allows you to 
manage resources for the various workloads more efficiently. The following 
diagram illustrates the main memory components in Flink:
+
+
+
+
+Flink: Total Process Memory
+
+
+
+The task manager process is a JVM process. On a high level, its memory 
consists of the *JVM Heap* and *Off-Heap* memory. These types of memory are 
consumed by Flink directly or by JVM for its specific purposes (i.e. metaspace 
etc). There are two major memory consumers within Flink: the user code of job 
operator tasks and the framework itself consuming memory for internal data 
structures, network buffers etc.

Review comment:
   ```suggestion
   The Task Manager process is a JVM process. On a high level, its memory 
consists of the *JVM Heap* and *Off-Heap* memory. These types of memory are 
consumed by Flink directly or by JVM for its specific purposes (i.e. metaspace 
etc). There are two major memory consumers within Flink: the user code of job 
operator tasks and the framework itself consuming memory for internal data 
structures, network buffers etc.
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-web] MarkSfik commented on a change in pull request #328: Add blog post: "Memory Management improvements with Apache Flink 1.10"

[GitBox]

MarkSfik commented on a change in pull request #328:
URL: https://github.com/apache/flink-web/pull/328#discussion_r411336895



##
File path: _posts/2020-04-17-memory-management-improvements-flink-1.10.md
##
@@ -0,0 +1,87 @@
+---
+layout: post
+title: "Memory Management improvements with Apache Flink 1.10"
+date: 2020-04-17T12:00:00.000Z
+authors:
+- andrey:
+  name: "Andrey Zagrebin"
+categories: news
+excerpt: This post discusses the recent changes to the memory model of the 
task managers and configuration options for your Flink applications in Flink 
1.10.
+---
+
+Apache Flink 1.10 comes with significant changes to the memory model of the 
task managers and configuration options for your Flink applications. These 
recently-introduced changes make Flink more adaptable to all kinds of 
deployment environments (e.g. Kubernetes, Yarn, Mesos), providing strict 
control over its memory consumption. In this post, we describe Flink’s memory 
model, as it stands in Flink 1.10, how to set up and manage memory consumption 
of your Flink applications and the recent changes the community implemented in 
the latest Apache Flink release. 
+
+## Introduction to Flink’s memory model
+
+Having a clear understanding of Apache Flink’s memory model allows you to 
manage resources for the various workloads more efficiently. The following 
diagram illustrates the main memory components in Flink:
+
+
+
+
+Flink: Total Process Memory
+
+
+
+The task manager process is a JVM process. On a high level, its memory 
consists of the *JVM Heap* and *Off-Heap* memory. These types of memory are 
consumed by Flink directly or by JVM for its specific purposes (i.e. metaspace 
etc). There are two major memory consumers within Flink: the user code of job 
operator tasks and the framework itself consuming memory for internal data 
structures, network buffers etc.
+
+**Please note that** the user code has direct access to all memory types: *JVM 
Heap, Direct* and *Native memory*. Therefore, Flink cannot really control its 
allocation and usage. There are however two types of Off-Heap memory which are 
consumed by tasks and controlled explicitly by Flink:
+
+- *Managed Off-Heap Memory*
+- *Network Buffers*
+
+The latter is part of the *JVM Direct Memory*, allocated for user record data 
exchange between operator tasks.
+
+## How to set up Flink memory
+
+With the latest release of Flink 1.10 and in order to provide better user 
experience, the framework comes with both high-level and fine-grained tuning of 
memory components. There are essentially three alternatives to setting up 
memory in task managers.

Review comment:
   ```suggestion
   With the latest release of Flink 1.10 and in order to provide better user 
experience, the framework comes with both high-level and fine-grained tuning of 
memory components. There are essentially three alternatives to setting up 
memory in Task Managers.
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-web] MarkSfik commented on a change in pull request #328: Add blog post: "Memory Management improvements with Apache Flink 1.10"

[GitBox]

MarkSfik commented on a change in pull request #328:
URL: https://github.com/apache/flink-web/pull/328#discussion_r411337034



##
File path: _posts/2020-04-17-memory-management-improvements-flink-1.10.md
##
@@ -0,0 +1,87 @@
+---
+layout: post
+title: "Memory Management improvements with Apache Flink 1.10"
+date: 2020-04-17T12:00:00.000Z
+authors:
+- andrey:
+  name: "Andrey Zagrebin"
+categories: news
+excerpt: This post discusses the recent changes to the memory model of the 
task managers and configuration options for your Flink applications in Flink 
1.10.
+---
+
+Apache Flink 1.10 comes with significant changes to the memory model of the 
task managers and configuration options for your Flink applications. These 
recently-introduced changes make Flink more adaptable to all kinds of 
deployment environments (e.g. Kubernetes, Yarn, Mesos), providing strict 
control over its memory consumption. In this post, we describe Flink’s memory 
model, as it stands in Flink 1.10, how to set up and manage memory consumption 
of your Flink applications and the recent changes the community implemented in 
the latest Apache Flink release. 
+
+## Introduction to Flink’s memory model
+
+Having a clear understanding of Apache Flink’s memory model allows you to 
manage resources for the various workloads more efficiently. The following 
diagram illustrates the main memory components in Flink:
+
+
+
+
+Flink: Total Process Memory
+
+
+
+The task manager process is a JVM process. On a high level, its memory 
consists of the *JVM Heap* and *Off-Heap* memory. These types of memory are 
consumed by Flink directly or by JVM for its specific purposes (i.e. metaspace 
etc). There are two major memory consumers within Flink: the user code of job 
operator tasks and the framework itself consuming memory for internal data 
structures, network buffers etc.
+
+**Please note that** the user code has direct access to all memory types: *JVM 
Heap, Direct* and *Native memory*. Therefore, Flink cannot really control its 
allocation and usage. There are however two types of Off-Heap memory which are 
consumed by tasks and controlled explicitly by Flink:
+
+- *Managed Off-Heap Memory*
+- *Network Buffers*
+
+The latter is part of the *JVM Direct Memory*, allocated for user record data 
exchange between operator tasks.
+
+## How to set up Flink memory
+
+With the latest release of Flink 1.10 and in order to provide better user 
experience, the framework comes with both high-level and fine-grained tuning of 
memory components. There are essentially three alternatives to setting up 
memory in task managers.
+
+The first two — and simplest — alternatives are configuring one of the two 
following options for total memory available for the task manager:

Review comment:
   ```suggestion
   The first two — and simplest — alternatives are configuring one of the two 
following options for total memory available for the Task Manager:
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-web] MarkSfik commented on a change in pull request #328: Add blog post: "Memory Management improvements with Apache Flink 1.10"

[GitBox]

MarkSfik commented on a change in pull request #328:
URL: https://github.com/apache/flink-web/pull/328#discussion_r411337742



##
File path: _posts/2020-04-17-memory-management-improvements-flink-1.10.md
##
@@ -0,0 +1,87 @@
+---
+layout: post
+title: "Memory Management improvements with Apache Flink 1.10"
+date: 2020-04-17T12:00:00.000Z
+authors:
+- andrey:
+  name: "Andrey Zagrebin"
+categories: news
+excerpt: This post discusses the recent changes to the memory model of the 
task managers and configuration options for your Flink applications in Flink 
1.10.
+---
+
+Apache Flink 1.10 comes with significant changes to the memory model of the 
task managers and configuration options for your Flink applications. These 
recently-introduced changes make Flink more adaptable to all kinds of 
deployment environments (e.g. Kubernetes, Yarn, Mesos), providing strict 
control over its memory consumption. In this post, we describe Flink’s memory 
model, as it stands in Flink 1.10, how to set up and manage memory consumption 
of your Flink applications and the recent changes the community implemented in 
the latest Apache Flink release. 
+
+## Introduction to Flink’s memory model
+
+Having a clear understanding of Apache Flink’s memory model allows you to 
manage resources for the various workloads more efficiently. The following 
diagram illustrates the main memory components in Flink:
+
+
+
+
+Flink: Total Process Memory
+
+
+
+The task manager process is a JVM process. On a high level, its memory 
consists of the *JVM Heap* and *Off-Heap* memory. These types of memory are 
consumed by Flink directly or by JVM for its specific purposes (i.e. metaspace 
etc). There are two major memory consumers within Flink: the user code of job 
operator tasks and the framework itself consuming memory for internal data 
structures, network buffers etc.
+
+**Please note that** the user code has direct access to all memory types: *JVM 
Heap, Direct* and *Native memory*. Therefore, Flink cannot really control its 
allocation and usage. There are however two types of Off-Heap memory which are 
consumed by tasks and controlled explicitly by Flink:
+
+- *Managed Off-Heap Memory*
+- *Network Buffers*
+
+The latter is part of the *JVM Direct Memory*, allocated for user record data 
exchange between operator tasks.
+
+## How to set up Flink memory
+
+With the latest release of Flink 1.10 and in order to provide better user 
experience, the framework comes with both high-level and fine-grained tuning of 
memory components. There are essentially three alternatives to setting up 
memory in task managers.
+
+The first two — and simplest — alternatives are configuring one of the two 
following options for total memory available for the task manager:
+
+- *Total Process Memory*: memory consumed by the Flink application and by the 
JVM to run the process.
+- *Total Flink Memory*: only memory consumed by the Flink application
+
+It is advisable to configure the *Total Flink Memory* for standalone 
deployments where explicitly declaring how much memory is given to Flink is a 
common practice, while the outer *JVM overhead* is of little interest. For the 
cases of deploying Flink in containerized environments (such as 
[Kubernetes](https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/kubernetes.html),
 
[Yarn](https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html)
 or 
[Mesos](https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/mesos.html)),
 the *Total Process Memory* option is recommended instead, because it becomes 
the size for the total memory of the requested container.
+
+If you want more fine-grained control over the size of *JVM Heap* and 
*Managed* Off-Heap, there is also the second alternative to configure both 
*[Task 
Heap](https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_setup.html#task-operator-heap-memory)*
 and *[Managed 
Memory](https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_setup.html#managed-memory)*.
 This alternative gives a clear separation between the heap memory and any 
other memory types.
+
+In line with the community’s efforts to [unify batch and stream 
processing](https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html),
 this model works universally for both scenarios. It allows sharing the *JVM 
Heap* memory between the user code of operator tasks in any workload and the 
heap state backend in stream processing scenarios. The *Managed Off-Heap 
Memory* can be used for batch spilling and for the RocksDB state backend in 
streaming.
+
+The remaining memory components are automatically adjusted either based on 
their default values or additionally configured parameters. Flink also checks 
the overall consistency. You can find more information about the different 
memory components in the corresponding 
[documentation](https://ci.apache.org/proj

[GitHub] [flink-web] MarkSfik commented on a change in pull request #328: Add blog post: "Memory Management improvements with Apache Flink 1.10"

[GitBox]

MarkSfik commented on a change in pull request #328:
URL: https://github.com/apache/flink-web/pull/328#discussion_r411337034



##
File path: _posts/2020-04-17-memory-management-improvements-flink-1.10.md
##
@@ -0,0 +1,87 @@
+---
+layout: post
+title: "Memory Management improvements with Apache Flink 1.10"
+date: 2020-04-17T12:00:00.000Z
+authors:
+- andrey:
+  name: "Andrey Zagrebin"
+categories: news
+excerpt: This post discusses the recent changes to the memory model of the 
task managers and configuration options for your Flink applications in Flink 
1.10.
+---
+
+Apache Flink 1.10 comes with significant changes to the memory model of the 
task managers and configuration options for your Flink applications. These 
recently-introduced changes make Flink more adaptable to all kinds of 
deployment environments (e.g. Kubernetes, Yarn, Mesos), providing strict 
control over its memory consumption. In this post, we describe Flink’s memory 
model, as it stands in Flink 1.10, how to set up and manage memory consumption 
of your Flink applications and the recent changes the community implemented in 
the latest Apache Flink release. 
+
+## Introduction to Flink’s memory model
+
+Having a clear understanding of Apache Flink’s memory model allows you to 
manage resources for the various workloads more efficiently. The following 
diagram illustrates the main memory components in Flink:
+
+
+
+
+Flink: Total Process Memory
+
+
+
+The task manager process is a JVM process. On a high level, its memory 
consists of the *JVM Heap* and *Off-Heap* memory. These types of memory are 
consumed by Flink directly or by JVM for its specific purposes (i.e. metaspace 
etc). There are two major memory consumers within Flink: the user code of job 
operator tasks and the framework itself consuming memory for internal data 
structures, network buffers etc.
+
+**Please note that** the user code has direct access to all memory types: *JVM 
Heap, Direct* and *Native memory*. Therefore, Flink cannot really control its 
allocation and usage. There are however two types of Off-Heap memory which are 
consumed by tasks and controlled explicitly by Flink:
+
+- *Managed Off-Heap Memory*
+- *Network Buffers*
+
+The latter is part of the *JVM Direct Memory*, allocated for user record data 
exchange between operator tasks.
+
+## How to set up Flink memory
+
+With the latest release of Flink 1.10 and in order to provide better user 
experience, the framework comes with both high-level and fine-grained tuning of 
memory components. There are essentially three alternatives to setting up 
memory in task managers.
+
+The first two — and simplest — alternatives are configuring one of the two 
following options for total memory available for the task manager:

Review comment:
   ```suggestion
   The first two — and simplest — alternatives are configuring one of the two 
following options for total memory available for the JVM process of the Task 
Manager:
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-web] MarkSfik commented on a change in pull request #328: Add blog post: "Memory Management improvements with Apache Flink 1.10"

[GitBox]

MarkSfik commented on a change in pull request #328:
URL: https://github.com/apache/flink-web/pull/328#discussion_r411339534



##
File path: _posts/2020-04-17-memory-management-improvements-flink-1.10.md
##
@@ -0,0 +1,87 @@
+---
+layout: post
+title: "Memory Management improvements with Apache Flink 1.10"
+date: 2020-04-17T12:00:00.000Z
+authors:
+- andrey:
+  name: "Andrey Zagrebin"
+categories: news
+excerpt: This post discusses the recent changes to the memory model of the 
task managers and configuration options for your Flink applications in Flink 
1.10.
+---
+
+Apache Flink 1.10 comes with significant changes to the memory model of the 
task managers and configuration options for your Flink applications. These 
recently-introduced changes make Flink more adaptable to all kinds of 
deployment environments (e.g. Kubernetes, Yarn, Mesos), providing strict 
control over its memory consumption. In this post, we describe Flink’s memory 
model, as it stands in Flink 1.10, how to set up and manage memory consumption 
of your Flink applications and the recent changes the community implemented in 
the latest Apache Flink release. 
+
+## Introduction to Flink’s memory model
+
+Having a clear understanding of Apache Flink’s memory model allows you to 
manage resources for the various workloads more efficiently. The following 
diagram illustrates the main memory components in Flink:
+
+
+
+
+Flink: Total Process Memory
+
+
+
+The task manager process is a JVM process. On a high level, its memory 
consists of the *JVM Heap* and *Off-Heap* memory. These types of memory are 
consumed by Flink directly or by JVM for its specific purposes (i.e. metaspace 
etc). There are two major memory consumers within Flink: the user code of job 
operator tasks and the framework itself consuming memory for internal data 
structures, network buffers etc.
+
+**Please note that** the user code has direct access to all memory types: *JVM 
Heap, Direct* and *Native memory*. Therefore, Flink cannot really control its 
allocation and usage. There are however two types of Off-Heap memory which are 
consumed by tasks and controlled explicitly by Flink:
+
+- *Managed Off-Heap Memory*
+- *Network Buffers*
+
+The latter is part of the *JVM Direct Memory*, allocated for user record data 
exchange between operator tasks.
+
+## How to set up Flink memory
+
+With the latest release of Flink 1.10 and in order to provide better user 
experience, the framework comes with both high-level and fine-grained tuning of 
memory components. There are essentially three alternatives to setting up 
memory in task managers.
+
+The first two — and simplest — alternatives are configuring one of the two 
following options for total memory available for the task manager:
+
+- *Total Process Memory*: memory consumed by the Flink application and by the 
JVM to run the process.
+- *Total Flink Memory*: only memory consumed by the Flink application

Review comment:
   ```suggestion
   - *Total Flink Memory*: only memory consumed only by the Flink Java 
application, including user code but excluding memory allocated by JVM to run it
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-web] MarkSfik commented on a change in pull request #328: Add blog post: "Memory Management improvements with Apache Flink 1.10"

[GitBox]

MarkSfik commented on a change in pull request #328:
URL: https://github.com/apache/flink-web/pull/328#discussion_r409549023



##
File path: _posts/2020-04-17-memory-management-improvements-flink-1.10.md
##
@@ -0,0 +1,87 @@
+---
+layout: post
+title: "Memory Management improvements with Apache Flink 1.10"
+date: 2020-04-17T12:00:00.000Z
+authors:
+- andrey:
+  name: "Andrey Zagrebin"
+categories: news
+excerpt: This post discusses the recent changes to the memory model of the 
task managers and configuration options for your Flink applications in Flink 
1.10.
+---
+
+Apache Flink 1.10 comes with significant changes to the memory model of the 
task managers and configuration options for your Flink applications. These 
recently-introduced changes make Flink more adaptable to all kinds of 
deployment environments (e.g. Kubernetes, Yarn, Mesos), providing strict 
control over its memory consumption. In this post, we describe Flink’s memory 
model, as it stands in Flink 1.10, how to set up and manage memory consumption 
of your Flink applications and the recent changes the community implemented in 
the latest Apache Flink release. 
+
+## Introduction to Flink’s memory model
+
+Having a clear understanding of Apache Flink’s memory model allows you to 
manage resources for the various workloads more efficiently. The following 
diagram illustrates the main memory components in Flink:
+
+
+
+
+Flink: Total Process Memory
+
+
+
+The task manager process is a JVM process. On a high level, its memory 
consists of the *JVM Heap* and *Off-Heap* memory. These types of memory are 
consumed by Flink directly or by JVM for its specific purposes (i.e. metaspace 
etc). There are two major memory consumers within Flink: the user code of job 
operator tasks and the framework itself consuming memory for internal data 
structures, network buffers etc.
+
+**Please note that** the user code has direct access to all memory types: *JVM 
Heap, Direct* and *Native memory*. Therefore, Flink cannot really control its 
allocation and usage. There are however two types of Off-Heap memory which are 
consumed by tasks and controlled explicitly by Flink:
+
+- *Managed Off-Heap Memory*
+- *Network Buffers*
+
+The latter is part of the *JVM Direct Memory*, allocated for user record data 
exchange between operator tasks.
+
+## How to set up Flink memory
+
+With the latest release of Flink 1.10 and in order to provide better user 
experience, the framework comes with both high-level and fine-grained tuning of 
memory components. There are essentially three alternatives to setting up 
memory in task managers.
+
+The first two — and simplest — alternatives are configuring one of the two 
following options for total memory available for the task manager:
+
+- *Total Process Memory*: memory consumed by the Flink application and by the 
JVM to run the process.

Review comment:
   ```suggestion
   - *Total Process Memory*: total memory consumed by the Flink Java 
application (including user code) and by the JVM to run the whole process.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-web] MarkSfik commented on a change in pull request #328: Add blog post: "Memory Management improvements with Apache Flink 1.10"

[GitBox]

MarkSfik commented on a change in pull request #328:
URL: https://github.com/apache/flink-web/pull/328#discussion_r411340267



##
File path: _posts/2020-04-17-memory-management-improvements-flink-1.10.md
##
@@ -0,0 +1,87 @@
+---
+layout: post
+title: "Memory Management improvements with Apache Flink 1.10"
+date: 2020-04-17T12:00:00.000Z
+authors:
+- andrey:
+  name: "Andrey Zagrebin"
+categories: news
+excerpt: This post discusses the recent changes to the memory model of the 
task managers and configuration options for your Flink applications in Flink 
1.10.
+---
+
+Apache Flink 1.10 comes with significant changes to the memory model of the 
task managers and configuration options for your Flink applications. These 
recently-introduced changes make Flink more adaptable to all kinds of 
deployment environments (e.g. Kubernetes, Yarn, Mesos), providing strict 
control over its memory consumption. In this post, we describe Flink’s memory 
model, as it stands in Flink 1.10, how to set up and manage memory consumption 
of your Flink applications and the recent changes the community implemented in 
the latest Apache Flink release. 
+
+## Introduction to Flink’s memory model
+
+Having a clear understanding of Apache Flink’s memory model allows you to 
manage resources for the various workloads more efficiently. The following 
diagram illustrates the main memory components in Flink:
+
+
+
+
+Flink: Total Process Memory
+
+
+
+The task manager process is a JVM process. On a high level, its memory 
consists of the *JVM Heap* and *Off-Heap* memory. These types of memory are 
consumed by Flink directly or by JVM for its specific purposes (i.e. metaspace 
etc). There are two major memory consumers within Flink: the user code of job 
operator tasks and the framework itself consuming memory for internal data 
structures, network buffers etc.
+
+**Please note that** the user code has direct access to all memory types: *JVM 
Heap, Direct* and *Native memory*. Therefore, Flink cannot really control its 
allocation and usage. There are however two types of Off-Heap memory which are 
consumed by tasks and controlled explicitly by Flink:
+
+- *Managed Off-Heap Memory*
+- *Network Buffers*
+
+The latter is part of the *JVM Direct Memory*, allocated for user record data 
exchange between operator tasks.
+
+## How to set up Flink memory
+
+With the latest release of Flink 1.10 and in order to provide better user 
experience, the framework comes with both high-level and fine-grained tuning of 
memory components. There are essentially three alternatives to setting up 
memory in task managers.
+
+The first two — and simplest — alternatives are configuring one of the two 
following options for total memory available for the task manager:
+
+- *Total Process Memory*: memory consumed by the Flink application and by the 
JVM to run the process.
+- *Total Flink Memory*: only memory consumed by the Flink application
+
+It is advisable to configure the *Total Flink Memory* for standalone 
deployments where explicitly declaring how much memory is given to Flink is a 
common practice, while the outer *JVM overhead* is of little interest. For the 
cases of deploying Flink in containerized environments (such as 
[Kubernetes](https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/kubernetes.html),
 
[Yarn](https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html)
 or 
[Mesos](https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/mesos.html)),
 the *Total Process Memory* option is recommended instead, because it becomes 
the size for the total memory of the requested container.

Review comment:
   ```suggestion
   It is advisable to configure the *Total Flink Memory* for standalone 
deployments where explicitly declaring how much memory is given to Flink is a 
common practice, while the outer *JVM overhead* is of little interest. For the 
cases of deploying Flink in containerized environments (such as 
[Kubernetes](https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/kubernetes.html),
 
[Yarn](https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html)
 or 
[Mesos](https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/mesos.html)),
 the *Total Process Memory* option is recommended instead, because it becomes 
the size for the total memory of the requested container. Containerized 
environments usually strictly enforce this memory limit. 
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (FLINK-17271) Translate new DataStream API tutorial

[David Anderson (Jira)]

David Anderson created FLINK-17271:
--

 Summary: Translate new DataStream API tutorial
 Key: FLINK-17271
 URL: https://issues.apache.org/jira/browse/FLINK-17271
 Project: Flink
  Issue Type: Improvement
  Components: chinese-translation
Reporter: David Anderson


tutorials/datastream_api.md needs to be translated. The zh file doesn't exist 
yet.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17272) param has space in config.sh

[Yu Wang (Jira)]

Yu Wang created FLINK-17272:
---

 Summary: param has space in config.sh
 Key: FLINK-17272
 URL: https://issues.apache.org/jira/browse/FLINK-17272
 Project: Flink
  Issue Type: Bug
Reporter: Yu Wang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17273) Fix not calling ResourceManager#closeTaskManagerConnection in KubernetesResourceManager in case of registered TaskExecutor failure

[Canbin Zheng (Jira)]

Canbin Zheng created FLINK-17273:


 Summary: Fix not calling 
ResourceManager#closeTaskManagerConnection in KubernetesResourceManager in case 
of registered TaskExecutor failure
 Key: FLINK-17273
 URL: https://issues.apache.org/jira/browse/FLINK-17273
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Kubernetes, Runtime / Coordination
Affects Versions: 1.10.0, 1.10.1
Reporter: Canbin Zheng
 Fix For: 1.11.0


At the moment, the {{KubernetesResourceManager}} does not call the method of 
{{ResourceManager#closeTaskManagerConnection}} once it detects that a currently 
registered task executor has failed. This ticket propoeses to fix this problem.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17274) Maven: Premature end of Content-Length delimited message body

[Robert Metzger (Jira)]

Robert Metzger created FLINK-17274:
--

 Summary: Maven: Premature end of Content-Length delimited message 
body
 Key: FLINK-17274
 URL: https://issues.apache.org/jira/browse/FLINK-17274
 Project: Flink
  Issue Type: Bug
  Components: Build System / Azure Pipelines
Affects Versions: 1.11.0
Reporter: Robert Metzger


CI: 
https://dev.azure.com/rmetzger/Flink/_build/results?buildId=7786&view=logs&j=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5&t=54421a62-0c80-5aad-3319-094ff69180bb

{code}
[ERROR] Failed to execute goal on project flink-connector-elasticsearch7_2.11: 
Could not resolve dependencies for project 
org.apache.flink:flink-connector-elasticsearch7_2.11:jar:1.11-SNAPSHOT: Could 
not transfer artifact org.apache.lucene:lucene-sandbox:jar:8.3.0 from/to 
alicloud-mvn-mirror 
(http://mavenmirror.alicloud.dak8s.net:/repository/maven-central/): GET 
request of: org/apache/lucene/lucene-sandbox/8.3.0/lucene-sandbox-8.3.0.jar 
from alicloud-mvn-mirror failed: Premature end of Content-Length delimited 
message body (expected: 289920; received: 239832 -> [Help 1]
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17275) Add core training exercises

[Nico Kruber (Jira)]

Nico Kruber created FLINK-17275:
---

 Summary: Add core training exercises
 Key: FLINK-17275
 URL: https://issues.apache.org/jira/browse/FLINK-17275
 Project: Flink
  Issue Type: New Feature
  Components: Training Excercises
Affects Versions: 1.11.0
Reporter: Nico Kruber
Assignee: Nico Kruber
 Fix For: 1.10.1, 1.11.0


Port the core training exercises, their descriptions, solutions, and tests from 
https://github.com/ververica/flink-training-exercises to Apache Flink.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17276) Add checkstyle to training exercises

[Nico Kruber (Jira)]

Nico Kruber created FLINK-17276:
---

 Summary: Add checkstyle to training exercises
 Key: FLINK-17276
 URL: https://issues.apache.org/jira/browse/FLINK-17276
 Project: Flink
  Issue Type: Improvement
  Components: Training Excercises
Affects Versions: 1.10.0, 1.11.0
Reporter: Nico Kruber
Assignee: Nico Kruber
 Fix For: 1.10.1, 1.11.0


Port Flink's checkstyle to the training exercises and adapt the code 
accordingly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17277) Apply IntelliJ recommendations to training exercises

[Nico Kruber (Jira)]

Nico Kruber created FLINK-17277:
---

 Summary: Apply IntelliJ recommendations to training exercises
 Key: FLINK-17277
 URL: https://issues.apache.org/jira/browse/FLINK-17277
 Project: Flink
  Issue Type: Improvement
  Components: Training Excercises
Reporter: Nico Kruber
Assignee: Nico Kruber
 Fix For: 1.10.1, 1.11.0


IntelliJ has a few recommendations on the original code of the training 
exercises. These should be addressed to serve as good reference code.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17278) Add Travis to the training exercises

[Nico Kruber (Jira)]

Nico Kruber created FLINK-17278:
---

 Summary: Add Travis to the training exercises
 Key: FLINK-17278
 URL: https://issues.apache.org/jira/browse/FLINK-17278
 Project: Flink
  Issue Type: Improvement
  Components: Training Excercises
Reporter: Nico Kruber
Assignee: Nico Kruber
 Fix For: 1.10.1, 1.11.0


This will run all the tests and verify code quality.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17279) Use gradle build scans

[Nico Kruber (Jira)]

Nico Kruber created FLINK-17279:
---

 Summary: Use gradle build scans
 Key: FLINK-17279
 URL: https://issues.apache.org/jira/browse/FLINK-17279
 Project: Flink
  Issue Type: Improvement
  Components: Training Excercises
Reporter: Nico Kruber
Assignee: Nico Kruber
 Fix For: 1.10.1, 1.11.0


Gradle build scans [1] add quick analysis into what happened if a CI build 
failed. It would upload a report with detailed info to [1].

See this for an example: https://gradle.com/s/g3tdhu47lntoc

[1] https://scans.gradle.com/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] FLIP-126: Unify (and separate) Watermark Assigners

[Aljoscha Krettek]

Hi Everyone!

We would like to start a discussion on "FLIP-126: Unify (and separate) 
Watermark Assigners" [1]. This work was started by Stephan in an 
experimental branch. I expanded on that work to provide a PoC for the 
changes proposed in this FLIP: [2].


Currently, we have two different flavours of Watermark 
Assigners: AssignerWithPunctuatedWatermarks 
and AssignerWithPeriodicWatermarks. Both of them extend 
from TimestampAssigner. This means that sources that want to support 
watermark assignment/extraction in the source need to support two 
separate interfaces, we have two operator implementations for the 
different flavours. Also, this makes features such as generic support 
for idleness detection more complicated to implemented because we again 
have to support two types of watermark assigners.


In this FLIP we propose two things:

Unify the Watermark Assigners into one Interface WatermarkGenerator
Separate this new interface from the TimestampAssigner
The motivation for the first is to simplify future implementations and 
code duplication. The motivation for the second point is again code 
deduplication, most assigners currently have to extend from some base 
timestamp extractor or duplicate the extraction logic, or users have to 
override an abstract method of the watermark assigner to provide the 
timestamp extraction logic.


Additionally, we propose to add a generic wrapping WatermarkGenerator 
that provides idleness detection, i.e. it can mark a stream/partition as 
idle if no data arrives after a configured timeout.


The "unify and separate" part refers to the fact that we want to unify 
punctuated and periodic assigners but at the same time split the 
timestamp assigner from the watermark generator.


Please find more details in the FLIP [1]. Looking forward to
your feedback.

Best,
Aljoscha

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-126%3A+Unify+%28and+separate%29+Watermark+Assigners


[2] https://github.com/aljoscha/flink/tree/stephan-event-time

[jira] [Created] (FLINK-17280) Migrate existing flink-benchmarks under apache project

[Yun Tang (Jira)]

Yun Tang created FLINK-17280:


 Summary: Migrate existing flink-benchmarks under apache project
 Key: FLINK-17280
 URL: https://issues.apache.org/jira/browse/FLINK-17280
 Project: Flink
  Issue Type: Improvement
  Components: Benchmarks
Reporter: Yun Tang


Current existing benchmark is under 
https://github.com/dataArtisans/flink-benchmarks and we need to move them to 
https://github.com/apache/flink-benchmarks



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17281) Add existing flink-benchmarks code

[Yun Tang (Jira)]

Yun Tang created FLINK-17281:


 Summary: Add existing flink-benchmarks code
 Key: FLINK-17281
 URL: https://issues.apache.org/jira/browse/FLINK-17281
 Project: Flink
  Issue Type: Sub-task
Reporter: Yun Tang


This issue focus on commit all existing code from 
https://github.com/dataArtisans/flink-benchmarks to current 
https://github.com/apache/flink-benchmarks



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17282) Add CI build for new flink-benchmarks repo

[Yun Tang (Jira)]

Yun Tang created FLINK-17282:


 Summary: Add CI build for new flink-benchmarks repo
 Key: FLINK-17282
 URL: https://issues.apache.org/jira/browse/FLINK-17282
 Project: Flink
  Issue Type: Sub-task
Reporter: Yun Tang


Use travis or azure pipelines as the CI for this new repo.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Creating a new repo to host Flink benchmarks

[Yun Tang]

Thanks for Gordon's help!

We will move on and create 
FLINK-17280 [1] to track the 
progress.


[1] https://issues.apache.org/jira/browse/FLINK-17280

Best
Yun Tang

From: Tzu-Li (Gordon) Tai 
Sent: Monday, April 20, 2020 18:03
To: dev 
Cc: Yun Tang 
Subject: Re: [DISCUSS] Creating a new repo to host Flink benchmarks

Repository has been created:
https://github.com/apache/flink-benchmarks

On Sat, Apr 18, 2020 at 5:10 PM Tzu-Li (Gordon) Tai 
mailto:tzuli...@apache.org>> wrote:
There doesn't seem to be any objections in not considering the flink-benchmarks 
migration as a significant change that requires a vote.

If no-one objects in the next days, I'll assume lazy consensus and create the 
new repo the following Monday (April 20).

On Tue, Apr 14, 2020 at 5:13 PM Tzu-Li (Gordon) Tai 
mailto:tzuli...@apache.org>> wrote:
+1 on creating a new repo, and migrating the flink-benchmarks there.

Regarding Chesnay's concern whether this requires a vote:
While the flink-benchmarks is indeed an existing external codebase, I'm not 
entirely sure we do need a vote here.
The bylaws state that:
"Adoption of large existing external codebase. This refers to contributions big 
enough that potentially change the shape and direction of the project with 
massive restructuring and future maintenance commitment."

The Flink benchmarks should not be a significant change to the shape / 
direction of the project.
What do others think?

On Tue, Apr 14, 2020 at 4:53 PM Chesnay Schepler 
mailto:ches...@apache.org>> wrote:
Isn't this technically the adoption of a new codebase?

On 14/04/2020 10:06, Yun Tang wrote:
> Hi Flink devs
>
> Thanks for the feedbacks! Overall, everyone who replied is positive about 
> creating such a separate repo to host the flink benchmarks.
>
> Since such an action does not necessarily need a vote (as defined by the 
> project bylaws), I'll proceed to ask PMC's help to create the repo under 
> Apache project.
>
> Best
> Yun Tang
>
> 
> From: Yu Li mailto:car...@gmail.com>>
> Sent: Friday, April 10, 2020 21:03
> To: dev mailto:dev@flink.apache.org>>
> Subject: Re: [DISCUSS] Creating a new repo to host Flink benchmarks
>
> +1 for the proposal, thanks for driving this Yun.
>
> Best Regards,
> Yu
>
>
> On Fri, 10 Apr 2020 at 11:25, Dian Fu 
> mailto:dian0511...@gmail.com>> wrote:
>
>> +1 for this proposal.
>>
>>> 在 2020年4月10日，上午10:58，Zhijiang 
>>> mailto:wangzhijiang...@aliyun.com>.INVALID> 写道：
>>>
>>> +1 for the proposal.
>>>
>>> Best,
>>> Zhijiang
>>>
>>>
>>> --
>>> From:Robert Metzger mailto:rmetz...@apache.org>>
>>> Send Time:2020 Apr. 10 (Fri.) 02:15
>>> To:dev mailto:dev@flink.apache.org>>
>>> Subject:Re: [DISCUSS] Creating a new repo to host Flink benchmarks
>>>
>>> +1 on creating the repo.
>>>
>>>
>>> On Thu, Apr 9, 2020 at 5:54 PM Till Rohrmann 
>>> mailto:trohrm...@apache.org>>
>> wrote:
 I think it is a good idea to make the benchmarks available to the
>> community
 via a repo under the Apache project and to make updating it part of the
 release process. Hence +1 for the proposal.

 Cheers,
 Till

 On Thu, Apr 9, 2020 at 4:01 PM Piotr Nowojski 
 mailto:pi...@ververica.com>>
>> wrote:
> Hi Yun Tang,
>
> Thanks for proposing the idea. Since we can not include benchmarks in
>> the
> Flink repository what you are proposing is the second best option.
>
> +1 from my side for the proposal.
>
> I think benchmarks have proven their value to justify this.
>
> Piotrek
>
>> On 9 Apr 2020, at 08:56, Yun Tang 
>> mailto:myas...@live.com>> wrote:
>>
>> Hi Flink devs,
>>
>> As Flink develops rapidly with more and more features added, how to
> ensure no performance regression existed has become more and more
> important. And we would like to create a new repo under apache project
>> to
> host previous flink-benchmarks [1] repo, which is inspired when we
 discuss
> under FLINK-16850 [2]
>> Some background context on flink-benchmarks, for those who are not
> familiar with the project yet:
>> - Current flink-benchmarks does not align with the Flink release,
>> which
> lead developers not easy to verify
>>   performance at specific Flink version because current
 flink-benchmarks
> always depends on the latest interfaces.
>> - Above problem could be solved well if we could ensure
 flink-benchmarks
> also create release branch when we
>>   releasing Flink. However, current flink-benchmarks repo is hosted
> under dataArtisans (the former name of
>>   ververica) project, which is not involved in Flink release manual
 [3].
> We propose to promote this repo under
>>   apache project so that release manager could have the right to
 release
> on flink-benchmark

[jira] [Created] (FLINK-17283) Improve and explain the solution to Long Rides training exercise

[David Anderson (Jira)]

David Anderson created FLINK-17283:
--

 Summary: Improve and explain the solution to Long Rides training 
exercise
 Key: FLINK-17283
 URL: https://issues.apache.org/jira/browse/FLINK-17283
 Project: Flink
  Issue Type: Improvement
  Components: Training Exercises
Reporter: David Anderson
Assignee: David Anderson


The Long Rides Alerts exercise for flink-training is missing a DISCUSSION.md. 
And the solution can be made a bit easier to explain.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17284) Support serial field type in PostgresCatalog

[Flavio Pompermaier (Jira)]

Flavio Pompermaier created FLINK-17284:
--

 Summary: Support serial field type in PostgresCatalog
 Key: FLINK-17284
 URL: https://issues.apache.org/jira/browse/FLINK-17284
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / JDBC
Affects Versions: 1.11.0
Reporter: Flavio Pompermaier


In the current version of the PostgresCatalog the serial type is not handled, 
while it can be safely mapped to INT.

See an example at  https://www.postgresqltutorial.com/postgresql-create-table/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17285) Add Chinese translation to Python Table API Documentations

[Zixuan Rao (Jira)]

Zixuan Rao created FLINK-17285:
--

 Summary: Add Chinese translation to Python Table API Documentations
 Key: FLINK-17285
 URL: https://issues.apache.org/jira/browse/FLINK-17285
 Project: Flink
  Issue Type: Task
  Components: chinese-translation
Reporter: Zixuan Rao


This issue is to add Chinese translation to Python Table API. Untranslated 
paragraphs currently in ```*-zh.md``` will be replaced with Chinese 
translations. Translation will be done with ```Google Translate``` and manual 
corrections. I plan to submit a pull request this week. Thanks. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17286) Integrate json to file system connector

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-17286:


 Summary: Integrate json to file system connector
 Key: FLINK-17286
 URL: https://issues.apache.org/jira/browse/FLINK-17286
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet, 
ORC, SequenceFile)
Reporter: Jingsong Lee
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Apply to contribute to Flink

[Lee Sysuke]

Hi Guys,

I want to contribute to Apache Flink.Would you please give me the
permission as a contributor?My JIRA ID is lyee.