[jira] [Created] (FLINK-14049) Update error message for failed partition updates to include task name

[Stephan Ewen (Jira)]

Stephan Ewen created FLINK-14049:


 Summary: Update error message for failed partition updates to 
include task name
 Key: FLINK-14049
 URL: https://issues.apache.org/jira/browse/FLINK-14049
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.9.0
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.9.1


The error message for failed partition updates does not include the task name.
That makes it useless during debugging.

Adding the task name is a simple addition that make this error message much 
more helpful.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (FLINK-14050) Refactor YarnClusterDescriptor inheritance

[TisonKun (Jira)]

TisonKun created FLINK-14050:


 Summary: Refactor YarnClusterDescriptor inheritance
 Key: FLINK-14050
 URL: https://issues.apache.org/jira/browse/FLINK-14050
 Project: Flink
  Issue Type: Sub-task
  Components: Client / Job Submission, Command Line Client
Affects Versions: 1.10.0
Reporter: TisonKun
 Fix For: 1.10.0


Currently, the inheritance looks like

{{AbstractYarnClusterDescriptor}}
-> {{YarnClusterDescriptor}}
-> {{TestingYarnClusterDescriptor}}
-> {{NonDeployingYarnClusterDescriptor}}
->-> {{NonDeployingDetachedYarnClusterDescriptor}}

With an investigation, I find

1. {{AbstractYarnClusterDescriptor}} is introduced for migration purpose and no 
need any more.
2. {{TestingYarnClusterDescriptor}} is redundant and can be replaced directly 
with {{YarnClusterDescriptor}}.
3. Some methods like {{#createYarnClusterClient}} have parameters that never 
used, which are for historical reasons.

Thus, I propose we refactor {{YarnClusterDescriptor}} inheritance 
{{YarnClusterDescriptor}}
-> {{NonDeployingYarnClusterDescriptor}}
->-> {{NonDeployingDetachedYarnClusterDescriptor}}

and also methods remove unused parameters.

CC [~kkl0u] [~aljoscha] [~till.rohrmann]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (FLINK-14051) Deploy job cluster in attached mode

[TisonKun (Jira)]

TisonKun created FLINK-14051:


 Summary: Deploy job cluster in attached mode
 Key: FLINK-14051
 URL: https://issues.apache.org/jira/browse/FLINK-14051
 Project: Flink
  Issue Type: Sub-task
  Components: Client / Job Submission, Command Line Client
Affects Versions: 1.10.0
Reporter: TisonKun
 Fix For: 1.10.0


While working on FLINK-14048 I revisit the problem we handle deploy logic in a 
complicated if-else branched in {{CliFrontend#runProgram}}. Previously we said 
even in per-job mode and attached we deploy a session cluster for historical 
reasons.

However, I notice that {{#deployJobCluster}} has a parameter {{boolean 
detached}}. Also it is used in sql-client package. So it looks like we can 
deploy job cluster in attached mode as we do in sql-client package.

However, as [~xccui] answered on mailing list 
[here|https://lists.apache.org/x/thread.html/5464459db08f2a756af0c61eb02d34a26f04c27c62140886cad52731@%3Cuser.flink.apache.org%3E],
 we support only standalone session cluster for sql-client. So maybe it is not 
our case. Anyway, if we cannot deploy job cluster in attached mode, I'd like to 
know the concrete reason.

CC [~till.rohrmann] [~twalthr]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[DISCUSS] Drop older versions of Kafka Connectors (0.9, 0.10) for Flink 1.10

[Stephan Ewen]

Hi all!

We still maintain connectors for Kafka 0.8 and 0.9 in Flink.
I would suggest to drop those with Flink 1.10 and start supporting only
Kafka 0.10 onwards.

Are there any concerns about this, or still a significant number of users
of these versions?

Best,
Stephan

Re: Call for approving Elasticsearch 7.x connector

[Aljoscha Krettek]

Hi,

Thanks for the heads up! I commented on the issue.

Best,
Aljoscha

> On 9. Sep 2019, at 10:38, vino yang  wrote:
> 
> Hi guys,
> 
> There is an issue about supporting Elasticsearch 7.x.[1]
> Based on our validation and discussion. We found that Elasticsearch 7.x
> does not guarantee API compatibility. Therefore, it does not have the
> ability to provide a universal connector like Kafka. It seems that we have
> to provide a new connector to support Elasticsearch 7.x.
> 
> Consider that Elasticsearch is a widely used system. There have been
> multiple user comments hoping to support Elasticsearch 7.x as soon as
> possible. Therefore, I hope this new connector will be approved as soon as
> possible, so that this work can be started.
> 
> Best,
> Vino
> 
> [1]: https://issues.apache.org/jira/browse/FLINK-13025

Re: [DISCUSS] Flink client api enhancement for downstream project

[Aljoscha Krettek]

Hi,

We could try and use the ASF slack for this purpose, that would probably be 
easiest. See https://s.apache.org/slack-invite. We could create a dedicated 
channel for our work and would still use the open ASF infrastructure and people 
can have a look if they are interested because discussion would be public. What 
do you think?

P.S. Committers/PMCs should should be able to login with their apache ID.

Best,
Aljoscha

> On 6. Sep 2019, at 14:24, Zili Chen  wrote:
> 
> Hi Aljoscha,
> 
> I'd like to gather all the ideas here and among documents, and draft a
> formal FLIP
> that keep us on the same page. Hopefully I start a FLIP thread in next week.
> 
> For the implementation or said POC part, I'd like to work with you guys who
> proposed
> the concept Executor to make sure that we go in the same direction. I'm
> wondering
> whether a dedicate thread or a Slack group is the proper one. In my opinion
> we can
> involve the team in a Slack group, concurrent with the FLIP process start
> our branch
> and once we reach a consensus on the FLIP, open an umbrella issue about the
> framework
> and start subtasks. What do you think?
> 
> Best,
> tison.
> 
> 
> Aljoscha Krettek  于2019年9月5日周四 下午9:39写道：
> 
>> Hi Tison,
>> 
>> To keep this moving forward, maybe you want to start working on a proof of
>> concept implementation for the new JobClient interface, maybe with a new
>> method executeAsync() in the environment that returns the JobClient and
>> implement the methods to see how that works and to see where we get. Would
>> you be interested in that?
>> 
>> Also, at some point we should collect all the ideas and start forming an
>> actual FLIP.
>> 
>> Best,
>> Aljoscha
>> 
>>> On 4. Sep 2019, at 12:04, Zili Chen  wrote:
>>> 
>>> Thanks for your update Kostas!
>>> 
>>> It looks good to me that clean up existing code paths as first
>>> pass. I'd like to help on review and file subtasks if I find ones.
>>> 
>>> Best,
>>> tison.
>>> 
>>> 
>>> Kostas Kloudas  于2019年9月4日周三 下午5:52写道：
>>> Here is the issue, and I will keep on updating it as I find more issues.
>>> 
>>> https://issues.apache.org/jira/browse/FLINK-13954
>>> 
>>> This will also cover the refactoring of the Executors that we discussed
>>> in this thread, without any additional functionality (such as the job
>> client).
>>> 
>>> Kostas
>>> 
>>> On Wed, Sep 4, 2019 at 11:46 AM Kostas Kloudas 
>> wrote:
 
 Great idea Tison!
 
 I will create the umbrella issue and post it here so that we are all
 on the same page!
 
 Cheers,
 Kostas
 
 On Wed, Sep 4, 2019 at 11:36 AM Zili Chen 
>> wrote:
> 
> Hi Kostas & Aljoscha,
> 
> I notice that there is a JIRA(FLINK-13946) which could be included
> in this refactor thread. Since we agree on most of directions in
> big picture, is it reasonable that we create an umbrella issue for
> refactor client APIs and also linked subtasks? It would be a better
> way that we join forces of our community.
> 
> Best,
> tison.
> 
> 
> Zili Chen  于2019年8月31日周六 下午12:52写道：
>> 
>> Great Kostas! Looking forward to your POC!
>> 
>> Best,
>> tison.
>> 
>> 
>> Jeff Zhang  于2019年8月30日周五 下午11:07写道：
>>> 
>>> Awesome, @Kostas Looking forward your POC.
>>> 
>>> Kostas Kloudas  于2019年8月30日周五 下午8:33写道：
>>> 
 Hi all,
 
 I am just writing here to let you know that I am working on a
>> POC that
 tries to refactor the current state of job submission in Flink.
 I want to stress out that it introduces NO CHANGES to the current
 behaviour of Flink. It just re-arranges things and introduces the
 notion of an Executor, which is the entity responsible for
>> taking the
 user-code and submitting it for execution.
 
 Given this, the discussion about the functionality that the
>> JobClient
 will expose to the user can go on independently and the same
 holds for all the open questions so far.
 
 I hope I will have some more new to share soon.
 
 Thanks,
 Kostas
 
 On Mon, Aug 26, 2019 at 4:20 AM Yang Wang 
>> wrote:
> 
> Hi Zili,
> 
> It make sense to me that a dedicated cluster is started for a
>> per-job
> cluster and will not accept more jobs.
> Just have a question about the command line.
> 
> Currently we could use the following commands to start
>> different
 clusters.
> *per-job cluster*
> ./bin/flink run -d -p 5 -ynm perjob-cluster1 -m yarn-cluster
> examples/streaming/WindowJoin.jar
> *session cluster*
> ./bin/flink run -p 5 -ynm session-cluster1 -m yarn-cluster
> examples/streaming/WindowJoin.jar
> 
> What will it look like after client enhancement?
> 
> 
> Best,
> Yang
> 
> Zili Ch

Re: [DISCUSS] Flink client api enhancement for downstream project

[Jeff Zhang]

+1 for using slack for instant communication

Aljoscha Krettek  于2019年9月11日周三 下午4:44写道：

> Hi,
>
> We could try and use the ASF slack for this purpose, that would probably
> be easiest. See https://s.apache.org/slack-invite. We could create a
> dedicated channel for our work and would still use the open ASF
> infrastructure and people can have a look if they are interested because
> discussion would be public. What do you think?
>
> P.S. Committers/PMCs should should be able to login with their apache ID.
>
> Best,
> Aljoscha
>
> > On 6. Sep 2019, at 14:24, Zili Chen  wrote:
> >
> > Hi Aljoscha,
> >
> > I'd like to gather all the ideas here and among documents, and draft a
> > formal FLIP
> > that keep us on the same page. Hopefully I start a FLIP thread in next
> week.
> >
> > For the implementation or said POC part, I'd like to work with you guys
> who
> > proposed
> > the concept Executor to make sure that we go in the same direction. I'm
> > wondering
> > whether a dedicate thread or a Slack group is the proper one. In my
> opinion
> > we can
> > involve the team in a Slack group, concurrent with the FLIP process start
> > our branch
> > and once we reach a consensus on the FLIP, open an umbrella issue about
> the
> > framework
> > and start subtasks. What do you think?
> >
> > Best,
> > tison.
> >
> >
> > Aljoscha Krettek  于2019年9月5日周四 下午9:39写道：
> >
> >> Hi Tison,
> >>
> >> To keep this moving forward, maybe you want to start working on a proof
> of
> >> concept implementation for the new JobClient interface, maybe with a new
> >> method executeAsync() in the environment that returns the JobClient and
> >> implement the methods to see how that works and to see where we get.
> Would
> >> you be interested in that?
> >>
> >> Also, at some point we should collect all the ideas and start forming an
> >> actual FLIP.
> >>
> >> Best,
> >> Aljoscha
> >>
> >>> On 4. Sep 2019, at 12:04, Zili Chen  wrote:
> >>>
> >>> Thanks for your update Kostas!
> >>>
> >>> It looks good to me that clean up existing code paths as first
> >>> pass. I'd like to help on review and file subtasks if I find ones.
> >>>
> >>> Best,
> >>> tison.
> >>>
> >>>
> >>> Kostas Kloudas  于2019年9月4日周三 下午5:52写道：
> >>> Here is the issue, and I will keep on updating it as I find more
> issues.
> >>>
> >>> https://issues.apache.org/jira/browse/FLINK-13954
> >>>
> >>> This will also cover the refactoring of the Executors that we discussed
> >>> in this thread, without any additional functionality (such as the job
> >> client).
> >>>
> >>> Kostas
> >>>
> >>> On Wed, Sep 4, 2019 at 11:46 AM Kostas Kloudas 
> >> wrote:
> 
>  Great idea Tison!
> 
>  I will create the umbrella issue and post it here so that we are all
>  on the same page!
> 
>  Cheers,
>  Kostas
> 
>  On Wed, Sep 4, 2019 at 11:36 AM Zili Chen 
> >> wrote:
> >
> > Hi Kostas & Aljoscha,
> >
> > I notice that there is a JIRA(FLINK-13946) which could be included
> > in this refactor thread. Since we agree on most of directions in
> > big picture, is it reasonable that we create an umbrella issue for
> > refactor client APIs and also linked subtasks? It would be a better
> > way that we join forces of our community.
> >
> > Best,
> > tison.
> >
> >
> > Zili Chen  于2019年8月31日周六 下午12:52写道：
> >>
> >> Great Kostas! Looking forward to your POC!
> >>
> >> Best,
> >> tison.
> >>
> >>
> >> Jeff Zhang  于2019年8月30日周五 下午11:07写道：
> >>>
> >>> Awesome, @Kostas Looking forward your POC.
> >>>
> >>> Kostas Kloudas  于2019年8月30日周五 下午8:33写道：
> >>>
>  Hi all,
> 
>  I am just writing here to let you know that I am working on a
> >> POC that
>  tries to refactor the current state of job submission in Flink.
>  I want to stress out that it introduces NO CHANGES to the current
>  behaviour of Flink. It just re-arranges things and introduces the
>  notion of an Executor, which is the entity responsible for
> >> taking the
>  user-code and submitting it for execution.
> 
>  Given this, the discussion about the functionality that the
> >> JobClient
>  will expose to the user can go on independently and the same
>  holds for all the open questions so far.
> 
>  I hope I will have some more new to share soon.
> 
>  Thanks,
>  Kostas
> 
>  On Mon, Aug 26, 2019 at 4:20 AM Yang Wang 
> >> wrote:
> >
> > Hi Zili,
> >
> > It make sense to me that a dedicated cluster is started for a
> >> per-job
> > cluster and will not accept more jobs.
> > Just have a question about the command line.
> >
> > Currently we could use the following commands to start
> >> different
>  clusters.
> > *per-job cluster*
> > ./bin/flink run -d -p 5 -ynm perjob-cluster1

Re: [DISCUSS] Flink client api enhancement for downstream project

[Zili Chen]

Hi Aljoscha,

I'm OK to use the ASF slack.

Best,
tison.


Jeff Zhang  于2019年9月11日周三 下午4:48写道：

> +1 for using slack for instant communication
>
> Aljoscha Krettek  于2019年9月11日周三 下午4:44写道：
>
>> Hi,
>>
>> We could try and use the ASF slack for this purpose, that would probably
>> be easiest. See https://s.apache.org/slack-invite. We could create a
>> dedicated channel for our work and would still use the open ASF
>> infrastructure and people can have a look if they are interested because
>> discussion would be public. What do you think?
>>
>> P.S. Committers/PMCs should should be able to login with their apache ID.
>>
>> Best,
>> Aljoscha
>>
>> > On 6. Sep 2019, at 14:24, Zili Chen  wrote:
>> >
>> > Hi Aljoscha,
>> >
>> > I'd like to gather all the ideas here and among documents, and draft a
>> > formal FLIP
>> > that keep us on the same page. Hopefully I start a FLIP thread in next
>> week.
>> >
>> > For the implementation or said POC part, I'd like to work with you guys
>> who
>> > proposed
>> > the concept Executor to make sure that we go in the same direction. I'm
>> > wondering
>> > whether a dedicate thread or a Slack group is the proper one. In my
>> opinion
>> > we can
>> > involve the team in a Slack group, concurrent with the FLIP process
>> start
>> > our branch
>> > and once we reach a consensus on the FLIP, open an umbrella issue about
>> the
>> > framework
>> > and start subtasks. What do you think?
>> >
>> > Best,
>> > tison.
>> >
>> >
>> > Aljoscha Krettek  于2019年9月5日周四 下午9:39写道：
>> >
>> >> Hi Tison,
>> >>
>> >> To keep this moving forward, maybe you want to start working on a
>> proof of
>> >> concept implementation for the new JobClient interface, maybe with a
>> new
>> >> method executeAsync() in the environment that returns the JobClient and
>> >> implement the methods to see how that works and to see where we get.
>> Would
>> >> you be interested in that?
>> >>
>> >> Also, at some point we should collect all the ideas and start forming
>> an
>> >> actual FLIP.
>> >>
>> >> Best,
>> >> Aljoscha
>> >>
>> >>> On 4. Sep 2019, at 12:04, Zili Chen  wrote:
>> >>>
>> >>> Thanks for your update Kostas!
>> >>>
>> >>> It looks good to me that clean up existing code paths as first
>> >>> pass. I'd like to help on review and file subtasks if I find ones.
>> >>>
>> >>> Best,
>> >>> tison.
>> >>>
>> >>>
>> >>> Kostas Kloudas  于2019年9月4日周三 下午5:52写道：
>> >>> Here is the issue, and I will keep on updating it as I find more
>> issues.
>> >>>
>> >>> https://issues.apache.org/jira/browse/FLINK-13954
>> >>>
>> >>> This will also cover the refactoring of the Executors that we
>> discussed
>> >>> in this thread, without any additional functionality (such as the job
>> >> client).
>> >>>
>> >>> Kostas
>> >>>
>> >>> On Wed, Sep 4, 2019 at 11:46 AM Kostas Kloudas 
>> >> wrote:
>> 
>>  Great idea Tison!
>> 
>>  I will create the umbrella issue and post it here so that we are all
>>  on the same page!
>> 
>>  Cheers,
>>  Kostas
>> 
>>  On Wed, Sep 4, 2019 at 11:36 AM Zili Chen 
>> >> wrote:
>> >
>> > Hi Kostas & Aljoscha,
>> >
>> > I notice that there is a JIRA(FLINK-13946) which could be included
>> > in this refactor thread. Since we agree on most of directions in
>> > big picture, is it reasonable that we create an umbrella issue for
>> > refactor client APIs and also linked subtasks? It would be a better
>> > way that we join forces of our community.
>> >
>> > Best,
>> > tison.
>> >
>> >
>> > Zili Chen  于2019年8月31日周六 下午12:52写道：
>> >>
>> >> Great Kostas! Looking forward to your POC!
>> >>
>> >> Best,
>> >> tison.
>> >>
>> >>
>> >> Jeff Zhang  于2019年8月30日周五 下午11:07写道：
>> >>>
>> >>> Awesome, @Kostas Looking forward your POC.
>> >>>
>> >>> Kostas Kloudas  于2019年8月30日周五 下午8:33写道：
>> >>>
>>  Hi all,
>> 
>>  I am just writing here to let you know that I am working on a
>> >> POC that
>>  tries to refactor the current state of job submission in Flink.
>>  I want to stress out that it introduces NO CHANGES to the current
>>  behaviour of Flink. It just re-arranges things and introduces the
>>  notion of an Executor, which is the entity responsible for
>> >> taking the
>>  user-code and submitting it for execution.
>> 
>>  Given this, the discussion about the functionality that the
>> >> JobClient
>>  will expose to the user can go on independently and the same
>>  holds for all the open questions so far.
>> 
>>  I hope I will have some more new to share soon.
>> 
>>  Thanks,
>>  Kostas
>> 
>>  On Mon, Aug 26, 2019 at 4:20 AM Yang Wang > >
>> >> wrote:
>> >
>> > Hi Zili,
>> >
>> > It make sense to me that a dedicated cluster is started for a
>> >> per-job
>> > cluster and will not accept m

[jira] [Created] (FLINK-14052) Generalize AkkaOptions to RpcOptions

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-14052:
-

 Summary: Generalize AkkaOptions to RpcOptions
 Key: FLINK-14052
 URL: https://issues.apache.org/jira/browse/FLINK-14052
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Configuration, Runtime / Coordination
Affects Versions: 1.9.0, 1.10.0
Reporter: Till Rohrmann


At the moment Flink's rpc configuration options are based on Akka specific 
configuration options defined in {{AkkaOptions}}. For example, the rpc timeout 
is configured via {{akka.ask.timeout}}. This has historical reasons because we 
introduced the {{RpcService}} after using Akka directly for rpc communication.

I think this confusing for users since the fact that we use Akka should only be 
an implementation detail. Instead we should have general rpc configuration 
options such as {{rpc.timeout}} which apply to all {{RpcService}} 
implementations and implementation specific configuration options such as 
{{rpc.akka.transport.heartbeat.interval}}.

Within the scope of this issue we could also rethink whether we don't copy the 
Akka specific configuration options names. For example 
{{rpc.akka.transport.heartbeat.interval}} should then be called 
{{rpc.akka.transport-failure-detector.heartbeat-interval}}. That way it would 
also be easy to simply parse and set them.

As a first step we need to list all RPC relevant configuration options 
currently used and decide whether it a RPC general option or an Akka specific 
one.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [DISCUSS] Flink client api enhancement for downstream project

[Kostas Kloudas]

+1 for creating a channel.

Kostas

On Wed, Sep 11, 2019 at 10:57 AM Zili Chen  wrote:
>
> Hi Aljoscha,
>
> I'm OK to use the ASF slack.
>
> Best,
> tison.
>
>
> Jeff Zhang  于2019年9月11日周三 下午4:48写道：
>>
>> +1 for using slack for instant communication
>>
>> Aljoscha Krettek  于2019年9月11日周三 下午4:44写道：
>>>
>>> Hi,
>>>
>>> We could try and use the ASF slack for this purpose, that would probably be 
>>> easiest. See https://s.apache.org/slack-invite. We could create a dedicated 
>>> channel for our work and would still use the open ASF infrastructure and 
>>> people can have a look if they are interested because discussion would be 
>>> public. What do you think?
>>>
>>> P.S. Committers/PMCs should should be able to login with their apache ID.
>>>
>>> Best,
>>> Aljoscha
>>>
>>> > On 6. Sep 2019, at 14:24, Zili Chen  wrote:
>>> >
>>> > Hi Aljoscha,
>>> >
>>> > I'd like to gather all the ideas here and among documents, and draft a
>>> > formal FLIP
>>> > that keep us on the same page. Hopefully I start a FLIP thread in next 
>>> > week.
>>> >
>>> > For the implementation or said POC part, I'd like to work with you guys 
>>> > who
>>> > proposed
>>> > the concept Executor to make sure that we go in the same direction. I'm
>>> > wondering
>>> > whether a dedicate thread or a Slack group is the proper one. In my 
>>> > opinion
>>> > we can
>>> > involve the team in a Slack group, concurrent with the FLIP process start
>>> > our branch
>>> > and once we reach a consensus on the FLIP, open an umbrella issue about 
>>> > the
>>> > framework
>>> > and start subtasks. What do you think?
>>> >
>>> > Best,
>>> > tison.
>>> >
>>> >
>>> > Aljoscha Krettek  于2019年9月5日周四 下午9:39写道：
>>> >
>>> >> Hi Tison,
>>> >>
>>> >> To keep this moving forward, maybe you want to start working on a proof 
>>> >> of
>>> >> concept implementation for the new JobClient interface, maybe with a new
>>> >> method executeAsync() in the environment that returns the JobClient and
>>> >> implement the methods to see how that works and to see where we get. 
>>> >> Would
>>> >> you be interested in that?
>>> >>
>>> >> Also, at some point we should collect all the ideas and start forming an
>>> >> actual FLIP.
>>> >>
>>> >> Best,
>>> >> Aljoscha
>>> >>
>>> >>> On 4. Sep 2019, at 12:04, Zili Chen  wrote:
>>> >>>
>>> >>> Thanks for your update Kostas!
>>> >>>
>>> >>> It looks good to me that clean up existing code paths as first
>>> >>> pass. I'd like to help on review and file subtasks if I find ones.
>>> >>>
>>> >>> Best,
>>> >>> tison.
>>> >>>
>>> >>>
>>> >>> Kostas Kloudas  于2019年9月4日周三 下午5:52写道：
>>> >>> Here is the issue, and I will keep on updating it as I find more issues.
>>> >>>
>>> >>> https://issues.apache.org/jira/browse/FLINK-13954
>>> >>>
>>> >>> This will also cover the refactoring of the Executors that we discussed
>>> >>> in this thread, without any additional functionality (such as the job
>>> >> client).
>>> >>>
>>> >>> Kostas
>>> >>>
>>> >>> On Wed, Sep 4, 2019 at 11:46 AM Kostas Kloudas 
>>> >> wrote:
>>> 
>>>  Great idea Tison!
>>> 
>>>  I will create the umbrella issue and post it here so that we are all
>>>  on the same page!
>>> 
>>>  Cheers,
>>>  Kostas
>>> 
>>>  On Wed, Sep 4, 2019 at 11:36 AM Zili Chen 
>>> >> wrote:
>>> >
>>> > Hi Kostas & Aljoscha,
>>> >
>>> > I notice that there is a JIRA(FLINK-13946) which could be included
>>> > in this refactor thread. Since we agree on most of directions in
>>> > big picture, is it reasonable that we create an umbrella issue for
>>> > refactor client APIs and also linked subtasks? It would be a better
>>> > way that we join forces of our community.
>>> >
>>> > Best,
>>> > tison.
>>> >
>>> >
>>> > Zili Chen  于2019年8月31日周六 下午12:52写道：
>>> >>
>>> >> Great Kostas! Looking forward to your POC!
>>> >>
>>> >> Best,
>>> >> tison.
>>> >>
>>> >>
>>> >> Jeff Zhang  于2019年8月30日周五 下午11:07写道：
>>> >>>
>>> >>> Awesome, @Kostas Looking forward your POC.
>>> >>>
>>> >>> Kostas Kloudas  于2019年8月30日周五 下午8:33写道：
>>> >>>
>>>  Hi all,
>>> 
>>>  I am just writing here to let you know that I am working on a
>>> >> POC that
>>>  tries to refactor the current state of job submission in Flink.
>>>  I want to stress out that it introduces NO CHANGES to the current
>>>  behaviour of Flink. It just re-arranges things and introduces the
>>>  notion of an Executor, which is the entity responsible for
>>> >> taking the
>>>  user-code and submitting it for execution.
>>> 
>>>  Given this, the discussion about the functionality that the
>>> >> JobClient
>>>  will expose to the user can go on independently and the same
>>>  holds for all the open questions so far.
>>> 
>>>  I hope I will have some more new to share soon.
>>> 
>>>  Thanks,
>>>  Kostas
>>

[ANNOUNCE] Zili Chen becomes a Flink committer

[Till Rohrmann]

Hi everyone,

I'm very happy to announce that Zili Chen (some of you might also know him
as Tison Kun) accepted the offer of the Flink PMC to become a committer of
the Flink project.

Zili Chen has been an active community member for almost 16 months now. He
helped pushing the Flip-6 effort over the finish line, ported a lot of
legacy code tests, removed a good part of the legacy code, contributed
numerous fixes, is involved in the Flink's client API refactoring, drives
the refactoring of Flink's HighAvailabilityServices and much more. Zili
Chen also helped the community by PR reviews, reporting Flink issues,
answering user mails and being very active on the dev mailing list.

Congratulations Zili Chen!

Best, Till
(on behalf of the Flink PMC)

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Forward Xu]

Congratulations Zili Chen!


Best,

Forward

Till Rohrmann  于2019年9月11日周三 下午5:23写道：

> Hi everyone,
>
> I'm very happy to announce that Zili Chen (some of you might also know him
> as Tison Kun) accepted the offer of the Flink PMC to become a committer of
> the Flink project.
>
> Zili Chen has been an active community member for almost 16 months now. He
> helped pushing the Flip-6 effort over the finish line, ported a lot of
> legacy code tests, removed a good part of the legacy code, contributed
> numerous fixes, is involved in the Flink's client API refactoring, drives
> the refactoring of Flink's HighAvailabilityServices and much more. Zili
> Chen also helped the community by PR reviews, reporting Flink issues,
> answering user mails and being very active on the dev mailing list.
>
> Congratulations Zili Chen!
>
> Best, Till
> (on behalf of the Flink PMC)
>

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Dian Fu]

Congratulations!

> 在 2019年9月11日，下午5:26，Jeff Zhang  写道：
> 
> Congratulations Zili Chen! 
> 
> Wesley Peng mailto:wes...@thepeng.eu>> 于2019年9月11日周三 
> 下午5:25写道：
> Hi
> 
> on 2019/9/11 17:22, Till Rohrmann wrote:
> > I'm very happy to announce that Zili Chen (some of you might also know 
> > him as Tison Kun) accepted the offer of the Flink PMC to become a 
> > committer of the Flink project.
> 
> Congratulations Zili Chen.
> 
> regards.
> 
> 
> -- 
> Best Regards
> 
> Jeff Zhang

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Terry Wang]

Congratulations!

Best,
Terry Wang



> 在 2019年9月11日，下午5:28，Dian Fu  写道：
> 
> Congratulations!
> 
>> 在 2019年9月11日，下午5:26，Jeff Zhang mailto:zjf...@gmail.com>> 
>> 写道：
>> 
>> Congratulations Zili Chen! 
>> 
>> Wesley Peng mailto:wes...@thepeng.eu>> 于2019年9月11日周三 
>> 下午5:25写道：
>> Hi
>> 
>> on 2019/9/11 17:22, Till Rohrmann wrote:
>> > I'm very happy to announce that Zili Chen (some of you might also know 
>> > him as Tison Kun) accepted the offer of the Flink PMC to become a 
>> > committer of the Flink project.
>> 
>> Congratulations Zili Chen.
>> 
>> regards.
>> 
>> 
>> -- 
>> Best Regards
>> 
>> Jeff Zhang
> 

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Zhu Zhu]

Congratulations Zili!

Thanks,
Zhu Zhu

Terry Wang  于2019年9月11日周三 下午5:34写道：

> Congratulations!
>
> Best,
> Terry Wang
>
>
>
> 在 2019年9月11日，下午5:28，Dian Fu  写道：
>
> Congratulations!
>
> 在 2019年9月11日，下午5:26，Jeff Zhang  写道：
>
> Congratulations Zili Chen!
>
> Wesley Peng  于2019年9月11日周三 下午5:25写道：
>
>> Hi
>>
>> on 2019/9/11 17:22, Till Rohrmann wrote:
>> > I'm very happy to announce that Zili Chen (some of you might also know
>> > him as Tison Kun) accepted the offer of the Flink PMC to become a
>> > committer of the Flink project.
>>
>> Congratulations Zili Chen.
>>
>> regards.
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>
>
>

Re: [DISCUSS] Features for Apache Flink 1.10

[Aljoscha Krettek]

Hi,

Thanks for putting together the list! And I’m +1 for the suggested 
release timeline and also for Gary and Yu as the release managers.


Best,
Aljoscha

On 9 Sep 2019, at 7:39, Yu Li wrote:


Hi Xuefu,

If I understand it correctly, the data type support work should be 
included
in the "Table API improvements->Finish type system" part, please check 
it

and let us know if anything missing there. Thanks.

Best Regards,
Yu


On Mon, 9 Sep 2019 at 11:14, Xuefu Z  wrote:

Looking at feature list, I don't see an item for complete the data 
type

support. Specifically, high precision timestamp is needed to Hive
integration, as it's so common. Missing it would damage the 
completeness of

our Hive effort.

Thanks,
Xuefu

On Sat, Sep 7, 2019 at 7:06 PM Xintong Song  
wrote:


Thanks Gray and Yu for compiling the feature list and kicking off 
this

discussion.

+1 for Gary and Yu being the release managers for Flink 1.10.

Thank you~

Xintong Song



On Sat, Sep 7, 2019 at 4:58 PM Till Rohrmann 

wrote:


Thanks for compiling the list of 1.10 efforts for the community 
Gary. I

think this helps a lot to better understand what the community is

currently

working on.

Thanks for volunteering as the release managers for the next major
release. +1 for Gary and Yu being the RMs for Flink 1.10.

Cheers,
Till

On Sat, Sep 7, 2019 at 7:26 AM Zhu Zhu  wrote:


Thanks Gary for kicking off this discussion.
Really appreciate that you and Yu offer to help to manage 1.10

release.


+1 for Gary and Yu as release managers.

Thanks,
Zhu Zhu

Dian Fu  于2019年9月7日周六 
下午12:26写道：



Hi Gary,

Thanks for kicking off the release schedule of 1.10. +1 for you 
and

Yu

Li

as the release manager.

The feature freeze/release time sounds reasonable.

Thanks,
Dian

在 2019年9月7日，上午11:30，Jark Wu  
写道：


Thanks Gary for kicking off the discussion for 1.10 release.

+1 for Gary and Yu as release managers. Thank you for you 
effort.


Best,
Jark


在 2019年9月7日，00:52，zhijiang 


写道：


Hi Gary,

Thanks for kicking off the features for next release 1.10.  I 
am

very

supportive of you and Yu Li to be the relaese managers.


Just mention another two improvements which want to be covered

in
FLINK-1.10 and I already confirmed with Piotr to reach an 
agreement

before.


1. Data serialize and copy only once for broadcast partition

[1]:

It
would improve the throughput performance greatly in broadcast 
mode

and

was

actually proposed in Flink-1.8. Most of works already done before

and

only

left the last critical jira/PR. It will not take much efforts to

make

it

ready.


2. Let Netty use Flink's buffers directly in credit-based mode

[2] :

It

could avoid memory copy from netty stack to flink managed network

buffer.

The obvious benefit is decreasing the direct memory overhead

greatly

in
large-scale jobs. I also heard of some user cases encounter 
direct

OOM

caused by netty memory overhead. Actually this improvment was

proposed

by
nico in FLINK-1.7 and always no time to focus then. Yun Gao 
already

submitted a PR half an year ago but have not been reviewed yet. I

could

help review the deign and PR codes to make it ready.


And you could make these two items as lowest priority if

possible.


[1] https://issues.apache.org/jira/browse/FLINK-10745
[2] https://issues.apache.org/jira/browse/FLINK-10742

Best,
Zhijiang


--

From:Gary Yao 
Send Time:2019年9月6日(星期五) 17:06
To:dev 
Cc:carp84 
Subject:[DISCUSS] Features for Apache Flink 1.10

Hi community,

Since Apache Flink 1.9.0 has been released more than 2 weeks

ago,

I

want to

start kicking off the discussion about what we want to achieve

for

the

1.10

release.

Based on discussions with various people as well as 
observations

from

mailing
list threads, Yu Li and I have compiled a list of features that

we

deem

important to be included in the next release. Note that the

features

presented
here are not meant to be exhaustive. As always, I am sure that

there

will be

other contributions that will make it into the next release.

This

email

thread
is merely to kick off a discussion, and to give users and

contributors

an
understanding where the focus of the next release lies. If 
there

is

anything
we have missed that somebody is working on, please reply to 
this

thread.



** Proposed features and focus

Following the contribution of Blink to Apache Flink, the

community

released

a
preview of the Blink SQL Query Processor, which offers better

SQL

coverage

and
improved performance for batch queries, in Flink 1.9.0. 
However,

the

integration of the Blink query processor is not fully completed

yet

as

there

are still pending tasks, such as implementing full TPC-DS

support.

With

the

next Flink release, we aim at finishing the Blink integration.

Furthermore, there are several ongoing work threads addressing

long-standing

issues reported by users, such as improving checkpointing under
backpressu

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[bupt_ljy]

Congratulations!


Best,
Jiayi Liao


 Original Message 
Sender: Till Rohrmann
Recipient: dev; user
Date: Wednesday, Sep 11, 2019 17:22
Subject: [ANNOUNCE] Zili Chen becomes a Flink committer


Hi everyone,


I'm very happy to announce that Zili Chen (some of you might also know him as 
Tison Kun) accepted the offer of the Flink PMC to become a committer of the 
Flink project.


Zili Chen has been an active community member for almost 16 months now. He 
helped pushing the Flip-6 effort over the finish line, ported a lot of legacy 
code tests, removed a good part of the legacy code, contributed numerous fixes, 
is involved in the Flink's client API refactoring, drives the refactoring of 
Flink's HighAvailabilityServices and much more. Zili Chen also helped the 
community by PR reviews, reporting Flink issues, answering user mails and being 
very active on the dev mailing list.


Congratulations Zili Chen!


Best, Till 

(on behalf of the Flink PMC)

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Benchao Li]

Congratulations Zili Chen!

bupt_ljy  于2019年9月11日周三 下午6:36写道：

> Congratulations!
>
>
> Best,
>
> Jiayi Liao
>
>  Original Message
> *Sender:* Till Rohrmann
> *Recipient:* dev; user
> *Date:* Wednesday, Sep 11, 2019 17:22
> *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer
>
> Hi everyone,
>
> I'm very happy to announce that Zili Chen (some of you might also know
> him as Tison Kun) accepted the offer of the Flink PMC to become a committer
> of the Flink project.
>
> Zili Chen has been an active community member for almost 16 months now.
> He helped pushing the Flip-6 effort over the finish line, ported a lot of
> legacy code tests, removed a good part of the legacy code, contributed
> numerous fixes, is involved in the Flink's client API refactoring, drives
> the refactoring of Flink's HighAvailabilityServices and much more. Zili
> Chen also helped the community by PR reviews, reporting Flink issues,
> answering user mails and being very active on the dev mailing list.
>
> Congratulations Zili Chen!
>
> Best, Till
> (on behalf of the Flink PMC)
>


-- 

Benchao Li
School of Electronics Engineering and Computer Science, Peking University
Tel:+86-15650713730
Email: libenc...@gmail.com; libenc...@pku.edu.cn

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Biao Liu]

Congrats Zili!

Thanks,
Biao /'bɪ.aʊ/



On Wed, 11 Sep 2019 at 18:43, Oytun Tez  wrote:

> Congratulations!
>
> ---
> Oytun Tez
>
> *M O T A W O R D*
> The World's Fastest Human Translation Platform.
> oy...@motaword.com — www.motaword.com
>
>
> On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy  wrote:
>
>> Congratulations!
>>
>>
>> Best,
>>
>> Jiayi Liao
>>
>>  Original Message
>> *Sender:* Till Rohrmann
>> *Recipient:* dev; user
>> *Date:* Wednesday, Sep 11, 2019 17:22
>> *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer
>>
>> Hi everyone,
>>
>> I'm very happy to announce that Zili Chen (some of you might also know
>> him as Tison Kun) accepted the offer of the Flink PMC to become a committer
>> of the Flink project.
>>
>> Zili Chen has been an active community member for almost 16 months now.
>> He helped pushing the Flip-6 effort over the finish line, ported a lot of
>> legacy code tests, removed a good part of the legacy code, contributed
>> numerous fixes, is involved in the Flink's client API refactoring, drives
>> the refactoring of Flink's HighAvailabilityServices and much more. Zili
>> Chen also helped the community by PR reviews, reporting Flink issues,
>> answering user mails and being very active on the dev mailing list.
>>
>> Congratulations Zili Chen!
>>
>> Best, Till
>> (on behalf of the Flink PMC)
>>
>

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Fabian Hueske]

Congrats Zili Chen :-)

Cheers, Fabian

Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu :

> Congrats Zili!
>
> Thanks,
> Biao /'bɪ.aʊ/
>
>
>
> On Wed, 11 Sep 2019 at 18:43, Oytun Tez  wrote:
>
>> Congratulations!
>>
>> ---
>> Oytun Tez
>>
>> *M O T A W O R D*
>> The World's Fastest Human Translation Platform.
>> oy...@motaword.com — www.motaword.com
>>
>>
>> On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy  wrote:
>>
>>> Congratulations!
>>>
>>>
>>> Best,
>>>
>>> Jiayi Liao
>>>
>>>  Original Message
>>> *Sender:* Till Rohrmann
>>> *Recipient:* dev; user
>>> *Date:* Wednesday, Sep 11, 2019 17:22
>>> *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer
>>>
>>> Hi everyone,
>>>
>>> I'm very happy to announce that Zili Chen (some of you might also know
>>> him as Tison Kun) accepted the offer of the Flink PMC to become a committer
>>> of the Flink project.
>>>
>>> Zili Chen has been an active community member for almost 16 months now.
>>> He helped pushing the Flip-6 effort over the finish line, ported a lot of
>>> legacy code tests, removed a good part of the legacy code, contributed
>>> numerous fixes, is involved in the Flink's client API refactoring, drives
>>> the refactoring of Flink's HighAvailabilityServices and much more. Zili
>>> Chen also helped the community by PR reviews, reporting Flink issues,
>>> answering user mails and being very active on the dev mailing list.
>>>
>>> Congratulations Zili Chen!
>>>
>>> Best, Till
>>> (on behalf of the Flink PMC)
>>>
>>

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Guowei Ma]

Congratulations Zili !

Best,
Guowei


Fabian Hueske  于2019年9月11日周三 下午7:02写道：

> Congrats Zili Chen :-)
>
> Cheers, Fabian
>
> Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu :
>
>> Congrats Zili!
>>
>> Thanks,
>> Biao /'bɪ.aʊ/
>>
>>
>>
>> On Wed, 11 Sep 2019 at 18:43, Oytun Tez  wrote:
>>
>>> Congratulations!
>>>
>>> ---
>>> Oytun Tez
>>>
>>> *M O T A W O R D*
>>> The World's Fastest Human Translation Platform.
>>> oy...@motaword.com — www.motaword.com
>>>
>>>
>>> On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy  wrote:
>>>
 Congratulations!


 Best,

 Jiayi Liao

  Original Message
 *Sender:* Till Rohrmann
 *Recipient:* dev; user
 *Date:* Wednesday, Sep 11, 2019 17:22
 *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer

 Hi everyone,

 I'm very happy to announce that Zili Chen (some of you might also know
 him as Tison Kun) accepted the offer of the Flink PMC to become a committer
 of the Flink project.

 Zili Chen has been an active community member for almost 16 months
 now. He helped pushing the Flip-6 effort over the finish line, ported a lot
 of legacy code tests, removed a good part of the legacy code, contributed
 numerous fixes, is involved in the Flink's client API refactoring, drives
 the refactoring of Flink's HighAvailabilityServices and much more. Zili
 Chen also helped the community by PR reviews, reporting Flink issues,
 answering user mails and being very active on the dev mailing list.

 Congratulations Zili Chen!

 Best, Till
 (on behalf of the Flink PMC)

>>>

Re: instable checkpointing after migration to flink 1.8

[Bekir Oguz]

Hi Yun,
first of all, this reported problem looks like resolved for 2 days already,
right after we changed the type and number of our nodes to give more heap
to task managers and have more task-managers as well.

Previously, our job was configured as parallelism 70, which was distributed
to 14 task-managers (5 slots/tm) each having 9GB heap.
After our config change, the job is now using parallelism 100, distributed
to 20 task-managers (5 slots/tm) each having 18GB heap.

This job has a keyed process function which manages ~400 million records in
its state, and creating 1 timer per record scheduled to trigger in 6 months
to check if the record is eligible to be wiped out from state or not. So we
have 400M records + 400M timers.

For user state, we use RocksDB backend with incremental checkpointing
enabled and using s3 as an external checkpoint location. RocksDB backend is
configured with predefined FLASH_SSD_OPTIMIZED values.

Before our config change, flink was managing 400M/14=28,5M records + 28,5M
timers in each task-manager with 9GB heap.
And after the config change, this is 400M/20=20M records + 20M timers in
each task-manager with 18GB heap.

So, we have less state to manage per task manager, and have more heap.
Apparently this fixes(!) the problem of long checkpointing durations (15
minutes) happening occasionally.

Coming back to your points:
1. Snapshot timers are indeed using HEAP which is the default. We can set
it to ROCKSDB to see if that change has an impact on the end-to-end
checkpoint duration. Do you think this change will also reduce the heap
usage?
2. I have collected and shared those logs under /tmp directory earlier and
noticed that snapshotting happens very fast, finishing in a second. But
what I noticed was, compaction kicking in during the snapshotting phase of
a long (15 minutes) checkpoint. But still, the time spent for snapshotting
was 1 second. I guess compaction has no impact there. And still do not know
why it took 15 mins to acknowledge for one task slot.

I have another question regarding this problem and our use of timers. Is
this a good practice to use timers like we do? Does the flink timer service
support having this many timers? One timer per record, which is 400 million
for us.

Looks like our problem is solved for the time being, but may appear again
since we still do not know the root cause.
About the use of timers: Could you please share your opinion on our timer
setup and maybe support us on my question on switching timers to use
rocksdb instead of heap?

Thanks a lot,
Bekir Oguz


On Thu, 5 Sep 2019 at 19:55, Yun Tang  wrote:

> Hi Bekir
>
> From what I could see, there should be two main factors influencing your
> time of sync execution checkpoint within that task.
>
>1. Snapshot timers in heap to S3 [1] (network IO)
>2. Creating local RocksDB checkpoint on disk [2] (disk IO)
>
> For the first part, unfortunately, there is no log or metrics could detect
> how long it takes.
> For the second part, you could login the machine where running that task,
> and find logs of RocksDB (default DB folder is
> {io.tmp.dirs}/flink-io-xxx/job-xxx/db and the log file name is *LOG*).
> You could check the interval of logs between "Started the snapshot process
> -- creating snapshot in directory" and "Snapshot DONE" to know how long
> RocksDB takes to create checkpoint on local disk.
>
> If we configure "state.backend.rocksdb.timer-service.factory" to
> "ROCKSDB", we could avoid the 1st part of time and this might be a solution
> to your problem. But to be honest, the implementation of timer snapshot
> code almost stay the same for Flink-1.6 and Flink-1.8 and should not be a
> regression.
>
> [1]
> https://github.com/apache/flink/blob/ba1cd43b6ed42070fd9013159a5b65adb18b3975/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java#L453
> [2]
> https://github.com/apache/flink/blob/ba1cd43b6ed42070fd9013159a5b65adb18b3975/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksIncrementalSnapshotStrategy.java#L249
>
> Best
> Yun Tang
> --
> *From:* Congxian Qiu 
> *Sent:* Thursday, September 5, 2019 10:38
> *To:* Bekir Oguz 
> *Cc:* Stephan Ewen ; dev ; Niels
> Alebregtse ; Vladislav Bakayev <
> vladislav.baka...@persgroep.net>
> *Subject:* Re: instable checkpointing after migration to flink 1.8
>
> Another information from our private emails
>
> there ALWAYS have Kafka AbstractCoordinator logs about lost connection to
> Kafka at the same time we have the checkpoints confirmed. Bekir checked the
> Kafka broker log, but did not find any interesting things there.
>
> Best,
> Congxian
>
>
> Congxian Qiu  于2019年9月5日周四 上午10:26写道：
>
> > Hi Bekir,
> >
> > If it is the storage place for timers, for RocksDBStateBackend, timers
> can
> > be stored in Heap or RocksDB[1][2]
> > [1]
> >
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/l

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[SHI Xiaogang]

Congratulations!

Regards,
Xiaogang

Guowei Ma  于2019年9月11日周三 下午7:07写道：

> Congratulations Zili !
>
> Best,
> Guowei
>
>
> Fabian Hueske  于2019年9月11日周三 下午7:02写道：
>
>> Congrats Zili Chen :-)
>>
>> Cheers, Fabian
>>
>> Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu > >:
>>
>>> Congrats Zili!
>>>
>>> Thanks,
>>> Biao /'bɪ.aʊ/
>>>
>>>
>>>
>>> On Wed, 11 Sep 2019 at 18:43, Oytun Tez  wrote:
>>>
 Congratulations!

 ---
 Oytun Tez

 *M O T A W O R D*
 The World's Fastest Human Translation Platform.
 oy...@motaword.com — www.motaword.com


 On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy  wrote:

> Congratulations!
>
>
> Best,
>
> Jiayi Liao
>
>  Original Message
> *Sender:* Till Rohrmann
> *Recipient:* dev; user
> *Date:* Wednesday, Sep 11, 2019 17:22
> *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer
>
> Hi everyone,
>
> I'm very happy to announce that Zili Chen (some of you might also
> know him as Tison Kun) accepted the offer of the Flink PMC to become a
> committer of the Flink project.
>
> Zili Chen has been an active community member for almost 16 months
> now. He helped pushing the Flip-6 effort over the finish line, ported a 
> lot
> of legacy code tests, removed a good part of the legacy code, contributed
> numerous fixes, is involved in the Flink's client API refactoring, drives
> the refactoring of Flink's HighAvailabilityServices and much more. Zili
> Chen also helped the community by PR reviews, reporting Flink issues,
> answering user mails and being very active on the dev mailing list.
>
> Congratulations Zili Chen!
>
> Best, Till
> (on behalf of the Flink PMC)
>

[jira] [Created] (FLINK-14053) blink planner dense_rank corner case bug

[jackylau (Jira)]

jackylau created FLINK-14053:


 Summary: blink planner dense_rank corner case bug
 Key: FLINK-14053
 URL: https://issues.apache.org/jira/browse/FLINK-14053
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.9.0
Reporter: jackylau
 Fix For: 1.10.0


sql :

val rank =
 """
 |SELECT
 | gradeId,
 | classId,
 | stuId,
 | score,
 | dense_rank() OVER (PARTITION BY gradeId, classId ORDER BY score asc) as 
dense_rank_num
 |FROM student
 |
 """.stripMargin

sample date:

row("grade2", "class2", "0006", 90),
row("grade1", "class2", "0007", 90),
row("grade1", "class1", "0001", 95),
row("grade1", "class1", "0002", 94),
row("grade1", "class1", "0003", 97),
row("grade1", "class1", "0004", 95),
row("grade1", "class1", "0005", 0)

the dense_rank ranks from 0, but it should be from 1

 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

[Fabian Hueske]

Hi all,

I'd like to add my opinion on this topic as well ;-)

In general, I think overriding built-in function with temp functions has a
couple of benefits but also a few challenges:

* Users can reimplement the behavior of a built-in functions of a different
system, e.g., for backward compatibility after a migration.
* I don't think that "accidental" overrides and surprising semantics are an
issue or dangerous. The user registered the temp function in the same
session and should therefore be aware of the changed semantics.
* I see that not all built-in functions can be overridden, like the CAST
example that Dawid gave. However, I think these should be a small fraction
and such functions could be blacklisted. Sure, that's not super consistent,
but should (IMO) not be a big issue in practice.
* Temp functions should be easy to use. Requiring a 3-part addressing makes
them a lot less user friendly, IMO. Users need to think about what catalog
and db to choose when registering them. Also using a temp function in a
query becomes less convenient. Moreover, I agree with Bowen's concerns that
a 3-part addressing scheme reduces the temporal appearance of the function.

>From the three possible solutions, my preference order is
1) 1-part address with override of built-in
2) 1-part address without override of built-in
3) 3-part address

Regarding the issue of external built-in functions, I don't think that
Timo's proposal of modules is fully orthogonal to this discussion.
A Hive function module could be an alternative to offering Hive functions
as part of Hive's catalog.
>From a user's point of view, I think that modules would be a "cleaner"
integration ("Why do I need a Hive catalog if all I want to do is apply a
Hive function on a Kafka table?").
However, the module approach clearly has the problem of dealing with
same-named functions in different modules (e.g., a Hive function and a
Flink built-in function).
The catalog approach as the benefit that functions can be addressed like
hiveCat::func (or a similar path).

I'm not sure what's the best solution here.

Cheers,
Fabian


Am Mo., 9. Sept. 2019 um 06:30 Uhr schrieb Bowen Li :

> Hi,
>
> W.r.t temp functions, I feel both options have their benefits and can
> theoretically achieve similar functionalities one way or another. In the
> end, it's more about use cases, users habits, and trade-offs.
>
> Re> Not always users are in full control of the catalog functions. There is
> also the case where different teams manage the catalog & use the catalog.
>
> Temp functions live within a session, and not within a catalog. Having
> 3-part paths may implies temp functions are tied to a catalog in two
> aspects.
> 1) it may indicate each catalog manages their temp functions, which is not
> true as we seem all agree they should reside at a central place, either in
> FunctionCatalog or CatalogManager
> 2) it may indicate there's some access control. When users are forbidden to
> manipulate some objects in the catalog that's managed by other teams, but
> are allowed to manipulate some other objects (temp functions in this case)
> belonging to the catalog in namespaces, users may think we introduced extra
> complexity and confusion with some kind of access control into the problem.
> It doesn't feel intuitive enough for end users.
>
> Thus, I'd be in favor of 1-part path for temporary functions, and other
> temp objects.
>
> Thanks,
> Bowen
>
>
>
> On Fri, Sep 6, 2019 at 2:16 AM Dawid Wysakowicz 
> wrote:
>
> > I agree the consequences of the decision are substantial. Let's see what
> > others think.
> >
> > -- Catalog functions are defined by users, and we suppose they can
> > drop/alter it in any way they want. Thus, overwriting a catalog function
> > doesn't seem to be a strong use case that we should be concerned about.
> > Rather, there are known use case for overwriting built-in functions.
> >
> > Not always users are in full control of the catalog functions. There is
> > also the case where different teams manage the catalog & use the catalog.
> > As for overriding built-in functions with 3-part approach user can always
> > use an equally named function from a catalog. E.g. to override
> >
> > *SELECT explode(arr) FROM ...*
> >
> > user can always write:
> >
> > *SELECT db.explode(arr) FROM ...*
> >
> > Best,
> >
> > Dawid
> > On 06/09/2019 10:54, Xuefu Z wrote:
> >
> > Hi Dawid,
> >
> > Thank you for your summary. While the only difference in the two
> proposals
> > is one- or three-part in naming, the consequence would be substantial.
> >
> > To me, there are two major use cases of temporary functions compared to
> > persistent ones:
> > 1. Temporary in nature and auto managed by the session. More often than
> > not, admin doesn't even allow user to create persistent functions.
> > 2. Provide an opportunity to overwriting system built-in functions.
> >
> > Since built-in functions has one-part name, requiring three-part name for
> > temporary functions eliminat

RE: [ANNOUNCE] Zili Chen becomes a Flink committer

[xingcanc]

Congratulations, Zili.

 

Best,

Xingcan

 

From: SHI Xiaogang  
Sent: Wednesday, September 11, 2019 7:43 AM
To: Guowei Ma 
Cc: Fabian Hueske ; Biao Liu ; Oytun Tez 
; bupt_ljy ; dev ; 
user ; Till Rohrmann 
Subject: Re: [ANNOUNCE] Zili Chen becomes a Flink committer

 

Congratulations!

 

Regards,

Xiaogang

 

Guowei Ma mailto:guowei@gmail.com> > 于2019年9月11日周三 
下午7:07写道：

Congratulations Zili !




Best,

Guowei

 

 

Fabian Hueske mailto:fhue...@gmail.com> > 于2019年9月11日周三 
下午7:02写道：

Congrats Zili Chen :-)

 

Cheers, Fabian

 

Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu mailto:mmyy1...@gmail.com> >:

Congrats Zili! 


 

Thanks,

Biao /'bɪ.aʊ/

 

 

 

On Wed, 11 Sep 2019 at 18:43, Oytun Tez mailto:oy...@motaword.com> > wrote:

Congratulations!


 

---

Oytun Tez

 

M O T A W O R D

The World's Fastest Human Translation Platform.

  oy...@motaword.com —   
www.motaword.com

 

 

On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy mailto:bupt_...@163.com> > wrote:

Congratulations!

 

Best,

Jiayi Liao

 

 Original Message  

Sender: Till Rohrmannmailto:trohrm...@apache.org> >

Recipient: devmailto:dev@flink.apache.org> >; 
usermailto:u...@flink.apache.org> >

Date: Wednesday, Sep 11, 2019 17:22

Subject: [ANNOUNCE] Zili Chen becomes a Flink committer

 

Hi everyone,

 

I'm very happy to announce that Zili Chen (some of you might also know him as 
Tison Kun) accepted the offer of the Flink PMC to become a committer of the 
Flink project.

 

Zili Chen has been an active community member for almost 16 months now. He 
helped pushing the Flip-6 effort over the finish line, ported a lot of legacy 
code tests, removed a good part of the legacy code, contributed numerous fixes, 
is involved in the Flink's client API refactoring, drives the refactoring of 
Flink's HighAvailabilityServices and much more. Zili Chen also helped the 
community by PR reviews, reporting Flink issues, answering user mails and being 
very active on the dev mailing list.

 

Congratulations Zili Chen!

 

Best, Till 

(on behalf of the Flink PMC)

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

[Dawid Wysakowicz]

Hi Fabian,
Thank you for your response.
Regarding the temporary function, just wanted to clarify one thing: the
3-part identifier does not mean the user always has to provide the catalog
& database explicitly. The same way user does not have to provide them in
e.g. when creating permanent table, view etc. It means though functions are
always stored within a database. The same way as all the permanent objects
and other temporary objects(tables, views). If not given explicitly the
current catalog & database would be used, both in the create statement or
when using the function.

Point taken though your preference would be to support overriding built-in
functions.

Best,
Dawid

On Wed, 11 Sep 2019, 21:14 Fabian Hueske,  wrote:

> Hi all,
>
> I'd like to add my opinion on this topic as well ;-)
>
> In general, I think overriding built-in function with temp functions has a
> couple of benefits but also a few challenges:
>
> * Users can reimplement the behavior of a built-in functions of a different
> system, e.g., for backward compatibility after a migration.
> * I don't think that "accidental" overrides and surprising semantics are an
> issue or dangerous. The user registered the temp function in the same
> session and should therefore be aware of the changed semantics.
> * I see that not all built-in functions can be overridden, like the CAST
> example that Dawid gave. However, I think these should be a small fraction
> and such functions could be blacklisted. Sure, that's not super consistent,
> but should (IMO) not be a big issue in practice.
> * Temp functions should be easy to use. Requiring a 3-part addressing makes
> them a lot less user friendly, IMO. Users need to think about what catalog
> and db to choose when registering them. Also using a temp function in a
> query becomes less convenient. Moreover, I agree with Bowen's concerns that
> a 3-part addressing scheme reduces the temporal appearance of the function.
>
> From the three possible solutions, my preference order is
> 1) 1-part address with override of built-in
> 2) 1-part address without override of built-in
> 3) 3-part address
>
> Regarding the issue of external built-in functions, I don't think that
> Timo's proposal of modules is fully orthogonal to this discussion.
> A Hive function module could be an alternative to offering Hive functions
> as part of Hive's catalog.
> From a user's point of view, I think that modules would be a "cleaner"
> integration ("Why do I need a Hive catalog if all I want to do is apply a
> Hive function on a Kafka table?").
> However, the module approach clearly has the problem of dealing with
> same-named functions in different modules (e.g., a Hive function and a
> Flink built-in function).
> The catalog approach as the benefit that functions can be addressed like
> hiveCat::func (or a similar path).
>
> I'm not sure what's the best solution here.
>
> Cheers,
> Fabian
>
>
> Am Mo., 9. Sept. 2019 um 06:30 Uhr schrieb Bowen Li :
>
> > Hi,
> >
> > W.r.t temp functions, I feel both options have their benefits and can
> > theoretically achieve similar functionalities one way or another. In the
> > end, it's more about use cases, users habits, and trade-offs.
> >
> > Re> Not always users are in full control of the catalog functions. There
> is
> > also the case where different teams manage the catalog & use the catalog.
> >
> > Temp functions live within a session, and not within a catalog. Having
> > 3-part paths may implies temp functions are tied to a catalog in two
> > aspects.
> > 1) it may indicate each catalog manages their temp functions, which is
> not
> > true as we seem all agree they should reside at a central place, either
> in
> > FunctionCatalog or CatalogManager
> > 2) it may indicate there's some access control. When users are forbidden
> to
> > manipulate some objects in the catalog that's managed by other teams, but
> > are allowed to manipulate some other objects (temp functions in this
> case)
> > belonging to the catalog in namespaces, users may think we introduced
> extra
> > complexity and confusion with some kind of access control into the
> problem.
> > It doesn't feel intuitive enough for end users.
> >
> > Thus, I'd be in favor of 1-part path for temporary functions, and other
> > temp objects.
> >
> > Thanks,
> > Bowen
> >
> >
> >
> > On Fri, Sep 6, 2019 at 2:16 AM Dawid Wysakowicz 
> > wrote:
> >
> > > I agree the consequences of the decision are substantial. Let's see
> what
> > > others think.
> > >
> > > -- Catalog functions are defined by users, and we suppose they can
> > > drop/alter it in any way they want. Thus, overwriting a catalog
> function
> > > doesn't seem to be a strong use case that we should be concerned about.
> > > Rather, there are known use case for overwriting built-in functions.
> > >
> > > Not always users are in full control of the catalog functions. There is
> > > also the case where different teams manage the catalog & use the
> catalog.
> > > As for overrid

[jira] [Created] (FLINK-14054) Enable checkpointing via job configuration

[Jun Qin (Jira)]

Jun Qin created FLINK-14054:
---

 Summary: Enable checkpointing via job configuration
 Key: FLINK-14054
 URL: https://issues.apache.org/jira/browse/FLINK-14054
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Configuration
Reporter: Jun Qin


Currently enabling checkpointing can only be done via the job code, see the 
following quote from this Flink 
[checkpointing|https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/state/checkpointing.html#enabling-and-configuring-checkpointing]
 doc:
{quote}By default, checkpointing is disabled. To enable checkpointing, call 
{{enableCheckpointing(n)}} on the {{StreamExecutionEnvironment}}, where _n_ is 
the checkpoint interval in milliseconds.
{quote}
This makes enabling checkingpointing after the job code has been released 
difficult: one has to change and rebuild the job code.

In addition, not only for developer, making checkpointing enabling configurable 
is also of interest for operation teams:
 * They may want to enable checkpointing for production but disable in test 
(e.g., to save storage space)
 * They may want to try out with and without checkpointing to evaluate the 
impact to the job behaviour and performance.  

Therefore, this request.  Thanks.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: instable checkpointing after migration to flink 1.8

[Yun Tang]

Hi Bekir

Changing the timer factory from HEAP to ROCKSDB would certainly help reduce 
your JVM heap usage. Since it would use RocksDB to store the timer state, you 
might come across performance regression as we need to poll timers from RocksDB 
instead of JVM heap.

From our experience, 20 million timers per task manager still acts a bit too 
much, could you reduce your window size to reduce the timers per window? By the 
way, timer coalescing [1] might be an idea to reduce timers. (This method could 
only take effect when user register timer currently).

[1] 
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#timer-coalescing

Best
Yun Tang


From: Bekir Oguz 
Sent: Wednesday, September 11, 2019 19:39
To: Yun Tang 
Cc: dev@flink.apache.org ; Stephan Ewen 
; Niels Alebregtse ; 
Vladislav Bakayev 
Subject: Re: instable checkpointing after migration to flink 1.8

Hi Yun,
first of all, this reported problem looks like resolved for 2 days already, 
right after we changed the type and number of our nodes to give more heap to 
task managers and have more task-managers as well.

Previously, our job was configured as parallelism 70, which was distributed to 
14 task-managers (5 slots/tm) each having 9GB heap.
After our config change, the job is now using parallelism 100, distributed to 
20 task-managers (5 slots/tm) each having 18GB heap.

This job has a keyed process function which manages ~400 million records in its 
state, and creating 1 timer per record scheduled to trigger in 6 months to 
check if the record is eligible to be wiped out from state or not. So we have 
400M records + 400M timers.

For user state, we use RocksDB backend with incremental checkpointing enabled 
and using s3 as an external checkpoint location. RocksDB backend is configured 
with predefined FLASH_SSD_OPTIMIZED values.

Before our config change, flink was managing 400M/14=28,5M records + 28,5M 
timers in each task-manager with 9GB heap.
And after the config change, this is 400M/20=20M records + 20M timers in each 
task-manager with 18GB heap.

So, we have less state to manage per task manager, and have more heap. 
Apparently this fixes(!) the problem of long checkpointing durations (15 
minutes) happening occasionally.

Coming back to your points:
1. Snapshot timers are indeed using HEAP which is the default. We can set it to 
ROCKSDB to see if that change has an impact on the end-to-end checkpoint 
duration. Do you think this change will also reduce the heap usage?
2. I have collected and shared those logs under /tmp directory earlier and 
noticed that snapshotting happens very fast, finishing in a second. But what I 
noticed was, compaction kicking in during the snapshotting phase of a long (15 
minutes) checkpoint. But still, the time spent for snapshotting was 1 second. I 
guess compaction has no impact there. And still do not know why it took 15 mins 
to acknowledge for one task slot.

I have another question regarding this problem and our use of timers. Is this a 
good practice to use timers like we do? Does the flink timer service support 
having this many timers? One timer per record, which is 400 million for us.

Looks like our problem is solved for the time being, but may appear again since 
we still do not know the root cause.
About the use of timers: Could you please share your opinion on our timer setup 
and maybe support us on my question on switching timers to use rocksdb instead 
of heap?

Thanks a lot,
Bekir Oguz


On Thu, 5 Sep 2019 at 19:55, Yun Tang 
mailto:myas...@live.com>> wrote:
Hi Bekir

From what I could see, there should be two main factors influencing your time 
of sync execution checkpoint within that task.

  1.  Snapshot timers in heap to S3 [1] (network IO)
  2.  Creating local RocksDB checkpoint on disk [2] (disk IO)

For the first part, unfortunately, there is no log or metrics could detect how 
long it takes.
For the second part, you could login the machine where running that task, and 
find logs of RocksDB (default DB folder is 
{io.tmp.dirs}/flink-io-xxx/job-xxx/db and the log file name is LOG). You could 
check the interval of logs between "Started the snapshot process -- creating 
snapshot in directory" and "Snapshot DONE" to know how long RocksDB takes to 
create checkpoint on local disk.

If we configure "state.backend.rocksdb.timer-service.factory" to "ROCKSDB", we 
could avoid the 1st part of time and this might be a solution to your problem. 
But to be honest, the implementation of timer snapshot code almost stay the 
same for Flink-1.6 and Flink-1.8 and should not be a regression.

[1] 
https://github.com/apache/flink/blob/ba1cd43b6ed42070fd9013159a5b65adb18b3975/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java#L453
[2] 
https://github.com/apache/flink/blob/ba1cd43b6ed42070fd9013159a5b65adb18b3975/flink-state-backends/flink-statebackend-rocksdb/src/main/jav

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

[Bowen Li]

Hi,

Thanks @Fabian @Dawid and everyone else for sharing your thoughts!

First, I'd like to take Hive built-in functions out of this FLIP to keep
our original scope and make it less controversial on a potential modular
approach. I will remove Hive built-in functions from the google doc.

Then the focus of debate is mainly function resolution order and temp
function namespace, which are somewhat related. I roughly summarized this
thread, and currently we are debating on two approaches with preference
from the following people:

Option 1:
Proposal: temp functions will be of 1-part path (function name only),
and can override built-in functions. The ambiguous function resolution
order is thus 1) temp functions 2) built-in functions 3) catalog functions
in the current catalog/database
Votes: Xuefu, Bowen, Fabian, Jark

Option 2:
Proposal: temp functions will be of 3-part path (with catalog,
database, and function name), and temp functions cannot override built-in
functions. The ambiguous function resolution order is thus 1) built-in
functions 2) temp functions (in 3-part path) 3) catalog functions in the
current catalog/database
Votes:  Dawid, Timo


Do you think we need a separate voting thread on the two options in the
community, or are we able to conclude from the above summary?



On Wed, Sep 11, 2019 at 8:09 AM Dawid Wysakowicz 
wrote:

> Hi Fabian,
> Thank you for your response.
> Regarding the temporary function, just wanted to clarify one thing: the
> 3-part identifier does not mean the user always has to provide the catalog
> & database explicitly. The same way user does not have to provide them in
> e.g. when creating permanent table, view etc. It means though functions are
> always stored within a database. The same way as all the permanent objects
> and other temporary objects(tables, views). If not given explicitly the
> current catalog & database would be used, both in the create statement or
> when using the function.
>
> Point taken though your preference would be to support overriding built-in
> functions.
>
> Best,
> Dawid
>
> On Wed, 11 Sep 2019, 21:14 Fabian Hueske,  wrote:
>
> > Hi all,
> >
> > I'd like to add my opinion on this topic as well ;-)
> >
> > In general, I think overriding built-in function with temp functions has
> a
> > couple of benefits but also a few challenges:
> >
> > * Users can reimplement the behavior of a built-in functions of a
> different
> > system, e.g., for backward compatibility after a migration.
> > * I don't think that "accidental" overrides and surprising semantics are
> an
> > issue or dangerous. The user registered the temp function in the same
> > session and should therefore be aware of the changed semantics.
> > * I see that not all built-in functions can be overridden, like the CAST
> > example that Dawid gave. However, I think these should be a small
> fraction
> > and such functions could be blacklisted. Sure, that's not super
> consistent,
> > but should (IMO) not be a big issue in practice.
> > * Temp functions should be easy to use. Requiring a 3-part addressing
> makes
> > them a lot less user friendly, IMO. Users need to think about what
> catalog
> > and db to choose when registering them. Also using a temp function in a
> > query becomes less convenient. Moreover, I agree with Bowen's concerns
> that
> > a 3-part addressing scheme reduces the temporal appearance of the
> function.
> >
> > From the three possible solutions, my preference order is
> > 1) 1-part address with override of built-in
> > 2) 1-part address without override of built-in
> > 3) 3-part address
> >
> > Regarding the issue of external built-in functions, I don't think that
> > Timo's proposal of modules is fully orthogonal to this discussion.
> > A Hive function module could be an alternative to offering Hive functions
> > as part of Hive's catalog.
> > From a user's point of view, I think that modules would be a "cleaner"
> > integration ("Why do I need a Hive catalog if all I want to do is apply a
> > Hive function on a Kafka table?").
> > However, the module approach clearly has the problem of dealing with
> > same-named functions in different modules (e.g., a Hive function and a
> > Flink built-in function).
> > The catalog approach as the benefit that functions can be addressed like
> > hiveCat::func (or a similar path).
> >
> > I'm not sure what's the best solution here.
> >
> > Cheers,
> > Fabian
> >
> >
> > Am Mo., 9. Sept. 2019 um 06:30 Uhr schrieb Bowen Li  >:
> >
> > > Hi,
> > >
> > > W.r.t temp functions, I feel both options have their benefits and can
> > > theoretically achieve similar functionalities one way or another. In
> the
> > > end, it's more about use cases, users habits, and trade-offs.
> > >
> > > Re> Not always users are in full control of the catalog functions.
> There
> > is
> > > also the case where different teams manage the catalog & use the
> catalog.
> > >
> > > Temp functions live within a session, and not within a catalog. H

[jira] [Created] (FLINK-14055) Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"

[Bowen Li (Jira)]

Bowen Li created FLINK-14055:


 Summary: Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
 Key: FLINK-14055
 URL: https://issues.apache.org/jira/browse/FLINK-14055
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Bowen Li


As FLINK-7151 adds basic function DDL to Flink, this ticket is to support 
dynamically loading functions from external source in function DDL with 
advanced syntax like 

 
{code:java}
CREATE FUNCTION func_name as class_name USING JAR/FILE/ACHIEVE 'xxx' [, 
JAR/FILE/ACHIEVE 'yyy'] ;

{code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (FLINK-14056) AtOm+Electron+TypedE-Mail

[KORAY DURGUT (Jira)]

KORAY DURGUT  created FLINK-14056:
-

 Summary: AtOm+Electron+TypedE-Mail
 Key: FLINK-14056
 URL: https://issues.apache.org/jira/browse/FLINK-14056
 Project: Flink
  Issue Type: Bug
  Components: API / Type Serialization System
Affects Versions: shaded-9.0
Reporter: KORAY DURGUT 
 Fix For: shaded-8.0






--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Checkpoint metrics.

[Jamie Grier]

Hey all,

I need to make sense of this behavior.  Any help would be appreciated.

Here’s an example of a set of Flink checkpoint metrics I don’t understand.
This is the first operator in a job and as you can see the end-to-end time
for the checkpoint is long, but it’s not explained by either sync, async,
or alignment times.  I’m not sure what to make of this.  It makes me think
I don’t understand the meaning of the metrics themselves.  In my
interpretation the end-to-end time should always be, roughly, the sum of
the other components — certainly in the case of a source task such as this.

Any thoughts or clarifications anyone can provide on this?  We have many
jobs with slow checkpoints that suffer from this sort of thing with metrics
that look similar.

-Jamie

Re: Checkpoint metrics.

[Seth Wiesman]

Great timing, I just debugged this on Monday. E2e time is checkpoint 
coordinator to checkpoint coordinator, so it includes RPC to the source and RPC 
from the operator back for the JM. 

Seth 

> On Sep 11, 2019, at 6:17 PM, Jamie Grier  wrote:
> 
> Hey all,
> 
> I need to make sense of this behavior.  Any help would be appreciated.
> 
> Here’s an example of a set of Flink checkpoint metrics I don’t understand.  
> This is the first operator in a job and as you can see the end-to-end time 
> for the checkpoint is long, but it’s not explained by either sync, async, or 
> alignment times.  I’m not sure what to make of this.  It makes me think I 
> don’t understand the meaning of the metrics themselves.  In my interpretation 
> the end-to-end time should always be, roughly, the sum of the other 
> components — certainly in the case of a source task such as this.
> 
> Any thoughts or clarifications anyone can provide on this?  We have many jobs 
> with slow checkpoints that suffer from this sort of thing with metrics that 
> look similar.
> 
> -Jamie
> 

[jira] [Created] (FLINK-14057) Add Remove Other Timers to TimerService

[Jesse Anderson (Jira)]

Jesse Anderson created FLINK-14057:
--

 Summary: Add Remove Other Timers to TimerService
 Key: FLINK-14057
 URL: https://issues.apache.org/jira/browse/FLINK-14057
 Project: Flink
  Issue Type: Improvement
Reporter: Jesse Anderson


The TimerService service has the ability to add timers with 
registerProcessingTimeTimer. This method can be called many times and have 
different timer times.

If you want to add a new timer and delete other timers, you have to keep track 
of all previous timer times and call deleteProcessingTimeTimer for each time. 
This method forces you to keep track of all previous (unexpired) timers for a 
key.

Instead, I suggest overloading registerProcessingTimeTimer with a second 
boolean argument that will remove all previous timers and set the new timer.

Note: although I'm using registerProcessingTimeTimer, this applies to 
registerEventTimeTimer as well.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Jark Wu]

Congratulations Zili!

Best,
Jark

On Wed, 11 Sep 2019 at 23:06,  wrote:

> Congratulations, Zili.
>
>
>
> Best,
>
> Xingcan
>
>
>
> *From:* SHI Xiaogang 
> *Sent:* Wednesday, September 11, 2019 7:43 AM
> *To:* Guowei Ma 
> *Cc:* Fabian Hueske ; Biao Liu ;
> Oytun Tez ; bupt_ljy ; dev <
> dev@flink.apache.org>; user ; Till Rohrmann <
> trohrm...@apache.org>
> *Subject:* Re: [ANNOUNCE] Zili Chen becomes a Flink committer
>
>
>
> Congratulations!
>
>
>
> Regards,
>
> Xiaogang
>
>
>
> Guowei Ma  于2019年9月11日周三 下午7:07写道：
>
> Congratulations Zili !
>
>
> Best,
>
> Guowei
>
>
>
>
>
> Fabian Hueske  于2019年9月11日周三 下午7:02写道：
>
> Congrats Zili Chen :-)
>
>
>
> Cheers, Fabian
>
>
>
> Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu :
>
> Congrats Zili!
>
>
>
> Thanks,
>
> Biao /'bɪ.aʊ/
>
>
>
>
>
>
>
> On Wed, 11 Sep 2019 at 18:43, Oytun Tez  wrote:
>
> Congratulations!
>
>
>
> ---
>
> Oytun Tez
>
>
>
> *M O T A W O R D*
>
> *The World's Fastest Human Translation Platform.*
>
> oy...@motaword.com — www.motaword.com
>
>
>
>
>
> On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy  wrote:
>
> Congratulations!
>
>
>
> Best,
>
> Jiayi Liao
>
>
>
>  Original Message
>
> *Sender:* Till Rohrmann
>
> *Recipient:* dev; user
>
> *Date:* Wednesday, Sep 11, 2019 17:22
>
> *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer
>
>
>
> Hi everyone,
>
>
>
> I'm very happy to announce that Zili Chen (some of you might also know
> him as Tison Kun) accepted the offer of the Flink PMC to become a committer
> of the Flink project.
>
>
>
> Zili Chen has been an active community member for almost 16 months now.
> He helped pushing the Flip-6 effort over the finish line, ported a lot of
> legacy code tests, removed a good part of the legacy code, contributed
> numerous fixes, is involved in the Flink's client API refactoring, drives
> the refactoring of Flink's HighAvailabilityServices and much more. Zili
> Chen also helped the community by PR reviews, reporting Flink issues,
> answering user mails and being very active on the dev mailing list.
>
>
>
> Congratulations Zili Chen!
>
>
>
> Best, Till
>
> (on behalf of the Flink PMC)
>
>

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Hequn Cheng]

Congratulations!

Best, Hequn

On Thu, Sep 12, 2019 at 9:24 AM Jark Wu  wrote:

> Congratulations Zili!
>
> Best,
> Jark
>
> On Wed, 11 Sep 2019 at 23:06,  wrote:
>
> > Congratulations, Zili.
> >
> >
> >
> > Best,
> >
> > Xingcan
> >
> >
> >
> > *From:* SHI Xiaogang 
> > *Sent:* Wednesday, September 11, 2019 7:43 AM
> > *To:* Guowei Ma 
> > *Cc:* Fabian Hueske ; Biao Liu ;
> > Oytun Tez ; bupt_ljy ; dev <
> > dev@flink.apache.org>; user ; Till Rohrmann <
> > trohrm...@apache.org>
> > *Subject:* Re: [ANNOUNCE] Zili Chen becomes a Flink committer
> >
> >
> >
> > Congratulations!
> >
> >
> >
> > Regards,
> >
> > Xiaogang
> >
> >
> >
> > Guowei Ma  于2019年9月11日周三 下午7:07写道：
> >
> > Congratulations Zili !
> >
> >
> > Best,
> >
> > Guowei
> >
> >
> >
> >
> >
> > Fabian Hueske  于2019年9月11日周三 下午7:02写道：
> >
> > Congrats Zili Chen :-)
> >
> >
> >
> > Cheers, Fabian
> >
> >
> >
> > Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu  >:
> >
> > Congrats Zili!
> >
> >
> >
> > Thanks,
> >
> > Biao /'bɪ.aʊ/
> >
> >
> >
> >
> >
> >
> >
> > On Wed, 11 Sep 2019 at 18:43, Oytun Tez  wrote:
> >
> > Congratulations!
> >
> >
> >
> > ---
> >
> > Oytun Tez
> >
> >
> >
> > *M O T A W O R D*
> >
> > *The World's Fastest Human Translation Platform.*
> >
> > oy...@motaword.com — www.motaword.com
> >
> >
> >
> >
> >
> > On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy  wrote:
> >
> > Congratulations!
> >
> >
> >
> > Best,
> >
> > Jiayi Liao
> >
> >
> >
> >  Original Message
> >
> > *Sender:* Till Rohrmann
> >
> > *Recipient:* dev; user
> >
> > *Date:* Wednesday, Sep 11, 2019 17:22
> >
> > *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer
> >
> >
> >
> > Hi everyone,
> >
> >
> >
> > I'm very happy to announce that Zili Chen (some of you might also know
> > him as Tison Kun) accepted the offer of the Flink PMC to become a
> committer
> > of the Flink project.
> >
> >
> >
> > Zili Chen has been an active community member for almost 16 months now.
> > He helped pushing the Flip-6 effort over the finish line, ported a lot of
> > legacy code tests, removed a good part of the legacy code, contributed
> > numerous fixes, is involved in the Flink's client API refactoring, drives
> > the refactoring of Flink's HighAvailabilityServices and much more. Zili
> > Chen also helped the community by PR reviews, reporting Flink issues,
> > answering user mails and being very active on the dev mailing list.
> >
> >
> >
> > Congratulations Zili Chen!
> >
> >
> >
> > Best, Till
> >
> > (on behalf of the Flink PMC)
> >
> >
>

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

[Becket Qin]

Hi Stephan,

Thanks for the volunteering to help.

Yes, the overhead would just be review capacity. In fact, I am not worrying
too much about the review capacity. That is just a one time cost. My
concern is mainly about the long term burden. Assume we have new source
interface ready in 1.10 with newly added Pulsar connectors in old
interface. Later on if we migrate Pulsar to new source interface, the old
Pulsar connector might be deprecated almost immediately after checked in,
but we may still have to maintain two code bases. For the existing
connectors, we have to do that anyways. But it would be good to avoid
introducing a new connector with the same problem.

Thanks,

Jiangjie (Becket) Qin

On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen  wrote:

> Hi all!
>
> Nice to see this lively discussion about the Pulsar connector.
> Some thoughts on the open questions:
>
> ## Contribute to Flink or maintain as a community package
>
> Looks like the discussion is more going towards contribution. I think that
> is good, especially if we think that we want to build a similarly deep
> integration with Pulsar as we have for example with Kafka. The connector
> already looks like a more thorough connector than many others we have in
> the repository.
>
> With either a repo split, or the new build system, I hope that the build
> overhead is not a problem.
>
> ## Committer Support
>
> Becket offered some help already, I can also help a bit. I hope that
> between us, we can cover this.
>
> ## Contribute now, or wait for FLIP-27
>
> As Becket said, FLIP-27 is actually making some PoC-ing progress, but will
> take 2 more months, I would estimate, before it is fully available.
>
> If we want to be on the safe side with the contribution, we should
> contribute the source sooner and adjust it later. That would also help us
> in case things get crazy towards the 1.10 feature freeze and it would be
> hard to find time to review the new changes.
> What would be the overhead of contributing now? Given that the code is
> already there, it looks like it would be only review capacity, right?
>
> Best,
> Stephan
>
> On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen 
> wrote:
>
> > Hi everyone!
> >
> > Thanks for your attention and the promotion of this work.
> >
> > We will prepare a FLIP as soon as possible for more specific discussions.
> >
> > For FLIP-27, it seems that we have not reached a consensus. Therefore,
> > I will explain all the functionalities of the existing connector in
> > the FLIP (including Source, Sink, and Catalog) to continue our
> > discussions in FLIP.
> >
> > Thanks for your kind help.
> >
> > Best,
> > Yijie
> >
> > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin  wrote:
> > >
> > > Hi Sijie,
> > >
> > > If we agree that the goal is to have Pulsar connector in 1.10, how
> about
> > we
> > > do the following:
> > >
> > > 0. Start a FLIP to add Pulsar connector to Flink main repo as it is a
> new
> > > public interface to Flink main repo.
> > > 1. Start to review the Pulsar sink right away as there is no change to
> > the
> > > sink interface so far.
> > > 2. Wait a little bit on FLIP-27. Flink 1.10 is going to be code freeze
> in
> > > late Nov and let's say we give a month to the development and review of
> > > Pulsar connector, we need to have FLIP-27 by late Oct. There are still
> 7
> > > weeks. Personally I think it is doable. If FLIP-27 is not ready by late
> > > Oct, we can review and check in Pulsar connector with the existing
> source
> > > interface. This means we will have Pulsar connector in Flink 1.10,
> either
> > > with or without FLIP-27.
> > >
> > > Because we are going to have Pulsar sink and source checked in
> > separately,
> > > it might make sense to have two FLIPs, one for Pulsar sink and another
> > for
> > > Pulsar source. And we can start the work on Pulsar sink right away.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo  wrote:
> > >
> > > > Thank you Bowen and Becket.
> > > >
> > > > What's the take from Flink community? Shall we wait for FLIP-27 or
> > shall we
> > > > proceed to next steps? And what the next steps are? :-)
> > > >
> > > > Thanks,
> > > > Sijie
> > > >
> > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li  wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I think having a Pulsar connector in Flink can be a good mutual
> > benefit
> > > > to
> > > > > both communities.
> > > > >
> > > > > Another perspective is that Pulsar connector is the 1st streaming
> > > > connector
> > > > > that integrates with Flink's metadata management system and Catalog
> > APIs.
> > > > > It'll be cool to see how the integration turns out and whether we
> > need to
> > > > > improve Flink Catalog stack, which are currently in Beta, to cater
> to
> > > > > streaming source/sink. Thus I'm in favor of merging Pulsar
> connector
> > into
> > > > > Flink 1.10.
> > > > >
> > > > > I'd suggest to submit smaller sized PRs, e.g. maybe one for basic
> > > > > 

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Rong Rong]

Congratulations Zili!

--
Rong

On Wed, Sep 11, 2019 at 6:26 PM Hequn Cheng  wrote:

> Congratulations!
>
> Best, Hequn
>
> On Thu, Sep 12, 2019 at 9:24 AM Jark Wu  wrote:
>
>> Congratulations Zili!
>>
>> Best,
>> Jark
>>
>> On Wed, 11 Sep 2019 at 23:06,  wrote:
>>
>> > Congratulations, Zili.
>> >
>> >
>> >
>> > Best,
>> >
>> > Xingcan
>> >
>> >
>> >
>> > *From:* SHI Xiaogang 
>> > *Sent:* Wednesday, September 11, 2019 7:43 AM
>> > *To:* Guowei Ma 
>> > *Cc:* Fabian Hueske ; Biao Liu ;
>> > Oytun Tez ; bupt_ljy ; dev <
>> > dev@flink.apache.org>; user ; Till Rohrmann <
>> > trohrm...@apache.org>
>> > *Subject:* Re: [ANNOUNCE] Zili Chen becomes a Flink committer
>> >
>> >
>> >
>> > Congratulations!
>> >
>> >
>> >
>> > Regards,
>> >
>> > Xiaogang
>> >
>> >
>> >
>> > Guowei Ma  于2019年9月11日周三 下午7:07写道：
>> >
>> > Congratulations Zili !
>> >
>> >
>> > Best,
>> >
>> > Guowei
>> >
>> >
>> >
>> >
>> >
>> > Fabian Hueske  于2019年9月11日周三 下午7:02写道：
>> >
>> > Congrats Zili Chen :-)
>> >
>> >
>> >
>> > Cheers, Fabian
>> >
>> >
>> >
>> > Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu <
>> mmyy1...@gmail.com>:
>> >
>> > Congrats Zili!
>> >
>> >
>> >
>> > Thanks,
>> >
>> > Biao /'bɪ.aʊ/
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Wed, 11 Sep 2019 at 18:43, Oytun Tez  wrote:
>> >
>> > Congratulations!
>> >
>> >
>> >
>> > ---
>> >
>> > Oytun Tez
>> >
>> >
>> >
>> > *M O T A W O R D*
>> >
>> > *The World's Fastest Human Translation Platform.*
>> >
>> > oy...@motaword.com — www.motaword.com
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy  wrote:
>> >
>> > Congratulations!
>> >
>> >
>> >
>> > Best,
>> >
>> > Jiayi Liao
>> >
>> >
>> >
>> >  Original Message
>> >
>> > *Sender:* Till Rohrmann
>> >
>> > *Recipient:* dev; user
>> >
>> > *Date:* Wednesday, Sep 11, 2019 17:22
>> >
>> > *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer
>> >
>> >
>> >
>> > Hi everyone,
>> >
>> >
>> >
>> > I'm very happy to announce that Zili Chen (some of you might also know
>> > him as Tison Kun) accepted the offer of the Flink PMC to become a
>> committer
>> > of the Flink project.
>> >
>> >
>> >
>> > Zili Chen has been an active community member for almost 16 months now.
>> > He helped pushing the Flip-6 effort over the finish line, ported a lot
>> of
>> > legacy code tests, removed a good part of the legacy code, contributed
>> > numerous fixes, is involved in the Flink's client API refactoring,
>> drives
>> > the refactoring of Flink's HighAvailabilityServices and much more. Zili
>> > Chen also helped the community by PR reviews, reporting Flink issues,
>> > answering user mails and being very active on the dev mailing list.
>> >
>> >
>> >
>> > Congratulations Zili Chen!
>> >
>> >
>> >
>> > Best, Till
>> >
>> > (on behalf of the Flink PMC)
>> >
>> >
>>
>

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Yangze Guo]

Congratulations!

Best,
Yangze Guo

On Thu, Sep 12, 2019 at 9:35 AM Rong Rong  wrote:
>
> Congratulations Zili!
>
> --
> Rong
>
> On Wed, Sep 11, 2019 at 6:26 PM Hequn Cheng  wrote:
>
> > Congratulations!
> >
> > Best, Hequn
> >
> > On Thu, Sep 12, 2019 at 9:24 AM Jark Wu  wrote:
> >
> >> Congratulations Zili!
> >>
> >> Best,
> >> Jark
> >>
> >> On Wed, 11 Sep 2019 at 23:06,  wrote:
> >>
> >> > Congratulations, Zili.
> >> >
> >> >
> >> >
> >> > Best,
> >> >
> >> > Xingcan
> >> >
> >> >
> >> >
> >> > *From:* SHI Xiaogang 
> >> > *Sent:* Wednesday, September 11, 2019 7:43 AM
> >> > *To:* Guowei Ma 
> >> > *Cc:* Fabian Hueske ; Biao Liu ;
> >> > Oytun Tez ; bupt_ljy ; dev <
> >> > dev@flink.apache.org>; user ; Till Rohrmann <
> >> > trohrm...@apache.org>
> >> > *Subject:* Re: [ANNOUNCE] Zili Chen becomes a Flink committer
> >> >
> >> >
> >> >
> >> > Congratulations!
> >> >
> >> >
> >> >
> >> > Regards,
> >> >
> >> > Xiaogang
> >> >
> >> >
> >> >
> >> > Guowei Ma  于2019年9月11日周三 下午7:07写道：
> >> >
> >> > Congratulations Zili !
> >> >
> >> >
> >> > Best,
> >> >
> >> > Guowei
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Fabian Hueske  于2019年9月11日周三 下午7:02写道：
> >> >
> >> > Congrats Zili Chen :-)
> >> >
> >> >
> >> >
> >> > Cheers, Fabian
> >> >
> >> >
> >> >
> >> > Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu <
> >> mmyy1...@gmail.com>:
> >> >
> >> > Congrats Zili!
> >> >
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > Biao /'bɪ.aʊ/
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Wed, 11 Sep 2019 at 18:43, Oytun Tez  wrote:
> >> >
> >> > Congratulations!
> >> >
> >> >
> >> >
> >> > ---
> >> >
> >> > Oytun Tez
> >> >
> >> >
> >> >
> >> > *M O T A W O R D*
> >> >
> >> > *The World's Fastest Human Translation Platform.*
> >> >
> >> > oy...@motaword.com — www.motaword.com
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy  wrote:
> >> >
> >> > Congratulations!
> >> >
> >> >
> >> >
> >> > Best,
> >> >
> >> > Jiayi Liao
> >> >
> >> >
> >> >
> >> >  Original Message
> >> >
> >> > *Sender:* Till Rohrmann
> >> >
> >> > *Recipient:* dev; user
> >> >
> >> > *Date:* Wednesday, Sep 11, 2019 17:22
> >> >
> >> > *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer
> >> >
> >> >
> >> >
> >> > Hi everyone,
> >> >
> >> >
> >> >
> >> > I'm very happy to announce that Zili Chen (some of you might also know
> >> > him as Tison Kun) accepted the offer of the Flink PMC to become a
> >> committer
> >> > of the Flink project.
> >> >
> >> >
> >> >
> >> > Zili Chen has been an active community member for almost 16 months now.
> >> > He helped pushing the Flip-6 effort over the finish line, ported a lot
> >> of
> >> > legacy code tests, removed a good part of the legacy code, contributed
> >> > numerous fixes, is involved in the Flink's client API refactoring,
> >> drives
> >> > the refactoring of Flink's HighAvailabilityServices and much more. Zili
> >> > Chen also helped the community by PR reviews, reporting Flink issues,
> >> > answering user mails and being very active on the dev mailing list.
> >> >
> >> >
> >> >
> >> > Congratulations Zili Chen!
> >> >
> >> >
> >> >
> >> > Best, Till
> >> >
> >> > (on behalf of the Flink PMC)
> >> >
> >> >
> >>
> >

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Paul Lam]

Congratulations Zili！

Best,
Paul Lam

> 在 2019年9月12日，09:34，Rong Rong  写道：
> 
> Congratulations Zili!
> 
> --
> Rong
> 
> On Wed, Sep 11, 2019 at 6:26 PM Hequn Cheng  > wrote:
> Congratulations!
> 
> Best, Hequn
> 
> On Thu, Sep 12, 2019 at 9:24 AM Jark Wu  > wrote:
> Congratulations Zili!
> 
> Best,
> Jark
> 
> On Wed, 11 Sep 2019 at 23:06,  > wrote:
> 
> > Congratulations, Zili.
> >
> >
> >
> > Best,
> >
> > Xingcan
> >
> >
> >
> > *From:* SHI Xiaogang  > >
> > *Sent:* Wednesday, September 11, 2019 7:43 AM
> > *To:* Guowei Ma mailto:guowei@gmail.com>>
> > *Cc:* Fabian Hueske mailto:fhue...@gmail.com>>; Biao 
> > Liu mailto:mmyy1...@gmail.com>>;
> > Oytun Tez mailto:oy...@motaword.com>>; bupt_ljy 
> > mailto:bupt_...@163.com>>; dev <
> > dev@flink.apache.org >; user 
> > mailto:u...@flink.apache.org>>; Till Rohrmann <
> > trohrm...@apache.org >
> > *Subject:* Re: [ANNOUNCE] Zili Chen becomes a Flink committer
> >
> >
> >
> > Congratulations!
> >
> >
> >
> > Regards,
> >
> > Xiaogang
> >
> >
> >
> > Guowei Ma mailto:guowei@gmail.com>> 
> > 于2019年9月11日周三 下午7:07写道：
> >
> > Congratulations Zili !
> >
> >
> > Best,
> >
> > Guowei
> >
> >
> >
> >
> >
> > Fabian Hueske mailto:fhue...@gmail.com>> 于2019年9月11日周三 
> > 下午7:02写道：
> >
> > Congrats Zili Chen :-)
> >
> >
> >
> > Cheers, Fabian
> >
> >
> >
> > Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu  > >:
> >
> > Congrats Zili!
> >
> >
> >
> > Thanks,
> >
> > Biao /'bɪ.aʊ/
> >
> >
> >
> >
> >
> >
> >
> > On Wed, 11 Sep 2019 at 18:43, Oytun Tez  > > wrote:
> >
> > Congratulations!
> >
> >
> >
> > ---
> >
> > Oytun Tez
> >
> >
> >
> > *M O T A W O R D*
> >
> > *The World's Fastest Human Translation Platform.*
> >
> > oy...@motaword.com  — www.motaword.com 
> > 
> >
> >
> >
> >
> >
> > On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy  > > wrote:
> >
> > Congratulations!
> >
> >
> >
> > Best,
> >
> > Jiayi Liao
> >
> >
> >
> >  Original Message
> >
> > *Sender:* Till Rohrmannmailto:trohrm...@apache.org>>
> >
> > *Recipient:* devmailto:dev@flink.apache.org>>; 
> > usermailto:u...@flink.apache.org>>
> >
> > *Date:* Wednesday, Sep 11, 2019 17:22
> >
> > *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer
> >
> >
> >
> > Hi everyone,
> >
> >
> >
> > I'm very happy to announce that Zili Chen (some of you might also know
> > him as Tison Kun) accepted the offer of the Flink PMC to become a committer
> > of the Flink project.
> >
> >
> >
> > Zili Chen has been an active community member for almost 16 months now.
> > He helped pushing the Flip-6 effort over the finish line, ported a lot of
> > legacy code tests, removed a good part of the legacy code, contributed
> > numerous fixes, is involved in the Flink's client API refactoring, drives
> > the refactoring of Flink's HighAvailabilityServices and much more. Zili
> > Chen also helped the community by PR reviews, reporting Flink issues,
> > answering user mails and being very active on the dev mailing list.
> >
> >
> >
> > Congratulations Zili Chen!
> >
> >
> >
> > Best, Till
> >
> > (on behalf of the Flink PMC)
> >
> >

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Becket Qin]

Congrats, Zili!

On Thu, Sep 12, 2019 at 9:39 AM Paul Lam  wrote:

> Congratulations Zili！
>
> Best,
> Paul Lam
>
> 在 2019年9月12日，09:34，Rong Rong  写道：
>
> Congratulations Zili!
>
> --
> Rong
>
> On Wed, Sep 11, 2019 at 6:26 PM Hequn Cheng  wrote:
>
>> Congratulations!
>>
>> Best, Hequn
>>
>> On Thu, Sep 12, 2019 at 9:24 AM Jark Wu  wrote:
>>
>>> Congratulations Zili!
>>>
>>> Best,
>>> Jark
>>>
>>> On Wed, 11 Sep 2019 at 23:06,  wrote:
>>>
>>> > Congratulations, Zili.
>>> >
>>> >
>>> >
>>> > Best,
>>> >
>>> > Xingcan
>>> >
>>> >
>>> >
>>> > *From:* SHI Xiaogang 
>>> > *Sent:* Wednesday, September 11, 2019 7:43 AM
>>> > *To:* Guowei Ma 
>>> > *Cc:* Fabian Hueske ; Biao Liu >> >;
>>> > Oytun Tez ; bupt_ljy ; dev <
>>> > dev@flink.apache.org>; user ; Till Rohrmann <
>>> > trohrm...@apache.org>
>>> > *Subject:* Re: [ANNOUNCE] Zili Chen becomes a Flink committer
>>> >
>>> >
>>> >
>>> > Congratulations!
>>> >
>>> >
>>> >
>>> > Regards,
>>> >
>>> > Xiaogang
>>> >
>>> >
>>> >
>>> > Guowei Ma  于2019年9月11日周三 下午7:07写道：
>>> >
>>> > Congratulations Zili !
>>> >
>>> >
>>> > Best,
>>> >
>>> > Guowei
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Fabian Hueske  于2019年9月11日周三 下午7:02写道：
>>> >
>>> > Congrats Zili Chen :-)
>>> >
>>> >
>>> >
>>> > Cheers, Fabian
>>> >
>>> >
>>> >
>>> > Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu <
>>> mmyy1...@gmail.com>:
>>> >
>>> > Congrats Zili!
>>> >
>>> >
>>> >
>>> > Thanks,
>>> >
>>> > Biao /'bɪ.aʊ/
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Wed, 11 Sep 2019 at 18:43, Oytun Tez  wrote:
>>> >
>>> > Congratulations!
>>> >
>>> >
>>> >
>>> > ---
>>> >
>>> > Oytun Tez
>>> >
>>> >
>>> >
>>> > *M O T A W O R D*
>>> >
>>> > *The World's Fastest Human Translation Platform.*
>>> >
>>> > oy...@motaword.com — www.motaword.com
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy  wrote:
>>> >
>>> > Congratulations!
>>> >
>>> >
>>> >
>>> > Best,
>>> >
>>> > Jiayi Liao
>>> >
>>> >
>>> >
>>> >  Original Message
>>> >
>>> > *Sender:* Till Rohrmann
>>> >
>>> > *Recipient:* dev; user
>>> >
>>> > *Date:* Wednesday, Sep 11, 2019 17:22
>>> >
>>> > *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer
>>> >
>>> >
>>> >
>>> > Hi everyone,
>>> >
>>> >
>>> >
>>> > I'm very happy to announce that Zili Chen (some of you might also know
>>> > him as Tison Kun) accepted the offer of the Flink PMC to become a
>>> committer
>>> > of the Flink project.
>>> >
>>> >
>>> >
>>> > Zili Chen has been an active community member for almost 16 months now.
>>> > He helped pushing the Flip-6 effort over the finish line, ported a lot
>>> of
>>> > legacy code tests, removed a good part of the legacy code, contributed
>>> > numerous fixes, is involved in the Flink's client API refactoring,
>>> drives
>>> > the refactoring of Flink's HighAvailabilityServices and much more. Zili
>>> > Chen also helped the community by PR reviews, reporting Flink issues,
>>> > answering user mails and being very active on the dev mailing list.
>>> >
>>> >
>>> >
>>> > Congratulations Zili Chen!
>>> >
>>> >
>>> >
>>> > Best, Till
>>> >
>>> > (on behalf of the Flink PMC)
>>> >
>>> >
>>>
>>
>

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

[Xintong Song]

True. It would be easier for the users to understand if JM and TM have
corresponding config options. Maybe we can have another FLIP addressing
this afterwards.

Thank you~

Xintong Song



On Mon, Sep 9, 2019 at 11:38 PM Stephan Ewen  wrote:

> One thing that I just came across: Some of these options should also have a
> corresponding value for the JobManager, like JVM overhead, metaspace,
> direct memory.
>
> On Fri, Sep 6, 2019 at 4:34 AM Xintong Song  wrote:
>
> > Thanks all for the votes.
> > So far, we have
> >
> >- 4 binding +1 votes (Stephan, Andrey, Till, Zhijiang)
> >- 2 un-binding +1 votes (Xintong, Yu)
> >- No -1 votes
> >
> > There are more than 3 binding +1 votes and no -1 votes, and the voting
> time
> > has past. According to the new bylaws, I'm happily to announce that
> FLIP-49
> > is approved to be adopted by Apache Flink.
> >
> > Regarding the minors mentioned during the voting, if there's no
> objection,
> > I would like to update the FLIP document with the followings
> >
> >- Exclude JVM Overhead from '-XX:MaxDirectMemorySize'
> >- Add a 'Follow-Up' section, with the follow-ups of web ui and
> >documentation issues
> >- Add a 'Limitation' section, with the Unsafe Java 12 support issue
> >
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Fri, Sep 6, 2019 at 10:28 AM Xintong Song 
> > wrote:
> >
> > > +1 (non-binding) from my side.
> > >
> > > @Yu, thanks for the vote.
> > > - The current FLIP document already mentioned the issue that Unsafe is
> > not
> > > supported in Java 12, in the section 'Unifying Explicit and Implicit
> > Memory
> > > Allocation'. It makes sense to me to emphasize this by adding a
> separate
> > > limitation section.
> > > - I think we should also update the FLIP document if we change the
> config
> > > names later in PRs. But I would not consider this as a major change to
> > the
> > > FLIP that requires another vote, especially when we already agreed
> during
> > > this vote to revisit the config names at implementation stage.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Fri, Sep 6, 2019 at 2:43 AM Yu Li  wrote:
> > >
> > >> +1 (non-binding)
> > >>
> > >> Minor:
> > >> 1. Is it worth a separate "Limitations" section to contain all
> relative
> > >> information like the Unsafe support issue in Java 12, just like many
> > other
> > >> FLIPs?
> > >> 2. About the config names, if we change them later in PR, does it mean
> > we
> > >> will need to update the FLIP document? If so, it seems we need another
> > >> vote
> > >> after the modification according to current bylaw? Or maybe we could
> > just
> > >> create a subpage under the FLIP and only re-vote on that part later?
> > >>
> > >> Thanks.
> > >>
> > >> Best Regards,
> > >> Yu
> > >>
> > >>
> > >> On Thu, 5 Sep 2019 at 00:52, Stephan Ewen  wrote:
> > >>
> > >> > Let's not block on config key names, just go ahead and we figure
> this
> > >> out
> > >> > concurrently or on the PR later.
> > >> >
> > >> >
> > >> > On Wed, Sep 4, 2019 at 3:48 PM Stephan Ewen 
> wrote:
> > >> >
> > >> > > Maybe to clear up confusion about my suggestion:
> > >> > >
> > >> > > I would vote to keep the name of the config parameter
> > >> > > "taskmanager.memory.network" because it is the same key as
> currently
> > >> > (good
> > >> > > to not break things unless good reason) and there currently is no
> > >> case or
> > >> > > even a concrete follow-up where we would actually differentiate
> > >> between
> > >> > > different types of network memory.
> > >> > >
> > >> > > I would suggest to not prematurely rename this because of
> something
> > >> that
> > >> > > might happen in the future. Experience shows that its better to do
> > >> these
> > >> > > things when the actual change comes.
> > >> > >
> > >> > > Side note: I am not sure "shuffle" is a good term in this
> context. I
> > >> have
> > >> > > so far only heard that in batch contexts, which is not what we do
> > >> here.
> > >> > One
> > >> > > more reason for me to not pre-maturely change names.
> > >> > >
> > >> > > On Wed, Sep 4, 2019 at 10:56 AM Xintong Song <
> tonysong...@gmail.com
> > >
> > >> > > wrote:
> > >> > >
> > >> > >> @till
> > >> > >>
> > >> > >> > Just to clarify Xintong, you suggest that Task off-heap memory
> > >> > >> represents
> > >> > >> > direct and native memory. Since we don't know how the user will
> > >> > allocate
> > >> > >> > the memory we will add this value to -XX:MaxDirectMemorySize so
> > >> that
> > >> > the
> > >> > >> > process won't fail if the user allocates only direct memory and
> > no
> > >> > >> native
> > >> > >> > memory. Is that correct?
> > >> > >> >
> > >> > >> Yes, this is what I mean.
> > >> > >>
> > >> > >>
> > >> > >> Thank you~
> > >> > >>
> > >> > >> Xintong Song
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >> On Wed, Sep 4, 2019 at 4:25 PM Till Rohrmann <
> trohrm...@apache.org
> > >
> > >> > >> wrote:
> > >> > >>
> > >> > >> > Just to clarify Xintong, y

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[zhijiang]

Congratulations Zili！
--
From:Becket Qin 
Send Time:2019年9月12日(星期四) 03:43
To:Paul Lam 
Cc:Rong Rong ; dev ; user 

Subject:Re: [ANNOUNCE] Zili Chen becomes a Flink committer

Congrats, Zili!
On Thu, Sep 12, 2019 at 9:39 AM Paul Lam  wrote:
Congratulations Zili！

Best,
Paul Lam

在 2019年9月12日，09:34，Rong Rong  写道：
Congratulations Zili!

--
Rong
On Wed, Sep 11, 2019 at 6:26 PM Hequn Cheng  wrote:
Congratulations!

Best, Hequn
On Thu, Sep 12, 2019 at 9:24 AM Jark Wu  wrote:
Congratulations Zili!

 Best,
 Jark

 On Wed, 11 Sep 2019 at 23:06,  wrote:

 > Congratulations, Zili.
 >
 >
 >
 > Best,
 >
 > Xingcan
 >
 >
 >
 > *From:* SHI Xiaogang 
 > *Sent:* Wednesday, September 11, 2019 7:43 AM
 > *To:* Guowei Ma 
 > *Cc:* Fabian Hueske ; Biao Liu ;
 > Oytun Tez ; bupt_ljy ; dev <
 > dev@flink.apache.org>; user ; Till Rohrmann <
 > trohrm...@apache.org>
 > *Subject:* Re: [ANNOUNCE] Zili Chen becomes a Flink committer
 >
 >
 >
 > Congratulations!
 >
 >
 >
 > Regards,
 >
 > Xiaogang
 >
 >
 >
 > Guowei Ma  于2019年9月11日周三 下午7:07写道：
 >
 > Congratulations Zili !
 >
 >
 > Best,
 >
 > Guowei
 >
 >
 >
 >
 >
 > Fabian Hueske  于2019年9月11日周三 下午7:02写道：
 >
 > Congrats Zili Chen :-)
 >
 >
 >
 > Cheers, Fabian
 >
 >
 >
 > Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu :
 >
 > Congrats Zili!
 >
 >
 >
 > Thanks,
 >
 > Biao /'bɪ.aʊ/
 >
 >
 >
 >
 >
 >
 >
 > On Wed, 11 Sep 2019 at 18:43, Oytun Tez  wrote:
 >
 > Congratulations!
 >
 >
 >
 > ---
 >
 > Oytun Tez
 >
 >
 >
 > *M O T A W O R D*
 >
 > *The World's Fastest Human Translation Platform.*
 >
 > oy...@motaword.com — www.motaword.com
 >
 >
 >
 >
 >
 > On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy  wrote:
 >
 > Congratulations!
 >
 >
 >
 > Best,
 >
 > Jiayi Liao
 >
 >
 >
 >  Original Message
 >
 > *Sender:* Till Rohrmann
 >
 > *Recipient:* dev; user
 >
 > *Date:* Wednesday, Sep 11, 2019 17:22
 >
 > *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer
 >
 >
 >
 > Hi everyone,
 >
 >
 >
 > I'm very happy to announce that Zili Chen (some of you might also know
 > him as Tison Kun) accepted the offer of the Flink PMC to become a committer
 > of the Flink project.
 >
 >
 >
 > Zili Chen has been an active community member for almost 16 months now.
 > He helped pushing the Flip-6 effort over the finish line, ported a lot of
 > legacy code tests, removed a good part of the legacy code, contributed
 > numerous fixes, is involved in the Flink's client API refactoring, drives
 > the refactoring of Flink's HighAvailabilityServices and much more. Zili
 > Chen also helped the community by PR reviews, reporting Flink issues,
 > answering user mails and being very active on the dev mailing list.
 >
 >
 >
 > Congratulations Zili Chen!
 >
 >
 >
 > Best, Till
 >
 > (on behalf of the Flink PMC)
 >
 >

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Yun Gao]

Congratulations Zili!

   Best,
   Yun


--
From:Yangze Guo 
Send Time:2019 Sep. 12 (Thu.) 09:38
To:dev 
Subject:Re: [ANNOUNCE] Zili Chen becomes a Flink committer

Congratulations!

Best,
Yangze Guo

On Thu, Sep 12, 2019 at 9:35 AM Rong Rong  wrote:
>
> Congratulations Zili!
>
> --
> Rong
>
> On Wed, Sep 11, 2019 at 6:26 PM Hequn Cheng  wrote:
>
> > Congratulations!
> >
> > Best, Hequn
> >
> > On Thu, Sep 12, 2019 at 9:24 AM Jark Wu  wrote:
> >
> >> Congratulations Zili!
> >>
> >> Best,
> >> Jark
> >>
> >> On Wed, 11 Sep 2019 at 23:06,  wrote:
> >>
> >> > Congratulations, Zili.
> >> >
> >> >
> >> >
> >> > Best,
> >> >
> >> > Xingcan
> >> >
> >> >
> >> >
> >> > *From:* SHI Xiaogang 
> >> > *Sent:* Wednesday, September 11, 2019 7:43 AM
> >> > *To:* Guowei Ma 
> >> > *Cc:* Fabian Hueske ; Biao Liu ;
> >> > Oytun Tez ; bupt_ljy ; dev <
> >> > dev@flink.apache.org>; user ; Till Rohrmann <
> >> > trohrm...@apache.org>
> >> > *Subject:* Re: [ANNOUNCE] Zili Chen becomes a Flink committer
> >> >
> >> >
> >> >
> >> > Congratulations!
> >> >
> >> >
> >> >
> >> > Regards,
> >> >
> >> > Xiaogang
> >> >
> >> >
> >> >
> >> > Guowei Ma  于2019年9月11日周三 下午7:07写道：
> >> >
> >> > Congratulations Zili !
> >> >
> >> >
> >> > Best,
> >> >
> >> > Guowei
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Fabian Hueske  于2019年9月11日周三 下午7:02写道：
> >> >
> >> > Congrats Zili Chen :-)
> >> >
> >> >
> >> >
> >> > Cheers, Fabian
> >> >
> >> >
> >> >
> >> > Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu <
> >> mmyy1...@gmail.com>:
> >> >
> >> > Congrats Zili!
> >> >
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > Biao /'bɪ.aʊ/
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Wed, 11 Sep 2019 at 18:43, Oytun Tez  wrote:
> >> >
> >> > Congratulations!
> >> >
> >> >
> >> >
> >> > ---
> >> >
> >> > Oytun Tez
> >> >
> >> >
> >> >
> >> > *M O T A W O R D*
> >> >
> >> > *The World's Fastest Human Translation Platform.*
> >> >
> >> > oy...@motaword.com — www.motaword.com
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy  wrote:
> >> >
> >> > Congratulations!
> >> >
> >> >
> >> >
> >> > Best,
> >> >
> >> > Jiayi Liao
> >> >
> >> >
> >> >
> >> >  Original Message
> >> >
> >> > *Sender:* Till Rohrmann
> >> >
> >> > *Recipient:* dev; user
> >> >
> >> > *Date:* Wednesday, Sep 11, 2019 17:22
> >> >
> >> > *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer
> >> >
> >> >
> >> >
> >> > Hi everyone,
> >> >
> >> >
> >> >
> >> > I'm very happy to announce that Zili Chen (some of you might also know
> >> > him as Tison Kun) accepted the offer of the Flink PMC to become a
> >> committer
> >> > of the Flink project.
> >> >
> >> >
> >> >
> >> > Zili Chen has been an active community member for almost 16 months now.
> >> > He helped pushing the Flip-6 effort over the finish line, ported a lot
> >> of
> >> > legacy code tests, removed a good part of the legacy code, contributed
> >> > numerous fixes, is involved in the Flink's client API refactoring,
> >> drives
> >> > the refactoring of Flink's HighAvailabilityServices and much more. Zili
> >> > Chen also helped the community by PR reviews, reporting Flink issues,
> >> > answering user mails and being very active on the dev mailing list.
> >> >
> >> >
> >> >
> >> > Congratulations Zili Chen!
> >> >
> >> >
> >> >
> >> > Best, Till
> >> >
> >> > (on behalf of the Flink PMC)
> >> >
> >> >
> >>
> >

Re: [VOTE] Release 1.8.2, release candidate #1

[Jark Wu]

+1 (non-binding)

- checked/verified signatures and hashes
- built from source, without Hadoop and using Scala 2.12
- checked that all POM files point to the same version
- started a cluster; WebUI is accessible, submit example jobs, no
suspicious log output
- manually verified the diff pom files between 1.8.1 and 1.8.2 to check
dependencies, looks good

Best,
Jark

On Wed, 11 Sep 2019 at 00:12, Till Rohrmann  wrote:

> +1 (binding)
>
> - verified checksums and signatures
> - no binary files in source release
> - built Flink from source release with Scala 2.12, running all tests
> - Verified that no new dependencies have been added
> - Executed simple example jobs locally (worked)
>
> Cheers,
> Till
>
> On Tue, Sep 10, 2019 at 8:00 AM Kurt Young  wrote:
>
> > +1 (binding)
> >
> > - build from source and passed all tests locally
> > - checked the difference between 1.8.1 and 1.8.2, no legal risk found
> > - went through all commits checked in between 1.8.1 and 1.8.2, make
> > sure all the issues set the proper "fixVersion" property
> >
> > Best,
> > Kurt
> >
> >
> > On Mon, Sep 9, 2019 at 8:45 PM Dian Fu  wrote:
> >
> > > +1 (non-binding)
> > >
> > > - built from source successfully (mvn clean install -DskipTests)
> > > - checked gpg signature and hashes of the source release and binary
> > > release packages
> > > - All artifacts have been deployed to the maven central repository
> > > - no new dependencies were added since 1.8.1
> > > - run a couple of tests in IDE success
> > >
> > > Regards,
> > > Dian
> > >
> > > > 在 2019年9月9日，下午2:28，jincheng sun  写道：
> > > >
> > > > +1 (binding)
> > > >
> > > > - checked signatures [SUCCESS]
> > > > - built from source without tests [SUCCESS]
> > > > - ran some tests in IDE [SUCCESS]
> > > > - start local cluster and submit word count example [SUCCESS]
> > > > - announcement PR for website looks good! (I have left a few
> comments)
> > > >
> > > > Best,
> > > > Jincheng
> > > >
> > > > Jark Wu  于2019年9月6日周五 下午8:47写道：
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> Please review and vote on the release candidate #1 for the version
> > > 1.8.2,
> > > >> as follows:
> > > >> [ ] +1, Approve the release
> > > >> [ ] -1, Do not approve the release (please provide specific
> comments)
> > > >>
> > > >>
> > > >> The complete staging area is available for your review, which
> > includes:
> > > >> * JIRA release notes [1],
> > > >> * the official Apache source release and binary convenience releases
> > to
> > > be
> > > >> deployed to dist.apache.org [2], which are signed with the key with
> > > >> fingerprint E2C45417BED5C104154F341085BACB5AEFAE3202 [3],
> > > >> * all artifacts to be deployed to the Maven Central Repository [4],
> > > >> * source code tag "release-1.8.2-rc1" [5],
> > > >> * website pull request listing the new release and adding
> announcement
> > > blog
> > > >> post [6].
> > > >>
> > > >> The vote will be open for at least 72 hours.
> > > >> Please cast your votes before *Sep. 11th 2019, 13:00 UTC*.
> > > >>
> > > >> It is adopted by majority approval, with at least 3 PMC affirmative
> > > votes.
> > > >>
> > > >> Thanks,
> > > >> Jark
> > > >>
> > > >> [1]
> > > >>
> > > >>
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12345670
> > > >> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.8.2-rc1/
> > > >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > >> [4]
> > > https://repository.apache.org/content/repositories/orgapacheflink-1262
> > > >> [5]
> > > >>
> > > >>
> > >
> >
> https://github.com/apache/flink/commit/6322618bb0f1b7942d86cb1b2b7bc55290d9e330
> > > >> [6] https://github.com/apache/flink-web/pull/262
> > > >>
> > >
> > >
> >
>

[RESULT] [VOTE] Release 1.8.2, release candidate #1

[Jark Wu]

I'm happy to announce that we have unanimously approved this release.

There are 5 approving votes, 3 of which are binding:
* jincheng (binding)
* Kurt (binding)
* Till (binding)
* Dian
* Jark

There are no disapproving votes.

Thanks everyone!

Re: [VOTE] FLIP-53: Fine Grained Operator Resource Management

[Xintong Song]

+1 from my side.

Thanks for bringing this up, Kurt. It's true that the accuracy loss of
floating values could cause problems in some cases. And I would consider
this as an implementation issue that can be discussed later in the PRs.

Thank you~

Xintong Song



On Sat, Sep 7, 2019 at 2:51 PM Kurt Young  wrote:

> +1 for FLIP-53.
>
> I would like to raise one minor concern regarding to implementing
> request absolute amount of memory case. Currently, it will be
> translated to a memory fraction during compile, and translate back
> to absolute value during execution. There is a risk that the user might
> get less than he requested due to floating point number problems.
>
> Best,
> Kurt
>
>
> On Fri, Sep 6, 2019 at 10:13 PM Andrey Zagrebin 
> wrote:
>
> > Thanks for starting the vote @Xintong
> >
> > +1 for the FLIP-53
> >
> > Best,
> > Andrey
> >
> > On Fri, Sep 6, 2019 at 3:53 PM Till Rohrmann 
> wrote:
> >
> > > Hi Xintong,
> > >
> > > thanks for starting this vote. The proposal looks good and, hence, +1
> for
> > > it.
> > >
> > > One comment I have is concerning the first implementation step. I would
> > > suggest to not add the flag allSourcesInSamePipelinedRegion to the
> > > ExecutionConfig because the ExecutionConfig is public API. Ideally we
> > keep
> > > this flag internal and don't expose it to the user.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Fri, Sep 6, 2019 at 1:47 PM Zhu Zhu  wrote:
> > >
> > > > Thanks Xintong for proposing this better resource management.
> > > > This helps a lot to users who want to better manage the job
> resources.
> > > And
> > > > would be even more useful if in the future we can have auto-tuning
> > > > mechanism for jobs.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks,
> > > > Zhu Zhu
> > > >
> > > > Xintong Song  于2019年9月6日周五 上午11:17写道：
> > > >
> > > > > Hi all,
> > > > >
> > > > > I would like to start the voting process for FLIP-53 [1], which is
> > > > > discussed and reached consensus in this thread [2].
> > > > >
> > > > > This voting will be open for at least 72 hours (excluding
> weekends).
> > > I'll
> > > > > try to close it Sep. 11, 04:00 UTC, unless there is an objection or
> > not
> > > > > enough votes.
> > > > >
> > > > > Thank you~
> > > > >
> > > > > Xintong Song
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
> > > > >
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-53-Fine-Grained-Resource-Management-td31831.html
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] FLIP-53: Fine Grained Operator Resource Management

[Xintong Song]

Thanks all for the votes.

So far, we have

   - 3 binding +1 votes (Till, Andrey and Kurt)
   - 2 un-binding +1 votes (Zhu and Xintong)
   - No -1 votes

There are 3 binding +1 votes and no -1 votes, and the voting time has past.
According to the community bylaws, I'm glad to announce that FLIP-53 is
approved to be adopted by Apache Flink.

Thank you~

Xintong Song



On Thu, Sep 12, 2019 at 10:31 AM Xintong Song  wrote:

> +1 from my side.
>
> Thanks for bringing this up, Kurt. It's true that the accuracy loss of
> floating values could cause problems in some cases. And I would consider
> this as an implementation issue that can be discussed later in the PRs.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Sat, Sep 7, 2019 at 2:51 PM Kurt Young  wrote:
>
>> +1 for FLIP-53.
>>
>> I would like to raise one minor concern regarding to implementing
>> request absolute amount of memory case. Currently, it will be
>> translated to a memory fraction during compile, and translate back
>> to absolute value during execution. There is a risk that the user might
>> get less than he requested due to floating point number problems.
>>
>> Best,
>> Kurt
>>
>>
>> On Fri, Sep 6, 2019 at 10:13 PM Andrey Zagrebin 
>> wrote:
>>
>> > Thanks for starting the vote @Xintong
>> >
>> > +1 for the FLIP-53
>> >
>> > Best,
>> > Andrey
>> >
>> > On Fri, Sep 6, 2019 at 3:53 PM Till Rohrmann 
>> wrote:
>> >
>> > > Hi Xintong,
>> > >
>> > > thanks for starting this vote. The proposal looks good and, hence, +1
>> for
>> > > it.
>> > >
>> > > One comment I have is concerning the first implementation step. I
>> would
>> > > suggest to not add the flag allSourcesInSamePipelinedRegion to the
>> > > ExecutionConfig because the ExecutionConfig is public API. Ideally we
>> > keep
>> > > this flag internal and don't expose it to the user.
>> > >
>> > > Cheers,
>> > > Till
>> > >
>> > > On Fri, Sep 6, 2019 at 1:47 PM Zhu Zhu  wrote:
>> > >
>> > > > Thanks Xintong for proposing this better resource management.
>> > > > This helps a lot to users who want to better manage the job
>> resources.
>> > > And
>> > > > would be even more useful if in the future we can have auto-tuning
>> > > > mechanism for jobs.
>> > > >
>> > > > +1 (non-binding)
>> > > >
>> > > > Thanks,
>> > > > Zhu Zhu
>> > > >
>> > > > Xintong Song  于2019年9月6日周五 上午11:17写道：
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > I would like to start the voting process for FLIP-53 [1], which is
>> > > > > discussed and reached consensus in this thread [2].
>> > > > >
>> > > > > This voting will be open for at least 72 hours (excluding
>> weekends).
>> > > I'll
>> > > > > try to close it Sep. 11, 04:00 UTC, unless there is an objection
>> or
>> > not
>> > > > > enough votes.
>> > > > >
>> > > > > Thank you~
>> > > > >
>> > > > > Xintong Song
>> > > > >
>> > > > >
>> > > > > [1]
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
>> > > > >
>> > > > > [2]
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-53-Fine-Grained-Resource-Management-td31831.html
>> > > > >
>> > > >
>> > >
>> >
>>
>

[jira] [Created] (FLINK-14058) FLIP-53 Fine Grained Operator Resource Management

[Xintong Song (Jira)]

Xintong Song created FLINK-14058:


 Summary: FLIP-53 Fine Grained Operator Resource Management
 Key: FLINK-14058
 URL: https://issues.apache.org/jira/browse/FLINK-14058
 Project: Flink
  Issue Type: New Feature
  Components: Runtime / Coordination
Affects Versions: 1.9.0
Reporter: Xintong Song
 Fix For: 1.10.0


This is the umbrella issue of 'FLIP-53: Fine Grained Operator Resource 
Management'.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (FLINK-14059) Introduce option allSourcesInSamePipelinedRegion in ExecutionConfig

[Xintong Song (Jira)]

Xintong Song created FLINK-14059:


 Summary: Introduce option allSourcesInSamePipelinedRegion in 
ExecutionConfig
 Key: FLINK-14059
 URL: https://issues.apache.org/jira/browse/FLINK-14059
 Project: Flink
  Issue Type: Sub-task
Reporter: Xintong Song


* Introduce option {{allSourcesInSamePipelinedRegion}} in {{ExecutionConfig}}
 * Set it to {{true}} by default
 * Set it to {{false}} for SQL/Table API bounded batch jobs by the Blink planner

This step should not introduce any behavior changes. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (FLINK-14061) Introduce managed memory fractions to StreamConfig

[Xintong Song (Jira)]

Xintong Song created FLINK-14061:


 Summary: Introduce managed memory fractions to StreamConfig
 Key: FLINK-14061
 URL: https://issues.apache.org/jira/browse/FLINK-14061
 Project: Flink
  Issue Type: Sub-task
Reporter: Xintong Song


Introduce {{fracManagedMemOnHeap}} and {{fracManagedMemOffHeap}} in 
{{StreamConfig}}, so they can be set by {{StreamingJobGraphGenerator}} and used 
by operators in runtime. 

This step should not introduce any behavior changes.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (FLINK-14063) Operators use fractions to decide how many managed memory to allocate

[Xintong Song (Jira)]

Xintong Song created FLINK-14063:


 Summary: Operators use fractions to decide how many managed memory 
to allocate
 Key: FLINK-14063
 URL: https://issues.apache.org/jira/browse/FLINK-14063
 Project: Flink
  Issue Type: Sub-task
Reporter: Xintong Song


* Operators allocate memory segments with the amount returned by 
{{MemoryManager#computeNumberOfPages}}.
 * Operators reserve memory with the amount returned by 
{{MemoryManager#computeMemorySize}}. 

This step activates the new fraction based managed memory.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (FLINK-14060) Set slot sharing groups according to pipelined regions

[Xintong Song (Jira)]

Xintong Song created FLINK-14060:


 Summary: Set slot sharing groups according to pipelined regions
 Key: FLINK-14060
 URL: https://issues.apache.org/jira/browse/FLINK-14060
 Project: Flink
  Issue Type: Sub-task
Reporter: Xintong Song


{{StreamingJobGraphGenerator}} set slot sharing group for operators at 
compiling time.
 * Identify pipelined regions, with respect to 
{{allSourcesInSamePipelinedRegion}}
 * Set slot sharing groups according to pipelined regions 
 ** By default, each pipelined region should go into a separate slot sharing 
group
 ** If the user sets operators in multiple pipelined regions into same slot 
sharing group, it should be respected

This step should not introduce any behavior changes, given that later scheduled 
pipelined regions can reuse slots from previous scheduled pipelined regions. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (FLINK-14062) Set managed memory fractions according to slot sharing groups

[Xintong Song (Jira)]

Xintong Song created FLINK-14062:


 Summary: Set managed memory fractions according to slot sharing 
groups
 Key: FLINK-14062
 URL: https://issues.apache.org/jira/browse/FLINK-14062
 Project: Flink
  Issue Type: Sub-task
Reporter: Xintong Song


* For operators with specified {{ResourceSpecs}}, calculate fractions according 
to operators {{ResourceSpecs}}
 * For operators with unknown {{ResourceSpecs}}, calculate fractions according 
to number of operators using managed memory

This step should not introduce any behavior changes.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Yun Tang]

Congratulations Zili

Best
Yun Tang

From: Yun Gao 
Sent: Thursday, September 12, 2019 10:12
To: dev 
Subject: Re: [ANNOUNCE] Zili Chen becomes a Flink committer

Congratulations Zili!

   Best,
   Yun


--
From:Yangze Guo 
Send Time:2019 Sep. 12 (Thu.) 09:38
To:dev 
Subject:Re: [ANNOUNCE] Zili Chen becomes a Flink committer

Congratulations!

Best,
Yangze Guo

On Thu, Sep 12, 2019 at 9:35 AM Rong Rong  wrote:
>
> Congratulations Zili!
>
> --
> Rong
>
> On Wed, Sep 11, 2019 at 6:26 PM Hequn Cheng  wrote:
>
> > Congratulations!
> >
> > Best, Hequn
> >
> > On Thu, Sep 12, 2019 at 9:24 AM Jark Wu  wrote:
> >
> >> Congratulations Zili!
> >>
> >> Best,
> >> Jark
> >>
> >> On Wed, 11 Sep 2019 at 23:06,  wrote:
> >>
> >> > Congratulations, Zili.
> >> >
> >> >
> >> >
> >> > Best,
> >> >
> >> > Xingcan
> >> >
> >> >
> >> >
> >> > *From:* SHI Xiaogang 
> >> > *Sent:* Wednesday, September 11, 2019 7:43 AM
> >> > *To:* Guowei Ma 
> >> > *Cc:* Fabian Hueske ; Biao Liu ;
> >> > Oytun Tez ; bupt_ljy ; dev <
> >> > dev@flink.apache.org>; user ; Till Rohrmann <
> >> > trohrm...@apache.org>
> >> > *Subject:* Re: [ANNOUNCE] Zili Chen becomes a Flink committer
> >> >
> >> >
> >> >
> >> > Congratulations!
> >> >
> >> >
> >> >
> >> > Regards,
> >> >
> >> > Xiaogang
> >> >
> >> >
> >> >
> >> > Guowei Ma  于2019年9月11日周三 下午7:07写道：
> >> >
> >> > Congratulations Zili !
> >> >
> >> >
> >> > Best,
> >> >
> >> > Guowei
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Fabian Hueske  于2019年9月11日周三 下午7:02写道：
> >> >
> >> > Congrats Zili Chen :-)
> >> >
> >> >
> >> >
> >> > Cheers, Fabian
> >> >
> >> >
> >> >
> >> > Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu <
> >> mmyy1...@gmail.com>:
> >> >
> >> > Congrats Zili!
> >> >
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > Biao /'bɪ.aʊ/
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Wed, 11 Sep 2019 at 18:43, Oytun Tez  wrote:
> >> >
> >> > Congratulations!
> >> >
> >> >
> >> >
> >> > ---
> >> >
> >> > Oytun Tez
> >> >
> >> >
> >> >
> >> > *M O T A W O R D*
> >> >
> >> > *The World's Fastest Human Translation Platform.*
> >> >
> >> > oy...@motaword.com — www.motaword.com
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy  wrote:
> >> >
> >> > Congratulations!
> >> >
> >> >
> >> >
> >> > Best,
> >> >
> >> > Jiayi Liao
> >> >
> >> >
> >> >
> >> >  Original Message
> >> >
> >> > *Sender:* Till Rohrmann
> >> >
> >> > *Recipient:* dev; user
> >> >
> >> > *Date:* Wednesday, Sep 11, 2019 17:22
> >> >
> >> > *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer
> >> >
> >> >
> >> >
> >> > Hi everyone,
> >> >
> >> >
> >> >
> >> > I'm very happy to announce that Zili Chen (some of you might also know
> >> > him as Tison Kun) accepted the offer of the Flink PMC to become a
> >> committer
> >> > of the Flink project.
> >> >
> >> >
> >> >
> >> > Zili Chen has been an active community member for almost 16 months now.
> >> > He helped pushing the Flip-6 effort over the finish line, ported a lot
> >> of
> >> > legacy code tests, removed a good part of the legacy code, contributed
> >> > numerous fixes, is involved in the Flink's client API refactoring,
> >> drives
> >> > the refactoring of Flink's HighAvailabilityServices and much more. Zili
> >> > Chen also helped the community by PR reviews, reporting Flink issues,
> >> > answering user mails and being very active on the dev mailing list.
> >> >
> >> >
> >> >
> >> > Congratulations Zili Chen!
> >> >
> >> >
> >> >
> >> > Best, Till
> >> >
> >> > (on behalf of the Flink PMC)
> >> >
> >> >
> >>
> >

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Xintong Song]

Congratulations Zili

Thank you~

Xintong Song



On Thu, Sep 12, 2019 at 11:03 AM Yun Tang  wrote:

> Congratulations Zili
>
> Best
> Yun Tang
> 
> From: Yun Gao 
> Sent: Thursday, September 12, 2019 10:12
> To: dev 
> Subject: Re: [ANNOUNCE] Zili Chen becomes a Flink committer
>
> Congratulations Zili!
>
>Best,
>Yun
>
>
> --
> From:Yangze Guo 
> Send Time:2019 Sep. 12 (Thu.) 09:38
> To:dev 
> Subject:Re: [ANNOUNCE] Zili Chen becomes a Flink committer
>
> Congratulations!
>
> Best,
> Yangze Guo
>
> On Thu, Sep 12, 2019 at 9:35 AM Rong Rong  wrote:
> >
> > Congratulations Zili!
> >
> > --
> > Rong
> >
> > On Wed, Sep 11, 2019 at 6:26 PM Hequn Cheng 
> wrote:
> >
> > > Congratulations!
> > >
> > > Best, Hequn
> > >
> > > On Thu, Sep 12, 2019 at 9:24 AM Jark Wu  wrote:
> > >
> > >> Congratulations Zili!
> > >>
> > >> Best,
> > >> Jark
> > >>
> > >> On Wed, 11 Sep 2019 at 23:06,  wrote:
> > >>
> > >> > Congratulations, Zili.
> > >> >
> > >> >
> > >> >
> > >> > Best,
> > >> >
> > >> > Xingcan
> > >> >
> > >> >
> > >> >
> > >> > *From:* SHI Xiaogang 
> > >> > *Sent:* Wednesday, September 11, 2019 7:43 AM
> > >> > *To:* Guowei Ma 
> > >> > *Cc:* Fabian Hueske ; Biao Liu <
> mmyy1...@gmail.com>;
> > >> > Oytun Tez ; bupt_ljy ; dev <
> > >> > dev@flink.apache.org>; user ; Till Rohrmann
> <
> > >> > trohrm...@apache.org>
> > >> > *Subject:* Re: [ANNOUNCE] Zili Chen becomes a Flink committer
> > >> >
> > >> >
> > >> >
> > >> > Congratulations!
> > >> >
> > >> >
> > >> >
> > >> > Regards,
> > >> >
> > >> > Xiaogang
> > >> >
> > >> >
> > >> >
> > >> > Guowei Ma  于2019年9月11日周三 下午7:07写道：
> > >> >
> > >> > Congratulations Zili !
> > >> >
> > >> >
> > >> > Best,
> > >> >
> > >> > Guowei
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Fabian Hueske  于2019年9月11日周三 下午7:02写道：
> > >> >
> > >> > Congrats Zili Chen :-)
> > >> >
> > >> >
> > >> >
> > >> > Cheers, Fabian
> > >> >
> > >> >
> > >> >
> > >> > Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu <
> > >> mmyy1...@gmail.com>:
> > >> >
> > >> > Congrats Zili!
> > >> >
> > >> >
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Biao /'bɪ.aʊ/
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Wed, 11 Sep 2019 at 18:43, Oytun Tez  wrote:
> > >> >
> > >> > Congratulations!
> > >> >
> > >> >
> > >> >
> > >> > ---
> > >> >
> > >> > Oytun Tez
> > >> >
> > >> >
> > >> >
> > >> > *M O T A W O R D*
> > >> >
> > >> > *The World's Fastest Human Translation Platform.*
> > >> >
> > >> > oy...@motaword.com — www.motaword.com
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy  wrote:
> > >> >
> > >> > Congratulations!
> > >> >
> > >> >
> > >> >
> > >> > Best,
> > >> >
> > >> > Jiayi Liao
> > >> >
> > >> >
> > >> >
> > >> >  Original Message
> > >> >
> > >> > *Sender:* Till Rohrmann
> > >> >
> > >> > *Recipient:* dev; user
> > >> >
> > >> > *Date:* Wednesday, Sep 11, 2019 17:22
> > >> >
> > >> > *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer
> > >> >
> > >> >
> > >> >
> > >> > Hi everyone,
> > >> >
> > >> >
> > >> >
> > >> > I'm very happy to announce that Zili Chen (some of you might also
> know
> > >> > him as Tison Kun) accepted the offer of the Flink PMC to become a
> > >> committer
> > >> > of the Flink project.
> > >> >
> > >> >
> > >> >
> > >> > Zili Chen has been an active community member for almost 16 months
> now.
> > >> > He helped pushing the Flip-6 effort over the finish line, ported a
> lot
> > >> of
> > >> > legacy code tests, removed a good part of the legacy code,
> contributed
> > >> > numerous fixes, is involved in the Flink's client API refactoring,
> > >> drives
> > >> > the refactoring of Flink's HighAvailabilityServices and much more.
> Zili
> > >> > Chen also helped the community by PR reviews, reporting Flink
> issues,
> > >> > answering user mails and being very active on the dev mailing list.
> > >> >
> > >> >
> > >> >
> > >> > Congratulations Zili Chen!
> > >> >
> > >> >
> > >> >
> > >> > Best, Till
> > >> >
> > >> > (on behalf of the Flink PMC)
> > >> >
> > >> >
> > >>
> > >
>
>

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Leonard Xu]

Congratulations Zili Chen ! ！

Best，
Leonard Xu
> On 2019年9月12日, at 上午11:02, Yun Tang  wrote:
> 
> Congratulations Zili
> 
> Best
> Yun Tang
> 
> From: Yun Gao  >
> Sent: Thursday, September 12, 2019 10:12
> To: dev mailto:dev@flink.apache.org>>
> Subject: Re: [ANNOUNCE] Zili Chen becomes a Flink committer
> 
> Congratulations Zili!
> 
>   Best,
>   Yun
> 
> 
> --
> From:Yangze Guo 
> Send Time:2019 Sep. 12 (Thu.) 09:38
> To:dev 
> Subject:Re: [ANNOUNCE] Zili Chen becomes a Flink committer
> 
> Congratulations!
> 
> Best,
> Yangze Guo
> 
> On Thu, Sep 12, 2019 at 9:35 AM Rong Rong  wrote:
>> 
>> Congratulations Zili!
>> 
>> --
>> Rong
>> 
>> On Wed, Sep 11, 2019 at 6:26 PM Hequn Cheng  wrote:
>> 
>>> Congratulations!
>>> 
>>> Best, Hequn
>>> 
>>> On Thu, Sep 12, 2019 at 9:24 AM Jark Wu  wrote:
>>> 
 Congratulations Zili!
 
 Best,
 Jark
 
 On Wed, 11 Sep 2019 at 23:06,  wrote:
 
> Congratulations, Zili.
> 
> 
> 
> Best,
> 
> Xingcan
> 
> 
> 
> *From:* SHI Xiaogang 
> *Sent:* Wednesday, September 11, 2019 7:43 AM
> *To:* Guowei Ma 
> *Cc:* Fabian Hueske ; Biao Liu ;
> Oytun Tez ; bupt_ljy ; dev <
> dev@flink.apache.org>; user ; Till Rohrmann <
> trohrm...@apache.org>
> *Subject:* Re: [ANNOUNCE] Zili Chen becomes a Flink committer
> 
> 
> 
> Congratulations!
> 
> 
> 
> Regards,
> 
> Xiaogang
> 
> 
> 
> Guowei Ma  于2019年9月11日周三 下午7:07写道：
> 
> Congratulations Zili !
> 
> 
> Best,
> 
> Guowei
> 
> 
> 
> 
> 
> Fabian Hueske  于2019年9月11日周三 下午7:02写道：
> 
> Congrats Zili Chen :-)
> 
> 
> 
> Cheers, Fabian
> 
> 
> 
> Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu <
 mmyy1...@gmail.com>:
> 
> Congrats Zili!
> 
> 
> 
> Thanks,
> 
> Biao /'bɪ.aʊ/
> 
> 
> 
> 
> 
> 
> 
> On Wed, 11 Sep 2019 at 18:43, Oytun Tez  wrote:
> 
> Congratulations!
> 
> 
> 
> ---
> 
> Oytun Tez
> 
> 
> 
> *M O T A W O R D*
> 
> *The World's Fastest Human Translation Platform.*
> 
> oy...@motaword.com  — www.motaword.com 
>  >
> 
> 
> 
> 
> 
> On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy  wrote:
> 
> Congratulations!
> 
> 
> 
> Best,
> 
> Jiayi Liao
> 
> 
> 
> Original Message
> 
> *Sender:* Till Rohrmann
> 
> *Recipient:* dev; user
> 
> *Date:* Wednesday, Sep 11, 2019 17:22
> 
> *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer
> 
> 
> 
> Hi everyone,
> 
> 
> 
> I'm very happy to announce that Zili Chen (some of you might also know
> him as Tison Kun) accepted the offer of the Flink PMC to become a
 committer
> of the Flink project.
> 
> 
> 
> Zili Chen has been an active community member for almost 16 months now.
> He helped pushing the Flip-6 effort over the finish line, ported a lot
 of
> legacy code tests, removed a good part of the legacy code, contributed
> numerous fixes, is involved in the Flink's client API refactoring,
 drives
> the refactoring of Flink's HighAvailabilityServices and much more. Zili
> Chen also helped the community by PR reviews, reporting Flink issues,
> answering user mails and being very active on the dev mailing list.
> 
> 
> 
> Congratulations Zili Chen!
> 
> 
> 
> Best, Till
> 
> (on behalf of the Flink PMC)

Re: [DISCUSS] Features for Apache Flink 1.10

[Biao Liu]

Thanks Gary for kicking off the discussion.

+1 for the feature freeze time. Also thanks Gary and Yu Li for volunteering
as the release manager.

BTW, I'm working on refactoring of `CheckpointCoordinator` [1]. It would be
great if it is included in 1.10.

1. https://issues.apache.org/jira/browse/FLINK-13698

Thanks,
Biao /'bɪ.aʊ/



On Wed, 11 Sep 2019 at 18:33, Aljoscha Krettek  wrote:

> Hi,
>
> Thanks for putting together the list! And I’m +1 for the suggested
> release timeline and also for Gary and Yu as the release managers.
>
> Best,
> Aljoscha
>
> On 9 Sep 2019, at 7:39, Yu Li wrote:
>
> > Hi Xuefu,
> >
> > If I understand it correctly, the data type support work should be
> > included
> > in the "Table API improvements->Finish type system" part, please check
> > it
> > and let us know if anything missing there. Thanks.
> >
> > Best Regards,
> > Yu
> >
> >
> > On Mon, 9 Sep 2019 at 11:14, Xuefu Z  wrote:
> >
> >> Looking at feature list, I don't see an item for complete the data
> >> type
> >> support. Specifically, high precision timestamp is needed to Hive
> >> integration, as it's so common. Missing it would damage the
> >> completeness of
> >> our Hive effort.
> >>
> >> Thanks,
> >> Xuefu
> >>
> >> On Sat, Sep 7, 2019 at 7:06 PM Xintong Song 
> >> wrote:
> >>
> >>> Thanks Gray and Yu for compiling the feature list and kicking off
> >>> this
> >>> discussion.
> >>>
> >>> +1 for Gary and Yu being the release managers for Flink 1.10.
> >>>
> >>> Thank you~
> >>>
> >>> Xintong Song
> >>>
> >>>
> >>>
> >>> On Sat, Sep 7, 2019 at 4:58 PM Till Rohrmann 
> >> wrote:
> >>>
>  Thanks for compiling the list of 1.10 efforts for the community
>  Gary. I
>  think this helps a lot to better understand what the community is
> >>> currently
>  working on.
> 
>  Thanks for volunteering as the release managers for the next major
>  release. +1 for Gary and Yu being the RMs for Flink 1.10.
> 
>  Cheers,
>  Till
> 
>  On Sat, Sep 7, 2019 at 7:26 AM Zhu Zhu  wrote:
> 
> > Thanks Gary for kicking off this discussion.
> > Really appreciate that you and Yu offer to help to manage 1.10
> >> release.
> >
> > +1 for Gary and Yu as release managers.
> >
> > Thanks,
> > Zhu Zhu
> >
> > Dian Fu  于2019年9月7日周六
> > 下午12:26写道：
> >
> >> Hi Gary,
> >>
> >> Thanks for kicking off the release schedule of 1.10. +1 for you
> >> and
> >>> Yu
>  Li
> >> as the release manager.
> >>
> >> The feature freeze/release time sounds reasonable.
> >>
> >> Thanks,
> >> Dian
> >>
> >>> 在 2019年9月7日，上午11:30，Jark Wu 
> >>> 写道：
> >>>
> >>> Thanks Gary for kicking off the discussion for 1.10 release.
> >>>
> >>> +1 for Gary and Yu as release managers. Thank you for you
> >>> effort.
> >>>
> >>> Best,
> >>> Jark
> >>>
> >>>
>  在 2019年9月7日，00:52，zhijiang
>  
> >>> 写道：
> 
>  Hi Gary,
> 
>  Thanks for kicking off the features for next release 1.10.  I
>  am
>  very
> >> supportive of you and Yu Li to be the relaese managers.
> 
>  Just mention another two improvements which want to be covered
> >> in
> >> FLINK-1.10 and I already confirmed with Piotr to reach an
> >> agreement
> > before.
> 
>  1. Data serialize and copy only once for broadcast partition
> >> [1]:
> >>> It
> >> would improve the throughput performance greatly in broadcast
> >> mode
> >>> and
> > was
> >> actually proposed in Flink-1.8. Most of works already done before
> >> and
> > only
> >> left the last critical jira/PR. It will not take much efforts to
> >> make
>  it
> >> ready.
> 
>  2. Let Netty use Flink's buffers directly in credit-based mode
> >>> [2] :
> > It
> >> could avoid memory copy from netty stack to flink managed network
>  buffer.
> >> The obvious benefit is decreasing the direct memory overhead
> >> greatly
> >>> in
> >> large-scale jobs. I also heard of some user cases encounter
> >> direct
> >>> OOM
> >> caused by netty memory overhead. Actually this improvment was
> >>> proposed
>  by
> >> nico in FLINK-1.7 and always no time to focus then. Yun Gao
> >> already
> >> submitted a PR half an year ago but have not been reviewed yet. I
> >>> could
> >> help review the deign and PR codes to make it ready.
> 
>  And you could make these two items as lowest priority if
> >> possible.
> 
>  [1] https://issues.apache.org/jira/browse/FLINK-10745
>  [2] https://issues.apache.org/jira/browse/FLINK-10742
> 
>  Best,
>  Zhijiang
> 
> >> --
>  From:Gary Yao 
>  Send Time:2019年9月6日(星期五) 17:06
>  To:dev 
>  Cc:carp84 
>  Su

Re: [DISCUSS] FLIP-63: Rework table partition support

[Biao Liu]

Hi Jingsong,

Thanks for explaining. It looks cool!

Thanks,
Biao /'bɪ.aʊ/



On Wed, 11 Sep 2019 at 11:37, JingsongLee 
wrote:

> Hi biao, thanks for your feedbacks:
>
> Actually, the runtime source partition of runtime is similar to split,
> which concerns data reading, parallelism and fault tolerance, all the
> runtime concepts.
> While table partition is only a virtual concept. Users are more likely to
> choose which partition to read and which partition to write. Users can
> manage their partitions.
> One is physical implementation correlation, the other is logical concept
> correlation.
> So I think they are two completely different things.
>
> About [2], The main problem is that how to write data to a catalog file
> system in stream mode, it is a general problem and has little to do with
> partition.
>
> [2]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-notifyOnMaster-for-notifyCheckpointComplete-td32769.html
>
> Best,
> Jingsong Lee
>
>
> --
> From:Biao Liu 
> Send Time:2019年9月10日(星期二) 14:57
> To:dev ; JingsongLee 
> Subject:Re: [DISCUSS] FLIP-63: Rework table partition support
>
> Hi Jingsong,
>
> Thank you for bringing this discussion. Since I don't have much experience
> of Flink table/SQL, I'll ask some questions from runtime or engine
> perspective.
>
> > ... where we describe how to partition support in flink and how to
> integrate to hive partition.
>
> FLIP-27 [1] introduces "partition" concept officially. The changes of
> FLIP-27 are not only about source interface but also about the whole
> infrastructure.
> Have you ever thought how to integrate your proposal with these changes?
> Or you just want to support "partition" in table layer, there will be no
> requirement of underlying infrastructure?
>
> I have seen a discussion [2] that seems be a requirement of infrastructure
> to support your proposal. So I have some concerns there might be some
> conflicts between this proposal and FLIP-27.
>
> 1.
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> 2.
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-notifyOnMaster-for-notifyCheckpointComplete-td32769.html
>
> Thanks,
> Biao /'bɪ.aʊ/
>
>
>
> On Fri, 6 Sep 2019 at 13:22, JingsongLee 
> wrote:
> Hi everyone, thank you for your comments. Mail name was updated
>  and streaming-related concepts were added.
>
>  We would like to start a discussion thread on "FLIP-63: Rework table
>  partition support"(Design doc: [1]), where we describe how to partition
>  support in flink and how to integrate to hive partition.
>
>  This FLIP addresses:
> - Introduce whole story about partition support.
> - Introduce and discuss DDL of partition support.
> - Introduce static and dynamic partition insert.
> - Introduce partition pruning
> - Introduce dynamic partition implementation
> - Introduce FileFormatSink to deal with streaming exactly-once and
>   partition-related logic.
>
>  Details can be seen in the design document.
>  Looking forward to your feedbacks. Thank you.
>
>  [1]
> https://docs.google.com/document/d/15R3vZ1R_pAHcvJkRx_CWleXgl08WL3k_ZpnWSdzP7GY/edit?usp=sharing
>
>  Best,
>  Jingsong Lee
>
>

[jira] [Created] (FLINK-14064) Support Project Push down to LookupableTableSource

[hailong wang (Jira)]

hailong wang created FLINK-14064:


 Summary: Support Project Push down to LookupableTableSource
 Key: FLINK-14064
 URL: https://issues.apache.org/jira/browse/FLINK-14064
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Affects Versions: 1.9.0
Reporter: hailong wang
 Fix For: 1.10.0


Consider such a temporal join sql:

 
{code:java}
SELECT T.id, D.id, D.name FROM T JOIN temporalTable for system_time AS OF 
T.proctime AS D ON upper(T.id) = upper(D.id)
{code}
When apply a sql function to equivalent condition such as 'upper', it doesn't 
support. In other words, temporal join don't support any transfrom on temporal  
table primary key field.

There are two reasons:

1、In CommonLookupJoin.getIdenticalSourceField method, we can not get the input 
index when expr is a rexCall except IN_FENNEL and CAST;

2、StreamExecLookupJoinRule doesn't push down project function to 
LookupableTableSource, so LookupFunction can not perceive the transfrom of 
primary key.

 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Interested in contributing and looking for good first issues

[Anoop Hallur]

Hello Flink Devs,
I am interested in contributing to Apache Flink and I am looking for
beginner friendly issues/tasks to hit the ground running. *Can somebody
recommend good issues for first time contributors* ?

Also, can somebody add me to JIRA project for Flink ? My JIRA username is
anoophallur

What I have done so far.

* I have read the "How to contribute" guide (at
https://flink.apache.org/contributing/how-to-contribute.html) and all it's
subpages.

* I have set up my development environment by following the guide at
https://cwiki.apache.org/confluence/display/FLINK/Setting+up+a+Flink+development+environment.
"mvn clean packages" builds and packages the build artifacts for me. All
the tests are passing locally as well.

* I have looked at the open issues at
https://issues.apache.org/jira/projects/FLINK/issues/FLINK-13613?filter=allopenissues
but
I am not sure which ones are good for new members. I use JVM based
languages + mvn for my day job, so I understand the overall project, but
not the relative importance of each of the issues.

* I have also read bits and pieces of documentation + the tutorials(at
https://training.ververica.com/intro/intro-1.html). I consider myself a
novice as a Flink consumer.

Thanks,
Anoop

*Anoop Hallur*
(001) 917-285-3445 | anoophal...@gmail.com | Skype: anoophallur

  

[jira] [Created] (FLINK-14065) Log metric name when the metric fails on registration/unregistration

[Zhilong Hong (Jira)]

Zhilong Hong created FLINK-14065:


 Summary: Log metric name when the metric fails on 
registration/unregistration
 Key: FLINK-14065
 URL: https://issues.apache.org/jira/browse/FLINK-14065
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Metrics
Affects Versions: 1.9.0
Reporter: Zhilong Hong
 Fix For: 1.10.0


When MetricGroup registers Metrics in MetricRegistryImpl, sometimes the 
registration fails due to exceptions. However, currently it only logs {{"Error 
while registering metric" }}with no more information, which is inconvenient for 
users to troubleshoot which metric fails and why it fails.

Also, the warning log in registration and unregistration are both "{{Error 
while registering metric}}". This will lead users to confusion (although users 
can locate the correct place according to the call stack).

So I propose to log metric name when the metrics fails on 
registration/unregistration.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Blink Planner HBaseUpsertTableSink Exception

[LakeShen]

Hi community , when I create the hbase sink table  in my  flink ddl sql
,just like this:





*create table sink_hbase_table(rowkey VARCHAR,cf
  row(kdt_it_count  bigint )) with (xx);*

and I run my flink task , it throws the exception like this :
*UpsertStreamTableSink requires that Table has a full primary keys if it is
updated.*
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlanInternal(StreamExecSink.scala:115)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlanInternal(StreamExecSink.scala:50)
at
org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:54)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlan(StreamExecSink.scala:50)
at
org.apache.flink.table.planner.delegation.StreamPlanner$$anonfun$translateToPlan$1.apply(StreamPlanner.scala:61)
at
org.apache.flink.table.planner.delegation.StreamPlanner$$anonfun$translateToPlan$1.apply(StreamPlanner.scala:60)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)

I saw the flink source code , I find that in HBaseUpsertTableSink , the
method setKeyFields doesnt' has any code content,in StreamExecSink class,I
saw the code content like this :
*//TODO UpsertStreamTableSink setKeyFields interface should be
Array[Array[String]]*
but now the  UpsertStreamTableSink setKeyFields interface is Array[String],
it seems like the conflict with the above content.
Could we use HBaseUpsertTableSink in our flink task?Thanks your reply.