[jira] [Created] (FLINK-16182) Remove invalid check in input type inference logic

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-16182:


 Summary: Remove invalid check in input type inference logic
 Key: FLINK-16182
 URL: https://issues.apache.org/jira/browse/FLINK-16182
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.11.0


In {{org.apache.flink.table.types.inference.TypeInferenceUtil#inferInputTypes}} 
we force that the input type inference must never return {{NULL}} type.

This precondition will never be satisfied when trying to infer output type 
based on surrounding info. This check is invalid anyway.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Kicking off the 1.11 release cycle

[Piotr Nowojski]

Hi,

Thank you for giving me and Zhijiang opportunity and entrusting us with the 
release :)

+1 for aiming with the feature freeze for and of April. I think all things 
consider that’s a good timing.

Piotrek

> On 20 Feb 2020, at 04:22, Yu Li  wrote:
> 
> Thanks Stephan for the kicking off! And thanks Piotr and Zhijiang for
> volunteering as the release managers!
> 
> Aiming for a shorter release cycle than 1.10 makes a lot of sense, and +1
> to target the feature freeze date around end of April.
> 
> Best Regards,
> Yu
> 
> 
> On Thu, 20 Feb 2020 at 10:37, Xintong Song  wrote:
> 
>> Thanks for the kicking off.
>> 
>> +1 for Zhijiang & Piotr (if he agrees to) being the release managers.
>> Thanks for the volunteering.
>> +1 for the *anticipated feature freeze date* end of April
>> 
>> Thank you~
>> 
>> Xintong Song
>> 
>> 
>> 
>> On Thu, Feb 20, 2020 at 10:30 AM Yuan Mei  wrote:
>> 
>>> +1
>>> 
>>> 3-month sounds a very reasonable duration for a release.
>>> 
>>> On Thu, Feb 20, 2020 at 9:53 AM tison  wrote:
>>> 
 Thanks for kicking off the discussion Stephan!
 
 +1 for the *anticipated feature freeze date* end of April and
 Zhijiang & Piotr to be the release managers.
 
 Best,
 tison.
 
 
 Dian Fu  于2020年2月20日周四 上午9:41写道：
 
> Thanks Stephan kicking off this discussion and Zhijiang volunteering
>> as
> one of the release managers.
> 
> +1 for the "feature freeze date" around end of April. There are still
 more
> than 2 months left, so I think it's a reasonable time.
> 
> Thanks,
> Dian
> 
>> 在 2020年2月19日，下午10:52，Aljoscha Krettek  写道：
>> 
>> +1
>> 
>> Although I would hope that it can be more than just "anticipated".
>> 
>> Best,
>> Aljoscha
>> 
>> On 19.02.20 15:40, Till Rohrmann wrote:
>>> Thanks for volunteering as one of our release managers Zhijiang.
>>> +1 for the *anticipated feature freeze date* end of April. As we
>> go
> along
>>> and collect more data points we might be able to strengthen our
>>> initial anticipation.
>>> Cheers,
>>> Till
>>> On Wed, Feb 19, 2020 at 4:44 AM Zhijiang <
>>> wangzhijiang...@aliyun.com
> .invalid>
>>> wrote:
 Thanks for kicking off the next release and the introduction,
 @Stephan!
 
 It's my pleasure to become the release manager and involve in
>> other
 community works. I am working together with @Piotr for a long
>> time,
 so
 very happy to cooperate for the release manager work again. The
> previous
 release work was always well done, and I can learn a lot from
>> these
> rich
 experiences.
 
 +1 for the "feature freeze date" around end of April.
 Although we have the FF SF in the meantime, fortunately there
>> are
>>> no
> long
 public holidays during this period in China.
 
 Best,
 Zhijiang
 
 
 
>> --
 From:Stephan Ewen 
 Send Time:2020 Feb. 19 (Wed.) 01:15
 To:dev 
 Cc:zhijiang ; pnowojski <
>> pnowoj...@apache.org
 
 Subject:[DISCUSS] Kicking off the 1.11 release cycle
 
 Hi all!
 
 Now that the 1.10 release is out (congrats again, everyone!), I
 wanted
> to
 bring up some questions about organizing the next release cycle.
 
 The most important questions at the beginning would be
  - Who would volunteer as Release Managers
  - When would be the release date.
 
 For the release managers, Piotrek and Zhijiang have mentioned
> previously
 that they would be interested, so I am copying them here to chime
>>> in.
 @Piotr and @Zhijiang could you confirm if you are interested in
 helping
 out with this?
 
 About the release date: By our original "3 months release cycle"
 assumption, we should aim for a release **Mid May**, meaning a
 feature
 freeze no later than end of April.
 That would be indeed a shorter release cycle than 1.10, and the
> assumption
 of a shorter testing period. But aiming for a shorter release
>> cycle
> than
 1.10 would actually be nice, in my opinion. 1.10 grew very big in
>>> the
> end,
 which caused also a very long testing period (plus Christmas and
> Chinese
 New Year are also partially to blame).
 
 The exact feature freeze date is anyways a community discussion
 later,
> but
 what do you think about starting with an "anticipated feature
>>> freeze
> date"
 around end of April, so that committers, contributors, and users
>>> know
 roughly what to expect?
 
 Best,
 Stephan
 
 
 
> 
> 
 
>>> 

[jira] [Created] (FLINK-16183) Make identifier parsing in Table API more lenient

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-16183:


 Summary: Make identifier parsing in Table API more lenient
 Key: FLINK-16183
 URL: https://issues.apache.org/jira/browse/FLINK-16183
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.11.0


I suggest to make the parsing logic for identifiers in Table API more lenient. 
We should not require users to escape any sql identifiers. 

It will make the identifiers not cross compatible between Table API and SQL, 
but it will improve user's experience and also will let us support parsing 
identifiers coming from Java's ExpressionParser (e.g. for a string array(..) it 
produces a lookup call with an "array" identifier which should be parsed)

I suggest doing it by extending the FlinkSqlParserImpl with a new logic for 
TableApiSqlIdentifier which would be very similar to CompoundIdentifier with 
slightly adjusted logic which would not discard reserved keywords.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16184) Submit job with fixed/configurable JobID

[lining (Jira)]

lining created FLINK-16184:
--

 Summary: Submit job  with fixed/configurable JobID
 Key: FLINK-16184
 URL: https://issues.apache.org/jira/browse/FLINK-16184
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: lining






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] FLIP-98: Better Back Pressure Detection

[Yadong Xie]

Hi all

I want to start the vote for FLIP-98, which proposes to provide better
backpressure detection experience in web UI.

To help everyone better understand the proposal, we spent some efforts on
making an online POC

previous web:
http://101.132.122.69:8081/#/job/add9d3ea1006af15b9f872987eb3befd/overview
POC web:
http://101.132.122.69:8081/web/#/job/add9d3ea1006af15b9f872987eb3befd/overview

The vote will last for at least 72 hours, following the consensus voting
process.

FLIP wiki:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection

Discussion thread:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html

Thanks,

Yadong

Re: [Discussion] Job generation / submission hooks & Atlas integration

[Gyula Fóra]

Hi all!

Thank you for the patience!

We have created a small design document for the change proposal detailing
the minimal required changes in Flink for the initial version of the Atlas
integration.

You can find the document here:
https://docs.google.com/document/d/1wSgzPdhcwt-SlNBBqL-Zb7g8fY6bN8JwHEg7GCdsBG8/edit?usp=sharing

It would be great if you could check it out and comment on it.
If we agree on the next steps I will start opening JIRA-s and PRs with the
proposed changes.

The document links to an already working Atlas hook prototype (and
accompanying flink fork). The links for that are also here:
Flink: https://github.com/gyfora/flink/tree/atlas-changes
Atlas: https://github.com/gyfora/atlas/tree/flink-bridge

Thank you!
Gyula

On Thu, Feb 13, 2020 at 4:43 PM Gyula Fóra  wrote:

> Thanks for the feedback Aljoscha!
>
> I have a POC ready with the Flink changes + the Atlas hook implementation.
> I will try to push this to a public repo tomorrow and we can discuss
> further based on that!
>
> Gyula
>
> On Thu, Feb 13, 2020, 15:26 Aljoscha Krettek  wrote:
>
>> I think exposing the Pipeline should be ok. Using the internal
>> StreamGraph might be problematic because this might change/break but
>> that's a problem of the external code.
>>
>> Aljoscha
>>
>> On 11.02.20 16:26, Gyula Fóra wrote:
>> > Hi All!
>> >
>> > I have made a prototype that simply adds a getPipeline() method to the
>> > JobClient interface. Then I could easily implement the Atlas hook using
>> the
>> > JobListener interface. I simply check if Pipeline is instanceof
>> StreamGraph
>> > and do the logic there.
>> >
>> > I think this is so far the cleanest approach and I much prefer this
>> > compared to working on the JobGraph directly which would expose even
>> more
>> > messy internals.
>> >
>> > Unfortunately this change alone is not enough for the integration as we
>> > need to make sure that all Sources/Sinks that we want to integrate to
>> atlas
>> > publicly expose some of their properties:
>> >
>> > - Kafka source/sink:
>> >- Kafka props
>> >- Topic(s) - this is tricky for sinks
>> > - FS source /sink:
>> >- Hadoop props
>> >- Base path for StreamingFileSink
>> >- Path for ContinuousMonitoringSource
>> >
>> > Most of these are straightforward changes, the only question is what we
>> > want to register in Atlas from the available connectors. Ideally users
>> > could also somehow register their own Atlas metadata for custom sources
>> and
>> > sinks, we could probably introduce an interface for that in Atlas.
>> >
>> > Cheers,
>> > Gyula
>> >
>> > On Fri, Feb 7, 2020 at 10:37 AM Gyula Fóra 
>> wrote:
>> >
>> >> Maybe we could improve the Pipeline interface in the long run, but as a
>> >> temporary solution the JobClient could expose a getPipeline() method.
>> >>
>> >> That way the implementation of the JobListener could check if its a
>> >> StreamGraph or a Plan.
>> >>
>> >> How bad does that sound?
>> >>
>> >> Gyula
>> >>
>> >> On Fri, Feb 7, 2020 at 10:19 AM Gyula Fóra 
>> wrote:
>> >>
>> >>> Hi Aljoscha!
>> >>>
>> >>> That's a valid concert but we should try to figure something out, many
>> >>> users need this before they can use Flink.
>> >>>
>> >>> I think the closest thing we have right now is the StreamGraph. In
>> >>> contrast with the JobGraph  the StreamGraph is pretty nice from a
>> metadata
>> >>> perspective :D
>> >>> The big downside of exposing the StreamGraph is that we don't have it
>> in
>> >>> batch. On the other hand we could expose the JobGraph but then the
>> >>> integration component would still have to do the heavy lifting for
>> batch
>> >>> and stream specific operators and UDFs.
>> >>>
>> >>> Instead of exposing either StreamGraph/JobGraph, we could come up
>> with a
>> >>> metadata like representation for the users but that would be like
>> >>> implementing Atlas integration itself without Atlas dependencies :D
>> >>>
>> >>> As a comparison point, this is how it works in Storm:
>> >>> Every operator (spout/bolt), stores a config map (string->string) with
>> >>> all the metadata such as operator class, and the operator specific
>> configs.
>> >>> The Atlas hook works on this map.
>> >>> This is very fragile and depends on a lot of internals. Kind of like
>> >>> exposing the JobGraph but much worse. I think we can do better.
>> >>>
>> >>> Gyula
>> >>>
>> >>> On Fri, Feb 7, 2020 at 9:55 AM Aljoscha Krettek 
>> >>> wrote:
>> >>>
>>  If we need it, we can probably beef up the JobListener to allow
>>  accessing some information about the whole graph or sources and
>> sinks.
>>  My only concern right now is that we don't have a stable interface
>> for
>>  our job graphs/pipelines right now.
>> 
>>  Best,
>>  Aljoscha
>> 
>>  On 06.02.20 23:00, Gyula Fóra wrote:
>> > Hi Jeff & Till!
>> >
>> > Thanks for the feedback, this is exactly the discussion I was
>> looking
>>  for.
>> > The JobListener looks very p

[VOTE] FLIP-99: Make Max Exception Configurable

[Yadong Xie]

Hi all

I want to start the vote for FLIP-99, which proposes to make the max
exception configurable in web UI.

To help everyone better understand the proposal, we spent some efforts on
making an online POC

previous web:
http://101.132.122.69:8081/#/job/543e9dc0cb2cca4433116007f0931d1a/exceptions
POC web:
http://101.132.122.69:8081/web/#/job/543e9dc0cb2cca4433116007f0931d1a/exceptions

The vote will last for at least 72 hours, following the consensus voting
process.

FLIP wiki:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-99%3A+Make+Max+Exception+Configurable

Discussion thread:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html

Thanks,

Yadong

[VOTE] FLIP-100: Add Attempt Information

[Yadong Xie]

Hi all

I want to start the vote for FLIP-100, which proposes to add attempt
information inside subtask and timeline in web UI.

To help everyone better understand the proposal, we spent some efforts on
making an online POC

Timeline Attempt (click the vertex timeline to see the differences):
previous web:
http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/timeline
POC web:
http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/timeline

Subtask Attempt (click the vertex and switch to subtask tab to see the
differences):
previous web:
http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/overview
POC web:
http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/overview


The vote will last for at least 72 hours, following the consensus voting
process.

FLIP wiki:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information

Discussion thread:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html

Thanks,

Yadong

[VOTE] FLIP-101: Add Pending Slots Detail

[Yadong Xie]

Hi all

I want to start the vote for FLIP-101, which proposes to add pending slots
information to help users check which vertex/subtask is blocked.

To help everyone better understand the proposal, we spent some efforts on
making an online POC

previous web:
http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
POC web:
http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots


The vote will last for at least 72 hours, following the consensus voting
process.

FLIP wiki:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail

Discussion thread:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html

Thanks,

Yadong

[VOTE] FLIP-102: Add More Metrics to TaskManager

[Yadong Xie]

Hi all

I want to start the vote for FLIP-102, which proposes to add more metrics
to the task manager in web UI.

To help everyone better understand the proposal, we spent some efforts on
making an online POC

previous web:
http://101.132.122.69:8081/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/metrics
POC web:
http://101.132.122.69:8081/web/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/metrics


The vote will last for at least 72 hours, following the consensus voting
process.

FLIP wiki:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager

Discussion thread:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html

Thanks,

Yadong

[VOTE] FLIP-103: Better TM/JM Log Display

[Yadong Xie]

Hi all

I want to start the vote for FLIP-103, which proposes to provide better
user log display UI in task manager and job manager.

To help everyone better understand the proposal, we spent some efforts on
making an online POC

Task manager:

previous web:
http://101.132.122.69:8081/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/logs
POC web:
http://101.132.122.69:8081/web/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/log

Job manager:

previous web: http://101.132.122.69:8081/#/job-manager/logs
POC web: http://101.132.122.69:8081/web/#/job-manager/log


The vote will last for at least 72 hours, following the consensus voting
process.

FLIP wiki:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143

Discussion thread:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html

Thanks,

Yadong

[jira] [Created] (FLINK-16185) Support JSON functions provides by calcite for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16185:
-

 Summary: Support JSON functions provides by calcite for blink 
planner
 Key: FLINK-16185
 URL: https://issues.apache.org/jira/browse/FLINK-16185
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / API
Reporter: Zili Chen
 Fix For: 1.11.0


Calcite 1.20 provides several built-in JSON functions(listed below). In order 
to enrich our builtin UDFs and according to our internal user experiences I'd 
propose to support those functions for blink planner.

cc [~jark] [~docete] what do you think?

JSON_EXISTS
JSON_QUERY
JSON_OBJECT
JSON_ARRAY
IS_JSON_ARRAY
IS_JSON_OBJECT
IS_JSON_SCALAR
IS_JSON_VALUE
IS_NOT_JSON_ARRAY
IS_NOT_JSON_OBJECT
IS_NOT_JSON_SCALAR
IS_NOT_JSON_VALUE
JSON_VALUE
JSON_VALUE_ANY
JSON_VALUE_EXPRESSION

JSON_OBJECTAGG
JSON_ARRAYAGG



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] FLIP-104: Add More Metrics to Jobmanager

[Yadong Xie]

Hi all

I want to start the vote for FLIP-104, which proposes to add more metrics
to job manager.

To help everyone better understand the proposal, we spent some efforts on
making an online POC

previous web: http://101.132.122.69:8081/#/job-manager/config
POC web: http://101.132.122.69:8081/web/#/job-manager/metrics


The vote will last for at least 72 hours, following the consensus voting
process.

FLIP wiki:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-104%3A+Add+More+Metrics+to+Jobmanager

Discussion thread:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html

Thanks,

Yadong

Re: [DISCUSS] Kicking off the 1.11 release cycle

[Zhu Zhu]

Thanks Piotr and Zhijiang for volunteering as the release managers!

+1 for aiming with the feature freeze date for end of April.

Thanks,
Zhu Zhu

Piotr Nowojski  于2020年2月20日周四 下午4:26写道：

> Hi,
>
> Thank you for giving me and Zhijiang opportunity and entrusting us with
> the release :)
>
> +1 for aiming with the feature freeze for and of April. I think all things
> consider that’s a good timing.
>
> Piotrek
>
> > On 20 Feb 2020, at 04:22, Yu Li  wrote:
> >
> > Thanks Stephan for the kicking off! And thanks Piotr and Zhijiang for
> > volunteering as the release managers!
> >
> > Aiming for a shorter release cycle than 1.10 makes a lot of sense, and +1
> > to target the feature freeze date around end of April.
> >
> > Best Regards,
> > Yu
> >
> >
> > On Thu, 20 Feb 2020 at 10:37, Xintong Song 
> wrote:
> >
> >> Thanks for the kicking off.
> >>
> >> +1 for Zhijiang & Piotr (if he agrees to) being the release managers.
> >> Thanks for the volunteering.
> >> +1 for the *anticipated feature freeze date* end of April
> >>
> >> Thank you~
> >>
> >> Xintong Song
> >>
> >>
> >>
> >> On Thu, Feb 20, 2020 at 10:30 AM Yuan Mei 
> wrote:
> >>
> >>> +1
> >>>
> >>> 3-month sounds a very reasonable duration for a release.
> >>>
> >>> On Thu, Feb 20, 2020 at 9:53 AM tison  wrote:
> >>>
>  Thanks for kicking off the discussion Stephan!
> 
>  +1 for the *anticipated feature freeze date* end of April and
>  Zhijiang & Piotr to be the release managers.
> 
>  Best,
>  tison.
> 
> 
>  Dian Fu  于2020年2月20日周四 上午9:41写道：
> 
> > Thanks Stephan kicking off this discussion and Zhijiang volunteering
> >> as
> > one of the release managers.
> >
> > +1 for the "feature freeze date" around end of April. There are still
>  more
> > than 2 months left, so I think it's a reasonable time.
> >
> > Thanks,
> > Dian
> >
> >> 在 2020年2月19日，下午10:52，Aljoscha Krettek  写道：
> >>
> >> +1
> >>
> >> Although I would hope that it can be more than just "anticipated".
> >>
> >> Best,
> >> Aljoscha
> >>
> >> On 19.02.20 15:40, Till Rohrmann wrote:
> >>> Thanks for volunteering as one of our release managers Zhijiang.
> >>> +1 for the *anticipated feature freeze date* end of April. As we
> >> go
> > along
> >>> and collect more data points we might be able to strengthen our
> >>> initial anticipation.
> >>> Cheers,
> >>> Till
> >>> On Wed, Feb 19, 2020 at 4:44 AM Zhijiang <
> >>> wangzhijiang...@aliyun.com
> > .invalid>
> >>> wrote:
>  Thanks for kicking off the next release and the introduction,
>  @Stephan!
> 
>  It's my pleasure to become the release manager and involve in
> >> other
>  community works. I am working together with @Piotr for a long
> >> time,
>  so
>  very happy to cooperate for the release manager work again. The
> > previous
>  release work was always well done, and I can learn a lot from
> >> these
> > rich
>  experiences.
> 
>  +1 for the "feature freeze date" around end of April.
>  Although we have the FF SF in the meantime, fortunately there
> >> are
> >>> no
> > long
>  public holidays during this period in China.
> 
>  Best,
>  Zhijiang
> 
> 
> 
> >> --
>  From:Stephan Ewen 
>  Send Time:2020 Feb. 19 (Wed.) 01:15
>  To:dev 
>  Cc:zhijiang ; pnowojski <
> >> pnowoj...@apache.org
> 
>  Subject:[DISCUSS] Kicking off the 1.11 release cycle
> 
>  Hi all!
> 
>  Now that the 1.10 release is out (congrats again, everyone!), I
>  wanted
> > to
>  bring up some questions about organizing the next release cycle.
> 
>  The most important questions at the beginning would be
>   - Who would volunteer as Release Managers
>   - When would be the release date.
> 
>  For the release managers, Piotrek and Zhijiang have mentioned
> > previously
>  that they would be interested, so I am copying them here to chime
> >>> in.
>  @Piotr and @Zhijiang could you confirm if you are interested in
>  helping
>  out with this?
> 
>  About the release date: By our original "3 months release cycle"
>  assumption, we should aim for a release **Mid May**, meaning a
>  feature
>  freeze no later than end of April.
>  That would be indeed a shorter release cycle than 1.10, and the
> > assumption
>  of a shorter testing period. But aiming for a shorter release
> >> cycle
> > than
>  1.10 would actually be nice, in my opinion. 1.10 grew very big in
> >>> the
> > end,
>  which caused also a very long testing period (plus Christmas and
> > Chines

Re: [DISCUSS] Extend (or maintain) "shell" script support for Windows

[Chesnay Schepler]

Realistically you can't really use the windows scripts in production 
since you can only start a standalone cluster consisting of 1 JM and 1 TM.


The .bat scripts are already limited to a significant degree.

On 20/02/2020 03:48, Xintong Song wrote:

I'm also in favor of dropping the bat scripts. I can hardly image
people using Flink Windows deployment in their production. But maybe
throwing a survey on the user ML to double check on that?

Thank you~

Xintong Song



On Thu, Feb 20, 2020 at 10:28 AM Yang Wang  wrote:


I tend to drop the windows bat script to start/stop the cluster. Since
there
are various windows versions(e.g. xp, win7, win10, etc.) and do not have
sensational compatibility. Also we do not have the e2e tests to guarantee
the scripts work well.

On the other hand, i completely agree that local deployment, especially in
the IDE, should work on windows. For standalone deployment, i think using
the docker is a good choice. I think many windows users are also using
docker. For Yarn/K8s deployment, i am not sure whether we need to support
to submit a Flink session/per-job cluster from windows machine.


Best,
Yang

Chesnay Schepler  于2020年2月20日周四 上午12:15写道：


Note that with FLINK-15925 we effectively broke the start-cluster.bat
script in 1.10.

Overall, I would be fine with dropping the .bat scripts because they are
a big maintenance problem for us.
We don't have anyone using these scripts regularly on our end (even I
don't).
If there were a way to unify the unix/windows scripts I would be all
ears, but this doesn't seem possible without lots and lots of branches.

However, we should definitely ensure that Flink continues to work in the
IDE on windows for training/demo purposes.

As for CI, with the new Azure setup this may actually be feasible now
from a CI budged perspective. But it would require a fair upfront time
investment to fix failing tests or adding assumptions for failing tests.
That said, this would have some nice benefits for finding resource
leaks, particularly in the filesystem.

On 19/02/2020 16:46, Aljoscha Krettek wrote:

Hi,

the background is this series of Jira Issues and PRs around extending
the .bat scripts for windows:
https://issues.apache.org/jira/browse/FLINK-5333.

I would like to resolve this, by either closing the Jira Issues as
"Won't Do" or finally merging these PRs. The questions I have are:

  - Should we add more full-featured (complicated?) windows scripts
that are essentially re-implementations of our existing "unix" scripts?
  - Would windows users use these or would they, by now, use the linux
subsystem for windows or cygwin?
  - Should we even remove the existing .bat scripts that we have?

Maintaining the windows scripts is hard because we only have one (I
think, Chesnay) developer on windows and no CI for windows.

What do you think?

Best,
Aljoscha

Re: [DISCUSS] Kicking off the 1.11 release cycle

[Jingsong Li]

Thanks Stephan for the kicking off.
Thanks Piotr and Zhijiang for volunteering.

+1 for aiming with the feature freeze date for end of April.
We should return to the quick release model. (3 months)

Best,
Jingsong Lee

On Thu, Feb 20, 2020 at 6:04 PM Zhu Zhu  wrote:

> Thanks Piotr and Zhijiang for volunteering as the release managers!
>
> +1 for aiming with the feature freeze date for end of April.
>
> Thanks,
> Zhu Zhu
>
> Piotr Nowojski  于2020年2月20日周四 下午4:26写道：
>
> > Hi,
> >
> > Thank you for giving me and Zhijiang opportunity and entrusting us with
> > the release :)
> >
> > +1 for aiming with the feature freeze for and of April. I think all
> things
> > consider that’s a good timing.
> >
> > Piotrek
> >
> > > On 20 Feb 2020, at 04:22, Yu Li  wrote:
> > >
> > > Thanks Stephan for the kicking off! And thanks Piotr and Zhijiang for
> > > volunteering as the release managers!
> > >
> > > Aiming for a shorter release cycle than 1.10 makes a lot of sense, and
> +1
> > > to target the feature freeze date around end of April.
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Thu, 20 Feb 2020 at 10:37, Xintong Song 
> > wrote:
> > >
> > >> Thanks for the kicking off.
> > >>
> > >> +1 for Zhijiang & Piotr (if he agrees to) being the release managers.
> > >> Thanks for the volunteering.
> > >> +1 for the *anticipated feature freeze date* end of April
> > >>
> > >> Thank you~
> > >>
> > >> Xintong Song
> > >>
> > >>
> > >>
> > >> On Thu, Feb 20, 2020 at 10:30 AM Yuan Mei 
> > wrote:
> > >>
> > >>> +1
> > >>>
> > >>> 3-month sounds a very reasonable duration for a release.
> > >>>
> > >>> On Thu, Feb 20, 2020 at 9:53 AM tison  wrote:
> > >>>
> >  Thanks for kicking off the discussion Stephan!
> > 
> >  +1 for the *anticipated feature freeze date* end of April and
> >  Zhijiang & Piotr to be the release managers.
> > 
> >  Best,
> >  tison.
> > 
> > 
> >  Dian Fu  于2020年2月20日周四 上午9:41写道：
> > 
> > > Thanks Stephan kicking off this discussion and Zhijiang
> volunteering
> > >> as
> > > one of the release managers.
> > >
> > > +1 for the "feature freeze date" around end of April. There are
> still
> >  more
> > > than 2 months left, so I think it's a reasonable time.
> > >
> > > Thanks,
> > > Dian
> > >
> > >> 在 2020年2月19日，下午10:52，Aljoscha Krettek  写道：
> > >>
> > >> +1
> > >>
> > >> Although I would hope that it can be more than just "anticipated".
> > >>
> > >> Best,
> > >> Aljoscha
> > >>
> > >> On 19.02.20 15:40, Till Rohrmann wrote:
> > >>> Thanks for volunteering as one of our release managers Zhijiang.
> > >>> +1 for the *anticipated feature freeze date* end of April. As we
> > >> go
> > > along
> > >>> and collect more data points we might be able to strengthen our
> > >>> initial anticipation.
> > >>> Cheers,
> > >>> Till
> > >>> On Wed, Feb 19, 2020 at 4:44 AM Zhijiang <
> > >>> wangzhijiang...@aliyun.com
> > > .invalid>
> > >>> wrote:
> >  Thanks for kicking off the next release and the introduction,
> >  @Stephan!
> > 
> >  It's my pleasure to become the release manager and involve in
> > >> other
> >  community works. I am working together with @Piotr for a long
> > >> time,
> >  so
> >  very happy to cooperate for the release manager work again. The
> > > previous
> >  release work was always well done, and I can learn a lot from
> > >> these
> > > rich
> >  experiences.
> > 
> >  +1 for the "feature freeze date" around end of April.
> >  Although we have the FF SF in the meantime, fortunately there
> > >> are
> > >>> no
> > > long
> >  public holidays during this period in China.
> > 
> >  Best,
> >  Zhijiang
> > 
> > 
> > 
> > >> --
> >  From:Stephan Ewen 
> >  Send Time:2020 Feb. 19 (Wed.) 01:15
> >  To:dev 
> >  Cc:zhijiang ; pnowojski <
> > >> pnowoj...@apache.org
> > 
> >  Subject:[DISCUSS] Kicking off the 1.11 release cycle
> > 
> >  Hi all!
> > 
> >  Now that the 1.10 release is out (congrats again, everyone!), I
> >  wanted
> > > to
> >  bring up some questions about organizing the next release cycle.
> > 
> >  The most important questions at the beginning would be
> >   - Who would volunteer as Release Managers
> >   - When would be the release date.
> > 
> >  For the release managers, Piotrek and Zhijiang have mentioned
> > > previously
> >  that they would be interested, so I am copying them here to
> chime
> > >>> in.
> >  @Piotr and @Zhijiang could you confirm if you are interested in
> >  helping
> >  out with this?
> > 
> > >>

Re: [DISCUSS] Kicking off the 1.11 release cycle

[lining jing]

Thanks for kicking off.

+1 for the *anticipated feature freeze date* end of April.

Zhu Zhu  于2020年2月20日周四 下午6:04写道：

> Thanks Piotr and Zhijiang for volunteering as the release managers!
>
> +1 for aiming with the feature freeze date for end of April.
>
> Thanks,
> Zhu Zhu
>
> Piotr Nowojski  于2020年2月20日周四 下午4:26写道：
>
> > Hi,
> >
> > Thank you for giving me and Zhijiang opportunity and entrusting us with
> > the release :)
> >
> > +1 for aiming with the feature freeze for and of April. I think all
> things
> > consider that’s a good timing.
> >
> > Piotrek
> >
> > > On 20 Feb 2020, at 04:22, Yu Li  wrote:
> > >
> > > Thanks Stephan for the kicking off! And thanks Piotr and Zhijiang for
> > > volunteering as the release managers!
> > >
> > > Aiming for a shorter release cycle than 1.10 makes a lot of sense, and
> +1
> > > to target the feature freeze date around end of April.
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Thu, 20 Feb 2020 at 10:37, Xintong Song 
> > wrote:
> > >
> > >> Thanks for the kicking off.
> > >>
> > >> +1 for Zhijiang & Piotr (if he agrees to) being the release managers.
> > >> Thanks for the volunteering.
> > >> +1 for the *anticipated feature freeze date* end of April
> > >>
> > >> Thank you~
> > >>
> > >> Xintong Song
> > >>
> > >>
> > >>
> > >> On Thu, Feb 20, 2020 at 10:30 AM Yuan Mei 
> > wrote:
> > >>
> > >>> +1
> > >>>
> > >>> 3-month sounds a very reasonable duration for a release.
> > >>>
> > >>> On Thu, Feb 20, 2020 at 9:53 AM tison  wrote:
> > >>>
> >  Thanks for kicking off the discussion Stephan!
> > 
> >  +1 for the *anticipated feature freeze date* end of April and
> >  Zhijiang & Piotr to be the release managers.
> > 
> >  Best,
> >  tison.
> > 
> > 
> >  Dian Fu  于2020年2月20日周四 上午9:41写道：
> > 
> > > Thanks Stephan kicking off this discussion and Zhijiang
> volunteering
> > >> as
> > > one of the release managers.
> > >
> > > +1 for the "feature freeze date" around end of April. There are
> still
> >  more
> > > than 2 months left, so I think it's a reasonable time.
> > >
> > > Thanks,
> > > Dian
> > >
> > >> 在 2020年2月19日，下午10:52，Aljoscha Krettek  写道：
> > >>
> > >> +1
> > >>
> > >> Although I would hope that it can be more than just "anticipated".
> > >>
> > >> Best,
> > >> Aljoscha
> > >>
> > >> On 19.02.20 15:40, Till Rohrmann wrote:
> > >>> Thanks for volunteering as one of our release managers Zhijiang.
> > >>> +1 for the *anticipated feature freeze date* end of April. As we
> > >> go
> > > along
> > >>> and collect more data points we might be able to strengthen our
> > >>> initial anticipation.
> > >>> Cheers,
> > >>> Till
> > >>> On Wed, Feb 19, 2020 at 4:44 AM Zhijiang <
> > >>> wangzhijiang...@aliyun.com
> > > .invalid>
> > >>> wrote:
> >  Thanks for kicking off the next release and the introduction,
> >  @Stephan!
> > 
> >  It's my pleasure to become the release manager and involve in
> > >> other
> >  community works. I am working together with @Piotr for a long
> > >> time,
> >  so
> >  very happy to cooperate for the release manager work again. The
> > > previous
> >  release work was always well done, and I can learn a lot from
> > >> these
> > > rich
> >  experiences.
> > 
> >  +1 for the "feature freeze date" around end of April.
> >  Although we have the FF SF in the meantime, fortunately there
> > >> are
> > >>> no
> > > long
> >  public holidays during this period in China.
> > 
> >  Best,
> >  Zhijiang
> > 
> > 
> > 
> > >> --
> >  From:Stephan Ewen 
> >  Send Time:2020 Feb. 19 (Wed.) 01:15
> >  To:dev 
> >  Cc:zhijiang ; pnowojski <
> > >> pnowoj...@apache.org
> > 
> >  Subject:[DISCUSS] Kicking off the 1.11 release cycle
> > 
> >  Hi all!
> > 
> >  Now that the 1.10 release is out (congrats again, everyone!), I
> >  wanted
> > > to
> >  bring up some questions about organizing the next release cycle.
> > 
> >  The most important questions at the beginning would be
> >   - Who would volunteer as Release Managers
> >   - When would be the release date.
> > 
> >  For the release managers, Piotrek and Zhijiang have mentioned
> > > previously
> >  that they would be interested, so I am copying them here to
> chime
> > >>> in.
> >  @Piotr and @Zhijiang could you confirm if you are interested in
> >  helping
> >  out with this?
> > 
> >  About the release date: By our original "3 months release cycle"
> >  assumption, we should aim for a release **Mid May**, meaning a
> > >

Re: [DISCUSS] Extend (or maintain) "shell" script support for Windows

[Stephan Ewen]

I think this question should go on the user mailing list, not on dev.

You can also use Twitter to collect some data points.

On Thu, Feb 20, 2020 at 11:07 AM Chesnay Schepler 
wrote:

> Realistically you can't really use the windows scripts in production
> since you can only start a standalone cluster consisting of 1 JM and 1 TM.
>
> The .bat scripts are already limited to a significant degree.
>
> On 20/02/2020 03:48, Xintong Song wrote:
> > I'm also in favor of dropping the bat scripts. I can hardly image
> > people using Flink Windows deployment in their production. But maybe
> > throwing a survey on the user ML to double check on that?
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Thu, Feb 20, 2020 at 10:28 AM Yang Wang 
> wrote:
> >
> >> I tend to drop the windows bat script to start/stop the cluster. Since
> >> there
> >> are various windows versions(e.g. xp, win7, win10, etc.) and do not have
> >> sensational compatibility. Also we do not have the e2e tests to
> guarantee
> >> the scripts work well.
> >>
> >> On the other hand, i completely agree that local deployment, especially
> in
> >> the IDE, should work on windows. For standalone deployment, i think
> using
> >> the docker is a good choice. I think many windows users are also using
> >> docker. For Yarn/K8s deployment, i am not sure whether we need to
> support
> >> to submit a Flink session/per-job cluster from windows machine.
> >>
> >>
> >> Best,
> >> Yang
> >>
> >> Chesnay Schepler  于2020年2月20日周四 上午12:15写道：
> >>
> >>> Note that with FLINK-15925 we effectively broke the start-cluster.bat
> >>> script in 1.10.
> >>>
> >>> Overall, I would be fine with dropping the .bat scripts because they
> are
> >>> a big maintenance problem for us.
> >>> We don't have anyone using these scripts regularly on our end (even I
> >>> don't).
> >>> If there were a way to unify the unix/windows scripts I would be all
> >>> ears, but this doesn't seem possible without lots and lots of branches.
> >>>
> >>> However, we should definitely ensure that Flink continues to work in
> the
> >>> IDE on windows for training/demo purposes.
> >>>
> >>> As for CI, with the new Azure setup this may actually be feasible now
> >>> from a CI budged perspective. But it would require a fair upfront time
> >>> investment to fix failing tests or adding assumptions for failing
> tests.
> >>> That said, this would have some nice benefits for finding resource
> >>> leaks, particularly in the filesystem.
> >>>
> >>> On 19/02/2020 16:46, Aljoscha Krettek wrote:
>  Hi,
> 
>  the background is this series of Jira Issues and PRs around extending
>  the .bat scripts for windows:
>  https://issues.apache.org/jira/browse/FLINK-5333.
> 
>  I would like to resolve this, by either closing the Jira Issues as
>  "Won't Do" or finally merging these PRs. The questions I have are:
> 
>    - Should we add more full-featured (complicated?) windows scripts
>  that are essentially re-implementations of our existing "unix"
> scripts?
>    - Would windows users use these or would they, by now, use the linux
>  subsystem for windows or cygwin?
>    - Should we even remove the existing .bat scripts that we have?
> 
>  Maintaining the windows scripts is hard because we only have one (I
>  think, Chesnay) developer on windows and no CI for windows.
> 
>  What do you think?
> 
>  Best,
>  Aljoscha
> 
> >>>
>
>

Re: [DISCUSS] FLIP-75: Flink Web UI Improvement Proposal

[Yadong Xie]

Hi all

Thanks to everyone involved in the discussion

I have started the voting thread for all the sub FLIPs (including FLIP-98,
FLIP-99, FLIP-100, FLIP-101, FLIP-102, FLIP-103, and FLIP-104)

Please take some time to vote on the FLIP you are interested in, thanks


lining jing  于2020年2月19日周三 下午3:54写道：

> I think we can create the PR for the GC status later if we could find an
> easy way to obtain it, before that the users could get GC logs from the
> FLIP-103.
>
> By the way, there is a similar topic 'FlameGraph In Job Vertex' in FLIP-75
> in the early discussion stage
>
> https://docs.google.com/document/d/1tIa8yN2prWWKJI_fa1u0t6h1r6RJpp56m48pXEyh6iI/edit#heading=h.6xcek9hyxzu0
> ,
> we move it into another FLIP to discuss later since FLIP-75 is heavy
> enough.
>
> And I have created FLINK-15422
>  to get more metrics
> for
> JVM's memory.
> Piotr Nowojski  于2020年2月18日周二 下午7:11写道：
>
> > Hi,
> >
> > A quick question/comment about FLIP-102. Have you thought about adding GC
> > stats? I’m not sure what’s easily do-able, but something that would allow
> > user to see GC issues (long/frequent pauses, lots of CPU time spent in
> the
> > GC) would be quite useful for analysing performance/stability issues,
> > without a need of connecting profilers in a distributed environment?
> >
> > Piotrek
> >
> > > On 10 Feb 2020, at 10:58, Yadong Xie  wrote:
> > >
> > > Hi all
> > > I have drafted the docs of top-level FLIPs for the individual changes
> > > proposed in FLIP-75.
> > > will update it to the cwiki page and start the voting stage soon if
> there
> > > is no objection.
> > >
> > >   - FLIP-98: Better Back Pressure Detection
> > >   <
> >
> https://docs.google.com/document/d/1b4GadCze-36x5TPHz6ie4WI9fOUuxWoT_rWWgeg68oo/edit?usp=sharing
> > >
> > >   - FLIP-99: Make Max Exception Configurable
> > >   <
> >
> https://docs.google.com/document/d/1tsPpTEx5WqliOAUC924xzRxYOalUuB-GoznGPcxSzJo/edit?usp=sharing
> > >
> > >   - FLIP-100: Add Attempt Information
> > >   <
> >
> https://docs.google.com/document/d/1Ww7biOr6WMVfoYhtBTJftRqEm9FGo33AXgYibdXy47Y/edit?usp=sharing
> > >
> > >   - FLIP-101: Add Pending Slots Tab in Job Detail
> > >   <
> >
> https://docs.google.com/document/d/1ttn7zIn_Z237JOHdmhiei6aCwKdjTU53I07XxA61Fis/edit?usp=sharing
> > >
> > >   - FLIP-102: Add More Metrics to TaskManager
> > >   <
> >
> https://docs.google.com/document/d/18yHdsqUJ1FmNRm0hyeCm3nWvPFpvpJgTJ8BYNAa6Ul8/edit?usp=sharing
> > >
> > >   - FLIP-103: Better Taskmanager Log Display
> > >   <
> >
> https://docs.google.com/document/d/16eEdW2KeLxvABdoXahx4MMMisW4_P9mKiqUE0F4GO1c/edit?usp=sharing
> > >
> > >   - FLIP-104: Add More Metrics to Jobmanager
> > >   <
> >
> https://docs.google.com/document/d/1Fak632iOroOLZFADqwZWu2SS-LUQqCLHnm8Vs3XM5to/edit?usp=sharing
> > >
> > >   - FLIP-105: Better Jobmanager Log Display
> > >   <
> >
> https://docs.google.com/document/d/1ayXaZflelaymQuF3l6UOuGEg6zzbSGGewoDq-9SBOPY/edit?usp=sharing
> > >
> > >
> > >
> > > Yadong Xie  于2020年2月9日周日 下午7:24写道：
> > >
> > >> Hi Till
> > >> I got your point, will create sub FLIPs and votings according to the
> > >> FLIP-75 and previous discussion soon.
> > >>
> > >> Till Rohrmann  于2020年2月9日周日 下午5:27写道：
> > >>
> > >>> Hi Yadong,
> > >>>
> > >>> I think it would be fine to simply link to this discussion thread to
> > keep
> > >>> the discussion history. Maybe an easier way would be to create
> > top-level
> > >>> FLIPs for the individual changes proposed in FLIP-75. The reason I'm
> > >>> proposing this is that it would be easier to vote on it and to
> > implement
> > >>> it
> > >>> because the scope is smaller. But maybe I'm wrong here and others
> could
> > >>> chime in to voice their opinion.
> > >>>
> > >>> Cheers,
> > >>> Till
> > >>>
> > >>> On Fri, Feb 7, 2020 at 9:58 AM Yadong Xie 
> wrote:
> > >>>
> >  Hi Till
> > 
> >  FLIP-75 has been open since September, and the design doc has been
> > >>> iterated
> >  over 3 versions and more than 20 patches.
> >  I had a try, but it is hard to split the design docs into sub FLIP
> and
> > >>> keep
> >  all the discussion history at the same time.
> > 
> >  Maybe it is better to start another discussion to talk about the
> > >>> individual
> >  sub FLIP voting? and make the next FLIP follow the new practice if
> >  possible.
> > 
> >  Till Rohrmann  于2020年2月3日周一 下午6:28写道：
> > 
> > > I think there is no such description because we never did it
> before.
> > I
> >  just
> > > figured that FLIP-75 could actually be a good candidate to start
> this
> > > practice. We would need a community discussion first, though.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Mon, Feb 3, 2020 at 10:28 AM Yadong Xie 
> > >>> wrote:
> > >
> > >> Hi Till
> > >> I didn’t find how to create of sub flip at cwiki.apache.org
> > >> do you mean to create 9 more FLIPS instead of FLIP-75?
> > >>>

[DISCUSS] Drop Savepoint Compatibility with Flink 1.2

[Stephan Ewen]

Hi all!

For some cleanup and simplifications, it would be helpful to drop Savepoint
compatibility with Flink version 1.2. That version was released almost
three years ago.

I would expect that no one uses that old version any more in a way that
they actively want to upgrade directly to 1.11.

Even if, there is still the way to first upgrade to another version (like
1.9) and then upgrade to 1.11 from there.

Any concerns to drop that support?

Best,
Stephan

[jira] [Created] (FLINK-16186) Speed up ElasticsearchITCase#testInvalidElasticsearchCluster

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-16186:


 Summary: Speed up 
ElasticsearchITCase#testInvalidElasticsearchCluster
 Key: FLINK-16186
 URL: https://issues.apache.org/jira/browse/FLINK-16186
 Project: Flink
  Issue Type: Bug
  Components: Connectors / ElasticSearch, Tests
Affects Versions: 1.9.0
Reporter: Chesnay Schepler


{{ElasticsearchITCase#testInvalidElasticsearchCluster}} runs for a significant 
time (30-40 seconds), as it tries to connect to a non-existent clusters.

We should find a way to reduce the timeout, or implement the test in some other 
fashion.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Improvements on FLIP Process

[Hequn Cheng]

Hi everyone,

I appreciate all the feedback! Let me try to explain more.

As there are a lot of people show concerns about the google doc.
Let’s make an assumption first that the comment on the google doc is not
allowed!
Let’s make an assumption first that the comment on the google doc is not
allowed!!
Let’s make an assumption first that the comment on the google doc is not
allowed!!!
(As for whether to allow (minor)comments on the google doc. I will raise
another discussion if we reach a consensus on this one.)

The main purpose of this discussion is to improve the current FLIP process.
Before we introduce the improvements, let’s first take a look at the
problems we have. This is the motivation why this discussion is raised. The
problems I find are listed as follows:
1. During the FLIP discussion, whenever there is a change, we need to pick
the change to the wiki page. This is somehow redundant.
2. Even after the proposal has been finalized, the wiki page can be changed
as it is editable to everyone. However, any change to an adopted FLIP
requires a new voting process.
3. The FLIP page is not well maintained, e.g., the status of many FLIPs
were not updated in time. You can find there are a lot of FLIPs with the
target release of 1.10 on the current wiki page.

Next, let’s see how the problems listed above can be solved by the new
process. I will list the new process again for discussion convenience.

1. Raise the discussion on the mailing list. The subject of the thread is
of the format [DISCUSS][FLIP] {your FLIP heading}. Also, the design doc
should follow the FLIP-Template strictly. (The [FLIP] tag is used to inform
people that it is a FLIP discussion and more attention should be paid.)
2. Create a FLIP wiki page(by a committer who wants to shepherd the FLIP)
once we reach an agreement on the discussion. We can simply copy the google
doc into the FLIP wiki page. The name of the shepherd should also be listed
on the wiki page.
3. Once the proposal is finalized, call a vote to have the proposal
adopted. It should be noted that we should always vote on a FLIP wiki page
instead of a google doc. The wiki page is the final version of the google
doc.

According to the new process:
Problem1 can be improved since we only create the FLIP wiki page when we
reached an agreement.
Problem2 can be improved since we can make the FLIP wiki page committer
editable. Not everyone can change the wiki page freely as it is now.
Problem3 can be improved as the FLIP wiki page is created and updated by
the committer. Committers are assumed to be more active than contributors.
This can also be easier to info the guy who is responsible for updating the
wiki page.

Looking forward to your feedback!

Best,
Hequn

On Wed, Feb 19, 2020 at 10:20 PM Till Rohrmann  wrote:

> +1 for Aljoscha's proposal.
>
> This, of course, does not mean that one cannot use Google docs in order to
> prepare the FLIP discussion.
>
> Cheers,
> Till
>
> On Wed, Feb 19, 2020 at 11:15 AM Xintong Song 
> wrote:
>
> > I'm also not a fun of discussing FLIPs with google docs.
> >
> > I think google docs is probably ok for smaller scope early discussions
> > before raising the discussion on the mailing list, when the draft is not
> > completed and is expected to change frequently. Once it is proposed to
> the
> > community, as many people already mentioned, google doc changes are very
> > hard to track.
> >
> > If I understand correctly, what Jincheng suggested is to use google doc
> but
> > not allowing discussions and modifications on it, except for minor
> issues.
> > Regarding that, my concerns are:
> > - How do we define "minor issues"? Are these typo and grammar issues
> only?
> >   - If so, I think it is the proposer's responsibility to provide
> > well-written docs with less such mistakes, if not none. Most editors
> > provide helpful spelling and grammar checks.
> >   - If not, then people may have different opinions on whether a comment
> is
> > minor or not.
> >
> > Another advantage for the wiki page is that, once you watch it you can
> > always get email notifications on modifications to that page. AFAIK, for
> > google doc you get notifications only if someone replies your comments,
> > unless you're the owner of the doc.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Wed, Feb 19, 2020 at 12:57 PM jincheng sun 
> > wrote:
> >
> > > Hi all,
> > >
> > > Thanks for bring up the discussion @Hequn!
> > >
> > > I agree with some concerns raised above, however, I would like to give
> my
> > > +1 for the proposal and I would like to share my thoughts:
> > >
> > > If I understand correctly, the proposal doesn’t encourage people to
> > discuss
> > > in the google doc, the first step of the proposal is to raise the
> > > discussion on the mailing list.
> > >
> > > It’s common sense to discuss on the mailing list even with a google
> doc.
> > > This is also the current status and works well. Most people know that
> we
> > > should focus the discussion

Re: [ANNOUNCE] Apache Flink-shaded 10.0 released

[Hequn Cheng]

Thanks a lot for the release Chesnay!
Also thanks to everyone who contributes to this release!

Best, Hequn

On Thu, Feb 20, 2020 at 11:11 AM Yu Li  wrote:

> Thanks Chesnay and all participants for making the release possible!
>
> Best Regards,
> Yu
>
>
> On Thu, 20 Feb 2020 at 09:50, Zhu Zhu  wrote:
>
> > Thanks Chesnay for the great work and everyone who helps with the
> > improvements and release!
> >
> > Thanks,
> > Zhu Zhu
> >
> > Dian Fu  于2020年2月20日周四 上午9:44写道：
> >
> > > Thanks Chesnay for the great work and everyone involved!
> > >
> > > Regards,
> > > Dian
> > >
> > > > 在 2020年2月20日，上午12:21，Zhijiang 
> 写道：
> > > >
> > > > Thanks Chesnay for making the release efficiently and also thanks to
> > all
> > > the other participants!
> > > >
> > > > Best,
> > > > Zhijiang
> > > >
> > > >
> > > > --
> > > > From:Till Rohrmann  trohrm...@apache.org
> > >>
> > > > Send Time:2020 Feb. 19 (Wed.) 22:21
> > > > To:dev mailto:dev@flink.apache.org>>
> > > > Subject:Re: [ANNOUNCE] Apache Flink-shaded 10.0 released
> > > >
> > > > Thanks for making the release possible Chesnay and everyone who was
> > > > involved!
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Wed, Feb 19, 2020 at 7:47 AM jincheng sun <
> sunjincheng...@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> Thanks a lot for the release Chesnay!
> > > >> And thanks to everyone who make this release possible!
> > > >>
> > > >> Best,
> > > >> Jincheng
> > > >>
> > > >>
> > > >> Chesnay Schepler  于2020年2月19日周三 上午12:45写道：
> > > >>
> > > >>> The Apache Flink community is very happy to announce the release of
> > > >>> Apache Flink-shaded 10.0.
> > > >>>
> > > >>> The flink-shaded project contains a number of shaded dependencies
> for
> > > >>> Apache Flink.
> > > >>>
> > > >>> Apache Flink(r) is an open-source stream processing framework for
> > > >>> distributed, high-performing, always-available, and accurate data
> > > >>> streaming applications.
> > > >>>
> > > >>> The release is available for download at:
> > > >>> https://flink.apache.org/downloads.html
> > > >>>
> > > >>> The full release notes are available in Jira:
> > > >>>
> > > >>>
> > > >>
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346746
> > > >>>
> > > >>> We would like to thank all contributors of the Apache Flink
> community
> > > >>> who made this release possible!
> > > >>>
> > > >>> Regards,
> > > >>> Chesnay
> > >
> > >
> >
>

Re: [DISCUSS] Kicking off the 1.11 release cycle

[Hequn Cheng]

Thanks a lot for kicking off the 1.11 release and thanks Piotr and Zhijiang
for volunteering as the release managers!
Also +1 for the *anticipated feature freeze date* around end of April. A
shorter release cycle would be nice.

Best,
Hequn

On Thu, Feb 20, 2020 at 6:08 PM lining jing  wrote:

> Thanks for kicking off.
>
> +1 for the *anticipated feature freeze date* end of April.
>
> Zhu Zhu  于2020年2月20日周四 下午6:04写道：
>
> > Thanks Piotr and Zhijiang for volunteering as the release managers!
> >
> > +1 for aiming with the feature freeze date for end of April.
> >
> > Thanks,
> > Zhu Zhu
> >
> > Piotr Nowojski  于2020年2月20日周四 下午4:26写道：
> >
> > > Hi,
> > >
> > > Thank you for giving me and Zhijiang opportunity and entrusting us with
> > > the release :)
> > >
> > > +1 for aiming with the feature freeze for and of April. I think all
> > things
> > > consider that’s a good timing.
> > >
> > > Piotrek
> > >
> > > > On 20 Feb 2020, at 04:22, Yu Li  wrote:
> > > >
> > > > Thanks Stephan for the kicking off! And thanks Piotr and Zhijiang for
> > > > volunteering as the release managers!
> > > >
> > > > Aiming for a shorter release cycle than 1.10 makes a lot of sense,
> and
> > +1
> > > > to target the feature freeze date around end of April.
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > >
> > > > On Thu, 20 Feb 2020 at 10:37, Xintong Song 
> > > wrote:
> > > >
> > > >> Thanks for the kicking off.
> > > >>
> > > >> +1 for Zhijiang & Piotr (if he agrees to) being the release
> managers.
> > > >> Thanks for the volunteering.
> > > >> +1 for the *anticipated feature freeze date* end of April
> > > >>
> > > >> Thank you~
> > > >>
> > > >> Xintong Song
> > > >>
> > > >>
> > > >>
> > > >> On Thu, Feb 20, 2020 at 10:30 AM Yuan Mei 
> > > wrote:
> > > >>
> > > >>> +1
> > > >>>
> > > >>> 3-month sounds a very reasonable duration for a release.
> > > >>>
> > > >>> On Thu, Feb 20, 2020 at 9:53 AM tison 
> wrote:
> > > >>>
> > >  Thanks for kicking off the discussion Stephan!
> > > 
> > >  +1 for the *anticipated feature freeze date* end of April and
> > >  Zhijiang & Piotr to be the release managers.
> > > 
> > >  Best,
> > >  tison.
> > > 
> > > 
> > >  Dian Fu  于2020年2月20日周四 上午9:41写道：
> > > 
> > > > Thanks Stephan kicking off this discussion and Zhijiang
> > volunteering
> > > >> as
> > > > one of the release managers.
> > > >
> > > > +1 for the "feature freeze date" around end of April. There are
> > still
> > >  more
> > > > than 2 months left, so I think it's a reasonable time.
> > > >
> > > > Thanks,
> > > > Dian
> > > >
> > > >> 在 2020年2月19日，下午10:52，Aljoscha Krettek  写道：
> > > >>
> > > >> +1
> > > >>
> > > >> Although I would hope that it can be more than just
> "anticipated".
> > > >>
> > > >> Best,
> > > >> Aljoscha
> > > >>
> > > >> On 19.02.20 15:40, Till Rohrmann wrote:
> > > >>> Thanks for volunteering as one of our release managers
> Zhijiang.
> > > >>> +1 for the *anticipated feature freeze date* end of April. As
> we
> > > >> go
> > > > along
> > > >>> and collect more data points we might be able to strengthen our
> > > >>> initial anticipation.
> > > >>> Cheers,
> > > >>> Till
> > > >>> On Wed, Feb 19, 2020 at 4:44 AM Zhijiang <
> > > >>> wangzhijiang...@aliyun.com
> > > > .invalid>
> > > >>> wrote:
> > >  Thanks for kicking off the next release and the introduction,
> > >  @Stephan!
> > > 
> > >  It's my pleasure to become the release manager and involve in
> > > >> other
> > >  community works. I am working together with @Piotr for a long
> > > >> time,
> > >  so
> > >  very happy to cooperate for the release manager work again.
> The
> > > > previous
> > >  release work was always well done, and I can learn a lot from
> > > >> these
> > > > rich
> > >  experiences.
> > > 
> > >  +1 for the "feature freeze date" around end of April.
> > >  Although we have the FF SF in the meantime, fortunately there
> > > >> are
> > > >>> no
> > > > long
> > >  public holidays during this period in China.
> > > 
> > >  Best,
> > >  Zhijiang
> > > 
> > > 
> > > 
> > > >> --
> > >  From:Stephan Ewen 
> > >  Send Time:2020 Feb. 19 (Wed.) 01:15
> > >  To:dev 
> > >  Cc:zhijiang ; pnowojski <
> > > >> pnowoj...@apache.org
> > > 
> > >  Subject:[DISCUSS] Kicking off the 1.11 release cycle
> > > 
> > >  Hi all!
> > > 
> > >  Now that the 1.10 release is out (congrats again, everyone!),
> I
> > >  wanted
> > > > to
> > >  bring up some questions about organizing the next release
> cycle.
> > > 
> > >  The most important questions at the beg

Re: [VOTE] Support Python ML Pipeline API

[Hequn Cheng]

Thanks all for the votes.

So far, we have
   - 4 binding +1 votes (Jincheng, Dian, Zhijiang and Rong)
   - 1 non-binding +1 votes (Yuan)
   - 0 -1 votes

I will move the FLIP to accepted!

Best, Hequn


On Sat, Feb 15, 2020 at 12:45 AM Rong Rong  wrote:

> +1 (binding)
>
> On Fri, Feb 14, 2020 at 7:15 AM Zhijiang  .invalid>
> wrote:
>
> > +1 (binding), it is valuable to enhance the python API for supporting ML
> > well.
> >
> > Best,
> > Zhijiang
> >
> >
> > --
> > From:Dian Fu 
> > Send Time:2020 Feb. 14 (Fri.) 15:07
> > To:dev 
> > Subject:Re: [VOTE] Support Python ML Pipeline API
> >
> > +1 (binding)
> >
> > Regards,
> > Dian
> >
> > > 在 2020年2月14日，下午2:49，Yuan Mei  写道：
> > >
> > > +1 vote
> > >
> > > This is one of the most important things for Flink ML framework to be
> > > widely adopted since most data scientists use python.
> > >
> > > Best
> > >
> > > Yuan
> > >
> > > On Fri, Feb 14, 2020 at 9:45 AM Hequn Cheng  wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> I'd like to start the vote of FLIP-96[1] which is discussed and
> reached
> > >> consensus in the discussion thread[2].
> > >> The vote will be open for at least 72 hours. Unless there is an
> > objection,
> > >> I will try to close it by Feb 19, 2020 02:00 UTC if we have received
> > >> sufficient votes.
> > >>
> > >> Thanks,
> > >> Hequn
> > >>
> > >> [1]
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-96%3A+Support+Python+ML+Pipeline+API
> > >> [2]
> > >>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-Python-ML-Pipeline-API-td37291.html
> > >>
> >
> >
>

[jira] [Created] (FLINK-16187) Support Python ML Pipeline API

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-16187:
---

 Summary: Support Python ML Pipeline API
 Key: FLINK-16187
 URL: https://issues.apache.org/jira/browse/FLINK-16187
 Project: Flink
  Issue Type: New Feature
  Components: API / Python
Reporter: Hequn Cheng
Assignee: Hequn Cheng


This is the umbrella Jira for FLIP96, which intends to support Python ML 
Pipeline API.

FLIP wiki page: 
[https://cwiki.apache.org/confluence/display/FLINK/FLIP-96%3A+Support+Python+ML+Pipeline+API]

FLIP-39 rebuilds the Flink ML pipeline on top of TableAPI and introduces a new 
set of Java APIs. As Python is widely used in ML areas, providing Python ML 
Pipeline APIs for Flink can not only make it easier to write ML jobs for Python 
users but also broaden the adoption of Flink ML. In this FLIP we propose to:
 * Add Python pipeline API according to Java pipeline API(we will adapt the 
Python pipeline API if Java pipeline API changes).
 * Support native Python Transformer/Estimator/Model, i.e., users can write not 
only Python Transformer/Estimator/Model wrappers for calling Java ones but also 
can write native Python Transformer/Estimator/Models.
 * Ease of use. Support keyword arguments when defining parameters.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16188) Make AutoClosableProcess constructor private

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-16188:


 Summary: Make AutoClosableProcess constructor private
 Key: FLINK-16188
 URL: https://issues.apache.org/jira/browse/FLINK-16188
 Project: Flink
  Issue Type: Improvement
  Components: Test Infrastructure
Affects Versions: 1.11.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.11.0


To exert more control on the creation of an {{AutoClosableProcess}}, make the 
constructor private and adjust existing usages to rely on the factory methods 
instead.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Apache Flink-shaded 10.0 released

[Xingbo Huang]

Thanks a lot Chesnay and all contributes to this release.

Best,
Xingbo

Hequn Cheng  于2020年2月20日周四 下午8:17写道：

> Thanks a lot for the release Chesnay!
> Also thanks to everyone who contributes to this release!
>
> Best, Hequn
>
> On Thu, Feb 20, 2020 at 11:11 AM Yu Li  wrote:
>
> > Thanks Chesnay and all participants for making the release possible!
> >
> > Best Regards,
> > Yu
> >
> >
> > On Thu, 20 Feb 2020 at 09:50, Zhu Zhu  wrote:
> >
> > > Thanks Chesnay for the great work and everyone who helps with the
> > > improvements and release!
> > >
> > > Thanks,
> > > Zhu Zhu
> > >
> > > Dian Fu  于2020年2月20日周四 上午9:44写道：
> > >
> > > > Thanks Chesnay for the great work and everyone involved!
> > > >
> > > > Regards,
> > > > Dian
> > > >
> > > > > 在 2020年2月20日，上午12:21，Zhijiang 
> > 写道：
> > > > >
> > > > > Thanks Chesnay for making the release efficiently and also thanks
> to
> > > all
> > > > the other participants!
> > > > >
> > > > > Best,
> > > > > Zhijiang
> > > > >
> > > > >
> > > > > --
> > > > > From:Till Rohrmann  > trohrm...@apache.org
> > > >>
> > > > > Send Time:2020 Feb. 19 (Wed.) 22:21
> > > > > To:dev mailto:dev@flink.apache.org>>
> > > > > Subject:Re: [ANNOUNCE] Apache Flink-shaded 10.0 released
> > > > >
> > > > > Thanks for making the release possible Chesnay and everyone who was
> > > > > involved!
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Wed, Feb 19, 2020 at 7:47 AM jincheng sun <
> > sunjincheng...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> Thanks a lot for the release Chesnay!
> > > > >> And thanks to everyone who make this release possible!
> > > > >>
> > > > >> Best,
> > > > >> Jincheng
> > > > >>
> > > > >>
> > > > >> Chesnay Schepler  于2020年2月19日周三 上午12:45写道：
> > > > >>
> > > > >>> The Apache Flink community is very happy to announce the release
> of
> > > > >>> Apache Flink-shaded 10.0.
> > > > >>>
> > > > >>> The flink-shaded project contains a number of shaded dependencies
> > for
> > > > >>> Apache Flink.
> > > > >>>
> > > > >>> Apache Flink(r) is an open-source stream processing framework for
> > > > >>> distributed, high-performing, always-available, and accurate data
> > > > >>> streaming applications.
> > > > >>>
> > > > >>> The release is available for download at:
> > > > >>> https://flink.apache.org/downloads.html
> > > > >>>
> > > > >>> The full release notes are available in Jira:
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346746
> > > > >>>
> > > > >>> We would like to thank all contributors of the Apache Flink
> > community
> > > > >>> who made this release possible!
> > > > >>>
> > > > >>> Regards,
> > > > >>> Chesnay
> > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Apache Flink Python API(PyFlink) 1.9.2 released

[Xingbo Huang]

Thanks a lot for the release.
Great Work, Jincheng!
Also thanks to participants who contribute to this release.

Best,
Xingbo


Till Rohrmann  于2020年2月18日周二 下午11:40写道：

> Thanks for updating the 1.9.2 release wrt Flink's Python API Jincheng!
>
> Cheers,
> Till
>
> On Thu, Feb 13, 2020 at 12:25 PM Hequn Cheng  wrote:
>
>> Thanks a lot for the release, Jincheng!
>> Also thanks to everyone that make this release possible!
>>
>> Best,
>> Hequn
>>
>> On Thu, Feb 13, 2020 at 2:18 PM Dian Fu  wrote:
>>
>> > Thanks for the great work, Jincheng.
>> >
>> > Regards,
>> > Dian
>> >
>> > 在 2020年2月13日，下午1:32，jincheng sun  写道：
>> >
>> > Hi everyone,
>> >
>> > The Apache Flink community is very happy to announce the release of
>> Apache
>> > Flink Python API(PyFlink) 1.9.2, which is the first release to PyPI for
>> the
>> > Apache Flink Python API 1.9 series.
>> >
>> > Apache Flink® is an open-source stream processing framework for
>> > distributed, high-performing, always-available, and accurate data
>> streaming
>> > applications.
>> >
>> > The release is available for download at:
>> >
>> > https://pypi.org/project/apache-flink/1.9.2/#files
>> >
>> > Or installed using pip command:
>> >
>> > pip install apache-flink==1.9.2
>> >
>> > We would like to thank all contributors of the Apache Flink community
>> who
>> > helped to verify this release and made this release possible!
>> >
>> > Best,
>> > Jincheng
>> >
>> >
>> >
>>
>

Re: [VOTE] FLIP-98: Better Back Pressure Detection

[Congxian Qiu]

+1 for this improvement, It's very helpful to know whether a task is
backpressured or not.

Best,
Congxian


Yadong Xie  于2020年2月20日周四 下午5:34写道：

> Hi all
>
> I want to start the vote for FLIP-98, which proposes to provide better
> backpressure detection experience in web UI.
>
> To help everyone better understand the proposal, we spent some efforts on
> making an online POC
>
> previous web:
> http://101.132.122.69:8081/#/job/add9d3ea1006af15b9f872987eb3befd/overview
> POC web:
>
> http://101.132.122.69:8081/web/#/job/add9d3ea1006af15b9f872987eb3befd/overview
>
> The vote will last for at least 72 hours, following the consensus voting
> process.
>
> FLIP wiki:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection
>
> Discussion thread:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
>
> Thanks,
>
> Yadong
>

[jira] [Created] (FLINK-16189) Move reponsibility fo copying distribution into FlinkResource

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-16189:


 Summary: Move reponsibility fo copying distribution into 
FlinkResource
 Key: FLINK-16189
 URL: https://issues.apache.org/jira/browse/FLINK-16189
 Project: Flink
  Issue Type: Improvement
  Components: Test Infrastructure
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.11.0


Currently, the {{FlinkDistribution}} is responsible for locating and copying 
the distribution for usage in tests.
This makes the class a bit inflexible to use, as for example a 
{{FlinkResource}} could not create 2 distributions in separate places.

I suggest to move these responsibilities into the {{FlinkResource}} 
implementations.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16190) Migrate existing java e2e tests away from FlinkDistribution

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-16190:


 Summary: Migrate existing java e2e tests away from 
FlinkDistribution
 Key: FLINK-16190
 URL: https://issues.apache.org/jira/browse/FLINK-16190
 Project: Flink
  Issue Type: Improvement
  Components: Test Infrastructure, Tests
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.11.0


A few prototype e2e tests work directly against the {{FlinkDistribution}}. 
These should be migrated to work against a {{FlinkResource}} instead.

This also entails providing some setup instructions for the distribution, e.g., 
copying jars from opt to lib.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Extend (or maintain) "shell" script support for Windows

[David Anderson]

We stopped using these windows scripts in our training sessions a couple of
years ago, and I can't remember seeing anyone use them at all since then.
While our community includes many developers using Windows for development,
in my experience pretty much everyone moves to some sort of linux-based
environment once they are done testing in the IDE, be that via WSL, Docker,
a local VM, or a staging server.

David

On Thu, Feb 20, 2020 at 11:09 AM Stephan Ewen  wrote:

> I think this question should go on the user mailing list, not on dev.
>
> You can also use Twitter to collect some data points.
>
> On Thu, Feb 20, 2020 at 11:07 AM Chesnay Schepler 
> wrote:
>
> > Realistically you can't really use the windows scripts in production
> > since you can only start a standalone cluster consisting of 1 JM and 1
> TM.
> >
> > The .bat scripts are already limited to a significant degree.
> >
> > On 20/02/2020 03:48, Xintong Song wrote:
> > > I'm also in favor of dropping the bat scripts. I can hardly image
> > > people using Flink Windows deployment in their production. But maybe
> > > throwing a survey on the user ML to double check on that?
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Thu, Feb 20, 2020 at 10:28 AM Yang Wang 
> > wrote:
> > >
> > >> I tend to drop the windows bat script to start/stop the cluster. Since
> > >> there
> > >> are various windows versions(e.g. xp, win7, win10, etc.) and do not
> have
> > >> sensational compatibility. Also we do not have the e2e tests to
> > guarantee
> > >> the scripts work well.
> > >>
> > >> On the other hand, i completely agree that local deployment,
> especially
> > in
> > >> the IDE, should work on windows. For standalone deployment, i think
> > using
> > >> the docker is a good choice. I think many windows users are also using
> > >> docker. For Yarn/K8s deployment, i am not sure whether we need to
> > support
> > >> to submit a Flink session/per-job cluster from windows machine.
> > >>
> > >>
> > >> Best,
> > >> Yang
> > >>
> > >> Chesnay Schepler  于2020年2月20日周四 上午12:15写道：
> > >>
> > >>> Note that with FLINK-15925 we effectively broke the start-cluster.bat
> > >>> script in 1.10.
> > >>>
> > >>> Overall, I would be fine with dropping the .bat scripts because they
> > are
> > >>> a big maintenance problem for us.
> > >>> We don't have anyone using these scripts regularly on our end (even I
> > >>> don't).
> > >>> If there were a way to unify the unix/windows scripts I would be all
> > >>> ears, but this doesn't seem possible without lots and lots of
> branches.
> > >>>
> > >>> However, we should definitely ensure that Flink continues to work in
> > the
> > >>> IDE on windows for training/demo purposes.
> > >>>
> > >>> As for CI, with the new Azure setup this may actually be feasible now
> > >>> from a CI budged perspective. But it would require a fair upfront
> time
> > >>> investment to fix failing tests or adding assumptions for failing
> > tests.
> > >>> That said, this would have some nice benefits for finding resource
> > >>> leaks, particularly in the filesystem.
> > >>>
> > >>> On 19/02/2020 16:46, Aljoscha Krettek wrote:
> >  Hi,
> > 
> >  the background is this series of Jira Issues and PRs around
> extending
> >  the .bat scripts for windows:
> >  https://issues.apache.org/jira/browse/FLINK-5333.
> > 
> >  I would like to resolve this, by either closing the Jira Issues as
> >  "Won't Do" or finally merging these PRs. The questions I have are:
> > 
> >    - Should we add more full-featured (complicated?) windows scripts
> >  that are essentially re-implementations of our existing "unix"
> > scripts?
> >    - Would windows users use these or would they, by now, use the
> linux
> >  subsystem for windows or cygwin?
> >    - Should we even remove the existing .bat scripts that we have?
> > 
> >  Maintaining the windows scripts is hard because we only have one (I
> >  think, Chesnay) developer on windows and no CI for windows.
> > 
> >  What do you think?
> > 
> >  Best,
> >  Aljoscha
> > 
> > >>>
> >
> >
>

Re: [VOTE] FLIP-99: Make Max Exception Configurable

[Congxian Qiu]

+1 to this feature.  this is helpful when troubleshooting.

Best,
Congxian


Yadong Xie  于2020年2月20日周四 下午5:39写道：

> Hi all
>
> I want to start the vote for FLIP-99, which proposes to make the max
> exception configurable in web UI.
>
> To help everyone better understand the proposal, we spent some efforts on
> making an online POC
>
> previous web:
>
> http://101.132.122.69:8081/#/job/543e9dc0cb2cca4433116007f0931d1a/exceptions
> POC web:
>
> http://101.132.122.69:8081/web/#/job/543e9dc0cb2cca4433116007f0931d1a/exceptions
>
> The vote will last for at least 72 hours, following the consensus voting
> process.
>
> FLIP wiki:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-99%3A+Make+Max+Exception+Configurable
>
> Discussion thread:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
>
> Thanks,
>
> Yadong
>

[jira] [Created] (FLINK-16191) Improve error message on Windows when RocksDB Paths are too long

[Stephan Ewen (Jira)]

Stephan Ewen created FLINK-16191:


 Summary: Improve error message on Windows when RocksDB Paths are 
too long
 Key: FLINK-16191
 URL: https://issues.apache.org/jira/browse/FLINK-16191
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / State Backends
Affects Versions: 1.10.0
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.10.1, 1.11.0


Paths on Windows have a length limit by default (247 chars).
If RocksDB tries to open a longer path, it throws a misleading exception 
(directory not found).

We can check for this situation and improve the error message.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Drop connectors for Elasticsearch 2.x and 5.x

[Robert Metzger]

Thank you all.

I have proposed a PR to upgrade the ES5 connector:
https://github.com/apache/flink/pull/11162

On Wed, Feb 19, 2020 at 3:02 PM Till Rohrmann  wrote:

> +1 for removing ES2 and updating ES5 in 1.11.
>
> +1 for revisiting the removal of ES5 for 1.12.
>
> Cheers,
> Till
>
> On Tue, Feb 18, 2020 at 1:28 PM Chesnay Schepler 
> wrote:
>
> > Since one of the reasons for dropping ES2 was that it blocks some
> > critical updates for the ES5 connector I'd prefer to keep ES5 around for
> > 1.11, and revisit this discussion for 1.12 .
> >
> > On 18/02/2020 13:03, Aljoscha Krettek wrote:
> > > Wouldn't removing the ES 2.x connector be enough because we can then
> > > update the ES 5.x connector? It seems there are some users that still
> > > want to use that one.
> > >
> > > Best,
> > > Aljoscha
> > >
> > > On 18.02.20 10:42, Robert Metzger wrote:
> > >> The ES5 connector is causing some problems on the CI system. It would
> be
> > >> nice if we could make a decision here soon. I don't want to invest
> time
> > >> into fixing it, if we are going to remove it.
> > >>
> > >> I'm still in favor of removing it. If we see that there's demand for
> the
> > >> 5.x connector after the 1.11 release, somebody can take the source and
> > >> contribute it to Apache Bahir or a GitHub account and then posts it to
> > >> flink-packages.org.
> > >>
> > >>
> > >> On Thu, Feb 13, 2020 at 3:34 PM Dawid Wysakowicz
> > >> 
> > >> wrote:
> > >>
> > >>> Sorry for late reply,
> > >>>
> > >>> @all I think there is a general consensus that we want to drop ES 2.x
> > >>> support. I created https://issues.apache.org/jira/browse/FLINK-16046
> > to
> > >>> track it.
> > >>>
> > >>>
> > >>> @Stephan @Chesnay @Itamar In our connectors we use Java High Level
> Rest
> > >>> Client. ES promises to maintain compatibility of it with any newer
> > >>> minor
> > >>> version of ES. So if we have 6.1 client we can use it with any 6.2,
> 6.3
> > >>> etc.
> > >>>
> > >>> ES provides also a low level rest client which does not include any
> > >>> direct es dependencies and can work with any version of ES. It does
> not
> > >>> provide any marshalling unmarshalling or higher level features as
> > >>> Chesnay said.
> > >>>
> > >>> Correct me if I am wrong @Itamar but your HTTP client is a simplified
> > >>> version of the ES's high level rest client with a subset of its
> > >>> features. I think it will still have the same problems as ES's High
> > >>> Level Rest Client's because ES does not guarantee that newer message
> > >>> formats will be compatible with older versions of ES or that message
> > >>> formats are compatible across major versions at all.
> > >>>
> > >>>
> > >>> @Stephan @Danny As for the 5.x connector. Any ideas how can we get
> > >>> user's feedback about it? I cross posted on the user mailing list
> with
> > >>> no luck so far. Personally I would be in favor of dropping the
> > >>> connector. Worst case scenario users still have the possibility of
> > >>> building the connector themselves from source with just bumping the
> > >>> flink's versions. As far as I can tell there were no changes to the
> > >>> code
> > >>> base for quite some time.
> > >>>
> > >>> Best,
> > >>>
> > >>> Dawid
> > >>>
> > >>> On 11/02/2020 10:46, Chesnay Schepler wrote:
> >  I suppose the downside in an HTTP ES sink is that you don't get
> _any_
> >  form of high-level API from ES, and we'd have to manually build an
> >  HTTP request that matches the ES format. Of course you also lose any
> >  client-side verification that the clients did, if there is any (but
> I
> >  guess the API itself prevented certain errors).
> > 
> >  On 11/02/2020 09:32, Stephan Ewen wrote:
> > > +1 to drop ES 2.x - unsure about 5.x (makes sense to get more user
> > > input
> > > for that one).
> > >
> > > @Itamar - if you would be interested in contributing a "universal"
> or
> > > "cross version" ES connector, that could be very interesting. Do
> you
> > > know
> > > if there are known performance issues or feature restrictions with
> > > that
> > > approach?
> > > @dawid what do you think about that?
> > >
> > >
> > > On Tue, Feb 11, 2020 at 6:28 AM Danny Chan 
> > >>> wrote:
> > >
> > >> 5.x seems to have a lot of users, is the 6.x completely
> > >> compatible with
> > >> 5.x ~
> > >>
> > >> Best,
> > >> Danny Chan
> > >> 在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz
> > >> ，写道：
> > >>> Hi all,
> > >>>
> > >>> As described in this
> > >>> https://issues.apache.org/jira/browse/FLINK-11720
> > >>> ticket our elasticsearch 5.x connector does not work out of the
> > >>> box on
> > >>> some systems and requires a version bump. This also happens for
> our
> > >>> e2e.
> > >>> We cannot bump the version in es 5.x connector, because 5.x
> > >>> connector
> > >>> shares a common class with 2.x that uses an API that was replaced

Re: [VOTE] FLIP-98: Better Back Pressure Detection

[Zhijiang]

+1 (binding).
It seems more clearly and directly to highlight the back pressured vertex in 
topology, which can raise the attention of users.

Best,
Zhijiang


--
From:Congxian Qiu 
Send Time:2020 Feb. 20 (Thu.) 21:49
To:dev@flink.apache.org 
Subject:Re: [VOTE] FLIP-98: Better Back Pressure Detection

+1 for this improvement, It's very helpful to know whether a task is
backpressured or not.

Best,
Congxian


Yadong Xie  于2020年2月20日周四 下午5:34写道：

> Hi all
>
> I want to start the vote for FLIP-98, which proposes to provide better
> backpressure detection experience in web UI.
>
> To help everyone better understand the proposal, we spent some efforts on
> making an online POC
>
> previous web:
> http://101.132.122.69:8081/#/job/add9d3ea1006af15b9f872987eb3befd/overview
> POC web:
>
> http://101.132.122.69:8081/web/#/job/add9d3ea1006af15b9f872987eb3befd/overview
>
> The vote will last for at least 72 hours, following the consensus voting
> process.
>
> FLIP wiki:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection
>
> Discussion thread:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
>
> Thanks,
>
> Yadong
>

Re: [DISCUSS] Improvements on FLIP Process

[David Anderson]

I'd like to bring up another problem that I see with the current FLIP
process, which relates to what happens after a FLIP has been implemented.

A few months ago I read through all of the then existing FLIPs. My reason
for doing so is that on a great many topics, the only "documentation" we
have is the FLIPs, and there were several parts of Flink that I wanted to
understand better.

But I put documentation in "quotes" above because there are significant
problems with trying to rely on the FLIPs to know what's going on:

(1) As has already been pointed out, in quite a few cases the status was
never updated.
(2) In some cases, what was ultimately implemented does not match the FLIP.
Often these discrepancies are pretty minor: for example, during the
implementation classes get renamed from what had been proposed, interfaces
get adjusted a bit, etc. And sometimes a FLIP is only partially
implemented.
(3) And I believe there are cases where past FLIPs have been made at least
partially obsolete by more recent development.

As a step toward making the FLIPs more useful, I would like to propose that
when the author(s) go back to update the status from Accepted to Released,
if they would add a section at the end outlining any ways in which the
implementation being released differs from what is described in the FLIP.

David


On Thu, Feb 20, 2020 at 1:05 PM Hequn Cheng  wrote:

> Hi everyone,
>
> I appreciate all the feedback! Let me try to explain more.
>
> As there are a lot of people show concerns about the google doc.
> Let’s make an assumption first that the comment on the google doc is not
> allowed!
> Let’s make an assumption first that the comment on the google doc is not
> allowed!!
> Let’s make an assumption first that the comment on the google doc is not
> allowed!!!
> (As for whether to allow (minor)comments on the google doc. I will raise
> another discussion if we reach a consensus on this one.)
>
> The main purpose of this discussion is to improve the current FLIP process.
> Before we introduce the improvements, let’s first take a look at the
> problems we have. This is the motivation why this discussion is raised. The
> problems I find are listed as follows:
> 1. During the FLIP discussion, whenever there is a change, we need to pick
> the change to the wiki page. This is somehow redundant.
> 2. Even after the proposal has been finalized, the wiki page can be changed
> as it is editable to everyone. However, any change to an adopted FLIP
> requires a new voting process.
> 3. The FLIP page is not well maintained, e.g., the status of many FLIPs
> were not updated in time. You can find there are a lot of FLIPs with the
> target release of 1.10 on the current wiki page.
>
> Next, let’s see how the problems listed above can be solved by the new
> process. I will list the new process again for discussion convenience.
>
> 1. Raise the discussion on the mailing list. The subject of the thread is
> of the format [DISCUSS][FLIP] {your FLIP heading}. Also, the design doc
> should follow the FLIP-Template strictly. (The [FLIP] tag is used to inform
> people that it is a FLIP discussion and more attention should be paid.)
> 2. Create a FLIP wiki page(by a committer who wants to shepherd the FLIP)
> once we reach an agreement on the discussion. We can simply copy the google
> doc into the FLIP wiki page. The name of the shepherd should also be listed
> on the wiki page.
> 3. Once the proposal is finalized, call a vote to have the proposal
> adopted. It should be noted that we should always vote on a FLIP wiki page
> instead of a google doc. The wiki page is the final version of the google
> doc.
>
> According to the new process:
> Problem1 can be improved since we only create the FLIP wiki page when we
> reached an agreement.
> Problem2 can be improved since we can make the FLIP wiki page committer
> editable. Not everyone can change the wiki page freely as it is now.
> Problem3 can be improved as the FLIP wiki page is created and updated by
> the committer. Committers are assumed to be more active than contributors.
> This can also be easier to info the guy who is responsible for updating the
> wiki page.
>
> Looking forward to your feedback!
>
> Best,
> Hequn
>
> On Wed, Feb 19, 2020 at 10:20 PM Till Rohrmann 
> wrote:
>
> > +1 for Aljoscha's proposal.
> >
> > This, of course, does not mean that one cannot use Google docs in order
> to
> > prepare the FLIP discussion.
> >
> > Cheers,
> > Till
> >
> > On Wed, Feb 19, 2020 at 11:15 AM Xintong Song 
> > wrote:
> >
> > > I'm also not a fun of discussing FLIPs with google docs.
> > >
> > > I think google docs is probably ok for smaller scope early discussions
> > > before raising the discussion on the mailing list, when the draft is
> not
> > > completed and is expected to change frequently. Once it is proposed to
> > the
> > > community, as many people already mentioned, google doc changes are
> very
> > > hard to track.
> > >
> > > If I understand correctly, what J

[jira] [Created] (FLINK-16192) Remove "Legacy State" and Checkpoint Compatibility with Flink 1.2

[Stephan Ewen (Jira)]

Stephan Ewen created FLINK-16192:


 Summary: Remove "Legacy State" and Checkpoint Compatibility with 
Flink 1.2
 Key: FLINK-16192
 URL: https://issues.apache.org/jira/browse/FLINK-16192
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Checkpointing
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.11.0


Legacy state is state produced by the {{Checkpointed}} interface, back in the 
days before Flink applications were re-scalable. The interface was deprecated 
in Flink 1.2, support dropped in Flink 1.4.

Removing legacy state leads to a lot of nice cleanups and simplifications in 
the code, especially in the checkpoint metadata handling. This was in fact the 
motivation, extending checkpoints to include "operator coordinators".

Removing support for this is in theory independent of removing support for 
Flink 1.2, but in practice all the testing savepoints for 1.2 compatibility use 
still this state. Dropping "legacy state" means either some very involved 
reworking of all the 1.2 compatibility tests and re-building the 1.2 snapshots, 
or dropping compatibility with Flink 1.2.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16193) Improve error messaging when a key is assigned to the wrong key group range

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-16193:


 Summary: Improve error messaging when a key is assigned to the 
wrong key group range
 Key: FLINK-16193
 URL: https://issues.apache.org/jira/browse/FLINK-16193
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream
Affects Versions: 1.11.0
Reporter: Seth Wiesman


Occasionally, users may run into an exception that reads something like:

 

{{java.lang.IllegalArgumentException: Key group 45 is not in 
KeyGroupRange\{startKeyGroup=0, endKeyGroup=42}}}

{{}}

This may be caused by a number of issues including:

1) Unstable hash and equals methods on their key objects

2) Improper use of DataStreamUtils#reinterpretAsKeyedStream

{{}}

{{Key group ranges are a fairly low level detail that most users will be 
unfamiliar with when working with Flink. We should offer more comprehensive 
error messaging that outlines likely causes and solutions.}}

{{}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] AssignerWithPeriodicWatermarks with max delay

[Eduardo Winpenny Tejedor]

Hi Till,

Thanks for setting me on the right path! I'll finish reading the FLIP and
Jira item and I'll contact Jiangjie Qin on how to best help.

Regards,
Eduardo


On Wed, 19 Feb 2020, 14:44 Till Rohrmann,  wrote:

> Hi Eduardo,
>
> great to hear that you want to contribute to Flink.
>
> I'm not too deeply involved in idle watermarks but I know that there is
> FLIP-27 [1] which has the aim of reworking Flink's source interface. Part
> of it is to solve the problem of idle watermarks. The idea would be to
> solve the problem for all sources instead of having to implement it for
> every watermark assigner. Please also take a look at the corresponding JIRA
> issue [2].
>
> Maybe you can check with the involved people how far we are with the
> implementation and whether it will cover the idle watermarks.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> [2] https://issues.apache.org/jira/browse/FLINK-10740
>
> Cheers,
> Till
>
> On Mon, Feb 17, 2020 at 10:22 PM Eduardo Winpenny Tejedor <
> eduardo.winpe...@gmail.com> wrote:
>
> > Hi all,
> >
> > I've been using Apache Flink for the last few months but I'm new to
> > the dev community. I'd like to contribute code (and possibly more) to
> > the community and I've been advised a good starting point would be
> > suggesting improvements for those areas that I found lacking. I'll
> > create a separate [DISCUSS] thread for each of those (if this is
> > indeed the process!).
> >
> > -- Problem statement --
> >
> > In my use cases I've had to output data at regular (event time)
> > intervals, regardless of whether there's been any events flowing
> > through the app. For those occasions when no events flow I've been
> > happy to delay the emission of data for some time. This amount of time
> > is reasonable and still several times larger than the worse delays of
> > my event bus. It also meets the business requirements :)
> >
> > Flink's documentation suggests marking a source as temporarily idle
> > for such occasions but to the best of my knowledge it will not advance
> > the watermark if there's no events at all flowing through the system.
> >
> >
> > -- Proposed solution --
> >
> > Provide all implementations of AssignerWithPeriodicWatermarks in the
> > Flink project a mechanism to specify a max time delay after which the
> > watermark will advance if no events have been processed. The watermark
> > will always stay as far as the specified delay when advanced in this
> > way.
> >
> > To achieve backward compatibility I suggest providing the
> > implementations of AssignerWithPeriodicWatermarks with a builder
> > method that'll allow to specify said max delay. Other options to
> > introducing this change in a non-invasive way are welcome.
> >
> > I'm hoping for your suggestions/comments/questions.
> >
> > Thanks,
> > Eduardo
> >
>

[jira] [Created] (FLINK-16194) Refactor the Kubernetes resouces construction architecture

[Canbin Zheng (Jira)]

Canbin Zheng created FLINK-16194:


 Summary: Refactor the Kubernetes resouces construction architecture
 Key: FLINK-16194
 URL: https://issues.apache.org/jira/browse/FLINK-16194
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Kubernetes
Affects Versions: 1.10.0
Reporter: Canbin Zheng
 Fix For: 1.11.0


So far, Flink has made efforts for the native integration of Kubernetes. 
However, it is always essential to evaluate the existing design and consider 
alternatives that have better design and are easier to maintain in the long 
run. We have suffered from some problems while developing new features base on 
the current code. Here is some of them:
 # We don’t have a unified monadic-step based orchestrator architecture to 
construct all the Kubernetes resources.
 ** There are inconsistencies between the orchestrator architecture that client 
uses to create the Kubernetes resources, and the orchestrator architecture that 
the master uses to create Pods; this confuses new contributors, as there is a 
cognitive burden to understand two architectural philosophies instead of one; 
for another, maintenance and new feature development become quite challenging.
 ** Pod construction is done in one step. With the introduction of new features 
for the Pod, the construction process could become far more complicated, and 
the functionality of a single class could explode, which hurts code 
readability, writability, and testability. At the moment, we have encountered 
such challenges and realized that it is not an easy thing to develop new 
features related to the Pod.
 ** The implementations of a specific feature are usually scattered in multiple 
decoration classes. For example, the current design uses a decoration class 
chain that contains five Decorator class to mount a configuration file to the 
Pod. If people would like to introduce other configuration files support, such 
as Hadoop configuration or Keytab files, they have no choice but to repeat the 
same tedious and scattered process.
 # We don’t have dedicated objects or tools for centrally parsing, verifying, 
and managing the Kubernetes parameters, which has raised some maintenance and 
inconsistency issues.
 ** There are many duplicated parsing and validating code, including settings 
of Image, ImagePullPolicy, ClusterID, ConfDir, Labels, etc. It not only harms 
readability and testability but also is prone to mistakes. Refer to issue 
FLINK-16025 for inconsistent parsing of the same parameter.
 ** The parameters are scattered so that some of the method signatures have to 
declare many unnecessary input parameters, such as 
FlinkMasterDeploymentDecorator#createJobManagerContainer.

 

For solving these issues, we propose to 
 # Introduce a unified monadic-step based orchestrator architecture that has a 
better, cleaner and consistent abstraction for the Kubernetes resources 
construction process. 
 # Add some dedicated tools for centrally parsing, verifying, and managing the 
Kubernetes parameters.

 

Refer to the design doc for the details, any feedback is welcome.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] FLINK-16194: Refactor the Kubernetes architecture design

[felixzheng zheng]

Hi everyone,

I would like to kick off a discussion on refactoring the existing
Kubernetes resources construction architecture design.

I created a design document [1] that clarifies our motivation to do this
and some improvement proposals for the new design.

Briefly, we would like to
1. Introduce a unified monadic-step based orchestrator architecture that
has a better, cleaner and consistent abstraction for the Kubernetes
resources construction process, both applicable to the client side and the
master side.
2. Add some dedicated tools for centrally parsing, verifying, and managing
the Kubernetes parameters.

It would be great to start the efforts before adding big features for the
native Kubernetes submodule, and Tison and I plan to work together for this
issue.

Please let me know your thoughts.

Regards,
Canbin Zheng

[1]
https://docs.google.com/document/d/1dFBjqho8IRyNWxKVhFnTf0qGCsgp72aKON4wUdHY5Pg/edit?usp=sharing
[2] https://issues.apache.org/jira/browse/FLINK-16194

Re: [ANNOUNCE] Apache Flink Python API(PyFlink) 1.9.2 released

[Xingbo Huang]

Hi Chad,
Beam Portability module solves the management of the Python runtime
environment and the communication between the Python runtime environment
Python VM and the Java runtime environment JVM in PyFlink. For details, you
can refer to FLIP-58[1] and the Blog[2].

Hope to resolve your doubts.

Best,
Xingbo

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
[2]
https://enjoyment.cool/2020/02/19/Deep-dive-how-to-support-Python-UDF-in-Apache-Flink-1-10/

Chad Dombrova  于2020年2月21日周五 上午12:16写道：

> Hi,
> Very cool. I’m curious about the relationship between this feature and
> Apache Beam.  What parts of Beam are used and for what?  Does this have any
> impact on existing Beam users like myself who use the Beam python API on
> top of Flink?  Can someone give me a brief overview or point me at the
> right documentation?
>
> Thanks!
> -chad
>
>
> On Thu, Feb 20, 2020 at 5:39 AM Xingbo Huang  wrote:
>
>> Thanks a lot for the release.
>> Great Work, Jincheng!
>> Also thanks to participants who contribute to this release.
>>
>> Best,
>> Xingbo
>>
>>
>> Till Rohrmann  于2020年2月18日周二 下午11:40写道：
>>
>>> Thanks for updating the 1.9.2 release wrt Flink's Python API Jincheng!
>>>
>>> Cheers,
>>> Till
>>>
>>> On Thu, Feb 13, 2020 at 12:25 PM Hequn Cheng  wrote:
>>>
 Thanks a lot for the release, Jincheng!
 Also thanks to everyone that make this release possible!

 Best,
 Hequn

 On Thu, Feb 13, 2020 at 2:18 PM Dian Fu  wrote:

 > Thanks for the great work, Jincheng.
 >
 > Regards,
 > Dian
 >
 > 在 2020年2月13日，下午1:32，jincheng sun  写道：
 >
 > Hi everyone,
 >
 > The Apache Flink community is very happy to announce the release of
 Apache
 > Flink Python API(PyFlink) 1.9.2, which is the first release to PyPI
 for the
 > Apache Flink Python API 1.9 series.
 >
 > Apache Flink® is an open-source stream processing framework for
 > distributed, high-performing, always-available, and accurate data
 streaming
 > applications.
 >
 > The release is available for download at:
 >
 > https://pypi.org/project/apache-flink/1.9.2/#files
 >
 > Or installed using pip command:
 >
 > pip install apache-flink==1.9.2
 >
 > We would like to thank all contributors of the Apache Flink community
 who
 > helped to verify this release and made this release possible!
 >
 > Best,
 > Jincheng
 >
 >
 >

>>>

Re: [DISCUSS] Improvements on FLIP Process

[Xintong Song]

+1 for David's proposal.

I think it happens a lot that we implement the FLIPs differently from the
original proposals, based on more understandings during the implementation.
Some times the differences are even not minors, which of course needs
consensus and maybe a vote on the mailing list. Moreover, some of the FLIP
wiki pages are referenced in user docs, release notes and blogposts. I
think it's important to keep the FLIPs updated.

We actually did that for FLIP-49 and FLIP-53 that released in 1.10,
updating the parts which are implemented differently according to ML
discussions. But we made the changed directly on the outdated contents. I
think, as David proposed, having a separate section and for such changes
might be better. We can add it to the FLIP template, to be filled in after
the FLIP is released.

@Hequn
What I don't get is that, if we do not allow comments on google docs, how
is it different from using wiki pages from the very beginning when
discussion is raised? Please correct me if I was wrong, it seems the
differences exist only if we make the wiki pages editable to committers
only. However, I don't see the necessity for having a strict wiki access,
because people can get notifications on modifications to the wiki pages by
simply *watch* them, unlike google docs where the people with edit access
can change the doc silently.

+1 to put the name of person who drives and is responsible for the FLIP doc
on the wiki page. Not sure whether it has to be a committer. This should
help clearly define who is responsible for keep the FLIP status updated,
and inform other people about whom to ping if they find a FLIP is not
properly updated.

Thank you~

Xintong Song



On Fri, Feb 21, 2020 at 1:52 AM David Anderson  wrote:

> I'd like to bring up another problem that I see with the current FLIP
> process, which relates to what happens after a FLIP has been implemented.
>
> A few months ago I read through all of the then existing FLIPs. My reason
> for doing so is that on a great many topics, the only "documentation" we
> have is the FLIPs, and there were several parts of Flink that I wanted to
> understand better.
>
> But I put documentation in "quotes" above because there are significant
> problems with trying to rely on the FLIPs to know what's going on:
>
> (1) As has already been pointed out, in quite a few cases the status was
> never updated.
> (2) In some cases, what was ultimately implemented does not match the FLIP.
> Often these discrepancies are pretty minor: for example, during the
> implementation classes get renamed from what had been proposed, interfaces
> get adjusted a bit, etc. And sometimes a FLIP is only partially
> implemented.
> (3) And I believe there are cases where past FLIPs have been made at least
> partially obsolete by more recent development.
>
> As a step toward making the FLIPs more useful, I would like to propose that
> when the author(s) go back to update the status from Accepted to Released,
> if they would add a section at the end outlining any ways in which the
> implementation being released differs from what is described in the FLIP.
>
> David
>
>
> On Thu, Feb 20, 2020 at 1:05 PM Hequn Cheng  wrote:
>
> > Hi everyone,
> >
> > I appreciate all the feedback! Let me try to explain more.
> >
> > As there are a lot of people show concerns about the google doc.
> > Let’s make an assumption first that the comment on the google doc is not
> > allowed!
> > Let’s make an assumption first that the comment on the google doc is not
> > allowed!!
> > Let’s make an assumption first that the comment on the google doc is not
> > allowed!!!
> > (As for whether to allow (minor)comments on the google doc. I will raise
> > another discussion if we reach a consensus on this one.)
> >
> > The main purpose of this discussion is to improve the current FLIP
> process.
> > Before we introduce the improvements, let’s first take a look at the
> > problems we have. This is the motivation why this discussion is raised.
> The
> > problems I find are listed as follows:
> > 1. During the FLIP discussion, whenever there is a change, we need to
> pick
> > the change to the wiki page. This is somehow redundant.
> > 2. Even after the proposal has been finalized, the wiki page can be
> changed
> > as it is editable to everyone. However, any change to an adopted FLIP
> > requires a new voting process.
> > 3. The FLIP page is not well maintained, e.g., the status of many FLIPs
> > were not updated in time. You can find there are a lot of FLIPs with the
> > target release of 1.10 on the current wiki page.
> >
> > Next, let’s see how the problems listed above can be solved by the new
> > process. I will list the new process again for discussion convenience.
> >
> > 1. Raise the discussion on the mailing list. The subject of the thread is
> > of the format [DISCUSS][FLIP] {your FLIP heading}. Also, the design doc
> > should follow the FLIP-Template strictly. (The [FLIP] tag is used to
> inform
> > people

Re: [DISCUSS] Kicking off the 1.11 release cycle

[Yang Wang]

Thanks Stephan for kicking off the 1.11 release and Zhijiang and Piotr for
the volunteering.
A short release cycle is really good. Since we will not push many features
simultaneously in
every release.

+1 for feature freeze date for end of April.


Best,
Yang

Hequn Cheng  于2020年2月20日周四 下午8:27写道：

> Thanks a lot for kicking off the 1.11 release and thanks Piotr and Zhijiang
> for volunteering as the release managers!
> Also +1 for the *anticipated feature freeze date* around end of April. A
> shorter release cycle would be nice.
>
> Best,
> Hequn
>
> On Thu, Feb 20, 2020 at 6:08 PM lining jing  wrote:
>
> > Thanks for kicking off.
> >
> > +1 for the *anticipated feature freeze date* end of April.
> >
> > Zhu Zhu  于2020年2月20日周四 下午6:04写道：
> >
> > > Thanks Piotr and Zhijiang for volunteering as the release managers!
> > >
> > > +1 for aiming with the feature freeze date for end of April.
> > >
> > > Thanks,
> > > Zhu Zhu
> > >
> > > Piotr Nowojski  于2020年2月20日周四 下午4:26写道：
> > >
> > > > Hi,
> > > >
> > > > Thank you for giving me and Zhijiang opportunity and entrusting us
> with
> > > > the release :)
> > > >
> > > > +1 for aiming with the feature freeze for and of April. I think all
> > > things
> > > > consider that’s a good timing.
> > > >
> > > > Piotrek
> > > >
> > > > > On 20 Feb 2020, at 04:22, Yu Li  wrote:
> > > > >
> > > > > Thanks Stephan for the kicking off! And thanks Piotr and Zhijiang
> for
> > > > > volunteering as the release managers!
> > > > >
> > > > > Aiming for a shorter release cycle than 1.10 makes a lot of sense,
> > and
> > > +1
> > > > > to target the feature freeze date around end of April.
> > > > >
> > > > > Best Regards,
> > > > > Yu
> > > > >
> > > > >
> > > > > On Thu, 20 Feb 2020 at 10:37, Xintong Song 
> > > > wrote:
> > > > >
> > > > >> Thanks for the kicking off.
> > > > >>
> > > > >> +1 for Zhijiang & Piotr (if he agrees to) being the release
> > managers.
> > > > >> Thanks for the volunteering.
> > > > >> +1 for the *anticipated feature freeze date* end of April
> > > > >>
> > > > >> Thank you~
> > > > >>
> > > > >> Xintong Song
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Thu, Feb 20, 2020 at 10:30 AM Yuan Mei  >
> > > > wrote:
> > > > >>
> > > > >>> +1
> > > > >>>
> > > > >>> 3-month sounds a very reasonable duration for a release.
> > > > >>>
> > > > >>> On Thu, Feb 20, 2020 at 9:53 AM tison 
> > wrote:
> > > > >>>
> > > >  Thanks for kicking off the discussion Stephan!
> > > > 
> > > >  +1 for the *anticipated feature freeze date* end of April and
> > > >  Zhijiang & Piotr to be the release managers.
> > > > 
> > > >  Best,
> > > >  tison.
> > > > 
> > > > 
> > > >  Dian Fu  于2020年2月20日周四 上午9:41写道：
> > > > 
> > > > > Thanks Stephan kicking off this discussion and Zhijiang
> > > volunteering
> > > > >> as
> > > > > one of the release managers.
> > > > >
> > > > > +1 for the "feature freeze date" around end of April. There are
> > > still
> > > >  more
> > > > > than 2 months left, so I think it's a reasonable time.
> > > > >
> > > > > Thanks,
> > > > > Dian
> > > > >
> > > > >> 在 2020年2月19日，下午10:52，Aljoscha Krettek 
> 写道：
> > > > >>
> > > > >> +1
> > > > >>
> > > > >> Although I would hope that it can be more than just
> > "anticipated".
> > > > >>
> > > > >> Best,
> > > > >> Aljoscha
> > > > >>
> > > > >> On 19.02.20 15:40, Till Rohrmann wrote:
> > > > >>> Thanks for volunteering as one of our release managers
> > Zhijiang.
> > > > >>> +1 for the *anticipated feature freeze date* end of April. As
> > we
> > > > >> go
> > > > > along
> > > > >>> and collect more data points we might be able to strengthen
> our
> > > > >>> initial anticipation.
> > > > >>> Cheers,
> > > > >>> Till
> > > > >>> On Wed, Feb 19, 2020 at 4:44 AM Zhijiang <
> > > > >>> wangzhijiang...@aliyun.com
> > > > > .invalid>
> > > > >>> wrote:
> > > >  Thanks for kicking off the next release and the
> introduction,
> > > >  @Stephan!
> > > > 
> > > >  It's my pleasure to become the release manager and involve
> in
> > > > >> other
> > > >  community works. I am working together with @Piotr for a
> long
> > > > >> time,
> > > >  so
> > > >  very happy to cooperate for the release manager work again.
> > The
> > > > > previous
> > > >  release work was always well done, and I can learn a lot
> from
> > > > >> these
> > > > > rich
> > > >  experiences.
> > > > 
> > > >  +1 for the "feature freeze date" around end of April.
> > > >  Although we have the FF SF in the meantime, fortunately
> there
> > > > >> are
> > > > >>> no
> > > > > long
> > > >  public holidays during this period in China.
> > > > 
> > > >  Best,
> > > >  Zhijiang
> > > > 
> > > > 
> > > > 
> > > > >> ---

Re: [VOTE] FLIP-103: Better TM/JM Log Display

[Yang Wang]

+1 non-binding

The log files list are shown in the new design. I like the idea. Since the
user defined logs(e.g. my-log.file)
could be found here now.


Best,
Yang

Yadong Xie  于2020年2月20日周四 下午5:56写道：

> Hi all
>
> I want to start the vote for FLIP-103, which proposes to provide better
> user log display UI in task manager and job manager.
>
> To help everyone better understand the proposal, we spent some efforts on
> making an online POC
>
> Task manager:
>
> previous web:
>
> http://101.132.122.69:8081/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/logs
> POC web:
>
> http://101.132.122.69:8081/web/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/log
>
> Job manager:
>
> previous web: http://101.132.122.69:8081/#/job-manager/logs
> POC web: http://101.132.122.69:8081/web/#/job-manager/log
>
>
> The vote will last for at least 72 hours, following the consensus voting
> process.
>
> FLIP wiki:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143
>
> Discussion thread:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
>
> Thanks,
>
> Yadong
>

Re: [VOTE] FLIP-102: Add More Metrics to TaskManager

[Xintong Song]

Thanks for driving this FLIP, Yadong.

+1 (non-binding) for the FLIP in general. I think this really helps our
users to understand and use the new FLIP-49 memory configuration.

I have a few minor comments.
- There's a frame "Other" in the frame "Non-Heap", besides "JVM Overhead"
and "JVM Metaspace". IIUC, the purpose of this is to explain the
mismatching between the metric "non-heap maximum" and the sum of the
configurations "JVM metaspace" & "JVM Overhead". However, from the
perspective of FLIP-49, JVM Overhead accounts for all the JVM non-heap
memory usages except for metaspace. The metrics does not match the
configuration because we did not set the a JVM parameter for "max non-heap
memory" (actually I'm not sure whether it can be specified in java 8). The
current UI might confuse people making them think there are other non-heap
memory usages not accounted by the configurations. Therefore, I would
suggest to remove the "Other" frame, but add another frame inside "JVM
Overhead", besides "Configuration", with "JVM limit" as the title and
"non-heap max metric minus metaspace configuration" as the value .

- In the final release, we have changed "shuffle memory" to "network
memory" because the latter is easier to understand for users. I think we
should be updated it in this FLIP as well.

- There's a typo "Directed" (should be "Direct") at the direct memory
metric.

Thank you~

Xintong Song



On Thu, Feb 20, 2020 at 5:52 PM Yadong Xie  wrote:

> Hi all
>
> I want to start the vote for FLIP-102, which proposes to add more metrics
> to the task manager in web UI.
>
> To help everyone better understand the proposal, we spent some efforts on
> making an online POC
>
> previous web:
>
> http://101.132.122.69:8081/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/metrics
> POC web:
>
> http://101.132.122.69:8081/web/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/metrics
>
>
> The vote will last for at least 72 hours, following the consensus voting
> process.
>
> FLIP wiki:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager
>
> Discussion thread:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
>
> Thanks,
>
> Yadong
>

Re: [VOTE] FLIP-103: Better TM/JM Log Display

[Xintong Song]

+1 (non-binding)

I like the ideas of having a list of all log files, and make them
downloadable.

Thank you~

Xintong Song



On Fri, Feb 21, 2020 at 10:24 AM Yang Wang  wrote:

> +1 non-binding
>
> The log files list are shown in the new design. I like the idea. Since the
> user defined logs(e.g. my-log.file)
> could be found here now.
>
>
> Best,
> Yang
>
> Yadong Xie  于2020年2月20日周四 下午5:56写道：
>
> > Hi all
> >
> > I want to start the vote for FLIP-103, which proposes to provide better
> > user log display UI in task manager and job manager.
> >
> > To help everyone better understand the proposal, we spent some efforts on
> > making an online POC
> >
> > Task manager:
> >
> > previous web:
> >
> >
> http://101.132.122.69:8081/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/logs
> > POC web:
> >
> >
> http://101.132.122.69:8081/web/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/log
> >
> > Job manager:
> >
> > previous web: http://101.132.122.69:8081/#/job-manager/logs
> > POC web: http://101.132.122.69:8081/web/#/job-manager/log
> >
> >
> > The vote will last for at least 72 hours, following the consensus voting
> > process.
> >
> > FLIP wiki:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143
> >
> > Discussion thread:
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> >
> > Thanks,
> >
> > Yadong
> >
>

[ANNOUNCE] Jingsong Lee becomes a Flink committer

[Kurt Young]

Hi everyone,

I'm very happy to announce that Jingsong Lee accepted the offer of the
Flink PMC to
become a committer of the Flink project.

Jingsong Lee has been an active community member for more than a year now.
He is
mainly focus on Flink SQL, played an essential role during blink planner
merging, drives
FLIP-63 and helped implementing rework expression design, and also
implemented
and fixed lots of features and bugs in Flink SQL. Moreover, he is very
active in both dev
and user mailing lists, helped discussing designs and answering users
questions, also
helped to verify various releases.

Congratulations Jingsong!

Best, Kurt
(on behalf of the Flink PMC)

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Benchao Li]

Congrats Jingsong, well deserved!

Kurt Young  于2020年2月21日周五 上午11:05写道：

> Hi everyone,
>
> I'm very happy to announce that Jingsong Lee accepted the offer of the
> Flink PMC to
> become a committer of the Flink project.
>
> Jingsong Lee has been an active community member for more than a year now.
> He is
> mainly focus on Flink SQL, played an essential role during blink planner
> merging, drives
> FLIP-63 and helped implementing rework expression design, and also
> implemented
> and fixed lots of features and bugs in Flink SQL. Moreover, he is very
> active in both dev
> and user mailing lists, helped discussing designs and answering users
> questions, also
> helped to verify various releases.
>
> Congratulations Jingsong!
>
> Best, Kurt
> (on behalf of the Flink PMC)
>
>
>

-- 

Benchao Li
School of Electronics Engineering and Computer Science, Peking University
Tel:+86-15650713730
Email: libenc...@gmail.com; libenc...@pku.edu.cn

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Xintong Song]

Congratulations, Jingsong. Well deserved~！


Thank you~

Xintong Song



On Fri, Feb 21, 2020 at 11:05 AM Kurt Young  wrote:

> Hi everyone,
>
> I'm very happy to announce that Jingsong Lee accepted the offer of the
> Flink PMC to
> become a committer of the Flink project.
>
> Jingsong Lee has been an active community member for more than a year now.
> He is
> mainly focus on Flink SQL, played an essential role during blink planner
> merging, drives
> FLIP-63 and helped implementing rework expression design, and also
> implemented
> and fixed lots of features and bugs in Flink SQL. Moreover, he is very
> active in both dev
> and user mailing lists, helped discussing designs and answering users
> questions, also
> helped to verify various releases.
>
> Congratulations Jingsong!
>
> Best, Kurt
> (on behalf of the Flink PMC)
>
>
>

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[tison]

Congrats Jingsong, well deserved!

Best,
tison.


Xintong Song  于2020年2月21日周五 上午11:09写道：

> Congratulations, Jingsong. Well deserved~！
>
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Feb 21, 2020 at 11:05 AM Kurt Young  wrote:
>
>> Hi everyone,
>>
>> I'm very happy to announce that Jingsong Lee accepted the offer of the
>> Flink PMC to
>> become a committer of the Flink project.
>>
>> Jingsong Lee has been an active community member for more than a year
>> now. He is
>> mainly focus on Flink SQL, played an essential role during blink planner
>> merging, drives
>> FLIP-63 and helped implementing rework expression design, and also
>> implemented
>> and fixed lots of features and bugs in Flink SQL. Moreover, he is very
>> active in both dev
>> and user mailing lists, helped discussing designs and answering users
>> questions, also
>> helped to verify various releases.
>>
>> Congratulations Jingsong!
>>
>> Best, Kurt
>> (on behalf of the Flink PMC)
>>
>>
>>

[DISCUSS] Support "create [temporary] table" and "create [temporary] view" statements

[Caizhi Weng]

Hi dear Flink developers,

I'm quite confused by the "create view" statement in SQL client.

>From traditional database we expect that a view behaves just like a table
except that it will be computed when used. We also expect that a view
should be created under a database just like a table.

But when I create a view in SQL client and switch to another database by
the "use" statement, the view is still there. It seems that this view is
created in all databases under the same catalog.

I look into the source code and discover that the view I created is
actually a temporary view. In
https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/common.html it
is stated that

> On the other hand, temporary tables are always stored in memory and only
> exist for the duration of the Flink session they are created within. These
> tables are not visible to other sessions. They are not bound to any catalog
> or database but can be created in the namespace of one. Temporary tables
> are not dropped if their corresponding database is removed.

So this seems to be the desired behavior.

What seems really strange is that in SQL client, by using the "create
table" statement I'll get a permanent table, while by using the "create
view" statement I'll get a temporary view, which is sort of inconsistent.

So I would like to discuss that

   1. Shall we make it clear and support create table / create temporary
   table / create view / create temporary view statements?
   2. If yes, shall we need to support them just in SQL client or in SQL
   parser, or in some other places?
   3. Shall catalog manager support creating both permanent and temporary
   views?

Thanks.

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Leonard Xu]

Congratulations！！ Jingsong.

Best, 
Leonard Xu

> 在 2020年2月21日，11:18，tison  写道：
> 
> Congrats Jingsong, well deserved!
> 
> Best,
> tison.
> 
> 
> Xintong Song mailto:tonysong...@gmail.com>> 
> 于2020年2月21日周五 上午11:09写道：
> Congratulations, Jingsong. Well deserved~！
> 
> Thank you~
> Xintong Song
> 
> 
> On Fri, Feb 21, 2020 at 11:05 AM Kurt Young  > wrote:
> Hi everyone, 
> 
> I'm very happy to announce that Jingsong Lee accepted the offer of the Flink 
> PMC to 
> become a committer of the Flink project.
> 
> Jingsong Lee has been an active community member for more than a year now. He 
> is 
> mainly focus on Flink SQL, played an essential role during blink planner 
> merging, drives 
> FLIP-63 and helped implementing rework expression design, and also 
> implemented 
> and fixed lots of features and bugs in Flink SQL. Moreover, he is very active 
> in both dev 
> and user mailing lists, helped discussing designs and answering users 
> questions, also 
> helped to verify various releases.
> 
> Congratulations Jingsong!
> 
> Best, Kurt
> (on behalf of the Flink PMC)
> 
> 

Re: [VOTE] FLIP-103: Better TM/JM Log Display

[lining jing]

+1 (non-binding)
It lists all log files, the user could see the GC log.

Xintong Song  于2020年2月21日周五 上午10:44写道：

> +1 (non-binding)
>
> I like the ideas of having a list of all log files, and make them
> downloadable.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Feb 21, 2020 at 10:24 AM Yang Wang  wrote:
>
> > +1 non-binding
> >
> > The log files list are shown in the new design. I like the idea. Since
> the
> > user defined logs(e.g. my-log.file)
> > could be found here now.
> >
> >
> > Best,
> > Yang
> >
> > Yadong Xie  于2020年2月20日周四 下午5:56写道：
> >
> > > Hi all
> > >
> > > I want to start the vote for FLIP-103, which proposes to provide better
> > > user log display UI in task manager and job manager.
> > >
> > > To help everyone better understand the proposal, we spent some efforts
> on
> > > making an online POC
> > >
> > > Task manager:
> > >
> > > previous web:
> > >
> > >
> >
> http://101.132.122.69:8081/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/logs
> > > POC web:
> > >
> > >
> >
> http://101.132.122.69:8081/web/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/log
> > >
> > > Job manager:
> > >
> > > previous web: http://101.132.122.69:8081/#/job-manager/logs
> > > POC web: http://101.132.122.69:8081/web/#/job-manager/log
> > >
> > >
> > > The vote will last for at least 72 hours, following the consensus
> voting
> > > process.
> > >
> > > FLIP wiki:
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143
> > >
> > > Discussion thread:
> > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > >
> > > Thanks,
> > >
> > > Yadong
> > >
> >
>

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Rui Li]

Congrats Jingsong!

On Fri, Feb 21, 2020 at 11:05 AM Kurt Young  wrote:

> Hi everyone,
>
> I'm very happy to announce that Jingsong Lee accepted the offer of the
> Flink PMC to
> become a committer of the Flink project.
>
> Jingsong Lee has been an active community member for more than a year now.
> He is
> mainly focus on Flink SQL, played an essential role during blink planner
> merging, drives
> FLIP-63 and helped implementing rework expression design, and also
> implemented
> and fixed lots of features and bugs in Flink SQL. Moreover, he is very
> active in both dev
> and user mailing lists, helped discussing designs and answering users
> questions, also
> helped to verify various releases.
>
> Congratulations Jingsong!
>
> Best, Kurt
> (on behalf of the Flink PMC)
>
>
>

-- 
Best regards!
Rui Li

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[zoudan]

Congratulations! Jingsong


Best,
Dan Zou

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Jark Wu]

Congratulations Jingsong! Well deserved.

Best,
Jark

On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:

> Congratulations! Jingsong
>
>
> Best,
> Dan Zou
>

Re: [VOTE] FLIP-103: Better TM/JM Log Display

[Kurt Young]

+1 (binding)


On Fri, Feb 21, 2020 at 11:25 AM lining jing  wrote:

> +1 (non-binding)
> It lists all log files, the user could see the GC log.
>
> Xintong Song  于2020年2月21日周五 上午10:44写道：
>
> > +1 (non-binding)
> >
> > I like the ideas of having a list of all log files, and make them
> > downloadable.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Fri, Feb 21, 2020 at 10:24 AM Yang Wang 
> wrote:
> >
> > > +1 non-binding
> > >
> > > The log files list are shown in the new design. I like the idea. Since
> > the
> > > user defined logs(e.g. my-log.file)
> > > could be found here now.
> > >
> > >
> > > Best,
> > > Yang
> > >
> > > Yadong Xie  于2020年2月20日周四 下午5:56写道：
> > >
> > > > Hi all
> > > >
> > > > I want to start the vote for FLIP-103, which proposes to provide
> better
> > > > user log display UI in task manager and job manager.
> > > >
> > > > To help everyone better understand the proposal, we spent some
> efforts
> > on
> > > > making an online POC
> > > >
> > > > Task manager:
> > > >
> > > > previous web:
> > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/logs
> > > > POC web:
> > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/web/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/log
> > > >
> > > > Job manager:
> > > >
> > > > previous web: http://101.132.122.69:8081/#/job-manager/logs
> > > > POC web: http://101.132.122.69:8081/web/#/job-manager/log
> > > >
> > > >
> > > > The vote will last for at least 72 hours, following the consensus
> > voting
> > > > process.
> > > >
> > > > FLIP wiki:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143
> > > >
> > > > Discussion thread:
> > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > > >
> > > > Thanks,
> > > >
> > > > Yadong
> > > >
> > >
> >
>

Re: [DISCUSS] Support "create [temporary] table" and "create [temporary] view" statements

[Jingsong Li]

Hi Caizhi, thanks for starting this discussion.

There is a FLIP-71 [1] to describe the whole story of view.

Sql-cli now implements a wrong way, and a separate way, which should be
deprecated and unified to TableEnvironment.

> Shall we make it clear and support create table / create temporary table
/ create view / create temporary view statements?

I think there are clear definitions for "create view" (permanent) and
"create temporary view" (temporary).

> If yes, shall we need to support them just in SQL client or in SQL
parser, or in some other places?

We should support in a unify way for TableEnvironment.sqlUpdate, and
SQL-CLI just pass to this method.

> Shall catalog manager support creating both permanent and temporary views?

Actually, it already supports these things, has done in FLIP-64 [2].

If you are interested, consider moving forward with the design and
implementation of FLIP-71. What do you think?

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-71+-+E2E+View+support+in+FLINK+SQL
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module


Best,
Jingsong Lee

On Fri, Feb 21, 2020 at 11:21 AM Caizhi Weng  wrote:

> Hi dear Flink developers,
>
> I'm quite confused by the "create view" statement in SQL client.
>
> From traditional database we expect that a view behaves just like a table
> except that it will be computed when used. We also expect that a view
> should be created under a database just like a table.
>
> But when I create a view in SQL client and switch to another database by
> the "use" statement, the view is still there. It seems that this view is
> created in all databases under the same catalog.
>
> I look into the source code and discover that the view I created is
> actually a temporary view. In
>
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/common.html
> it
> is stated that
>
> > On the other hand, temporary tables are always stored in memory and only
> > exist for the duration of the Flink session they are created within.
> These
> > tables are not visible to other sessions. They are not bound to any
> catalog
> > or database but can be created in the namespace of one. Temporary tables
> > are not dropped if their corresponding database is removed.
>
> So this seems to be the desired behavior.
>
> What seems really strange is that in SQL client, by using the "create
> table" statement I'll get a permanent table, while by using the "create
> view" statement I'll get a temporary view, which is sort of inconsistent.
>
> So I would like to discuss that
>
>1. Shall we make it clear and support create table / create temporary
>table / create view / create temporary view statements?
>2. If yes, shall we need to support them just in SQL client or in SQL
>parser, or in some other places?
>3. Shall catalog manager support creating both permanent and temporary
>views?
>
> Thanks.
>


-- 
Best, Jingsong Lee

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Dian Fu]

Congrats Jingsong!

> 在 2020年2月21日，上午11:39，Jark Wu  写道：
> 
> Congratulations Jingsong! Well deserved.
> 
> Best,
> Jark
> 
> On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
> 
>> Congratulations! Jingsong
>> 
>> 
>> Best,
>> Dan Zou
>> 

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[wenlong.lwl]

Congrats Jingsong!

On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:

> Congrats Jingsong!
>
> > 在 2020年2月21日，上午11:39，Jark Wu  写道：
> >
> > Congratulations Jingsong! Well deserved.
> >
> > Best,
> > Jark
> >
> > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
> >
> >> Congratulations! Jingsong
> >>
> >>
> >> Best,
> >> Dan Zou
> >>
>
>

Re: [VOTE] FLIP-102: Add More Metrics to TaskManager

[Kurt Young]

Some questions related to "managed memory":

1. Should the managed memory be part of direct memory?
2. Should the shuffle memory also be part of the managed memory?

Best,
Kurt


On Fri, Feb 21, 2020 at 10:41 AM Xintong Song  wrote:

> Thanks for driving this FLIP, Yadong.
>
> +1 (non-binding) for the FLIP in general. I think this really helps our
> users to understand and use the new FLIP-49 memory configuration.
>
> I have a few minor comments.
> - There's a frame "Other" in the frame "Non-Heap", besides "JVM Overhead"
> and "JVM Metaspace". IIUC, the purpose of this is to explain the
> mismatching between the metric "non-heap maximum" and the sum of the
> configurations "JVM metaspace" & "JVM Overhead". However, from the
> perspective of FLIP-49, JVM Overhead accounts for all the JVM non-heap
> memory usages except for metaspace. The metrics does not match the
> configuration because we did not set the a JVM parameter for "max non-heap
> memory" (actually I'm not sure whether it can be specified in java 8). The
> current UI might confuse people making them think there are other non-heap
> memory usages not accounted by the configurations. Therefore, I would
> suggest to remove the "Other" frame, but add another frame inside "JVM
> Overhead", besides "Configuration", with "JVM limit" as the title and
> "non-heap max metric minus metaspace configuration" as the value .
>
> - In the final release, we have changed "shuffle memory" to "network
> memory" because the latter is easier to understand for users. I think we
> should be updated it in this FLIP as well.
>
> - There's a typo "Directed" (should be "Direct") at the direct memory
> metric.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Thu, Feb 20, 2020 at 5:52 PM Yadong Xie  wrote:
>
> > Hi all
> >
> > I want to start the vote for FLIP-102, which proposes to add more metrics
> > to the task manager in web UI.
> >
> > To help everyone better understand the proposal, we spent some efforts on
> > making an online POC
> >
> > previous web:
> >
> >
> http://101.132.122.69:8081/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/metrics
> > POC web:
> >
> >
> http://101.132.122.69:8081/web/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/metrics
> >
> >
> > The vote will last for at least 72 hours, following the consensus voting
> > process.
> >
> > FLIP wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager
> >
> > Discussion thread:
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> >
> > Thanks,
> >
> > Yadong
> >
>

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Xingbo Huang]

Congratulations Jingsong! Well deserved.

Best,
Xingbo

wenlong.lwl  于2020年2月21日周五 上午11:43写道：

> Congrats Jingsong!
>
> On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:
>
> > Congrats Jingsong!
> >
> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
> > >
> > > Congratulations Jingsong! Well deserved.
> > >
> > > Best,
> > > Jark
> > >
> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
> > >
> > >> Congratulations! Jingsong
> > >>
> > >>
> > >> Best,
> > >> Dan Zou
> > >>
> >
> >
>

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Jeff Zhang]

Congratulations！Jingsong. You deserve it

wenlong.lwl  于2020年2月21日周五 上午11:43写道：

> Congrats Jingsong!
>
> On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:
>
> > Congrats Jingsong!
> >
> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
> > >
> > > Congratulations Jingsong! Well deserved.
> > >
> > > Best,
> > > Jark
> > >
> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
> > >
> > >> Congratulations! Jingsong
> > >>
> > >>
> > >> Best,
> > >> Dan Zou
> > >>
> >
> >
>


-- 
Best Regards

Jeff Zhang

Re: [VOTE] FLIP-100: Add Attempt Information

[Kurt Young]

Hi Yadong,

Thanks for the proposal, it's a useful feature, especially for batch jobs.
But according
to the examples you gave, I can't tell whether i got required information
from that.
Can you replace the demo job to a more complex batch job and then we can
see some
differences of start/stop time of different tasks and attempts?

Best,
Kurt


On Thu, Feb 20, 2020 at 5:46 PM Yadong Xie  wrote:

> Hi all
>
> I want to start the vote for FLIP-100, which proposes to add attempt
> information inside subtask and timeline in web UI.
>
> To help everyone better understand the proposal, we spent some efforts on
> making an online POC
>
> Timeline Attempt (click the vertex timeline to see the differences):
> previous web:
> http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/timeline
> POC web:
>
> http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/timeline
>
> Subtask Attempt (click the vertex and switch to subtask tab to see the
> differences):
> previous web:
> http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/overview
> POC web:
>
> http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/overview
>
>
> The vote will last for at least 72 hours, following the consensus voting
> process.
>
> FLIP wiki:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information
>
> Discussion thread:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
>
> Thanks,
>
> Yadong
>

Re: [VOTE] FLIP-98: Better Back Pressure Detection

[Kurt Young]

+1 (binding)



On Fri, Feb 21, 2020 at 1:09 AM Zhijiang 
wrote:

> +1 (binding).
> It seems more clearly and directly to highlight the back pressured vertex
> in topology, which can raise the attention of users.
>
> Best,
> Zhijiang
>
>
> --
> From:Congxian Qiu 
> Send Time:2020 Feb. 20 (Thu.) 21:49
> To:dev@flink.apache.org 
> Subject:Re: [VOTE] FLIP-98: Better Back Pressure Detection
>
> +1 for this improvement, It's very helpful to know whether a task is
> backpressured or not.
>
> Best,
> Congxian
>
>
> Yadong Xie  于2020年2月20日周四 下午5:34写道：
>
> > Hi all
> >
> > I want to start the vote for FLIP-98, which proposes to provide better
> > backpressure detection experience in web UI.
> >
> > To help everyone better understand the proposal, we spent some efforts on
> > making an online POC
> >
> > previous web:
> >
> http://101.132.122.69:8081/#/job/add9d3ea1006af15b9f872987eb3befd/overview
> > POC web:
> >
> >
> http://101.132.122.69:8081/web/#/job/add9d3ea1006af15b9f872987eb3befd/overview
> >
> > The vote will last for at least 72 hours, following the consensus voting
> > process.
> >
> > FLIP wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection
> >
> > Discussion thread:
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> >
> > Thanks,
> >
> > Yadong
> >
>
>

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[godfrey he]

Congrats Jingsong! Well deserved.

Best,
godfrey

Jeff Zhang  于2020年2月21日周五 上午11:49写道：

> Congratulations！Jingsong. You deserve it
>
> wenlong.lwl  于2020年2月21日周五 上午11:43写道：
>
>> Congrats Jingsong!
>>
>> On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:
>>
>> > Congrats Jingsong!
>> >
>> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
>> > >
>> > > Congratulations Jingsong! Well deserved.
>> > >
>> > > Best,
>> > > Jark
>> > >
>> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
>> > >
>> > >> Congratulations! Jingsong
>> > >>
>> > >>
>> > >> Best,
>> > >> Dan Zou
>> > >>
>> >
>> >
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: [VOTE] FLIP-102: Add More Metrics to TaskManager

[Xintong Song]

>
> 1. Should the managed memory be part of direct memory?
>
The answer is no. Managed memory is currently allocated by accessing to
private field of Unsafe. It is not accounted for in JVM's direct memory
limit and corresponding metrics. To that end, it is equivalent to
native memory.


> 2. Should the shuffle memory also be part of the managed memory?

I don't think so. Shuffle (Network) memory is allocated with direct
buffers, and accounted for in JVM's direct memory limit and corresponding
metrics. Moreover, the FLIP-49 memory model expose network memory and
managed memory as two independent components of the overall memory
footprint.


Thank you~

Xintong Song



On Fri, Feb 21, 2020 at 11:45 AM Kurt Young  wrote:

> Some questions related to "managed memory":
>
> 1. Should the managed memory be part of direct memory?
> 2. Should the shuffle memory also be part of the managed memory?
>
> Best,
> Kurt
>
>
> On Fri, Feb 21, 2020 at 10:41 AM Xintong Song 
> wrote:
>
> > Thanks for driving this FLIP, Yadong.
> >
> > +1 (non-binding) for the FLIP in general. I think this really helps our
> > users to understand and use the new FLIP-49 memory configuration.
> >
> > I have a few minor comments.
> > - There's a frame "Other" in the frame "Non-Heap", besides "JVM Overhead"
> > and "JVM Metaspace". IIUC, the purpose of this is to explain the
> > mismatching between the metric "non-heap maximum" and the sum of the
> > configurations "JVM metaspace" & "JVM Overhead". However, from the
> > perspective of FLIP-49, JVM Overhead accounts for all the JVM non-heap
> > memory usages except for metaspace. The metrics does not match the
> > configuration because we did not set the a JVM parameter for "max
> non-heap
> > memory" (actually I'm not sure whether it can be specified in java 8).
> The
> > current UI might confuse people making them think there are other
> non-heap
> > memory usages not accounted by the configurations. Therefore, I would
> > suggest to remove the "Other" frame, but add another frame inside "JVM
> > Overhead", besides "Configuration", with "JVM limit" as the title and
> > "non-heap max metric minus metaspace configuration" as the value .
> >
> > - In the final release, we have changed "shuffle memory" to "network
> > memory" because the latter is easier to understand for users. I think we
> > should be updated it in this FLIP as well.
> >
> > - There's a typo "Directed" (should be "Direct") at the direct memory
> > metric.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Thu, Feb 20, 2020 at 5:52 PM Yadong Xie  wrote:
> >
> > > Hi all
> > >
> > > I want to start the vote for FLIP-102, which proposes to add more
> metrics
> > > to the task manager in web UI.
> > >
> > > To help everyone better understand the proposal, we spent some efforts
> on
> > > making an online POC
> > >
> > > previous web:
> > >
> > >
> >
> http://101.132.122.69:8081/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/metrics
> > > POC web:
> > >
> > >
> >
> http://101.132.122.69:8081/web/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/metrics
> > >
> > >
> > > The vote will last for at least 72 hours, following the consensus
> voting
> > > process.
> > >
> > > FLIP wiki:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager
> > >
> > > Discussion thread:
> > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > >
> > > Thanks,
> > >
> > > Yadong
> > >
> >
>

[jira] [Created] (FLINK-16195) append constant field to unique key set on project when deriving unique keys in FlinkRelMdUniqueKeys

[godfrey he (Jira)]

godfrey he created FLINK-16195:
--

 Summary: append constant field to unique key set on project when 
deriving unique keys in FlinkRelMdUniqueKeys
 Key: FLINK-16195
 URL: https://issues.apache.org/jira/browse/FLINK-16195
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Reporter: godfrey he


currently, `FlinkRelMdUniqueKeys` only supports deriving unique keys on 
non-constant fields. such as: `select a, b, 1, count(*) from T group by a, b`, 
currently the derived unique keys is `a, b`. However `a, b, 1` is also a unique 
key, and the result is `a, b` and `a, b, 1`.
note: Ideally, the planner does not require the constant key in the unique key 
set, all constant values are pulled up or removed as much as possible. Supports 
this improvement to handle some corner cases in cbo.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Support "create [temporary] table" and "create [temporary] view" statements

[Jark Wu]

Hi Caizhi,

Thanks for raising this. I also think it is a bug in SQL CLI that CREATE
VIEW creates a temporary view. We should fix that.
If we want to fix it, we have to support create permanent view on
TableEnvironment first which is not supported yet.

Best,
Jark

On Fri, 21 Feb 2020 at 11:40, Jingsong Li  wrote:

> Hi Caizhi, thanks for starting this discussion.
>
> There is a FLIP-71 [1] to describe the whole story of view.
>
> Sql-cli now implements a wrong way, and a separate way, which should be
> deprecated and unified to TableEnvironment.
>
> > Shall we make it clear and support create table / create temporary table
> / create view / create temporary view statements?
>
> I think there are clear definitions for "create view" (permanent) and
> "create temporary view" (temporary).
>
> > If yes, shall we need to support them just in SQL client or in SQL
> parser, or in some other places?
>
> We should support in a unify way for TableEnvironment.sqlUpdate, and
> SQL-CLI just pass to this method.
>
> > Shall catalog manager support creating both permanent and temporary
> views?
>
> Actually, it already supports these things, has done in FLIP-64 [2].
>
> If you are interested, consider moving forward with the design and
> implementation of FLIP-71. What do you think?
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-71+-+E2E+View+support+in+FLINK+SQL
> [2]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module?src=contextnavpagetreemode
> >
>
> Best,
> Jingsong Lee
>
> On Fri, Feb 21, 2020 at 11:21 AM Caizhi Weng  wrote:
>
> > Hi dear Flink developers,
> >
> > I'm quite confused by the "create view" statement in SQL client.
> >
> > From traditional database we expect that a view behaves just like a table
> > except that it will be computed when used. We also expect that a view
> > should be created under a database just like a table.
> >
> > But when I create a view in SQL client and switch to another database by
> > the "use" statement, the view is still there. It seems that this view is
> > created in all databases under the same catalog.
> >
> > I look into the source code and discover that the view I created is
> > actually a temporary view. In
> >
> >
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/common.html
> > it
> > is stated that
> >
> > > On the other hand, temporary tables are always stored in memory and
> only
> > > exist for the duration of the Flink session they are created within.
> > These
> > > tables are not visible to other sessions. They are not bound to any
> > catalog
> > > or database but can be created in the namespace of one. Temporary
> tables
> > > are not dropped if their corresponding database is removed.
> >
> > So this seems to be the desired behavior.
> >
> > What seems really strange is that in SQL client, by using the "create
> > table" statement I'll get a permanent table, while by using the "create
> > view" statement I'll get a temporary view, which is sort of inconsistent.
> >
> > So I would like to discuss that
> >
> >1. Shall we make it clear and support create table / create temporary
> >table / create view / create temporary view statements?
> >2. If yes, shall we need to support them just in SQL client or in SQL
> >parser, or in some other places?
> >3. Shall catalog manager support creating both permanent and temporary
> >views?
> >
> > Thanks.
> >
>
>
> --
> Best, Jingsong Lee
>

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Zhenghua Gao]

Congrats Jingsong！


*Best Regards,*
*Zhenghua Gao*


On Fri, Feb 21, 2020 at 11:59 AM godfrey he  wrote:

> Congrats Jingsong! Well deserved.
>
> Best,
> godfrey
>
> Jeff Zhang  于2020年2月21日周五 上午11:49写道：
>
>> Congratulations！Jingsong. You deserve it
>>
>> wenlong.lwl  于2020年2月21日周五 上午11:43写道：
>>
>>> Congrats Jingsong!
>>>
>>> On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:
>>>
>>> > Congrats Jingsong!
>>> >
>>> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
>>> > >
>>> > > Congratulations Jingsong! Well deserved.
>>> > >
>>> > > Best,
>>> > > Jark
>>> > >
>>> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
>>> > >
>>> > >> Congratulations! Jingsong
>>> > >>
>>> > >>
>>> > >> Best,
>>> > >> Dan Zou
>>> > >>
>>> >
>>> >
>>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Zhijiang]

Congrats Jingsong! Welcome on board!

Best,
Zhijiang


--
From:Zhenghua Gao 
Send Time:2020 Feb. 21 (Fri.) 12:49
To:godfrey he 
Cc:dev ; user 
Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congrats Jingsong！


Best Regards,
Zhenghua Gao

On Fri, Feb 21, 2020 at 11:59 AM godfrey he  wrote:

Congrats Jingsong! Well deserved.

Best,
godfrey
Jeff Zhang  于2020年2月21日周五 上午11:49写道：
Congratulations！Jingsong. You deserve it 

wenlong.lwl  于2020年2月21日周五 上午11:43写道：
Congrats Jingsong!

 On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:

 > Congrats Jingsong!
 >
 > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
 > >
 > > Congratulations Jingsong! Well deserved.
 > >
 > > Best,
 > > Jark
 > >
 > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
 > >
 > >> Congratulations! Jingsong
 > >>
 > >>
 > >> Best,
 > >> Dan Zou
 > >>
 >
 >


-- 
Best Regards

Jeff Zhang

Re: [VOTE] FLIP-98: Better Back Pressure Detection

[Jark Wu]

+1 (binding)

On Fri, 21 Feb 2020 at 11:51, Kurt Young  wrote:

> +1 (binding)
>
>
>
> On Fri, Feb 21, 2020 at 1:09 AM Zhijiang  .invalid>
> wrote:
>
> > +1 (binding).
> > It seems more clearly and directly to highlight the back pressured vertex
> > in topology, which can raise the attention of users.
> >
> > Best,
> > Zhijiang
> >
> >
> > --
> > From:Congxian Qiu 
> > Send Time:2020 Feb. 20 (Thu.) 21:49
> > To:dev@flink.apache.org 
> > Subject:Re: [VOTE] FLIP-98: Better Back Pressure Detection
> >
> > +1 for this improvement, It's very helpful to know whether a task is
> > backpressured or not.
> >
> > Best,
> > Congxian
> >
> >
> > Yadong Xie  于2020年2月20日周四 下午5:34写道：
> >
> > > Hi all
> > >
> > > I want to start the vote for FLIP-98, which proposes to provide better
> > > backpressure detection experience in web UI.
> > >
> > > To help everyone better understand the proposal, we spent some efforts
> on
> > > making an online POC
> > >
> > > previous web:
> > >
> >
> http://101.132.122.69:8081/#/job/add9d3ea1006af15b9f872987eb3befd/overview
> > > POC web:
> > >
> > >
> >
> http://101.132.122.69:8081/web/#/job/add9d3ea1006af15b9f872987eb3befd/overview
> > >
> > > The vote will last for at least 72 hours, following the consensus
> voting
> > > process.
> > >
> > > FLIP wiki:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection
> > >
> > > Discussion thread:
> > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > >
> > > Thanks,
> > >
> > > Yadong
> > >
> >
> >
>

Re: [VOTE] FLIP-99: Make Max Exception Configurable

[Jark Wu]

+1 (binding)

Best,
Jark

On Thu, 20 Feb 2020 at 23:17, Congxian Qiu  wrote:

> +1 to this feature.  this is helpful when troubleshooting.
>
> Best,
> Congxian
>
>
> Yadong Xie  于2020年2月20日周四 下午5:39写道：
>
> > Hi all
> >
> > I want to start the vote for FLIP-99, which proposes to make the max
> > exception configurable in web UI.
> >
> > To help everyone better understand the proposal, we spent some efforts on
> > making an online POC
> >
> > previous web:
> >
> >
> http://101.132.122.69:8081/#/job/543e9dc0cb2cca4433116007f0931d1a/exceptions
> > POC web:
> >
> >
> http://101.132.122.69:8081/web/#/job/543e9dc0cb2cca4433116007f0931d1a/exceptions
> >
> > The vote will last for at least 72 hours, following the consensus voting
> > process.
> >
> > FLIP wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-99%3A+Make+Max+Exception+Configurable
> >
> > Discussion thread:
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> >
> > Thanks,
> >
> > Yadong
> >
>

Re: [DISCUSS] FLINK-16194: Refactor the Kubernetes architecture design

[Yang Wang]

 Hi Canbing,


Thanks a lot for sharing your thoughts to improve the Flink on K8s native
integration.
Frankly speaking, your discussion title confuses me and i am wondering
whether you
want to refactor the whole design. However, after i dive into the details
and the provided
document, i realize that mostly you want to improve the implementation.


Regarding your two main points.

>> Introduce a unified monadic-step based orchestrator architecture that
has a better,
cleaner and consistent abstraction for the Kubernetes resources
construction process,
both applicable to the client side and the master side.

When i introduce the decorator for the K8s in Flink, there is always a
guideline in my mind
that it should be easy for extension and adding new features. Just as you
say, we have lots
of functions to support and the K8s is also evolving very fast. The current
`ConfigMapDecorator`,
`FlinkMasterDeploymentDecorator`, `TaskManagerPodDecorator` is a basic
implementation with
some prerequisite parameters. Of course we could chain more decorators to
construct the K8s
resources. For example, InitializerDecorator -> OwnerReferenceDecorator ->
FlinkMasterDeploymentDecorator -> InitContainerDecorator ->
SidecarDecorator -> etc.

I am little sceptical about splitting every parameter into a single
decorator.  Since it does
not take too much benefits. But i agree with moving some common parameters
into separate
decorators(e.g. volume mount). Also introducing the `~Builder` class and
spinning off the chaining
decorator calls from `Fabric8FlinkKubeClient` make sense to me.



>> Add some dedicated tools for centrally parsing, verifying, and managing
the Kubernetes parameters.

Currently, we always get the parameters directly from Flink configuration(
e.g. `flinkConfig.getString(KubernetesConfigOptions.CONTAINER_IMAGE)`). I
think it could be improved
by introducing some dedicated conf parser classes. It is a good idea.


Best,
Yang




felixzheng zheng  于2020年2月21日周五 上午9:31写道：

> Hi everyone,
>
> I would like to kick off a discussion on refactoring the existing
> Kubernetes resources construction architecture design.
>
> I created a design document [1] that clarifies our motivation to do this
> and some improvement proposals for the new design.
>
> Briefly, we would like to
> 1. Introduce a unified monadic-step based orchestrator architecture that
> has a better, cleaner and consistent abstraction for the Kubernetes
> resources construction process, both applicable to the client side and the
> master side.
> 2. Add some dedicated tools for centrally parsing, verifying, and managing
> the Kubernetes parameters.
>
> It would be great to start the efforts before adding big features for the
> native Kubernetes submodule, and Tison and I plan to work together for this
> issue.
>
> Please let me know your thoughts.
>
> Regards,
> Canbin Zheng
>
> [1]
>
> https://docs.google.com/document/d/1dFBjqho8IRyNWxKVhFnTf0qGCsgp72aKON4wUdHY5Pg/edit?usp=sharing
> [2] https://issues.apache.org/jira/browse/FLINK-16194
>

Re: [VOTE] FLIP-100: Add Attempt Information

[Jark Wu]

Thanks Yadong,

I think one of the purpose to find the failed attempts is that we want to
look into the logs in the failed node (node may change across attempts).
However, from the POC, I didn't find a log link on each attempts. Maybe we
can add such feature?

Best,
Jark

On Fri, 21 Feb 2020 at 11:51, Kurt Young  wrote:

> Hi Yadong,
>
> Thanks for the proposal, it's a useful feature, especially for batch jobs.
> But according
> to the examples you gave, I can't tell whether i got required information
> from that.
> Can you replace the demo job to a more complex batch job and then we can
> see some
> differences of start/stop time of different tasks and attempts?
>
> Best,
> Kurt
>
>
> On Thu, Feb 20, 2020 at 5:46 PM Yadong Xie  wrote:
>
> > Hi all
> >
> > I want to start the vote for FLIP-100, which proposes to add attempt
> > information inside subtask and timeline in web UI.
> >
> > To help everyone better understand the proposal, we spent some efforts on
> > making an online POC
> >
> > Timeline Attempt (click the vertex timeline to see the differences):
> > previous web:
> >
> http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/timeline
> > POC web:
> >
> >
> http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/timeline
> >
> > Subtask Attempt (click the vertex and switch to subtask tab to see the
> > differences):
> > previous web:
> >
> http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/overview
> > POC web:
> >
> >
> http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/overview
> >
> >
> > The vote will last for at least 72 hours, following the consensus voting
> > process.
> >
> > FLIP wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information
> >
> > Discussion thread:
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> >
> > Thanks,
> >
> > Yadong
> >
>

Re: [VOTE] FLIP-101: Add Pending Slots Detail

[Jark Wu]

Thanks Yadong,

I think a pending slot view will be helpful. But will it be verbose when
there is no pending slot, but a "pending slot" in the tab?
What do you think to show the pending slot page when click the "?" on the
vertex status?

Best,
Jark

On Thu, 20 Feb 2020 at 17:50, Yadong Xie  wrote:

> Hi all
>
> I want to start the vote for FLIP-101, which proposes to add pending slots
> information to help users check which vertex/subtask is blocked.
>
> To help everyone better understand the proposal, we spent some efforts on
> making an online POC
>
> previous web:
> http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
> POC web:
>
> http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots
>
>
> The vote will last for at least 72 hours, following the consensus voting
> process.
>
> FLIP wiki:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail
>
> Discussion thread:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
>
> Thanks,
>
> Yadong
>

Re: [VOTE] FLIP-103: Better TM/JM Log Display

[Jark Wu]

+1 (binding)

A minor question: I didn't find the GC log in the POC web. Is it only
generated when some configuration is enabled?

Best,
Jark

On Fri, 21 Feb 2020 at 11:40, Kurt Young  wrote:

> +1 (binding)
>
>
> On Fri, Feb 21, 2020 at 11:25 AM lining jing 
> wrote:
>
> > +1 (non-binding)
> > It lists all log files, the user could see the GC log.
> >
> > Xintong Song  于2020年2月21日周五 上午10:44写道：
> >
> > > +1 (non-binding)
> > >
> > > I like the ideas of having a list of all log files, and make them
> > > downloadable.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Fri, Feb 21, 2020 at 10:24 AM Yang Wang 
> > wrote:
> > >
> > > > +1 non-binding
> > > >
> > > > The log files list are shown in the new design. I like the idea.
> Since
> > > the
> > > > user defined logs(e.g. my-log.file)
> > > > could be found here now.
> > > >
> > > >
> > > > Best,
> > > > Yang
> > > >
> > > > Yadong Xie  于2020年2月20日周四 下午5:56写道：
> > > >
> > > > > Hi all
> > > > >
> > > > > I want to start the vote for FLIP-103, which proposes to provide
> > better
> > > > > user log display UI in task manager and job manager.
> > > > >
> > > > > To help everyone better understand the proposal, we spent some
> > efforts
> > > on
> > > > > making an online POC
> > > > >
> > > > > Task manager:
> > > > >
> > > > > previous web:
> > > > >
> > > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/logs
> > > > > POC web:
> > > > >
> > > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/web/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/log
> > > > >
> > > > > Job manager:
> > > > >
> > > > > previous web: http://101.132.122.69:8081/#/job-manager/logs
> > > > > POC web: http://101.132.122.69:8081/web/#/job-manager/log
> > > > >
> > > > >
> > > > > The vote will last for at least 72 hours, following the consensus
> > > voting
> > > > > process.
> > > > >
> > > > > FLIP wiki:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143
> > > > >
> > > > > Discussion thread:
> > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Yadong
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] FLIP-101: Add Pending Slots Detail

[Kurt Young]

I agree with Jark, even if we have pending slots now, a dedicated tab seems
to be too much.

Best,
Kurt


On Fri, Feb 21, 2020 at 2:12 PM Jark Wu  wrote:

> Thanks Yadong,
>
> I think a pending slot view will be helpful. But will it be verbose when
> there is no pending slot, but a "pending slot" in the tab?
> What do you think to show the pending slot page when click the "?" on the
> vertex status?
>
> Best,
> Jark
>
> On Thu, 20 Feb 2020 at 17:50, Yadong Xie  wrote:
>
> > Hi all
> >
> > I want to start the vote for FLIP-101, which proposes to add pending
> slots
> > information to help users check which vertex/subtask is blocked.
> >
> > To help everyone better understand the proposal, we spent some efforts on
> > making an online POC
> >
> > previous web:
> >
> http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
> > POC web:
> >
> >
> http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots
> >
> >
> > The vote will last for at least 72 hours, following the consensus voting
> > process.
> >
> > FLIP wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail
> >
> > Discussion thread:
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> >
> > Thanks,
> >
> > Yadong
> >
>

Re: [Discuss] Update the pull request description template.

[Yang Wang]

I second xintong's suggestion. When i open a PR, i also check the item list
in the template. It help to
know whether i should test the PR in a real cluster(Yarn/K8s/Mesos). Or i
should be more careful
when touching the per-record code paths.If we have some dependencies
changes, i will need to check
the generated jar as expected.


Best,
Yang

Xintong Song  于2020年2月20日周四 上午10:33写道：

> Thanks for the feedbacks, Chesnay and Till. And thanks for the pointer,
> Congxian.
>
> I don't know how often committers and reviewers checks and benefits from
> the PR description. From your feedbacks and the number of responses to this
> discussion, it's probably not often.
>
> However, as a contributor and speaking only for myself, I actually find the
> PR template very helpful. I use it as a checking list for opening a PR.
> Filling in the template forces me to revisit the important things, e.g.,
> have I added enough test cases to cover the all the important changes, does
> this change need to be validated with a real deployment (if it touches the
> deployment and recovery). An experienced developer might be able to check
> these things without such a checking list, but there might be more primary
> developers that can benefit from it.
>
>
> Therefore, if we agree that PR template is less useful for reviewers, I
> would like to propose to reposition it as a contributor checking list. The
> following are some examples of how the existing items might be
> repositioned.
>
>
> - The runtime per-record code paths (performance sensitive): (yes / no /
> don't know). If yes, please check the following items.
>
> - Is there a good reason to do that?
> - Is there an alternative non pre-record approach?
>
> - Is Java stream or Optional used in the per-recode code path? (Those
> should be avoid according to the code style and quality guide[1])
>
> - Do we know the exact impact on performance? (Maybe point to the
> performance benchmarks)
>
>
> - Anything that affects deployment or recovery: JobManager (and its
> components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no /
> don't know). If yes, please check the following items.
>
> - Has this PR been validated with a real deployment?
>
> - Has this PR been validated with the failover scenarios?
>
> - Does this PR requires any specific version or configuration of an
> external system? E.g., Kubernetes/Yarn/Mesos/ZooKeeper APIs not supported
> by all the versions that Flink claims to be compatible with.
>
>
> WDYT?
>
>
> Thank you~
>
> Xintong Song
>
>
> [1]https://flink.apache.org/contributing/code-style-and-quality-java.html
>
> On Wed, Feb 19, 2020 at 9:24 PM Till Rohrmann 
> wrote:
>
> > I actually wanted to second Chesnay but apparently my impression is a bit
> > wrong. Out of the last 10 closed PRs (admittedly a small sample size)
> only
> > 2 did not fill out the template. I did not check for correctness though.
> >
> > Assuming that people use the template, I believe it is a good idea to
> > update it. One thing to consider is whether we wanna keep the S3 item or
> > want to generalize it. I think there was some reason why we explicitly
> > added it to the template but I cannot really remember.
> >
> > Cheers,
> > Till
> >
> > On Mon, Feb 17, 2020 at 3:02 PM Congxian Qiu 
> > wrote:
> >
> > > JFYI, there is an issue[1] which I think is related to this thread
> > > [1] https://issues.apache.org/jira/browse/FLINK-15977
> > >
> > > Best,
> > > Congxian
> > >
> > >
> > > Chesnay Schepler  于2020年2月17日周一 下午9:08写道：
> > >
> > > > I think it should just be removed since 99% of pull requests ignore
> it
> > > > anyway.
> > > >
> > > > On 17/02/2020 13:31, Xintong Song wrote:
> > > > > Hi all,
> > > > >
> > > > > It seems our PR description template is a bit outdated, and I would
> > > like
> > > > to
> > > > > propose updating it.
> > > > >
> > > > > I was working on a Kubernetes related PR, and realized that our PR
> > > > > description does not mention the new Kubernetes integration
> > questioning
> > > > > about deployment related changes. Currently is is as follows:
> > > > >
> > > > >> Anything that affects deployment or recovery: JobManager (and its
> > > > >> components), Checkpointing, Yarn/Mesos, ZooKeeper:
> > > > >>
> > > > > In addition to outdated contents, there might be other stuff that
> we
> > > want
> > > > > to add to the template. For example, I would suggest add a question
> > > about
> > > > > whether there are any memory allocation introduced by the PR, so we
> > > > review
> > > > > them carefully and avoid problems due to un-accounted memory
> > > allocations
> > > > > like FLINK-15981. (To be fair, for FLINK-15981 the memory
> allocation
> > > was
> > > > > introduced before we start to account for everything memory usage,
> > but
> > > > > noticing such memory allocations early should help us prevent
> similar
> > > > > problems in the future).
> > > > >
> > > > > Therefore, I'd also like to collect ideas on how do you think the
> > > > template

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Yang Wang]

Congratulations！Jingsong. Well deserved.


Best,
Yang

Zhijiang  于2020年2月21日周五 下午1:18写道：

> Congrats Jingsong! Welcome on board!
>
> Best,
> Zhijiang
>
> --
> From:Zhenghua Gao 
> Send Time:2020 Feb. 21 (Fri.) 12:49
> To:godfrey he 
> Cc:dev ; user 
> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>
> Congrats Jingsong！
>
>
> *Best Regards,*
> *Zhenghua Gao*
>
>
> On Fri, Feb 21, 2020 at 11:59 AM godfrey he  wrote:
> Congrats Jingsong! Well deserved.
>
> Best,
> godfrey
>
> Jeff Zhang  于2020年2月21日周五 上午11:49写道：
> Congratulations！Jingsong. You deserve it
>
> wenlong.lwl  于2020年2月21日周五 上午11:43写道：
> Congrats Jingsong!
>
> On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:
>
> > Congrats Jingsong!
> >
> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
> > >
> > > Congratulations Jingsong! Well deserved.
> > >
> > > Best,
> > > Jark
> > >
> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
> > >
> > >> Congratulations! Jingsong
> > >>
> > >>
> > >> Best,
> > >> Dan Zou
> > >>
> >
> >
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>
>

Re: [VOTE] FLIP-98: Better Back Pressure Detection

[Yang Wang]

+1 non-binding

It is really straightforward.

Best,
Yang

Jark Wu  于2020年2月21日周五 下午1:52写道：

> +1 (binding)
>
> On Fri, 21 Feb 2020 at 11:51, Kurt Young  wrote:
>
> > +1 (binding)
> >
> >
> >
> > On Fri, Feb 21, 2020 at 1:09 AM Zhijiang  > .invalid>
> > wrote:
> >
> > > +1 (binding).
> > > It seems more clearly and directly to highlight the back pressured
> vertex
> > > in topology, which can raise the attention of users.
> > >
> > > Best,
> > > Zhijiang
> > >
> > >
> > > --
> > > From:Congxian Qiu 
> > > Send Time:2020 Feb. 20 (Thu.) 21:49
> > > To:dev@flink.apache.org 
> > > Subject:Re: [VOTE] FLIP-98: Better Back Pressure Detection
> > >
> > > +1 for this improvement, It's very helpful to know whether a task is
> > > backpressured or not.
> > >
> > > Best,
> > > Congxian
> > >
> > >
> > > Yadong Xie  于2020年2月20日周四 下午5:34写道：
> > >
> > > > Hi all
> > > >
> > > > I want to start the vote for FLIP-98, which proposes to provide
> better
> > > > backpressure detection experience in web UI.
> > > >
> > > > To help everyone better understand the proposal, we spent some
> efforts
> > on
> > > > making an online POC
> > > >
> > > > previous web:
> > > >
> > >
> >
> http://101.132.122.69:8081/#/job/add9d3ea1006af15b9f872987eb3befd/overview
> > > > POC web:
> > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/web/#/job/add9d3ea1006af15b9f872987eb3befd/overview
> > > >
> > > > The vote will last for at least 72 hours, following the consensus
> > voting
> > > > process.
> > > >
> > > > FLIP wiki:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection
> > > >
> > > > Discussion thread:
> > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > > >
> > > > Thanks,
> > > >
> > > > Yadong
> > > >
> > >
> > >
> >
>

Re: [VOTE] FLIP-104: Add More Metrics to Jobmanager

[Jark Wu]

Thanks Yadong,

I think we can use different color to distinguish the memory usage (from
green to red?).
Besides, I think we should add an unit on the "Garbage Collection" ->
"Time", it's hard to know what the value mean.
Would be better to display the value like "10ms", "5ns".

Best,
Jark

On Thu, 20 Feb 2020 at 17:58, Yadong Xie  wrote:

> Hi all
>
> I want to start the vote for FLIP-104, which proposes to add more metrics
> to job manager.
>
> To help everyone better understand the proposal, we spent some efforts on
> making an online POC
>
> previous web: http://101.132.122.69:8081/#/job-manager/config
> POC web: http://101.132.122.69:8081/web/#/job-manager/metrics
>
>
> The vote will last for at least 72 hours, following the consensus voting
> process.
>
> FLIP wiki:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-104%3A+Add+More+Metrics+to+Jobmanager
>
> Discussion thread:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
>
> Thanks,
>
> Yadong
>

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Hequn Cheng]

Congratulations Jingsong! Well deserved.

Best,
Hequn

On Fri, Feb 21, 2020 at 2:42 PM Yang Wang  wrote:

> Congratulations！Jingsong. Well deserved.
>
>
> Best,
> Yang
>
> Zhijiang  于2020年2月21日周五 下午1:18写道：
>
>> Congrats Jingsong! Welcome on board!
>>
>> Best,
>> Zhijiang
>>
>> --
>> From:Zhenghua Gao 
>> Send Time:2020 Feb. 21 (Fri.) 12:49
>> To:godfrey he 
>> Cc:dev ; user 
>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>>
>> Congrats Jingsong！
>>
>>
>> *Best Regards,*
>> *Zhenghua Gao*
>>
>>
>> On Fri, Feb 21, 2020 at 11:59 AM godfrey he  wrote:
>> Congrats Jingsong! Well deserved.
>>
>> Best,
>> godfrey
>>
>> Jeff Zhang  于2020年2月21日周五 上午11:49写道：
>> Congratulations！Jingsong. You deserve it
>>
>> wenlong.lwl  于2020年2月21日周五 上午11:43写道：
>> Congrats Jingsong!
>>
>> On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:
>>
>> > Congrats Jingsong!
>> >
>> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
>> > >
>> > > Congratulations Jingsong! Well deserved.
>> > >
>> > > Best,
>> > > Jark
>> > >
>> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
>> > >
>> > >> Congratulations! Jingsong
>> > >>
>> > >>
>> > >> Best,
>> > >> Dan Zou
>> > >>
>> >
>> >
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>>
>>

Re: [VOTE] FLIP-103: Better TM/JM Log Display

[Yang Wang]

Hi Jark,

The GC log is not enabled by default. If the users want, he/she should
configure it by the following config option.
env.java.opts.jobmanager: -XX:+PrintGCDetails -XX:+PrintGCDateStamps
-Xloggc:/opt/flink/log/jobmanager-gc.log -XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=512M

Best,
Yang

Jark Wu  于2020年2月21日周五 下午2:19写道：

> +1 (binding)
>
> A minor question: I didn't find the GC log in the POC web. Is it only
> generated when some configuration is enabled?
>
> Best,
> Jark
>
> On Fri, 21 Feb 2020 at 11:40, Kurt Young  wrote:
>
> > +1 (binding)
> >
> >
> > On Fri, Feb 21, 2020 at 11:25 AM lining jing 
> > wrote:
> >
> > > +1 (non-binding)
> > > It lists all log files, the user could see the GC log.
> > >
> > > Xintong Song  于2020年2月21日周五 上午10:44写道：
> > >
> > > > +1 (non-binding)
> > > >
> > > > I like the ideas of having a list of all log files, and make them
> > > > downloadable.
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Fri, Feb 21, 2020 at 10:24 AM Yang Wang 
> > > wrote:
> > > >
> > > > > +1 non-binding
> > > > >
> > > > > The log files list are shown in the new design. I like the idea.
> > Since
> > > > the
> > > > > user defined logs(e.g. my-log.file)
> > > > > could be found here now.
> > > > >
> > > > >
> > > > > Best,
> > > > > Yang
> > > > >
> > > > > Yadong Xie  于2020年2月20日周四 下午5:56写道：
> > > > >
> > > > > > Hi all
> > > > > >
> > > > > > I want to start the vote for FLIP-103, which proposes to provide
> > > better
> > > > > > user log display UI in task manager and job manager.
> > > > > >
> > > > > > To help everyone better understand the proposal, we spent some
> > > efforts
> > > > on
> > > > > > making an online POC
> > > > > >
> > > > > > Task manager:
> > > > > >
> > > > > > previous web:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/logs
> > > > > > POC web:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/web/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/log
> > > > > >
> > > > > > Job manager:
> > > > > >
> > > > > > previous web: http://101.132.122.69:8081/#/job-manager/logs
> > > > > > POC web: http://101.132.122.69:8081/web/#/job-manager/log
> > > > > >
> > > > > >
> > > > > > The vote will last for at least 72 hours, following the consensus
> > > > voting
> > > > > > process.
> > > > > >
> > > > > > FLIP wiki:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143
> > > > > >
> > > > > > Discussion thread:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Yadong
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] FLIP-103: Better TM/JM Log Display

[Yangze Guo]

+1 (non-binding)

It will make debugging more convenient and hence improve usability.

Best,
Yangze Guo

On Fri, Feb 21, 2020 at 2:50 PM Yang Wang  wrote:
>
> Hi Jark,
>
> The GC log is not enabled by default. If the users want, he/she should
> configure it by the following config option.
> env.java.opts.jobmanager: -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> -Xloggc:/opt/flink/log/jobmanager-gc.log -XX:+UseGCLogFileRotation
> -XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=512M
>
> Best,
> Yang
>
> Jark Wu  于2020年2月21日周五 下午2:19写道：
>
> > +1 (binding)
> >
> > A minor question: I didn't find the GC log in the POC web. Is it only
> > generated when some configuration is enabled?
> >
> > Best,
> > Jark
> >
> > On Fri, 21 Feb 2020 at 11:40, Kurt Young  wrote:
> >
> > > +1 (binding)
> > >
> > >
> > > On Fri, Feb 21, 2020 at 11:25 AM lining jing 
> > > wrote:
> > >
> > > > +1 (non-binding)
> > > > It lists all log files, the user could see the GC log.
> > > >
> > > > Xintong Song  于2020年2月21日周五 上午10:44写道：
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > I like the ideas of having a list of all log files, and make them
> > > > > downloadable.
> > > > >
> > > > > Thank you~
> > > > >
> > > > > Xintong Song
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Feb 21, 2020 at 10:24 AM Yang Wang 
> > > > wrote:
> > > > >
> > > > > > +1 non-binding
> > > > > >
> > > > > > The log files list are shown in the new design. I like the idea.
> > > Since
> > > > > the
> > > > > > user defined logs(e.g. my-log.file)
> > > > > > could be found here now.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Yang
> > > > > >
> > > > > > Yadong Xie  于2020年2月20日周四 下午5:56写道：
> > > > > >
> > > > > > > Hi all
> > > > > > >
> > > > > > > I want to start the vote for FLIP-103, which proposes to provide
> > > > better
> > > > > > > user log display UI in task manager and job manager.
> > > > > > >
> > > > > > > To help everyone better understand the proposal, we spent some
> > > > efforts
> > > > > on
> > > > > > > making an online POC
> > > > > > >
> > > > > > > Task manager:
> > > > > > >
> > > > > > > previous web:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > http://101.132.122.69:8081/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/logs
> > > > > > > POC web:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > http://101.132.122.69:8081/web/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/log
> > > > > > >
> > > > > > > Job manager:
> > > > > > >
> > > > > > > previous web: http://101.132.122.69:8081/#/job-manager/logs
> > > > > > > POC web: http://101.132.122.69:8081/web/#/job-manager/log
> > > > > > >
> > > > > > >
> > > > > > > The vote will last for at least 72 hours, following the consensus
> > > > > voting
> > > > > > > process.
> > > > > > >
> > > > > > > FLIP wiki:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143
> > > > > > >
> > > > > > > Discussion thread:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Yadong
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >