[jira] [Created] (FLINK-17062) Fix the conversion from Java row type to Python row type

[Dian Fu (Jira)]

Dian Fu created FLINK-17062:
---

 Summary: Fix the conversion from Java row type to Python row type
 Key: FLINK-17062
 URL: https://issues.apache.org/jira/browse/FLINK-17062
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.9.0
Reporter: Dian Fu
Assignee: Dian Fu
 Fix For: 1.9.3, 1.10.1, 1.11.0


It iterate over the result of FieldsDataType.getFieldDataTypes when converting 
Java row type to Python row type. The result is non-deterministic as the result 
of FieldsDataType.getFieldDataTypes is of type map.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17063) Make null typ be a possible result of a type inference

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-17063:


 Summary: Make null typ be a possible result of a type inference
 Key: FLINK-17063
 URL: https://issues.apache.org/jira/browse/FLINK-17063
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.11.0


There are use cases that can benefit from making a null type a valid result of 
input and output type inference.

Example. We can use null type in {{VALUES}} clause for a temporary placeholder 
that will be later inferred based on types of other rows.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17064) Improve ExpressionConverter

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-17064:


 Summary: Improve ExpressionConverter
 Key: FLINK-17064
 URL: https://issues.apache.org/jira/browse/FLINK-17064
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.11.0


There are couple of issues with the {{ExpressionResolver}}:
1. There is a lot of code duplication
2. Precision of certain types might get lost e.g. BINARY, CHAR



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-123: DDL and DML compatibility for Hive connector

[Rui Li]

Hi Kurt,

Thanks for the feedback and that's a good idea. I have updated the FLIP and
added tables in the "Limited Scope" section to list the supported and
unsupported features. Please have a look and let me know if that makes
sense. Thanks.

On Wed, Apr 8, 2020 at 2:19 PM Kurt Young  wrote:

> Hi Rui,
>
> Thanks for bringing up this discussion and it makes sense to me though i
> have one comment about the FLIP.
> There are a few places in the design document saying some features will not
> be supported or not included in
> this FLIP, but I don't get what will be supported exactly. I can imagine
> other users will also have such confusion.
> Could you add a table or a list of syntax which will be supported?
>
> Best,
> Kurt
>
>
> On Wed, Apr 1, 2020 at 4:24 PM Rui Li  wrote:
>
> > Hi devs,
> >
> > I'd like to start a discussion about bringing DDL & DML compatibility for
> > Hive connector. The proposal mainly aims to implement more DDLs for Hive
> > connector and allow users to write Hive syntax when using the Hive
> dialect.
> > Hopefully this will make it easier for users to migrate to Flink, with
> > fewer SQL statements that need to be changed.
> >
> > Please find more details in the FLIP wiki [1]. Feedbacks and suggestions
> > are appreciated.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-123%3A+DDL+and+DML+compatibility+for+Hive+connector
> >
> > --
> > Cheers,
> > Rui Li
> >
>


-- 
Best regards!
Rui Li

Re: Configuring autolinks to Flink JIRA ticket in github repos

[Yun Tang]

Hi community

The autolink to Flink JIRA ticket has taken effect. You could refer to the 
commit details page[1] to see all Flink JIRA titles within commits has the 
hyper link underline. Moreover, you don't need to use markdown language to 
create hyper link to Flink JIRA ticket when discussing in the pull requests. 
e.g FLINK-16850 could point to the link instead of 
[FLINK-16850](https://issues.apache.org/jira/browse/FLINK-16850)


[1] https://github.com/apache/flink/commits/master

Best
Yun Tang


From: Till Rohrmann 
Sent: Thursday, April 2, 2020 23:11
To: dev 
Subject: Re: Configuring autolinks to Flink JIRA ticket in github repos

Nice, this is a cool feature. Thanks for asking INFRA for it.

Cheers,
Till

On Wed, Apr 1, 2020 at 6:52 PM Yun Tang  wrote:

> Hi community.
>
> I noticed that Github supports autolink reference recently [1]. This is
> helpful to allow developers could open Jira ticket link from pull requests
> title directly when accessing github repo.
>
> I have already created INFRA-20055 [2] to ask for configuration for seven
> Flink related github repos. Hope it could be resolved soon 🙂
>
>
> [1]
> https://help.github.com/en/github/administering-a-repository/configuring-autolinks-to-reference-external-resources
> [2] https://issues.apache.org/jira/browse/INFRA-20055
>
> Best
> Yun Tang
>

Re: Configuring autolinks to Flink JIRA ticket in github repos

[Jark Wu]

Thanks Yun,

This's a great feature! I was surprised by the autolink feature yesterday
(didn't know your work at that time).

Best,
Jark

On Thu, 9 Apr 2020 at 16:12, Yun Tang  wrote:

> Hi community
>
> The autolink to Flink JIRA ticket has taken effect. You could refer to the
> commit details page[1] to see all Flink JIRA titles within commits has the
> hyper link underline. Moreover, you don't need to use markdown language to
> create hyper link to Flink JIRA ticket when discussing in the pull
> requests. e.g FLINK-16850 could point to the link instead of [FLINK-16850](
> https://issues.apache.org/jira/browse/FLINK-16850)
>
>
> [1] https://github.com/apache/flink/commits/master
>
> Best
> Yun Tang
>
> 
> From: Till Rohrmann 
> Sent: Thursday, April 2, 2020 23:11
> To: dev 
> Subject: Re: Configuring autolinks to Flink JIRA ticket in github repos
>
> Nice, this is a cool feature. Thanks for asking INFRA for it.
>
> Cheers,
> Till
>
> On Wed, Apr 1, 2020 at 6:52 PM Yun Tang  wrote:
>
> > Hi community.
> >
> > I noticed that Github supports autolink reference recently [1]. This is
> > helpful to allow developers could open Jira ticket link from pull
> requests
> > title directly when accessing github repo.
> >
> > I have already created INFRA-20055 [2] to ask for configuration for seven
> > Flink related github repos. Hope it could be resolved soon 🙂
> >
> >
> > [1]
> >
> https://help.github.com/en/github/administering-a-repository/configuring-autolinks-to-reference-external-resources
> > [2] https://issues.apache.org/jira/browse/INFRA-20055
> >
> > Best
> > Yun Tang
> >
>

[jira] [Created] (FLINK-17065) Adds notes of Python versions supported by PyFlink in doc

[Huang Xingbo (Jira)]

Huang Xingbo created FLINK-17065:


 Summary: Adds notes of Python versions supported by PyFlink in doc
 Key: FLINK-17065
 URL: https://issues.apache.org/jira/browse/FLINK-17065
 Project: Flink
  Issue Type: Improvement
  Components: API / Python, Documentation
Reporter: Huang Xingbo
 Fix For: 1.9.3, 1.10.1, 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17066) Limits the PyArrow version to 0.13

[Huang Xingbo (Jira)]

Huang Xingbo created FLINK-17066:


 Summary: Limits the PyArrow version to 0.13
 Key: FLINK-17066
 URL: https://issues.apache.org/jira/browse/FLINK-17066
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Reporter: Huang Xingbo
 Fix For: 1.10.1


We need to limits the version of PyArrow to 0.13 in PyFlink 1.10. The bug[1] 
comes from the dependency of beam 2.15 which has been resolved in beam 2.17.

 [1] https://issues.apache.org/jira/browse/BEAM-8368



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Configuring autolinks to Flink JIRA ticket in github repos

[Yu Li]

Great! Thanks for the efforts Yun.

Best Regards,
Yu


On Thu, 9 Apr 2020 at 16:15, Jark Wu  wrote:

> Thanks Yun,
>
> This's a great feature! I was surprised by the autolink feature yesterday
> (didn't know your work at that time).
>
> Best,
> Jark
>
> On Thu, 9 Apr 2020 at 16:12, Yun Tang  wrote:
>
> > Hi community
> >
> > The autolink to Flink JIRA ticket has taken effect. You could refer to
> the
> > commit details page[1] to see all Flink JIRA titles within commits has
> the
> > hyper link underline. Moreover, you don't need to use markdown language
> to
> > create hyper link to Flink JIRA ticket when discussing in the pull
> > requests. e.g FLINK-16850 could point to the link instead of
> [FLINK-16850](
> > https://issues.apache.org/jira/browse/FLINK-16850)
> >
> >
> > [1] https://github.com/apache/flink/commits/master
> >
> > Best
> > Yun Tang
> >
> > 
> > From: Till Rohrmann 
> > Sent: Thursday, April 2, 2020 23:11
> > To: dev 
> > Subject: Re: Configuring autolinks to Flink JIRA ticket in github repos
> >
> > Nice, this is a cool feature. Thanks for asking INFRA for it.
> >
> > Cheers,
> > Till
> >
> > On Wed, Apr 1, 2020 at 6:52 PM Yun Tang  wrote:
> >
> > > Hi community.
> > >
> > > I noticed that Github supports autolink reference recently [1]. This is
> > > helpful to allow developers could open Jira ticket link from pull
> > requests
> > > title directly when accessing github repo.
> > >
> > > I have already created INFRA-20055 [2] to ask for configuration for
> seven
> > > Flink related github repos. Hope it could be resolved soon 🙂
> > >
> > >
> > > [1]
> > >
> >
> https://help.github.com/en/github/administering-a-repository/configuring-autolinks-to-reference-external-resources
> > > [2] https://issues.apache.org/jira/browse/INFRA-20055
> > >
> > > Best
> > > Yun Tang
> > >
> >
>

[DISCUSS] Generating java code?

[Niels Basjes]

Hi,

I'm working on https://issues.apache.org/jira/browse/FLINK-16871 to make
more build time variables (like the scala version) into the code available
at runtime.

During the review process there was discussion around a basic question: *Is
generating java code during the build ok?*
See

   - https://github.com/apache/flink/pull/11245#discussion_r400035133
   - https://github.com/apache/flink/pull/11592
   - https://github.com/apache/flink/pull/11592#issuecomment-610282450

As suggested by Chesnay Schepler I'm putting this question to the mailing
list.
  https://github.com/apache/flink/pull/11592#issuecomment-610963947

The main discussion was around the ease of use when running in an IDE like
IntelliJ.

So essentially we have two solution directions available:

   1. *Generate a properties file and then use the classloader to load this
   file as a resource and then parse it as a property file.*
   This is the currently used solution direction for this part of the code.
   A rough version of this (to be improved) :
   
https://github.com/apache/flink/commit/47099f663b7644056e9d87b262cd4dba034f513e
   This method has several effects:
  1. The developer can run the project immediately from within the IDE
  as fallback values are provided if the 'actual' values are missing.
  2. This property file (with stuff that should never be overwritten)
  can be modified by placing a different one in the classpath. In
fact it IS
  modified in the flink-dist as it generates a new file with the same name
  into the binary distribution (I consider this to be bad).
  3. Loading resources means loading, parsing and a lot of error
  handling. Lots of things "can be null" or  be a default value. So the
  values are unreliable and lots of code needs to handle this. In fact when
  running from IntelliJ the properties file is generated poorly most of the
  time, only during a normal maven build will it work correctly.
   2. *Generate a Java source file and then simply compile this and make it
   part of the project.*
   Essentially the same model as you would have when using Apache Avro,
   Protobuf, Antlr 4 and javacc (several of those are used in Flink!).
   A rough version of this (to be improved) :
   
https://github.com/apache/flink/commit/d215e4df60dc9d647dcee1aa9a2114cbf49d0566
   This method has several effects:
   1. The developer MUST run 'mvn generate-sources' before the actual the
  project immediately from within the IDE as fallback values are
provided if
  the 'actual' values are missing.
  2. The code/test will not run until this step is done.
  3. Because the file is generated by a plugin it is always correct. As
  a consequence all variables are always available and the downstream users
  no longer have to handle the "can be null" or "default value" situations.

So is generating code similar to what I created a desired change?
My opinion is that it is the better solution, the data available is more
reliable and as a consequence the rest of the code is simpler.
It would probably mean that during development of flink you should be aware
of this and do an 'extra step' to get it running.

What do you guys think?

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

contributor apply

[zhangzhanhua]

Hi Guys,

I want to contribute to Apache Flink.
Would you please give me the permission as a contributor?
My JIRA ID is zhangzhanhua.

[jira] [Created] (FLINK-17067) CatalogManager#createTable and createTemporaryTable should provide consistent semantics

[Zhenghua Gao (Jira)]

Zhenghua Gao created FLINK-17067:


 Summary: CatalogManager#createTable and createTemporaryTable 
should provide consistent semantics
 Key: FLINK-17067
 URL: https://issues.apache.org/jira/browse/FLINK-17067
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Affects Versions: 1.10.0
Reporter: Zhenghua Gao
 Fix For: 1.11.0


Currently CatalogManager#createTable provides  [IF NOT EXISTS] semantic and 
CatalogManager#createTemporaryTable provides [OR REPLACE] semantic. IMO they 
should provide consistent semantics: [IF NOT EXISTS] or [OR REPLACE] or BOTH.

I prefer [IF NOT EXISTS] since we didn't support [OR REPLACE] in table DDL(and 
view DDL) currently.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Configuring autolinks to Flink JIRA ticket in github repos

[Dian Fu]

Cool! Thanks Yun for this effort. Very useful feature.

Regards,
Dian

> 在 2020年4月9日，下午4:32，Yu Li  写道：
> 
> Great! Thanks for the efforts Yun.
> 
> Best Regards,
> Yu
> 
> 
> On Thu, 9 Apr 2020 at 16:15, Jark Wu  wrote:
> 
>> Thanks Yun,
>> 
>> This's a great feature! I was surprised by the autolink feature yesterday
>> (didn't know your work at that time).
>> 
>> Best,
>> Jark
>> 
>> On Thu, 9 Apr 2020 at 16:12, Yun Tang  wrote:
>> 
>>> Hi community
>>> 
>>> The autolink to Flink JIRA ticket has taken effect. You could refer to
>> the
>>> commit details page[1] to see all Flink JIRA titles within commits has
>> the
>>> hyper link underline. Moreover, you don't need to use markdown language
>> to
>>> create hyper link to Flink JIRA ticket when discussing in the pull
>>> requests. e.g FLINK-16850 could point to the link instead of
>> [FLINK-16850](
>>> https://issues.apache.org/jira/browse/FLINK-16850)
>>> 
>>> 
>>> [1] https://github.com/apache/flink/commits/master
>>> 
>>> Best
>>> Yun Tang
>>> 
>>> 
>>> From: Till Rohrmann 
>>> Sent: Thursday, April 2, 2020 23:11
>>> To: dev 
>>> Subject: Re: Configuring autolinks to Flink JIRA ticket in github repos
>>> 
>>> Nice, this is a cool feature. Thanks for asking INFRA for it.
>>> 
>>> Cheers,
>>> Till
>>> 
>>> On Wed, Apr 1, 2020 at 6:52 PM Yun Tang  wrote:
>>> 
 Hi community.
 
 I noticed that Github supports autolink reference recently [1]. This is
 helpful to allow developers could open Jira ticket link from pull
>>> requests
 title directly when accessing github repo.
 
 I have already created INFRA-20055 [2] to ask for configuration for
>> seven
 Flink related github repos. Hope it could be resolved soon 🙂
 
 
 [1]
 
>>> 
>> https://help.github.com/en/github/administering-a-repository/configuring-autolinks-to-reference-external-resources
 [2] https://issues.apache.org/jira/browse/INFRA-20055
 
 Best
 Yun Tang
 
>>> 
>> 

[jira] [Created] (FLINK-17068) ERROR at teardown of TableConfigTests.test_get_set_decimal_context

[Robert Metzger (Jira)]

Robert Metzger created FLINK-17068:
--

 Summary: ERROR at teardown of 
TableConfigTests.test_get_set_decimal_context
 Key: FLINK-17068
 URL: https://issues.apache.org/jira/browse/FLINK-17068
 Project: Flink
  Issue Type: Bug
  Components: API / Python, Tests
Affects Versions: 1.11.0
Reporter: Robert Metzger


CI run: 
https://dev.azure.com/rmetzger/Flink/_build/results?buildId=7243&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=14487301-07d2-5d56-5690-6dfab9ffd4d9

{code}
2020-04-09T00:34:15.9084299Z  ERRORS 

2020-04-09T00:34:15.9085728Z __ ERROR at teardown of 
TableConfigTests.test_get_set_decimal_context __
2020-04-09T00:34:15.9086216Z 
2020-04-09T00:34:15.9086725Z self = 
2020-04-09T00:34:15.9087144Z 
2020-04-09T00:34:15.9087457Z def __enter__(self):
2020-04-09T00:34:15.9087787Z try:
2020-04-09T00:34:15.9091929Z >   return next(self.gen)
2020-04-09T00:34:15.9092634Z E   OSError: [Errno 9] Bad file descriptor
2020-04-09T00:34:15.9092863Z 
2020-04-09T00:34:15.9093134Z 
dev/.conda/envs/3.5/lib/python3.5/contextlib.py:59: OSError
2020-04-09T00:34:15.9093548Z __ ERROR at setup of 
TableConfigTests.test_get_set_idle_state_retention_time ___
2020-04-09T00:34:15.9093803Z 
2020-04-09T00:34:15.9094082Z self = 
2020-04-09T00:34:15.9094313Z 
2020-04-09T00:34:15.9094502Z def __enter__(self):
2020-04-09T00:34:15.9094862Z try:
2020-04-09T00:34:15.9095088Z >   return next(self.gen)
2020-04-09T00:34:15.9095707Z E   OSError: [Errno 9] Bad file descriptor
2020-04-09T00:34:15.9095913Z 
2020-04-09T00:34:15.9096203Z 
dev/.conda/envs/3.5/lib/python3.5/contextlib.py:59: OSError
2020-04-09T00:34:15.9096818Z _ ERROR at teardown of 
TableConfigTests.test_get_set_idle_state_retention_time _
2020-04-09T00:34:15.9100686Z 
2020-04-09T00:34:15.9101687Z self = 
2020-04-09T00:34:15.9102005Z 
2020-04-09T00:34:15.9102193Z def __enter__(self):
2020-04-09T00:34:15.9102415Z try:
2020-04-09T00:34:15.9102741Z >   return next(self.gen)
2020-04-09T00:34:15.9103144Z E   OSError: [Errno 9] Bad file descriptor
2020-04-09T00:34:15.9103367Z 
2020-04-09T00:34:15.9103786Z 
dev/.conda/envs/3.5/lib/python3.5/contextlib.py:59: OSError
2020-04-09T00:34:15.9104185Z  ERROR at setup of 
TableConfigTests.test_get_set_local_timezone 
2020-04-09T00:34:15.9104999Z 
2020-04-09T00:34:15.9105287Z self = 
2020-04-09T00:34:15.9105531Z 
2020-04-09T00:34:15.9105707Z def __enter__(self):
2020-04-09T00:34:15.9105924Z try:
2020-04-09T00:34:15.9106138Z >   return next(self.gen)
2020-04-09T00:34:15.9106555Z E   OSError: [Errno 9] Bad file descriptor
2020-04-09T00:34:15.9106858Z 
2020-04-09T00:34:15.9107159Z 
dev/.conda/envs/3.5/lib/python3.5/contextlib.py:59: OSError
2020-04-09T00:34:15.9107675Z __ ERROR at teardown of 
TableConfigTests.test_get_set_local_timezone ___
2020-04-09T00:34:15.9107983Z 
2020-04-09T00:34:15.9108350Z self = 
2020-04-09T00:34:15.9108699Z 
2020-04-09T00:34:15.9108983Z def __enter__(self):
2020-04-09T00:34:15.9109311Z try:
2020-04-09T00:34:15.9109566Z >   return next(self.gen)
2020-04-09T00:34:15.9109872Z E   OSError: [Errno 9] Bad file descriptor
2020-04-09T00:34:15.9110082Z 
2020-04-09T00:34:15.9110349Z 
dev/.conda/envs/3.5/lib/python3.5/contextlib.py:59: OSError
2020-04-09T00:34:15.9111098Z __ ERROR at setup of 
TableConfigTests.test_get_set_max_generated_code_length ___
2020-04-09T00:34:15.9111479Z 
2020-04-09T00:34:15.9111740Z self = 
2020-04-09T00:34:15.9112010Z 
2020-04-09T00:34:15.9112297Z def __enter__(self):
2020-04-09T00:34:15.9112571Z try:
2020-04-09T00:34:15.9112803Z >   return next(self.gen)
2020-04-09T00:34:15.9113114Z E   OSError: [Errno 9] Bad file descriptor
2020-04-09T00:34:15.9113353Z 
2020-04-09T00:34:15.9113737Z 
dev/.conda/envs/3.5/lib/python3.5/contextlib.py:59: OSError
2020-04-09T00:34:15.9114282Z _ ERROR at teardown of 
TableConfigTests.test_get_set_max_generated_code_length _
2020-04-09T00:34:15.9114652Z 
2020-04-09T00:34:15.9114929Z self = 
2020-04-09T00:34:15.9115169Z 
2020-04-09T00:34:15.9115460Z def __enter__(self):
2020-04-09T00:34:15.9115756Z try:
2020-04-09T00:34:15.9115989Z >   return next(self.gen)
2020-04-09T00:34:15.9116279Z E   OSError: [Errno 9] Bad file descriptor
2020-04-09T00:34:15.9116579Z 
2020-04-09T00:34:15.9116944Z 
dev/.conda/envs/3.5/lib/python3.5/contextlib.py:59: OSError
2020-04-09T00:34:15.9117387Z __ ERROR at setup of 
TableConfigTests.test_get_set_null_check __
2020-04-09T00:34:15.9117640Z 
2020-04-09T00:34:15.9117908Z self = 
2020-04-09T00:34:15.9118258Z 
2020-04-09T00:34:15.9118519Z def __enter__(self):
2020-04-09T00:34:15.9118827Z try:
2020-04-0

[jira] [Created] (FLINK-17069) DefaultClusterClientServiceLoader should provide more information in case of missing dependencies

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-17069:


 Summary: DefaultClusterClientServiceLoader should provide more 
information in case of missing dependencies
 Key: FLINK-17069
 URL: https://issues.apache.org/jira/browse/FLINK-17069
 Project: Flink
  Issue Type: Improvement
  Components: Command Line Client
Affects Versions: 1.10.0
Reporter: Chesnay Schepler


If the loading of a {{ClusterClientFactory}} fails with a 
{{NoClassDefFoundError}} all we print is {{"Could not load factory due to 
missing dependencies."}}.

This just happened to me and I have no idea what is going one.
Which factory did it try?
What was missing?
Is this a problem for my particular submission?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Configuring autolinks to Flink JIRA ticket in github repos

[Aljoscha Krettek]

That is very nice! Thanks for taking care of this ~3q

On 09.04.20 11:08, Dian Fu wrote:

Cool! Thanks Yun for this effort. Very useful feature.

Regards,
Dian


在 2020年4月9日，下午4:32，Yu Li  写道：

Great! Thanks for the efforts Yun.

Best Regards,
Yu


On Thu, 9 Apr 2020 at 16:15, Jark Wu  wrote:


Thanks Yun,

This's a great feature! I was surprised by the autolink feature yesterday
(didn't know your work at that time).

Best,
Jark

On Thu, 9 Apr 2020 at 16:12, Yun Tang  wrote:


Hi community

The autolink to Flink JIRA ticket has taken effect. You could refer to

the

commit details page[1] to see all Flink JIRA titles within commits has

the

hyper link underline. Moreover, you don't need to use markdown language

to

create hyper link to Flink JIRA ticket when discussing in the pull
requests. e.g FLINK-16850 could point to the link instead of

[FLINK-16850](

https://issues.apache.org/jira/browse/FLINK-16850)


[1] https://github.com/apache/flink/commits/master

Best
Yun Tang


From: Till Rohrmann 
Sent: Thursday, April 2, 2020 23:11
To: dev 
Subject: Re: Configuring autolinks to Flink JIRA ticket in github repos

Nice, this is a cool feature. Thanks for asking INFRA for it.

Cheers,
Till

On Wed, Apr 1, 2020 at 6:52 PM Yun Tang  wrote:


Hi community.

I noticed that Github supports autolink reference recently [1]. This is
helpful to allow developers could open Jira ticket link from pull

requests

title directly when accessing github repo.

I have already created INFRA-20055 [2] to ask for configuration for

seven

Flink related github repos. Hope it could be resolved soon 🙂


[1]




https://help.github.com/en/github/administering-a-repository/configuring-autolinks-to-reference-external-resources

[2] https://issues.apache.org/jira/browse/INFRA-20055

Best
Yun Tang

Re: Configuring autolinks to Flink JIRA ticket in github repos

[Hequn Cheng]

It’s much more convenient now. Thanks you!

> On Apr 9, 2020, at 8:01 PM, Aljoscha Krettek  wrote:
> 
> That is very nice! Thanks for taking care of this ~3q
> 
> On 09.04.20 11:08, Dian Fu wrote:
>> Cool! Thanks Yun for this effort. Very useful feature.
>> Regards,
>> Dian
>>> 在 2020年4月9日，下午4:32，Yu Li  写道：
>>> 
>>> Great! Thanks for the efforts Yun.
>>> 
>>> Best Regards,
>>> Yu
>>> 
>>> 
>>> On Thu, 9 Apr 2020 at 16:15, Jark Wu  wrote:
>>> 
 Thanks Yun,
 
 This's a great feature! I was surprised by the autolink feature yesterday
 (didn't know your work at that time).
 
 Best,
 Jark
 
 On Thu, 9 Apr 2020 at 16:12, Yun Tang  wrote:
 
> Hi community
> 
> The autolink to Flink JIRA ticket has taken effect. You could refer to
 the
> commit details page[1] to see all Flink JIRA titles within commits has
 the
> hyper link underline. Moreover, you don't need to use markdown language
 to
> create hyper link to Flink JIRA ticket when discussing in the pull
> requests. e.g FLINK-16850 could point to the link instead of
 [FLINK-16850](
> https://issues.apache.org/jira/browse/FLINK-16850)
> 
> 
> [1] https://github.com/apache/flink/commits/master
> 
> Best
> Yun Tang
> 
> 
> From: Till Rohrmann 
> Sent: Thursday, April 2, 2020 23:11
> To: dev 
> Subject: Re: Configuring autolinks to Flink JIRA ticket in github repos
> 
> Nice, this is a cool feature. Thanks for asking INFRA for it.
> 
> Cheers,
> Till
> 
> On Wed, Apr 1, 2020 at 6:52 PM Yun Tang  wrote:
> 
>> Hi community.
>> 
>> I noticed that Github supports autolink reference recently [1]. This is
>> helpful to allow developers could open Jira ticket link from pull
> requests
>> title directly when accessing github repo.
>> 
>> I have already created INFRA-20055 [2] to ask for configuration for
 seven
>> Flink related github repos. Hope it could be resolved soon 🙂
>> 
>> 
>> [1]
>> 
> 
 https://help.github.com/en/github/administering-a-repository/configuring-autolinks-to-reference-external-resources
>> [2] https://issues.apache.org/jira/browse/INFRA-20055
>> 
>> Best
>> Yun Tang
>> 
> 
 
> 

Re: Configuring autolinks to Flink JIRA ticket in github repos

[Xingbo Huang]

Thanks Yun,

Good Work. It is very convenient to link to JIRA from corresponding PR
currently.

Best,
Xingbo

Hequn Cheng  于2020年4月9日周四 下午8:16写道：

> It’s much more convenient now. Thanks you!
>
> > On Apr 9, 2020, at 8:01 PM, Aljoscha Krettek 
> wrote:
> >
> > That is very nice! Thanks for taking care of this ~3q
> >
> > On 09.04.20 11:08, Dian Fu wrote:
> >> Cool! Thanks Yun for this effort. Very useful feature.
> >> Regards,
> >> Dian
> >>> 在 2020年4月9日，下午4:32，Yu Li  写道：
> >>>
> >>> Great! Thanks for the efforts Yun.
> >>>
> >>> Best Regards,
> >>> Yu
> >>>
> >>>
> >>> On Thu, 9 Apr 2020 at 16:15, Jark Wu  wrote:
> >>>
>  Thanks Yun,
> 
>  This's a great feature! I was surprised by the autolink feature
> yesterday
>  (didn't know your work at that time).
> 
>  Best,
>  Jark
> 
>  On Thu, 9 Apr 2020 at 16:12, Yun Tang  wrote:
> 
> > Hi community
> >
> > The autolink to Flink JIRA ticket has taken effect. You could refer
> to
>  the
> > commit details page[1] to see all Flink JIRA titles within commits
> has
>  the
> > hyper link underline. Moreover, you don't need to use markdown
> language
>  to
> > create hyper link to Flink JIRA ticket when discussing in the pull
> > requests. e.g FLINK-16850 could point to the link instead of
>  [FLINK-16850](
> > https://issues.apache.org/jira/browse/FLINK-16850)
> >
> >
> > [1] https://github.com/apache/flink/commits/master
> >
> > Best
> > Yun Tang
> >
> > 
> > From: Till Rohrmann 
> > Sent: Thursday, April 2, 2020 23:11
> > To: dev 
> > Subject: Re: Configuring autolinks to Flink JIRA ticket in github
> repos
> >
> > Nice, this is a cool feature. Thanks for asking INFRA for it.
> >
> > Cheers,
> > Till
> >
> > On Wed, Apr 1, 2020 at 6:52 PM Yun Tang  wrote:
> >
> >> Hi community.
> >>
> >> I noticed that Github supports autolink reference recently [1].
> This is
> >> helpful to allow developers could open Jira ticket link from pull
> > requests
> >> title directly when accessing github repo.
> >>
> >> I have already created INFRA-20055 [2] to ask for configuration for
>  seven
> >> Flink related github repos. Hope it could be resolved soon 🙂
> >>
> >>
> >> [1]
> >>
> >
> 
> https://help.github.com/en/github/administering-a-repository/configuring-autolinks-to-reference-external-resources
> >> [2] https://issues.apache.org/jira/browse/INFRA-20055
> >>
> >> Best
> >> Yun Tang
> >>
> >
> 
> >
>
>

Re: [VOTE] FLIP-120: Support conversion between PyFlink Table and Pandas DataFrame

[Xingbo Huang]

Hi Dian,

+1 (non-binding)
Thanks a lot for driving this.

Best,
Xingbo

Dian Fu  于2020年4月8日周三 上午10:03写道：

> Hi all,
>
> I'd like to start the vote for FLIP-120[1] which is discussed and reached
> consensus in the discussion thread[2].
>
> The vote will be open for at least 72 hours unless there is an objection
> or we have not received sufficient votes.
>
> Regards,
> Dian
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-120%3A+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-120:+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
> >
> [2]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-120-Support-conversion-between-PyFlink-Table-and-Pandas-DataFrame-tt39611.html
> <
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-120-Support-conversion-between-PyFlink-Table-and-Pandas-DataFrame-tt39611.html
> >

[jira] [Created] (FLINK-17070) Kubernetes session CLI ignores stop/help command in detached mode

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-17070:


 Summary: Kubernetes session CLI ignores stop/help command in 
detached mode
 Key: FLINK-17070
 URL: https://issues.apache.org/jira/browse/FLINK-17070
 Project: Flink
  Issue Type: Bug
  Components: Command Line Client, Deployment / Kubernetes
Affects Versions: 1.11.0
Reporter: Chesnay Schepler


The CLI experience for the KubernetesSessionCli is far from ideal.

Getting the help information should be easier than
{code}
$ echo 'help' | ./bin/kubernetes-session.sh -Dexecution.attached=true
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17071) Kubernetes session CLI logging output is either misleading or concerning

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-17071:


 Summary: Kubernetes session CLI logging output is either 
misleading or concerning
 Key: FLINK-17071
 URL: https://issues.apache.org/jira/browse/FLINK-17071
 Project: Flink
  Issue Type: Bug
  Components: Command Line Client, Deployment / Kubernetes
Affects Versions: 1.11.0
Reporter: Chesnay Schepler


When running any command against the KubernetesSessionCLI it prints a log 
message about having created a session cluster.

This should certainly not appear when running a stop/help command.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] FLIP-71: E2E View support in Flink SQL

[Zhenghua Gao]

Hi all,

I'd like to start the vote for FLIP-71[1] which adds E2E view support in
Flink SQL.
This FLIP is discussed in the thread[2].

The vote will be open for at least 72 hours. Unless there is an objection.
I will try to
close it by April 13, 2020 09:00 UTC if we have received sufficient votes.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-71%3A+E2E+View+support+in+FLINK+SQL

[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-71-E2E-View-support-in-Flink-SQL-td33131.html#a39787

*Best Regards,*
*Zhenghua Gao*

Re: [ANNOUNCE] Apache Flink Stateful Functions 2.0.0 released

[Zhijiang]

Great work! Thanks Gordon for the continuous efforts for enhancing stateful 
functions and the efficient release!
Wish stateful functions becoming more and more popular in users.

Best,
Zhijiang


--
From:Yun Tang 
Send Time:2020 Apr. 9 (Thu.) 00:17
To:Till Rohrmann ; dev 
Cc:Oytun Tez ; user 
Subject:Re: [ANNOUNCE] Apache Flink Stateful Functions 2.0.0 released

Excited to see the stateful functions release!
Thanks for the great work of manager Gordon and everyone who ever contributed 
to this.

Best
Yun Tang

From: Till Rohrmann 
Sent: Wednesday, April 8, 2020 14:30
To: dev 
Cc: Oytun Tez ; user 
Subject: Re: [ANNOUNCE] Apache Flink Stateful Functions 2.0.0 released

Great news! Thanks a lot for being our release manager Gordon and to everyone 
who helped with the release.

Cheers,
Till

On Wed, Apr 8, 2020 at 3:57 AM Congxian Qiu 
mailto:qcx978132...@gmail.com>> wrote:
Thanks a lot for the release and your great job, Gordon!
Also thanks to everyone who made this release possible!

Best,
Congxian


Oytun Tez mailto:oy...@motaword.com>> 于2020年4月8日周三 上午2:55写道：

> I should also add, I couldn't agree more with this sentence in the release
> article: "state access/updates and messaging need to be integrated."
>
> This is something we strictly enforce in our Flink case, where we do not
> refer to anything external for storage, use Flink as our DB.
>
>
>
>  --
>
> [image: MotaWord]
> Oytun Tez
> M O T A W O R D | CTO & Co-Founder
> oy...@motaword.com
>
>   
>
>
> On Tue, Apr 7, 2020 at 12:26 PM Oytun Tez 
> mailto:oy...@motaword.com>> wrote:
>
>> Great news! Thank you all.
>>
>> On Tue, Apr 7, 2020 at 12:23 PM Marta Paes Moreira 
>> mailto:ma...@ververica.com>>
>> wrote:
>>
>>> Thank you for managing the release, Gordon — you did a tremendous job!
>>> And to everyone else who worked on pushing it through.
>>>
>>> Really excited about the new use cases that StateFun 2.0 unlocks for
>>> Flink users and beyond!
>>>
>>>
>>> Marta
>>>
>>> On Tue, Apr 7, 2020 at 4:47 PM Hequn Cheng 
>>> mailto:he...@apache.org>> wrote:
>>>
 Thanks a lot for the release and your great job, Gordon!
 Also thanks to everyone who made this release possible!

 Best,
 Hequn

 On Tue, Apr 7, 2020 at 8:58 PM Tzu-Li (Gordon) Tai 
 mailto:tzuli...@apache.org>>
 wrote:

> The Apache Flink community is very happy to announce the release of
> Apache Flink Stateful Functions 2.0.0.
>
> Stateful Functions is an API that simplifies building distributed
> stateful applications.
> It's based on functions with persistent state that can interact
> dynamically with strong consistency guarantees.
>
> Please check out the release blog post for an overview of the release:
> https://flink.apache.org/news/2020/04/07/release-statefun-2.0.0.html
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Maven artifacts for Stateful Functions can be found at:
> https://search.maven.org/search?q=g:org.apache.flink%20statefun
>
> Python SDK for Stateful Functions published to the PyPI index can be
> found at:
> https://pypi.org/project/apache-flink-statefun/
>
> Official Docker image for building Stateful Functions applications is
> currently being published to Docker Hub.
> Dockerfiles for this release can be found at:
> https://github.com/apache/flink-statefun-docker/tree/master/2.0.0
> Progress for creating the Docker Hub repository can be tracked at:
> https://github.com/docker-library/official-images/pull/7749
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346878
>
> We would like to thank all contributors of the Apache Flink community
> who made this release possible!
>
> Cheers,
> Gordon
>
 --
>>  --
>>
>> [image: MotaWord]
>> Oytun Tez
>> M O T A W O R D | CTO & Co-Founder
>> oy...@motaword.com
>>
>>   
>>
>

Re: Configuring autolinks to Flink JIRA ticket in github repos

[Zhijiang]

Very nice work! Thanks Yun finding this feature and making it happen!

Best,
Zhijiang


--
From:Xingbo Huang 
Send Time:2020 Apr. 9 (Thu.) 20:23
To:dev 
Subject:Re: Configuring autolinks to Flink JIRA ticket in github repos

Thanks Yun,

Good Work. It is very convenient to link to JIRA from corresponding PR
currently.

Best,
Xingbo

Hequn Cheng  于2020年4月9日周四 下午8:16写道：

> It’s much more convenient now. Thanks you!
>
> > On Apr 9, 2020, at 8:01 PM, Aljoscha Krettek 
> wrote:
> >
> > That is very nice! Thanks for taking care of this ~3q
> >
> > On 09.04.20 11:08, Dian Fu wrote:
> >> Cool! Thanks Yun for this effort. Very useful feature.
> >> Regards,
> >> Dian
> >>> 在 2020年4月9日，下午4:32，Yu Li  写道：
> >>>
> >>> Great! Thanks for the efforts Yun.
> >>>
> >>> Best Regards,
> >>> Yu
> >>>
> >>>
> >>> On Thu, 9 Apr 2020 at 16:15, Jark Wu  wrote:
> >>>
>  Thanks Yun,
> 
>  This's a great feature! I was surprised by the autolink feature
> yesterday
>  (didn't know your work at that time).
> 
>  Best,
>  Jark
> 
>  On Thu, 9 Apr 2020 at 16:12, Yun Tang  wrote:
> 
> > Hi community
> >
> > The autolink to Flink JIRA ticket has taken effect. You could refer
> to
>  the
> > commit details page[1] to see all Flink JIRA titles within commits
> has
>  the
> > hyper link underline. Moreover, you don't need to use markdown
> language
>  to
> > create hyper link to Flink JIRA ticket when discussing in the pull
> > requests. e.g FLINK-16850 could point to the link instead of
>  [FLINK-16850](
> > https://issues.apache.org/jira/browse/FLINK-16850)
> >
> >
> > [1] https://github.com/apache/flink/commits/master
> >
> > Best
> > Yun Tang
> >
> > 
> > From: Till Rohrmann 
> > Sent: Thursday, April 2, 2020 23:11
> > To: dev 
> > Subject: Re: Configuring autolinks to Flink JIRA ticket in github
> repos
> >
> > Nice, this is a cool feature. Thanks for asking INFRA for it.
> >
> > Cheers,
> > Till
> >
> > On Wed, Apr 1, 2020 at 6:52 PM Yun Tang  wrote:
> >
> >> Hi community.
> >>
> >> I noticed that Github supports autolink reference recently [1].
> This is
> >> helpful to allow developers could open Jira ticket link from pull
> > requests
> >> title directly when accessing github repo.
> >>
> >> I have already created INFRA-20055 [2] to ask for configuration for
>  seven
> >> Flink related github repos. Hope it could be resolved soon 🙂
> >>
> >>
> >> [1]
> >>
> >
> 
> https://help.github.com/en/github/administering-a-repository/configuring-autolinks-to-reference-external-resources
> >> [2] https://issues.apache.org/jira/browse/INFRA-20055
> >>
> >> Best
> >> Yun Tang
> >>
> >
> 
> >
>
>

Re: [VOTE] FLIP-71: E2E View support in Flink SQL

[Timo Walther]

+1 (binding)

Thanks for your efforts.

Regards,
Timo


On 09.04.20 14:46, Zhenghua Gao wrote:

Hi all,

I'd like to start the vote for FLIP-71[1] which adds E2E view support in
Flink SQL.
This FLIP is discussed in the thread[2].

The vote will be open for at least 72 hours. Unless there is an objection.
I will try to
close it by April 13, 2020 09:00 UTC if we have received sufficient votes.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-71%3A+E2E+View+support+in+FLINK+SQL

[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-71-E2E-View-support-in-Flink-SQL-td33131.html#a39787

*Best Regards,*
*Zhenghua Gao*

Re: contributor apply

[Timo Walther]

Hi,

welcome to our community! You don't need to be a contributor in JIRA to 
contribute to Apache Flink. Just ping a committer if your want to work 
on some JIRA ticket and someone will assign the ticket to you.


Regards,
Timo

On 09.04.20 11:03, zhangzhanhua wrote:

Hi Guys,

I want to contribute to Apache Flink.
Would you please give me the permission as a contributor?
My JIRA ID is zhangzhanhua.

Re: [DISCUSS] Creating a new repo to host Flink benchmarks

[Piotr Nowojski]

Hi Yun Tang,

Thanks for proposing the idea. Since we can not include benchmarks in the Flink 
repository what you are proposing is the second best option.

+1 from my side for the proposal.

I think benchmarks have proven their value to justify this.

Piotrek

> On 9 Apr 2020, at 08:56, Yun Tang  wrote:
> 
> Hi Flink devs,
> 
> As Flink develops rapidly with more and more features added, how to ensure no 
> performance regression existed has become more and more important. And we 
> would like to create a new repo under apache project to host previous 
> flink-benchmarks [1] repo, which is inspired when we discuss under 
> FLINK-16850 [2]
> 
> Some background context on flink-benchmarks, for those who are not familiar 
> with the project yet:
> 
> - Current flink-benchmarks does not align with the Flink release, which lead 
> developers not easy to verify
>   performance at specific Flink version because current flink-benchmarks 
> always depends on the latest interfaces.
> - Above problem could be solved well if we could ensure flink-benchmarks also 
> create release branch when we
>   releasing Flink. However, current flink-benchmarks repo is hosted under 
> dataArtisans (the former name of
>   ververica) project, which is not involved in Flink release manual [3]. We 
> propose to promote this repo under
>   apache project so that release manager could have the right to release on 
> flink-benchmarks.
> - The reason why we not involve flink-benchmarks into the apache/flink repo 
> is because it heavily depends on
>   JMH [4], which is under GPLv2 license.
> 
> What do you think?
> 
> Best,
> Yun Tang
> 
> [1] https://github.com/dataArtisans/flink-benchmarks
> [2] https://issues.apache.org/jira/browse/FLINK-16850
> [3] https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release
> [4] https://openjdk.java.net/projects/code-tools/jmh/
> 
> 

[jira] [Created] (FLINK-17072) Let Dispatcher and ResourceManager pick random endpoint id

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-17072:
-

 Summary: Let Dispatcher and ResourceManager pick random endpoint id
 Key: FLINK-17072
 URL: https://issues.apache.org/jira/browse/FLINK-17072
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
 Fix For: 1.11.0


As it is recommended by Akka, actors should not reuse the same actor name of a 
previous instance. The problem is that one can receive messages which were 
intended for the previous instance. Hence, I propose to let the 
{{ResourceManager}} and the {{Dispatcher}} pick a random name.

A nice side effect is that the {{TestingMiniCluster}} needs no longer to use 
specialized factories for these two components in order to avoid name clashes 
when starting stand-by RM and {{Dispatchers}}. Moreover, it simplifies the 
setup of these components.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17073) Slow checkpoint cleanup causing OOMs

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-17073:
-

 Summary: Slow checkpoint cleanup causing OOMs
 Key: FLINK-17073
 URL: https://issues.apache.org/jira/browse/FLINK-17073
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing, Runtime / Coordination
Affects Versions: 1.10.0, 1.9.0, 1.8.0, 1.7.3, 1.11.0
Reporter: Till Rohrmann
 Fix For: 1.9.3, 1.10.1, 1.11.0


A user reported that he sees a decline in checkpoint cleanup speed when 
upgrading from Flink 1.7.2 to 1.10.0. The result is that a lot of cleanup tasks 
are waiting in the execution queue occupying memory. Ultimately, the JM process 
dies with an OOM.

Compared to Flink 1.7.2, we introduced a dedicated {{ioExecutor}} which is used 
by the {{HighAvailabilityServices}}. Before, we use the {{AkkaRpcService}} 
thread pool which was a {{ForkJoinPool}} with a max parallelism of 64. Now it 
is a {{FixedThreadPool}} with as many threads as CPU cores. This change might 
have caused the decline in completed checkpoint discard throughput. This 
suspicion needs to be validated before trying to fix it!

[1] 
https://lists.apache.org/thread.html/r390e5d775878918edca0b6c9f18de96f828c266a888e34ed30ce8494%40%3Cuser.flink.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-123: DDL and DML compatibility for Hive connector

[Jingsong Li]

Thanks Rui for diving.

+1 for this proposal.

There are still lots of people who love Hive SQL.
And I have seen some people support HQL on presto. Presto, as a famous
computing engine, also supports ANSI SQL as we do. This is quite different
from HQL.

Do you think we must need import `FlinkHiveSqlParserImpl`? This will bother
planner code, if possible, I think it is better to keep dialect things in
sql-parer.
What do you think?

Best,
Jingsong Lee

On Thu, Apr 9, 2020 at 3:56 PM Rui Li  wrote:

> Hi Kurt,
>
> Thanks for the feedback and that's a good idea. I have updated the FLIP and
> added tables in the "Limited Scope" section to list the supported and
> unsupported features. Please have a look and let me know if that makes
> sense. Thanks.
>
> On Wed, Apr 8, 2020 at 2:19 PM Kurt Young  wrote:
>
> > Hi Rui,
> >
> > Thanks for bringing up this discussion and it makes sense to me though i
> > have one comment about the FLIP.
> > There are a few places in the design document saying some features will
> not
> > be supported or not included in
> > this FLIP, but I don't get what will be supported exactly. I can imagine
> > other users will also have such confusion.
> > Could you add a table or a list of syntax which will be supported?
> >
> > Best,
> > Kurt
> >
> >
> > On Wed, Apr 1, 2020 at 4:24 PM Rui Li  wrote:
> >
> > > Hi devs,
> > >
> > > I'd like to start a discussion about bringing DDL & DML compatibility
> for
> > > Hive connector. The proposal mainly aims to implement more DDLs for
> Hive
> > > connector and allow users to write Hive syntax when using the Hive
> > dialect.
> > > Hopefully this will make it easier for users to migrate to Flink, with
> > > fewer SQL statements that need to be changed.
> > >
> > > Please find more details in the FLIP wiki [1]. Feedbacks and
> suggestions
> > > are appreciated.
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-123%3A+DDL+and+DML+compatibility+for+Hive+connector
> > >
> > > --
> > > Cheers,
> > > Rui Li
> > >
> >
>
>
> --
> Best regards!
> Rui Li
>


-- 
Best, Jingsong Lee

Re: [VOTE] FLIP-71: E2E View support in Flink SQL

[Jingsong Li]

+1

Hope we can finish it in 1.11, we have been looking forward to it for a
long time.

Best,
Jingsong Lee

On Thu, Apr 9, 2020 at 9:23 PM Timo Walther  wrote:

> +1 (binding)
>
> Thanks for your efforts.
>
> Regards,
> Timo
>
>
> On 09.04.20 14:46, Zhenghua Gao wrote:
> > Hi all,
> >
> > I'd like to start the vote for FLIP-71[1] which adds E2E view support in
> > Flink SQL.
> > This FLIP is discussed in the thread[2].
> >
> > The vote will be open for at least 72 hours. Unless there is an
> objection.
> > I will try to
> > close it by April 13, 2020 09:00 UTC if we have received sufficient
> votes.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-71%3A+E2E+View+support+in+FLINK+SQL
> >
> > [2]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-71-E2E-View-support-in-Flink-SQL-td33131.html#a39787
> >
> > *Best Regards,*
> > *Zhenghua Gao*
> >
>
>

-- 
Best, Jingsong Lee

[jira] [Created] (FLINK-17074) Deprecate DataStream.keyBy() that use tuple/expression keys

[Aljoscha Krettek (Jira)]

Aljoscha Krettek created FLINK-17074:


 Summary: Deprecate DataStream.keyBy() that use tuple/expression 
keys
 Key: FLINK-17074
 URL: https://issues.apache.org/jira/browse/FLINK-17074
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream
 Environment: Currently you can either specify a {{KeySelector}}, tuple 
positions, and expression keys? I think {{KeySelectors}} are strictly superior 
and with lambdas (or function references) quite easy to use.  Tuple/expression 
keys use reflection underneath to do the field accesses, so performance is 
strictly worse. Also, when using a {{KeySelector}} you will have a meaningful 
key type {{KEY}} in your operations while for tuple/expression keys the key 
type is simply {{Tuple}}.

Tuple/expression keys were introduced before Java got support for lambdas in 
Java 8 and before we added the Table API. Nowadays, using a lambda is little 
more typing than using an expression key but is (possibly) faster and more type 
safe. The Table API should be used for these more expression-based/relational 
use cases.
Reporter: Aljoscha Krettek






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-111: Docker image unification

[Ismaël Mejía]

+1 (non-binding)
Great work Andrey, pretty excited about this happening!


On Wed, Apr 8, 2020 at 4:20 AM Canbin Zheng  wrote:
>
> Thanks for the FLIP Andrey.
>
> +1 (non-binding) from my side.
>
> Regards,
> Canbin Zheng
>
> Yang Wang  于2020年4月8日周三 上午9:57写道：
>
> > Thanks Andrey for the efforts of docker image unification.
> >
> > +1 (non-binding)
> >
> > Best,
> > Yang
> >
> > Till Rohrmann  于2020年4月7日周二 下午11:04写道：
> >
> > > Thanks for driving this effort Andrey.
> > >
> > > +1 (binding)
> > >
> > > Cheers,
> > > Till
> > >
> > > On Tue, Apr 7, 2020 at 4:48 PM Ufuk Celebi  wrote:
> > >
> > > > Thanks for starting this FLIP.
> > > >
> > > > +1
> > > >
> > > > On Tue, Apr 7, 2020 at 11:29 AM Andrey Zagrebin 
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > As discussed in these threads [1] and [2],
> > > > > we suggest to unify the docker topic in Flink for users [3].
> > > > >
> > > > > This mainly means refactoring of the existing code and introducing
> > more
> > > > > docs as a first step.
> > > > > The effort should enable further improvements and follow-ups for the
> > > > docker
> > > > > integration with Flink.
> > > > >
> > > > > The integration with docker in Flink is currently addressed in many
> > > > places
> > > > > which often intersect, repeat each other or apply different
> > approaches.
> > > > > This makes it really hard to follow the whole topic for users and
> > > > > maintainers. This FLIP suggests how to unify this topic. It means
> > > having
> > > > > one place which has the *Dockerfile*, all necessary scripts and docs
> > > > > following each other in a consistent way without any repetitions or
> > > > > contradictions.
> > > > >
> > > > > The idea is to keep all docker related resources in
> > apache/flink-docker
> > > > > . It already has a detailed
> > > > > Dockerfile which is well suited for common use cases or at least
> > serves
> > > > as
> > > > > a good starting point. The suggestion is to make it extensible for
> > > other
> > > > > concerns which are currently addressed in other places.
> > > > >
> > > > > Best,
> > > > > Andrey
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-111-Docker-image-unification-td38444.html#a39822
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-111-Docker-image-unification-td38444i20.html#a39950
> > > > > [3]
> > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification
> > > > >
> > > >
> > >
> >

Re: [DISCUSS] FLIP-111: Docker image unification

[Ismaël Mejía]

I am coming extremely late to this discussion since the vote already started but
it is great we are finally getting into unification Enthusiast +1. Kudos to
Andrey and the rest of the community for bringing all the useful and different
perspectives.

I just want to bring information on two tickets that were created on parallel to
the discussion that were mostly migrated from the old docker-flink repo that
both Patrick and me have been maintaining for the last 3 years (now repatriated
into the flink-docker repo):

FLINK-16260 Add docker images based on Java 11 (PR ready)
FLINK-16846 Add python docker images

Both are related to the current discussion. The one Java 11 address just the
users wishes. We can see this as a good way to validate our support and offer it
to users but of course this should not be the default image that FLIP-111 will
be based on until the community agrees on it (probably in the future).

The python one definitely deserves more discussion. Today the official Flink
docker images do not contain python so users must extend them to contain python.
Since there are so many nice improvements on Flink for Python, is maybe the time
to release images with python support? This of course brings other questions of
which versions to support and how are we going to test them. So maybe we should
open a specific thread or FLIP on that once FLIP-111 is done.

On Wed, Apr 8, 2020 at 4:35 AM Canbin Zheng  wrote:
>
> Hi, all,
>
> Thanks for the reply, Andrey!
>
> I have filed two new tickets tracking the problems:
> 1. FLINK-17033  for
> upgrading base Java Docker image, I pointed out some other problems
> the openjdk:8-jre-alpine could have in the ticket‘s description.
> 2. FLINK-17034  for
> suggesting executing the container CMD under TINI.
>
> Regards,
> Canbin Zheng
>
> Andrey Zagrebin  于2020年4月7日周二 下午4:58写道：
>
> > Hi all,
> >
> > Thanks for the further feedback Niels and Canbin.
> >
> > @Niels
> >
> > I agree with Till, the comments about docker tags are valid concerns and we
> > can discuss them in dedicated ML threads
> > in parallel or after the general unification of Dockerfiles suggested by
> > this FLIP.
> >
> > One thing to add about point 4. The native Kubernetes integration does not
> > support a job mode at the moment.
> > This is not only about the image. As I understand, even if you pack the job
> > artefacts into the image, the native Kubernetes integration will start a
> > session cluster.
> > This will be a follow-up for the native Kubernetes integration.
> > cc @Yang Wang
> >
> > @Canbin
> >
> > I think you raise valid concerns. It makes sense to create JIRA issues for
> > them.
> > One for the alpine image problem and one to suggest the TINI as a blocker
> > for FLINK-15843  and
> > slow pod shutdown.
> > We can discuss and address them in parallel or after the general
> > unification of Dockerfiles suggested by this FLIP.
> >
> > I will start a separate voting thread for this FLIP.
> >
> > Cheers,
> > Andrey
> >
> >
> > On Mon, Apr 6, 2020 at 5:49 PM Canbin Zheng 
> > wrote:
> >
> > > Hi, all
> > >
> > > Thanks a lot for this FLIP and all the fruitable discussion. I am not
> > sure
> > > whether the following questions are in the scope of this FLIP, but I
> > still
> > > expect your reply:
> > >
> > >1. Which docker base image do we plan to use for Java? As far as I
> > >see, openjdk:8-jre-alpine[1] is not officially supported by the
> > OpenJDK
> > >project anymore; openjdk:8-jre is larger than openjdk:8-jre-slim in
> > size so
> > >that we use the latter one in our internal branch and it works fine
> > so far.
> > >2. Is it possible that we execute the container CMD under *TINI*[2]
> > >instead of the shell for better hygiene? As far as I see, the
> > container of
> > >the JM or TMs is running in the shell form and it could not receive
> > the
> > >*TERM* signal when the pod is deleted[3]. Some of the problems are as
> > >follows:
> > >   - The JM and the TMs could have no chance of cleanup, I used to
> > >   create FLINK-15843[4] for tracking this problem.
> > >   - The pod could take a long time(up to 40 seconds) to be deleted
> > >   after the K8s API Server receives the deletion request.
> > >
> > >At the moment, we use *TINI* in our internal branch for the
> > > native K8s setup and it solves the problems mentioned above.
> > >
> > > [1]
> > >
> > >
> > https://github.com/docker-library/docs/blob/master/openjdk/README.md#supported-tags-and-respective-dockerfile-links
> > >
> > >
> > https://github.com/docker-library/openjdk/commit/3eb0351b208d739fac35345c85e3c6237c2114ec#diff-f95ffa3d134732c33f7b8368e099
> > >  [2]
> > > https://github.com/krallin/tini
> > >  [3]
> > > https://docs.docker.com/engine/reference/commandline/kill/
> > >  [4]
> > > https://is

Re: Configuring autolinks to Flink JIRA ticket in github repos

[Till Rohrmann]

Thanks Yun. This is very helpful :-)

Cheers,
Till

On Thu, Apr 9, 2020 at 3:16 PM Zhijiang 
wrote:

> Very nice work! Thanks Yun finding this feature and making it happen!
>
> Best,
> Zhijiang
>
>
> --
> From:Xingbo Huang 
> Send Time:2020 Apr. 9 (Thu.) 20:23
> To:dev 
> Subject:Re: Configuring autolinks to Flink JIRA ticket in github repos
>
> Thanks Yun,
>
> Good Work. It is very convenient to link to JIRA from corresponding PR
> currently.
>
> Best,
> Xingbo
>
> Hequn Cheng  于2020年4月9日周四 下午8:16写道：
>
> > It’s much more convenient now. Thanks you!
> >
> > > On Apr 9, 2020, at 8:01 PM, Aljoscha Krettek 
> > wrote:
> > >
> > > That is very nice! Thanks for taking care of this ~3q
> > >
> > > On 09.04.20 11:08, Dian Fu wrote:
> > >> Cool! Thanks Yun for this effort. Very useful feature.
> > >> Regards,
> > >> Dian
> > >>> 在 2020年4月9日，下午4:32，Yu Li  写道：
> > >>>
> > >>> Great! Thanks for the efforts Yun.
> > >>>
> > >>> Best Regards,
> > >>> Yu
> > >>>
> > >>>
> > >>> On Thu, 9 Apr 2020 at 16:15, Jark Wu  wrote:
> > >>>
> >  Thanks Yun,
> > 
> >  This's a great feature! I was surprised by the autolink feature
> > yesterday
> >  (didn't know your work at that time).
> > 
> >  Best,
> >  Jark
> > 
> >  On Thu, 9 Apr 2020 at 16:12, Yun Tang  wrote:
> > 
> > > Hi community
> > >
> > > The autolink to Flink JIRA ticket has taken effect. You could refer
> > to
> >  the
> > > commit details page[1] to see all Flink JIRA titles within commits
> > has
> >  the
> > > hyper link underline. Moreover, you don't need to use markdown
> > language
> >  to
> > > create hyper link to Flink JIRA ticket when discussing in the pull
> > > requests. e.g FLINK-16850 could point to the link instead of
> >  [FLINK-16850](
> > > https://issues.apache.org/jira/browse/FLINK-16850)
> > >
> > >
> > > [1] https://github.com/apache/flink/commits/master
> > >
> > > Best
> > > Yun Tang
> > >
> > > 
> > > From: Till Rohrmann 
> > > Sent: Thursday, April 2, 2020 23:11
> > > To: dev 
> > > Subject: Re: Configuring autolinks to Flink JIRA ticket in github
> > repos
> > >
> > > Nice, this is a cool feature. Thanks for asking INFRA for it.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Wed, Apr 1, 2020 at 6:52 PM Yun Tang  wrote:
> > >
> > >> Hi community.
> > >>
> > >> I noticed that Github supports autolink reference recently [1].
> > This is
> > >> helpful to allow developers could open Jira ticket link from pull
> > > requests
> > >> title directly when accessing github repo.
> > >>
> > >> I have already created INFRA-20055 [2] to ask for configuration
> for
> >  seven
> > >> Flink related github repos. Hope it could be resolved soon 🙂
> > >>
> > >>
> > >> [1]
> > >>
> > >
> > 
> >
> https://help.github.com/en/github/administering-a-repository/configuring-autolinks-to-reference-external-resources
> > >> [2] https://issues.apache.org/jira/browse/INFRA-20055
> > >>
> > >> Best
> > >> Yun Tang
> > >>
> > >
> > 
> > >
> >
> >
>
>

Re: [DISCUSS] Creating a new repo to host Flink benchmarks

[Till Rohrmann]

I think it is a good idea to make the benchmarks available to the community
via a repo under the Apache project and to make updating it part of the
release process. Hence +1 for the proposal.

Cheers,
Till

On Thu, Apr 9, 2020 at 4:01 PM Piotr Nowojski  wrote:

> Hi Yun Tang,
>
> Thanks for proposing the idea. Since we can not include benchmarks in the
> Flink repository what you are proposing is the second best option.
>
> +1 from my side for the proposal.
>
> I think benchmarks have proven their value to justify this.
>
> Piotrek
>
> > On 9 Apr 2020, at 08:56, Yun Tang  wrote:
> >
> > Hi Flink devs,
> >
> > As Flink develops rapidly with more and more features added, how to
> ensure no performance regression existed has become more and more
> important. And we would like to create a new repo under apache project to
> host previous flink-benchmarks [1] repo, which is inspired when we discuss
> under FLINK-16850 [2]
> >
> > Some background context on flink-benchmarks, for those who are not
> familiar with the project yet:
> >
> > - Current flink-benchmarks does not align with the Flink release, which
> lead developers not easy to verify
> >   performance at specific Flink version because current flink-benchmarks
> always depends on the latest interfaces.
> > - Above problem could be solved well if we could ensure flink-benchmarks
> also create release branch when we
> >   releasing Flink. However, current flink-benchmarks repo is hosted
> under dataArtisans (the former name of
> >   ververica) project, which is not involved in Flink release manual [3].
> We propose to promote this repo under
> >   apache project so that release manager could have the right to release
> on flink-benchmarks.
> > - The reason why we not involve flink-benchmarks into the apache/flink
> repo is because it heavily depends on
> >   JMH [4], which is under GPLv2 license.
> >
> > What do you think?
> >
> > Best,
> > Yun Tang
> >
> > [1] https://github.com/dataArtisans/flink-benchmarks
> > [2] https://issues.apache.org/jira/browse/FLINK-16850
> > [3]
> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release
> > [4] https://openjdk.java.net/projects/code-tools/jmh/
> >
> >
>
>

[jira] [Created] (FLINK-17075) Add task status reconciliation between TM and JM

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-17075:
-

 Summary: Add task status reconciliation between TM and JM
 Key: FLINK-17075
 URL: https://issues.apache.org/jira/browse/FLINK-17075
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.10.0, 1.11.0
Reporter: Till Rohrmann


In order to harden the TM and JM communication I suggest to let the 
{{TaskExecutor}} send the task statuses back to the {{JobMaster}} as part of 
the heartbeat payload (similar to FLINK-11059). This would allow to reconcile 
the states of both components in case that a status update message was lost as 
described by a user on the ML.

https://lists.apache.org/thread.html/ra9ed70866381f0ef0f4779633346722ccab3dc0d6dbacce04080b74e%40%3Cuser.flink.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[PROPOSAL] Contribute training materials to Apache Flink

[David Anderson]

Dear Flink Community!

For some time now Ververica has been hosting some freely available Apache
Flink training materials at https://training.ververica.com. This includes
tutorial content covering the core concepts of the DataStream API, and
hands-on exercises that accompany those explanations.

Website: https://training.ververica.com
Website repo: https://github.com/dataArtisans/flink-training
Exercises: repo: https://github.com/ververica/flink-training-exercises

We would like to contribute this training content to Apache Flink. By doing
so, we hope to make it even easier for folks to get started with Flink.
Especially during this time when so many are working from home, we'd like
to get this self-paced training course out where more people will see it.

If the community wants these training materials, then this also raises the
question of where to put them. We are thinking it would be best to
integrate the website content into flink.apache.org, and to add the
exercises to flink-playgrounds -- but these points can be discussed
separately once we've established that the community wants this content.

Looking forward to hearing what you think!

Best regards,
David

Re: [DISCUSS] Creating a new repo to host Flink benchmarks

[Robert Metzger]

+1 on creating the repo.


On Thu, Apr 9, 2020 at 5:54 PM Till Rohrmann  wrote:

> I think it is a good idea to make the benchmarks available to the community
> via a repo under the Apache project and to make updating it part of the
> release process. Hence +1 for the proposal.
>
> Cheers,
> Till
>
> On Thu, Apr 9, 2020 at 4:01 PM Piotr Nowojski  wrote:
>
> > Hi Yun Tang,
> >
> > Thanks for proposing the idea. Since we can not include benchmarks in the
> > Flink repository what you are proposing is the second best option.
> >
> > +1 from my side for the proposal.
> >
> > I think benchmarks have proven their value to justify this.
> >
> > Piotrek
> >
> > > On 9 Apr 2020, at 08:56, Yun Tang  wrote:
> > >
> > > Hi Flink devs,
> > >
> > > As Flink develops rapidly with more and more features added, how to
> > ensure no performance regression existed has become more and more
> > important. And we would like to create a new repo under apache project to
> > host previous flink-benchmarks [1] repo, which is inspired when we
> discuss
> > under FLINK-16850 [2]
> > >
> > > Some background context on flink-benchmarks, for those who are not
> > familiar with the project yet:
> > >
> > > - Current flink-benchmarks does not align with the Flink release, which
> > lead developers not easy to verify
> > >   performance at specific Flink version because current
> flink-benchmarks
> > always depends on the latest interfaces.
> > > - Above problem could be solved well if we could ensure
> flink-benchmarks
> > also create release branch when we
> > >   releasing Flink. However, current flink-benchmarks repo is hosted
> > under dataArtisans (the former name of
> > >   ververica) project, which is not involved in Flink release manual
> [3].
> > We propose to promote this repo under
> > >   apache project so that release manager could have the right to
> release
> > on flink-benchmarks.
> > > - The reason why we not involve flink-benchmarks into the apache/flink
> > repo is because it heavily depends on
> > >   JMH [4], which is under GPLv2 license.
> > >
> > > What do you think?
> > >
> > > Best,
> > > Yun Tang
> > >
> > > [1] https://github.com/dataArtisans/flink-benchmarks
> > > [2] https://issues.apache.org/jira/browse/FLINK-16850
> > > [3]
> >
> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release
> > > [4] https://openjdk.java.net/projects/code-tools/jmh/
> > >
> > >
> >
> >
>

Re: [PROPOSAL] Contribute training materials to Apache Flink

[Niels Basjes]

Hi,

Sounds like a very nice thing to have as part of the project ecosystem.

Niels

On Thu, Apr 9, 2020 at 8:10 PM David Anderson  wrote:

> Dear Flink Community!
>
> For some time now Ververica has been hosting some freely available Apache
> Flink training materials at https://training.ververica.com. This includes
> tutorial content covering the core concepts of the DataStream API, and
> hands-on exercises that accompany those explanations.
>
> Website: https://training.ververica.com
> Website repo: https://github.com/dataArtisans/flink-training
> Exercises: repo: https://github.com/ververica/flink-training-exercises
>
> We would like to contribute this training content to Apache Flink. By doing
> so, we hope to make it even easier for folks to get started with Flink.
> Especially during this time when so many are working from home, we'd like
> to get this self-paced training course out where more people will see it.
>
> If the community wants these training materials, then this also raises the
> question of where to put them. We are thinking it would be best to
> integrate the website content into flink.apache.org, and to add the
> exercises to flink-playgrounds -- but these points can be discussed
> separately once we've established that the community wants this content.
>
> Looking forward to hearing what you think!
>
> Best regards,
> David
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [PROPOSAL] Contribute training materials to Apache Flink

[Seth Wiesman]

Hi David,

+1 to add to the project.

I agree that flink.apache.org and flink playgrounds respectively are the
best places to host this content.

On Thu, Apr 9, 2020 at 2:56 PM Niels Basjes  wrote:

> Hi,
>
> Sounds like a very nice thing to have as part of the project ecosystem.
>
> Niels
>
> On Thu, Apr 9, 2020 at 8:10 PM David Anderson  wrote:
>
>> Dear Flink Community!
>>
>> For some time now Ververica has been hosting some freely available Apache
>> Flink training materials at https://training.ververica.com. This includes
>> tutorial content covering the core concepts of the DataStream API, and
>> hands-on exercises that accompany those explanations.
>>
>> Website: https://training.ververica.com
>> Website repo: https://github.com/dataArtisans/flink-training
>> Exercises: repo: https://github.com/ververica/flink-training-exercises
>>
>> We would like to contribute this training content to Apache Flink. By
>> doing
>> so, we hope to make it even easier for folks to get started with Flink.
>> Especially during this time when so many are working from home, we'd like
>> to get this self-paced training course out where more people will see it.
>>
>> If the community wants these training materials, then this also raises the
>> question of where to put them. We are thinking it would be best to
>> integrate the website content into flink.apache.org, and to add the
>> exercises to flink-playgrounds -- but these points can be discussed
>> separately once we've established that the community wants this content.
>>
>> Looking forward to hearing what you think!
>>
>> Best regards,
>> David
>>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>

[jira] [Created] (FLINK-17076) Revamp Kafka Connector Documentation

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-17076:


 Summary: Revamp Kafka Connector Documentation
 Key: FLINK-17076
 URL: https://issues.apache.org/jira/browse/FLINK-17076
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka, Documentation
Reporter: Seth Wiesman


The Kafka connector is one of the most popular connectors in Flink. The 
documentation has grown organically over the years as new versions and features 
have become supported. The page is still based on the kafka 0.8 connector with 
asides for newer versions. 

Since Kafka 1.0 was released in 2017, I believe the page should be restructured 
around the universal connector and Kafka(De)SerializationSchema's with clear 
examples.  




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17077) FLINK_CONF_DIR environment variable to locate flink-conf.yaml in Docker container

[Eui Heo (Jira)]

Eui Heo created FLINK-17077:
---

 Summary: FLINK_CONF_DIR environment variable to locate 
flink-conf.yaml in Docker container
 Key: FLINK-17077
 URL: https://issues.apache.org/jira/browse/FLINK-17077
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Docker
Reporter: Eui Heo


To use flink-conf.yaml outside Flink home directory, we should use 
FLINK_CONF_DIR.
But despite of FLINK_CONF_DIR is provided, docker-entrypoint.sh in official 
flink-docker doesn't know FLINK_CONF_DIR and it is ignored when append 
additional flink properties to flink-conf.yaml. It would be good to use 
FLINK_CONF_DIR for the location of flink-conf.yaml, if user provide it.

https://github.com/apache/flink-docker/blob/master/docker-entrypoint.sh#L23



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-124: Add open/close and Collector to (De)SerializationSchema

[SHI Xiaogang]

Hi,

I don't think the proposal is a good solution to the problems. I am in
favour of using a ProcessFunction chained to the source/sink function to
serialize/deserialize the records, instead of embedding (de)serialization
schema in source/sink function.

Message packing is heavily used in our production environment to allow
compression and improve throughput. As buffered messages have to be
delivered when the time exceeds the limit, timers are also required in our
cases. I think it's also a common need for other users.

In the this proposal, with more components added into the context, in the
end we will find the serialization/deserialization schema is just another
wrapper of ProcessFunction.

Regards,
Xiaogang

Aljoscha Krettek  于2020年4月7日周二 下午6:34写道：

> On 07.04.20 08:45, Dawid Wysakowicz wrote:
>
> > @Jark I was aware of the implementation of SinkFunction, but it was a
> > conscious choice to not do it that way.
> >
> > Personally I am against giving a default implementation to both the new
> > and old methods. This results in an interface that by default does
> > nothing or notifies the user only in the runtime, that he/she has not
> > implemented a method of the interface, which does not sound like a good
> > practice to me. Moreover I believe the method without a Collector will
> > still be the preferred method by many users. Plus it communicates
> > explicitly what is the minimal functionality required by the interface.
> > Nevertheless I am happy to hear other opinions.
>
> Dawid and I discussed this before. I did the extension of the
> SinkFunction but by now I think it's better to do it this way, because
> otherwise you can implement the interface without implementing any methods.
>
> > @all I also prefer the buffering approach. Let's wait a day or two more
> > to see if others think differently.
>
> I'm also in favour of buffering outside the lock.
>
> Also, +1 to this FLIP.
>
> Best,
> Aljoscha
>

Re: [VOTE] FLIP-71: E2E View support in Flink SQL

[Danny Chan]

+1 from my side.

Best,
Danny Chan
在 2020年4月9日 +0800 PM9:23，Timo Walther ，写道：
> +1 (binding)
>
> Thanks for your efforts.
>
> Regards,
> Timo
>
>
> On 09.04.20 14:46, Zhenghua Gao wrote:
> > Hi all,
> >
> > I'd like to start the vote for FLIP-71[1] which adds E2E view support in
> > Flink SQL.
> > This FLIP is discussed in the thread[2].
> >
> > The vote will be open for at least 72 hours. Unless there is an objection.
> > I will try to
> > close it by April 13, 2020 09:00 UTC if we have received sufficient votes.
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-71%3A+E2E+View+support+in+FLINK+SQL
> >
> > [2]
> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-71-E2E-View-support-in-Flink-SQL-td33131.html#a39787
> >
> > *Best Regards,*
> > *Zhenghua Gao*
> >
>

[VOTE] FLIP-113: Supports Dynamic Table Options for Flink SQL

[Danny Chan]

Hi all,

I would like to start the vote for FLIP-113 [1], which is discussed and
reached a consensus in the discussion thread [2].

The vote will be open until April 13nd (72h), unless there is an
objection or not enough votes.

Best,
Danny Chan

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL
[2]
https://lists.apache.org/thread.html/r94af5d3d97e76e7dd9df68cb0becc7ba74d15591a8fab84c72fa%40%3Cdev.flink.apache.org%3E

[jira] [Created] (FLINK-17078) Logging output is misleading when executing bin/flink -e kubernetes-session without specifying cluster-id

[Canbin Zheng (Jira)]

Canbin Zheng created FLINK-17078:


 Summary: Logging output is misleading when executing bin/flink -e 
kubernetes-session without specifying cluster-id
 Key: FLINK-17078
 URL: https://issues.apache.org/jira/browse/FLINK-17078
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Kubernetes
Reporter: Canbin Zheng
 Fix For: 1.11.0


When executing the following command:

 
{code:java}
./bin/flink run -d -e kubernetes-session 
examples/streaming/SocketWindowWordCount.jar --hostname 172.16.0.6 --port 12345
{code}
The exception stack would be:
{quote}org.apache.flink.client.program.ProgramInvocationException: The main 
method caused an error: 
org.apache.flink.client.deployment.ClusterRetrieveException: Could not get the 
rest endpoint of flink-cluster-6fa5f5dc-4b50-48c3-b57f-d91e686fa474
 at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
 at 
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
 at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:148)
 at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:662)
 at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
 at 
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:893)
 at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:966)
 at 
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
 at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:966)
Caused by: java.lang.RuntimeException: 
org.apache.flink.client.deployment.ClusterRetrieveException: Could not get the 
rest endpoint of flink-cluster-6fa5f5dc-4b50-48c3-b57f-d91e686fa474
 at 
org.apache.flink.kubernetes.KubernetesClusterDescriptor.lambda$createClusterClientProvider$0(KubernetesClusterDescriptor.java:94)
 at 
org.apache.flink.kubernetes.KubernetesClusterDescriptor.retrieve(KubernetesClusterDescriptor.java:118)
 at 
org.apache.flink.kubernetes.KubernetesClusterDescriptor.retrieve(KubernetesClusterDescriptor.java:59)
 at 
org.apache.flink.client.deployment.executors.AbstractSessionClusterExecutor.execute(AbstractSessionClusterExecutor.java:63)
 at 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1756)
 at 
org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:106)
 at 
org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:72)
 at 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1643)
 at 
org.apache.flink.streaming.examples.socket.SocketWindowWordCount.main(SocketWindowWordCount.java:92)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
 ... 8 more
Caused by: org.apache.flink.client.deployment.ClusterRetrieveException: Could 
not get the rest endpoint of flink-cluster-6fa5f5dc-4b50-48c3-b57f-d91e686fa474
 ... 22 more
{quote}
The logging output is misleading, we'd better throw an exception indicating 
that people should explicitly specify the value of {{kubernetes.cluster-id}}.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Creating a new repo to host Flink benchmarks

[Zhijiang]

+1 for the proposal.

Best,
Zhijiang 


--
From:Robert Metzger 
Send Time:2020 Apr. 10 (Fri.) 02:15
To:dev 
Subject:Re: [DISCUSS] Creating a new repo to host Flink benchmarks

+1 on creating the repo.


On Thu, Apr 9, 2020 at 5:54 PM Till Rohrmann  wrote:

> I think it is a good idea to make the benchmarks available to the community
> via a repo under the Apache project and to make updating it part of the
> release process. Hence +1 for the proposal.
>
> Cheers,
> Till
>
> On Thu, Apr 9, 2020 at 4:01 PM Piotr Nowojski  wrote:
>
> > Hi Yun Tang,
> >
> > Thanks for proposing the idea. Since we can not include benchmarks in the
> > Flink repository what you are proposing is the second best option.
> >
> > +1 from my side for the proposal.
> >
> > I think benchmarks have proven their value to justify this.
> >
> > Piotrek
> >
> > > On 9 Apr 2020, at 08:56, Yun Tang  wrote:
> > >
> > > Hi Flink devs,
> > >
> > > As Flink develops rapidly with more and more features added, how to
> > ensure no performance regression existed has become more and more
> > important. And we would like to create a new repo under apache project to
> > host previous flink-benchmarks [1] repo, which is inspired when we
> discuss
> > under FLINK-16850 [2]
> > >
> > > Some background context on flink-benchmarks, for those who are not
> > familiar with the project yet:
> > >
> > > - Current flink-benchmarks does not align with the Flink release, which
> > lead developers not easy to verify
> > >   performance at specific Flink version because current
> flink-benchmarks
> > always depends on the latest interfaces.
> > > - Above problem could be solved well if we could ensure
> flink-benchmarks
> > also create release branch when we
> > >   releasing Flink. However, current flink-benchmarks repo is hosted
> > under dataArtisans (the former name of
> > >   ververica) project, which is not involved in Flink release manual
> [3].
> > We propose to promote this repo under
> > >   apache project so that release manager could have the right to
> release
> > on flink-benchmarks.
> > > - The reason why we not involve flink-benchmarks into the apache/flink
> > repo is because it heavily depends on
> > >   JMH [4], which is under GPLv2 license.
> > >
> > > What do you think?
> > >
> > > Best,
> > > Yun Tang
> > >
> > > [1] https://github.com/dataArtisans/flink-benchmarks
> > > [2] https://issues.apache.org/jira/browse/FLINK-16850
> > > [3]
> >
> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release
> > > [4] https://openjdk.java.net/projects/code-tools/jmh/
> > >
> > >
> >
> >
>

[DISCUSS] Releasing Flink 1.9.3

[Dian Fu]

Hi all,

It has been more than two months since we released Flink 1.9.2. There are 
already 36 improvements/bugfixes in the release-1.9 branch. Therefore, I 
propose to create the next bugfix release 1.9.3 for Flink 1.9.

Most notable fixes are:

- [FLINK-15085] HistoryServer dashboard config json out of sync
- [FLINK-15575] Azure Filesystem Shades Wrong Package "httpcomponents"
- [FLINK-15638] releasing/create_release_branch.sh does not set version in 
flink-python/pyflink/version.py
- [FLINK-16242] BinaryGeneric serialization error cause checkpoint failure
- [FLINK-16573] Kinesis consumer does not properly shutdown RecordFetcher 
threads
- [FLINK-16047] Blink planner produces wrong aggregate results with state clean 
up
- [FLINK-16860] TableException: Failed to push filter into table source! when 
upgrading flink to 1.9.2
- [FLINK-16916] The logic of NullableSerializer#copy is wrong
- [FLINK-16389] Bump Kafka 0.10 to 0.10.2.2
- [FLINK-15812] HistoryServer archiving is done in Dispatcher main thread
- [FLINK-17062] Fix the conversion from Java row type to Python row type

Furthermore, there is one blocker issue which should be merged before 1.9.3 
release:

- [FLINK-16576] State inconsistency on restore with memory state backends 
(reviewing)

I would volunteer as the release manager and kick off the release process. What 
do you think?

Please let me know if there are any concerns or any other blocker issues need 
to be fixed in 1.9.3. Thanks.

Appreciated if there is any PMC could help with the final steps of the release 
process.

Regards,
Dian

Re: [DISCUSS] Creating a new repo to host Flink benchmarks

[Dian Fu]

+1 for this proposal. 

> 在 2020年4月10日，上午10:58，Zhijiang  写道：
> 
> +1 for the proposal.
> 
> Best,
> Zhijiang 
> 
> 
> --
> From:Robert Metzger 
> Send Time:2020 Apr. 10 (Fri.) 02:15
> To:dev 
> Subject:Re: [DISCUSS] Creating a new repo to host Flink benchmarks
> 
> +1 on creating the repo.
> 
> 
> On Thu, Apr 9, 2020 at 5:54 PM Till Rohrmann  wrote:
> 
>> I think it is a good idea to make the benchmarks available to the community
>> via a repo under the Apache project and to make updating it part of the
>> release process. Hence +1 for the proposal.
>> 
>> Cheers,
>> Till
>> 
>> On Thu, Apr 9, 2020 at 4:01 PM Piotr Nowojski  wrote:
>> 
>>> Hi Yun Tang,
>>> 
>>> Thanks for proposing the idea. Since we can not include benchmarks in the
>>> Flink repository what you are proposing is the second best option.
>>> 
>>> +1 from my side for the proposal.
>>> 
>>> I think benchmarks have proven their value to justify this.
>>> 
>>> Piotrek
>>> 
 On 9 Apr 2020, at 08:56, Yun Tang  wrote:
 
 Hi Flink devs,
 
 As Flink develops rapidly with more and more features added, how to
>>> ensure no performance regression existed has become more and more
>>> important. And we would like to create a new repo under apache project to
>>> host previous flink-benchmarks [1] repo, which is inspired when we
>> discuss
>>> under FLINK-16850 [2]
 
 Some background context on flink-benchmarks, for those who are not
>>> familiar with the project yet:
 
 - Current flink-benchmarks does not align with the Flink release, which
>>> lead developers not easy to verify
  performance at specific Flink version because current
>> flink-benchmarks
>>> always depends on the latest interfaces.
 - Above problem could be solved well if we could ensure
>> flink-benchmarks
>>> also create release branch when we
  releasing Flink. However, current flink-benchmarks repo is hosted
>>> under dataArtisans (the former name of
  ververica) project, which is not involved in Flink release manual
>> [3].
>>> We propose to promote this repo under
  apache project so that release manager could have the right to
>> release
>>> on flink-benchmarks.
 - The reason why we not involve flink-benchmarks into the apache/flink
>>> repo is because it heavily depends on
  JMH [4], which is under GPLv2 license.
 
 What do you think?
 
 Best,
 Yun Tang
 
 [1] https://github.com/dataArtisans/flink-benchmarks
 [2] https://issues.apache.org/jira/browse/FLINK-16850
 [3]
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release
 [4] https://openjdk.java.net/projects/code-tools/jmh/
 
 
>>> 
>>> 
>> 
> 

Re: [DISCUSS] Releasing Flink 1.9.3

[jincheng sun]

Hi Dian,

Thanks for bring up the discussion. I would like to give you a hand at the
last stage when the RC is finished.  :)

Best,
Jincheng



Dian Fu  于2020年4月10日周五 上午11:08写道：

> Hi all,
>
> It has been more than two months since we released Flink 1.9.2. There are
> already 36 improvements/bugfixes in the release-1.9 branch. Therefore, I
> propose to create the next bugfix release 1.9.3 for Flink 1.9.
>
> Most notable fixes are:
>
> - [FLINK-15085] HistoryServer dashboard config json out of sync
> - [FLINK-15575] Azure Filesystem Shades Wrong Package "httpcomponents"
> - [FLINK-15638] releasing/create_release_branch.sh does not set version in
> flink-python/pyflink/version.py
> - [FLINK-16242] BinaryGeneric serialization error cause checkpoint failure
> - [FLINK-16573] Kinesis consumer does not properly shutdown RecordFetcher
> threads
> - [FLINK-16047] Blink planner produces wrong aggregate results with state
> clean up
> - [FLINK-16860] TableException: Failed to push filter into table source!
> when upgrading flink to 1.9.2
> - [FLINK-16916] The logic of NullableSerializer#copy is wrong
> - [FLINK-16389] Bump Kafka 0.10 to 0.10.2.2
> - [FLINK-15812] HistoryServer archiving is done in Dispatcher main thread
> - [FLINK-17062] Fix the conversion from Java row type to Python row type
>
> Furthermore, there is one blocker issue which should be merged before
> 1.9.3 release:
>
> - [FLINK-16576] State inconsistency on restore with memory state backends
> (reviewing)
>
> I would volunteer as the release manager and kick off the release process.
> What do you think?
>
> Please let me know if there are any concerns or any other blocker issues
> need to be fixed in 1.9.3. Thanks.
>
> Appreciated if there is any PMC could help with the final steps of the
> release process.
>
> Regards,
> Dian

Re: [DISCUSS] FLIP-118: Improve Flink’s ID system

[Yangze Guo]

Hi there,

Sorry for the belated reply. I just make a preliminary investigation
of the infect of refactoring IntermediateResultPartitionID.

The key change is making it being composed of IntermediateDataSetID
and a partitionNum.
public class IntermediateResultPartitionID {
   private final IntermediateDataSetID intermediateDataSetID;
   private final int partitionNum;
}

In this benchmark, I use examples/streaming/WordCount.jar as the test
job and run Flink on Yarn. All the configurations are kept default
except for "taskmanager.numberOfTaskSlots", which is set to 2.

The result shows it does have an impact on performance.
- After this change, the maximum parallelism went from 760 to 685,
which limited by the total number of network buffers. For the same
parallelism, user needs more network buffer than before.
- The netty message "PartitionRequest" and "TaskEventRequest" increase
by 4 bytes. For "PartitionRequest", it means 7% increase.
- The TDD takes longer to construct. With 600 parallelisms, it takes
twice as long to construct TDD than before.

Details record in
https://docs.google.com/spreadsheets/d/13lt6J29uikWoux79047lkvbGi2fS4ioWHSbvT1kTL2M/edit?usp=sharing

The same issue could happen in ExecutionAttemptID, which will increase
the "PartitionRequest" and "TaskEventRequest" by 8 bytes(subtaskIndex
and attemptNumber). But it may not infect the TDD as much as
IntermediateResultPartitionID, since there is only one
ExecutionAttemptID per TDD.

After that preliminary investigation, I tend to not refactor
ExecutionAttemptID and IntermediateResultPartitionID at the moment or
treat it as future work.

WDYT? @ZhuZhu @Till

Best,
Yangze Guo

On Wed, Apr 1, 2020 at 11:53 AM Zhu Zhu  wrote:
>
> >> However, it seems the JobVertexID is derived from hashcode ...
> You are right. JobVertexID is widely used and reworking it may affect the
> public
> interfaces, e.g. REST api. We can take it as a long term goal and exclude
> it from this FLIP.
> This same applies to IntermediateDataSetID, which can be also composed of a
> JobID
> and an index as Till proposed.
>
> Thanks,
> Zhu Zhu
>
> Till Rohrmann  于2020年3月31日周二 下午8:36写道：
>
> > For the IntermediateDataSetID I was just thinking that it might actually be
> > interesting to know which job produced the result which, by using cluster
> > partitions, could be used across different jobs. Not saying that we have to
> > do it, though.
> >
> > A small addition to Zhu Zhu's comment about TDD sizes: For the problem with
> > too large TDDs there is already an issue [1]. The current suspicion is that
> > the size of TDDs for jobs with a large parallelism can indeed become
> > problematic for Flink. Hence, it would be great to investigate the impacts
> > of the proposed changes.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-16069
> >
> > Cheers,
> > Till
> >
> > On Tue, Mar 31, 2020 at 11:50 AM Yangze Guo  wrote:
> >
> > > Hi, Zhu,
> > >
> > > Thanks for the feedback.
> > >
> > > > make JobVertexID a composition of JobID and a topology index
> > > I think it is a good idea. However, it seems the JobVertexID is
> > > derived from hashcode which used to identify them across submission.
> > > I'm not familiar with that component though. I prefer to keep this
> > > idea out the scope of this FLIP if no one could help us to figure it
> > > out.
> > >
> > > > How about we still keep IntermediateDataSetID independent from
> > > JobVertexID,
> > > > but just print the producing relationships in logs? I think keeping
> > > > IntermediateDataSetID independent may be better considering the cross
> > job
> > > > result usages in interactive query cases.
> > > I think you are right. I'll keep IntermediateDataSetID independent
> > > from JobVertexID.
> > >
> > > > The new IDs will become larger with this rework.
> > > Yes, I also have the same concern. Benchmark is necessary, I'll try to
> > > provide one during the implementation phase.
> > >
> > >
> > > Best,
> > > Yangze Guo
> > >
> > > On Tue, Mar 31, 2020 at 4:55 PM Zhu Zhu  wrote:
> > > >
> > > > Thanks for proposing this improvement Yangze. Big +1 for the overall
> > > > proposal. It can help a lot in troubleshooting.
> > > >
> > > > Here are a few questions for it:
> > > > 1. Shall we make JobVertexID a composition of JobID and a topology
> > index?
> > > > This would help in the session cluster case, so that we can identify
> > > which
> > > > tasks are from which jobs along with the rework of ExecutionAttemptID.
> > > >
> > > > 2. You mentioned that "Add the producer info to the string literal of
> > > > IntermediateDataSetID". Do you mean to make IntermediateDataSetID a
> > > > composition of JobVertexID and a consumer index?
> > > > How about we still keep IntermediateDataSetID independent from
> > > JobVertexID,
> > > > but just print the producing relationships in logs? I think keeping
> > > > IntermediateDataSetID independent may be better considering the cross
> > job
> > > > result usages in interactive query cases.

Re: [DISCUSS] Releasing Flink 1.9.3

[Dian Fu]

Hi Jincheng,

Thanks a lot for offering help. It would be very helpful. Thanks again!

Regards,
Dian

> 在 2020年4月10日，下午12:46，jincheng sun  写道：
> 
> Hi Dian,
> 
> Thanks for bring up the discussion. I would like to give you a hand at the
> last stage when the RC is finished.  :)
> 
> Best,
> Jincheng
> 
> 
> 
> Dian Fu  于2020年4月10日周五 上午11:08写道：
> 
>> Hi all,
>> 
>> It has been more than two months since we released Flink 1.9.2. There are
>> already 36 improvements/bugfixes in the release-1.9 branch. Therefore, I
>> propose to create the next bugfix release 1.9.3 for Flink 1.9.
>> 
>> Most notable fixes are:
>> 
>> - [FLINK-15085] HistoryServer dashboard config json out of sync
>> - [FLINK-15575] Azure Filesystem Shades Wrong Package "httpcomponents"
>> - [FLINK-15638] releasing/create_release_branch.sh does not set version in
>> flink-python/pyflink/version.py
>> - [FLINK-16242] BinaryGeneric serialization error cause checkpoint failure
>> - [FLINK-16573] Kinesis consumer does not properly shutdown RecordFetcher
>> threads
>> - [FLINK-16047] Blink planner produces wrong aggregate results with state
>> clean up
>> - [FLINK-16860] TableException: Failed to push filter into table source!
>> when upgrading flink to 1.9.2
>> - [FLINK-16916] The logic of NullableSerializer#copy is wrong
>> - [FLINK-16389] Bump Kafka 0.10 to 0.10.2.2
>> - [FLINK-15812] HistoryServer archiving is done in Dispatcher main thread
>> - [FLINK-17062] Fix the conversion from Java row type to Python row type
>> 
>> Furthermore, there is one blocker issue which should be merged before
>> 1.9.3 release:
>> 
>> - [FLINK-16576] State inconsistency on restore with memory state backends
>> (reviewing)
>> 
>> I would volunteer as the release manager and kick off the release process.
>> What do you think?
>> 
>> Please let me know if there are any concerns or any other blocker issues
>> need to be fixed in 1.9.3. Thanks.
>> 
>> Appreciated if there is any PMC could help with the final steps of the
>> release process.
>> 
>> Regards,
>> Dian

Re: [VOTE] FLIP-111: Docker image unification

[Yangze Guo]

+1 (non-binding)
Thanks for driving this, Andrey.

Best,
Yangze Guo

On Thu, Apr 9, 2020 at 11:33 PM Ismaël Mejía  wrote:
>
> +1 (non-binding)
> Great work Andrey, pretty excited about this happening!
>
>
> On Wed, Apr 8, 2020 at 4:20 AM Canbin Zheng  wrote:
> >
> > Thanks for the FLIP Andrey.
> >
> > +1 (non-binding) from my side.
> >
> > Regards,
> > Canbin Zheng
> >
> > Yang Wang  于2020年4月8日周三 上午9:57写道：
> >
> > > Thanks Andrey for the efforts of docker image unification.
> > >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Yang
> > >
> > > Till Rohrmann  于2020年4月7日周二 下午11:04写道：
> > >
> > > > Thanks for driving this effort Andrey.
> > > >
> > > > +1 (binding)
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Tue, Apr 7, 2020 at 4:48 PM Ufuk Celebi  wrote:
> > > >
> > > > > Thanks for starting this FLIP.
> > > > >
> > > > > +1
> > > > >
> > > > > On Tue, Apr 7, 2020 at 11:29 AM Andrey Zagrebin 
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > As discussed in these threads [1] and [2],
> > > > > > we suggest to unify the docker topic in Flink for users [3].
> > > > > >
> > > > > > This mainly means refactoring of the existing code and introducing
> > > more
> > > > > > docs as a first step.
> > > > > > The effort should enable further improvements and follow-ups for the
> > > > > docker
> > > > > > integration with Flink.
> > > > > >
> > > > > > The integration with docker in Flink is currently addressed in many
> > > > > places
> > > > > > which often intersect, repeat each other or apply different
> > > approaches.
> > > > > > This makes it really hard to follow the whole topic for users and
> > > > > > maintainers. This FLIP suggests how to unify this topic. It means
> > > > having
> > > > > > one place which has the *Dockerfile*, all necessary scripts and docs
> > > > > > following each other in a consistent way without any repetitions or
> > > > > > contradictions.
> > > > > >
> > > > > > The idea is to keep all docker related resources in
> > > apache/flink-docker
> > > > > > . It already has a detailed
> > > > > > Dockerfile which is well suited for common use cases or at least
> > > serves
> > > > > as
> > > > > > a good starting point. The suggestion is to make it extensible for
> > > > other
> > > > > > concerns which are currently addressed in other places.
> > > > > >
> > > > > > Best,
> > > > > > Andrey
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-111-Docker-image-unification-td38444.html#a39822
> > > > > > [2]
> > > > > >
> > > > > >
> > > > >
> > > >
> > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-111-Docker-image-unification-td38444i20.html#a39950
> > > > > > [3]
> > > > > >
> > > > > >
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification
> > > > > >
> > > > >
> > > >
> > >

[jira] [Created] (FLINK-17079) CsvTableSinkFactoryBase add numFiles and writeMode config when createTableSink

[sulei (Jira)]

sulei created FLINK-17079:
-

 Summary: CsvTableSinkFactoryBase add numFiles and writeMode config 
when createTableSink
 Key: FLINK-17079
 URL: https://issues.apache.org/jira/browse/FLINK-17079
 Project: Flink
  Issue Type: Wish
Reporter: sulei


when CsvTableSinkFactoryBase create csvTableSink, the numFiles and writeMode 
use the default value and could not change. this Issue makes the numFiles and 
writeMode could change by user's config.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)