[jira] [Created] (FLINK-17080) Utils.CollectHelper is possible to throw NPE if the sink is up but not initialized when the job fails

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-17080:
---

 Summary: Utils.CollectHelper is possible to throw NPE if the sink 
is up but not initialized when the job fails
 Key: FLINK-17080
 URL: https://issues.apache.org/jira/browse/FLINK-17080
 Project: Flink
  Issue Type: Bug
  Components: API / DataSet
Affects Versions: 1.11.0
Reporter: Caizhi Weng


When the sink using {{Utils.CollectHelper}} is up but not initialized and the 
job fails due to other operators, it is possible that the {{close()}} method of 
{{Utils.CollectHelper}} is called when {{open()}} is not called.

Currently {{Utils.CollectHelper}} does not deal with this case and will throw 
NPE.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17081) Batch test classes in Blink planner does not extend TestLogger

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-17081:
---

 Summary: Batch test classes in Blink planner does not extend 
TestLogger
 Key: FLINK-17081
 URL: https://issues.apache.org/jira/browse/FLINK-17081
 Project: Flink
  Issue Type: Test
  Components: Tests
Affects Versions: 1.11.0
Reporter: Caizhi Weng


Currently streaming test classes in Blink planner has extended {{TestLogger}} 
while batch test classes hasn't. They should also extend {{TestLogger}} for 
better debugging.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-105: Support to Interpret and Emit Changelog in Flink SQL

[Jark Wu]

Hi,

After a short offline discussion with Kurt, It seems that I misunderstood
Kurt's meaning.
Kurt meant: is `format=debezium` is enough, or split into two options
`format=debezium` and `format.encoding=json`.

Debezium not only support JSON encoding, but also Avro. Canal supports JSON
and Protobuf. So a single `format=debezium` is not enough (in the long
term).
The reason I proposed a single option `format=debezium-json` instead of two:
 - It's simpler to write a single option instead of two, we also make this
design decision for "connector" and "version".
 - I didn't find a good name for separate option keys, because JSON is also
a format, not an encoding, but `format.format=json` is weird.

Hi everyone,

If there are no further concerns, I would like to start a voting thread by
tomorrow.

Best,
Jark



On Wed, 8 Apr 2020 at 15:37, Jark Wu  wrote:

> Hi Kurt,
>
> The JSON encoding of Debezium can be configured to include or exclude the
> message schema using the `value.converter.schemas.enable` properties.
> That's why we propose to have a `format.schema-include` property key to
> config how to parse the json.
>
> Besides, the encoding format of debezium is stable and unified across
> different databases (MySQL, Oracle, SQL Server, DB2, PostgresSQL).
> However, because of the limitation of some special databases, some
> databases CDC encoding are different (Cassandra and MongoDB).
> If we want to support them in the future, we can introduce an optional
> property key, e.g. `format.encoding-connector=mongodb`, to recognize this
> special encoding.
>
> Canal currently only support to capture changes from MySQL, so there is
> only one encoding in Canal. But both Canal and Debezium may evolve their
> encoding in the future.
> We can also introduce a `format.encoding-version` in the future if needed.
>
> Best,
> Jark
>
>
> On Wed, 8 Apr 2020 at 14:26, Kurt Young  wrote:
>
>> One minor comment, is there any other encoding or format in debezium? I'm
>> asking because the format
>> name is debezium-json, i'm wondering whether debezium is enough. This also
>> applies to canal.
>>
>> Best,
>> Kurt
>>
>>
>> On Tue, Apr 7, 2020 at 11:49 AM Jark Wu  wrote:
>>
>> > Hi everyone,
>> >
>> > Since this FLIP was proposed, the community has discussed a lot about
>> the
>> > first approach: introducing new TableSource and TableSink interfaces to
>> > support changelog.
>> > And yes, that is FLIP-95 which has been accepted last week. So most of
>> the
>> > work has been merged into FLIP-95.
>> >
>> > In order to support the goal of FLIP-105, there is still a little
>> things to
>> > discuss: how to connect external CDC formats.
>> > We propose to introduce 2 new formats: Debezium format and Canal format.
>> > They are the most popular CDC tools according to the survey in user [1]
>> and
>> > user-zh [2] mailing list.
>> >
>> > I have updated the FLIP:
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL
>> >
>> > Welcome feedbacks!
>> >
>> > Best,
>> > Jark
>> >
>> > [1]:
>> >
>> >
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/SURVEY-What-Change-Data-Capture-tools-are-you-using-td33569.html
>> > [2]: http://apache-flink.147419.n8.nabble.com/SURVEY-CDC-td1910.html
>> >
>> >
>> > On Fri, 14 Feb 2020 at 22:08, Jark Wu  wrote:
>> >
>> > > Hi everyone,
>> > >
>> > > I would like to start discussion about how to support interpreting
>> > > external changelog into Flink SQL, and how to emit changelog from
>> Flink
>> > SQL.
>> > >
>> > > This topic has already been mentioned several times in the past. CDC
>> > > (Change Data Capture) data has been a very important streaming data in
>> > the
>> > > world. Connect to CDC is a significant feature for Flink, it fills the
>> > > missing piece for Flink's streaming processing.
>> > >
>> > > In FLIP-105, we propose 2 approaches to achieve.
>> > > One is introducing new TableSource interface (higher priority),
>> > > the other is introducing new SQL syntax to interpret and emit
>> changelog.
>> > >
>> > > FLIP-105:
>> > >
>> >
>> https://docs.google.com/document/d/1onyIUUdWAHfr_Yd5nZOE7SOExBc6TiW5C4LiL5FrjtQ/edit#
>> > >
>> > > Thanks for any feedback!
>> > >
>> > > Best,
>> > > Jark
>> > >
>> >
>>
>

[jira] [Created] (FLINK-17082) Remove mocking from SQL client tests

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-17082:


 Summary: Remove mocking from SQL client tests
 Key: FLINK-17082
 URL: https://issues.apache.org/jira/browse/FLINK-17082
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Client, Tests
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-123: DDL and DML compatibility for Hive connector

[Rui Li]

Hi Jingsong,

Thanks for your input. I'd prefer to keep Hive syntax to a separate parser,
because 1) Supporting both Hive and Flink syntax in one parser makes the
grammar complicated and less efficient, e.g. we may need a bigger LOOKAHEAD
value to resolve conflicts. 2) It can be difficult to add new features to a
syntax without interfering with the other one. 3) It'll be more difficult
to turn off Hive syntax when Hive dialect is not used.
The separate parser only requires minimum change in the planner, which I
think is acceptable compared to the benefits it brings us.

On Thu, Apr 9, 2020 at 10:42 PM Jingsong Li  wrote:

> Thanks Rui for diving.
>
> +1 for this proposal.
>
> There are still lots of people who love Hive SQL.
> And I have seen some people support HQL on presto. Presto, as a famous
> computing engine, also supports ANSI SQL as we do. This is quite different
> from HQL.
>
> Do you think we must need import `FlinkHiveSqlParserImpl`? This will bother
> planner code, if possible, I think it is better to keep dialect things in
> sql-parer.
> What do you think?
>
> Best,
> Jingsong Lee
>
> On Thu, Apr 9, 2020 at 3:56 PM Rui Li  wrote:
>
> > Hi Kurt,
> >
> > Thanks for the feedback and that's a good idea. I have updated the FLIP
> and
> > added tables in the "Limited Scope" section to list the supported and
> > unsupported features. Please have a look and let me know if that makes
> > sense. Thanks.
> >
> > On Wed, Apr 8, 2020 at 2:19 PM Kurt Young  wrote:
> >
> > > Hi Rui,
> > >
> > > Thanks for bringing up this discussion and it makes sense to me though
> i
> > > have one comment about the FLIP.
> > > There are a few places in the design document saying some features will
> > not
> > > be supported or not included in
> > > this FLIP, but I don't get what will be supported exactly. I can
> imagine
> > > other users will also have such confusion.
> > > Could you add a table or a list of syntax which will be supported?
> > >
> > > Best,
> > > Kurt
> > >
> > >
> > > On Wed, Apr 1, 2020 at 4:24 PM Rui Li  wrote:
> > >
> > > > Hi devs,
> > > >
> > > > I'd like to start a discussion about bringing DDL & DML compatibility
> > for
> > > > Hive connector. The proposal mainly aims to implement more DDLs for
> > Hive
> > > > connector and allow users to write Hive syntax when using the Hive
> > > dialect.
> > > > Hopefully this will make it easier for users to migrate to Flink,
> with
> > > > fewer SQL statements that need to be changed.
> > > >
> > > > Please find more details in the FLIP wiki [1]. Feedbacks and
> > suggestions
> > > > are appreciated.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-123%3A+DDL+and+DML+compatibility+for+Hive+connector
> > > >
> > > > --
> > > > Cheers,
> > > > Rui Li
> > > >
> > >
> >
> >
> > --
> > Best regards!
> > Rui Li
> >
>
>
> --
> Best, Jingsong Lee
>


-- 
Best regards!
Rui Li

[jira] [Created] (FLINK-17083) Restrict users of accumulators not to return null in Accumulator#getLocalValue

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-17083:
---

 Summary: Restrict users of accumulators not to return null in 
Accumulator#getLocalValue
 Key: FLINK-17083
 URL: https://issues.apache.org/jira/browse/FLINK-17083
 Project: Flink
  Issue Type: Improvement
  Components: API / Core
Affects Versions: 1.11.0
Reporter: Caizhi Weng


[This discussion|https://issues.apache.org/jira/browse/FLINK-13880] raises a 
problem that, if a user returns a null value in {{Accumulator#getLocalValue}}, 
he will be notified that the {{failureCause}} should not be null when fetching 
the accumulator result, which seems to be really wired.

The problem is that we're not explicitly restricting the users not to return 
null values in {{Accumulator#getLocalValue}}. [~aljoscha] is it legal to return 
null values? If not, we should explicitly throws a related exception to the 
users instead of giving the users a somewhat unrelated exception.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-113: Supports Dynamic Table Options for Flink SQL

[Timo Walther]

+1 (binding)

Thanks for the healthy discussion. I think this feature can be useful 
during the development of a pipeline.


Regards,
Timo

On 10.04.20 03:34, Danny Chan wrote:

Hi all,

I would like to start the vote for FLIP-113 [1], which is discussed and
reached a consensus in the discussion thread [2].

The vote will be open until April 13nd (72h), unless there is an
objection or not enough votes.

Best,
Danny Chan

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL
[2]
https://lists.apache.org/thread.html/r94af5d3d97e76e7dd9df68cb0becc7ba74d15591a8fab84c72fa%40%3Cdev.flink.apache.org%3E

[jira] [Created] (FLINK-17084) Implement type inference for ROW/ARRAY/MAP constructors

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-17084:


 Summary: Implement type inference for ROW/ARRAY/MAP constructors
 Key: FLINK-17084
 URL: https://issues.apache.org/jira/browse/FLINK-17084
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.11.0


We can already implement input type inference and output type strategy for 
function:
BuiltinFunctionDefinitions.ROW
BuiltinFunctionDefinitions.ARRAY
BuiltinFunctionDefinitions.MAP





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Javadocs Broken?

[tison]

Hi guys,

Right now when I click "JavaDocs" in out docsite[1] it jumps to a page[2] I
think is definitely not
out api documentation. Any thoughts?

Best,
tison.

[1] https://ci.apache.org/projects/flink/flink-docs-master/
[2] https://ci.apache.org/projects/flink/flink-docs-master/api/java/

Re: Javadocs Broken?

[tison]

s/out/our/

[jira] [Created] (FLINK-17085) Remove FlinkKubeClient#handleException since it does not do anything

[Yang Wang (Jira)]

Yang Wang created FLINK-17085:
-

 Summary: Remove FlinkKubeClient#handleException since it does not 
do anything
 Key: FLINK-17085
 URL: https://issues.apache.org/jira/browse/FLINK-17085
 Project: Flink
  Issue Type: Task
  Components: Deployment / Kubernetes
Reporter: Yang Wang


Follow up the discussion in this PR[1].

Currently, {{FlinkKubeClient#handleException}} just log the exception and do 
not have any substantial operations. Since all the exception has already been 
handled, for example, {{KubernetesClusterDescriptor#createClusterClientProvider 
#killCluster}}, then it should be removed.
 Moreover, the {{handleException}} will cause the exception duplication in CLI. 

 

[1]. [https://github.com/apache/flink/pull/11427#discussion_r404221581]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17086) Flink sql client not able to read parquet hive table because `HiveMapredSplitReader` not supports name mapping reading for parquet format.

[Lei Wang (Jira)]

Lei Wang created FLINK-17086:


 Summary: Flink sql client not able to read parquet hive table 
because  `HiveMapredSplitReader` not supports name mapping reading for parquet 
format.
 Key: FLINK-17086
 URL: https://issues.apache.org/jira/browse/FLINK-17086
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.10.0
Reporter: Lei Wang


When writing hive table with parquet format, flink sql client not able to read 
it correctly because HiveMapredSplitReader not supports name mapping reading 
for parquet format.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-113: Supports Dynamic Table Options for Flink SQL

[Jark Wu]

+1 from my side (binding)

Best,
Jark

On Fri, 10 Apr 2020 at 17:03, Timo Walther  wrote:

> +1 (binding)
>
> Thanks for the healthy discussion. I think this feature can be useful
> during the development of a pipeline.
>
> Regards,
> Timo
>
> On 10.04.20 03:34, Danny Chan wrote:
> > Hi all,
> >
> > I would like to start the vote for FLIP-113 [1], which is discussed and
> > reached a consensus in the discussion thread [2].
> >
> > The vote will be open until April 13nd (72h), unless there is an
> > objection or not enough votes.
> >
> > Best,
> > Danny Chan
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL
> > [2]
> >
> https://lists.apache.org/thread.html/r94af5d3d97e76e7dd9df68cb0becc7ba74d15591a8fab84c72fa%40%3Cdev.flink.apache.org%3E
> >
>
>

Re: [VOTE] FLIP-108: Add GPU support in Flink

[Xintong Song]

Sorry to pull this back. I have some concerns about the recent updated
interface details.

- Should we have a factory interface for `ExternalResourceDriver`, that
takes the configuration and returns a driver instance? Otherwise, if we are
creating the driver instance with reflection, we kind of implicitly
requires the driver to have a public non-argument constructor. If we
decided to go with this approach, then we will not need
`ExternalResourceDriver#open`.
- Not sure about the necessity of `ExternalResourceDriver#close`. I would
suggest to avoid introduce more interfaces if not absolutely necessary.
- `ExternalResourceDriver#retrieveResourceInfo` should not take
`ResourceProfile` as argument. This exposes more information than it needs.
In addition, it requires the runtime/core to understand how to properly
wrap the external resource into `ResourceProfile`. E.g.,
`ResourceProfile#extendedResources` takes `Resource`, which is an abstract
class. Runtime/core has to known which implementation of `Resource` to use.
- Do we really need `ExternalResourceInfo#getInformation`? I think it
should be good enough to make `ExternalResourceInfo` an empty interface.
User can define their own `ExternalResourceInfo` implementation and how it
is used by the operator user codes.

Thank you~

Xintong Song



On Thu, Apr 9, 2020 at 2:25 PM Becket Qin  wrote:

> +1
>
> Thanks for driving this effort, Ynagze. The latest FLIP wiki looks good to
> me.
>
> Cheers,
>
> Jiangjie (Becket) Qin
>
> On Wed, Apr 8, 2020 at 4:10 PM Yangze Guo  wrote:
>
> > Edit: RuntimeContext interface
> > From: Map>
> > getExternalResourceInfo(ResourceSpec resourceSpec);
> > To: Map> getExternalResourceInfo();
> >
> > Best,
> > Yangze Guo
> >
> >
> > On Wed, Apr 8, 2020 at 11:36 AM Yangze Guo  wrote:
> > >
> > > Hi, there
> > >
> > > I have updated the FLIP, mainly target to make it more detailed and
> > > clear. The general design is not changed, but there are still some
> > > changes need to be notified here:
> > >
> > > - Change the `ExternalResourceDriver` from abstract class to
> > > interface, since it does not have an abstract implementation. Add life
> > > cycle method `open` and `close`.
> > >
> > > - Specify the method added to the RuntimeContext from which user get
> > > the information of external resources.
> > > Map>
> > > getExternalResourceInfo(ResourceSpec resourceSpec);
> > >
> > > - Add "String getInformation()" method to `ExternalResourceInfo`
> > interface.
> > >
> > > - Treat adding external resource info to RestAPI/WebUI as a future
> work.
> > >
> > > If you have any new concerns after that change, please mentioned here.
> > > Sorry for disturbing you.
> > >
> > > Best,
> > > Yangze Guo
> > >
> > > On Wed, Apr 8, 2020 at 9:55 AM Yang Wang 
> wrote:
> > > >
> > > > Thanks Yangze for the efforts to support GPU extended resources.
> > > >
> > > > +1 for this FLIP
> > > >
> > > >
> > > > Best,
> > > > Yang
> > > >
> > > > Till Rohrmann  于2020年4月2日周四 下午11:10写道：
> > > >
> > > > > Thanks for driving this effort Yangze.
> > > > >
> > > > > +1
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Wed, Apr 1, 2020 at 12:41 PM Canbin Zheng <
> felixzhen...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Thanks Yangze for driving the initial CPU support!
> > > > > > +1 (non-binding) from my side.
> > > > > >
> > > > > >
> > > > > > Xintong Song  于2020年4月1日周三 下午6:36写道：
> > > > > >
> > > > > > > Thanks Yangze, the FLIP looks good to me.
> > > > > > > +1 (non-binding) from my side.
> > > > > > >
> > > > > > > Thank you~
> > > > > > >
> > > > > > > Xintong Song
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Apr 1, 2020 at 5:22 PM Yangze Guo 
> > wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > I'd like to start the vote of FLIP-108 [1], which adds GPU
> > support in
> > > > > > > > Flink. This FLIP is discussed in the thread[2].
> > > > > > > >
> > > > > > > > The vote will be open for at least 72 hours. Unless there is
> an
> > > > > > > objection,
> > > > > > > > I will try to close it by April 4, 2020 10:00 UTC if we have
> > received
> > > > > > > > sufficient votes.
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink
> > > > > > > > [2]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-108-Add-GPU-support-in-Flink-td38286.html
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yangze Guo
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
>

Re: [VOTE] FLIP-113: Supports Dynamic Table Options for Flink SQL

[Kurt Young]

+1

Best,
Kurt


On Fri, Apr 10, 2020 at 6:01 PM Jark Wu  wrote:

> +1 from my side (binding)
>
> Best,
> Jark
>
> On Fri, 10 Apr 2020 at 17:03, Timo Walther  wrote:
>
> > +1 (binding)
> >
> > Thanks for the healthy discussion. I think this feature can be useful
> > during the development of a pipeline.
> >
> > Regards,
> > Timo
> >
> > On 10.04.20 03:34, Danny Chan wrote:
> > > Hi all,
> > >
> > > I would like to start the vote for FLIP-113 [1], which is discussed and
> > > reached a consensus in the discussion thread [2].
> > >
> > > The vote will be open until April 13nd (72h), unless there is an
> > > objection or not enough votes.
> > >
> > > Best,
> > > Danny Chan
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL
> > > [2]
> > >
> >
> https://lists.apache.org/thread.html/r94af5d3d97e76e7dd9df68cb0becc7ba74d15591a8fab84c72fa%40%3Cdev.flink.apache.org%3E
> > >
> >
> >
>

Re: [VOTE] FLIP-108: Add GPU support in Flink

[Yangze Guo]

Thanks for the feedback, Xintong.

- Should we have a factory interface for `ExternalResourceDriver`, that
takes the configuration and returns a driver instance? Otherwise, if we are
creating the driver instance with reflection, we kind of implicitly
requires the driver to have a public non-argument constructor. If we
decided to go with this approach, then we will not need
`ExternalResourceDriver#open`.

True, we could have an `ExternalResourceDriverFactory`, like
interface ExternalResourceDriverFactory {
ExternalResourceDriver fromConfiguration(Configuration config);
}
Regarding the configuration, the user should provide
"external-resource.{resourceName}.driver-factory.class" instead.

- Not sure about the necessity of `ExternalResourceDriver#close`. I would
suggest to avoid introduce more interfaces if not absolutely necessary.

I add `ExternalResourceDriver#close` in case user needs to clean up
internal states and any other resources. It's true that it may not
absolutely necessary for our GPUDriver. From my side, I'm ok to remove
it.

- `ExternalResourceDriver#retrieveResourceInfo` should not take
`ResourceProfile` as argument. This exposes more information than it needs.
In addition, it requires the runtime/core to understand how to properly
wrap the external resource into `ResourceProfile`. E.g.,
`ResourceProfile#extendedResources` takes `Resource`, which is an abstract
class. Runtime/core has to known which implementation of `Resource` to use.

True, at the moment, I think the amount of the resource is enough for
the `ExternalResourceDriver#retrieveResourceInfo`. In the future, if
the fine-grained external resource management is supported, the amount
of the resource seems to be enough either. If we want to leverage some
external resources which could not be measured by a single long value,
we might enrich this. But I'd like to keep it out of the scope of this
FLIP.

- Do we really need `ExternalResourceInfo#getInformation`? I think it
should be good enough to make `ExternalResourceInfo` an empty interface.
User can define their own `ExternalResourceInfo` implementation and how it
is used by the operator user codes.

Sounds good.

Best,
Yangze Guo

On Fri, Apr 10, 2020 at 6:04 PM Xintong Song  wrote:
>
> Sorry to pull this back. I have some concerns about the recent updated
> interface details.
>
> - Should we have a factory interface for `ExternalResourceDriver`, that
> takes the configuration and returns a driver instance? Otherwise, if we are
> creating the driver instance with reflection, we kind of implicitly
> requires the driver to have a public non-argument constructor. If we
> decided to go with this approach, then we will not need
> `ExternalResourceDriver#open`.
> - Not sure about the necessity of `ExternalResourceDriver#close`. I would
> suggest to avoid introduce more interfaces if not absolutely necessary.
> - `ExternalResourceDriver#retrieveResourceInfo` should not take
> `ResourceProfile` as argument. This exposes more information than it needs.
> In addition, it requires the runtime/core to understand how to properly
> wrap the external resource into `ResourceProfile`. E.g.,
> `ResourceProfile#extendedResources` takes `Resource`, which is an abstract
> class. Runtime/core has to known which implementation of `Resource` to use.
> - Do we really need `ExternalResourceInfo#getInformation`? I think it
> should be good enough to make `ExternalResourceInfo` an empty interface.
> User can define their own `ExternalResourceInfo` implementation and how it
> is used by the operator user codes.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Thu, Apr 9, 2020 at 2:25 PM Becket Qin  wrote:
>
> > +1
> >
> > Thanks for driving this effort, Ynagze. The latest FLIP wiki looks good to
> > me.
> >
> > Cheers,
> >
> > Jiangjie (Becket) Qin
> >
> > On Wed, Apr 8, 2020 at 4:10 PM Yangze Guo  wrote:
> >
> > > Edit: RuntimeContext interface
> > > From: Map>
> > > getExternalResourceInfo(ResourceSpec resourceSpec);
> > > To: Map> getExternalResourceInfo();
> > >
> > > Best,
> > > Yangze Guo
> > >
> > >
> > > On Wed, Apr 8, 2020 at 11:36 AM Yangze Guo  wrote:
> > > >
> > > > Hi, there
> > > >
> > > > I have updated the FLIP, mainly target to make it more detailed and
> > > > clear. The general design is not changed, but there are still some
> > > > changes need to be notified here:
> > > >
> > > > - Change the `ExternalResourceDriver` from abstract class to
> > > > interface, since it does not have an abstract implementation. Add life
> > > > cycle method `open` and `close`.
> > > >
> > > > - Specify the method added to the RuntimeContext from which user get
> > > > the information of external resources.
> > > > Map>
> > > > getExternalResourceInfo(ResourceSpec resourceSpec);
> > > >
> > > > - Add "String getInformation()" method to `ExternalResourceInfo`
> > > interface.
> > > >
> > > > - Treat adding external resource info to RestAPI/WebUI as a future
> > work.
> > > >
> > > > If you have any new conc

Re: [DISCUSS] Creating a new repo to host Flink benchmarks

[Yu Li]

+1 for the proposal, thanks for driving this Yun.

Best Regards,
Yu


On Fri, 10 Apr 2020 at 11:25, Dian Fu  wrote:

> +1 for this proposal.
>
> > 在 2020年4月10日，上午10:58，Zhijiang  写道：
> >
> > +1 for the proposal.
> >
> > Best,
> > Zhijiang
> >
> >
> > --
> > From:Robert Metzger 
> > Send Time:2020 Apr. 10 (Fri.) 02:15
> > To:dev 
> > Subject:Re: [DISCUSS] Creating a new repo to host Flink benchmarks
> >
> > +1 on creating the repo.
> >
> >
> > On Thu, Apr 9, 2020 at 5:54 PM Till Rohrmann 
> wrote:
> >
> >> I think it is a good idea to make the benchmarks available to the
> community
> >> via a repo under the Apache project and to make updating it part of the
> >> release process. Hence +1 for the proposal.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Thu, Apr 9, 2020 at 4:01 PM Piotr Nowojski 
> wrote:
> >>
> >>> Hi Yun Tang,
> >>>
> >>> Thanks for proposing the idea. Since we can not include benchmarks in
> the
> >>> Flink repository what you are proposing is the second best option.
> >>>
> >>> +1 from my side for the proposal.
> >>>
> >>> I think benchmarks have proven their value to justify this.
> >>>
> >>> Piotrek
> >>>
>  On 9 Apr 2020, at 08:56, Yun Tang  wrote:
> 
>  Hi Flink devs,
> 
>  As Flink develops rapidly with more and more features added, how to
> >>> ensure no performance regression existed has become more and more
> >>> important. And we would like to create a new repo under apache project
> to
> >>> host previous flink-benchmarks [1] repo, which is inspired when we
> >> discuss
> >>> under FLINK-16850 [2]
> 
>  Some background context on flink-benchmarks, for those who are not
> >>> familiar with the project yet:
> 
>  - Current flink-benchmarks does not align with the Flink release,
> which
> >>> lead developers not easy to verify
>   performance at specific Flink version because current
> >> flink-benchmarks
> >>> always depends on the latest interfaces.
>  - Above problem could be solved well if we could ensure
> >> flink-benchmarks
> >>> also create release branch when we
>   releasing Flink. However, current flink-benchmarks repo is hosted
> >>> under dataArtisans (the former name of
>   ververica) project, which is not involved in Flink release manual
> >> [3].
> >>> We propose to promote this repo under
>   apache project so that release manager could have the right to
> >> release
> >>> on flink-benchmarks.
>  - The reason why we not involve flink-benchmarks into the apache/flink
> >>> repo is because it heavily depends on
>   JMH [4], which is under GPLv2 license.
> 
>  What do you think?
> 
>  Best,
>  Yun Tang
> 
>  [1] https://github.com/dataArtisans/flink-benchmarks
>  [2] https://issues.apache.org/jira/browse/FLINK-16850
>  [3]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release
>  [4] https://openjdk.java.net/projects/code-tools/jmh/
> 
> 
> >>>
> >>>
> >>
> >
>
>

[jira] [Created] (FLINK-17087) Use constant port for rest.port when it's set as 0 on Kubernetes

[Canbin Zheng (Jira)]

Canbin Zheng created FLINK-17087:


 Summary: Use constant port for rest.port when it's set as 0 on 
Kubernetes
 Key: FLINK-17087
 URL: https://issues.apache.org/jira/browse/FLINK-17087
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Kubernetes
Reporter: Canbin Zheng
 Fix For: 1.11.0


If people set {{rest.port}} to 0 when deploying a native K8s session cluster as 
the following command does,

 
{code:java}
./bin/kubernetes-session.sh -Dkubernetes.cluster-id=felix1 -Drest.port=0 ...
{code}
the submission client will throw an Exception as follows:

 
{quote}org.apache.flink.client.deployment.ClusterDeploymentException: Could not 
create Kubernetes cluster felix1
 at 
org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployClusterInternal(KubernetesClusterDescriptor.java:189)
 at 
org.apache.flink.kubernetes.KubernetesClusterDescriptor.deploySessionCluster(KubernetesClusterDescriptor.java:129)
 at 
org.apache.flink.kubernetes.cli.KubernetesSessionCli.run(KubernetesSessionCli.java:108)
 at 
org.apache.flink.kubernetes.cli.KubernetesSessionCli.lambda$main$0(KubernetesSessionCli.java:185)
 at 
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
 at 
org.apache.flink.kubernetes.cli.KubernetesSessionCli.main(KubernetesSessionCli.java:185)

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure 
executing: POST at: https://xxx/apis/apps/v1/namespaces/default/deployments. 
Message: Deployment.apps "felix1" is invalid: 
spec.template.spec.containers[0].ports[0].containerPort: Required value. 
Received status: Status(apiVersion=v1, code=422, 
details=StatusDetails(causes=[StatusCause(field=spec.template.spec.containers[0].ports[0].containerPort,
 message=Required value, reason=FieldValueRequired, additionalProperties={})], 
group=apps, kind=Deployment, name=felix1, retryAfterSeconds=null, uid=null, 
additionalProperties={}), kind=Status, message=Deployment.apps "felix1" is 
invalid: spec.template.spec.containers[0].ports[0].containerPort: Required 
value, metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, 
additionalProperties={}), reason=Invalid, status=Failure, 
additionalProperties={}).
 at 
io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510)
 at 
io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:449)
 at 
io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413)
 at 
io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372)
 at 
io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:241)
 at 
io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:798)
 at 
io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:328)
 at 
io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:324)
 at 
org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.createJobManagerComponent(Fabric8FlinkKubeClient.java:83)
 at 
org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployClusterInternal(KubernetesClusterDescriptor.java:184)
 ... 5 more
{quote}
 

As we can see, the exception message is unintuitive and may confuse a variety 
of users. 

Therefore, this ticket proposes to use a fixed port instead if people set it as 
0, like what we have done for the {{blob.server.port}} and the 
{{taskmanager.rpc.port}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-124: Add open/close and Collector to (De)SerializationSchema

[Jark Wu]

Hi Xiaogang,

I think this proposal doesn't conflict with your use case, you can still
chain a ProcessFunction after a source which emits raw data.
But I'm not in favor of chaining ProcessFunction after source, and we
should avoid that, because:

1) For correctness, it is necessary to perform the watermark generation as
early as possible in order to be close to the actual data
 generation within a source's data partition. This is also the purpose of
per-partition watermark and event-time alignment.
 Many on going FLIPs (e.g. FLIP-27, FLIP-95) works a lot on this effort.
Deseriazing records and generating watermark in chained
 ProcessFunction makes it difficult to do per-partition watermark in the
future.
2) In Flink SQL, a source should emit the deserialized row instead of raw
data. Otherwise, users have to define raw byte[] as the
 single column of the defined table, and parse them in queries, which is
very inconvenient.

Best,
Jark

On Fri, 10 Apr 2020 at 09:18, SHI Xiaogang  wrote:

> Hi,
>
> I don't think the proposal is a good solution to the problems. I am in
> favour of using a ProcessFunction chained to the source/sink function to
> serialize/deserialize the records, instead of embedding (de)serialization
> schema in source/sink function.
>
> Message packing is heavily used in our production environment to allow
> compression and improve throughput. As buffered messages have to be
> delivered when the time exceeds the limit, timers are also required in our
> cases. I think it's also a common need for other users.
>
> In the this proposal, with more components added into the context, in the
> end we will find the serialization/deserialization schema is just another
> wrapper of ProcessFunction.
>
> Regards,
> Xiaogang
>
> Aljoscha Krettek  于2020年4月7日周二 下午6:34写道：
>
> > On 07.04.20 08:45, Dawid Wysakowicz wrote:
> >
> > > @Jark I was aware of the implementation of SinkFunction, but it was a
> > > conscious choice to not do it that way.
> > >
> > > Personally I am against giving a default implementation to both the new
> > > and old methods. This results in an interface that by default does
> > > nothing or notifies the user only in the runtime, that he/she has not
> > > implemented a method of the interface, which does not sound like a good
> > > practice to me. Moreover I believe the method without a Collector will
> > > still be the preferred method by many users. Plus it communicates
> > > explicitly what is the minimal functionality required by the interface.
> > > Nevertheless I am happy to hear other opinions.
> >
> > Dawid and I discussed this before. I did the extension of the
> > SinkFunction but by now I think it's better to do it this way, because
> > otherwise you can implement the interface without implementing any
> methods.
> >
> > > @all I also prefer the buffering approach. Let's wait a day or two more
> > > to see if others think differently.
> >
> > I'm also in favour of buffering outside the lock.
> >
> > Also, +1 to this FLIP.
> >
> > Best,
> > Aljoscha
> >
>

[jira] [Created] (FLINK-17088) Search TaskManager log content based on keywords function

[pingle wang (Jira)]

pingle wang created FLINK-17088:
---

 Summary: Search TaskManager log content based on keywords function
 Key: FLINK-17088
 URL: https://issues.apache.org/jira/browse/FLINK-17088
 Project: Flink
  Issue Type: New Feature
  Components: Runtime / REST, Runtime / Web Frontend
Affects Versions: 1.10.0
Reporter: pingle wang
 Fix For: 1.11.0
 Attachments: image-2020-04-11-01-30-09-551.png, 
image-2020-04-11-01-30-21-681.png, image-2020-04-11-01-30-38-046.png

Usually the taskmanager log file has a lot of content. There are several ways 
to search for content keywords:
The first one: in browser ctrl + f
The second type: use shell command operations on the server ip, such as grep -A
The third kind: Log files are connected to Elasticsearch through Logstash / 
Filebeat

These all increase the user's operation, which brings inconvenience, so the 
search function is developed to query the log content.


Description:
 1. web input box word means search keyword, the default value is empty
2. The input box lines indicates how many lines follow the keyword, similar 
to 'grep -A 10 Exception', lines default value is 0.
 3. If the search returns empty, display as 'No content matching the search 
keywords'

 

Function display:

!image-2020-04-11-01-30-09-551.png!

!image-2020-04-11-01-30-21-681.png!

!image-2020-04-11-01-30-38-046.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [PROPOSAL] Contribute training materials to Apache Flink

[Eduardo Winpenny Tejedor]

+1 to the idea, David regarding your question, could it go in a
quickstart/get started with Flink section? Next to or replacing the current
'Getting started' section?

On Thu, 9 Apr 2020, 22:11 Seth Wiesman,  wrote:

> Hi David,
>
> +1 to add to the project.
>
> I agree that flink.apache.org and flink playgrounds respectively are the
> best places to host this content.
>
> On Thu, Apr 9, 2020 at 2:56 PM Niels Basjes  wrote:
>
>> Hi,
>>
>> Sounds like a very nice thing to have as part of the project ecosystem.
>>
>> Niels
>>
>> On Thu, Apr 9, 2020 at 8:10 PM David Anderson 
>> wrote:
>>
>>> Dear Flink Community!
>>>
>>> For some time now Ververica has been hosting some freely available Apache
>>> Flink training materials at https://training.ververica.com. This
>>> includes
>>> tutorial content covering the core concepts of the DataStream API, and
>>> hands-on exercises that accompany those explanations.
>>>
>>> Website: https://training.ververica.com
>>> Website repo: https://github.com/dataArtisans/flink-training
>>> Exercises: repo: https://github.com/ververica/flink-training-exercises
>>>
>>> We would like to contribute this training content to Apache Flink. By
>>> doing
>>> so, we hope to make it even easier for folks to get started with Flink.
>>> Especially during this time when so many are working from home, we'd like
>>> to get this self-paced training course out where more people will see it.
>>>
>>> If the community wants these training materials, then this also raises
>>> the
>>> question of where to put them. We are thinking it would be best to
>>> integrate the website content into flink.apache.org, and to add the
>>> exercises to flink-playgrounds -- but these points can be discussed
>>> separately once we've established that the community wants this content.
>>>
>>> Looking forward to hearing what you think!
>>>
>>> Best regards,
>>> David
>>>
>>
>>
>> --
>> Best regards / Met vriendelijke groeten,
>>
>> Niels Basjes
>>
>

[jira] [Created] (FLINK-17089) Checkpoint fail because RocksDBException: Error While opening a file for sequentially reading

[Lu Niu (Jira)]

Lu Niu created FLINK-17089:
--

 Summary: Checkpoint fail because RocksDBException: Error While 
opening a file for sequentially reading
 Key: FLINK-17089
 URL: https://issues.apache.org/jira/browse/FLINK-17089
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Reporter: Lu Niu


we use incremental rocksdb state backend. Flink job checkpoint throws following 
exception after running for about 20 hours:
{code:java}
Caused by: org.rocksdb.RocksDBException: While opening a file for sequentially 
reading: 
/data/foo/bar/usercache/xxx/appcache/application_1584397637704_9072/flink-io-4e2294f0-7e9b-4102-b079-1089f23c47aa/job_d781983f4967703b0480c7943e8100af_op_KeyedProcessOperator_b9daf26d7397cd4b00184cc833054139__27_60__uuid_dee7e33b-9bce-42f3-909a-f6fa4ab52d8c/db/MANIFEST-06:
 No such file or directory at org.rocksdb.Checkpoint.createCheckpoint(Native 
Method)   at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:51)  
at 
org.apache.flink.contrib.streaming.state.snapshot.RocksIncrementalSnapshotStrategy.takeDBNativeCheckpoint(RocksIncrementalSnapshotStrategy.java:249)
 at 
org.apache.flink.contrib.streaming.state.snapshot.RocksIncrementalSnapshotStrategy.doSnapshot(RocksIncrementalSnapshotStrategy.java:160)
 at 
org.apache.flink.contrib.streaming.state.snapshot.RocksDBSnapshotStrategyBase.snapshot(RocksDBSnapshotStrategyBase.java:126)
 at 
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:439)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:411)
   ... 17 more
{code}
This failure consistent happens until the job restarts.

Some findings:

Jobmanager log shows each time the error came from different subTask:
{code:java}
// grep jobManager log on appcache/application_1584397637704_9622
Caused by: org.rocksdb.RocksDBException: While opening a file for sequentially 
reading: 
/data/nvme3n1/nm-local-dir/usercache/dkapoor/appcache/application_1584397637704_9622/flink-io-c42b6665-0170-4dc9-9933-8abd78812fd5/job_03a4b302f44a8d9f5b31693a80bde30c_op_KeyedProcessOperator_b9daf26d7397cd4b00184cc833054139__5_60__uuid_fa8124e4-1678-4555-a90a-8eec4d974a22/db/MANIFEST-06:
 No such file or directory
Caused by: org.rocksdb.RocksDBException: While opening a file for sequentially 
reading: 
/data/nvme3n1/nm-local-dir/usercache/dkapoor/appcache/application_1584397637704_9622/flink-io-a8dfe34d-909e-4aea-8d20-c89199b20856/job_03a4b302f44a8d9f5b31693a80bde30c_op_KeyedProcessOperator_b9daf26d7397cd4b00184cc833054139__4_60__uuid_12fc9764-418e-4802-800e-3623e385743f/db/MANIFEST-06:
 No such file or directory
Caused by: org.rocksdb.RocksDBException: While opening a file for sequentially 
reading: 
/data/nvme1n1/nm-local-dir/usercache/dkapoor/appcache/application_1584397637704_9622/flink-io-e98c35d7-586a-4edb-9eba-99c6fd823540/job_03a4b302f44a8d9f5b31693a80bde30c_op_KeyedProcessOperator_b9daf26d7397cd4b00184cc833054139__9_60__uuid_f52a3f02-aa12-4285-b594-b94e1b0f8ba7/db/MANIFEST-06:
 No such file or directory
Caused by: org.rocksdb.RocksDBException: While opening a file for sequentially 
reading: 
/data/nvme3n1/nm-local-dir/usercache/dkapoor/appcache/application_1584397637704_9622/flink-io-a2887f93-1c75-48b1-8b67-72acdc69ce1b/job_03a4b302f44a8d9f5b31693a80bde30c_op_KeyedProcessOperator_b9daf26d7397cd4b00184cc833054139__2_60__uuid_6a8267eb-aa04-48a3-b82f-7b5b9f21c8e0/db/MANIFEST-06:
 No such file or directory
Caused by: org.rocksdb.RocksDBException: While opening a file for sequentially 
reading: 
/data/nvme2n1/nm-local-dir/usercache/dkapoor/appcache/application_1584397637704_9622/flink-io-27e797c3-de39-4140-84e8-b94e640154cc/job_03a4b302f44a8d9f5b31693a80bde30c_op_KeyedProcessOperator_b9daf26d7397cd4b00184cc833054139__1_60__uuid_fde8b198-32d8-4e0c-a412-f316a4fe1e3e/db/MANIFEST-06:
 No such file or directory
Caused by: org.rocksdb.RocksDBException: While opening a file for sequentially 
reading: 
/data/nvme1n1/nm-local-dir/usercache/dkapoor/appcache/application_1584397637704_9622/flink-io-e98c35d7-586a-4edb-9eba-99c6fd823540/job_03a4b302f44a8d9f5b31693a80bde30c_op_KeyedProcessOperator_b9daf26d7397cd4b00184cc833054139__9_60__uuid_f52a3f02-aa12-4285-b594-b94e1b0f8ba7/db/MANIFEST-06:
 No such file or directory
Caused by: org.rocksdb.RocksDBException: While opening a file for sequentially 
reading: 
/data/nvme2n1/nm-local-dir/usercache/dkapoor/appcache/application_1584397637704_9622/flink-io-7be6a975-c0cd-4083-a1c3-b47e4c8fbb1b/job_03a4b302f44a8d9f5b31693a80bde30c_op_KeyedProcessOperator_b9daf26d7397cd4b00184cc833054139__13_60__uuid_d779fe65-181f-40d2-b32e-e17a023c128d/db/MANIFEST-06:
 No such file or directory
Caused by: org.rocksdb.RocksDBException: While opening a file for sequentially 
reading: 
/data/nvme1n1/nm-local-dir/usercache/dkap

Re: [PROPOSAL] Contribute training materials to Apache Flink

[David Anderson]

Yes, the training materials could become part of the docs, as you suggest.
The reason why I'm somewhat resistant to this idea is that the
documentation is versioned. It will be easier to maintain the training
content and its exercises and examples if we only have to be concerned with
having one version (for the latest Flink release), and I don't really see
any added value to the community in keeping older versions of the training
online.


On Fri, Apr 10, 2020 at 8:46 PM Eduardo Winpenny Tejedor <
eduardo.winpe...@gmail.com> wrote:

> +1 to the idea, David regarding your question, could it go in a
> quickstart/get started with Flink section? Next to or replacing the current
> 'Getting started' section?
>
> On Thu, 9 Apr 2020, 22:11 Seth Wiesman,  wrote:
>
>> Hi David,
>>
>> +1 to add to the project.
>>
>> I agree that flink.apache.org and flink playgrounds respectively are the
>> best places to host this content.
>>
>> On Thu, Apr 9, 2020 at 2:56 PM Niels Basjes  wrote:
>>
>>> Hi,
>>>
>>> Sounds like a very nice thing to have as part of the project ecosystem.
>>>
>>> Niels
>>>
>>> On Thu, Apr 9, 2020 at 8:10 PM David Anderson 
>>> wrote:
>>>
 Dear Flink Community!

 For some time now Ververica has been hosting some freely available
 Apache
 Flink training materials at https://training.ververica.com. This
 includes
 tutorial content covering the core concepts of the DataStream API, and
 hands-on exercises that accompany those explanations.

 Website: https://training.ververica.com
 Website repo: https://github.com/dataArtisans/flink-training
 Exercises: repo: https://github.com/ververica/flink-training-exercises

 We would like to contribute this training content to Apache Flink. By
 doing
 so, we hope to make it even easier for folks to get started with Flink.
 Especially during this time when so many are working from home, we'd
 like
 to get this self-paced training course out where more people will see
 it.

 If the community wants these training materials, then this also raises
 the
 question of where to put them. We are thinking it would be best to
 integrate the website content into flink.apache.org, and to add the
 exercises to flink-playgrounds -- but these points can be discussed
 separately once we've established that the community wants this content.

 Looking forward to hearing what you think!

 Best regards,
 David

>>>
>>>
>>> --
>>> Best regards / Met vriendelijke groeten,
>>>
>>> Niels Basjes
>>>
>>

Re: [VOTE] FLIP-108: Add GPU support in Flink

[Yangze Guo]

Hi there,
I've updated the FLIP accordingly. Please take a look. If you have any
further concerns please let me know.

Best,
Yangze Guo

On Fri, Apr 10, 2020 at 6:40 PM Yangze Guo  wrote:
>
> Thanks for the feedback, Xintong.
>
> - Should we have a factory interface for `ExternalResourceDriver`, that
> takes the configuration and returns a driver instance? Otherwise, if we are
> creating the driver instance with reflection, we kind of implicitly
> requires the driver to have a public non-argument constructor. If we
> decided to go with this approach, then we will not need
> `ExternalResourceDriver#open`.
>
> True, we could have an `ExternalResourceDriverFactory`, like
> interface ExternalResourceDriverFactory {
> ExternalResourceDriver fromConfiguration(Configuration config);
> }
> Regarding the configuration, the user should provide
> "external-resource.{resourceName}.driver-factory.class" instead.
>
> - Not sure about the necessity of `ExternalResourceDriver#close`. I would
> suggest to avoid introduce more interfaces if not absolutely necessary.
>
> I add `ExternalResourceDriver#close` in case user needs to clean up
> internal states and any other resources. It's true that it may not
> absolutely necessary for our GPUDriver. From my side, I'm ok to remove
> it.
>
> - `ExternalResourceDriver#retrieveResourceInfo` should not take
> `ResourceProfile` as argument. This exposes more information than it needs.
> In addition, it requires the runtime/core to understand how to properly
> wrap the external resource into `ResourceProfile`. E.g.,
> `ResourceProfile#extendedResources` takes `Resource`, which is an abstract
> class. Runtime/core has to known which implementation of `Resource` to use.
>
> True, at the moment, I think the amount of the resource is enough for
> the `ExternalResourceDriver#retrieveResourceInfo`. In the future, if
> the fine-grained external resource management is supported, the amount
> of the resource seems to be enough either. If we want to leverage some
> external resources which could not be measured by a single long value,
> we might enrich this. But I'd like to keep it out of the scope of this
> FLIP.
>
> - Do we really need `ExternalResourceInfo#getInformation`? I think it
> should be good enough to make `ExternalResourceInfo` an empty interface.
> User can define their own `ExternalResourceInfo` implementation and how it
> is used by the operator user codes.
>
> Sounds good.
>
> Best,
> Yangze Guo
>
> On Fri, Apr 10, 2020 at 6:04 PM Xintong Song  wrote:
> >
> > Sorry to pull this back. I have some concerns about the recent updated
> > interface details.
> >
> > - Should we have a factory interface for `ExternalResourceDriver`, that
> > takes the configuration and returns a driver instance? Otherwise, if we are
> > creating the driver instance with reflection, we kind of implicitly
> > requires the driver to have a public non-argument constructor. If we
> > decided to go with this approach, then we will not need
> > `ExternalResourceDriver#open`.
> > - Not sure about the necessity of `ExternalResourceDriver#close`. I would
> > suggest to avoid introduce more interfaces if not absolutely necessary.
> > - `ExternalResourceDriver#retrieveResourceInfo` should not take
> > `ResourceProfile` as argument. This exposes more information than it needs.
> > In addition, it requires the runtime/core to understand how to properly
> > wrap the external resource into `ResourceProfile`. E.g.,
> > `ResourceProfile#extendedResources` takes `Resource`, which is an abstract
> > class. Runtime/core has to known which implementation of `Resource` to use.
> > - Do we really need `ExternalResourceInfo#getInformation`? I think it
> > should be good enough to make `ExternalResourceInfo` an empty interface.
> > User can define their own `ExternalResourceInfo` implementation and how it
> > is used by the operator user codes.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Thu, Apr 9, 2020 at 2:25 PM Becket Qin  wrote:
> >
> > > +1
> > >
> > > Thanks for driving this effort, Ynagze. The latest FLIP wiki looks good to
> > > me.
> > >
> > > Cheers,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Wed, Apr 8, 2020 at 4:10 PM Yangze Guo  wrote:
> > >
> > > > Edit: RuntimeContext interface
> > > > From: Map>
> > > > getExternalResourceInfo(ResourceSpec resourceSpec);
> > > > To: Map> getExternalResourceInfo();
> > > >
> > > > Best,
> > > > Yangze Guo
> > > >
> > > >
> > > > On Wed, Apr 8, 2020 at 11:36 AM Yangze Guo  wrote:
> > > > >
> > > > > Hi, there
> > > > >
> > > > > I have updated the FLIP, mainly target to make it more detailed and
> > > > > clear. The general design is not changed, but there are still some
> > > > > changes need to be notified here:
> > > > >
> > > > > - Change the `ExternalResourceDriver` from abstract class to
> > > > > interface, since it does not have an abstract implementation. Add life
> > > > > cycle method `open` and `close`.
> > > > >
> > > > > - Specify the method

Re: [VOTE] FLIP-108: Add GPU support in Flink

[Becket Qin]

Good feedback form Xintong. The latest FLIP looks good to me.

Thanks,

Jiangjie (Becket) Qin

On Sat, Apr 11, 2020 at 9:20 AM Yangze Guo  wrote:

> Hi there,
> I've updated the FLIP accordingly. Please take a look. If you have any
> further concerns please let me know.
>
> Best,
> Yangze Guo
>
> On Fri, Apr 10, 2020 at 6:40 PM Yangze Guo  wrote:
> >
> > Thanks for the feedback, Xintong.
> >
> > - Should we have a factory interface for `ExternalResourceDriver`, that
> > takes the configuration and returns a driver instance? Otherwise, if we
> are
> > creating the driver instance with reflection, we kind of implicitly
> > requires the driver to have a public non-argument constructor. If we
> > decided to go with this approach, then we will not need
> > `ExternalResourceDriver#open`.
> >
> > True, we could have an `ExternalResourceDriverFactory`, like
> > interface ExternalResourceDriverFactory {
> > ExternalResourceDriver fromConfiguration(Configuration config);
> > }
> > Regarding the configuration, the user should provide
> > "external-resource.{resourceName}.driver-factory.class" instead.
> >
> > - Not sure about the necessity of `ExternalResourceDriver#close`. I would
> > suggest to avoid introduce more interfaces if not absolutely necessary.
> >
> > I add `ExternalResourceDriver#close` in case user needs to clean up
> > internal states and any other resources. It's true that it may not
> > absolutely necessary for our GPUDriver. From my side, I'm ok to remove
> > it.
> >
> > - `ExternalResourceDriver#retrieveResourceInfo` should not take
> > `ResourceProfile` as argument. This exposes more information than it
> needs.
> > In addition, it requires the runtime/core to understand how to properly
> > wrap the external resource into `ResourceProfile`. E.g.,
> > `ResourceProfile#extendedResources` takes `Resource`, which is an
> abstract
> > class. Runtime/core has to known which implementation of `Resource` to
> use.
> >
> > True, at the moment, I think the amount of the resource is enough for
> > the `ExternalResourceDriver#retrieveResourceInfo`. In the future, if
> > the fine-grained external resource management is supported, the amount
> > of the resource seems to be enough either. If we want to leverage some
> > external resources which could not be measured by a single long value,
> > we might enrich this. But I'd like to keep it out of the scope of this
> > FLIP.
> >
> > - Do we really need `ExternalResourceInfo#getInformation`? I think it
> > should be good enough to make `ExternalResourceInfo` an empty interface.
> > User can define their own `ExternalResourceInfo` implementation and how
> it
> > is used by the operator user codes.
> >
> > Sounds good.
> >
> > Best,
> > Yangze Guo
> >
> > On Fri, Apr 10, 2020 at 6:04 PM Xintong Song 
> wrote:
> > >
> > > Sorry to pull this back. I have some concerns about the recent updated
> > > interface details.
> > >
> > > - Should we have a factory interface for `ExternalResourceDriver`, that
> > > takes the configuration and returns a driver instance? Otherwise, if
> we are
> > > creating the driver instance with reflection, we kind of implicitly
> > > requires the driver to have a public non-argument constructor. If we
> > > decided to go with this approach, then we will not need
> > > `ExternalResourceDriver#open`.
> > > - Not sure about the necessity of `ExternalResourceDriver#close`. I
> would
> > > suggest to avoid introduce more interfaces if not absolutely necessary.
> > > - `ExternalResourceDriver#retrieveResourceInfo` should not take
> > > `ResourceProfile` as argument. This exposes more information than it
> needs.
> > > In addition, it requires the runtime/core to understand how to properly
> > > wrap the external resource into `ResourceProfile`. E.g.,
> > > `ResourceProfile#extendedResources` takes `Resource`, which is an
> abstract
> > > class. Runtime/core has to known which implementation of `Resource` to
> use.
> > > - Do we really need `ExternalResourceInfo#getInformation`? I think it
> > > should be good enough to make `ExternalResourceInfo` an empty
> interface.
> > > User can define their own `ExternalResourceInfo` implementation and
> how it
> > > is used by the operator user codes.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Thu, Apr 9, 2020 at 2:25 PM Becket Qin 
> wrote:
> > >
> > > > +1
> > > >
> > > > Thanks for driving this effort, Ynagze. The latest FLIP wiki looks
> good to
> > > > me.
> > > >
> > > > Cheers,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Wed, Apr 8, 2020 at 4:10 PM Yangze Guo 
> wrote:
> > > >
> > > > > Edit: RuntimeContext interface
> > > > > From: Map>
> > > > > getExternalResourceInfo(ResourceSpec resourceSpec);
> > > > > To: Map>
> getExternalResourceInfo();
> > > > >
> > > > > Best,
> > > > > Yangze Guo
> > > > >
> > > > >
> > > > > On Wed, Apr 8, 2020 at 11:36 AM Yangze Guo 
> wrote:
> > > > > >
> > > > > > Hi, there
> > > > > >
> > > > > > I have updated the FLIP

Re: [DISCUSS] FLIP-71 - E2E View support in Flink SQL

[Jark Wu]

Sorry for the late reply,

I have some concern around "Supporting SHOW VIEWS|DESCRIBE VIEW name".
Currently, in SQL CLI, the "SHOW TABLES" will also list views and "DESCRIBE
name" can also describe a view.
Shall we remove the view support in those commands if we want to support a
dedicate "SHOW VIEWS|DESCRIBE VIEW name"?

Brest,
Jark

On Wed, 8 Apr 2020 at 23:49, Timo Walther  wrote:

> I didn't know that. We should definitely implement this asap. Please
> open a JIRA issue.
>
> Thanks,
> Timo
>
>
> On 08.04.20 14:29, Zhenghua Gao wrote:
> > Hi Timo,
> >
> > Actually "TEMPORARY" is not supported in table DDL now.
> > But you are right I could support "CREATE TEMPORARY VIEW" in this FLIP.
> > And may be we should open a separate JIRA ticket to track supporting it
> in
> > table DDL？
> >
> > *Best Regards,*
> > *Zhenghua Gao*
> >
> >
> > On Wed, Apr 8, 2020 at 7:48 PM Timo Walther  wrote:
> >
> >> Hi Zhenghua,
> >>
> >> FLINK-10232 is quite old and a lot of stuff was discussed and agreed on
> >> since then. I don't like to postpone the 'TEMPORARY' keyword because it
> >> is a important concept that is already part of the Table API (see
> >> TableEnvironment.createTemporaryView) and in function DDL and table DDL.
> >> It is not complicated to supported it in this FLIP and just a couple of
> >> line of code more.
> >>
> >> Regards,
> >> Timo
> >>
> >> On 08.04.20 11:27, Zhenghua Gao wrote:
> >>> Another concern about "CREATE DDL" is:
> >>>
> >>> FLINK-10232 proposes using "IF NOT EXISTS" to control the behavior
> when a
> >>> view or table with the same name already exists.
> >>> And "OR REPLACE" for type/library/function DDL.
> >>>
> >>> @godfrey he  I will keep the "IF NOT EXISTS"
> syntax
> >>> and postpone the "OR REPLACE" syntax until we need it.
> >>>
> >>>
> >>> *Best Regards,*
> >>> *Zhenghua Gao*
> >>>
> >>>
> >>> On Wed, Apr 8, 2020 at 4:46 PM Zhenghua Gao  wrote:
> >>>
>  Hi Timo,
> 
>  Shall we postpone the support of 'TEMPORARY' keyword since it's not
>  mentioned in FLINK-10232?
>  
> 
>  *Best Regards,*
>  *Zhenghua Gao*
> 
> 
>  On Wed, Apr 8, 2020 at 3:30 PM Timo Walther 
> wrote:
> 
> > Hi Zhenghua,
> >
> > VIEWS should also support the TEMPORARY keyword according to FLIP-64.
> >
> > Otherwise the FLIP looks good to me.
> >
> > Regards,
> > Timo
> >
> >
> > On 08.04.20 07:31, Zhenghua Gao wrote:
> >> @Danny Chan   you‘re right. I have updated
> the
> > doc.
> >>
> >> *Best Regards,*
> >> *Zhenghua Gao*
> >>
> >>
> >> On Wed, Apr 8, 2020 at 1:20 PM Danny Chan 
> >> wrote:
> >>
> >>> +1 for the proposal, a small concern for drop view statement:
> >>>
> >>> dropViewStatement:
> >>>  DROP VIEW name [ IF EXISTS ]
> >>> I think the drop statement should be
> >>> DROP VIEW [ IF EXISTS ] name
> >>>
> >>> Best,
> >>> Danny Chan
> >>> 在 2020年4月8日 +0800 AM11:54，Kurt Young ，写道：
>  This FLIP seems to be quite straightforward, +1 from my side.
> 
>  Best,
>  Kurt
> 
> 
>  On Tue, Apr 7, 2020 at 8:42 PM Zhenghua Gao 
> >> wrote:
> 
> > forward the reply to ML too.
> >
> >
> > *Best Regards,*
> > *Zhenghua Gao*
> >
> >
> > -- Forwarded message -
> > From: Zhenghua Gao 
> > Date: Tue, Apr 7, 2020 at 8:40 PM
> > Subject: Re: [DISCUSS] FLIP-71 - E2E View support in Flink SQL
> > To: godfrey he 
> >
> >
> >>> regarding to "Interoperability between Flink and Hive is not
> > guaranteed", can you explain this more?
> > We have several limitations of interoperability between flink
> >> objects
> >>> and
> > hive objects (tables, functions, etc).
> > So we don't promise the interoperability of views between flink
> and
> >>> hive
> > since a view is defined base on these objects.
> >
> >>> "CREATE VIEW [ IF NOT EXISTS ]"
> > This should be "CREATE VIEW [OR REPLACE]".
> >
> >>> "DESC"
> > It's a shortcut of "DESCRIBE" in SQL Client (See desc table xxx).
> > In DDL, we should only support "SHOW VIEWS" and "DESCRIBE VIEW
> >> xxx".
> >
> > I have updated the design doc, thanks.
> >
> > *Best Regards,*
> > *Zhenghua Gao*
> >
> >
> > On Tue, Apr 7, 2020 at 8:09 PM godfrey he 
> > wrote:
> >
> >> Hi Zhenghua,
> >>
> >> Thanks for driving this. It's one step forward that
> >> TableEnvironment
> >> supports more complete SQLs.
> >> I have a few minor questions:
> >> 1. regarding to "Interoperability between Flink and Hive is not
> >> guaranteed", can you explain this more?
>

[VOTE] FLIP-105: Support to Interpret and Emit Changelog in Flink SQL

[Jark Wu]

Hi all,

I would like to start the vote for FLIP-105 [1], which is discussed and
reached a consensus in the discussion thread [2].

The vote will be open for at least 72h, unless there is an objection or not
enough votes.

Thanks,
Jark

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-105-Support-to-Interpret-and-Emit-Changelog-in-Flink-SQL-td37665.html

Re: [DISCUSS] FLIP-71 - E2E View support in Flink SQL

[Zhenghua Gao]

Hi Jark,

>> Shall we remove the view support in those commands if we want to support
a
>> dedicate "SHOW VIEWS|DESCRIBE VIEW name"?
Yes, we should correct those commands in SQL client. Will open tickets
after the vote.


*Best Regards,*
*Zhenghua Gao*


On Sat, Apr 11, 2020 at 11:25 AM Jark Wu  wrote:

> Sorry for the late reply,
>
> I have some concern around "Supporting SHOW VIEWS|DESCRIBE VIEW name".
> Currently, in SQL CLI, the "SHOW TABLES" will also list views and "DESCRIBE
> name" can also describe a view.
> Shall we remove the view support in those commands if we want to support a
> dedicate "SHOW VIEWS|DESCRIBE VIEW name"?
>
> Brest,
> Jark
>
> On Wed, 8 Apr 2020 at 23:49, Timo Walther  wrote:
>
> > I didn't know that. We should definitely implement this asap. Please
> > open a JIRA issue.
> >
> > Thanks,
> > Timo
> >
> >
> > On 08.04.20 14:29, Zhenghua Gao wrote:
> > > Hi Timo,
> > >
> > > Actually "TEMPORARY" is not supported in table DDL now.
> > > But you are right I could support "CREATE TEMPORARY VIEW" in this FLIP.
> > > And may be we should open a separate JIRA ticket to track supporting it
> > in
> > > table DDL？
> > >
> > > *Best Regards,*
> > > *Zhenghua Gao*
> > >
> > >
> > > On Wed, Apr 8, 2020 at 7:48 PM Timo Walther 
> wrote:
> > >
> > >> Hi Zhenghua,
> > >>
> > >> FLINK-10232 is quite old and a lot of stuff was discussed and agreed
> on
> > >> since then. I don't like to postpone the 'TEMPORARY' keyword because
> it
> > >> is a important concept that is already part of the Table API (see
> > >> TableEnvironment.createTemporaryView) and in function DDL and table
> DDL.
> > >> It is not complicated to supported it in this FLIP and just a couple
> of
> > >> line of code more.
> > >>
> > >> Regards,
> > >> Timo
> > >>
> > >> On 08.04.20 11:27, Zhenghua Gao wrote:
> > >>> Another concern about "CREATE DDL" is:
> > >>>
> > >>> FLINK-10232 proposes using "IF NOT EXISTS" to control the behavior
> > when a
> > >>> view or table with the same name already exists.
> > >>> And "OR REPLACE" for type/library/function DDL.
> > >>>
> > >>> @godfrey he  I will keep the "IF NOT EXISTS"
> > syntax
> > >>> and postpone the "OR REPLACE" syntax until we need it.
> > >>>
> > >>>
> > >>> *Best Regards,*
> > >>> *Zhenghua Gao*
> > >>>
> > >>>
> > >>> On Wed, Apr 8, 2020 at 4:46 PM Zhenghua Gao 
> wrote:
> > >>>
> >  Hi Timo,
> > 
> >  Shall we postpone the support of 'TEMPORARY' keyword since it's not
> >  mentioned in FLINK-10232?
> >  
> > 
> >  *Best Regards,*
> >  *Zhenghua Gao*
> > 
> > 
> >  On Wed, Apr 8, 2020 at 3:30 PM Timo Walther 
> > wrote:
> > 
> > > Hi Zhenghua,
> > >
> > > VIEWS should also support the TEMPORARY keyword according to
> FLIP-64.
> > >
> > > Otherwise the FLIP looks good to me.
> > >
> > > Regards,
> > > Timo
> > >
> > >
> > > On 08.04.20 07:31, Zhenghua Gao wrote:
> > >> @Danny Chan   you‘re right. I have updated
> > the
> > > doc.
> > >>
> > >> *Best Regards,*
> > >> *Zhenghua Gao*
> > >>
> > >>
> > >> On Wed, Apr 8, 2020 at 1:20 PM Danny Chan 
> > >> wrote:
> > >>
> > >>> +1 for the proposal, a small concern for drop view statement:
> > >>>
> > >>> dropViewStatement:
> > >>>  DROP VIEW name [ IF EXISTS ]
> > >>> I think the drop statement should be
> > >>> DROP VIEW [ IF EXISTS ] name
> > >>>
> > >>> Best,
> > >>> Danny Chan
> > >>> 在 2020年4月8日 +0800 AM11:54，Kurt Young ，写道：
> >  This FLIP seems to be quite straightforward, +1 from my side.
> > 
> >  Best,
> >  Kurt
> > 
> > 
> >  On Tue, Apr 7, 2020 at 8:42 PM Zhenghua Gao 
> > >> wrote:
> > 
> > > forward the reply to ML too.
> > >
> > >
> > > *Best Regards,*
> > > *Zhenghua Gao*
> > >
> > >
> > > -- Forwarded message -
> > > From: Zhenghua Gao 
> > > Date: Tue, Apr 7, 2020 at 8:40 PM
> > > Subject: Re: [DISCUSS] FLIP-71 - E2E View support in Flink SQL
> > > To: godfrey he 
> > >
> > >
> > >>> regarding to "Interoperability between Flink and Hive is not
> > > guaranteed", can you explain this more?
> > > We have several limitations of interoperability between flink
> > >> objects
> > >>> and
> > > hive objects (tables, functions, etc).
> > > So we don't promise the interoperability of views between flink
> > and
> > >>> hive
> > > since a view is defined base on these objects.
> > >
> > >>> "CREATE VIEW [ IF NOT EXISTS ]"
> > > This should be "CREATE VIEW [OR REPLACE]".
> > >
> > >>> "DESC"
> > > It's a shortcut of "DESCRIBE" in SQL Client (See desc table
> xxx).
> > > In DDL, we should only support "S

[jira] [Created] (FLINK-17090) Harden preckeck for the KubernetesConfigOptions.JOB_MANAGER_CPU

[Canbin Zheng (Jira)]

Canbin Zheng created FLINK-17090:


 Summary: Harden preckeck for the 
KubernetesConfigOptions.JOB_MANAGER_CPU
 Key: FLINK-17090
 URL: https://issues.apache.org/jira/browse/FLINK-17090
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Kubernetes
Affects Versions: 1.10.0
Reporter: Canbin Zheng
 Fix For: 1.11.0


If people specify a negative value for the config option of 
{{KubernetesConfigOptions#JOB_MANAGER_CPU}} as what the following command does,
{code:java}
./bin/kubernetes-session.sh -Dkubernetes.jobmanager.cpu=-3.0 
-Dkubernetes.cluster-id=...{code}
then it will throw an exception as follows:
{quote}org.apache.flink.client.deployment.ClusterDeploymentException: Could not 
create Kubernetes cluster "felix1".
 at 
org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployClusterInternal(KubernetesClusterDescriptor.java:192)
 at 
org.apache.flink.kubernetes.KubernetesClusterDescriptor.deploySessionCluster(KubernetesClusterDescriptor.java:129)
 at 
org.apache.flink.kubernetes.cli.KubernetesSessionCli.run(KubernetesSessionCli.java:108)
 at 
org.apache.flink.kubernetes.cli.KubernetesSessionCli.lambda$main$0(KubernetesSessionCli.java:185)
 at 
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
 at 
org.apache.flink.kubernetes.cli.KubernetesSessionCli.main(KubernetesSessionCli.java:185)

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure 
executing: POST at: 
[https://cls-cf5wqdwy.ccs.tencent-cloud.com/apis/apps/v1/namespaces/default/deployments].
 Message: Deployment.apps "felix1" is invalid: 
[spec.template.spec.containers[0].resources.limits[cpu]: Invalid value: "-3": 
must be greater than or equal to 0, 
spec.template.spec.containers[0].resources.requests[cpu]: Invalid value: "-3": 
must be greater than or equal to 0]. Received status: Status(apiVersion=v1, 
code=422, 
details=StatusDetails(causes=[StatusCause(field=spec.template.spec.containers[0].resources.limits[cpu],
 message=Invalid value: "-3": must be greater than or equal to 0, 
reason=FieldValueInvalid, additionalProperties={}), 
StatusCause(field=spec.template.spec.containers[0].resources.requests[cpu], 
message=Invalid value: "-3": must be greater than or equal to 0, 
reason=FieldValueInvalid, additionalProperties={})], group=apps, 
kind=Deployment, name=felix1, retryAfterSeconds=null, uid=null, 
additionalProperties={}), kind=Status, message=Deployment.apps "felix1" is 
invalid: [spec.template.spec.containers[0].resources.limits[cpu]: Invalid 
value: "-3": must be greater than or equal to 0, 
spec.template.spec.containers[0].resources.requests[cpu]: Invalid value: "-3": 
must be greater than or equal to 0], metadata=ListMeta(_continue=null, 
resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, 
status=Failure, additionalProperties={}).
 at 
io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510)
 at 
io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:449)
 at 
io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413)
 at 
io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372)
 at 
io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:241)
 at 
io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:798)
 at 
io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:328)
 at 
io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:324)
 at 
org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.createJobManagerComponent(Fabric8FlinkKubeClient.java:83)
 at 
org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployClusterInternal(KubernetesClusterDescriptor.java:182)
{quote}
Since there is a gap in the configuration model between the flink-side and the 
k8s-side, this ticket proposes to harden precheck in the flink k8s parameters 
parsing tool and throw a more user-friendly exception message like "the value 
of {{kubernetes.jobmanager.cpu}} must be greater than or equal to 0".



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17091) AvroRowSerializationSchema doesn't support new Timestamp conversion classes

[Paul Lin (Jira)]

Paul Lin created FLINK-17091:


 Summary: AvroRowSerializationSchema doesn't support new Timestamp 
conversion classes
 Key: FLINK-17091
 URL: https://issues.apache.org/jira/browse/FLINK-17091
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.10.0
Reporter: Paul Lin


AvroRowSerializationSchema doesn't know how to convert the new physical classes 
of Timestamp (eg. java.time.Date) into Avro's int/long based timestamp.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)