Re: [RESULT][VOTE] Migrate to sponsored Travis account

[Chesnay Schepler]

Yes we can do that; for the time being you can add an empty commit to 
re-trigger the CI.



On 08/07/2019 03:49, Congxian Qiu wrote:

As we used flink bot to trigger the CI test, could we add a command for
flink bot to retrigger the CI(sometimes we may encounter some flaky tests)

Best,
Congxian


Chesnay Schepler  于2019年7月8日周一 上午5:01写道：


The vote has passed unanimously in favor of migrating to a separate
Travis account.

I will now set things up such that no PullRequest is no longer run on
the ASF servers.
This is a major setup in reducing our usage of ASF resources.
For the time being we'll use free Travis plan for flink-ci (i.e. 5
workers, which is the same the ASF gives us). Over the course of the
next week we'll setup the Ververica subscription to increase this limit.

  From now now, a bot will mirror all new and updated PullRequests to a
mirror repository (https://github.com/flink-ci/flink-ci) and write an
update into the PR once the build is complete.
I have ran the bots for the past 3 days in parallel to our existing
Travis and it was working without major issues.

The biggest change that contributors will see is that there's no longer
a icon next to each commit. We may revisit this in the future.

I'll setup a repo with the source of the bot later.

On 04/07/2019 10:46, Chesnay Schepler wrote:

I've raised a JIRA
with INFRA to
inquire whether it would be possible to switch to a different Travis
account, and if so what steps would need to be taken.
We need a proper confirmation from INFRA since we are not in full
control of the flink repository (for example, we cannot access the
settings page).

If this is indeed possible, Ververica is willing sponsor a Travis
account for the Flink project.
This would provide us with more than enough resources than we need.

Since this makes the project more reliant on resources provided by
external companies I would like to vote on this.

Please vote on this proposal, as follows:
[ ] +1, Approve the migration to a Ververica-sponsored Travis account,
provided that INFRA approves
[ ] -1, Do not approach the migration to a Ververica-sponsored Travis
account

The vote will be open for at least 24h, and until we have confirmation
from INFRA. The voting period may be shorter than the usual 3 days
since our current is effectively not working.

On 04/07/2019 06:51, Bowen Li wrote:

Re: > Are they using their own Travis CI pool, or did the switch to
an entirely different CI service?

I reached out to Wes and Krisztián from Apache Arrow PMC. They are
currently moving away from ASF's Travis to their own in-house metal
machines at [1] with custom CI application at [2]. They've seen
significant improvement w.r.t both much higher performance and
basically no resource waiting time, "night-and-day" difference
quoting Wes.

Re: > If we can just switch to our own Travis pool, just for our
project, then this might be something we can do fairly quickly?

I believe so, according to [3] and [4]


[1] https://ci.ursalabs.org/ 
[2] https://github.com/ursa-labs/ursabot
[3]


https://docs.travis-ci.com/user/migrate/open-source-repository-migration

[4]

https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com



On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler mailto:ches...@apache.org>> wrote:

 Are they using their own Travis CI pool, or did the switch to an
 entirely different CI service?

 If we can just switch to our own Travis pool, just for our
 project, then
 this might be something we can do fairly quickly?

 On 03/07/2019 05:55, Bowen Li wrote:
 > I responded in the INFRA ticket [1] that I believe they are
 using a wrong
 > metric against Flink and the total build time is a completely
 different
 > thing than guaranteed build capacity.
 >
 > My response:
 >
 > "As mentioned above, since I started to pay attention to Flink's
 build
 > queue a few tens of days ago, I'm in Seattle and I saw no build
 was kicking
 > off in PST daytime in weekdays for Flink. Our teammates in China
 and Europe
 > have also reported similar observations. So we need to evaluate
 how the
 > large total build time came from - if 1) your number and 2) our
 > observations from three locations that cover pretty much a full
 day, are
 > all true, I **guess** one reason can be that - highly likely the
 extra
 > build time came from weekends when other Apache projects may be
 idle and
 > Flink just drains hard its congested queue.
 >
 > Please be aware of that we're not complaining about the lack of
 resources
 > in general, I'm complaining about the lack of **stable,
dedicated**
 > resources. An example for the latter one is, currently even if
 no build is
 > in Flink's queue and I submit a request to be the queue head in
PST
 > morning, my build won't even start in 6-8

[jira] [Created] (FLINK-13139) Various Hive tests fail on Travis

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13139:
-

 Summary: Various Hive tests fail on Travis
 Key: FLINK-13139
 URL: https://issues.apache.org/jira/browse/FLINK-13139
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive
Affects Versions: 1.9.0
Reporter: Till Rohrmann
 Fix For: 1.9.0


Various Hive related tests fail on Travis:

{code}
06:06:49.654 [ERROR] Errors: 
06:06:49.654 [ERROR]   HiveInputFormatTest.createCatalog:66 » Catalog Failed 
to create Hive Metastore...
06:06:49.654 [ERROR]   HiveTableFactoryTest.init:55 » Catalog Failed to create 
Hive Metastore client
06:06:49.654 [ERROR]   HiveTableOutputFormatTest.createCatalog:72 » Catalog 
Failed to create Hive Met...
06:06:49.654 [ERROR]   HiveTableSinkTest.createCatalog:72 » Catalog Failed to 
create Hive Metastore c...
06:06:49.654 [ERROR]   HiveTableSourceTest.createCatalog:67 » Catalog Failed 
to create Hive Metastore...
06:06:49.654 [ERROR]   HiveCatalogGenericMetadataTest.init:49 » Catalog Failed 
to create Hive Metasto...
06:06:49.654 [ERROR]   HiveCatalogHiveMetadataTest.init:55 » Catalog Failed to 
create Hive Metastore ...
06:06:49.654 [ERROR]   HiveGenericUDFTest.testCeil:193->init:387 » 
ExceptionInInitializer
06:06:49.654 [ERROR]   HiveGenericUDFTest.testDecode:160 » NoClassDefFound 
Could not initialize class...
06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFArray_singleArray:202->init:237 
» NoClassDefFound Cou...
06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFBin:60->init:237 » 
NoClassDefFound Could not initiali...
06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFConv:67->init:237 » 
NoClassDefFound Could not initial...
06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFJson:85->init:237 » 
NoClassDefFound Could not initial...
06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFMinute:126->init:237 » 
ExceptionInInitializer
06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFRand:51->init:237 » 
NoClassDefFound Could not initial...
06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFRegExpExtract:153->init:237 » 
NoClassDefFound Could n...
06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFToInteger:188->init:237 » 
NoClassDefFound Could not i...
06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFUnbase64:166->init:237 » 
NoClassDefFound Could not in...
06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFUnhex:177->init:237 » 
NoClassDefFound Could not initi...
06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFWeekOfYear:139->init:237 » 
NoClassDefFound Could not ...
{code}

https://api.travis-ci.org/v3/job/555252043/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13140) PushFilterIntoTableSourceScanRuleTest.testWithUd fails on Travis

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13140:
-

 Summary: PushFilterIntoTableSourceScanRuleTest.testWithUd fails on 
Travis
 Key: FLINK-13140
 URL: https://issues.apache.org/jira/browse/FLINK-13140
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.9.0
Reporter: Till Rohrmann
 Fix For: 1.9.0


{{PushFilterIntoTableSourceScanRuleTest.testWithUd}} fails on Travis with

{code}
06:22:11.290 [ERROR] Failures: 
06:22:11.290 [ERROR]   PushFilterIntoTableSourceScanRuleTest.testWithUdf:93 
planAfter 
expected:<...ssions$utils$Func1$$[a39386268ffec8461452460bcbe089ad]($2), 32)])
   +- Lo...> but 
was:<...ssions$utils$Func1$$[8a89e0d5a022a06a00c7734a25295ff4]($2), 32)])
   +- Lo...>
{code}

https://api.travis-ci.org/v3/job/555252046/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13141) Remove getBufferSize method from BufferPoolFactory

[zhijiang (JIRA)]

zhijiang created FLINK-13141:


 Summary: Remove getBufferSize method from BufferPoolFactory
 Key: FLINK-13141
 URL: https://issues.apache.org/jira/browse/FLINK-13141
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network
Reporter: zhijiang
Assignee: zhijiang


This is just a refactor work to make the interfacer of BufferPoolFactory more 
simple and clean.

BufferPoolFactory#getBufferSize is only used for creating subpartitions in 
ResultPartitionFactory. We could pass the network buffer size from 
NettyShuffleEnvironmentConfiguration while constructing the 
ResultPartitionFactory, then the interface method getBufferSize could be 
removed form BufferPoolFactory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13142) Fix unstable AggregateITCase#testNestedGroupByAgg IT case

[Jark Wu (JIRA)]

Jark Wu created FLINK-13142:
---

 Summary: Fix unstable AggregateITCase#testNestedGroupByAgg IT case 
 Key: FLINK-13142
 URL: https://issues.apache.org/jira/browse/FLINK-13142
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.9.0
Reporter: Jark Wu






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13143) Clear CheckpointExceptionHandler relevant classes

[vinoyang (JIRA)]

vinoyang created FLINK-13143:


 Summary: Clear CheckpointExceptionHandler relevant classes
 Key: FLINK-13143
 URL: https://issues.apache.org/jira/browse/FLINK-13143
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Checkpointing
Reporter: vinoyang
Assignee: vinoyang


Since FLINK-11662 has been merged, we can clear {{CheckpointExceptionHandler}} 
relevant classes.

{{CheckpointExceptionHandler}} used to implement 
{{setFailOnCheckpointingErrors}}. Now, it has only one implementation which is 
{{DecliningCheckpointExceptionHandler}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13144) Add takeAsList to Table

[Jeff Zhang (JIRA)]

Jeff Zhang created FLINK-13144:
--

 Summary: Add takeAsList to Table
 Key: FLINK-13144
 URL: https://issues.apache.org/jira/browse/FLINK-13144
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Affects Versions: 1.9.0
Reporter: Jeff Zhang


So that user can preview the data easily. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Apply for being a contributor

[hongton122]

Hi,


I want to contribute to Apache Flink.
Would you please give me the contributor permission?
My JIRA ID is XD122, my fullname is Xiang Dong.






Tanks very much;




 





 

[jira] [Created] (FLINK-13145) Run HA dataset/datastream E2E test with new RestartPipelinedRegionStrategy

[Gary Yao (JIRA)]

Gary Yao created FLINK-13145:


 Summary: Run HA dataset/datastream E2E test with new 
RestartPipelinedRegionStrategy
 Key: FLINK-13145
 URL: https://issues.apache.org/jira/browse/FLINK-13145
 Project: Flink
  Issue Type: Task
  Components: Runtime / Coordination, Tests
Affects Versions: 1.9.0
Reporter: Gary Yao
Assignee: Gary Yao
 Fix For: 1.9.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [jira] [Created] (FLINK-13139) Various Hive tests fail on Travis

[不常用邮箱]

Hello all:

I meet this problem with can’t import follow class in test file.
org.apache.flink.table.catalog.hive.HiveCatalog;
org.apache.flink.table.catalog.hive.HiveTestUtils;
org.apache.flink.table.catalog.hive.HivePartitionConfig;
org.apache.flink.table.catalog.hive.HiveCatalogConfig;
org.apache.flink.table.catalog.hive.HiveDatabaseConfig;

And even themselves can’t import each other. Is that some setting wrong with my 
editor?
I search the problem in google, told that may occur when copy files?
Is someone can fix this? 

Thanks.
Louis



-- 
Louis
Email： xu_soft39211...@163.com

> On Jul 8, 2019, at 16:05, Till Rohrmann (JIRA)  wrote:
> 
> Till Rohrmann created FLINK-13139:
> -
> 
> Summary: Various Hive tests fail on Travis
> Key: FLINK-13139
> URL: https://issues.apache.org/jira/browse/FLINK-13139
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Hive
>Affects Versions: 1.9.0
>Reporter: Till Rohrmann
> Fix For: 1.9.0
> 
> 
> Various Hive related tests fail on Travis:
> 
> {code}
> 06:06:49.654 [ERROR] Errors: 
> 06:06:49.654 [ERROR]   HiveInputFormatTest.createCatalog:66 » Catalog Failed 
> to create Hive Metastore...
> 06:06:49.654 [ERROR]   HiveTableFactoryTest.init:55 » Catalog Failed to 
> create Hive Metastore client
> 06:06:49.654 [ERROR]   HiveTableOutputFormatTest.createCatalog:72 » Catalog 
> Failed to create Hive Met...
> 06:06:49.654 [ERROR]   HiveTableSinkTest.createCatalog:72 » Catalog Failed 
> to create Hive Metastore c...
> 06:06:49.654 [ERROR]   HiveTableSourceTest.createCatalog:67 » Catalog Failed 
> to create Hive Metastore...
> 06:06:49.654 [ERROR]   HiveCatalogGenericMetadataTest.init:49 » Catalog 
> Failed to create Hive Metasto...
> 06:06:49.654 [ERROR]   HiveCatalogHiveMetadataTest.init:55 » Catalog Failed 
> to create Hive Metastore ...
> 06:06:49.654 [ERROR]   HiveGenericUDFTest.testCeil:193->init:387 » 
> ExceptionInInitializer
> 06:06:49.654 [ERROR]   HiveGenericUDFTest.testDecode:160 » NoClassDefFound 
> Could not initialize class...
> 06:06:49.654 [ERROR]   
> HiveSimpleUDFTest.testUDFArray_singleArray:202->init:237 » NoClassDefFound 
> Cou...
> 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFBin:60->init:237 » 
> NoClassDefFound Could not initiali...
> 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFConv:67->init:237 » 
> NoClassDefFound Could not initial...
> 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFJson:85->init:237 » 
> NoClassDefFound Could not initial...
> 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFMinute:126->init:237 » 
> ExceptionInInitializer
> 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFRand:51->init:237 » 
> NoClassDefFound Could not initial...
> 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFRegExpExtract:153->init:237 
> » NoClassDefFound Could n...
> 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFToInteger:188->init:237 » 
> NoClassDefFound Could not i...
> 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFUnbase64:166->init:237 » 
> NoClassDefFound Could not in...
> 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFUnhex:177->init:237 » 
> NoClassDefFound Could not initi...
> 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFWeekOfYear:139->init:237 » 
> NoClassDefFound Could not ...
> {code}
> 
> https://api.travis-ci.org/v3/job/555252043/log.txt
> 
> 
> 
> --
> This message was sent by Atlassian JIRA
> (v7.6.3#76005)

Re: [DISCUSS]Support Upsert mode for Streaming Non-window FlatAggregate

[Jark Wu]

Thanks Jincheng,

I think approach 3 there is no ambiguity in the semantics and can better
explain a deterministic result, although it is not so easy to use.
So I'm +1 to approach 3.

Best,
Jark

On Fri, 5 Jul 2019 at 18:13, jincheng sun  wrote:

> Hi,
>
> @Kurt @Jark thanks again for your comments.
>
> @Kurt
> Separating key and non-key for UDTAGG can definitely provide more
> information for the system, however, it will also add more burden to our
> users and bring some code reuse problems. BTW, approach 3 can also be used
> to separate UDTAGG into keyed or non-keyed as we can check whether the key
> list is empty. So from this point of view, we can use approach 3 to solve
> your problem.
>
> @Jark
> It's great that the TopN in Blink can decide the key automatically. But,
> I'd like to point out another case that the keys cannot be decided by the
> system, i.e., can only be decided by the user. For example, for the TopN,
> let's say top1 for better understanding. Support the Top1 outputs three
> columns(rankid, value, seller_name), and the user wants to upsert the
> result either with key of rankid or with the key of rankid+seller_name.
> 1. With the key of rankid: In this case, the user just wants to get the
> top 1 record.
> 2. With the key of rankid+seller_name: In this case, the user wants to get
> all seller_names that have ever belong to top1. This can not be solved by
> the approach 3 if using only one function. However, it is very easy to
> implement with the withKey approach.
>
> Even though, I have thought more clearly about these things and find more
> interesting things that I want to share with you all. For the TopN example
> which I listed above, it may also lead to a problem in which batch and
> streaming are not unified.
>
> To make it worse, the upsert sink is not supported in batch and we even
> don't have any clear implementation plan about how to support upsert on the
> batch, the unification problem for `withKeys` approach becomes hang in
> doubt.
>
> So, to avoid the unification problem, I think we can also use the approach
> 3. It is more rigorous although less flexible compared to the `withKeys`
> approach.
>
> Meanwhile, I will think more about the unification problem later. Maybe
> new ideas about it may come through. :)
>
> Best，
> Jincheng
>
> Jark Wu  于2019年7月5日周五 上午10:48写道：
>
>> Hi Hequn,
>>
>> > If the TopN table aggregate function
>> > outputs three columns(rankid, time, value), either rankid or
>> rankid+time could be
>> > used as the key. Which one to be chosen is more likely to be decided by
>> the user
>> > according to his business.
>> In this case, the TopN table aggregate function should return two sets of
>> unique key, one is "rankid", another is "rankid, time".
>> This will be more align with current TopN node in blink planner and let
>> optimizer to decide which key based on the downstream information (column
>> selection, sink's primary key).
>>
>>
>> Best,
>> Jark
>>
>> On Fri, 5 Jul 2019 at 00:05, Hequn Cheng  wrote:
>>
>>> Hi Kurt and Jark,
>>>
>>> Thanks a lot for your great inputs!
>>>
>>> The keys of the query may not strongly be related to the UDTAGG.
>>> It may also be related to the corresponding scenarios that a user wants
>>> to achieve.
>>>
>>> For example, take TopN again as an example. If the TopN table aggregate
>>> function
>>> outputs three columns(rankid, time, value), either rankid or rankid+time
>>> could be
>>> used as the key. Which one to be chosen is more likely to be decided by
>>> the user
>>> according to his business.
>>>
>>> Best, Hequn
>>>
>>> On Thu, Jul 4, 2019 at 8:11 PM Jark Wu  wrote:
>>>
 Hi jingcheng,

 I agree with Kurt's point. As you said "the user must know the keys of
 the output of UDTAGG clearly".
 If I understand correctly, the key information is strongly relative to
 the UDTAGG implementation.
 Users may call `flatAggregate` on a UDTAGG instance with different keys
 which may result in a wrong result.
 So I think it would be better to couple key information with UDTAGG
 interface (i.e. "Approach 3" in your design doc).

 Regards,
 Jark

 On Thu, 4 Jul 2019 at 18:06, Kurt Young  wrote:

> Hi Jincheng,
>
> Thanks for the clarification. I think you just pointed out my concern
> by
> yourself:
>
> > When a user uses a User-defined table aggregate function (UDTAGG), he
> must understand the behavior of the UDTAGG, including the return type
> and
> the characteristics of the returned data. such as the key fields.
>
> This indicates that the UDTAGG is somehow be classified to different
> types,
> one will no key, one with key information. So the developer of the
> UDTAGG
> should choose which type of this function should be. In this case,
> my question would be, why don't we have explicit information about keys
> such as we split UDTAGG to keyed UDTAGG and non-keyed UDTAGG. So the
> us

[jira] [Created] (FLINK-13146) GET request of: org/apache/avro/avro/1.8.2/avro-1.8.2.jar from google-maven-central failed

[vinoyang (JIRA)]

vinoyang created FLINK-13146:


 Summary: GET request of: org/apache/avro/avro/1.8.2/avro-1.8.2.jar 
from google-maven-central failed
 Key: FLINK-13146
 URL: https://issues.apache.org/jira/browse/FLINK-13146
 Project: Flink
  Issue Type: Test
  Components: Travis
Reporter: vinoyang


{code:java}
10:16:08.407 [ERROR] Failed to execute goal 
org.apache.avro:avro-maven-plugin:1.8.2:schema (default) on project flink-avro: 
Execution default of goal org.apache.avro:avro-maven-plugin:1.8.2:schema 
failed: Plugin org.apache.avro:avro-maven-plugin:1.8.2 or one of its 
dependencies could not be resolved: Failure to transfer 
org.apache.avro:avro:jar:1.8.2 from 
https://maven-central.storage-download.googleapis.com/repos/central/data/ was 
cached in the local repository, resolution will not be reattempted until the 
update interval of google-maven-central has elapsed or updates are forced. 
Original error: Could not transfer artifact org.apache.avro:avro:jar:1.8.2 
from/to google-maven-central 
(https://maven-central.storage-download.googleapis.com/repos/central/data/): 
GET request of: org/apache/avro/avro/1.8.2/avro-1.8.2.jar from 
google-maven-central failed -> [Help 1]
{code}
log details: [https://api.travis-ci.org/v3/job/555691054/log.txt]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Apply for being a contributor

[highfei2011]

Hi ,


   Good evening ! I am from China. I want to contribute to Apache Flink.Would 
you please give me the contributor permission? My JIRA ID is 
highfei2...@126.com, my fullname is Jeff Yang. Thank you very much !
   I am looking forward to hearing from your soon.

  Yours sincerely,

   Jeff Yang

[jira] [Created] (FLINK-13147) Duplicated code in 'if' conditional statement

[lixiaobao (JIRA)]

lixiaobao created FLINK-13147:
-

 Summary:  Duplicated code in 'if' conditional statement
 Key: FLINK-13147
 URL: https://issues.apache.org/jira/browse/FLINK-13147
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive
Affects Versions: 1.8.1, 1.8.0, 1.7.2, 1.6.4, 1.6.3
Reporter: lixiaobao
 Fix For: 1.7.3, 1.9.0


{code:java}
} else if (type.equals(LogicalTypeRoot.VARBINARY)
 || type.equals(LogicalTypeRoot.BINARY)
 || type.equals(LogicalTypeRoot.BINARY)) {
 if (colStat instanceof CatalogColumnStatisticsDataBinary) {
 CatalogColumnStatisticsDataBinary binaryColumnStatsData = 
(CatalogColumnStatisticsDataBinary) colStat;
 BinaryColumnStatsData binaryColumnStats = new 
BinaryColumnStatsData(binaryColumnStatsData.getMaxLength(), 
binaryColumnStatsData.getAvgLength(), binaryColumnStatsData.getNullCount());
 return ColumnStatisticsData.binaryStats(binaryColumnStats);
 }{code}
The class HiveStatsUtil have  duplicated code in 'if' conditional statement.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13148) Expose WindowedStream.sideOutputLateData() from CoGroupedStreams

[Congxian Qiu(klion26) (JIRA)]

Congxian Qiu(klion26) created FLINK-13148:
-

 Summary: Expose WindowedStream.sideOutputLateData() from 
CoGroupedStreams
 Key: FLINK-13148
 URL: https://issues.apache.org/jira/browse/FLINK-13148
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream
Reporter: Congxian Qiu(klion26)
Assignee: Congxian Qiu(klion26)


As FLINK-10050 supported {{alloedLateness}}, but we can not get the side output 
containing the late data, this issue wants to fix it.

For implementation, I want to add an input parameter {{OutputTag}} in 
{{WithWindow}} as following
{code:java}
protected WithWindow(DataStream input1,
DataStream input2,
KeySelector keySelector1,
KeySelector keySelector2,
TypeInformation keyType,
WindowAssigner, W> windowAssigner,
Trigger, ? super W> trigger,
Evictor, ? super W> evictor,
Time allowedLateness,
OutputTage> outputTag) {
  ...
}
{code}
 and add a function sideOutputLateData(OutputTag outputTag) in with Window
{code:java}
public WithWindow sideOutputLateData(OutputTag> outputTag) {
   ...
}
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13149) improve flink ui to make it easier to be used by other projects

[Jie TANG (JIRA)]

Jie TANG created FLINK-13149:


 Summary: improve flink ui to make it easier to be used by other 
projects
 Key: FLINK-13149
 URL: https://issues.apache.org/jira/browse/FLINK-13149
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Web Frontend
Affects Versions: 1.9.0
Reporter: Jie TANG
 Fix For: 1.9.0
 Attachments: Architecture.png, snapshot-1.png, snapshot-2.png

The new web UI looks nice, but there are still some problems when I try to 
integrate it into the other frontend project, I think we can make some changes 
in order to make it easier to be customized.

*These changes will not bring break changes and it will also not affect the 
user interface.*

 
 * Migrate the code for {{app.config.ts}} to {{config.service.ts}} to make it 
configurable in angular DI system.

 * Add property named {{JOB_PREFIX}} to {{config.service.ts}} for 
{{job.service.ts}} and {{metrics.service.ts}} to make them configurable

 * Add optional param to the url to hide menu and header to make it possible 
for users want to embed the flink ui as an iframe in other website

 * Update Angular version to 8.0 (no break changes)

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13150) defaultCatalogName and defaultDatabaseName in TableEnvImpl is not updated after they are updated in TableEnvironment

[Jeff Zhang (JIRA)]

Jeff Zhang created FLINK-13150:
--

 Summary: defaultCatalogName and defaultDatabaseName in 
TableEnvImpl is not updated after they are updated in TableEnvironment
 Key: FLINK-13150
 URL: https://issues.apache.org/jira/browse/FLINK-13150
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.9.0
Reporter: Jeff Zhang


defaultCatalogName and defaultDatabaseName in TableEnvImpl are initialized when 
it is created and never changed even when they are updated in TableEnvironment.

The will cause issues that we may register table to the wrong catalog after we 
changed the defaultCatalogName and defaultDatabaseName 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13151) Rename JobManager (Process) to Flink Master

[Konstantin Knauf (JIRA)]

Konstantin Knauf created FLINK-13151:


 Summary: Rename JobManager (Process) to Flink Master
 Key: FLINK-13151
 URL: https://issues.apache.org/jira/browse/FLINK-13151
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation, Runtime / Coordination
Reporter: Konstantin Knauf


In FLINK-12652 we coined the term "Flink Master" to describe the master process 
of a Flink cluster (containing JobManagers, Dispatcher and ResourceManager). 
This needs to be reflected in the rest of the documentation, scripts (where 
possible) and code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[FLIP-47] Savepoints vs Checkpoints

[Kostas Kloudas]

Hi Devs,

Currently there is a number of efforts around checkpoints/savepoints, as
reflected by the number of FLIPs. From a quick look FLIP-34, FLIP-41,
FLIP-43, and FLIP-45 are all directly related to these topics. This
reflects the importance of these two notions/features to the users of the
framework.

Although many efforts are centred around these notions, their semantics and
the interplay between them is not always clearly defined. This makes them
difficult to explain them to the users (all the different combinations of
state-backends, formats and tradeoffs) and in some cases it may have
negative effects to the users (e.g. the already-fixed-some-time-ago issue
of savepoints not being considered for recovery although they committed
side-effects).

FLIP-47 [1] and the related Document [2] is aiming at starting a discussion
around the semantics of savepoints/checkpoints and their interplay, and to
some extent help us fix the future steps concerning these notions. As an
example, should we work towards bringing them closer, or moving them
further apart.

This is not a complete proposal (by no means), as many of the practical
implications can only be fleshed out after we agree on the basic semantics
and the general frame around these notions. To that end, there are no
concrete implementation steps and the FLIP is going to be updated as the
discussion continues.

I am really looking forward to your opinions on the topic.

Cheers,
Kostas

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-47%3A+Checkpoints+vs.+Savepoints
[2]
https://docs.google.com/document/d/1_1FF8D3u0tT_zHWtB-hUKCP_arVsxlmjwmJ-TvZd4fs/edit?usp=sharing

[jira] [Created] (FLINK-13152) Unable to run query when using HiveCatalog and DataSet api

[Jeff Zhang (JIRA)]

Jeff Zhang created FLINK-13152:
--

 Summary: Unable to run query when using HiveCatalog and DataSet api
 Key: FLINK-13152
 URL: https://issues.apache.org/jira/browse/FLINK-13152
 Project: Flink
  Issue Type: Bug
Reporter: Jeff Zhang


{code:java}
ERROR [2019-07-08 22:09:22,200] ({ParallelScheduler-Worker-1} 
FlinkSqlInterrpeter.java[runSqlList]:107) - Fail to run sql:select * from a
org.apache.flink.table.api.TableException: Cannot generate a valid execution 
plan for the given query:

FlinkLogicalTableSourceScan(table=[[hive, default, a]], fields=[tt], 
source=[HiveTableSource(tt)])

This exception indicates that the query uses an unsupported SQL feature.
Please check the documentation for the set of currently supported SQL features.
at org.apache.flink.table.plan.Optimizer.runVolcanoPlanner(Optimizer.scala:245)
at 
org.apache.flink.table.plan.Optimizer.optimizePhysicalPlan(Optimizer.scala:170)
at org.apache.flink.table.plan.BatchOptimizer.optimize(BatchOptimizer.scala:57)
at 
org.apache.flink.table.api.internal.BatchTableEnvImpl.translate(BatchTableEnvImpl.scala:258)
at 
org.apache.flink.table.api.scala.internal.BatchTableEnvironmentImpl.toDataSet(BatchTableEnvironmentImpl.scala:66){code}
This is the exception I hit, and the below is code to reproduce the issue.

I suspect it is because I am using DataSet and HiveCatalog together
{code:java}

def showTable(table: Table): String = {
  val columnNames: Array[String] = table.getSchema.getFieldNames
  val dsRow: DataSet[Row] = btenv.toDataSet[Row](table)
  val rows = dsRow.first(maxResult).collect()
}{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] How to Deal with Split/Select in DataStream API

[Aljoscha Krettek]

I think this would benefit from a FLIP, that neatly sums up the options, and 
which then gives us also a point where we can vote and ratify a decision.

As a gut feeling, I most like Option 3). Initially I would have preferred 
option 1) (because of a sense of API purity), but by now I think it’s good that 
users have this simpler option.

Aljoscha 

> On 8. Jul 2019, at 06:39, Xingcan Cui  wrote:
> 
> Hi all,
> 
> Thanks for your participation.
> 
> In this thread, we got one +1 for option 1 and option 3, respectively. In the 
> original thread[1], we got two +1 for option 1, one +1 for option 2, and five 
> +1 and one -1 for option 3.
> 
> To summarize,
> 
> Option 1 (port side output to flatMap and deprecate split/select): three +1
> Option 2 (introduce a new split/select and deprecate existing one): one +1
> Option 3 ("correct" the existing split/select): six +1 and one -1
> 
> It seems that most people involved are in favor of "correcting" the existing 
> split/select. However, this will definitely break the API compatibility, in a 
> subtle way.
> 
> IMO, the real behavior of consecutive split/select's has never been 
> thoroughly clarified. Even in the community, it hard to say that we come into 
> a consensus on its real semantics[2-4]. Though the initial design is not 
> ambiguous, there's no doubt that its concept has drifted. 
> 
> As the split/select is quite an ancient API, I cc'ed this to more members. It 
> couldn't be better if you can share your opinions on this.
> 
> Thanks,
> Xingcan
> 
> [1] 
> https://lists.apache.org/thread.html/f94ea5c97f96c705527dcc809b0e2b69e87a4c5d400cb7c61859e1f4@%3Cdev.flink.apache.org%3E
>  
> 
> [2] https://issues.apache.org/jira/browse/FLINK-1772 
> 
> [3] https://issues.apache.org/jira/browse/FLINK-5031 
> 
> [4] https://issues.apache.org/jira/browse/FLINK-11084 
> 
> 
> 
>> On Jul 5, 2019, at 12:04 AM, 杨力 > > wrote:
>> 
>> I prefer the 1) approach. I used to carry fields, which is needed only for 
>> splitting, in the outputs of flatMap functions. Replacing it with outputTags 
>> would simplify data structures.
>> 
>> Xingcan Cui mailto:xingc...@gmail.com> 
>> >> 于 2019年7月5日周五 
>> 上午2:20写道：
>> Hi folks,
>> 
>> Two weeks ago, I started a thread [1] discussing whether we should discard 
>> the split/select methods (which have been marked as deprecation since v1.7) 
>> in DataStream API. 
>> 
>> The fact is, these methods will cause "unexpected" results when using 
>> consecutively (e.g., ds.split(a).select(b).split(c).select(d)) or 
>> multi-times on the same target (e.g., ds.split(a).select(b), 
>> ds.split(c).select(d)). The reason is that following the initial design, the 
>> new split/select logic will always override the existing one on the same 
>> target operator, rather than append to it. Some users may not be aware of 
>> that, but if you do, a current solution would be to use the more powerful 
>> side output feature [2].
>> 
>> FLINK-11084  added some 
>> restrictions to the existing split/select logic and suggest to replace it 
>> with side output in the future. However, considering that the side output is 
>> currently only available in the process function layer and the split/select 
>> could have been widely used in many real-world applications, we'd like to 
>> start a vote andlisten to the community on how to deal with them.
>> 
>> In the discussion thread [1], we proposed three solutions as follows. All of 
>> them are feasible but have different impacts on the public API.
>> 
>> 1) Port the side output feature to DataStream API's flatMap and replace 
>> split/select with it.
>> 
>> 2) Introduce a dedicated function in DataStream API (with the "correct" 
>> behavior but a different name) that can be used to replace the existing 
>> split/select.
>> 
>> 3) Keep split/select but change the behavior/semantic to be "correct".
>> 
>> Note that this is just a vote for gathering information, so feel free to 
>> participate and share your opinions.
>> 
>> The voting time will end on July 7th 17:00 EDT.
>> 
>> Thanks,
>> Xingcan
>> 
>> [1] 
>> https://lists.apache.org/thread.html/f94ea5c97f96c705527dcc809b0e2b69e87a4c5d400cb7c61859e1f4@%3Cdev.flink.apache.org%3E
>>  
>> >  
>> >
>> [2] 
>> h

Re: [DISCUSS] Graceful Shutdown Handling by UDFs.

[Kostas Kloudas]

Hi Konstantin,

Yes you are right that `cancel` falls under the "abrupt" termination of a
job.
I will update the FLIP accordingly.

Cheers,
Kostas

On Sun, Jul 7, 2019 at 11:38 AM Konstantin Knauf 
wrote:

> Hi Klou,
>
> +1 for this proposal. I am missing any mention of "cancel" in the design
> document though. In my understanding we are not planning to deprecate
> "cancel" completely (just cancel-with-savepoint, which is superseded by
> "stop"). In any case we should consider it in the design document. It seems
> to me that "cancel" should be consider an ungraceful shutdown, so that the
> Job could be restarted from last (retained) checkpoint (as right now).
>
> Cheers,
>
> Konstantin
>
> On Thu, Jul 4, 2019 at 3:21 PM Kostas Kloudas  wrote:
>
> > Hi all,
> >
> > In many cases, UDFs (User Defined Functions) need to be able to perform
> > application-specific actions when they stop in an orderly manner.
> > Currently, Flink's UDFs, and more specifically the RichFunction which
> > exposes lifecycle-related hooks, only have a close() method which is
> called
> > in any case of job termination. This includes any form of orderly
> > termination (STOP or End-Of-Stream) and termination due to an error.
> >
> >
> > The FLIP in [1] and the design document in [2] propose the addition of an
> > interface that will allow UDFs that implement it to perform application
> > specific logic in the case of graceful termination. These cases include
> > DRAIN and SUSPEND for streaming jobs (see FLIP-34), but also reaching the
> > End-Of-Stream for jobs with finite sources.
> >
> > Let's have a lively discussion to solve this issue that has been around
> for
> > quite some time.
> >
> > Cheers,
> > Kostas
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs
> >
> > [2]
> >
> >
> https://docs.google.com/document/d/1SXfhmeiJfWqi2ITYgCgAoSDUv5PNq1T8Zu01nR5Ebog/edit?usp=sharing
> >
>
>
> --
>
> Konstantin Knauf | Solutions Architect
>
> +49 160 91394525
>
>
> Planned Absences: 10.08.2019 - 31.08.2019, 05.09. - 06.09.2010
>
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
>
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>

[jira] [Created] (FLINK-13153) SplitAggregateITCase.testMinMaxWithRetraction failed on Travis

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13153:
-

 Summary: SplitAggregateITCase.testMinMaxWithRetraction failed on 
Travis
 Key: FLINK-13153
 URL: https://issues.apache.org/jira/browse/FLINK-13153
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.9.0
Reporter: Till Rohrmann
 Fix For: 1.9.0


{{SplitAggregateITCase.testMinMaxWithRetraction}} failed on Travis with

{code}
Failures: 
10:50:43.355 [ERROR]   SplitAggregateITCase.testMinMaxWithRetraction:195 
expected: but was:
{code}

https://api.travis-ci.org/v3/job/554991853/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13154) Broken links in documentation

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13154:
-

 Summary: Broken links in documentation
 Key: FLINK-13154
 URL: https://issues.apache.org/jira/browse/FLINK-13154
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.9.0
Reporter: Till Rohrmann
 Fix For: 1.9.0


Flink's broken link verification failed on Travis as it has discovered some 
broken links. We need to fix this for the upcoming release.

{code}
[2019-07-07 10:54:21] ERROR `/tutorials/datastream_api.html' not found.
[2019-07-07 10:54:21] ERROR `/tutorials/local_setup.html' not found.
[2019-07-07 10:54:21] ERROR `/tutorials/flink_on_windows.html' not found.
[2019-07-07 10:54:21] ERROR `/examples' not found.
[2019-07-07 10:54:23] ERROR `/zh/dev/connectors/pubsub.html' not found.
[2019-07-07 10:54:24] ERROR `/ops/filesystems.html' not found.
[2019-07-07 10:54:24] ERROR `/dev/table/catalog.html' not found.
[2019-07-07 10:54:28] ERROR `/zh/tutorials/datastream_api.html' not found.
[2019-07-07 10:54:28] ERROR `/zh/tutorials/local_setup.html' not found.
http://localhost:4000/tutorials/datastream_api.html:
Remote file does not exist -- broken link!!!
http://localhost:4000/tutorials/local_setup.html:
Remote file does not exist -- broken link!!!
--
http://localhost:4000/tutorials/flink_on_windows.html:
Remote file does not exist -- broken link!!!
--
http://localhost:4000/examples:
Remote file does not exist -- broken link!!!
--
http://localhost:4000/zh/dev/connectors/pubsub.html:
Remote file does not exist -- broken link!!!
--
http://localhost:4000/ops/filesystems.html:
Remote file does not exist -- broken link!!!
--
http://localhost:4000/dev/table/catalog.html:
Remote file does not exist -- broken link!!!
--
http://localhost:4000/zh/tutorials/datastream_api.html:
Remote file does not exist -- broken link!!!
http://localhost:4000/zh/tutorials/local_setup.html:
Remote file does not exist -- broken link!!!
{code}

https://api.travis-ci.org/v3/job/554991858/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13155) SQL Client end-to-end test fails on Travis

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13155:
-

 Summary: SQL Client end-to-end test fails on Travis
 Key: FLINK-13155
 URL: https://issues.apache.org/jira/browse/FLINK-13155
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Affects Versions: 1.9.0
Reporter: Till Rohrmann
 Fix For: 1.9.0


The {{SQL Client end-to-end test}} which executes 
{{test-scripts/test_sql_client.sh}} fails on Travis with non-empty out files.

https://api.travis-ci.org/v3/job/554991859/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13156) Elasticsearch (v1.7.1) sink end-to-end test failed on Travis

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13156:
-

 Summary: Elasticsearch (v1.7.1) sink end-to-end test failed on 
Travis
 Key: FLINK-13156
 URL: https://issues.apache.org/jira/browse/FLINK-13156
 Project: Flink
  Issue Type: Bug
  Components: Connectors / ElasticSearch
Affects Versions: 1.9.0
Reporter: Till Rohrmann
 Fix For: 1.9.0


The {{Elasticsearch (v1.7.1) sink end-to-end test}} failed on Travis because 
the test waited for Elasticsearch records indefinitely.

https://api.travis-ci.org/v3/job/554991871/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [RESULT][VOTE] Migrate to sponsored Travis account

[Chesnay Schepler]

I have temporarily re-enabled running PR builds on the ASF account; 
migrating to the Travis subscription caused some issues in the bot that 
I have to fix first.


On 07/07/2019 23:01, Chesnay Schepler wrote:
The vote has passed unanimously in favor of migrating to a separate 
Travis account.


I will now set things up such that no PullRequest is no longer run on 
the ASF servers.

This is a major setup in reducing our usage of ASF resources.
For the time being we'll use free Travis plan for flink-ci (i.e. 5 
workers, which is the same the ASF gives us). Over the course of the 
next week we'll setup the Ververica subscription to increase this limit.


From now now, a bot will mirror all new and updated PullRequests to a 
mirror repository (https://github.com/flink-ci/flink-ci) and write an 
update into the PR once the build is complete.
I have ran the bots for the past 3 days in parallel to our existing 
Travis and it was working without major issues.


The biggest change that contributors will see is that there's no 
longer a icon next to each commit. We may revisit this in the future.


I'll setup a repo with the source of the bot later.

On 04/07/2019 10:46, Chesnay Schepler wrote:
I've raised a JIRA 
with INFRA to 
inquire whether it would be possible to switch to a different Travis 
account, and if so what steps would need to be taken.
We need a proper confirmation from INFRA since we are not in full 
control of the flink repository (for example, we cannot access the 
settings page).


If this is indeed possible, Ververica is willing sponsor a Travis 
account for the Flink project.

This would provide us with more than enough resources than we need.

Since this makes the project more reliant on resources provided by 
external companies I would like to vote on this.


Please vote on this proposal, as follows:
[ ] +1, Approve the migration to a Ververica-sponsored Travis 
account, provided that INFRA approves
[ ] -1, Do not approach the migration to a Ververica-sponsored Travis 
account


The vote will be open for at least 24h, and until we have 
confirmation from INFRA. The voting period may be shorter than the 
usual 3 days since our current is effectively not working.


On 04/07/2019 06:51, Bowen Li wrote:
Re: > Are they using their own Travis CI pool, or did the switch to 
an entirely different CI service?


I reached out to Wes and Krisztián from Apache Arrow PMC. They are 
currently moving away from ASF's Travis to their own in-house metal 
machines at [1] with custom CI application at [2]. They've seen 
significant improvement w.r.t both much higher performance and 
basically no resource waiting time, "night-and-day" difference 
quoting Wes.


Re: > If we can just switch to our own Travis pool, just for our 
project, then this might be something we can do fairly quickly?


I believe so, according to [3] and [4]


[1] https://ci.ursalabs.org/ 
[2] https://github.com/ursa-labs/ursabot
[3] 
https://docs.travis-ci.com/user/migrate/open-source-repository-migration 

[4] 
https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com




On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler > wrote:


    Are they using their own Travis CI pool, or did the switch to an
    entirely different CI service?

    If we can just switch to our own Travis pool, just for our
    project, then
    this might be something we can do fairly quickly?

    On 03/07/2019 05:55, Bowen Li wrote:
    > I responded in the INFRA ticket [1] that I believe they are
    using a wrong
    > metric against Flink and the total build time is a completely
    different
    > thing than guaranteed build capacity.
    >
    > My response:
    >
    > "As mentioned above, since I started to pay attention to Flink's
    build
    > queue a few tens of days ago, I'm in Seattle and I saw no build
    was kicking
    > off in PST daytime in weekdays for Flink. Our teammates in China
    and Europe
    > have also reported similar observations. So we need to evaluate
    how the
    > large total build time came from - if 1) your number and 2) our
    > observations from three locations that cover pretty much a full
    day, are
    > all true, I **guess** one reason can be that - highly likely the
    extra
    > build time came from weekends when other Apache projects may be
    idle and
    > Flink just drains hard its congested queue.
    >
    > Please be aware of that we're not complaining about the lack of
    resources
    > in general, I'm complaining about the lack of **stable, 
dedicated**

    > resources. An example for the latter one is, currently even if
    no build is
    > in Flink's queue and I submit a request to be the queue head 
in PST

    > morning, my build won't even start in 6-8+h. That is an absurd
    amount of
    > waiting time.
    >
    > That's saying, if ASF INFRA decides to adopt a quota system a

Re: [DISCUSS] Adopting a Code Style and Quality Guide

[Robert Metzger]

I've converted the google document into a markdown-based pull request here:
https://github.com/apache/flink-web/pull/224

I didn't put a ton of effort into splitting the guide into multiple pages.
It would be nice to reach consensus on the exact split in the PR.

On Thu, Jun 27, 2019 at 10:48 PM Hugo Louro  wrote:

> +1. Thanks for working on the guide. It's very thorough and a good resource
> to learn good practices from.
>
> I would like use this thread as a placeholder for a couple of topics that
> may be deserving of further discussion on different threads:
>   - What's the best way to keep track of checkstyle version updates. For
> instance, currently there is a PR
>  proposing checkstyle to be
> updated because version 8.12 is no longer supported
>  - When classes import shaded dependencies, it is not trivial for IntelliJ
> to download and associate sources and javadocs, which makes debugging and
> navigate the code base harder. I tried installing the version of the
> library using maven from the CLI, and associate the sources "manually" on
> IntelliJ, but it seems it does not work (perhaps IntelliJ issue). Does
> anyone know of a good solution? If so, we should added here
> <
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/flinkDev/ide_setup.html#intellij-idea
> >.
> I can volunteer for that if you tell me how to do it :)
> - did the community evaluate naming test methods according to these
> conventions  ?
>
> Thanks
>
> On Mon, Jun 24, 2019 at 11:38 AM Stephan Ewen  wrote:
>
> > I think it makes sense to also start individual [DISCUSS] threads about
> > extensions and follow-ups.
> > Various suggestions came up, partly as comments in the doc, some as
> > questions in other threads.
> >
> > Examples:
> >   - use of null in return types versus Optional versus @Nullable/@Nonnull
> >   - initialization of collection sizes
> >   - logging
> >
> > I think these would be best discussed in separate threads each.
> > So, for contributors to whom these issues are dear, feel free to spawn
> > these additional threads.
> > (Bear in mind it is close to 1.9 feature freeze time, so please leave
> this
> > discussions a bit of time so that all community members have a chance to
> > participate)
> >
> >
> >
> > On Mon, Jun 24, 2019 at 7:51 PM Stephan Ewen  wrote:
> >
> > > Thanks for the pointer David.
> > >
> > > I was not aware of this tool and I have no experience with its
> practical
> > > impact. For example I would be curious what the effect of this is for
> > > existing PRs, merge conflicts, etc.
> > >
> > > If you want, feel free to start another discuss thread on the
> > introduction
> > > of such a tool.
> > >
> > > On Sun, Jun 23, 2019 at 6:32 PM David Morávek  wrote:
> > >
> > >> I love this kind of unification being pushed forward, the benefit it
> has
> > >> on
> > >> code reviews is enormous. Just my two cents, did you guys think about
> > >> adopting an automatic code formatter for java, the same way as Apache
> > Beam
> > >> did?
> > >>
> > >> It is super easy to use for contributors as they don't need to keep
> any
> > >> particular coding style in mind and they can only focus on
> functionality
> > >> they want to fix, and it's also great for reviewers, because they only
> > see
> > >> the important changes. This also eliminates need for any special
> editor
> > /
> > >> checkstyle configs as the code formatting is part of the build itself.
> > >>
> > >> The one Beam uses is https://github.com/diffplug/spotless with
> > >> GoogleJavaFormat, it may be worth to look into.
> > >>
> > >> Best,
> > >> David
> > >>
> > >> On Fri, Jun 21, 2019 at 4:40 PM Stephan Ewen 
> wrote:
> > >>
> > >> > Thanks, everyone, for the positive feedback :-)
> > >> >
> > >> > @Robert - It probably makes sense to break this down into various
> > pages,
> > >> > like PR, general code style guide, Java, component specific guides,
> > >> > formats, etc.
> > >> >
> > >> > Best,
> > >> > Stephan
> > >> >
> > >> >
> > >> > On Fri, Jun 21, 2019 at 4:29 PM Robert Metzger  >
> > >> > wrote:
> > >> >
> > >> > > It seems that the discussion around this topic has settled.
> > >> > >
> > >> > > I'm going to turn the Google Doc into a markdown file (maybe also
> > >> > multiple,
> > >> > > I'll try out different things) and then open a pull request for
> the
> > >> Flink
> > >> > > website.
> > >> > > I'll post a link to the PR here once I'm done.
> > >> > >
> > >> > > On Fri, Jun 14, 2019 at 9:36 AM zhijiang <
> > wangzhijiang...@aliyun.com
> > >> > > .invalid>
> > >> > > wrote:
> > >> > >
> > >> > > > Thanks for providing this useful guide for benefiting both
> > >> contributors
> > >> > > > and committers in consistency.
> > >> > > >
> > >> > > > I just reviewed and learned the whole doc which covers a lot of
> > >> > > > information. Wish it further categoried and put onto somewhere
> for
> > >> > easily
> > >> > > > traced future.
> > >> 

Re: [jira] [Created] (FLINK-13139) Various Hive tests fail on Travis

[Bowen Li]

Hi Louis,


Thanks for reporting the issue. The problem is due to Hive 2.3.4 not
compatible with Hadoop 2.4, and requires at least 2.7. I'm not sure yet why
the build succeeded most of the time on Travis CI and only fails
occasionally, maybe it's because the build process
has flink-shaded-hadoop-2-uber in multiple hadoop 2 versions (2.4, 2.7,
etc) in its classpath somehow.

It's been fixed in FLINK-13134
.




On Mon, Jul 8, 2019 at 3:15 AM 不常用邮箱  wrote:

> Hello all:
>
> I meet this problem with can’t import follow class in test file.
> org.apache.flink.table.catalog.hive.HiveCatalog;
> org.apache.flink.table.catalog.hive.HiveTestUtils;
> org.apache.flink.table.catalog.hive.HivePartitionConfig;
> org.apache.flink.table.catalog.hive.HiveCatalogConfig;
> org.apache.flink.table.catalog.hive.HiveDatabaseConfig;
>
> And even themselves can’t import each other. Is that some setting wrong
> with my editor?
> I search the problem in google, told that may occur when copy files?
> Is someone can fix this?
>
> Thanks.
> Louis
>
>
>
> --
> Louis
> Email： xu_soft39211...@163.com
>
> > On Jul 8, 2019, at 16:05, Till Rohrmann (JIRA)  wrote:
> >
> > Till Rohrmann created FLINK-13139:
> > -
> >
> > Summary: Various Hive tests fail on Travis
> > Key: FLINK-13139
> > URL: https://issues.apache.org/jira/browse/FLINK-13139
> > Project: Flink
> >  Issue Type: Bug
> >  Components: Connectors / Hive
> >Affects Versions: 1.9.0
> >Reporter: Till Rohrmann
> > Fix For: 1.9.0
> >
> >
> > Various Hive related tests fail on Travis:
> >
> > {code}
> > 06:06:49.654 [ERROR] Errors:
> > 06:06:49.654 [ERROR]   HiveInputFormatTest.createCatalog:66 » Catalog
> Failed to create Hive Metastore...
> > 06:06:49.654 [ERROR]   HiveTableFactoryTest.init:55 » Catalog Failed to
> create Hive Metastore client
> > 06:06:49.654 [ERROR]   HiveTableOutputFormatTest.createCatalog:72 »
> Catalog Failed to create Hive Met...
> > 06:06:49.654 [ERROR]   HiveTableSinkTest.createCatalog:72 » Catalog
> Failed to create Hive Metastore c...
> > 06:06:49.654 [ERROR]   HiveTableSourceTest.createCatalog:67 » Catalog
> Failed to create Hive Metastore...
> > 06:06:49.654 [ERROR]   HiveCatalogGenericMetadataTest.init:49 » Catalog
> Failed to create Hive Metasto...
> > 06:06:49.654 [ERROR]   HiveCatalogHiveMetadataTest.init:55 » Catalog
> Failed to create Hive Metastore ...
> > 06:06:49.654 [ERROR]   HiveGenericUDFTest.testCeil:193->init:387 »
> ExceptionInInitializer
> > 06:06:49.654 [ERROR]   HiveGenericUDFTest.testDecode:160 »
> NoClassDefFound Could not initialize class...
> > 06:06:49.654 [ERROR]
>  HiveSimpleUDFTest.testUDFArray_singleArray:202->init:237 »
> NoClassDefFound Cou...
> > 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFBin:60->init:237 »
> NoClassDefFound Could not initiali...
> > 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFConv:67->init:237 »
> NoClassDefFound Could not initial...
> > 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFJson:85->init:237 »
> NoClassDefFound Could not initial...
> > 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFMinute:126->init:237 »
> ExceptionInInitializer
> > 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFRand:51->init:237 »
> NoClassDefFound Could not initial...
> > 06:06:49.654 [ERROR]
>  HiveSimpleUDFTest.testUDFRegExpExtract:153->init:237 » NoClassDefFound
> Could n...
> > 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFToInteger:188->init:237
> » NoClassDefFound Could not i...
> > 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFUnbase64:166->init:237
> » NoClassDefFound Could not in...
> > 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFUnhex:177->init:237 »
> NoClassDefFound Could not initi...
> > 06:06:49.654 [ERROR]   HiveSimpleUDFTest.testUDFWeekOfYear:139->init:237
> » NoClassDefFound Could not ...
> > {code}
> >
> > https://api.travis-ci.org/v3/job/555252043/log.txt
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v7.6.3#76005)
>
>

Re: [RESULT][VOTE] Migrate to sponsored Travis account

[Chesnay Schepler]

The kinks have been worked out; the bot is running again and pr builds 
are yet again no longer running on ASF resources.


PRs are mirrored to: https://github.com/flink-ci/flink
Bot source: https://github.com/flink-ci/ci-bot

On 08/07/2019 17:14, Chesnay Schepler wrote:
I have temporarily re-enabled running PR builds on the ASF account; 
migrating to the Travis subscription caused some issues in the bot 
that I have to fix first.


On 07/07/2019 23:01, Chesnay Schepler wrote:
The vote has passed unanimously in favor of migrating to a separate 
Travis account.


I will now set things up such that no PullRequest is no longer run on 
the ASF servers.

This is a major setup in reducing our usage of ASF resources.
For the time being we'll use free Travis plan for flink-ci (i.e. 5 
workers, which is the same the ASF gives us). Over the course of the 
next week we'll setup the Ververica subscription to increase this limit.


From now now, a bot will mirror all new and updated PullRequests to a 
mirror repository (https://github.com/flink-ci/flink-ci) and write an 
update into the PR once the build is complete.
I have ran the bots for the past 3 days in parallel to our existing 
Travis and it was working without major issues.


The biggest change that contributors will see is that there's no 
longer a icon next to each commit. We may revisit this in the future.


I'll setup a repo with the source of the bot later.

On 04/07/2019 10:46, Chesnay Schepler wrote:
I've raised a JIRA 
with INFRA to 
inquire whether it would be possible to switch to a different Travis 
account, and if so what steps would need to be taken.
We need a proper confirmation from INFRA since we are not in full 
control of the flink repository (for example, we cannot access the 
settings page).


If this is indeed possible, Ververica is willing sponsor a Travis 
account for the Flink project.

This would provide us with more than enough resources than we need.

Since this makes the project more reliant on resources provided by 
external companies I would like to vote on this.


Please vote on this proposal, as follows:
[ ] +1, Approve the migration to a Ververica-sponsored Travis 
account, provided that INFRA approves
[ ] -1, Do not approach the migration to a Ververica-sponsored 
Travis account


The vote will be open for at least 24h, and until we have 
confirmation from INFRA. The voting period may be shorter than the 
usual 3 days since our current is effectively not working.


On 04/07/2019 06:51, Bowen Li wrote:
Re: > Are they using their own Travis CI pool, or did the switch to 
an entirely different CI service?


I reached out to Wes and Krisztián from Apache Arrow PMC. They are 
currently moving away from ASF's Travis to their own in-house metal 
machines at [1] with custom CI application at [2]. They've seen 
significant improvement w.r.t both much higher performance and 
basically no resource waiting time, "night-and-day" difference 
quoting Wes.


Re: > If we can just switch to our own Travis pool, just for our 
project, then this might be something we can do fairly quickly?


I believe so, according to [3] and [4]


[1] https://ci.ursalabs.org/ 
[2] https://github.com/ursa-labs/ursabot
[3] 
https://docs.travis-ci.com/user/migrate/open-source-repository-migration 

[4] 
https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com




On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler 
mailto:ches...@apache.org>> wrote:


    Are they using their own Travis CI pool, or did the switch to an
    entirely different CI service?

    If we can just switch to our own Travis pool, just for our
    project, then
    this might be something we can do fairly quickly?

    On 03/07/2019 05:55, Bowen Li wrote:
    > I responded in the INFRA ticket [1] that I believe they are
    using a wrong
    > metric against Flink and the total build time is a completely
    different
    > thing than guaranteed build capacity.
    >
    > My response:
    >
    > "As mentioned above, since I started to pay attention to Flink's
    build
    > queue a few tens of days ago, I'm in Seattle and I saw no build
    was kicking
    > off in PST daytime in weekdays for Flink. Our teammates in China
    and Europe
    > have also reported similar observations. So we need to evaluate
    how the
    > large total build time came from - if 1) your number and 2) our
    > observations from three locations that cover pretty much a full
    day, are
    > all true, I **guess** one reason can be that - highly likely the
    extra
    > build time came from weekends when other Apache projects may be
    idle and
    > Flink just drains hard its congested queue.
    >
    > Please be aware of that we're not complaining about the lack of
    resources
    > in general, I'm complaining about the lack of **stable, 
dedicated**

    > resources. An example for the latter one is, currently even

[jira] [Created] (FLINK-13157) reeanble unit test HiveInputFormatTest.testReadComplextDataTypeFromHiveInputFormat() and resolve conflicts

[Bowen Li (JIRA)]

Bowen Li created FLINK-13157:


 Summary: reeanble unit test 
HiveInputFormatTest.testReadComplextDataTypeFromHiveInputFormat() and resolve 
conflicts
 Key: FLINK-13157
 URL: https://issues.apache.org/jira/browse/FLINK-13157
 Project: Flink
  Issue Type: Task
  Components: Connectors / Hive
Reporter: Bowen Li
Assignee: zjuwangg
 Fix For: 1.9.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Proposal for Flink job execution/availability metrics impovement

[Becket Qin]

Hi Hwanju, thanks for the reply.

Regarding the first two proposals, my main concern is whether it is
necessary to have something deeply coupled with Flink runtime. To some
extent, the SLA metrics are kind of custom metrics. It would be good if we
can support custom metrics in general, instead of only for job state
changes. I am wondering if something similar to following would  meet your
requirements.

1. Define an *Incident* interface in Flink
- The incident may have multiple subclasses, e.g. job state change,
checkpiont failure, and other user defined subclasses.
2. Introduce an IncidentListener interface, which is responsible for
handling *incidents *in Flink.
- Each IncidentListener is responsible for handling one type of
*Incident*.
- An IncidentListener will be provided a context to leverage some
services in Flink. For now, we can just expose Metric reporting service.
3. Introduce an IncidentListenerRegistry to allow register
IncidentListeners for an incident type.
- Multiple IncidentListeners can be registered for each incident type.
- When an incident occurs, the registered IncidentListeners will be
invoked.
- users may configure IncidentListeners in the configuration.

The above mechanism has two benefits:
1. Users can provide arbitrary logic to handle incidents, including
calculating SLA and so on.
2. In the future we can add more incident types to address other custom
incident handling use cases.

There might be some details to be sorted out, such as where should the
incident handler run? JM or TM? But those are probably more of detail
design and implementation choices.

Regarding the 3rd doc, I think it is useful to introduce a progress
monitoring mechanism. And inserting a in-flow control event also align with
the fundamental design of Flink. So in general I think it is a good
addition. One thing not quite clear to me is which part in the proposal is
intended to be done inside Flink and which part might be built as an
ecosystem project / pluggable? For example, if we reuse the mechanism
above, can we do the following:

1. Introduce a *ProgressMonitoringIncident *in Flink
*-* Each operator will timstamp the incident when the incident flows
through the operator. Eventually, there will be N such incidents, where N
is the total parallelism of the sink nodes.
- The SinkOperator will invoke the ProgressMonitoringIncidentListener
to handle all such incidents and perform corresponding analysis (maybe the
ExecutionGraph is needed in the context).
2. Users may provide custom logic to analyze the progress.

Thanks,

Jiangjie (Becket) Qin

On Wed, Jul 3, 2019 at 4:53 PM Hwanju Kim  wrote:

> Hi,
>
> I am sharing the last doc, which is about progress monitor (a little
> deferred sharing by my vacation):
>
> https://docs.google.com/document/d/1Ov9A7V2tMs4uVimcSeHL5eftRJ3MCJBiVSFNdz8rmjU/edit?usp=sharing
>
> This last one seems like pretty independent from the first two (execution
> tracking and exception classifier), so it could be completely decoupled.
> We may want to leave this proposal, apart from the first two, more
> discussed here in dev list rather than diving deeper into implementation.
> (there are obviously more several challenging things)
>
> Thanks,
> Hwanju
>
>
> 2019년 5월 28일 (화) 오후 11:41, Hwanju Kim 님이 작성:
>
> > (Somehow my email has failed to be sent multiple times, so I am using my
> > personal email account)
> >
> > Hi,
> >
> > Piotrek - Thanks for the feedback! I revised the doc as commented.
> >
> > Here's the second part about exception classification -
> >
> https://docs.google.com/document/d/1pcHg9F3GoDDeVD5GIIo2wO67Hmjgy0-hRDeuFnrMgT4/edit?usp=sharing
> > I put cross-links between the first and the second.
> >
> > Thanks,
> > Hwanju
> >
> > 2019년 5월 24일 (금) 오전 3:57, Piotr Nowojski 님이 작성:
> >
> >> Hi Hwanju,
> >>
> >> I looked through the document, however I’m not the best person to
> >> review/judge/discuss about implementation details here. I hope that
> Chesney
> >> will be able to help in this regard.
> >>
> >> Piotrek
> >>
> >> > On 24 May 2019, at 09:09, Kim, Hwanju 
> >> wrote:
> >> >
> >> > Hi,
> >> >
> >> > As suggested by Piotrek, the first part, execution state tracking, is
> >> now split to a separate doc:
> >> >
> >>
> https://docs.google.com/document/d/1oLF3w1wYyr8vqoFoQZhw1QxTofmAtlD8IF694oPLjNI/edit?usp=sharing
> >> >
> >> > We'd appreciate any feedback. I am still using the same email thread
> to
> >> provide a full context, but please let me know if it's better to have a
> >> separate email thread as well. We will be sharing the remaining parts
> once
> >> ready.
> >> >
> >> > Thanks,
> >> > Hwanju
> >> >
> >> > On 5/17/19, 12:59 AM, "Piotr Nowojski"  wrote:
> >> >
> >> >Hi Hwanju & Chesney,
> >> >
> >> >Regarding various things that both of you mentioned, like
> accounting
> >> of state restoration separately or batch scheduling, we can always
> >> acknowledge some limitations of the initial approach and maybe we can
> >> ad

Re: [DISCUSS] ARM support for Flink

[Xiyuan Wang]

Thanks for your help. I built the frocksdb locally on ARM and all the
related tests are passed now. Except some tests which can be fixed easily,
it seems that both building and testing are ran well on ARM.

Basing on my test, Is it possible to support Flink on ARM officailly? Seem
the worklist is not too long. And I can help with the CI testing part.

Need Flink team's idea.

Thanks.

Dian Fu  于2019年7月8日周一 上午10:23写道：

> Hi Xiyuan,
>
> Thanks for bring the discussion.
>
> WRT the exception, it's because the native bundled in the rocksdb jar file
> isn't compiled with cross platform support. You can refer [1] for how to
> build rocksdb which has ARM platform.
>
> WRT ARM support, the rocksdb currently used in Flink is hosted in the
> Ververica git [2], so it won't be difficult to make it support ARM.
> However, I guess this git exists just for temporary [3], not because we
> want to add much feature in rocksdb.
>
> [1] https://github.com/facebook/rocksdb/issues/678 <
> https://github.com/facebook/rocksdb/issues/678>
> [2] https://github.com/dataArtisans/frocksdb <
> https://github.com/dataArtisans/frocksdb>
> [3] https://issues.apache.org/jira/browse/FLINK-10471 <
> https://issues.apache.org/jira/browse/FLINK-10471>
>
> Regards,
> Dian
>
> > 在 2019年7月8日，上午9:17，Xiyuan Wang  写道：
> >
> > Hi Flink:
> >  Recently we meet a problem that we have to test and run Flink on ARM
> > arch. While after searching Flink community, I didn’t find an official
> ARM
> > release version.
> >
> > Since Flink is made by Java and Scala language which can be ran
> > cross-platform usually, I think Flink can be built and ran on ARM
> directly
> > as well. Then with my local test, Flink was built and deployed success as
> > expected. But some tests were failed due to ARM arch. For example:
> >
> > 1. MemoryArchitectureTest.testArchitectureNotUnknown:34 Values should be
> > different. Actual: UNKNOWN
> > 2. [ERROR]
> >
> testIterator(org.apache.flink.contrib.streaming.state.RocksDBRocksStateKeysIteratorTest)
> > Time elapsed: 0.234 s  <<< ERROR!
> > java.io.IOException: Could not load the native RocksDB library
> > at
> >
> org.apache.flink.contrib.streaming.state.RocksDBRocksStateKeysIteratorTest.testIteratorHelper(RocksDBRocksStateKeysIteratorTest.java:90)
> > at
> >
> org.apache.flink.contrib.streaming.state.RocksDBRocksStateKeysIteratorTest.testIterator(RocksDBRocksStateKeysIteratorTest.java:63)
> > Caused by: java.lang.UnsatisfiedLinkError:
> >
> /tmp/rocksdb-lib-81ca7930b92af2cca143a050c0338d34/librocksdbjni-linux64.so:
> >
> /tmp/rocksdb-lib-81ca7930b92af2cca143a050c0338d34/librocksdbjni-linux64.so:
> > cannot open shared object file: No such file or directory (Possible
> cause:
> > can't load AMD 64-bit .so on a AARCH64-bit platform)
> > …
> >
> >  Since the test isn’t passed totally, we are not sure if Flink 100%
> > support ARM or not. Is it possible for Flink to support ARM release
> > officially? I guess it may be not a very huge work basing on Java. I
> notice
> > that Flink now uses trivis-ci which is X86 only for build & test check.
> Is
> > it possible to add an ARM arch CI as well? It can be non-voting first.
> Then
> > we can keep monitoring and fixing ARM related error. One day it’s stable
> > enough, we can remove the non-voting tag and create Flink ARM release.
> >
> >  There is an open source CI community called OpenLab[1] which can provide
> > CI function and ARM resource to Flink by free. I’m one of the OpenLab
> > members. If Flink commun think ARM support is fine, I can keep helping
> > Flink to build and maintain the ARM CI job. There is an  POC for Flink
> ARM
> > build job made by me on OpenLab system[2] and a live demo which built and
> > run on an ARM VM[3]. You can take a look first.
> >
> > Eager to get everybody’s feedback. Any question is welcome.
> >
> > Thanks.
> >
> > [1]: https://openlabtesting.org/
> > [2]: https://github.com/theopenlab/flink/pull/1
> > [3]: http://114.115.168.52:8081/#/overview
>
>

[jira] [Created] (FLINK-13158) Remove WebMonitor interface from webmonitor package

[vinoyang (JIRA)]

vinoyang created FLINK-13158:


 Summary: Remove WebMonitor interface from webmonitor package
 Key: FLINK-13158
 URL: https://issues.apache.org/jira/browse/FLINK-13158
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / REST
Reporter: vinoyang
Assignee: vinoyang


Currently, the interface {{WebMonitor}} has already not been used anymore. IMO, 
we can remove it from the source code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13159) java.lang.ClassNotFoundException when restore job

[kring (JIRA)]

kring created FLINK-13159:
-

 Summary: java.lang.ClassNotFoundException when restore job
 Key: FLINK-13159
 URL: https://issues.apache.org/jira/browse/FLINK-13159
 Project: Flink
  Issue Type: Bug
Reporter: kring


{code:java}
//代码占位符
{code}
java.lang.Exception: Exception while creating StreamOperatorStateContext. at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:195)
 at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:250)
 at 
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
 at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289) 
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) at 
java.lang.Thread.run(Thread.java:748) Caused by: 
org.apache.flink.util.FlinkException: Could not restore keyed state backend for 
WindowOperator_b398b3dd4c544ddf2d47a0cc47d332f4_(1/6) from any of the 1 prov 
ided restore options. at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
 at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:307)
 at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:135)
 ... 5 common frames omitted Caused by: 
org.apache.flink.runtime.state.BackendBuildingException: Failed when trying to 
restore heap backend at 
org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.build(HeapKeyedStateBackendBuilder.java:130)
 at 
org.apache.flink.runtime.state.filesystem.FsStateBackend.createKeyedStateBackend(FsStateBackend.java:489)
 at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:291)
 at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
 at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
 ... 7 common frames omitted Caused by: java.lang.RuntimeException: Cannot 
instantiate class. at 
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:384)
 at 
org.apache.flink.runtime.state.heap.StateTableByKeyGroupReaders.lambda$createV2PlusReader$0(StateTableByKeyGroupReaders.java:74)
 at 
org.apache.flink.runtime.state.KeyGroupPartitioner$PartitioningResultKeyGroupReader.readMappingsInKeyGroup(KeyGroupPartitioner.java:297)
 at 
org.apache.flink.runtime.state.heap.HeapRestoreOperation.readKeyGroupStateData(HeapRestoreOperation.java:290)
 at 
org.apache.flink.runtime.state.heap.HeapRestoreOperation.readStateHandleStateData(HeapRestoreOperation.java:251)
 at 
org.apache.flink.runtime.state.heap.HeapRestoreOperation.restore(HeapRestoreOperation.java:153)
 at 
org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.build(HeapKeyedStateBackendBuilder.java:127)
 ... 11 common frames omitted Caused by: java.lang.ClassNotFoundException: xxx 
at java.lang.Class.forName0(Native Method) at 
java.lang.Class.forName(Class.java:348) at 
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:382)
 ... 17 common frames omitted



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13160) HiveTaleSource should implment PartitionableTableSource

[zjuwangg (JIRA)]

zjuwangg created FLINK-13160:


 Summary: HiveTaleSource should implment PartitionableTableSource
 Key: FLINK-13160
 URL: https://issues.apache.org/jira/browse/FLINK-13160
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Hive
Reporter: zjuwangg
Assignee: zjuwangg






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] How to Deal with Split/Select in DataStream API

[Xingcan Cui]

Hi Aljoscha,

Thanks for your response.

With all this preliminary information collected, I’ll start a formal process.

Thank everybody for your attention.

Best,
Xingcan

> On Jul 8, 2019, at 10:17 AM, Aljoscha Krettek  wrote:
> 
> I think this would benefit from a FLIP, that neatly sums up the options, and 
> which then gives us also a point where we can vote and ratify a decision.
> 
> As a gut feeling, I most like Option 3). Initially I would have preferred 
> option 1) (because of a sense of API purity), but by now I think it’s good 
> that users have this simpler option.
> 
> Aljoscha 
> 
>> On 8. Jul 2019, at 06:39, Xingcan Cui > > wrote:
>> 
>> Hi all,
>> 
>> Thanks for your participation.
>> 
>> In this thread, we got one +1 for option 1 and option 3, respectively. In 
>> the original thread[1], we got two +1 for option 1, one +1 for option 2, and 
>> five +1 and one -1 for option 3.
>> 
>> To summarize,
>> 
>> Option 1 (port side output to flatMap and deprecate split/select): three +1
>> Option 2 (introduce a new split/select and deprecate existing one): one +1
>> Option 3 ("correct" the existing split/select): six +1 and one -1
>> 
>> It seems that most people involved are in favor of "correcting" the existing 
>> split/select. However, this will definitely break the API compatibility, in 
>> a subtle way.
>> 
>> IMO, the real behavior of consecutive split/select's has never been 
>> thoroughly clarified. Even in the community, it hard to say that we come 
>> into a consensus on its real semantics[2-4]. Though the initial design is 
>> not ambiguous, there's no doubt that its concept has drifted. 
>> 
>> As the split/select is quite an ancient API, I cc'ed this to more members. 
>> It couldn't be better if you can share your opinions on this.
>> 
>> Thanks,
>> Xingcan
>> 
>> [1] 
>> https://lists.apache.org/thread.html/f94ea5c97f96c705527dcc809b0e2b69e87a4c5d400cb7c61859e1f4@%3Cdev.flink.apache.org%3E
>>  
>> 
>> [2] https://issues.apache.org/jira/browse/FLINK-1772 
>> 
>> [3] https://issues.apache.org/jira/browse/FLINK-5031 
>> 
>> [4] https://issues.apache.org/jira/browse/FLINK-11084 
>> 
>> 
>> 
>>> On Jul 5, 2019, at 12:04 AM, 杨力 >> > wrote:
>>> 
>>> I prefer the 1) approach. I used to carry fields, which is needed only for 
>>> splitting, in the outputs of flatMap functions. Replacing it with 
>>> outputTags would simplify data structures.
>>> 
>>> Xingcan Cui mailto:xingc...@gmail.com> 
>>> >> 于 2019年7月5日周五 
>>> 上午2:20写道：
>>> Hi folks,
>>> 
>>> Two weeks ago, I started a thread [1] discussing whether we should discard 
>>> the split/select methods (which have been marked as deprecation since v1.7) 
>>> in DataStream API. 
>>> 
>>> The fact is, these methods will cause "unexpected" results when using 
>>> consecutively (e.g., ds.split(a).select(b).split(c).select(d)) or 
>>> multi-times on the same target (e.g., ds.split(a).select(b), 
>>> ds.split(c).select(d)). The reason is that following the initial design, 
>>> the new split/select logic will always override the existing one on the 
>>> same target operator, rather than append to it. Some users may not be aware 
>>> of that, but if you do, a current solution would be to use the more 
>>> powerful side output feature [2].
>>> 
>>> FLINK-11084 >> > added some 
>>> restrictions to the existing split/select logic and suggest to replace it 
>>> with side output in the future. However, considering that the side output 
>>> is currently only available in the process function layer and the 
>>> split/select could have been widely used in many real-world applications, 
>>> we'd like to start a vote andlisten to the community on how to deal with 
>>> them.
>>> 
>>> In the discussion thread [1], we proposed three solutions as follows. All 
>>> of them are feasible but have different impacts on the public API.
>>> 
>>> 1) Port the side output feature to DataStream API's flatMap and replace 
>>> split/select with it.
>>> 
>>> 2) Introduce a dedicated function in DataStream API (with the "correct" 
>>> behavior but a different name) that can be used to replace the existing 
>>> split/select.
>>> 
>>> 3) Keep split/select but change the behavior/semantic to be "correct".
>>> 
>>> Note that this is just a vote for gathering information, so feel free to 
>>> participate and share your opinions.
>>> 
>>> The voting time will end on July 7th 17:00 EDT.
>>> 
>>> Thanks,
>>> Xingcan
>>> 
>>> [1] 
>>> https://lists.apache.org/thread.html/f94ea5c97f96c705527dcc809b0e2b69e87a4c5d40

[jira] [Created] (FLINK-13161) numBuckets calculate wrong in BinaryHashBucketArea

[Louis Xu (JIRA)]

Louis Xu created FLINK-13161:


 Summary: numBuckets calculate wrong in BinaryHashBucketArea
 Key: FLINK-13161
 URL: https://issues.apache.org/jira/browse/FLINK-13161
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Runtime
Affects Versions: 1.9.0
Reporter: Louis Xu
Assignee: Louis Xu
 Fix For: 1.9.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13162) Default value of slot.idle.timeout is too large for batch job

[Jeff Zhang (JIRA)]

Jeff Zhang created FLINK-13162:
--

 Summary: Default value of slot.idle.timeout is too large for batch 
job
 Key: FLINK-13162
 URL: https://issues.apache.org/jira/browse/FLINK-13162
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Configuration, Runtime / Coordination
Affects Versions: 1.9.0
Reporter: Jeff Zhang


The default value of slot.idle.timeout is 50 seconds, it is too large for batch 
job.

It will cause downstream vertex unable to start even the upstream vertex is 
finished. It has to wait for 50 seconds. The default value of this kind of 
configuration doesn't make sense for batch job.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13163) Default value of slot.request.timeout is too small for batch job

[Jeff Zhang (JIRA)]

Jeff Zhang created FLINK-13163:
--

 Summary: Default value of slot.request.timeout is too small for 
batch job
 Key: FLINK-13163
 URL: https://issues.apache.org/jira/browse/FLINK-13163
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Configuration, Runtime / Coordination
Affects Versions: 1.9.0
Reporter: Jeff Zhang


The default value of slot.request.timeout is 5 minutes. It will cause the flink 
job fail if downstream vertex can not get resources in 5 minutes. Ideally, for 
batch job, it should wait there indefinitely. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)