[GitHub] [flink] godfreyhe commented on a change in pull request #14606: [Flink-20876][table-planner-blink] Separate the implementation of StreamExecTemporalJoin
godfreyhe commented on a change in pull request #14606:
URL: https://github.com/apache/flink/pull/14606#discussion_r557118638
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/batch/BatchPhysicalHashJoin.scala
##
@@ -162,38 +155,40 @@ class BatchPhysicalHashJoin(
}
override def translateToExecNode(): ExecNode[_] = {
-val nonEquiPredicates = if (!joinInfo.isEqui) {
- joinInfo.getRemaining(getCluster.getRexBuilder)
-} else {
- null
-}
+JoinUtil.validateJoinSpec(
+ joinSpec,
+ FlinkTypeFactory.toLogicalRowType(left.getRowType),
+ FlinkTypeFactory.toLogicalRowType(right.getRowType))
Review comment:
`BatchExecHashJoin` operator have validated when translating to plan, is
this validation needed ? if it needed, I would like to do the validation when
constructing this node
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/batch/BatchPhysicalHashJoin.scala
##
@@ -162,38 +155,40 @@ class BatchPhysicalHashJoin(
}
override def translateToExecNode(): ExecNode[_] = {
-val nonEquiPredicates = if (!joinInfo.isEqui) {
- joinInfo.getRemaining(getCluster.getRexBuilder)
-} else {
- null
-}
+JoinUtil.validateJoinSpec(
+ joinSpec,
+ FlinkTypeFactory.toLogicalRowType(left.getRowType),
+ FlinkTypeFactory.toLogicalRowType(right.getRowType))
+
val mq = getCluster.getMetadataQuery
val leftRowSize = Util.first(mq.getAverageRowSize(left), 24).toInt
val leftRowCount = Util.first(mq.getRowCount(left), 20).toLong
val rightRowSize = Util.first(mq.getAverageRowSize(right), 24).toInt
val rightRowCount = Util.first(mq.getRowCount(right), 20).toLong
+val (leftEdge, rightEdge) = getInputEdges
new BatchExecHashJoin(
-JoinTypeUtil.getFlinkJoinType(joinType),
-leftKeys,
-rightKeys,
-filterNulls,
-nonEquiPredicates,
+joinSpec,
leftRowSize,
rightRowSize,
leftRowCount,
rightRowCount,
leftIsBuild,
tryDistinctBuildRow,
-getInputEdges,
+leftEdge,
Review comment:
`BatchExecSortMergeJoin` and `BatchExecNestedLoopJoin` are missed.
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/rules/physical/stream/StreamPhysicalTemporalJoinRule.scala
##
@@ -20,19 +20,20 @@ package
org.apache.flink.table.planner.plan.rules.physical.stream
import org.apache.flink.table.planner.plan.nodes.FlinkRelNode
import org.apache.flink.table.planner.plan.nodes.logical._
-import
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTemporalJoin
+import
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalTemporalJoin
import org.apache.flink.table.planner.plan.utils.TemporalJoinUtil
+import org.apache.flink.util.Preconditions.checkState
+
import org.apache.calcite.plan.{RelOptRule, RelOptRuleCall, RelTraitSet}
import org.apache.calcite.rel.RelNode
import org.apache.calcite.rel.core.JoinRelType
-import org.apache.flink.util.Preconditions.checkState
/**
- * Rule that matches a temporal join node and converts it to
[[StreamExecTemporalJoin]],
- * the temporal join node is a [[FlinkLogicalJoin]] which contains
[[TemporalJoinCondition]].
+ * Rule that matches a temporal join node and converts it to
[[StreamPhysicalTemporalJoin]],
+ * the temporal join node is a [[FlinkLogicalJoin]] which contains
[[TEMPORAL_JOIN_CONDITION]].
*/
-class StreamExecTemporalJoinRule
- extends StreamExecJoinRuleBase("StreamExecJoinRuleBase") {
+class StreamPhysicalTemporalJoinRule
+ extends StreamPhysicalJoinRuleBase("StreamPhysicalJoinRuleBase") {
Review comment:
The description should be `StreamPhysicalTemporalJoinRule`
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/utils/JoinSpec.java
##
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.plan
[GitHub] [flink] flinkbot edited a comment on pull request #14536: [FLINK-20812][Connector][Hbase] hbase in sql mode,can use 'properties.*' add Configuration parameter.
flinkbot edited a comment on pull request #14536: URL: https://github.com/apache/flink/pull/14536#issuecomment-752909965 ## CI report: * a4c6b2e3d222c1679ce19d21a7f108d63d8dc3fc UNKNOWN * e8c1e77209e80aa39985342f01c3d8d566220d1a UNKNOWN * ba7aceff1a94c93ce89ed15359c992f62ad83e93 UNKNOWN * 3c42f8358ae07557917ce71eae8d092ed501b45d UNKNOWN * 477cd8c0b5b31588f7bd0174e0d87393a6df19ca UNKNOWN * 6558b9f806137c611bd648d3b9672c415a23b061 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12004) * cc47bd3f4429803694fed9fb9853827b11ed124c UNKNOWN * 3baca24234153ee39be0bc8ddd6a4e565e3d1eca Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12021) * a5cb7af54aab9fcd01b83e348949fc3623d34d26 UNKNOWN * da3e25fe1e5fa5031be6ba0ddacc4a27fefeacd1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14616: [FLINK-20879][table-api] Use MemorySize type instead of String type for memory ConfigOption in ExecutionConfigOptions
flinkbot edited a comment on pull request #14616: URL: https://github.com/apache/flink/pull/14616#issuecomment-758526377 ## CI report: * 448c026a402e045e050f405daf934a8a7c880c9d UNKNOWN * aac6ca6fa0792f2ecb4d982545fea6403115f24b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12031) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14614: [FLINK-20911][state-backend-rocksdb] Support configuration of RocksDB log level
flinkbot edited a comment on pull request #14614: URL: https://github.com/apache/flink/pull/14614#issuecomment-758512892 ## CI report: * 879417b9fbfe847c1a90de5caef7817edd10d69a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11931) * f3a5b0e4feb3bf16edecba4946fc0d160d7a4bce Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12030) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14627: [FLINK-20946][python] Optimize Python ValueState Implementation In PyFlink
flinkbot edited a comment on pull request #14627: URL: https://github.com/apache/flink/pull/14627#issuecomment-759272918 ## CI report: * a0e6a39605a3d1a80f75f8764534dfb82d08c31f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11971) * ff00387c962b937a5dabaaeb9241acfdbc9e49ab Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12032) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20967) Add Documentation for hbase connector new Function
WeiNan Zhao created FLINK-20967: --- Summary: Add Documentation for hbase connector new Function Key: FLINK-20967 URL: https://issues.apache.org/jira/browse/FLINK-20967 Project: Flink Issue Type: Task Components: Connectors / HBase, Documentation Affects Versions: 1.13.0 Reporter: WeiNan Zhao Fix For: 1.13.0 hbase connector version(1.4,2.2) can use 'properties.*' add configuration, need to add Documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20938) Implement Flink's own tencent COS filesystem
[ https://issues.apache.org/jira/browse/FLINK-20938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264678#comment-17264678 ] Shaun commented on FLINK-20938: --- * Development needs support > Implement Flink's own tencent COS filesystem > > > Key: FLINK-20938 > URL: https://issues.apache.org/jira/browse/FLINK-20938 > Project: Flink > Issue Type: New Feature > Components: FileSystems >Affects Versions: 1.12.0 >Reporter: hayden zhou >Priority: Major > > Tencent's COS is widely used among China's cloud users, and Hadoop supports > Tencent COS since 3.3.0. > Open this jira to wrap CosNFileSystem in FLINK(similar to oss support), so > that users can read from & write to COS more easily in FLINK. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14638: [FLINK-20966][table-planner-blink] Rename Stream(/Batch)ExecIntermediateTableScan to Stream(/Batch)PhysicalIntermediateTableScan
flinkbot edited a comment on pull request #14638: URL: https://github.com/apache/flink/pull/14638#issuecomment-759991130 ## CI report: * 99448538758f042069a6e9eb6e319726a54ca92c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12033) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-20967) Add documentation for the new introduced 'properties.*' option in HBase connector
[ https://issues.apache.org/jira/browse/FLINK-20967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-20967: --- Assignee: WeiNan Zhao > Add documentation for the new introduced 'properties.*' option in HBase > connector > - > > Key: FLINK-20967 > URL: https://issues.apache.org/jira/browse/FLINK-20967 > Project: Flink > Issue Type: Task > Components: Connectors / HBase, Documentation, Table SQL / Ecosystem >Affects Versions: 1.13.0 >Reporter: WeiNan Zhao >Assignee: WeiNan Zhao >Priority: Major > Fix For: 1.13.0 > > > hbase connector version(1.4,2.2) can use 'properties.*' add configuration, > need to add Documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20967) Add documentation for the new introduced 'properties.*' option in HBase connector
[ https://issues.apache.org/jira/browse/FLINK-20967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-20967: Summary: Add documentation for the new introduced 'properties.*' option in HBase connector (was: Add Documentation for hbase connector new Function) > Add documentation for the new introduced 'properties.*' option in HBase > connector > - > > Key: FLINK-20967 > URL: https://issues.apache.org/jira/browse/FLINK-20967 > Project: Flink > Issue Type: Task > Components: Connectors / HBase, Documentation >Affects Versions: 1.13.0 >Reporter: WeiNan Zhao >Priority: Major > Fix For: 1.13.0 > > > hbase connector version(1.4,2.2) can use 'properties.*' add configuration, > need to add Documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20967) Add documentation for the new introduced 'properties.*' option in HBase connector
[ https://issues.apache.org/jira/browse/FLINK-20967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-20967: Component/s: Table SQL / Ecosystem > Add documentation for the new introduced 'properties.*' option in HBase > connector > - > > Key: FLINK-20967 > URL: https://issues.apache.org/jira/browse/FLINK-20967 > Project: Flink > Issue Type: Task > Components: Connectors / HBase, Documentation, Table SQL / Ecosystem >Affects Versions: 1.13.0 >Reporter: WeiNan Zhao >Priority: Major > Fix For: 1.13.0 > > > hbase connector version(1.4,2.2) can use 'properties.*' add configuration, > need to add Documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20812) Support 'properties.*' option to pass through all the HBase properties
[ https://issues.apache.org/jira/browse/FLINK-20812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-20812: Summary: Support 'properties.*' option to pass through all the HBase properties (was: flink connector hbase(1.4,2.2) too few control parameters provided) > Support 'properties.*' option to pass through all the HBase properties > -- > > Key: FLINK-20812 > URL: https://issues.apache.org/jira/browse/FLINK-20812 > Project: Flink > Issue Type: Improvement > Components: Connectors / HBase, Table SQL / Ecosystem >Reporter: WeiNan Zhao >Assignee: WeiNan Zhao >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > When I use the cdh cluster, I need to use kerberos authentication, and I need > to add some kerberos authentication parameters of hbase, but the current > hbase connector structure does not provide this entry, I wonder if it can be > modified, if possible, I can submit for hbase connector a pr. > e.g hbase parameter > hbase.security.authentication='kerberos', > hbase.master.kerberos.principal='...', > hbase.kerberos.regionserver.principal='...', > hbase.security.auth.enable = 'true', > hbase.sasl.clientconfig = 'Client' -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20966) Rename Stream(/Batch)ExecIntermediateTableScan to Stream(/Batch)PhysicalIntermediateTableScan
[ https://issues.apache.org/jira/browse/FLINK-20966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] godfrey he reassigned FLINK-20966: -- Assignee: godfrey he > Rename Stream(/Batch)ExecIntermediateTableScan to > Stream(/Batch)PhysicalIntermediateTableScan > - > > Key: FLINK-20966 > URL: https://issues.apache.org/jira/browse/FLINK-20966 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong commented on pull request #14616: [FLINK-20879][table-api] Use MemorySize type instead of String type for memory ConfigOption in ExecutionConfigOptions
wuchong commented on pull request #14616: URL: https://github.com/apache/flink/pull/14616#issuecomment-760011288 The compile is failed again... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20968) Remove legacy exec nodes
godfrey he created FLINK-20968: -- Summary: Remove legacy exec nodes Key: FLINK-20968 URL: https://issues.apache.org/jira/browse/FLINK-20968 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: godfrey he Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] gaoyunhaii commented on pull request #13377: [FLINK-18592] bugfix for StreamingFileSink
gaoyunhaii commented on pull request #13377: URL: https://github.com/apache/flink/pull/13377#issuecomment-760012794 > Hi,@gaoyunhaii,could you rebase the branch to resovle the conflicts? Hi, @wangjian2019 should be the author of the PR, and @wangjian2019 could you help with the rebase~? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14536: [FLINK-20812][Connector][Hbase] hbase in sql mode,can use 'properties.*' add Configuration parameter.
flinkbot edited a comment on pull request #14536: URL: https://github.com/apache/flink/pull/14536#issuecomment-752909965 ## CI report: * a4c6b2e3d222c1679ce19d21a7f108d63d8dc3fc UNKNOWN * e8c1e77209e80aa39985342f01c3d8d566220d1a UNKNOWN * ba7aceff1a94c93ce89ed15359c992f62ad83e93 UNKNOWN * 3c42f8358ae07557917ce71eae8d092ed501b45d UNKNOWN * 477cd8c0b5b31588f7bd0174e0d87393a6df19ca UNKNOWN * 6558b9f806137c611bd648d3b9672c415a23b061 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12004) * cc47bd3f4429803694fed9fb9853827b11ed124c UNKNOWN * 3baca24234153ee39be0bc8ddd6a4e565e3d1eca Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12021) * a5cb7af54aab9fcd01b83e348949fc3623d34d26 UNKNOWN * da3e25fe1e5fa5031be6ba0ddacc4a27fefeacd1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12029) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20969) Update Scala 2.12 version to 2.12.13
Erwan Loisant created FLINK-20969: - Summary: Update Scala 2.12 version to 2.12.13 Key: FLINK-20969 URL: https://issues.apache.org/jira/browse/FLINK-20969 Project: Flink Issue Type: Task Components: Build System Reporter: Erwan Loisant Flink is compiled with Scala 2.12.7, that was released in 2018. The latest version of the 2.12 series is 2.12.13, and upgrading will bring build performance improvements as well as improving compatibility for projects using more recent versions of Scala. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14626: [FLINK-20948][table-planner-blink] Introduce StreamPhysicalDeduplicate, and make StreamExecDeduplicate only extended from ExecNode
flinkbot edited a comment on pull request #14626: URL: https://github.com/apache/flink/pull/14626#issuecomment-759272820 ## CI report: * 1387747f865a79d120ee7f5d6f24685c90e0076f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11970) * c5c5c0750d0bca932e319cb507e54897e0b2c4ec Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12023) * 9bf89577783692484d25df4186459cc1aaeb6000 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12034) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14631: [FLINK-20885][canal][json] Deserialization exception when using 'canal-json.table.include' to filter out the binlog of the specified
flinkbot edited a comment on pull request #14631: URL: https://github.com/apache/flink/pull/14631#issuecomment-759388845 ## CI report: * e283ba5376620acf2206a9c9ff7a4cdc9ba9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12018) * 571a38f52edb662a2b5c8a157ef96a52ad5ddb68 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12027) * da3e25fe1e5fa5031be6ba0ddacc4a27fefeacd1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14637: [FLINK-20949][table-planner-blink] Separate the implementation of sink nodes
flinkbot edited a comment on pull request #14637: URL: https://github.com/apache/flink/pull/14637#issuecomment-759980085 ## CI report: * 8b2bf9a0a0d1c74ea4a6d1712dccb415dcd34a4d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12028) * afd1a28c76fbfc06338673821228f041812618cb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20969) Update Scala 2.12 version to 2.12.13
[ https://issues.apache.org/jira/browse/FLINK-20969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwan Loisant updated FLINK-20969: -- Component/s: (was: Build System) API / Scala > Update Scala 2.12 version to 2.12.13 > > > Key: FLINK-20969 > URL: https://issues.apache.org/jira/browse/FLINK-20969 > Project: Flink > Issue Type: Task > Components: API / Scala >Reporter: Erwan Loisant >Priority: Major > > Flink is compiled with Scala 2.12.7, that was released in 2018. > The latest version of the 2.12 series is 2.12.13, and upgrading will bring > build performance improvements as well as improving compatibility for > projects using more recent versions of Scala. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] xiaoHoly commented on pull request #14616: [FLINK-20879][table-api] Use MemorySize type instead of String type for memory ConfigOption in ExecutionConfigOptions
xiaoHoly commented on pull request #14616: URL: https://github.com/apache/flink/pull/14616#issuecomment-760015927 > The compile is failed again... fix again This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] erwan opened a new pull request #14639: [FLINK-20969][API / Scala] Upgrade Scala 2.12 version to 2.12.13
erwan opened a new pull request #14639: URL: https://github.com/apache/flink/pull/14639 ## What is the purpose of the change Flink is compiled with Scala 2.12.7, that was released in 2018. The latest version of the 2.12 series is 2.12.13, and upgrading will bring build performance improvements as well as improving compatibility for projects using more recent versions of Scala. ## Brief change log - Bump Scala 2.12 version to 2.12.13 - Fix tests because of a call that has been removed. Note that the `finish()` method already existed in 2.12.7, and all it did was calling `printSummary()`. ## Verifying this change This change doesn't introduce any new behavior, so it is covered by the existing test suite of the Scala API. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20969) Update Scala 2.12 version to 2.12.13
[ https://issues.apache.org/jira/browse/FLINK-20969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-20969: --- Labels: pull-request-available (was: ) > Update Scala 2.12 version to 2.12.13 > > > Key: FLINK-20969 > URL: https://issues.apache.org/jira/browse/FLINK-20969 > Project: Flink > Issue Type: Task > Components: API / Scala >Reporter: Erwan Loisant >Priority: Major > Labels: pull-request-available > > Flink is compiled with Scala 2.12.7, that was released in 2018. > The latest version of the 2.12 series is 2.12.13, and upgrading will bring > build performance improvements as well as improving compatibility for > projects using more recent versions of Scala. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14639: [FLINK-20969][API / Scala] Upgrade Scala 2.12 version to 2.12.13
flinkbot commented on pull request #14639: URL: https://github.com/apache/flink/pull/14639#issuecomment-760019540 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 8b40b5ac2faf8d07403c16dcd975ddee473ef8a1 (Thu Jan 14 08:33:13 UTC 2021) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-20969).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-13414) Add support for Scala 2.13
[ https://issues.apache.org/jira/browse/FLINK-13414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264691#comment-17264691 ] Erwan Loisant commented on FLINK-13414: --- As we're working on upping the Scala versions, I've also created a ticket and a PR to upgrade the 2.12 version: * https://issues.apache.org/jira/browse/FLINK-20969 * [https://github.com/apache/flink/pull/14639] > Add support for Scala 2.13 > -- > > Key: FLINK-13414 > URL: https://issues.apache.org/jira/browse/FLINK-13414 > Project: Flink > Issue Type: New Feature > Components: API / Scala >Reporter: Chaoran Yu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20840) Projection pushdown doesn't work in temporal(lookup) join
[
https://issues.apache.org/jira/browse/FLINK-20840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258216#comment-17258216
]
Leonard Xu edited comment on FLINK-20840 at 1/14/21, 8:33 AM:
--
[~fsk119]
The rule `FlinkProjectJoinTransposeRule` was added in logical optimize phase
which uses volcano optimizer, the optimized plan is uncertain for example above
case.
was (Author: leonard xu):
[~fsk119]
The rule `FlinkProjectJoinTransposeRule` was added in logical rewrite phase
which uses volcano optimizer, the optimized plan is uncertain for example above
case.
> Projection pushdown doesn't work in temporal(lookup) join
> --
>
> Key: FLINK-20840
> URL: https://issues.apache.org/jira/browse/FLINK-20840
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.12.0
>Reporter: Leonard Xu
>Assignee: Leonard Xu
>Priority: Major
> Fix For: 1.13.0
>
>
> {code:java}
> sql 1:
> |SELECT T.*, D.id
> |FROM MyTable AS T
> |JOIN LookupTable FOR SYSTEM_TIME AS OF T.proctime AS D
> |ON T.a = D.id
> optmized plan:
> Calc(select=[a, b, c, PROCTIME_MATERIALIZE(proctime) AS proctime, rowtime,
> id])
> +- LookupJoin(table=[default_catalog.default_database.LookupTable],
> joinType=[InnerJoin], async=[false], lookup=[id=a], select=[a, b, c,
> proctime, rowtime, id, name, age])
>+- DataStreamScan(table=[[default_catalog, default_database, MyTable]],
> fields=[a, b, c, proctime, rowtime])
> sql 2:
> |SELECT T.a, D.id
> |FROM MyTable AS T
> |JOIN LookupTable FOR SYSTEM_TIME AS OF T.proctime AS D
> |ON T.a = D.id
> optmized plan:
> LookupJoin(table=[default_catalog.default_database.LookupTable],
> joinType=[InnerJoin], async=[false], lookup=[id=a], select=[a, id])
> +- Calc(select=[a])
>+- DataStreamScan(table=[[default_catalog, default_database, MyTable]],
> fields=[a, b, c, proctime, rowtime])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20969) Update Scala 2.12 version to 2.12.13
[
https://issues.apache.org/jira/browse/FLINK-20969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264694#comment-17264694
]
Erwan Loisant commented on FLINK-20969:
---
I'm getting a compatibility warning, what would be the right way to fix that?
{{[ERROR] Failed to execute goal
com.github.siom79.japicmp:japicmp-maven-plugin:0.11.0:cmp (default) on project
flink-scala_2.12: Breaking the build because there is at least one
incompatibility:
org.apache.flink.api.scala.CoGroupDataSet.$anonfun$buildGroupSortList$1(org.apache.flink.api.common.typeinfo.TypeInformation,java.util.ArrayList,scala.Tuple2):METHOD_REMOVED,org.apache.flink.api.scala.typeutils.CaseClassTypeInfo.$anonfun$getFlatFields$1(int,java.util.List,scala.runtime.IntRef,org.apache.flink.api.common.typeinfo.TypeInformation):METHOD_REMOVED,org.apache.flink.api.scala.typeutils.CaseClassTypeInfo.$anonfun$getFlatFields$1$adapted(int,java.util.List,scala.runtime.IntRef,org.apache.flink.api.common.typeinfo.TypeInformation):METHOD_REMOVED
-> [Help 1]}}
> Update Scala 2.12 version to 2.12.13
>
>
> Key: FLINK-20969
> URL: https://issues.apache.org/jira/browse/FLINK-20969
> Project: Flink
> Issue Type: Task
> Components: API / Scala
>Reporter: Erwan Loisant
>Priority: Major
> Labels: pull-request-available
>
> Flink is compiled with Scala 2.12.7, that was released in 2018.
> The latest version of the 2.12 series is 2.12.13, and upgrading will bring
> build performance improvements as well as improving compatibility for
> projects using more recent versions of Scala.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhuxiaoshang opened a new pull request #14640: [FLINK-20937][Table API/SQL]Drop table can fails, when i create a tab…
zhuxiaoshang opened a new pull request #14640: URL: https://github.com/apache/flink/pull/14640 …le with "watermark for proctime()" syntax ## What is the purpose of the change *don't need to resolve schema when drop table * ## Brief change log *(for example:)* - *don't need to resolve schema when drop table* ## Verifying this change This change added tests and can be verified as follows: *(example:)* - testDropTableWithWatermakForProcTime ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no)no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no)no - The serializers: (yes / no / don't know)no - The runtime per-record code paths (performance sensitive): (yes / no / don't know)no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know)no - The S3 file system connector: (yes / no / don't know)no ## Documentation - Does this pull request introduce a new feature? (yes / no)no - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)no This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20937) Drop table can fails, when i create a table with "watermark for proctime()" syntax
[ https://issues.apache.org/jira/browse/FLINK-20937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-20937: --- Labels: pull-request-available starter (was: starter) > Drop table can fails, when i create a table with "watermark for proctime()" > syntax > -- > > Key: FLINK-20937 > URL: https://issues.apache.org/jira/browse/FLINK-20937 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.12.0 >Reporter: MarcelLeon >Assignee: zhuxiaoshang >Priority: Major > Labels: pull-request-available, starter > Fix For: 1.13.0 > > Attachments: drop table fail.png > > > In sql-client: > I create a kafka table with "watermark for proctime()" syntax. > Then i try execute a query or drop it, it report fails. > I know point watermark with process_time is meaningless, but while i drop it > doesn't work , it will affect use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14640: [FLINK-20937][Table API/SQL]Drop table can fails, when i create a tab…
flinkbot commented on pull request #14640: URL: https://github.com/apache/flink/pull/14640#issuecomment-760025295 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit cd36249da9dfea6f8a8ce9eccfe89c220535131e (Thu Jan 14 08:43:56 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14530: [FLINK-20348][kafka] Make "schema-registry.subject" optional for Kafka sink with avro-confluent format
flinkbot edited a comment on pull request #14530: URL: https://github.com/apache/flink/pull/14530#issuecomment-752828495 ## CI report: * eaa0445691254f544c231ec0d2d55519af277b33 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12025) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14606: [Flink-20876][table-planner-blink] Separate the implementation of StreamExecTemporalJoin
flinkbot edited a comment on pull request #14606: URL: https://github.com/apache/flink/pull/14606#issuecomment-757754275 ## CI report: * 3776b52cfe3535dcc193b3a922a7d1d658126d66 UNKNOWN * 155b18c169e45a97cd52c5b43883d5cf6b79f038 UNKNOWN * 29868c9db791dc78af63512150e7f5c6a82950ea Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11996) * 3e0f602eef1ff57ac68984e2be4f784dd33ac7f6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14616: [FLINK-20879][table-api] Use MemorySize type instead of String type for memory ConfigOption in ExecutionConfigOptions
flinkbot edited a comment on pull request #14616: URL: https://github.com/apache/flink/pull/14616#issuecomment-758526377 ## CI report: * 448c026a402e045e050f405daf934a8a7c880c9d UNKNOWN * aac6ca6fa0792f2ecb4d982545fea6403115f24b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12031) * f429967b7680aa4dce4212ddb01b2c107b3b7117 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14631: [FLINK-20885][canal][json] Deserialization exception when using 'canal-json.table.include' to filter out the binlog of the specified
flinkbot edited a comment on pull request #14631: URL: https://github.com/apache/flink/pull/14631#issuecomment-759388845 ## CI report: * e283ba5376620acf2206a9c9ff7a4cdc9ba9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12018) * 571a38f52edb662a2b5c8a157ef96a52ad5ddb68 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12027) * da3e25fe1e5fa5031be6ba0ddacc4a27fefeacd1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12029) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12035) * 5adfea8032e4ed53eb1e30f3d864f08271389a8b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14637: [FLINK-20949][table-planner-blink] Separate the implementation of sink nodes
flinkbot edited a comment on pull request #14637: URL: https://github.com/apache/flink/pull/14637#issuecomment-759980085 ## CI report: * 8b2bf9a0a0d1c74ea4a6d1712dccb415dcd34a4d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12028) * afd1a28c76fbfc06338673821228f041812618cb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12036) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14639: [FLINK-20969][API / Scala] Upgrade Scala 2.12 version to 2.12.13
flinkbot commented on pull request #14639: URL: https://github.com/apache/flink/pull/14639#issuecomment-760032294 ## CI report: * 8b40b5ac2faf8d07403c16dcd975ddee473ef8a1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Reopened] (FLINK-19158) Revisit java e2e download timeouts
[
https://issues.apache.org/jira/browse/FLINK-19158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann reopened FLINK-19158:
---
[~rmetzger] could you please take another look?
> Revisit java e2e download timeouts
> --
>
> Key: FLINK-19158
> URL: https://issues.apache.org/jira/browse/FLINK-19158
> Project: Flink
> Issue Type: Improvement
> Components: Build System
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> Consider this failed test case
> {code}
> Test testHBase(org.apache.flink.tests.util.hbase.SQLClientHBaseITCase) is
> running.
>
> 09:38:38,719 [main] INFO
> org.apache.flink.tests.util.cache.PersistingDownloadCache[] - Downloading
> https://archive.apache.org/dist/hbase/1.4.3/hbase-1.4.3-bin.tar.gz.
> 09:40:38,732 [main] ERROR
> org.apache.flink.tests.util.hbase.SQLClientHBaseITCase [] -
>
> Test testHBase(org.apache.flink.tests.util.hbase.SQLClientHBaseITCase) failed
> with:
> java.io.IOException: Process ([wget, -q, -P,
> /home/vsts/work/1/e2e_cache/downloads/1598516010,
> https://archive.apache.org/dist/hbase/1.4.3/hbase-1.4.3-bin.tar.gz]) exceeded
> timeout (12) or number of retries (3).
> at
> org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlockingWithRetry(AutoClosableProcess.java:148)
> at
> org.apache.flink.tests.util.cache.AbstractDownloadCache.getOrDownload(AbstractDownloadCache.java:127)
> at
> org.apache.flink.tests.util.cache.PersistingDownloadCache.getOrDownload(PersistingDownloadCache.java:36)
> at
> org.apache.flink.tests.util.hbase.LocalStandaloneHBaseResource.setupHBaseDist(LocalStandaloneHBaseResource.java:76)
> at
> org.apache.flink.tests.util.hbase.LocalStandaloneHBaseResource.before(LocalStandaloneHBaseResource.java:70)
> at
> org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:46)
> {code}
> It seems that the download has not been retried. The download might be stuck?
> I would propose to set a timeout per try and increase the total time from 2
> to 5 minutes.
> This example is from:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=6267&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=ff888d9b-cd34-53cc-d90f-3e446d355529
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-20835) Implement FineGrainedSlotManager
[ https://issues.apache.org/jira/browse/FLINK-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song reassigned FLINK-20835: Assignee: Yangze Guo > Implement FineGrainedSlotManager > > > Key: FLINK-20835 > URL: https://issues.apache.org/jira/browse/FLINK-20835 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Yangze Guo >Assignee: Yangze Guo >Priority: Major > Fix For: 1.13.0 > > > Introduce SlotManager plugin for fine-grained resource management. > - Request slot from TaskExecutor with the actual resource profiles. > - Use ResourceTracker to bookkeep the resource requirements > - Introduce TaskExecutorTracker, which bookkeep the total / available > resource and slot assignment for registered task executor. > ** Bookkeep task manager total and available resources. > ** Bookkeep slot allocations and assignments. > ** Intorduce PendingTaskManager. > ** Map registered task executors to matching PendingTaskManager. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20970) DECIMAL(10, 0) can not be GROUP BY key.
Wong Mulan created FLINK-20970: -- Summary: DECIMAL(10, 0) can not be GROUP BY key. Key: FLINK-20970 URL: https://issues.apache.org/jira/browse/FLINK-20970 Project: Flink Issue Type: Bug Components: Table SQL / Runtime Affects Versions: 1.10.1 Reporter: Wong Mulan Attachments: image-2021-01-14-17-06-28-648.png If value which type is not DECIMAL(38, 18) is GROUP BY key, the result will be -1. So, only DECIMAL(38, 18) can be GROUP BY key? !image-2021-01-14-17-06-28-648.png|thumbnail! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14616: [FLINK-20879][table-api] Use MemorySize type instead of String type for memory ConfigOption in ExecutionConfigOptions
flinkbot edited a comment on pull request #14616: URL: https://github.com/apache/flink/pull/14616#issuecomment-758526377 ## CI report: * 448c026a402e045e050f405daf934a8a7c880c9d UNKNOWN * aac6ca6fa0792f2ecb4d982545fea6403115f24b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12031) * f429967b7680aa4dce4212ddb01b2c107b3b7117 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12037) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14639: [FLINK-20969][API / Scala] Upgrade Scala 2.12 version to 2.12.13
flinkbot edited a comment on pull request #14639: URL: https://github.com/apache/flink/pull/14639#issuecomment-760032294 ## CI report: * 8b40b5ac2faf8d07403c16dcd975ddee473ef8a1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12038) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14640: [FLINK-20937][Table API/SQL]Drop table can fails, when i create a tab…
flinkbot commented on pull request #14640: URL: https://github.com/apache/flink/pull/14640#issuecomment-760055828 ## CI report: * cd36249da9dfea6f8a8ce9eccfe89c220535131e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20970) DECIMAL(10, 0) can not be GROUP BY key.
[ https://issues.apache.org/jira/browse/FLINK-20970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wong Mulan updated FLINK-20970: --- Description: If value which type is not DECIMAL(38, 18) is GROUP BY key, the result will be -1. So, only DECIMAL(38, 18) can be GROUP BY key? Whatever the value is, it will be return -1. !image-2021-01-14-17-06-28-648.png|thumbnail! was: If value which type is not DECIMAL(38, 18) is GROUP BY key, the result will be -1. So, only DECIMAL(38, 18) can be GROUP BY key? !image-2021-01-14-17-06-28-648.png|thumbnail! > DECIMAL(10, 0) can not be GROUP BY key. > --- > > Key: FLINK-20970 > URL: https://issues.apache.org/jira/browse/FLINK-20970 > Project: Flink > Issue Type: Bug > Components: Table SQL / Runtime >Affects Versions: 1.10.1 >Reporter: Wong Mulan >Priority: Major > Attachments: image-2021-01-14-17-06-28-648.png > > > If value which type is not DECIMAL(38, 18) is GROUP BY key, the result will > be -1. > So, only DECIMAL(38, 18) can be GROUP BY key? > Whatever the value is, it will be return -1. > !image-2021-01-14-17-06-28-648.png|thumbnail! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] HuangXingBo opened a new pull request #14641: [FLINK-20751][python][table-planner-blink] Port batch python group aggregate nodes to Java
HuangXingBo opened a new pull request #14641: URL: https://github.com/apache/flink/pull/14641 ## What is the purpose of the change *This pull request will Port batch python group aggregate nodes to Java* ## Brief change log - *Port batch python group aggregate nodes to Java* ## Verifying this change This change added tests and can be verified as follows: - *Original Tests* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20341) Rework Deployment / Coordination Documentation
[ https://issues.apache.org/jira/browse/FLINK-20341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann closed FLINK-20341. - Resolution: Fixed Done since all subtasks have been completed. > Rework Deployment / Coordination Documentation > -- > > Key: FLINK-20341 > URL: https://issues.apache.org/jira/browse/FLINK-20341 > Project: Flink > Issue Type: Task > Components: Documentation, Runtime / Coordination >Reporter: Robert Metzger >Assignee: Robert Metzger >Priority: Major > Fix For: 1.13.0 > > > Problems: > - Clusters & Deployment pages are very inhomogeneous > - Overview page has good intentions, but is a huge wall of text > - Native K8s and YARN have a “Background / Internals” page > - difference between Local Cluster and Standalone Cluster is unclear > Goals: > - Deploying a Flink cluster is one of the first tasks when getting to know > Flink. We need proper guidance for making these steps a success > - We need a proper separation between general concepts (HA, session/perjob > mode) and implementations of them (ZK HA, K8s HA, YARN session, …). Also > orthogonal aspects such as FileSystems, Plugins, Security etc. > Related work: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-42%3A+Rework+Flink+Documentation > (see “Deployment Section”) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20751) Port batch python group aggregate nodes to Java
[ https://issues.apache.org/jira/browse/FLINK-20751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-20751: --- Labels: pull-request-available (was: ) > Port batch python group aggregate nodes to Java > --- > > Key: FLINK-20751 > URL: https://issues.apache.org/jira/browse/FLINK-20751 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: godfrey he >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14640: [FLINK-20937][Table API/SQL]Drop table can fails, when i create a tab…
flinkbot edited a comment on pull request #14640: URL: https://github.com/apache/flink/pull/14640#issuecomment-760055828 ## CI report: * cd36249da9dfea6f8a8ce9eccfe89c220535131e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12039) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20971) Allow SQL expressions in Table API
Timo Walther created FLINK-20971:
Summary: Allow SQL expressions in Table API
Key: FLINK-20971
URL: https://issues.apache.org/jira/browse/FLINK-20971
Project: Flink
Issue Type: Sub-task
Components: Table SQL / API
Reporter: Timo Walther
Assignee: Timo Walther
Both FLIP-136 and FLIP-129 assume that a table schema is able to store both a
SQL expression and a Table API one. Therefore we should introduce an expression
in Table API that allows to use SQL expressions. This is also useful for other
use cases where people prefer SQL expressions in Table API.
An expression `callSql(String)` next to `call(udf)` can offer scalar operations
such as:
{code}
table.select(callSql("1+1"), 12, call(Udf.class, $("f0")))
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20944) Launching in application mode requesting a ClusterIP rest service type results in an Exception
[
https://issues.apache.org/jira/browse/FLINK-20944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-20944:
--
Fix Version/s: 1.12.2
1.13.0
> Launching in application mode requesting a ClusterIP rest service type
> results in an Exception
> --
>
> Key: FLINK-20944
> URL: https://issues.apache.org/jira/browse/FLINK-20944
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Affects Versions: 1.12.0
> Environment: Kubernetes 1.17
> Flink 1.12
> Running ./bin/flink from an Ubuntu 18.04 host
>Reporter: Jason Brome
>Priority: Critical
> Fix For: 1.13.0, 1.12.2
>
>
> Run a Flink job in Kubernetes in application mode, specifying
> kubernetes.rest-service.exposed.type=ClusterIP, results in the job being
> started, however the call to ./bin/flink throws an UnknownHostException
> Exception on the client.
> Command line:
> {{./bin/flink run-application --target kubernetes-application
> -Dkubernetes.cluster-id=myjob-qa
> -Dkubernetes.container.image=_SOME_REDACTED_PATH/somrepo/someimage_
> -Dkubernetes.service-account=flink-service-account
> -Dkubernetes.namespace=myjob-qa
> -Dkubernetes.rest-service.exposed.type=ClusterIP local:///opt/flink}}
> {{/usrlib/my-job.jar}}
> Output:
> 2021-01-12 20:29:19,047 INFO
> org.apache.flink.kubernetes.utils.KubernetesUtils [] - Kubernetes deployment
> requires a fixed port. Configuration blob.server.port will be set to 6124
> 2021-01-12 20:29:19,048 INFO
> org.apache.flink.kubernetes.utils.KubernetesUtils [] - Kubernetes deployment
> requires a fixed port. Configuration taskmanager.rpc.port will be set to 6122
> 2021-01-12 20:29:20,369 ERROR
> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient [] - A
> Kubernetes exception occurred.
> java.net.UnknownHostException: myjob-qa-rest.myjob-qa: Name or service not
> known
> at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0_275]
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
> ~[?:1.8.0_275]
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
> ~[?:1.8.0_275]
> at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[?:1.8.0_275]
> at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_275]
> at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_275]
> at java.net.InetAddress.getByName(InetAddress.java:1077) ~[?:1.8.0_275]
> at
> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.getWebMonitorAddress(HighAvailabilityServicesUtils.java:193)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.kubernetes.KubernetesClusterDescriptor.lambda$createClusterClientProvider$0(KubernetesClusterDescriptor.java:114)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployApplicationCluster(KubernetesClusterDescriptor.java:185)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:64)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.client.cli.CliFrontend.runApplication(CliFrontend.java:207)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:974)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1047)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> [flink-dist_2.12-1.12.0.jar:1.12.0]
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1047)
> [flink-dist_2.12-1.12.0.jar:1.12.0]
>
> The program finished with the following exception:
> java.lang.RuntimeException:
> org.apache.flink.client.deployment.ClusterRetrieveException: Could not create
> the RestClusterClient.
> at
> org.apache.flink.kubernetes.KubernetesClusterDescriptor.lambda$createClusterClientProvider$0(KubernetesClusterDescriptor.java:118)
> at
> org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployApplicationCluster(KubernetesClusterDescriptor.java:185)
> at
> org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:64)
> at
> org.apache.flink.client.cli.CliFrontend.runApplication(CliFrontend.java:207)
> at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:974)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1047)
> at
> org.apache.flink.runtime.security.contexts.NoOpS
[jira] [Updated] (FLINK-20944) Launching in application mode requesting a ClusterIP rest service type results in an Exception
[
https://issues.apache.org/jira/browse/FLINK-20944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-20944:
--
Priority: Critical (was: Major)
> Launching in application mode requesting a ClusterIP rest service type
> results in an Exception
> --
>
> Key: FLINK-20944
> URL: https://issues.apache.org/jira/browse/FLINK-20944
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Affects Versions: 1.12.0
> Environment: Kubernetes 1.17
> Flink 1.12
> Running ./bin/flink from an Ubuntu 18.04 host
>Reporter: Jason Brome
>Priority: Critical
>
> Run a Flink job in Kubernetes in application mode, specifying
> kubernetes.rest-service.exposed.type=ClusterIP, results in the job being
> started, however the call to ./bin/flink throws an UnknownHostException
> Exception on the client.
> Command line:
> {{./bin/flink run-application --target kubernetes-application
> -Dkubernetes.cluster-id=myjob-qa
> -Dkubernetes.container.image=_SOME_REDACTED_PATH/somrepo/someimage_
> -Dkubernetes.service-account=flink-service-account
> -Dkubernetes.namespace=myjob-qa
> -Dkubernetes.rest-service.exposed.type=ClusterIP local:///opt/flink}}
> {{/usrlib/my-job.jar}}
> Output:
> 2021-01-12 20:29:19,047 INFO
> org.apache.flink.kubernetes.utils.KubernetesUtils [] - Kubernetes deployment
> requires a fixed port. Configuration blob.server.port will be set to 6124
> 2021-01-12 20:29:19,048 INFO
> org.apache.flink.kubernetes.utils.KubernetesUtils [] - Kubernetes deployment
> requires a fixed port. Configuration taskmanager.rpc.port will be set to 6122
> 2021-01-12 20:29:20,369 ERROR
> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient [] - A
> Kubernetes exception occurred.
> java.net.UnknownHostException: myjob-qa-rest.myjob-qa: Name or service not
> known
> at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0_275]
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
> ~[?:1.8.0_275]
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
> ~[?:1.8.0_275]
> at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[?:1.8.0_275]
> at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_275]
> at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_275]
> at java.net.InetAddress.getByName(InetAddress.java:1077) ~[?:1.8.0_275]
> at
> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.getWebMonitorAddress(HighAvailabilityServicesUtils.java:193)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.kubernetes.KubernetesClusterDescriptor.lambda$createClusterClientProvider$0(KubernetesClusterDescriptor.java:114)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployApplicationCluster(KubernetesClusterDescriptor.java:185)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:64)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.client.cli.CliFrontend.runApplication(CliFrontend.java:207)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:974)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1047)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> [flink-dist_2.12-1.12.0.jar:1.12.0]
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1047)
> [flink-dist_2.12-1.12.0.jar:1.12.0]
>
> The program finished with the following exception:
> java.lang.RuntimeException:
> org.apache.flink.client.deployment.ClusterRetrieveException: Could not create
> the RestClusterClient.
> at
> org.apache.flink.kubernetes.KubernetesClusterDescriptor.lambda$createClusterClientProvider$0(KubernetesClusterDescriptor.java:118)
> at
> org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployApplicationCluster(KubernetesClusterDescriptor.java:185)
> at
> org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:64)
> at
> org.apache.flink.client.cli.CliFrontend.runApplication(CliFrontend.java:207)
> at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:974)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1047)
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
[GitHub] [flink] flinkbot commented on pull request #14641: [FLINK-20751][python][table-planner-blink] Port batch python group aggregate nodes to Java
flinkbot commented on pull request #14641: URL: https://github.com/apache/flink/pull/14641#issuecomment-760081285 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit e0c4115590cdbea89d30285c22f30208cf3895ca (Thu Jan 14 09:43:30 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-20751).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20919) After the flink batch job is completed, the yarn application cannot be completed.
[
https://issues.apache.org/jira/browse/FLINK-20919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264722#comment-17264722
]
Till Rohrmann commented on FLINK-20919:
---
Could you maybe also share the client log file of your submission [~Janze]?
Maybe there is indeed a bug in the client.
> After the flink batch job is completed, the yarn application cannot be
> completed.
> -
>
> Key: FLINK-20919
> URL: https://issues.apache.org/jira/browse/FLINK-20919
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.2
>Reporter: Wu
>Priority: Major
> Attachments: flink-Overview.png, flink-completed.png,
> flink-jobManager.png, jobmanager.log, kaLr8Coy.png
>
>
> I submit flink batch job in yarn-cluster mode. After the flink batch job is
> completed, the yarn application cannot be completed. The yarn application
> still occupies a vcore. How to automatically close the yarn application.
>
> {code:java}
> //代码占位符
> EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inBatchMode().build();
> TableEnvironment tableEnv = TableEnvironment.create(settings);
> tableEnv.executeSql("create table file_table");
> tableEnv.executeSql("create table print_table");
> String sql = "select count(1) from file_table";
> Table table = tableEnv.sqlQuery(sql);
> tableEnv.createTemporaryView("t", table);
> tableEnv.from("t").executeInsert("print_table");
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20927) Update configuration option in YarnConfigOptions class
[ https://issues.apache.org/jira/browse/FLINK-20927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264726#comment-17264726 ] Till Rohrmann commented on FLINK-20927: --- You can update them but it is not very important [~jiang7chengzitc]. > Update configuration option in YarnConfigOptions class > -- > > Key: FLINK-20927 > URL: https://issues.apache.org/jira/browse/FLINK-20927 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.12.0 >Reporter: Ruguo Yu >Priority: Minor > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-01-11-18-36-20-723.png, > image-2021-01-11-18-36-44-811.png > > > There are many configuration options that use a not recommended method to > build in _YarnConfigOptions_ class, mainly include the following: > 1. Use the deprecated method 'defaultValue' directly instead of specifying > the data type. > !image-2021-01-11-18-36-20-723.png|width=842,height=90! > 2. Use String instead of Enum types. > !image-2021-01-11-18-36-44-811.png|width=1128,height=139! > > Therefore, for the above problem, we can update it with the new API, which > can also be consistent with KubernetesConfigOptions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] AHeise commented on a change in pull request #13551: [FLINK-19520][configuration] Add randomization of checkpoint config.
AHeise commented on a change in pull request #13551:
URL: https://github.com/apache/flink/pull/13551#discussion_r557266156
##
File path:
flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/streaming/util/TestStreamEnvironment.java
##
@@ -69,13 +74,28 @@ public static void setAsContext(
TestStreamEnvironment env =
new TestStreamEnvironment(
miniCluster, parallelism, jarFiles,
classpaths);
+randomize(conf);
env.configure(conf, env.getUserClassloader());
return env;
};
initializeContextEnvironment(factory);
}
+/**
+ * Randomizes configuration on test case level even if mini cluster is
used in a class rule.
+ *
+ * @param conf
+ */
+private static void randomize(Configuration conf) {
+if (Randomization) {
+final String testName = TestNameProvider.getCurrentTestName();
Review comment:
To make it independent of `TestLogger` we could use
[RunListener](https://junit.org/junit4/javadoc/4.12/org/junit/runner/notification/RunListener.html)
and then have a singleton listener to fetch the name.
However, I didn't go fully into this direction as it's quite a bit of
fiddling in junit4 and I hope to eventually migrate to junit 5. Then I'd check
if it can be done in a better way.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14606: [Flink-20876][table-planner-blink] Separate the implementation of StreamExecTemporalJoin
flinkbot edited a comment on pull request #14606: URL: https://github.com/apache/flink/pull/14606#issuecomment-757754275 ## CI report: * 3776b52cfe3535dcc193b3a922a7d1d658126d66 UNKNOWN * 155b18c169e45a97cd52c5b43883d5cf6b79f038 UNKNOWN * 29868c9db791dc78af63512150e7f5c6a82950ea Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11996) * 3e0f602eef1ff57ac68984e2be4f784dd33ac7f6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12040) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14614: [FLINK-20911][state-backend-rocksdb] Support configuration of RocksDB log level
flinkbot edited a comment on pull request #14614: URL: https://github.com/apache/flink/pull/14614#issuecomment-758512892 ## CI report: * f3a5b0e4feb3bf16edecba4946fc0d160d7a4bce Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12030) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14631: [FLINK-20885][canal][json] Deserialization exception when using 'canal-json.table.include' to filter out the binlog of the specified
flinkbot edited a comment on pull request #14631: URL: https://github.com/apache/flink/pull/14631#issuecomment-759388845 ## CI report: * e283ba5376620acf2206a9c9ff7a4cdc9ba9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12018) * 571a38f52edb662a2b5c8a157ef96a52ad5ddb68 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12027) * da3e25fe1e5fa5031be6ba0ddacc4a27fefeacd1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12029) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12035) * 5adfea8032e4ed53eb1e30f3d864f08271389a8b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12041) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14637: [FLINK-20949][table-planner-blink] Separate the implementation of sink nodes
flinkbot edited a comment on pull request #14637: URL: https://github.com/apache/flink/pull/14637#issuecomment-759980085 ## CI report: * afd1a28c76fbfc06338673821228f041812618cb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12036) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14641: [FLINK-20751][python][table-planner-blink] Port batch python group aggregate nodes to Java
flinkbot commented on pull request #14641: URL: https://github.com/apache/flink/pull/14641#issuecomment-760086252 ## CI report: * e0c4115590cdbea89d30285c22f30208cf3895ca UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #13551: [FLINK-19520][configuration] Add randomization of checkpoint config.
AHeise commented on a change in pull request #13551: URL: https://github.com/apache/flink/pull/13551#discussion_r557270935 ## File path: flink-runtime/pom.xml ## @@ -509,6 +509,16 @@ under the License. true +
[jira] [Commented] (FLINK-20927) Update configuration option in YarnConfigOptions class
[ https://issues.apache.org/jira/browse/FLINK-20927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264732#comment-17264732 ] Ruguo Yu commented on FLINK-20927: -- Ok [~trohrmann] , could you assign it to me? THX! > Update configuration option in YarnConfigOptions class > -- > > Key: FLINK-20927 > URL: https://issues.apache.org/jira/browse/FLINK-20927 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.12.0 >Reporter: Ruguo Yu >Priority: Minor > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-01-11-18-36-20-723.png, > image-2021-01-11-18-36-44-811.png > > > There are many configuration options that use a not recommended method to > build in _YarnConfigOptions_ class, mainly include the following: > 1. Use the deprecated method 'defaultValue' directly instead of specifying > the data type. > !image-2021-01-11-18-36-20-723.png|width=842,height=90! > 2. Use String instead of Enum types. > !image-2021-01-11-18-36-44-811.png|width=1128,height=139! > > Therefore, for the above problem, we can update it with the new API, which > can also be consistent with KubernetesConfigOptions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] chaozwn opened a new pull request #14642: [FLINK-20812][doc] HBase properties 'properties.*' function Description
chaozwn opened a new pull request #14642: URL: https://github.com/apache/flink/pull/14642 ## What is the purpose of the change add change doc. ## Brief change log ## Verifying this change ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? no This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14642: [FLINK-20812][doc] HBase properties 'properties.*' function Description
flinkbot commented on pull request #14642: URL: https://github.com/apache/flink/pull/14642#issuecomment-760092328 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 5ef442a3e3f007dbaa364b533d280343ea6a88be (Thu Jan 14 10:03:33 UTC 2021) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] chaozwn closed pull request #14642: [FLINK-20812][doc] HBase properties 'properties.*' function Description
chaozwn closed pull request #14642: URL: https://github.com/apache/flink/pull/14642 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #13551: [FLINK-19520][configuration] Add randomization of checkpoint config.
zentol commented on a change in pull request #13551:
URL: https://github.com/apache/flink/pull/13551#discussion_r557279916
##
File path:
flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/streaming/util/TestStreamEnvironment.java
##
@@ -69,13 +74,28 @@ public static void setAsContext(
TestStreamEnvironment env =
new TestStreamEnvironment(
miniCluster, parallelism, jarFiles,
classpaths);
+randomize(conf);
env.configure(conf, env.getUserClassloader());
return env;
};
initializeContextEnvironment(factory);
}
+/**
+ * Randomizes configuration on test case level even if mini cluster is
used in a class rule.
+ *
+ * @param conf
+ */
+private static void randomize(Configuration conf) {
+if (Randomization) {
+final String testName = TestNameProvider.getCurrentTestName();
Review comment:
yeah I'd revisit it once junit5 is in.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-20969) Update Scala 2.12 version to 2.12.13
[ https://issues.apache.org/jira/browse/FLINK-20969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-20969. Resolution: Duplicate > Update Scala 2.12 version to 2.12.13 > > > Key: FLINK-20969 > URL: https://issues.apache.org/jira/browse/FLINK-20969 > Project: Flink > Issue Type: Task > Components: API / Scala >Reporter: Erwan Loisant >Priority: Major > Labels: pull-request-available > > Flink is compiled with Scala 2.12.7, that was released in 2018. > The latest version of the 2.12 series is 2.12.13, and upgrading will bring > build performance improvements as well as improving compatibility for > projects using more recent versions of Scala. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14614: [FLINK-20911][state-backend-rocksdb] Support configuration of RocksDB log level
flinkbot edited a comment on pull request #14614: URL: https://github.com/apache/flink/pull/14614#issuecomment-758512892 ## CI report: * f3a5b0e4feb3bf16edecba4946fc0d160d7a4bce Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12030) * 926ffcf5f7b81e189e72339c7579fd73dd6715e9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14616: [FLINK-20879][table-api] Use MemorySize type instead of String type for memory ConfigOption in ExecutionConfigOptions
flinkbot edited a comment on pull request #14616: URL: https://github.com/apache/flink/pull/14616#issuecomment-758526377 ## CI report: * 448c026a402e045e050f405daf934a8a7c880c9d UNKNOWN * f429967b7680aa4dce4212ddb01b2c107b3b7117 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12037) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14641: [FLINK-20751][python][table-planner-blink] Port batch python group aggregate nodes to Java
flinkbot edited a comment on pull request #14641: URL: https://github.com/apache/flink/pull/14641#issuecomment-760086252 ## CI report: * e0c4115590cdbea89d30285c22f30208cf3895ca Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12042) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #14630: [FLINK-20915][docker] Move docker entrypoint to distribution
zentol commented on a change in pull request #14630:
URL: https://github.com/apache/flink/pull/14630#discussion_r557283929
##
File path: flink-dist/src/main/flink-bin/bin/docker-entrypoint.sh
##
@@ -0,0 +1,176 @@
+#!/usr/bin/env bash
+
+###
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+###
+
+###
+# This script is called by the Flink docker images when starting a process.
+# It contains docker-specific features, and hard-codes a few options.
+#
+# Globals:
+# FLINK_HOME - (required) path to the Flink home
directory
+# ENABLE_BUILT_IN_PLUGINS - semi-colon (;) separated list of
plugins to enable, e.g., "flink-plugin1.jar;flink-plugin2.jar"
+# FLINK_PROPERTIES- additional flink-conf.yaml entries
as a multi-line string
+# JOB_MANAGER_RPC_ADDRESS - RPC address of the job manager
+# TASK_MANAGER_NUMBER_OF_TASK_SLOTS - number of slots for task executors
+###
+
+COMMAND_STANDALONE="standalone-job"
+# Deprecated, should be remove in Flink release 1.13
+COMMAND_NATIVE_KUBERNETES="native-k8s"
+COMMAND_HISTORY_SERVER="history-server"
+
+args=("$@")
+echo "${args[@]}"
Review comment:
my bad, this was for debugging.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #14630: [FLINK-20915][docker] Move docker entrypoint to distribution
zentol commented on pull request #14630: URL: https://github.com/apache/flink/pull/14630#issuecomment-760098909 @wangyang0918 I have outlined this approach and already gotten their approval. https://github.com/docker-library/official-images/pull/9249#issuecomment-756902869 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol edited a comment on pull request #14630: [FLINK-20915][docker] Move docker entrypoint to distribution
zentol edited a comment on pull request #14630: URL: https://github.com/apache/flink/pull/14630#issuecomment-760098909 @wangyang0918 I have outlined this approach and already gotten their approval. https://github.com/docker-library/official-images/pull/9249#issuecomment-756902869 (note that I had the same hunch as you though!) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
huajiewang created FLINK-20972: -- Summary: TwoPhaseCommitSinkFunction Output a large amount of EventData Key: FLINK-20972 URL: https://issues.apache.org/jira/browse/FLINK-20972 Project: Flink Issue Type: Improvement Components: API / DataStream Affects Versions: 1.12.0 Environment: flink 1.4.0 + Reporter: huajiewang when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete, Maybe A large number of EventData will be output (log.info) ,which will cause IO bottleneck and disk waste -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zentol edited a comment on pull request #14630: [FLINK-20915][docker] Move docker entrypoint to distribution
zentol edited a comment on pull request #14630: URL: https://github.com/apache/flink/pull/14630#issuecomment-760098909 @wangyang0918 I have outlined this approach and already gotten their approval. https://github.com/docker-library/official-images/pull/9249#issuecomment-756902869 (note that I had the same hunch as you though, and was actually surprised that they were fine with it) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16267) Flink uses more memory than taskmanager.memory.process.size in Kubernetes
[
https://issues.apache.org/jira/browse/FLINK-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264740#comment-17264740
]
Yordan Pavlov commented on FLINK-16267:
---
[~azagrebin] Do we consider the problem with RocksDB described above as
non-Flink, if not should we re-open this?
> Flink uses more memory than taskmanager.memory.process.size in Kubernetes
> -
>
> Key: FLINK-16267
> URL: https://issues.apache.org/jira/browse/FLINK-16267
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.0
>Reporter: ChangZhuo Chen (陳昌倬)
>Priority: Major
> Attachments: flink-conf_1.10.0.yaml, flink-conf_1.9.1.yaml,
> oomkilled_taskmanager.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This issue is from
> [https://stackoverflow.com/questions/60336764/flink-uses-more-memory-than-taskmanager-memory-process-size-in-kubernetes]
> h1. Description
> * In Flink 1.10.0, we try to use `taskmanager.memory.process.size` to limit
> the resource used by taskmanager to ensure they are not killed by Kubernetes.
> However, we still get lots of taskmanager `OOMKilled`. The setup is in the
> following section.
> * The taskmanager log is in attachment [^oomkilled_taskmanager.log].
> h2. Kubernete
> * The Kubernetes setup is the same as described in
> [https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/kubernetes.html].
> * The following is resource configuration for taskmanager deployment in
> Kubernetes:
> {{resources:}}
> {{ requests:}}
> {{ cpu: 1000m}}
> {{ memory: 4096Mi}}
> {{ limits:}}
> {{ cpu: 1000m}}
> {{ memory: 4096Mi}}
> h2. Flink Docker
> * The Flink docker is built by the following Docker file.
> {{FROM flink:1.10-scala_2.11}}
> RUN mkdir -p /opt/flink/plugins/s3 &&
> ln -s /opt/flink/opt/flink-s3-fs-presto-1.10.0.jar /opt/flink/plugins/s3/
> {{RUN ln -s /opt/flink/opt/flink-metrics-prometheus-1.10.0.jar
> /opt/flink/lib/}}
> h2. Flink Configuration
> * The following are all memory related configurations in `flink-conf.yaml`
> in 1.10.0:
> {{jobmanager.heap.size: 820m}}
> {{taskmanager.memory.jvm-metaspace.size: 128m}}
> {{taskmanager.memory.process.size: 4096m}}
> * We use RocksDB and we don't set `state.backend.rocksdb.memory.managed` in
> `flink-conf.yaml`.
> ** Use S3 as checkpoint storage.
> * The code uses DateStream API
> ** input/output are both Kafka.
> h2. Project Dependencies
> * The following is our dependencies.
> {{val flinkVersion = "1.10.0"}}{{libraryDependencies +=
> "com.squareup.okhttp3" % "okhttp" % "4.2.2"}}
> {{libraryDependencies += "com.typesafe" % "config" % "1.4.0"}}
> {{libraryDependencies += "joda-time" % "joda-time" % "2.10.5"}}
> {{libraryDependencies += "org.apache.flink" %% "flink-connector-kafka" %
> flinkVersion}}
> {{libraryDependencies += "org.apache.flink" % "flink-metrics-dropwizard" %
> flinkVersion}}
> {{libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion
> % "provided"}}
> {{libraryDependencies += "org.apache.flink" %% "flink-statebackend-rocksdb"
> % flinkVersion % "provided"}}
> {{libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" %
> flinkVersion % "provided"}}
> {{libraryDependencies += "org.json4s" %% "json4s-jackson" % "3.6.7"}}
> {{libraryDependencies += "org.log4s" %% "log4s" % "1.8.2"}}
> {{libraryDependencies += "org.rogach" %% "scallop" % "3.3.1"}}
> h2. Previous Flink 1.9.1 Configuration
> * The configuration we used in Flink 1.9.1 are the following. It does not
> have `OOMKilled`.
> h3. Kubernetes
> {{resources:}}
> {{ requests:}}
> {{ cpu: 1200m}}
> {{ memory: 2G}}
> {{ limits:}}
> {{ cpu: 1500m}}
> {{ memory: 2G}}
> h3. Flink 1.9.1
> {{jobmanager.heap.size: 820m}}
> {{taskmanager.heap.size: 1024m}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20969) Update Scala 2.12 version to 2.12.13
[ https://issues.apache.org/jira/browse/FLINK-20969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264743#comment-17264743 ] Aljoscha Krettek commented on FLINK-20969: -- [~eloisant] Please continue on FLINK-12461. This issue is a bit more complicated, sorry for that. 😅 > Update Scala 2.12 version to 2.12.13 > > > Key: FLINK-20969 > URL: https://issues.apache.org/jira/browse/FLINK-20969 > Project: Flink > Issue Type: Task > Components: API / Scala >Reporter: Erwan Loisant >Priority: Major > Labels: pull-request-available > > Flink is compiled with Scala 2.12.7, that was released in 2018. > The latest version of the 2.12 series is 2.12.13, and upgrading will bring > build performance improvements as well as improving compatibility for > projects using more recent versions of Scala. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13247) Implement external shuffle service for YARN
[ https://issues.apache.org/jira/browse/FLINK-13247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264747#comment-17264747 ] Marek Simunek commented on FLINK-13247: --- Hi, is there some raw plan when it will be finished? [~ssy] > Implement external shuffle service for YARN > --- > > Key: FLINK-13247 > URL: https://issues.apache.org/jira/browse/FLINK-13247 > Project: Flink > Issue Type: New Feature > Components: Runtime / Network >Reporter: MalcolmSanders >Assignee: MalcolmSanders >Priority: Minor > > Flink batch job users could achieve better cluster utilization and job > throughput throught external shuffle service because the producers of > intermedia result partitions can be released once intermedia result > partitions have been persisted on disks. In > [FLINK-10653|https://issues.apache.org/jira/browse/FLINK-10653], [~zjwang] > has introduced pluggable shuffle manager architecture which abstracts the > process of data transfer between stages from flink runtime as shuffle > service. I propose to YARN implementation for flink external shuffle service > since YARN is widely used in various companies. > The basic idea is as follows: > (1) Producers write intermedia result partitions to local disks assigned by > NodeManager; > (2) Yarn shuffle servers, deployed on each NodeManager as an auxiliary > service, are acknowledged of intermedia result partition descriptions by > producers; > (3) Consumers fetch intermedia result partition from yarn shuffle servers; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14614: [FLINK-20911][state-backend-rocksdb] Support configuration of RocksDB log level
flinkbot edited a comment on pull request #14614: URL: https://github.com/apache/flink/pull/14614#issuecomment-758512892 ## CI report: * f3a5b0e4feb3bf16edecba4946fc0d160d7a4bce Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12030) * 926ffcf5f7b81e189e72339c7579fd73dd6715e9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12043) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14630: [FLINK-20915][docker] Move docker entrypoint to distribution
flinkbot edited a comment on pull request #14630: URL: https://github.com/apache/flink/pull/14630#issuecomment-759372895 ## CI report: * 656628f8e564ef8ee29032d392f1485a8b0d9eea UNKNOWN * b3370da98fb5903795c396172842e33c2bbf7575 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12009) * a19d8569e3badcb3c1877a9df03ab388edacc0c2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] aljoscha opened a new pull request #14643: [FLINK-20491] Support Broadcast Operation in BATCH execution mode (backport)
aljoscha opened a new pull request #14643: URL: https://github.com/apache/flink/pull/14643 This is a backport of #14312 for Flink 1.12. The changes applied cleanly. Please see the ML discussion, where there was consensus for this backport: https://lists.apache.org/thread.html/rb0de666ca8d1c34ab3a042fc265d89765a68e385c521522a63130200%40%3Cdev.flink.apache.org%3E This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16444) Count the read/write/seek/next latency of RocksDB as metrics
[ https://issues.apache.org/jira/browse/FLINK-16444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264751#comment-17264751 ] fanrui commented on FLINK-16444: +1, some latency metrics are very useful. Use flink-benckmark to make a benchmark. When RocksDB Statistics is enabled (StatsLevel.ALL), the update and value performance of ValueState is reduced by more than 10%. Is it better to do some sampling on the Flink side? For example: every 100 calls to the `ValueState.update` method, latency is collected once. > Count the read/write/seek/next latency of RocksDB as metrics > > > Key: FLINK-16444 > URL: https://issues.apache.org/jira/browse/FLINK-16444 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > Fix For: 1.13.0 > > > Currently, user cannot know the read/write/seek/next latency of RocksDB, we > could add these helpful metrics to know the overall state performance. To not > affect the action performance much, we could introduce counter to only record > the latency at interval of some actions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[ https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huajiewang updated FLINK-20972: --- Attachment: Jdbc2PCSinkFunction.scala > TwoPhaseCommitSinkFunction Output a large amount of EventData > - > > Key: FLINK-20972 > URL: https://issues.apache.org/jira/browse/FLINK-20972 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.12.0 > Environment: flink 1.4.0 + >Reporter: huajiewang >Priority: Minor > Labels: easyfix > Attachments: Jdbc2PCSinkFunction.scala > > Original Estimate: 1h > Remaining Estimate: 1h > > when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete, > Maybe A large number of EventData will be output (log.info) > ,which will cause IO bottleneck and disk waste > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20927) Update configuration option in YarnConfigOptions class
[ https://issues.apache.org/jira/browse/FLINK-20927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264754#comment-17264754 ] Till Rohrmann commented on FLINK-20927: --- I've assigned it to you. > Update configuration option in YarnConfigOptions class > -- > > Key: FLINK-20927 > URL: https://issues.apache.org/jira/browse/FLINK-20927 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.12.0 >Reporter: Ruguo Yu >Assignee: Ruguo Yu >Priority: Minor > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-01-11-18-36-20-723.png, > image-2021-01-11-18-36-44-811.png > > > There are many configuration options that use a not recommended method to > build in _YarnConfigOptions_ class, mainly include the following: > 1. Use the deprecated method 'defaultValue' directly instead of specifying > the data type. > !image-2021-01-11-18-36-20-723.png|width=842,height=90! > 2. Use String instead of Enum types. > !image-2021-01-11-18-36-44-811.png|width=1128,height=139! > > Therefore, for the above problem, we can update it with the new API, which > can also be consistent with KubernetesConfigOptions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huajiewang updated FLINK-20972:
---
Description:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
Maybe A large number of EventData will be output (log.info)
,which will cause IO bottleneck and disk waste
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolder{handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4
was:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
Maybe A large number of EventData will be output (log.info)
,which will cause IO bottleneck and disk waste
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix
> Attachments: Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
> Maybe A large number of EventData will be output (log.info)
> ,which will cause IO bottleneck and disk waste
>
>
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolder{handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-20927) Update configuration option in YarnConfigOptions class
[ https://issues.apache.org/jira/browse/FLINK-20927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann reassigned FLINK-20927: - Assignee: Ruguo Yu > Update configuration option in YarnConfigOptions class > -- > > Key: FLINK-20927 > URL: https://issues.apache.org/jira/browse/FLINK-20927 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.12.0 >Reporter: Ruguo Yu >Assignee: Ruguo Yu >Priority: Minor > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-01-11-18-36-20-723.png, > image-2021-01-11-18-36-44-811.png > > > There are many configuration options that use a not recommended method to > build in _YarnConfigOptions_ class, mainly include the following: > 1. Use the deprecated method 'defaultValue' directly instead of specifying > the data type. > !image-2021-01-11-18-36-20-723.png|width=842,height=90! > 2. Use String instead of Enum types. > !image-2021-01-11-18-36-44-811.png|width=1128,height=139! > > Therefore, for the above problem, we can update it with the new API, which > can also be consistent with KubernetesConfigOptions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13247) Implement external shuffle service for YARN
[ https://issues.apache.org/jira/browse/FLINK-13247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264757#comment-17264757 ] Till Rohrmann commented on FLINK-13247: --- At the moment I am not aware that the community is actively working on it [~marek.simunek]. But I am pulling in [~maguowei] who should know better than me. > Implement external shuffle service for YARN > --- > > Key: FLINK-13247 > URL: https://issues.apache.org/jira/browse/FLINK-13247 > Project: Flink > Issue Type: New Feature > Components: Runtime / Network >Reporter: MalcolmSanders >Assignee: MalcolmSanders >Priority: Minor > > Flink batch job users could achieve better cluster utilization and job > throughput throught external shuffle service because the producers of > intermedia result partitions can be released once intermedia result > partitions have been persisted on disks. In > [FLINK-10653|https://issues.apache.org/jira/browse/FLINK-10653], [~zjwang] > has introduced pluggable shuffle manager architecture which abstracts the > process of data transfer between stages from flink runtime as shuffle > service. I propose to YARN implementation for flink external shuffle service > since YARN is widely used in various companies. > The basic idea is as follows: > (1) Producers write intermedia result partitions to local disks assigned by > NodeManager; > (2) Yarn shuffle servers, deployed on each NodeManager as an auxiliary > service, are acknowledged of intermedia result partition descriptions by > producers; > (3) Consumers fetch intermedia result partition from yarn shuffle servers; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14643: [FLINK-20491] Support Broadcast Operation in BATCH execution mode (backport)
flinkbot commented on pull request #14643: URL: https://github.com/apache/flink/pull/14643#issuecomment-760109092 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit db5d6010fd2f61610e2baa74e32b6bfbc3834782 (Thu Jan 14 10:34:15 UTC 2021) **Warnings:** * Documentation files were touched, but no `.zh.md` files: Update Chinese documentation or file Jira ticket. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20974) docker容器pyflink中stream任务取消之后会留下几个僵尸进程
he jie created FLINK-20974: -- Summary: docker容器pyflink中stream任务取消之后会留下几个僵尸进程 Key: FLINK-20974 URL: https://issues.apache.org/jira/browse/FLINK-20974 Project: Flink Issue Type: Bug Components: API / Python Affects Versions: 1.12.0 Reporter: he jie pyflink在stream流任务取消之后会留下几个 python，bash ，tee的僵尸进程 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20973) failed to submit flink-on-yarn job on Windows due to error of classPath
kunnni created FLINK-20973:
--
Summary: failed to submit flink-on-yarn job on Windows due to
error of classPath
Key: FLINK-20973
URL: https://issues.apache.org/jira/browse/FLINK-20973
Project: Flink
Issue Type: Bug
Components: Deployment / YARN
Affects Versions: 1.11.3, 1.10.1
Reporter: kunnni
When I submitted flink-on-yarn job on Windows (Operating system of yarn cluster
is Linux), I got error message: “Error: Could not find or load main class
org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint”.
then I found out that Windows has different classPath delimiter with Linux as
in Windows is ";", but in Linux is ":". the main cause is that classPath was
built on Windows, but was splitted on Linux.
the solution is to replace value of File.pathSeparator with
ApplicationConstants.CLASS_PATH_SEPARATOR in class of
org.apache.flink.yarn.YarnClusterDescriptor and org.apache.flink.yarn.Utils.
sourceCode is displayed below：
{code:java}
private ApplicationReport startAppMaster(..) throws Exception {
..
// classpath assembler
StringBuilder classPathBuilder = new StringBuilder();
if (userJarInclusion == YarnConfigOptions.UserJarInclusion.FIRST) {
for (String userClassPath : userClassPaths)
{ classPathBuilder.append(userClassPath).append(File.pathSeparator); }
}
for (String classPath : systemClassPaths)
{ classPathBuilder.append(classPath).append(File.pathSeparator); }
..
// Setup CLASSPATH and environment variables for ApplicationMaster
final Map appMasterEnv = new HashMap<>();
// set user specified app master environment variables
appMasterEnv.putAll(
BootstrapTools.getEnvironmentVariables(ResourceManagerOptions.CONTAINERIZED_MASTER_ENV_PREFIX,
configuration));
// set Flink app class path
appMasterEnv.put(YarnConfigKeys.ENV_FLINK_CLASSPATH,
classPathBuilder.toString());
..
}{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264765#comment-17264765
]
Yun Gao commented on FLINK-20972:
-
Hi [~benjobs], which log is outputted a lot~? Do you refer to
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint {}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId);
This one ?
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix
> Attachments: Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
> Maybe A large number of EventData will be output (log.info)
> ,which will cause IO bottleneck and disk waste
>
>
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolder{handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] leonardBang opened a new pull request #14644: [FLINK-20840][table-planner] Properly transpose projection to join
leonardBang opened a new pull request #14644: URL: https://github.com/apache/flink/pull/14644 ## What is the purpose of the change * Currently the rule `FlinkProjectJoinTransposeRule` was added in logical optimize phase which uses volcano optimizer, the optimized plan is uncertain because it uses CBO model. ## Brief change log - Add a `PROJECT_REWRITE` phase in hep optimizer which uses RBO model. ## Verifying this change * The tests in TemporalJoinTest/LookupJoinTest cover this change ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): ( no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] leonardBang commented on pull request #14644: [FLINK-20840][table-planner] Properly transpose projection to join
leonardBang commented on pull request #14644: URL: https://github.com/apache/flink/pull/14644#issuecomment-760112428 cc： @godfreyhe could you help review this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20974) docker容器pyflink中stream任务取消之后会留下几个僵尸进程
[ https://issues.apache.org/jira/browse/FLINK-20974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz closed FLINK-20974. Resolution: Invalid Please use English in JIRA. > docker容器pyflink中stream任务取消之后会留下几个僵尸进程 > - > > Key: FLINK-20974 > URL: https://issues.apache.org/jira/browse/FLINK-20974 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.12.0 >Reporter: he jie >Priority: Major > > pyflink在stream流任务取消之后会留下几个 python，bash ，tee的僵尸进程 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20840) Projection pushdown doesn't work in temporal(lookup) join
[
https://issues.apache.org/jira/browse/FLINK-20840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-20840:
---
Labels: pull-request-available (was: )
> Projection pushdown doesn't work in temporal(lookup) join
> --
>
> Key: FLINK-20840
> URL: https://issues.apache.org/jira/browse/FLINK-20840
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.12.0
>Reporter: Leonard Xu
>Assignee: Leonard Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> {code:java}
> sql 1:
> |SELECT T.*, D.id
> |FROM MyTable AS T
> |JOIN LookupTable FOR SYSTEM_TIME AS OF T.proctime AS D
> |ON T.a = D.id
> optmized plan:
> Calc(select=[a, b, c, PROCTIME_MATERIALIZE(proctime) AS proctime, rowtime,
> id])
> +- LookupJoin(table=[default_catalog.default_database.LookupTable],
> joinType=[InnerJoin], async=[false], lookup=[id=a], select=[a, b, c,
> proctime, rowtime, id, name, age])
>+- DataStreamScan(table=[[default_catalog, default_database, MyTable]],
> fields=[a, b, c, proctime, rowtime])
> sql 2:
> |SELECT T.a, D.id
> |FROM MyTable AS T
> |JOIN LookupTable FOR SYSTEM_TIME AS OF T.proctime AS D
> |ON T.a = D.id
> optmized plan:
> LookupJoin(table=[default_catalog.default_database.LookupTable],
> joinType=[InnerJoin], async=[false], lookup=[id=a], select=[a, id])
> +- Calc(select=[a])
>+- DataStreamScan(table=[[default_catalog, default_database, MyTable]],
> fields=[a, b, c, proctime, rowtime])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20975) HiveTableSourceITCase.testPartitionFilter fails on AZP
Till Rohrmann created FLINK-20975:
-
Summary: HiveTableSourceITCase.testPartitionFilter fails on AZP
Key: FLINK-20975
URL: https://issues.apache.org/jira/browse/FLINK-20975
Project: Flink
Issue Type: Bug
Components: Connectors / Hive
Affects Versions: 1.13.0
Reporter: Till Rohrmann
Fix For: 1.13.0
The test {{HiveTableSourceITCase.testPartitionFilter}} fails on AZP with the
following exception:
{code}
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.junit.Assert.assertFalse(Assert.java:74)
at
org.apache.flink.connectors.hive.HiveTableSourceITCase.testPartitionFilter(HiveTableSourceITCase.java:278)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tillrohrmann commented on pull request #14593: [FLINK-20903] Remove unnecessary methods from SchedulerNG
tillrohrmann commented on pull request #14593: URL: https://github.com/apache/flink/pull/14593#issuecomment-760112835 Thanks for the review @XComp. The failing test case seems to be unrelated. Merging it now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann closed pull request #14593: [FLINK-20903] Remove unnecessary methods from SchedulerNG
tillrohrmann closed pull request #14593: URL: https://github.com/apache/flink/pull/14593 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20903) Remove SchedulerNG.initialize method
[
https://issues.apache.org/jira/browse/FLINK-20903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-20903.
-
Resolution: Fixed
Fixed via
7a001c6c7a813c8af55ac798547c5036446df71d
b580b7def435deac11cba8ff04c3b718cb568c48
a4beb1d1ec6a0bf74ac3ec5a79db349ae4a8267a
0464532030736727fa5cd0910b0d4b6b004e4904
> Remove SchedulerNG.initialize method
>
>
> Key: FLINK-20903
> URL: https://issues.apache.org/jira/browse/FLINK-20903
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.13.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> The {{SchedulerNG}} no longer needs the {{initialize}} method because the
> {{JobMaster}} is now started with a valid {{MainThreadExecutor}}. Hence, we
> can simplify the {{SchedulerNG}} by passing the
> {{ComponentMainThreadExecutor}} to the constructor of the implementations.
> Similarly, we can remove the {{SchedulerNG.registerJobStatusListener}} method.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
