[jira] [Created] (FLINK-22448) FlinkRexUtil create Sarg String array elemet supplement space
Junning Liang created FLINK-22448:
-
Summary: FlinkRexUtil create Sarg String array elemet supplement
space
Key: FLINK-22448
URL: https://issues.apache.org/jira/browse/FLINK-22448
Project: Flink
Issue Type: Bug
Reporter: Junning Liang
Fix For: 1.13.0
As we know, the new version of Calcite introduces the {{SEARCH}} rex call to
express range conditions. But when i used string array to express range in the
StreamSQL, i found that some string in the array had problems with the
completion length by using space.
the following query:
{code:java}
create view tempView as
select * from sourceTable where action in ('systemnotifyv2', 'session_auth',
'create_session', 'close_single_chat'){code}
after Sarg operator created, the result is :
{code:java}
create view tempView as
select * from sourceTable where action in ('systemnotifyv2', 'session_auth ',
'create_session', 'close_single_chat')
{code}
I debuged to see why dose the happans. After calling rexBuilder.makeLiteral in
FlinkRexUtil#expandSearchOperands, the string 'session_auth' became
'session_auth '.And i also found that the type and length of the string array
were determined by the first string in the array.Just like my example above,
the type of the array was Char and the length of the array was 14.the length of
'session_auth' string was 12 so that calcite would supplement 2 space to make
it meet the length of 14.
Now, i All I can think of is adding trim parameter to remove the space。do you
have a better way to fix or avoid the problem happens?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22449) Casting an invalid constant string to int throws exception from SinkNotNullEnforcer
Caizhi Weng created FLINK-22449:
---
Summary: Casting an invalid constant string to int throws
exception from SinkNotNullEnforcer
Key: FLINK-22449
URL: https://issues.apache.org/jira/browse/FLINK-22449
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 1.13.0
Reporter: Caizhi Weng
Add the following test case to {{CalcITCase}} to reproduce this bug:
{code:scala}
@Test
def myTest(): Unit = {
checkResult("SELECT CAST('haha' AS INT)", Seq(row(null)))
}
{code}
The exception stack is
{code}
Caused by: org.apache.flink.table.api.TableException: Column 'EXPR$0' is NOT
NULL, however, a null value is being written into it. You can set job
configuration 'table.exec.sink.not-null-enforcer'='drop' to suppress this
exception and drop such records silently.
at
org.apache.flink.table.runtime.operators.sink.SinkNotNullEnforcer.filter(SinkNotNullEnforcer.java:56)
at
org.apache.flink.table.runtime.operators.sink.SinkNotNullEnforcer.filter(SinkNotNullEnforcer.java:30)
at
org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:38)
at
org.apache.flink.streaming.runtime.tasks.ChainingOutput.pushToOperator(ChainingOutput.java:112)
at
org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:93)
at
org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:39)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28)
at BatchExecCalc$33.processElement(Unknown Source)
at
org.apache.flink.streaming.runtime.tasks.ChainingOutput.pushToOperator(ChainingOutput.java:112)
at
org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:93)
at
org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:39)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28)
at
org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:317)
at
org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:411)
at
org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:92)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)
{code}
This is because the result type of CAST is inferred as NOT NULL (see
{{SqlCastFunction#inferReturnType}} and
{{StandardConvertletTable#convertCast}}, the nullability is the same with the
input argument), however parsing an invalid string to int will produce null
values.
One way I could think of is to change the result type of CAST to always
nullable (at least for some specific types of casting, for example casting from
string to int), but as CAST is a very low-level function this might have a big
impact (for example, if a rule adds casting, the resulting row type might not
be equal to the original row type due to mismatch in nullability).
So it seems that at the current stage we should set all columns in a select
sink to be nullable. However this indicates that one cannot truly rely on the
nullability of any result type.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22450) SQL parser fails to parse subquery containing INTERSECT in a view
Caizhi Weng created FLINK-22450:
---
Summary: SQL parser fails to parse subquery containing INTERSECT
in a view
Key: FLINK-22450
URL: https://issues.apache.org/jira/browse/FLINK-22450
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 1.13.0
Reporter: Caizhi Weng
Add the following test case to {{TableEnvironmentITCase}} to reproduce this bug.
{code:scala}
@Test
def myTest(): Unit = {
tEnv.executeSql(
"""
|CREATE TABLE T1 (
| a INT,
| b BIGINT
|) WITH (
| 'connector'='values'
|)
|""".stripMargin)
tEnv.executeSql(
"""
|CREATE TABLE T2 (
| c INT,
| d BIGINT
|) WITH (
| 'connector'='values'
|)
|""".stripMargin)
tEnv.executeSql(
"""
|CREATE TABLE T3 (
| c INT,
| d BIGINT
|) WITH (
| 'connector'='values'
|)
|""".stripMargin)
tEnv.executeSql("CREATE VIEW myView AS SELECT * FROM T1, (SELECT * FROM T2
WHERE c > 0 INTERSECT SELECT * FROM T3 WHERE c > 0) WHERE a = c")
System.out.println(tEnv.explainSql("SELECT * FROM myView"))
}
{code}
The exception stack is
{code}
org.apache.flink.table.api.SqlParserException: SQL parse failed. Encountered ",
SELECT" at line 2, column 10.
Was expecting one of:
"AS" ...
"EXCEPT" ...
"EXTEND" ...
"FETCH" ...
"FOR" ...
"GROUP" ...
"HAVING" ...
"INTERSECT" ...
"LIMIT" ...
"MATCH_RECOGNIZE" ...
"OFFSET" ...
"ORDER" ...
"PIVOT" ...
"MINUS" ...
"TABLESAMPLE" ...
"UNION" ...
"WHERE" ...
"WINDOW" ...
"(" ...
...
...
...
...
...
...
"/*+" ...
"NATURAL" ...
"JOIN" ...
"INNER" ...
"LEFT" ...
"RIGHT" ...
"FULL" ...
"CROSS" ...
"," ...
"," ...
"," ...
"," ...
"," ...
"," ...
"," "LATERAL" ...
"," "(" ...
"," "UNNEST" ...
"," "TABLE" ...
"OUTER" ...
"." ...
at
org.apache.flink.table.planner.parse.CalciteParser.parse(CalciteParser.java:56)
at
org.apache.flink.table.planner.utils.Expander.expanded(Expander.java:83)
at
org.apache.flink.table.planner.operations.SqlToOperationConverter.convertViewQuery(SqlToOperationConverter.java:849)
at
org.apache.flink.table.planner.operations.SqlToOperationConverter.convertCreateView(SqlToOperationConverter.java:819)
at
org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:248)
at
org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:99)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:723)
at
org.apache.flink.table.api.TableEnvironmentITCase.myTest(TableEnvironmentITCase.scala:116)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.
[jira] [Created] (FLINK-22451) Support (*) as parameter of table UserDefinedFunction
Yi Tang created FLINK-22451:
---
Summary: Support (*) as parameter of table UserDefinedFunction
Key: FLINK-22451
URL: https://issues.apache.org/jira/browse/FLINK-22451
Project: Flink
Issue Type: Improvement
Reporter: Yi Tang
For now, one can use star (*) to act as a wild card, selecting all of the
columns in the table.
{code}
Table result = orders.select($("*"));
{code}
When one use a star (*) as parameter of an UDF, it will fail
{{ReferenceResolverRule}} in on
{code}
"Cannot resolve field [*], input field list:[...]."
{code}
The cause is that, the parameter of an UDF is not expanded in
{{StarReferenceFlatteningRule}}
I think we can support to expand the star parameter to the real fields list if
it is the only parameter of the UDF.
then the parameters can be received by
{code}
eval(@DataTypeHint(inputGroup = InputGroup.ANY) Object... row)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22452) Support specifying custom transactional.id prefix in FlinkKafkaProducer
Wenhao Ji created FLINK-22452: - Summary: Support specifying custom transactional.id prefix in FlinkKafkaProducer Key: FLINK-22452 URL: https://issues.apache.org/jira/browse/FLINK-22452 Project: Flink Issue Type: Improvement Components: Connectors / Kafka Affects Versions: 1.12.2 Reporter: Wenhao Ji Currently, the "transactional.id"s of the Kafka producers in FlinkKafkaProducer are generated based on the task name. This mechanism has some limitations: * It will exceed Kafka's limitation if the task name is too long. (resolved in FLINK-17691) * They will very likely clash each other if the job topologies are similar. (discussed in FLINK-11654) * Only certain "transactional.id" may be authorized by [Prefixed ACLs|https://docs.confluent.io/platform/current/kafka/authorization.html#prefixed-acls] on the target Kafka cluster. Besides, the spring community has introduced the [setTransactionIdPrefix|https://docs.spring.io/spring-kafka/api/org/springframework/kafka/core/DefaultKafkaProducerFactory.html#setTransactionIdPrefix(java.lang.String)] method to their Kafka client. Therefore, I think it will be necessary to have this feature in the Flink Kafka connector. As discussed in FLINK-11654, the possible solution will be, * either introduce an additional method called setTransactionalIdPrefix(String) in the FlinkKafkaProducer, * or use the existing "transactional.id" properties as the prefix. And the behavior of the "transactional.id" generation will be * keep the behavior as it was if absent, * use the one if present as the prefix for the TransactionalIdsGenerator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22453) Can not stop job when not ust "-m" option
Liu created FLINK-22453:
---
Summary: Can not stop job when not ust "-m" option
Key: FLINK-22453
URL: https://issues.apache.org/jira/browse/FLINK-22453
Project: Flink
Issue Type: Bug
Components: API / Core
Affects Versions: 1.12.2
Reporter: Liu
flink version: 1.12.2
yarn version : 3.1.1 (hdp 3.1.5)
h3. Starting a Flink Session on YARN
when i use ' flink stop xxx', comond line output:
{code:java}
//代码占位符
Setting HBASE_CONF_DIR=/etc/hbase/conf because no HBASE_CONF_DIR was set.SLF4J:
Class path contains multiple SLF4J bindings.SLF4J: Found binding in
[jar:file:/data/flink-1.12.2/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
Found binding in
[jar:file:/usr/hdp/3.1.5.0-152/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]2021-04-25 16:10:43,369 INFO
org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn
properties file under /tmp/.yarn-properties-flink.2021-04-25 16:10:43,369 INFO
org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn
properties file under /tmp/.yarn-properties-flink.Suspending job
"a81c0fe295871ef278a119cd44206216" with a savepoint.2021-04-25 16:10:45,126
INFO org.apache.hadoop.yarn.client.AHSProxy [] -
Connecting to Application History server at
adt-bd-c1-nn03.internal/172.20.33.149:102002021-04-25 16:10:45,174 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - No path for
the flink jar passed. Using the location of class
org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2021-04-25
16:10:45,520 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - Found Web Interface adt-bd-c1-flink06.internal:43379 of application
'application_1618023905026_0005'.
The program
finished with the following exception:
org.apache.flink.util.FlinkException: Could not stop with a savepoint job
"a81c0fe295871ef278a119cd44206216". at
org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:581) at
org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1002)
at org.apache.flink.client.cli.CliFrontend.stop(CliFrontend.java:569) at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1069) at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
at java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)Caused
by: java.util.concurrent.ExecutionException:
java.util.concurrent.TimeoutException at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at
org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:579) ...
9 moreCaused by: java.util.concurrent.TimeoutException at
org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1220)
at
org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
at
org.apache.flink.runtime.concurrent.FutureUtils.lambda$orTimeout$15(FutureUtils.java:582)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
flink can't stop the job,
but when i user 'flink stop -m jobmanager.server.host:port x' , it
work well.
'-m' is an option args, The old version does not have this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22454) Ignore CAST condition in temporal join
Shengkai Fang created FLINK-22454:
-
Summary: Ignore CAST condition in temporal join
Key: FLINK-22454
URL: https://issues.apache.org/jira/browse/FLINK-22454
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Affects Versions: 1.13.0
Reporter: Shengkai Fang
Please add test in {{LookupJoinTest}}
{code:java}
def before(): Unit ={
util.addDataStream[(Int, String, Long)](
"MyTable", 'a, 'b, 'c, 'proctime.proctime, 'rowtime.rowtime)
if (legacyTableSource) {
TestTemporalTable.createTemporaryTable(util.tableEnv, "LookupTable")
} else {
util.addTable(
"""
|CREATE TABLE LookupTable (
| `id` DECIMAL(38, 10),
| `to_qty` DECIMAL(38, 10),
| `name` STRING,
| `age` INT,
| `id_int` as CAST(`id` AS INT)
|) WITH (
| 'connector' = 'values'
|)
|""".stripMargin)
}
{code}
{code:java}
@Test
def test(): Unit = {
val sql =
"""
|SELECT MyTable.b, LookupTable.`to_qty`
|FROM MyTable
|LEFT JOIN LookupTable FOR SYSTEM_TIME AS OF MyTable.`proctime`
|ON MyTable.a = CAST(LookupTable.`id` as INT)
|""".stripMargin
util.tableEnv.sqlQuery(sql).explain()
}
{code}
The exception stack is
{code}
org.apache.flink.table.api.TableException: Temporal table join requires
equivalent condition of the same type, but the condition is
a[INT]=id[DECIMAL(38, 10)]
at
org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecLookupJoin.validateLookupKeyType(CommonExecLookupJoin.java:303)
at
org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecLookupJoin.translateToPlanInternal(CommonExecLookupJoin.java:222)
at
org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:134)
at
org.apache.flink.table.planner.plan.nodes.exec.ExecEdge.translateToPlan(ExecEdge.java:247)
at
org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecCalc.translateToPlanInternal(CommonExecCalc.java:88)
at
org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:134)
at
org.apache.flink.table.planner.delegation.StreamPlanner$$anonfun$translateToPlan$1.apply(StreamPlanner.scala:70)
at
org.apache.flink.table.planner.delegation.StreamPlanner$$anonfun$translateToPlan$1.apply(StreamPlanner.scala:69)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
org.apache.flink.table.planner.delegation.StreamPlanner.translateToPlan(StreamPlanner.scala:69)
at
org.apache.flink.table.planner.delegation.StreamPlanner.explain(StreamPlanner.scala:104)
at
org.apache.flink.table.planner.delegation.StreamPlanner.explain(StreamPlanner.scala:46)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.explainInternal(TableEnvironmentImpl.java:691)
at
org.apache.flink.table.api.internal.TableImpl.explain(TableImpl.java:582)
at
org.apache.flink.table.planner.plan.stream.sql.join.LookupJoinTest.test(LookupJoinTest.scala:197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
[jira] [Created] (FLINK-22455) FlinkRelBuilder#windowAggregate will throw ClassCastException when function reuse
tartarus created FLINK-22455:
Summary: FlinkRelBuilder#windowAggregate will throw
ClassCastException when function reuse
Key: FLINK-22455
URL: https://issues.apache.org/jira/browse/FLINK-22455
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Reporter: tartarus
Attachments: FlinkRelBuilderTest.scala
If the input parameter aggCalls of FlinkRelBuilder#windowAggregate contains the
same aggregate function. Then it will throw ClassCastException, because of the
optimization of aggregate function reuse. We did not judge the return value
type, but direct type conversion;
{code:java}
val aggregate = super.transform(
new UnaryOperator[RelBuilder.Config] {
override def apply(t: RelBuilder.Config)
: RelBuilder.Config = t.withPruneInputOfAggregate(false)
})
.push(build())
.aggregate(groupKey, aggCalls)
.build()
.asInstanceOf[LogicalAggregate]
{code}
I wrote a test that triggered this problem.
You can use the attached code to reproduce this problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [VOTE] Release 1.13.0, release candidate #2
Hi Rui, Documentation is not a part of the release package. The changes to the documentation will take effect in 24 hours and can be accessed on the web [1] then. Docs can even be updated after 1.13 is released. So we don't need to cancel the RC for this. Best, Jark [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/hive/overview/ On Sun, 25 Apr 2021 at 13:47, Rui Li wrote: > Hi Dawid, > > I have just merged the doc change for hive dialect [1], which talks about > how to use hive dialect to run hive queries. Do you think we can have > another RC to include it? Sorry for the trouble. > > [1] https://issues.apache.org/jira/browse/FLINK-22119 > > On Sat, Apr 24, 2021 at 1:36 AM Dawid Wysakowicz > wrote: > > > Hi everyone, > > Please review and vote on the release candidate #2 for the version > 1.13.0, > > as follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > The complete staging area is available for your review, which includes: > > * JIRA release notes [1], > > * the official Apache source release and binary convenience releases to > be > > deployed to dist.apache.org [2], which are signed with the key with > > fingerprint 31D2DD10BFC15A2D [3], > > * all artifacts to be deployed to the Maven Central Repository [4], > > * source code tag "release-1.13.0-rc2" [5], > > * website pull request listing the new release and adding announcement > > blog post [6]. > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Thanks, > > Dawid Wysakowicz > > > > [1] > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12349287 > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.13.0-rc2/ > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > [4] > > https://repository.apache.org/content/repositories/orgapacheflink-1420/ > > [5] https://github.com/apache/flink/tree/release-1.13.0-rc2 > > [6] https://github.com/apache/flink-web/pull/436 > > > > > -- > Best regards! > Rui Li >
Re: [DISCUSS]FLIP-150: Introduce Hybrid Source
Hi Becket, I agree and am not planning to hard wire a specific combination of sources (like S3 + Kafka). That also wouldn't help for the use case I want to address, because there are customized connectors that we need to be able to plug in. Rather, the suggested simplification would be for the flexibility of switching mechanism. The prototype already supports fixed start positions and checkpoint conversion for any combination of sources; no need to undo that. But for testing/example purposes, we will need to settle on a specific combination. Thomas On Sat, Apr 24, 2021 at 8:20 PM Becket Qin wrote: > > Sorry for the late reply. Starting from a specific connector sounds > reasonable to me. > > That said, I would suggest to keep the possibility of future generalization > as much as possible. We have already seen some variation of source > combinations from different users, HDFS + Kafka, S3 + Kafka, S3 + SQL > Binlog, etc. So it would be good if we can reuse some base abstraction in > the future instead of having to write each combination from scratch. > > Thanks, > > Jiangjie (Becket) Qin > > On Sat, Apr 17, 2021 at 7:34 PM Stephan Ewen wrote: > > > Thanks, Thomas! > > > > @Becket and @Nicholas - would you be ok with that approach? > > > > > > On Thu, Apr 15, 2021 at 6:30 PM Thomas Weise wrote: > > > > > Hi Stephan, > > > > > > Thanks for the feedback! > > > > > > I agree with the approach of starting with a simple implementation > > > that can address a well understood, significant portion of use cases. > > > > > > I'm planning to continue work on the prototype that I had shared. > > > There is production level usage waiting for it fairly soon. I expect > > > to open a PR in the coming weeks. > > > > > > Thomas > > > > > > > > > > > > > > > > > > On Tue, Apr 13, 2021 at 12:15 PM Stephan Ewen wrote: > > > > > > > > Thanks all for this discussion. Looks like there are lots of ideas and > > > > folks that are eager to do things, so let's see how we can get this > > > moving. > > > > > > > > My take on this is the following: > > > > > > > > There will probably not be one Hybrid source, but possibly multiple > > ones, > > > > because of different strategies/requirements. > > > > - One may be very simple, with switching points known up-front. > > Would > > > > be good to have this in a very simple implementation. > > > > - There may be one where the switch is dynamic and the readers need > > > to > > > > report back where they left off. > > > > - There may be one that switches back and forth multiple times > > > during a > > > > job, for example Kakfa going to DFS whenever it falls behind retention, > > > in > > > > order to catch up again. > > > > > > > > This also seems hard to "design on paper"; I expect there are nuances > > in > > > a > > > > production setup that affect some details of the design. So I'd feel > > most > > > > comfortable in adding a variant of the hybrid source to Flink that has > > > been > > > > used already in a real use case (not necessarily in production, but > > maybe > > > > in a testing/staging environment, so it seems to meet all > > requirements). > > > > > > > > > > > > What do you think about the following approach? > > > > - If there is a tested PoC, let's try to get it contributed to Flink > > > > without trying to make it much more general. > > > > - When we see similar but a bit different requirements for another > > > hybrid > > > > source, then let's try to evolve the contributed one. > > > > - If we see new requirements that are so different that they don't > > fit > > > > well with the existing hybrid source, then let us look at building a > > > second > > > > hybrid source for those requirements. > > > > > > > > We need to make connector contributions in general more easy, and I > > think > > > > it is not a bad thing to end up with different approaches and see how > > > these > > > > play out against each other when being used by users. For example > > > switching > > > > with known boundaries, dynamic switching, back-and-forth-switching, > > etc. > > > > (I know some committers are planning to do some work on making > > > > connector contributions easier, with standardized testing frameworks, > > > > decoupled CI, etc.) > > > > > > > > Best, > > > > Stephan > > > > > > > > > > > > On Thu, Mar 25, 2021 at 4:41 AM Thomas Weise wrote: > > > > > > > > > Hi, > > > > > > > > > > As mentioned in my previous email, I had been working on a prototype > > > for > > > > > the hybrid source. > > > > > > > > > > You can find it at https://github.com/tweise/flink/pull/1 > > > > > > > > > > It contains: > > > > > * Switching with configurable chain of sources > > > > > * Fixed or dynamic start positions > > > > > * Test with MockSource and FileSource > > > > > > > > > > The purpose of the above PR is to gather feedback and help drive > > > consensus > > > > > on the FLIP. > > > > > > > > > > * How to support a dynamic start position within the source chain
Re: [DISCUSS]FLIP-150: Introduce Hybrid Source
Thanks for the clarification, Thomas. Yes, that makes sense to me. Cheers, Jiangjie (Becket) Qin On Mon, Apr 26, 2021 at 1:03 AM Thomas Weise wrote: > Hi Becket, > > I agree and am not planning to hard wire a specific combination of > sources (like S3 + Kafka). That also wouldn't help for the use case I > want to address, because there are customized connectors that we need > to be able to plug in. > > Rather, the suggested simplification would be for the flexibility of > switching mechanism. > > The prototype already supports fixed start positions and checkpoint > conversion for any combination of sources; no need to undo that. > > But for testing/example purposes, we will need to settle on a specific > combination. > > Thomas > > On Sat, Apr 24, 2021 at 8:20 PM Becket Qin wrote: > > > > Sorry for the late reply. Starting from a specific connector sounds > > reasonable to me. > > > > That said, I would suggest to keep the possibility of future > generalization > > as much as possible. We have already seen some variation of source > > combinations from different users, HDFS + Kafka, S3 + Kafka, S3 + SQL > > Binlog, etc. So it would be good if we can reuse some base abstraction in > > the future instead of having to write each combination from scratch. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > On Sat, Apr 17, 2021 at 7:34 PM Stephan Ewen wrote: > > > > > Thanks, Thomas! > > > > > > @Becket and @Nicholas - would you be ok with that approach? > > > > > > > > > On Thu, Apr 15, 2021 at 6:30 PM Thomas Weise wrote: > > > > > > > Hi Stephan, > > > > > > > > Thanks for the feedback! > > > > > > > > I agree with the approach of starting with a simple implementation > > > > that can address a well understood, significant portion of use cases. > > > > > > > > I'm planning to continue work on the prototype that I had shared. > > > > There is production level usage waiting for it fairly soon. I expect > > > > to open a PR in the coming weeks. > > > > > > > > Thomas > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 13, 2021 at 12:15 PM Stephan Ewen > wrote: > > > > > > > > > > Thanks all for this discussion. Looks like there are lots of ideas > and > > > > > folks that are eager to do things, so let's see how we can get this > > > > moving. > > > > > > > > > > My take on this is the following: > > > > > > > > > > There will probably not be one Hybrid source, but possibly multiple > > > ones, > > > > > because of different strategies/requirements. > > > > > - One may be very simple, with switching points known up-front. > > > Would > > > > > be good to have this in a very simple implementation. > > > > > - There may be one where the switch is dynamic and the readers > need > > > > to > > > > > report back where they left off. > > > > > - There may be one that switches back and forth multiple times > > > > during a > > > > > job, for example Kakfa going to DFS whenever it falls behind > retention, > > > > in > > > > > order to catch up again. > > > > > > > > > > This also seems hard to "design on paper"; I expect there are > nuances > > > in > > > > a > > > > > production setup that affect some details of the design. So I'd > feel > > > most > > > > > comfortable in adding a variant of the hybrid source to Flink that > has > > > > been > > > > > used already in a real use case (not necessarily in production, but > > > maybe > > > > > in a testing/staging environment, so it seems to meet all > > > requirements). > > > > > > > > > > > > > > > What do you think about the following approach? > > > > > - If there is a tested PoC, let's try to get it contributed to > Flink > > > > > without trying to make it much more general. > > > > > - When we see similar but a bit different requirements for > another > > > > hybrid > > > > > source, then let's try to evolve the contributed one. > > > > > - If we see new requirements that are so different that they > don't > > > fit > > > > > well with the existing hybrid source, then let us look at building > a > > > > second > > > > > hybrid source for those requirements. > > > > > > > > > > We need to make connector contributions in general more easy, and I > > > think > > > > > it is not a bad thing to end up with different approaches and see > how > > > > these > > > > > play out against each other when being used by users. For example > > > > switching > > > > > with known boundaries, dynamic switching, back-and-forth-switching, > > > etc. > > > > > (I know some committers are planning to do some work on making > > > > > connector contributions easier, with standardized testing > frameworks, > > > > > decoupled CI, etc.) > > > > > > > > > > Best, > > > > > Stephan > > > > > > > > > > > > > > > On Thu, Mar 25, 2021 at 4:41 AM Thomas Weise > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > As mentioned in my previous email, I had been working on a > prototype > > > > for > > > > > > the hybrid source. > > > > > > > > > > > > You ca
[jira] [Created] (FLINK-22456) Support InitializeOnMaster and FinalizeOnMaster to be used in InputFormat
Li created FLINK-22456: -- Summary: Support InitializeOnMaster and FinalizeOnMaster to be used in InputFormat Key: FLINK-22456 URL: https://issues.apache.org/jira/browse/FLINK-22456 Project: Flink Issue Type: Improvement Components: Runtime / Task Reporter: Li In _InputOutputFormatVertex_, _initializeGlobal_ and _finalizeGlobal_ are only called when the Format is _OutputFormat_, however _InputFormat_ is not be called. In FLINK-1722, its say _HadoopOutputFormats_ ues it to do something before and after the task. And they only support _initializeGlobal_ and _finalizeGlobal_ in _OutputFormat_. I don't know why _InputFormat_ doesn't support, anyone can tell me why? But I think _InitializeOnMaster_ and _FinalizeOnMaster_ should also be supported in _InputFormat_. For example, an offline task in _JdbcInputFormat_, user can use _initializeGlobal_ to query the total counts of this task, and then user can create InputSplits by total counts. While task running, user can add progress indicators metric by calculating the total number of records divided by the current number of reads, and even the remaining time of the task can be estimated. It is very helpful for users to view task progress and remaining time through external systems. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22457) KafkaSourceLegacyITCase.testMultipleSourcesOnePartition fails because of timeout
Guowei Ma created FLINK-22457:
-
Summary: KafkaSourceLegacyITCase.testMultipleSourcesOnePartition
fails because of timeout
Key: FLINK-22457
URL: https://issues.apache.org/jira/browse/FLINK-22457
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.14.0
Reporter: Guowei Ma
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17140&view=logs&j=1fc6e7bf-633c-5081-c32a-9dea24b05730&t=80a658d1-f7f6-5d93-2758-53ac19fd5b19&l=7045
{code:java}
Apr 24 23:47:33 [ERROR] Tests run: 21, Failures: 0, Errors: 1, Skipped: 0, Time
elapsed: 174.335 s <<< FAILURE! - in
org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase
Apr 24 23:47:33 [ERROR]
testMultipleSourcesOnePartition(org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase)
Time elapsed: 60.019 s <<< ERROR!
Apr 24 23:47:33 org.junit.runners.model.TestTimedOutException: test timed out
after 6 milliseconds
Apr 24 23:47:33 at sun.misc.Unsafe.park(Native Method)
Apr 24 23:47:33 at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
Apr 24 23:47:33 at
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
Apr 24 23:47:33 at
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
Apr 24 23:47:33 at
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
Apr 24 23:47:33 at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
Apr 24 23:47:33 at
org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:49)
Apr 24 23:47:33 at
org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runMultipleSourcesOnePartitionExactlyOnceTest(KafkaConsumerTestBase.java:1112)
Apr 24 23:47:33 at
org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase.testMultipleSourcesOnePartition(KafkaSourceLegacyITCase.java:87)
Apr 24 23:47:33 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
Apr 24 23:47:33 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Apr 24 23:47:33 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Apr 24 23:47:33 at java.lang.reflect.Method.invoke(Method.java:498)
Apr 24 23:47:33 at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
Apr 24 23:47:33 at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Apr 24 23:47:33 at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
Apr 24 23:47:33 at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
Apr 24 23:47:33 at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
Apr 24 23:47:33 at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
Apr 24 23:47:33 at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
Apr 24 23:47:33 at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22458) Failed when operating empty catalog table
Spongebob created FLINK-22458:
-
Summary: Failed when operating empty catalog table
Key: FLINK-22458
URL: https://issues.apache.org/jira/browse/FLINK-22458
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Affects Versions: 1.12.2
Environment: Flink: 1.12.2
Reporter: Spongebob
The pipline might like: HiveTable -> FlinkCatalogTable(might be empty) ->
HiveTable
It runs normally when the FlinkCatalogTable is not empty, But When
FlinkCatalogTable is empty, Jobmanager throws this exception:
{code:java}
java.lang.Exception: Failed to finalize execution on master at
org.apache.flink.runtime.executiongraph.ExecutionGraph.vertexFinished(ExecutionGraph.java:1373)
~[flownData-1.0-jar-with-dependencies.jar:?] at
org.apache.flink.runtime.executiongraph.ExecutionVertex.executionFinished(ExecutionVertex.java:877)
~[flownData-1.0-jar-with-dependencies.jar:?] at
org.apache.flink.runtime.executiongraph.Execution.markFinished(Execution.java:1241)
~[flownData-1.0-jar-with-dependencies.jar:?] at
org.apache.flink.runtime.executiongraph.ExecutionGraph.updateStateInternal(ExecutionGraph.java:1610)
~[flownData-1.0-jar-with-dependencies.jar:?] at
org.apache.flink.runtime.executiongraph.ExecutionGraph.updateState(ExecutionGraph.java:1584)
~[flownData-1.0-jar-with-dependencies.jar:?] at
org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:663)
~[flownData-1.0-jar-with-dependencies.jar:?] at
org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:89)
~[flownData-1.0-jar-with-dependencies.jar:?] at
org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:447)
~[flownData-1.0-jar-with-dependencies.jar:?] at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_251] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_251] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_251] at java.lang.reflect.Method.invoke(Method.java:498)
~[?:1.8.0_251] at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305)
~[flownData-1.0-jar-with-dependencies.jar:?] at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212)
~[flownData-1.0-jar-with-dependencies.jar:?] at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
~[flownData-1.0-jar-with-dependencies.jar:?] at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
~[flownData-1.0-jar-with-dependencies.jar:?] at
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
[flownData-1.0-jar-with-dependencies.jar:?] at
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
[flownData-1.0-jar-with-dependencies.jar:?] at
scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
[flownData-1.0-jar-with-dependencies.jar:?] at
scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
[flownData-1.0-jar-with-dependencies.jar:?] at
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
[flownData-1.0-jar-with-dependencies.jar:?] at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
[flownData-1.0-jar-with-dependencies.jar:?] at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
[flownData-1.0-jar-with-dependencies.jar:?] at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
[flownData-1.0-jar-with-dependencies.jar:?] at
akka.actor.Actor.aroundReceive(Actor.scala:517)
[flownData-1.0-jar-with-dependencies.jar:?] at
akka.actor.Actor.aroundReceive$(Actor.scala:515)
[flownData-1.0-jar-with-dependencies.jar:?] at
akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
[flownData-1.0-jar-with-dependencies.jar:?] at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
[flownData-1.0-jar-with-dependencies.jar:?] at
akka.actor.ActorCell.invoke(ActorCell.scala:561)
[flownData-1.0-jar-with-dependencies.jar:?] at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
[flownData-1.0-jar-with-dependencies.jar:?] at
akka.dispatch.Mailbox.run(Mailbox.scala:225)
[flownData-1.0-jar-with-dependencies.jar:?] at
akka.dispatch.Mailbox.exec(Mailbox.scala:235)
[flownData-1.0-jar-with-dependencies.jar:?] at
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[flownData-1.0-jar-with-dependencies.jar:?] at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[flownData-1.0-jar-with-dependencies.jar:?] at
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[flownData-1.0-jar-with-dependencies.jar:?] at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[flownData-1.0-jar-with-dependencies.jar:?] Caused by:
o
[jira] [Created] (FLINK-22459) FlinkKafkaProducerITCase testScaleUpAfterScalingDown failed because of Timeout expired after 60000milliseconds while awaiting EndTxn(COMMIT)
Guowei Ma created FLINK-22459:
-
Summary: FlinkKafkaProducerITCase testScaleUpAfterScalingDown
failed because of Timeout expired after 6milliseconds while awaiting
EndTxn(COMMIT)
Key: FLINK-22459
URL: https://issues.apache.org/jira/browse/FLINK-22459
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.12.2
Reporter: Guowei Ma
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17180&view=logs&j=1fc6e7bf-633c-5081-c32a-9dea24b05730&t=80a658d1-f7f6-5d93-2758-53ac19fd5b19&l=6694
{code:java}
[ERROR] Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
261.699 s <<< FAILURE! - in
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase
[ERROR]
testScaleUpAfterScalingDown(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
Time elapsed: 67.768 s <<< ERROR!
org.apache.kafka.common.errors.TimeoutException: Timeout expired after
6milliseconds while awaiting EndTxn(COMMIT)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] FlinkKafkaProducerITCase.testScaleUpAfterScalingDown » Timeout
Timeout expired...
[INFO]
[ERROR] Tests run: 136, Failures: 0, Errors: 1, Skipped: 0
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22460) Conversion to relational algebra failed caused by ''
Haiwei Zhou created FLINK-22460:
---
Summary: Conversion to relational algebra failed caused by ''
Key: FLINK-22460
URL: https://issues.apache.org/jira/browse/FLINK-22460
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Affects Versions: 1.12.1
Reporter: Haiwei Zhou
Flink complains that an insert sql doesn't match the table schema. The
validated type is missing a "NOT NULL" modifier.
{code:java}
py4j.protocol.Py4JJavaError: An error occurred while calling o18.executeSql.
: java.lang.AssertionError: Conversion to relational algebra failed to preserve
datatypes:
validated type:
RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" request, CHAR(7)
CHARACTER SET "UTF-16LE" NOT NULL EXPR$1, BIGINT number, TIMESTAMP(3)
start_time, TIMESTAMP(3) end_time) NOT NULL
converted type:
RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" request, CHAR(7)
CHARACTER SET "UTF-16LE" NOT NULL EXPR$1, BIGINT NOT NULL number, TIMESTAMP(3)
start_time, TIMESTAMP(3) end_time) NOT
NULL{code}
{code:java}
table_env.execute_sql('''
CREATE TABLE preload_stats (
lineitems STRING,
itype STRING,
number BIGINT NOT NULL,
start_time TIMESTAMP(3),
end_time TIMESTAMP(3)
)'''
table_env.execute_sql(
"SELECT request, 'request', number, start_time, end_time "
"FROM result_1 ").execute_insert('preload_stats')
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22461) Resuming Savepoint (file, async, scale down) end-to-end test failed because of binding port fail
Guowei Ma created FLINK-22461:
-
Summary: Resuming Savepoint (file, async, scale down) end-to-end
test failed because of binding port fail
Key: FLINK-22461
URL: https://issues.apache.org/jira/browse/FLINK-22461
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing, Tests
Affects Versions: 1.11.3
Reporter: Guowei Ma
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17179&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=2b7514ee-e706-5046-657b-3430666e7bd9&l=1495
{code:java}
2021-04-25 20:27:11,375 INFO org.apache.flink.core.fs.FileSystem
[] - Hadoop is not in the classpath/dependencies. The extended set
of supported File Systems via Hadoop is not available.
2021-04-25 20:27:11,420 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Install
security context.
2021-04-25 20:27:11,441 INFO
org.apache.flink.runtime.security.modules.HadoopModuleFactory [] - Cannot
create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2021-04-25 20:27:11,444 INFO
org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file
will be created as /tmp/jaas-1367939375368529920.conf.
2021-04-25 20:27:11,450 INFO
org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory [] -
Cannot install HadoopSecurityContext because Hadoop cannot be found in the
Classpath.
2021-04-25 20:27:11,454 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Initializing
cluster services.
2021-04-25 20:27:11,483 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils[] - Trying to
start actor system, external address localhost:6123, bind address 0.0.0.0:6123.
2021-04-25 20:27:12,014 INFO akka.event.slf4j.Slf4jLogger
[] - Slf4jLogger started
2021-04-25 20:27:12,040 INFO akka.remote.Remoting
[] - Starting remoting
2021-04-25 20:27:12,127 ERROR akka.remote.transport.netty.NettyTransport
[] - failed to bind to /0.0.0.0:6123, shutting down Netty transport
2021-04-25 20:27:12,138 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Shutting
StandaloneSessionClusterEntrypoint down with application status FAILED.
Diagnostics java.net.BindException: Could not start actor system on any port in
port range 6123
at
org.apache.flink.runtime.clusterframework.BootstrapTools.startRemoteActorSystem(BootstrapTools.java:173)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils$AkkaRpcServiceBuilder.createAndStart(AkkaRpcServiceUtils.java:363)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils.createRemoteRpcService(AkkaRpcServiceUtils.java:91)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:278)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:223)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:177)
at
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:174)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:577)
at
org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint.main(StandaloneSessionClusterEntrypoint.java:67)
.
2021-04-25 20:27:12,139 INFO
akka.remote.RemoteActorRefProvider$RemotingTerminator[] - Shutting down
remote daemon.
2021-04-25 20:27:12,141 INFO
akka.remote.RemoteActorRefProvider$RemotingTerminator[] - Remote daemon
shut down; proceeding with flushing remote transports.
2021-04-25 20:27:12,143 ERROR akka.remote.Remoting
[] - Remoting system has been terminated abrubtly. Attempting to
shut down transports
2021-04-25 20:27:12,143 INFO
akka.remote.RemoteActorRefProvider$RemotingTerminator[] - Remoting shut
down.
2021-04-25 20:27:12,187 ERROR
org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Could not
start cluster entrypoint StandaloneSessionClusterEntrypoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to
initialize the cluster entrypoint StandaloneSessionClusterEntrypoint.
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:200)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:577)
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at
org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypo
Re: [VOTE] Release 1.12.3, release candidate #1
+1 (binding) - Verified the checksum and signature - Reviewed the website PR and have left a few comments - Installed PyFlink with the source package on Mac OS and everything works well - Ran a PyFlink example which uses both Python Table API and Python DataStream API Regards, Dian > 2021年4月24日 上午5:52，Arvid Heise 写道： > > Hi everyone, > Please review and vote on the release candidate #1 for the version 1.12.3, > as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release and binary convenience releases to be > deployed to dist.apache.org [2], which are signed with the key with > fingerprint 476DAA5D1FF08189 [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tag "release-1.2.3-rc3" [5], > * website pull request listing the new release and adding announcement blog > post [6]. > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Thanks, > Your friendly release manager Arvid > > [1] > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12349691 > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.12.3-rc1/ > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] https://repository.apache.org/content/repositories/orgapacheflink-1419 > [5] https://github.com/apache/flink/releases/tag/release-1.12.3-rc1 > [6] https://github.com/apache/flink-web/pull/437
[jira] [Created] (FLINK-22462) JdbcExactlyOnceSinkE2eTest.testInsert failed because of too many clients.
Guowei Ma created FLINK-22462:
-
Summary: JdbcExactlyOnceSinkE2eTest.testInsert failed because of
too many clients.
Key: FLINK-22462
URL: https://issues.apache.org/jira/browse/FLINK-22462
Project: Flink
Issue Type: Bug
Components: Connectors / JDBC
Affects Versions: 1.14.0
Reporter: Guowei Ma
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17178&view=logs&j=ba53eb01-1462-56a3-8e98-0dd97fbcaab5&t=bfbc6239-57a0-5db0-63f3-41551b4f7d51&l=13514
{code:java}
Apr 25 23:05:31 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
elapsed: 138.743 s <<< FAILURE! - in
org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest
Apr 25 23:05:31 [ERROR]
testInsert(org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest) Time
elapsed: 137.267 s <<< ERROR!
Apr 25 23:05:31 org.postgresql.util.PSQLException: FATAL: sorry, too many
clients already
Apr 25 23:05:31 at
org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:524)
Apr 25 23:05:31 at
org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:145)
Apr 25 23:05:31 at
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:196)
Apr 25 23:05:31 at
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
Apr 25 23:05:31 at
org.postgresql.jdbc.PgConnection.(PgConnection.java:211)
Apr 25 23:05:31 at org.postgresql.Driver.makeConnection(Driver.java:459)
Apr 25 23:05:31 at org.postgresql.Driver.connect(Driver.java:261)
Apr 25 23:05:31 at
java.sql.DriverManager.getConnection(DriverManager.java:664)
Apr 25 23:05:31 at
java.sql.DriverManager.getConnection(DriverManager.java:247)
Apr 25 23:05:31 at
org.apache.flink.connector.jdbc.xa.JdbcXaFacadeTestHelper.getInsertedIds(JdbcXaFacadeTestHelper.java:81)
Apr 25 23:05:31 at
org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest.testInsert(JdbcExactlyOnceSinkE2eTest.java:119)
Apr 25 23:05:31 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22463) IllegalArgumentException is thrown in WindowAttachedWindowingStrategy when two phase is enabled for distinct agg
godfrey he created FLINK-22463:
--
Summary: IllegalArgumentException is thrown in
WindowAttachedWindowingStrategy when two phase is enabled for distinct agg
Key: FLINK-22463
URL: https://issues.apache.org/jira/browse/FLINK-22463
Project: Flink
Issue Type: Bug
Reporter: godfrey he
Caused by: java.lang.IllegalArgumentException
at
org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:122)
at
org.apache.flink.table.planner.plan.logical.WindowAttachedWindowingStrategy.(WindowAttachedWindowingStrategy.java:51)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
The reason is the {{windowStart}} may be {{-1}} when two phase is enabled for
distinct agg, see
[TwoStageOptimizedWindowAggregateRule.java#L143|https://github.com/apache/flink/blob/a3363b91b144edfbae5ab114984ded622d3f8fbc/flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/rules/physical/stream/TwoStageOptimizedWindowAggregateRule.java#L143]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22464) OperatorEventSendingCheckpointITCase.testOperatorEventLostWithReaderFailure hangs with `AdaptiveScheduler`
Guowei Ma created FLINK-22464:
-
Summary:
OperatorEventSendingCheckpointITCase.testOperatorEventLostWithReaderFailure
hangs with `AdaptiveScheduler`
Key: FLINK-22464
URL: https://issues.apache.org/jira/browse/FLINK-22464
Project: Flink
Issue Type: Bug
Components: Tests
Affects Versions: 1.14.0
Reporter: Guowei Ma
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17178&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=a0a633b8-47ef-5c5a-2806-3c13b9e48228&l=8171
{code:java}
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22465) KafkaSourceITCase.testValueOnlyDeserializer hangs
Guowei Ma created FLINK-22465:
-
Summary: KafkaSourceITCase.testValueOnlyDeserializer hangs
Key: FLINK-22465
URL: https://issues.apache.org/jira/browse/FLINK-22465
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.14.0
Reporter: Guowei Ma
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17104&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5&l=28977
{code:java}
at
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
at
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
at java.util.Iterator.forEachRemaining(Iterator.java:115)
at
org.apache.flink.connector.kafka.source.KafkaSourceITCase.testValueOnlyDeserializer(KafkaSourceITCase.java:155)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.jav
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22466) KafkaSourceLegacyITCase.testOneToOneSources fail because the OperatorEvent lost
Guowei Ma created FLINK-22466:
-
Summary: KafkaSourceLegacyITCase.testOneToOneSources fail because
the OperatorEvent lost
Key: FLINK-22466
URL: https://issues.apache.org/jira/browse/FLINK-22466
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.13.0
Reporter: Guowei Ma
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17110&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5&l=7010
{code:java}
2021-04-23T14:31:37.1620668Z Apr 23 14:31:37 [INFO] Running
org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase
2021-04-23T14:32:27.0398155Z java.util.concurrent.ExecutionException:
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2021-04-23T14:32:27.0400673Zat
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
2021-04-23T14:32:27.0401550Zat
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
2021-04-23T14:32:27.0402365Zat
org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:49)
2021-04-23T14:32:27.0403227Zat
org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runOneToOneExactlyOnceTest(KafkaConsumerTestBase.java:1009)
2021-04-23T14:32:27.0403937Zat
org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase.testOneToOneSources(KafkaSourceLegacyITCase.java:77)
2021-04-23T14:32:27.0404881Zat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-04-23T14:32:27.0405293Zat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-04-23T14:32:27.0406792Zat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-04-23T14:32:27.0407333Zat
java.lang.reflect.Method.invoke(Method.java:498)
2021-04-23T14:32:27.0407743Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2021-04-23T14:32:27.0408318Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2021-04-23T14:32:27.0408784Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2021-04-23T14:32:27.0409246Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2021-04-23T14:32:27.0409742Zat
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
2021-04-23T14:32:27.0410251Zat
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
2021-04-23T14:32:27.0410727Zat
java.util.concurrent.FutureTask.run(FutureTask.java:266)
2021-04-23T14:32:27.0411065Zat java.lang.Thread.run(Thread.java:748)
2021-04-23T14:32:27.0411430Z Caused by:
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2021-04-23T14:32:27.0411931Zat
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2021-04-23T14:32:27.0412631Zat
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
2021-04-23T14:32:27.0413144Zat
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
2021-04-23T14:32:27.0413605Zat
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
2021-04-23T14:32:27.0414063Zat
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-04-23T14:32:27.0414497Zat
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-04-23T14:32:27.0415002Zat
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
2021-04-23T14:32:27.0415526Zat
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2021-04-23T14:32:27.0416026Zat
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2021-04-23T14:32:27.0416498Zat
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-04-23T14:32:27.0417403Zat
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-04-23T14:32:27.0417945Zat
org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1081)
2021-04-23T14:32:27.0418481Zat
akka.dispatch.OnComplete.internal(Future.scala:264)
2021-04-23T14:32:27.0418820Zat
akka.dispatch.OnComplete.internal(Future.scala:261)
2021-04-23T14:32:27.0419188Zat
akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
2021-04-23T14:32:27.0419574Zat
akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
2021-04-23T14:32:27.0419956Zat
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-04-23T14:32:27.0420414Zat
org.apache.flink.runtime.concurrent.Executors$
[jira] [Created] (FLINK-22467) OverAggregateITCase.testRowNumberOnOver is unstable
Jingsong Lee created FLINK-22467: Summary: OverAggregateITCase.testRowNumberOnOver is unstable Key: FLINK-22467 URL: https://issues.apache.org/jira/browse/FLINK-22467 Project: Flink Issue Type: Test Components: Table SQL / Planner Reporter: Jingsong Lee Fix For: 1.14.0 assertEquals(expected, sink.getAppendResults) Results should be sorted. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22468) Make StateFun build with Java 11
Tzu-Li (Gordon) Tai created FLINK-22468: --- Summary: Make StateFun build with Java 11 Key: FLINK-22468 URL: https://issues.apache.org/jira/browse/FLINK-22468 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-3.1.0, statefun-3.0.1 StateFun is currently not building with Java 11 due to the removal of javax.annotations in JDK 11. This should be fixable by manually adding the dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22469) HistoryServer starts with NoSuchFileException
Gabor Somogyi created FLINK-22469:
-
Summary: HistoryServer starts with NoSuchFileException
Key: FLINK-22469
URL: https://issues.apache.org/jira/browse/FLINK-22469
Project: Flink
Issue Type: Bug
Components: Runtime / Web Frontend
Affects Versions: 1.12.2
Reporter: Gabor Somogyi
When history server started initially it throws the following exception:
{code:java}
2021-04-23 23:25:17,487 ERROR
org.apache.flink.runtime.webmonitor.history.HistoryServerArchiveFetcher [] -
Failed to update job overview.
java.nio.file.NoSuchFileException:
/var/folders/jd/35_sh46s7zq0qc6khfw8hc80gn/T/flink-web-history-35a77053-0e52-4cb3-8d22-626e3ce3cbd7/jobs/overview.json
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
~[?:1.8.0_282]
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
~[?:1.8.0_282]
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
~[?:1.8.0_282]
at
sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
~[?:1.8.0_282]
at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_282]
at java.nio.file.Files.createFile(Files.java:632) ~[?:1.8.0_282]
at
org.apache.flink.runtime.webmonitor.history.HistoryServer.createOrGetFile(HistoryServer.java:324)
~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.webmonitor.history.HistoryServerArchiveFetcher.updateJobOverview(HistoryServerArchiveFetcher.java:464)
[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.webmonitor.history.HistoryServerArchiveFetcher.access$000(HistoryServerArchiveFetcher.java:74)
[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.webmonitor.history.HistoryServerArchiveFetcher$JobArchiveFetcherTask.(HistoryServerArchiveFetcher.java:199)
[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.webmonitor.history.HistoryServerArchiveFetcher.(HistoryServerArchiveFetcher.java:124)
[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.webmonitor.history.HistoryServer.(HistoryServer.java:230)
[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.webmonitor.history.HistoryServer.(HistoryServer.java:146)
[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:130)
[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:127)
[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.webmonitor.history.HistoryServer.main(HistoryServer.java:126)
[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
{code}
The issue is that "webDi"r not yet created by "HistoryServer" when
"HistoryServerArchiveFetcher" tries to reach "jobs/overview.json" file in it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [VOTE] Release 1.13.0, release candidate #2
I see. Thanks Jark for the clarification. On Sun, Apr 25, 2021 at 10:36 PM Jark Wu wrote: > Hi Rui, > > Documentation is not a part of the release package. > The changes to the documentation will take effect in 24 hours > and can be accessed on the web [1] then. > Docs can even be updated after 1.13 is released. > So we don't need to cancel the RC for this. > > Best, > Jark > > [1]: > > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/hive/overview/ > > On Sun, 25 Apr 2021 at 13:47, Rui Li wrote: > > > Hi Dawid, > > > > I have just merged the doc change for hive dialect [1], which talks about > > how to use hive dialect to run hive queries. Do you think we can have > > another RC to include it? Sorry for the trouble. > > > > [1] https://issues.apache.org/jira/browse/FLINK-22119 > > > > On Sat, Apr 24, 2021 at 1:36 AM Dawid Wysakowicz > > > wrote: > > > > > Hi everyone, > > > Please review and vote on the release candidate #2 for the version > > 1.13.0, > > > as follows: > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > > > > The complete staging area is available for your review, which includes: > > > * JIRA release notes [1], > > > * the official Apache source release and binary convenience releases to > > be > > > deployed to dist.apache.org [2], which are signed with the key with > > > fingerprint 31D2DD10BFC15A2D [3], > > > * all artifacts to be deployed to the Maven Central Repository [4], > > > * source code tag "release-1.13.0-rc2" [5], > > > * website pull request listing the new release and adding announcement > > > blog post [6]. > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > approval, with at least 3 PMC affirmative votes. > > > > > > Thanks, > > > Dawid Wysakowicz > > > > > > [1] > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12349287 > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.13.0-rc2/ > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > > [4] > > > > https://repository.apache.org/content/repositories/orgapacheflink-1420/ > > > [5] https://github.com/apache/flink/tree/release-1.13.0-rc2 > > > [6] https://github.com/apache/flink-web/pull/436 > > > > > > > > > -- > > Best regards! > > Rui Li > > > -- Best regards! Rui Li
Re: when should `FlinkYarnSessionCli` be included for parsing CLI arguments?
Hi, Tony. What is the version of your flink-dist. AFAIK, this issue should be addressed in FLINK-15852[1]. Could you give the client log of case 2(set the log level to DEBUG would be better). [1] https://issues.apache.org/jira/browse/FLINK-15852 Best, Yangze Guo On Sun, Apr 25, 2021 at 11:33 AM Tony Wei wrote: > > Hi Experts, > > I recently tried to run yarn-application mode on my yarn cluster, and I had a > problem related to configuring `execution.target`. > After reading the source code and doing some experiments, I found that there > should be some room of improvement for `FlinkYarnSessionCli` or > `AbstractYarnCli`. > > My experiments are: > > setting `execution.target: yarn-application` in flink-conf.yaml and run > `flink run-application -t yarn-application`: run job successfully. > > `FlinkYarnSessionCli` is not active > `GenericCLI` is active > > setting `execution.target: yarn-per-job` in flink-conf.yaml and run `flink > run-application -t yarn-application`: run job failed > > failed due to `ClusterDeploymentException` [1] > `FlinkYarnSessionCli` is active > > setting `execution.target: yarn-application` in flink-conf.yaml and run > `flink run -t yarn-per-job`: run job successfully. > > `FlinkYarnSessionCli` is not active > `GenericCLI` is active > > setting `execution.target: yarn-per-job` in flink-conf.yaml and run `flink > run -t yarn-per-job`: run job successfully. > > `FlinkYarnSessionCli` is active > > From `AbstractYarnCli#isActive` [2] and `FlinkYarnSessionCli#isActive` [3], > `FlinkYarnSessionCli` will be active when `execution.target` is specified > with `yarn-per-job` or `yarn-session`. > > According to the flink official document [4], I thought the 2nd experiment > should also work well, but it didn't. >> >> The --target will overwrite the execution.target specified in the >> config/flink-config.yaml. > > > The root cause is that `FlinkYarnSessionCli` only overwrite the > `execution.target` with `yarn-session` or `yarn-per-job` [5], but no > `yarn-application`. > So, my question is > > should we use `FlinkYarnSessionCli` in case 2? > if we should, how we can improve `FlinkYarnSessionCli` so that we can > overwrite `execution.target` via `--target`? > > and one more improvement, the config description for `execution.target` [6] > should include `yarn-application` as well. > > [1] > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L439-L447 > [2] > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/cli/AbstractYarnCli.java#L54-L66 > [3] > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java#L373-L377 > [4] > https://ci.apache.org/projects/flink/flink-docs-stable/deployment/cli.html#selecting-deployment-targets > [5] > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java#L397-L413 > [6] > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/DeploymentOptions.java#L41-L46 > > best regards, >
