Require help regarding possible issue/bug I'm facing while using Flink

2022-03-06 Thread Chia De Xun . 
Greetings,

I'm facing a difficult issue/bug while working with Flink. Would definitely
appreciate some official expert help on this issue. I have posted my
problem on StackOverflow
,
but have no replies at the moment.

Let me know if you have any questions/clarifications for me! It would be
best appreciated.

Best Regards,
De Xun

[jira] [Created] (FLINK-26499) Azure hang due to unable to download akka artifact

2022-03-06 Thread Yun Gao (Jira) 
Yun Gao created FLINK-26499:
---

 Summary: Azure hang due to unable to download akka artifact
 Key: FLINK-26499
 URL: https://issues.apache.org/jira/browse/FLINK-26499
 Project: Flink
  Issue Type: Bug
  Components: Build System / Azure Pipelines
Affects Versions: 1.15.0
Reporter: Yun Gao


{code:java}
Mar 03 14:23:14 "main" #1 prio=5 os_prio=0 tid=0x7f9d8800b800 nid=0x4dc5 in 
Object.wait() [0x7f9d8ebd5000]
Mar 03 14:23:14java.lang.Thread.State: WAITING (on object monitor)
Mar 03 14:23:14 at java.lang.Object.wait(Native Method)
Mar 03 14:23:14 - waiting on <0x8d7e9ae0> (a 
java.lang.UNIXProcess)
Mar 03 14:23:14 at java.lang.Object.wait(Object.java:502)
Mar 03 14:23:14 at java.lang.UNIXProcess.waitFor(UNIXProcess.java:395)
Mar 03 14:23:14 - locked <0x8d7e9ae0> (a java.lang.UNIXProcess)
Mar 03 14:23:14 at 
org.apache.flink.runtime.rpc.akka.FallbackAkkaRpcSystemLoader.downloadDependencies(FallbackAkkaRpcSystemLoader.java:149)
Mar 03 14:23:14 at 
org.apache.flink.runtime.rpc.akka.FallbackAkkaRpcSystemLoader.loadRpcSystem(FallbackAkkaRpcSystemLoader.java:75)
Mar 03 14:23:14 at 
org.apache.flink.runtime.rpc.RpcSystem.load(RpcSystem.java:101)
Mar 03 14:23:14 at 
org.apache.flink.runtime.rpc.RpcSystem.load(RpcSystem.java:84)
Mar 03 14:23:14 at 
org.apache.flink.runtime.testutils.MiniClusterResource.(MiniClusterResource.java:73)
Mar 03 14:23:14 at 
org.apache.flink.test.util.AbstractTestBase.(AbstractTestBase.java:66)
Mar 03 14:23:14 at sun.misc.Unsafe.ensureClassInitialized(Native Method)
Mar 03 14:23:14 at 
sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
Mar 03 14:23:14 at 
sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:156)
Mar 03 14:23:14 at 
java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
Mar 03 14:23:14 at 
java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
Mar 03 14:23:14 at java.lang.reflect.Field.get(Field.java:393)
Mar 03 14:23:14 at 
org.junit.runners.model.FrameworkField.get(FrameworkField.java:92)
Mar 03 14:23:14 at 
org.junit.runners.model.TestClass.collectAnnotatedFieldValues(TestClass.java:249)
Mar 03 14:23:14 at 
org.junit.runners.ParentRunner.classRules(ParentRunner.java:280)
Mar 03 14:23:14 at 
org.junit.runners.ParentRunner.withClassRules(ParentRunner.java:268)
Mar 03 14:23:14 at 
org.junit.runners.ParentRunner.classBlock(ParentRunner.java:217)
Mar 03 14:23:14 at 
org.junit.runners.ParentRunner.run(ParentRunner.java:412)
Mar 03 14:23:14 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
Mar 03 14:23:14 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
Mar 03 14:23:14 at 
org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
Mar 03 14:23:14 at 
org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
Mar 03 14:23:14 at 
org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
Mar 03 14:23:14 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
Mar 03 14:23:14 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
Mar 03 14:23:14 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
Mar 03 14:23:14 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator$$Lambda$157/260840925.accept(Unknown
 Source)
Mar 03 14:23:14 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
Mar 03 14:23:14 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
Mar 03 14:23:14 at 
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
Mar 03 14:23:14 at 
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
 {code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=32486&view=logs&j=a57e0635-3fad-5b08-57c7-a4142d7d6fa9&t=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7&l=60259



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26500) AdaptiveSchedulerClusterITCase.testAutomaticScaleUp failed on azure

2022-03-06 Thread Yun Gao (Jira) 
Yun Gao created FLINK-26500:
---

 Summary: AdaptiveSchedulerClusterITCase.testAutomaticScaleUp 
failed on azure
 Key: FLINK-26500
 URL: https://issues.apache.org/jira/browse/FLINK-26500
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.15.0
Reporter: Yun Gao


{code:java}
Mar 03 13:38:24 [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time 
elapsed: 21.854 s <<< FAILURE! - in 
org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerClusterITCase
Mar 03 13:38:24 [ERROR] 
org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerClusterITCase.testAutomaticScaleUp
  Time elapsed: 16.035 s  <<< ERROR!
Mar 03 13:38:24 java.util.concurrent.TimeoutException: Condition was not met in 
given timeout.
Mar 03 13:38:24 at 
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:167)
Mar 03 13:38:24 at 
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:145)
Mar 03 13:38:24 at 
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:137)
Mar 03 13:38:24 at 
org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerClusterITCase.waitUntilParallelismForVertexReached(AdaptiveSchedulerClusterITCase.java:267)
Mar 03 13:38:24 at 
org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerClusterITCase.testAutomaticScaleUp(AdaptiveSchedulerClusterITCase.java:147)
Mar 03 13:38:24 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
Mar 03 13:38:24 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Mar 03 13:38:24 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Mar 03 13:38:24 at java.lang.reflect.Method.invoke(Method.java:498)
Mar 03 13:38:24 at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
Mar 03 13:38:24 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Mar 03 13:38:24 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
Mar 03 13:38:24 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
Mar 03 13:38:24 at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
Mar 03 13:38:24 at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
Mar 03 13:38:24 at 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
Mar 03 13:38:24 at 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
Mar 03 13:38:24 at 
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
Mar 03 13:38:24 at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
Mar 03 13:38:24 at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
Mar 03 13:38:24 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
Mar 03 13:38:24 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
Mar 03 13:38:24 at 
org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
Mar 03 13:38:24 at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
Mar 03 13:38:24 at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
Mar 03 13:38:24 at 
org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
Mar 03 13:38:24 at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
Mar 03 13:38:24 at 
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
Mar 03 13:38:24 at 
org.junit.runners.ParentRunner.run(ParentRunner.java:413)
Mar 03 13:38:24 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
Mar 03 13:38:24 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
Mar 03 13:38:24 at 
org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
 {code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=32488&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=24c3384f-1bcb-57b3-224f-51bf973bbee8&l=9618



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26501) Quickstarts Scala nightly end-to-end test failed on azure due to checkponts failed and logs contains exceptions

2022-03-06 Thread Yun Gao (Jira) 
Yun Gao created FLINK-26501:
---

 Summary: Quickstarts Scala nightly end-to-end test failed on azure 
due to checkponts failed and logs contains exceptions
 Key: FLINK-26501
 URL: https://issues.apache.org/jira/browse/FLINK-26501
 Project: Flink
  Issue Type: Bug
  Components: API / Scala, Runtime / Checkpointing
Affects Versions: 1.14.3
Reporter: Yun Gao


{code:java}
2022-03-05T02:35:36.4040037Z Mar 05 02:35:36 2022-03-05 02:35:34,334 INFO  
org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Triggering 
checkpoint 1 (type=CHECKPOINT) @ 1646447734295 for job 
b236087395260dc34648b84c2b86d6e8.
2022-03-05T02:35:36.4041701Z Mar 05 02:35:36 2022-03-05 02:35:34,387 INFO  
org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Decline 
checkpoint 1 by task e8a324cae6bf452d32db6797bbbafad0 of job 
b236087395260dc34648b84c2b86d6e8 at 127.0.0.1:45911-0a50f5 @ localhost 
(dataPort=44047).
2022-03-05T02:35:36.4043279Z Mar 05 02:35:36 
org.apache.flink.util.SerializedThrowable: Task name with subtask : Source: 
Sequence Source (Deprecated) -> Map -> Sink: Unnamed (1/1)#0 Failure reason: 
Checkpoint was declined (task is closing)
2022-03-05T02:35:36.4044531Z Mar 05 02:35:36at 
org.apache.flink.runtime.taskmanager.Task.declineCheckpoint(Task.java:1389) 
~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
2022-03-05T02:35:36.4045729Z Mar 05 02:35:36at 
org.apache.flink.runtime.taskmanager.Task.declineCheckpoint(Task.java:1382) 
~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
2022-03-05T02:35:36.4047172Z Mar 05 02:35:36at 
org.apache.flink.runtime.taskmanager.Task.triggerCheckpointBarrier(Task.java:1348)
 ~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
2022-03-05T02:35:36.4049092Z Mar 05 02:35:36at 
org.apache.flink.runtime.taskexecutor.TaskExecutor.triggerCheckpoint(TaskExecutor.java:956)
 ~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
2022-03-05T02:35:36.4050158Z Mar 05 02:35:36at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322]
2022-03-05T02:35:36.4050929Z Mar 05 02:35:36at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_322]
2022-03-05T02:35:36.4051776Z Mar 05 02:35:36at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_322]
2022-03-05T02:35:36.4052559Z Mar 05 02:35:36at 
java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322]
2022-03-05T02:35:36.4053373Z Mar 05 02:35:36at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:316)
 ~[?:?]
2022-03-05T02:35:36.4054849Z Mar 05 02:35:36at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
 ~[?:?]
2022-03-05T02:35:36.4055685Z Mar 05 02:35:36at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:314)
 ~[?:?]
2022-03-05T02:35:36.4056461Z Mar 05 02:35:36at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
 ~[?:?]
2022-03-05T02:35:36.4057219Z Mar 05 02:35:36at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
 ~[?:?]
2022-03-05T02:35:36.4057899Z Mar 05 02:35:36at 
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) ~[?:?]
2022-03-05T02:35:36.4059666Z Mar 05 02:35:36at 
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) ~[?:?]
2022-03-05T02:35:36.4061005Z Mar 05 02:35:36at 
scala.PartialFunction.applyOrElse(PartialFunction.scala:123) 
~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
2022-03-05T02:35:36.4062324Z Mar 05 02:35:36at 
scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) 
~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
2022-03-05T02:35:36.4063941Z Mar 05 02:35:36at 
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) ~[?:?]
2022-03-05T02:35:36.4065009Z Mar 05 02:35:36at 
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) 
~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
2022-03-05T02:35:36.4066205Z Mar 05 02:35:36at 
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) 
~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
2022-03-05T02:35:36.4067514Z Mar 05 02:35:36at 
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) 
~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
2022-03-05T02:35:36.4068255Z Mar 05 02:35:36at 
akka.actor.Actor.aroundReceive(Actor.scala:537) ~[?:?]
2022-03-05T02:35:36.4069019Z Mar 05 02:35:36at 
akka.actor.Actor.aroundReceive$(Actor.scala:535) ~[?:?]
2022-03-05T02:35:36.4069638Z Mar 05 02:35:36at 
akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) ~[?:?]
2022-03-05T02:35:36.4070271Z Mar 05 02:35:36at 
akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)

[jira] [Created] (FLINK-26502) Multiple component leader election has different close/stop behavior

2022-03-06 Thread Niklas Semmler (Jira) 
Niklas Semmler created FLINK-26502:
--

 Summary: Multiple component leader election has different 
close/stop behavior 
 Key: FLINK-26502
 URL: https://issues.apache.org/jira/browse/FLINK-26502
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.15.0
Reporter: Niklas Semmler


The new multi component leader election driver behaves different when the 
service is closed.

The 
[ZooKeeperLeaderElectionDriver#close|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/leaderelection/ZooKeeperLeaderElectionDriver.java#L120]
 method closes the Zookeeper connection. In contrast, the 
[MultipleComponentLeaderElectionDriverAdapter#close|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/leaderelection/MultipleComponentLeaderElectionDriverAdapter.java#L55]
 (also a LeaderElectionDriver object) only unregisters the event handler. It 
relies on the 
[ZooKeeperMultipleComponentLeaderElectionDriver#close|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/leaderelection/ZooKeeperMultipleComponentLeaderElectionDriver.java#L106]
 method to close the Zookeeper connection. Currently this method is only called 
when the HighAvailabilityServices are stopped. This difference in behavior 
means that previously enabled tests are not working anymore FLINK-25235.

Together with [~dmvk], I propose that we align the behavior of the 
MultipleComponentLeaderElectionDriverAdapter to the existing behavior. We can 
do so by closing the Zookeeper connection once all components that are part of 
the multiple component leader election have stopped.

PS: I use ZooKeeper here as an example, the same can be said about Kubernetes 
as well.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26503) Multi-network deployments may lead to hard-to-debug issues

2022-03-06 Thread Niklas Semmler (Jira) 
Niklas Semmler created FLINK-26503:
--

 Summary: Multi-network deployments may lead to hard-to-debug issues
 Key: FLINK-26503
 URL: https://issues.apache.org/jira/browse/FLINK-26503
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.15.0
Reporter: Niklas Semmler


The way the TaskManager defines it's receiving network address may create 
hard-to-debug issues for deployment setups where a TaskManager can reach a 
JobManager over more than one interface. Additionally. the following conditions 
have to be met:
* The JobManager is unreachable for some time on the TaskManager's start-up
* The receiving address {{taskmanager.host}} is not specified while the 
{{taskmanager.bind-host}} is set to a concrete value (and not {{0.0.0.0}}).

*Background*
A TaskManager has two settings to configure its network: 
{{taskmanager.bind-host}} and {{taskmanager.host}}. 
* {{taskmanager.bind-host}} decides to what interfaces the TaskManager binds on 
start-up. When not set it defaults to {{0.0.0.0}}, the most permissive 
setting). With FLINK-24474, this setting is now set to {{localhost}} via the 
configuration file. This line is disabled for docker setups.
* {{taskmanager.host}} specifies what the TaskManager announces as receiving 
address (i.e., where it wants to be reached by the JobManager). When not set, 
the TaskManager cycles through the available interfaces (a slight 
simplification) and attempts to connect to the JobManager via each interface. 
It chooses the address of the interface that allows a successful connection. If 
no connection can be made before a timeout it chooses the address of the 
hostname.

In network scenarios with multiple interfaces connecting TaskManager to the 
JobManager, a user could specify an address & interface with 
{{taskmanager.bind-host}} that is different to the address & network defined by 
the {{taskmanager.host}}.

*Example*
A straightforward way to create such a scenario is to set the 
{{taskmanager.bind-host}} to {{localhost}}. In this situation (and under the 
conditions mentioned above), the TaskManager will fall back to the external IP 
address of the hostname while binding to the localhost. This means that the 
JobManager will be able to receive packets from the TaskManager, but not 
respond.  

*Proposed solution*
Can we help users debug these scenarios, by adding log messages? Could we note 
that the IP address of the TaskManager is not on the JobManager's network (as 
defined by the interface the JobManager binds to) and give this as a hint?



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26504) Fix the incorrect type error in unbounded Python UDAF

2022-03-06 Thread Huang Xingbo (Jira) 
Huang Xingbo created FLINK-26504:


 Summary: Fix the incorrect type error in unbounded Python UDAF
 Key: FLINK-26504
 URL: https://issues.apache.org/jira/browse/FLINK-26504
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.13.2, 1.14.0, 1.15.0
Reporter: Huang Xingbo
Assignee: Huang Xingbo
 Fix For: 1.15.0, 1.13.7, 1.14.5


The stack trace is 
{code:java}
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: 
Error received from SDK harness for instruction 1: Traceback (most recent call 
last):
  File 
"/usr/local/python3/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
 line 289, in _execute
response = task()
  File 
"/usr/local/python3/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
 line 362, in 
lambda: self.create_worker().do_instruction(request), request)
  File 
"/usr/local/python3/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
 line 607, in do_instruction
getattr(request, request_type), request.instruction_id)
  File 
"/usr/local/python3/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
 line 644, in process_bundle
bundle_processor.process_bundle(instruction_id))
  File 
"/usr/local/python3/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 1000, in process_bundle
element.data)
  File 
"/usr/local/python3/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 228, in process_encoded
self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py", line 357, in 
apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 359, in 
apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 221, in 
apache_beam.runners.worker.operations.SingletonConsumerSet.receive
  File "pyflink/fn_execution/beam/beam_operations_fast.pyx", line 71, in 
pyflink.fn_execution.beam.beam_operations_fast.FunctionOperation.process
  File "pyflink/fn_execution/beam/beam_operations_fast.pyx", line 84, in 
pyflink.fn_execution.beam.beam_operations_fast.FunctionOperation.process
  File 
"/usr/local/python3/lib/python3.7/site-packages/pyflink/fn_execution/operations.py",
 line 115, in process_element
return self.func(value)
  File 
"/usr/local/python3/lib/python3.7/site-packages/pyflink/fn_execution/operations.py",
 line 384, in process_element_or_timer
self.group_agg_function.on_timer(input_data[3])
TypeError: Argument 'key' has incorrect type (expected 
pyflink.fn_execution.coder_impl_fast.InternalRow, got Row)

at 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:60)
at 
org.apache.beam.runners.fnexecution.control.SdkHarnessClient$BundleProcessor$ActiveBundle.close(SdkHarnessClient.java:504)
at 
org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory$1.close(DefaultJobBundleFactory.java:555)
at 
org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.finishBundle(BeamPythonFunctionRunner.java:381)
{code}
The condition for this bug to occur is that state clean is triggered. The 
workaround is to disable table.exec.state.ttl





--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26505) Support non equality condition in Hive dialect

2022-03-06 Thread luoyuxia (Jira) 
luoyuxia created FLINK-26505:


 Summary: Support non equality condition in Hive dialect
 Key: FLINK-26505
 URL: https://issues.apache.org/jira/browse/FLINK-26505
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Hive
Reporter: luoyuxia
 Fix For: 1.15.0


It'll throw exception with the following sql which can be executed in hive:
{code:java}
select count(1)
from
  (select key
  from t1
  where key = 0) t1
left semi join
  (select key
  from t2
  where key = 0) t2
on 1 = 1;
{code}






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26506) Support StreamExecutionEnvironment.registerCachedFile in Python DataStream API

2022-03-06 Thread Dian Fu (Jira) 
Dian Fu created FLINK-26506:
---

 Summary: Support StreamExecutionEnvironment.registerCachedFile in 
Python DataStream API
 Key: FLINK-26506
 URL: https://issues.apache.org/jira/browse/FLINK-26506
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Reporter: Dian Fu
 Fix For: 1.16.0


This API is missed in Python DataStream API.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [ANNOUNCE] New Apache Flink Committer - David Morávek

2022-03-06 Thread David Morávek 
Thanks everyone!

Best,
D.

On Sun 6. 3. 2022 at 9:07, Yuan Mei  wrote:

> Congratulations, David!
>
> Best Regards,
> Yuan
>
> On Sat, Mar 5, 2022 at 8:13 PM Roman Khachatryan  wrote:
>
> > Congratulations, David!
> >
> > Regards,
> > Roman
> >
> > On Fri, Mar 4, 2022 at 7:54 PM Austin Cawley-Edwards
> >  wrote:
> > >
> > > Congrats David!
> > >
> > > On Fri, Mar 4, 2022 at 12:18 PM Zhilong Hong 
> > wrote:
> > >
> > > > Congratulations, David!
> > > >
> > > > Best,
> > > > Zhilong
> > > >
> > > > On Sat, Mar 5, 2022 at 1:09 AM Piotr Nowojski 
> > > > wrote:
> > > >
> > > > > Congratulations :)
> > > > >
> > > > > pt., 4 mar 2022 o 16:04 Aitozi  napisał(a):
> > > > >
> > > > > > Congratulations David!
> > > > > >
> > > > > > Ingo Bürk  于2022年3月4日周五 22:56写道：
> > > > > >
> > > > > > > Congrats, David!
> > > > > > >
> > > > > > > On 04.03.22 12:34, Robert Metzger wrote:
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > On behalf of the PMC, I'm very happy to announce David
> Morávek
> > as a
> > > > > new
> > > > > > > > Flink committer.
> > > > > > > >
> > > > > > > > His first contributions to Flink date back to 2019. He has
> been
> > > > > > > > increasingly active with reviews and driving major
> initiatives
> > in
> > > > the
> > > > > > > > community. David brings valuable experience from being a
> > committer
> > > > in
> > > > > > the
> > > > > > > > Apache Beam project to Flink.
> > > > > > > >
> > > > > > > >
> > > > > > > > Please join me in congratulating David for becoming a Flink
> > > > > committer!
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Robert
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>