[jira] [Created] (FLINK-16279) Per job Yarn application leak in normal execution mode.
Wenlong Lyu created FLINK-16279:
---
Summary: Per job Yarn application leak in normal execution mode.
Key: FLINK-16279
URL: https://issues.apache.org/jira/browse/FLINK-16279
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Wenlong Lyu
I run a job in yarn per job mode using {{env.executeAsync}}, the job failed but
the yarn cluster didn't be destroyed.
After some research on the code, I found that:
when running in attached mode, MiniDispatcher will neve set {{shutDownfuture}}
before received a request from job client.
{code}
if (executionMode == ClusterEntrypoint.ExecutionMode.NORMAL) {
// terminate the MiniDispatcher once we served the
first JobResult successfully
jobResultFuture.thenAccept((JobResult result) -> {
ApplicationStatus status =
result.getSerializedThrowable().isPresent() ?
ApplicationStatus.FAILED :
ApplicationStatus.SUCCEEDED;
LOG.debug("Shutting down per-job cluster
because someone retrieved the job result.");
shutDownFuture.complete(status);
});
}
{code}
However, when running in async mode(submit job by env.executeAsync), there may
be no request from job client because when a user find that the job is failed
from job client, he may never request the result again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16864) Add idle metrics for Task
Wenlong Lyu created FLINK-16864: --- Summary: Add idle metrics for Task Key: FLINK-16864 URL: https://issues.apache.org/jira/browse/FLINK-16864 Project: Flink Issue Type: Improvement Components: Runtime / Task Reporter: Wenlong Lyu Currently there is no metric for user to measure how busy a task is concretely, which is important for user to decide how to tune a job. We would like to propose adding an IdleTime which measure idle time of a task including the time cost for mail processor to wait for new mail and the time cost in record writer to waiting a new buffer. With the idle time: 1. when a job can not catch up with the speed of data generating, the vertex which idle time is near to zero is the bottle neck of the job. 2. when a job is not busy, idle time can be used to guide user how much he can scale down the job. In addition, measuring idle time can have little impaction on the performance of the job, because when a task is busy, we don't touch the code to measure wait-time in mailbox. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17012) Expose cost of task initialization
Wenlong Lyu created FLINK-17012: --- Summary: Expose cost of task initialization Key: FLINK-17012 URL: https://issues.apache.org/jira/browse/FLINK-17012 Project: Flink Issue Type: Improvement Components: Runtime / Metrics Reporter: Wenlong Lyu Currently a task switches to running before fully initialized, does not take state initialization and operator initialization(#open ) in to account, which may take long time to finish. As a result, there would be a weird phenomenon that all tasks are running but throughput is 0. I think it could be good if we can expose the initialization stage of tasks. What to you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17040) SavepointWriterITCase broken
Wenlong Lyu created FLINK-17040: --- Summary: SavepointWriterITCase broken Key: FLINK-17040 URL: https://issues.apache.org/jira/browse/FLINK-17040 Project: Flink Issue Type: Bug Reporter: Wenlong Lyu I think it is because of the change of flink-16537 which create partition writer in beforeInvoke [~zhijiang] Caused by: java.lang.UnsupportedOperationException: This method should never be called at org.apache.flink.state.api.runtime.SavepointEnvironment.getAllWriters(SavepointEnvironment.java:242) at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:439) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47) at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:433) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:454) at org.apache.flink.state.api.output.BoundedOneInputStreamTaskRunner.mapPartition(BoundedOneInputStreamTaskRunner.java:76) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17109) FlinkPipelineTranslationUtil#getJobGraph doesn't consider Jars and Classpaths in configuration.
Wenlong Lyu created FLINK-17109: --- Summary: FlinkPipelineTranslationUtil#getJobGraph doesn't consider Jars and Classpaths in configuration. Key: FLINK-17109 URL: https://issues.apache.org/jira/browse/FLINK-17109 Project: Flink Issue Type: Improvement Components: Client / Job Submission Reporter: Wenlong Lyu I think this method should be removed, and we should use PipelineExecutorUtils#getJobGraph instead? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17110) Make StreamExecutionEnvironment#configure also affects configuration
Wenlong Lyu created FLINK-17110: --- Summary: Make StreamExecutionEnvironment#configure also affects configuration Key: FLINK-17110 URL: https://issues.apache.org/jira/browse/FLINK-17110 Project: Flink Issue Type: Improvement Reporter: Wenlong Lyu If StreamExecutionEnvironment#configure can also affect the configuration in StreamExecutionEnvironment, we can easily not only add some library jars or classpaths dynamically according to the job we want to run which is quite important for a platform product, but also optimize some runtime configuration in program. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17141) Name of SQL Operator is too long
Wenlong Lyu created FLINK-17141: --- Summary: Name of SQL Operator is too long Key: FLINK-17141 URL: https://issues.apache.org/jira/browse/FLINK-17141 Project: Flink Issue Type: Improvement Reporter: Wenlong Lyu the name of the operator contains the detail logic of the operator, which make it very large when there are a lot of columns. It is a disaster for logging and web ui, also can cost a lot of memory because we use the name widely such as ExecutionVertex and failover message etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17386) Exception in HadoopSecurityContextFactory.createContext while no shaded-hadoop-lib provided.
Wenlong Lyu created FLINK-17386: --- Summary: Exception in HadoopSecurityContextFactory.createContext while no shaded-hadoop-lib provided. Key: FLINK-17386 URL: https://issues.apache.org/jira/browse/FLINK-17386 Project: Flink Issue Type: Bug Reporter: Wenlong Lyu java.io.IOException: Process execution failed due error. Error output:java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.UserGroupInformation\n\tat org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory.createContext(HadoopSecurityContextFactory.java:59)\n\tat org.apache.flink.runtime.security.SecurityUtils.installContext(SecurityUtils.java:92)\n\tat org.apache.flink.runtime.security.SecurityUtils.install(SecurityUtils.java:60)\n\tat org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:964)\n\n\tat com.alibaba.flink.vvr.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:144)\n\tat com.alibaba.flink.vvr.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:126)\n\tat com.alibaba.flink.vvr.VVRCompileTest.runSingleJobCompileCheck(VVRCompileTest.java:173)\n\tat com.alibaba.flink.vvr.VVRCompileTest.lambda$runJobsCompileCheck$0(VVRCompileTest.java:101)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)\n\tat java.lang.Thread.run(Thread.java:834) I think it is because exception throw in the static code block of UserInformation, we should catch Throwable instead of Exception in HadoopSecurityContextFactory#createContext? [~rongrong] what do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-4271) There is no way to set parallelism of operators such as map and window produced by
Wenlong Lyu created FLINK-4271: -- Summary: There is no way to set parallelism of operators such as map and window produced by Key: FLINK-4271 URL: https://issues.apache.org/jira/browse/FLINK-4271 Project: Flink Issue Type: Bug Components: DataStream API Reporter: Wenlong Lyu Currently, CoGroupStreams package the map/keyBy/window operators with a human friendly interface, like: dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the intermediate operators and final window operators can not be accessed by users, and we cannot set attributes of the operators, which make co-group hard to use in production environment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4375) defined rpc protocols provided by job manager
Wenlong Lyu created FLINK-4375: -- Summary: defined rpc protocols provided by job manager Key: FLINK-4375 URL: https://issues.apache.org/jira/browse/FLINK-4375 Project: Flink Issue Type: Sub-task Components: Distributed Coordination Reporter: Wenlong Lyu Assignee: Wenlong Lyu job manager RPC server needs to implement a job control protocol, resource user protocol, task control protocol, 1. job controller: cancelJob, suspendJob, etc. 2. resource user: slotFailed(notify slot failure), slotAvailable(offer slot), etc. 3. task controller: updateTaskState, updateResultPartitionInfo, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4376) implement job manager init procedure
Wenlong Lyu created FLINK-4376: -- Summary: implement job manager init procedure Key: FLINK-4376 URL: https://issues.apache.org/jira/browse/FLINK-4376 Project: Flink Issue Type: Sub-task Components: Cluster Management Reporter: Wenlong Lyu JobManager should be created with a jobgraph and a resource manager gateway, the init procedure may be as follows: 1. init job manager rpc service and related rpc endpoints 2. try to elect as a leader job manager of the given job graph 3. create the slot pool manager with the given resource manager gateway 4. create the scheduler with the slot pool manager 5. create/restore execution graph from the job graph 6. start execution graph scheduling and run the job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4378) Enable RollingSink to custom HDFS client configuration
Wenlong Lyu created FLINK-4378:
--
Summary: Enable RollingSink to custom HDFS client configuration
Key: FLINK-4378
URL: https://issues.apache.org/jira/browse/FLINK-4378
Project: Flink
Issue Type: Improvement
Components: filesystem-connector
Reporter: Wenlong Lyu
Assignee: Wenlong Lyu
Optimizing the configuration of hdfs client in different situation, such as
{{io.file.buffer.size}} can make rolling sink perform better.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4406) Implement job master registration at resource manager
Wenlong Lyu created FLINK-4406: -- Summary: Implement job master registration at resource manager Key: FLINK-4406 URL: https://issues.apache.org/jira/browse/FLINK-4406 Project: Flink Issue Type: Sub-task Components: Cluster Management Reporter: Wenlong Lyu Job Master needs to register to Resource Manager when starting and then watches leadership changes of RM, and trigger re-registration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4414) Remove restriction on RpcService.getAddress
Wenlong Lyu created FLINK-4414:
--
Summary: Remove restriction on RpcService.getAddress
Key: FLINK-4414
URL: https://issues.apache.org/jira/browse/FLINK-4414
Project: Flink
Issue Type: Sub-task
Reporter: Wenlong Lyu
Assignee: Wenlong Lyu
currently {{RpcService}} provide only address of the endpoint, I think rpc
service serve both the endpoint create on it and the remote gateway create on
it, so it is ok to offer the getAddress to all {{RpcGateway}} created on the
rpc service including the server and client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4443) Add support in RpcCompletenessTest for inheritance of RpcGateway and RpcEndpoint
Wenlong Lyu created FLINK-4443:
--
Summary: Add support in RpcCompletenessTest for inheritance of
RpcGateway and RpcEndpoint
Key: FLINK-4443
URL: https://issues.apache.org/jira/browse/FLINK-4443
Project: Flink
Issue Type: Sub-task
Reporter: Wenlong Lyu
Assignee: Wenlong Lyu
RpcCompletenessTest needs to support RpcGateway which is composited by some
basic functions like the example following:
{code:java}
public interface ExecutionStateListener extends RpcGateway {
public void notifyExecutionStateChanges();
}
public interface JobStateListener extends RpcGateway {
public void notifyJobStateChanges();
}
public interface JobWatcher extends ExecutionStateListener, JobStateListener,
RpcGateway {
}
public class JobWatcherEndpoint extends RpcEndpoint {
protected JobWatcherEndpoint(RpcService rpcService) {
super(rpcService);
}
@RpcMethod
public void notifyExecutionStateChanges() {
}
@RpcMethod
public void notifyJobStateChanges() {
}
}
public class AttachedJobClient extends JobWatcherEndpoint {
protected JobClient(RpcService rpcService) {
super(rpcService);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-5284) Make output of bucketing sink compatible with other processing framework like mapreduce
Wenlong Lyu created FLINK-5284: -- Summary: Make output of bucketing sink compatible with other processing framework like mapreduce Key: FLINK-5284 URL: https://issues.apache.org/jira/browse/FLINK-5284 Project: Flink Issue Type: Improvement Components: filesystem-connector Reporter: Wenlong Lyu Assignee: Wenlong Lyu Currently bucketing sink cannot move the in-progress and pending files to final output when the stream finished, and when recovering, the current output file will contain some invalid content, which can only be identified by the file-length meta file. These make the final output of the job incompatible to other processing framework like mapreduce. There are two things to do to solve the problem: 1. add direct output option to bucketing sink, which writes output to the final file, and delete/truncate the some file when fail over. direct output will be quite useful specially for finite stream job, which can enable user to migrate there batch job to streaming, taking advantage of features such as checkpointing. 2. add truncate by copy option to enable bucketing sink to resize output file by copying content valid in current file instead of creating a length meta file. truncate by copy will make some more extra IO operation, but can make the output more clean. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-5749) Unset HADOOP_HOME and HADOOP_CONF_DIR from system environment when building flink
Wenlong Lyu created FLINK-5749: -- Summary: Unset HADOOP_HOME and HADOOP_CONF_DIR from system environment when building flink Key: FLINK-5749 URL: https://issues.apache.org/jira/browse/FLINK-5749 Project: Flink Issue Type: Bug Reporter: Wenlong Lyu Assignee: Wenlong Lyu Currently when we are trying to build flink on a machine with HADOOP_HOME environment variable set, Test data will be written to HDFS, instead of local tmp dir which is expected. This will cause tests failed. I suggest unset HADOOP_HOME and HADOOP_CONF_DIR environment variable in pom, to make sure maven run the test cases in the exactly way we want. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5815) Add resource files configuration for Yarn Mode
Wenlong Lyu created FLINK-5815: -- Summary: Add resource files configuration for Yarn Mode Key: FLINK-5815 URL: https://issues.apache.org/jira/browse/FLINK-5815 Project: Flink Issue Type: Improvement Components: Client, Distributed Coordination Reporter: Wenlong Lyu Assignee: Wenlong Lyu Currently in flink, when we want to setup a resource file to distributed cache, we need to make the file accessible remotely by a url, which is often difficult to maintain a service like that. What's more, when we want do add some extra jar files to job classpath, we need to copy the jar files to blob server when submitting the jobgraph. In yarn, especially in flip-6, the blob server is not running yet when we try to start a flink job. Yarn has a efficient distributed cache implementation for application running on it, what's more we can be easily share the files stored in hdfs in different application by distributed cache without extra IO operations. I suggest to introduce -yfiles, -ylibjars -yarchives options to FlinkYarnCLI to enable yarn user setup their job resource files by yarn distributed cache. The options is compatible with what is used in mapreduce, which make it easy to use for yarn user who generally has experience on using mapreduce. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5817) Fix test concurrent execution failure by test dir conflicts.
Wenlong Lyu created FLINK-5817: -- Summary: Fix test concurrent execution failure by test dir conflicts. Key: FLINK-5817 URL: https://issues.apache.org/jira/browse/FLINK-5817 Project: Flink Issue Type: Bug Reporter: Wenlong Lyu Assignee: Wenlong Lyu Currently when different users build flink on the same machine, failure may happen because some test utilities create test file using the fixed name, which will cause file access failing when different user processing the same file at the same time. We have found errors from AbstractTestBase, IOManagerTest, FileCacheTest. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5989) Protobuf in akka needs to be shaded
Wenlong Lyu created FLINK-5989:
--
Summary: Protobuf in akka needs to be shaded
Key: FLINK-5989
URL: https://issues.apache.org/jira/browse/FLINK-5989
Project: Flink
Issue Type: Improvement
Components: Build System
Reporter: Wenlong Lyu
Currently akka introduces dependency on protobuf, which is a common jar used in
many systems, I think we need to use a shaded akka like what we do in
dependency on hadoop to avoid version conflicts with user code.
{code}
[INFO] +- com.data-artisans:flakka-actor_2.10:jar:2.3-custom:compile
[INFO] | \- com.typesafe:config:jar:1.2.1:compile
[INFO] +- com.data-artisans:flakka-remote_2.10:jar:2.3-custom:compile
[INFO] | +- io.netty:netty:jar:3.8.0.Final:compile
[INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] | \- org.uncommons.maths:uncommons-maths:jar:1.2.2a:compile
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6162) Fix bug in ByteArrayOutputStreamWithPos#setPosition
Wenlong Lyu created FLINK-6162:
--
Summary: Fix bug in ByteArrayOutputStreamWithPos#setPosition
Key: FLINK-6162
URL: https://issues.apache.org/jira/browse/FLINK-6162
Project: Flink
Issue Type: Bug
Components: Core
Reporter: Wenlong Lyu
Assignee: Wenlong Lyu
Currently the precondition check in setPosition will fail when the buffer is
full:
{{Preconditions.checkArgument(position < getEndPosition(), "Position out of
bounds.");}}
We should allow the expected position to be the end position
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-20736) Separate the implementation of BatchExecLimit and StreamExecLimit
Wenlong Lyu created FLINK-20736: --- Summary: Separate the implementation of BatchExecLimit and StreamExecLimit Key: FLINK-20736 URL: https://issues.apache.org/jira/browse/FLINK-20736 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20766) Separate the implementation of stream sort nodes
Wenlong Lyu created FLINK-20766: --- Summary: Separate the implementation of stream sort nodes Key: FLINK-20766 URL: https://issues.apache.org/jira/browse/FLINK-20766 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Wenlong Lyu Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20782) Separate the implementation of BatchExecRank
Wenlong Lyu created FLINK-20782: --- Summary: Separate the implementation of BatchExecRank Key: FLINK-20782 URL: https://issues.apache.org/jira/browse/FLINK-20782 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu Fix For: 1.13.0 separate the ExeNode and PhysicalNode of Rank in batch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20783) Separate the implementation of BatchExec nodes for Join
Wenlong Lyu created FLINK-20783: --- Summary: Separate the implementation of BatchExec nodes for Join Key: FLINK-20783 URL: https://issues.apache.org/jira/browse/FLINK-20783 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20876) Separate the implementation of StreamExecTemporalJoin
Wenlong Lyu created FLINK-20876: --- Summary: Separate the implementation of StreamExecTemporalJoin Key: FLINK-20876 URL: https://issues.apache.org/jira/browse/FLINK-20876 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20925) Separate the implementation of StreamExecLookup and BatchExecLookup
Wenlong Lyu created FLINK-20925: --- Summary: Separate the implementation of StreamExecLookup and BatchExecLookup Key: FLINK-20925 URL: https://issues.apache.org/jira/browse/FLINK-20925 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-26770) Nullable ArrayData should not be Object[]
Wenlong Lyu created FLINK-26770:
---
Summary: Nullable ArrayData should not be Object[]
Key: FLINK-26770
URL: https://issues.apache.org/jira/browse/FLINK-26770
Project: Flink
Issue Type: Bug
Affects Versions: 1.15.0
Reporter: Wenlong Lyu
sql:
"INSERT INTO %s "
+ " (a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r) values ("
+ "1,'dim',cast(20.2007 as
double),false,652482,cast('2020-07-08' as date),'source_test',cast('2020-07-10
16:28:07.737' as timestamp),"
+ "cast(8.58965 as float),cast(ARRAY [464,98661,32489] as
array),cast(ARRAY [8589934592,8589934593,8589934594] as array),"
+ "ARRAY[cast(8.58967 as float),cast(96.4667 as
float),cast(9345.16 as float)], ARRAY [cast(587897.4646746 as
double),cast(792343.646446 as double),cast(76.46464 as double)],"
+ "cast(ARRAY [true,true,false,true] as
array),cast(ARRAY ['monday','saturday','sunday'] as
array),true,cast(8119.21 as numeric(6,2)), cast('2020-07-10
16:28:07.737' as timestamp)"
+ ")";
error:
Caused by: java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to
[Ljava.lang.Integer;
at
org.apache.flink.table.data.GenericArrayData.toIntArray(GenericArrayData.java:297)
~[flink-table-common-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
related codegen result:
objArray$81 = new Object[result$76.size()];
for ( i$82 = 0; i$82 < result$76.size(); i$82++) {
if (!result$76.isNullAt(i$82)) {
objArray$81[i$82] = result$76.getBoolean(i$82);
cause:
ArrayToArrayCastRule#arrayElementType use Object when a column is
nullable, but GenericArrayData only accepts array with specified types, like
Integer[], I think we should follow CodeGenUtils#boxedTypeTermForType
[~slinkydeveloper]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-25070) FLIP-195: Improve the name and structure of vertex and operator name for job
Wenlong Lyu created FLINK-25070: --- Summary: FLIP-195: Improve the name and structure of vertex and operator name for job Key: FLINK-25070 URL: https://issues.apache.org/jira/browse/FLINK-25070 Project: Flink Issue Type: Improvement Components: API / DataStream, Runtime / Web Frontend, Table SQL / Runtime Reporter: Wenlong Lyu this is an umbrella issue tracking the improvement of operator/vertex names in flink -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25072) Introduce description for operator
Wenlong Lyu created FLINK-25072: --- Summary: Introduce description for operator Key: FLINK-25072 URL: https://issues.apache.org/jira/browse/FLINK-25072 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25073) Introduce Tree Mode
Wenlong Lyu created FLINK-25073: --- Summary: Introduce Tree Mode Key: FLINK-25073 URL: https://issues.apache.org/jira/browse/FLINK-25073 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25074) Simplify name of window operators in DS by moving details to description
Wenlong Lyu created FLINK-25074: --- Summary: Simplify name of window operators in DS by moving details to description Key: FLINK-25074 URL: https://issues.apache.org/jira/browse/FLINK-25074 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25076) Simplify name of SQL operators
Wenlong Lyu created FLINK-25076: --- Summary: Simplify name of SQL operators Key: FLINK-25076 URL: https://issues.apache.org/jira/browse/FLINK-25076 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25118) Add vertex index as prefix in vertex name
Wenlong Lyu created FLINK-25118: --- Summary: Add vertex index as prefix in vertex name Key: FLINK-25118 URL: https://issues.apache.org/jira/browse/FLINK-25118 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25123) Improve expression description in SQL operator
Wenlong Lyu created FLINK-25123: --- Summary: Improve expression description in SQL operator Key: FLINK-25123 URL: https://issues.apache.org/jira/browse/FLINK-25123 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Wenlong Lyu -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25198) add document about how to debug with the name and description
Wenlong Lyu created FLINK-25198: --- Summary: add document about how to debug with the name and description Key: FLINK-25198 URL: https://issues.apache.org/jira/browse/FLINK-25198 Project: Flink Issue Type: Sub-task Components: Documentation Reporter: Wenlong Lyu Fix For: 1.15.0 the doc could in the debugging section https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/debugging/debugging_event_time/ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25600) Support new statement set syntax in sql client and update docs
Wenlong Lyu created FLINK-25600: --- Summary: Support new statement set syntax in sql client and update docs Key: FLINK-25600 URL: https://issues.apache.org/jira/browse/FLINK-25600 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu this is a follow up of FLINK-25392, to finish adding the new statement set: 1. the new statement set need multi line parsing support in sql client, which is not supported currently: execute statement set begin insert xxx; insert xxx; end; 2. we need to update the doc to introduce the new syntax -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-21010) Separete the implementation of StreamExecJoin
Wenlong Lyu created FLINK-21010: --- Summary: Separete the implementation of StreamExecJoin Key: FLINK-21010 URL: https://issues.apache.org/jira/browse/FLINK-21010 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Wenlong Lyu Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21011) Separete the implementation of StreamExecJoin
Wenlong Lyu created FLINK-21011: --- Summary: Separete the implementation of StreamExecJoin Key: FLINK-21011 URL: https://issues.apache.org/jira/browse/FLINK-21011 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Wenlong Lyu Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21704) Support StreamExecChangelogNormalize json serde
Wenlong Lyu created FLINK-21704: --- Summary: Support StreamExecChangelogNormalize json serde Key: FLINK-21704 URL: https://issues.apache.org/jira/browse/FLINK-21704 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21785) Support StreamExecCorrelate json serde
Wenlong Lyu created FLINK-21785: --- Summary: Support StreamExecCorrelate json serde Key: FLINK-21785 URL: https://issues.apache.org/jira/browse/FLINK-21785 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21805) Support StreamExecRank and StreamExecSortLimit Json ser/de
Wenlong Lyu created FLINK-21805: --- Summary: Support StreamExecRank and StreamExecSortLimit Json ser/de Key: FLINK-21805 URL: https://issues.apache.org/jira/browse/FLINK-21805 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21813) Support StreamExecOverAggregate json ser/de
Wenlong Lyu created FLINK-21813: --- Summary: Support StreamExecOverAggregate json ser/de Key: FLINK-21813 URL: https://issues.apache.org/jira/browse/FLINK-21813 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21815) Support StreamExecUnion json ser/de
Wenlong Lyu created FLINK-21815: --- Summary: Support StreamExecUnion json ser/de Key: FLINK-21815 URL: https://issues.apache.org/jira/browse/FLINK-21815 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21816) Support StreamExecGroupTableAggregate json ser/de
Wenlong Lyu created FLINK-21816: --- Summary: Support StreamExecGroupTableAggregate json ser/de Key: FLINK-21816 URL: https://issues.apache.org/jira/browse/FLINK-21816 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21843) Support StreamExecGroupWindowAggregate json ser/de
Wenlong Lyu created FLINK-21843: --- Summary: Support StreamExecGroupWindowAggregate json ser/de Key: FLINK-21843 URL: https://issues.apache.org/jira/browse/FLINK-21843 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21870) RelDataType of TableFunction different in local and CI
Wenlong Lyu created FLINK-21870: --- Summary: RelDataType of TableFunction different in local and CI Key: FLINK-21870 URL: https://issues.apache.org/jira/browse/FLINK-21870 Project: Flink Issue Type: Sub-task Reporter: Wenlong Lyu Locally the StructKind of RelDataType is None, however in CI environment, the StructKind is FULL_QUOLIFIED, see the case in CorrelateJsonPlanTest#testRegisterByClass -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22020) Reflections ClassNotFound when trying to deserialize a json plan in deployment envrionment
Wenlong Lyu created FLINK-22020: --- Summary: Reflections ClassNotFound when trying to deserialize a json plan in deployment envrionment Key: FLINK-22020 URL: https://issues.apache.org/jira/browse/FLINK-22020 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.13.0 Reporter: Wenlong Lyu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-8200) RocksDBAsyncSnapshotTest should use temp fold instead of fold with fixed name
Wenlong Lyu created FLINK-8200: -- Summary: RocksDBAsyncSnapshotTest should use temp fold instead of fold with fixed name Key: FLINK-8200 URL: https://issues.apache.org/jira/browse/FLINK-8200 Project: Flink Issue Type: Bug Reporter: Wenlong Lyu The following case failed when different user run the test in the same machine. Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.226 sec <<< FAILURE! - in org.apache.flink.contrib.streaming.state.RocksDBAsyncSnapshotTest testCleanupOfSnapshotsInFailureCase(org.apache.flink.contrib.streaming.state.RocksDBAsyncSnapshotTest) Time elapsed: 0.023 sec <<< ERROR! java.io.IOException: No local storage directories available. Local DB files directory 'file:/tmp/foobar' does not exist and cannot be created. at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.lazyInitializeForJob(RocksDBStateBackend.java:251) at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:300) at org.apache.flink.contrib.streaming.state.RocksDBAsyncSnapshotTest.testCleanupOfSnapshotsInFailureCase(RocksDBAsyncSnapshotTest.java:338) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8201) YarnResourceManagerTest causes license checking failure
Wenlong Lyu created FLINK-8201:
--
Summary: YarnResourceManagerTest causes license checking failure
Key: FLINK-8201
URL: https://issues.apache.org/jira/browse/FLINK-8201
Project: Flink
Issue Type: Bug
Reporter: Wenlong Lyu
YarnResourceManagerTest generates a temporary taskmanager config file in
flink-yarn module
root folder and never clear it, which makes license checking fail when we run
{{mvn clean verify}} multiple times in the same source folder.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
