[jira] [Created] (FLINK-36380) Cannot determine simple type name "org"
mumu created FLINK-36380: Summary: Cannot determine simple type name "org" Key: FLINK-36380 URL: https://issues.apache.org/jira/browse/FLINK-36380 Project: Flink Issue Type: Bug Reporter: mumu Flink SQL task upgraded from 1.13.1 to 1.18.1,with the following error message : kafka to kafka {code:java} java.lang.RuntimeException: Could not instantiate generated class 'WatermarkGenerator$5' at org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:74) ~[flink-table-runtime-1.18.1.jar:1.18.1] at org.apache.flink.table.runtime.generated.GeneratedWatermarkGeneratorSupplier.createWatermarkGenerator(GeneratedWatermarkGeneratorSupplier.java:69) ~[flink-table-runtime-1.18.1.jar:1.18.1] at org.apache.flink.streaming.api.operators.source.ProgressiveTimestampsAndWatermarks.createMainOutput(ProgressiveTimestampsAndWatermarks.java:109) ~[flink-dist_2.12-1.18.1.jar:1.18.1] at org.apache.flink.streaming.api.operators.SourceOperator.initializeMainOutput(SourceOperator.java:462) ~[flink-dist_2.12-1.18.1.jar:1.18.1] at org.apache.flink.streaming.api.operators.SourceOperator.emitNextNotReading(SourceOperator.java:438) ~[flink-dist_2.12-1.18.1.jar:1.18.1] at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:414) ~[flink-dist_2.12-1.18.1.jar:1.18.1] at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68) ~[flink-dist_2.12-1.18.1.jar:1.18.1] at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist_2.12-1.18.1.jar:1.18.1] at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:562) ~[flink-dist_2.12-1.18.1.jar:1.18.1] at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) ~[flink-dist_2.12-1.18.1.jar:1.18.1] at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:858) ~[flink-dist_2.12-1.18.1.jar:1.18.1] at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:807) ~[flink-dist_2.12-1.18.1.jar:1.18.1] at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970) ~[flink-dist_2.12-1.18.1.jar:1.18.1] at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:949) ~[flink-dist_2.12-1.18.1.jar:1.18.1] at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763) ~[flink-dist_2.12-1.18.1.jar:1.18.1] at org.apache.flink.runtime.taskmanager.Task.run(Task.java:579) ~[flink-dist_2.12-1.18.1.jar:1.18.1] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_66] Caused by: org.apache.flink.util.FlinkRuntimeException: org.apache.flink.api.common.InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue. at org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:94) ~[flink-table-runtime-1.18.1.jar:1.18.1] at org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:101) ~[flink-table-runtime-1.18.1.jar:1.18.1] at org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:68) ~[flink-table-runtime-1.18.1.jar:1.18.1] ... 16 more Caused by: org.apache.flink.shaded.guava31.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.flink.api.common.InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue. at org.apache.flink.shaded.guava31.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2055) ~[flink-cdc-pipeline-connector-mysql-3.0.1.jar:3.0.1] at org.apache.flink.shaded.guava31.com.google.common.cache.LocalCache.get(LocalCache.java:3966) ~[flink-cdc-pipeline-connector-mysql-3.0.1.jar:3.0.1] at org.apache.flink.shaded.guava31.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4863) ~[flink-cdc-pipeline-connector-mysql-3.0.1.jar:3.0.1] at org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:92) ~[flink-table-runtime-1.18.1.jar:1.18.1] at org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:101) ~[flink-table-runtime-1.18.1.jar:1.18.1] at org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:68) ~[flink-table-runtime-1.18.1.jar:1.18.1] ... 16 more Caused by: org.apache.flink.api.common.InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue. at org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:107) ~[flink-table-runtime-1.18.1.jar:1.18.1] at org.apache.flink.table.runtime.generated.CompileUtils.lambda$compile$0(CompileUtils.java:92) ~[flink-table-runt
[jira] [Created] (FLINK-36378) type extraction problem between java BigDecimal and sql decimal
Jacky Lau created FLINK-36378: - Summary: type extraction problem between java BigDecimal and sql decimal Key: FLINK-36378 URL: https://issues.apache.org/jira/browse/FLINK-36378 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 2.0-preview Reporter: Jacky Lau Fix For: 2.0-preview add following to ValueDataTypeConverterTest of(new BigDecimal("0.000"), DataTypes.DECIMAL(4, 3)) org.apache.flink.table.api.ValidationException: Decimal scale must be between 0 and the precision 1 (both inclusive). spark 0.000 -> decimal(3, 3) calcite 0.000 -> decimal(4, 3) so follow the calcite -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36379) Improve (Global)Committer with UC disabled
Arvid Heise created FLINK-36379: --- Summary: Improve (Global)Committer with UC disabled Key: FLINK-36379 URL: https://issues.apache.org/jira/browse/FLINK-36379 Project: Flink Issue Type: Bug Affects Versions: 2.0-preview Reporter: Arvid Heise Assignee: Arvid Heise FLINK-36287 disabled UC for all inter-sink connections to adhere to the contract of notifyCheckpointCompleted. This allows us to remove some special casing and improve global committer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36381) Remove skip tag in python module caused by deprecating legacy csv source
xuyang created FLINK-36381: -- Summary: Remove skip tag in python module caused by deprecating legacy csv source Key: FLINK-36381 URL: https://issues.apache.org/jira/browse/FLINK-36381 Project: Flink Issue Type: Technical Debt Components: API / Python Reporter: xuyang Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36376) More friendly error or warn message for misconfigured statebackend with async state processing
Zakelly Lan created FLINK-36376: --- Summary: More friendly error or warn message for misconfigured statebackend with async state processing Key: FLINK-36376 URL: https://issues.apache.org/jira/browse/FLINK-36376 Project: Flink Issue Type: Sub-task Reporter: Zakelly Lan Assignee: Zakelly Lan -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36377) Support the use of the LAST_VALUE aggregate function on ROW type data
Yang Li created FLINK-36377: --- Summary: Support the use of the LAST_VALUE aggregate function on ROW type data Key: FLINK-36377 URL: https://issues.apache.org/jira/browse/FLINK-36377 Project: Flink Issue Type: Improvement Components: Runtime / State Backends Reporter: Yang Li h2. Introduction In Flink, after applying a group by, users may use LAST_VALUE to process certain fields to ensure that all fields have corresponding aggregation functions. Currently, LAST_VALUE does not support the ROW type syntax, so users always apply the LAST_VALUE function to each individual field separately, as shown below. SELECT LAST_VALUE(bool_a) AS last_bool_a, LAST_VALUE(int2_b) AS last_int2_b, LAST_VALUE(int4_c) AS last_int4_c, LAST_VALUE(int8_d) AS last_int8_d, LAST_VALUE(float4_e) AS last_float4_e, LAST_VALUE(float4_f) AS last_float4_f, LAST_VALUE(numeric_g) AS last_numeric_g, LAST_VALUE(text_m) AS last_text_m, LAST_VALUE(varchar_p) AS last_varchar_p, date_h FROM source_table GROUP BY date_h If the upstream operator is a retract stream, this approach will lead to redundant StateMap traversal. To facilitate retraction, Flink's internal {{LastValueWithRetractAggFunction}} will store all historical data related to the primary key. When the last value is deleted, it will traverse all keys in the {{orderToValue}} (which maps timestamps to data) and this {{MapView}} is stored in the form of {{{}StateMap{}}}. More {{LAST_VALUE}} functions leads to more times the read and write operations of RocksDB. Therefore, I advocate for handling {{ROW}} types with {{{}LAST_VALUE{}}}, allowing support for all fields with just one {{LAST_VALUE}} function as below. SELECT LAST_VALUE( ROW( bool_a, int2_b, int4_c, int8_d, float4_e, float4_f, numeric_g, text_m, varchar_p ) ) AS row_data, date_h FROM source_table GROUP BY date_h The experiment indicates that applying the {{ROW}} type to the {{LAST_VALUE}} function can improve the processing speed for retract streams, but has no effect on append-only streams. h2. Evaluation: The throughput of jobs was compared based on whether the {{ROW}} type was used in the {{LAST_VALUE}} function, considering both retract and append-only scenarios. h3. Retraction Use a deduplication operator to convert the append-only stream generated by datagen into a retract stream. h4. LAST_VALUE with Separated Fields !https://xiaomi.f.mioffice.cn/space/api/box/stream/download/asynccode/?code=N2UxY2VkZjAwOWM0M2M4ODYxNzMwZjA3ZDYxNTYzOTVfbElkb2JxUkwzY3hHNEZ4TXZYNUxHaEVQdVk3M25mcWZfVG9rZW46Ym94azRqS3VCeXpQSFl1YjVhaVF0NVBMcHhjXzE3MjczMzU0NjE6MTcyNzMzOTA2MV9WNA|width=903,height=236! h4. LAST_VALUE with ROW !https://xiaomi.f.mioffice.cn/space/api/box/stream/download/asynccode/?code=MTNkOTI3OTRhYmU3ZmYxMzE4NDhkOWU1NDkxYzZmNTBfNUtKVGp0b0lOd1MyUVpDbnR0SWJ4aGxQd3QwNTZmZXdfVG9rZW46Ym94azR1UTVZUGZFdnFXMktBdWdHMjd6cHVmXzE3MjczMzU0NjE6MTcyNzMzOTA2MV9WNA|width=894,height=233! h4. Summary he two jobs show little difference in throughput (Row 4817: Mean 1808). Through flame graph analysis, applying the ROW type to the LAST_VALUE function reduces the consumption of the aggregate function calls to accumulate, with CPU usage for accumulate being (ROW 20.02%: Separated 66.98%). LastValueWithRetractAccumulator uses MapState storage MapView. Therefore, updating the LastValueWithRetractAccumulator requires reading from or writing to RocksDB. h3. AppendOnly h4. LAST_VALUE with Separated Fields !https://xiaomi.f.mioffice.cn/space/api/box/stream/download/asynccode/?code=OGExYjk0NTQ3NDNmZDQ4OTI5ODhiZTliM2QzODM5YmVfcE1LQTZSRElaRkZpNmhsRkxndzhSRGxQdFJrTFdZQ0lfVG9rZW46Ym94azRVYWlsWU5UTmJ6VHhRTmRaTnJHRHBmXzE3MjczMzU0NjE6MTcyNzMzOTA2MV9WNA|width=870,height=228! h4. LAST_VALUE with ROW !https://xiaomi.f.mioffice.cn/space/api/box/stream/download/asynccode/?code=N2JkZjFiMDZjODU4MmEyNGNmNjgzNzQyYzM4YmYzZTlfNUt3VVc1ZmZHeFpoSWRvdm1HYU0yRldxcGtLM1MyY09fVG9rZW46Ym94azRqcTZvNW04NU1KTXpoQUZEa0g4R29mXzE3MjczMzU0NjE6MTcyNzMzOTA2MV9WNA|width=864,height=230! h4. Summary The two jobs show little difference in throughput (Row 13411: Mean 10673). Further examination of the flame graphs for both processes reveals that the bottleneck for both jobs lies in getting {{RocksDBValueState}} which is called by {{{}GroupFunction{}}}. Using {{ROW}} aggregation does not yield significant optimization in this part. I suspect it's because Flink uses RowData to store data from multiple Accumulators, and every time the {{accState}} invokes the {{value}} method, it reads all the Accumulators at the same time. Therefore, the use of ROW optimization might not be very effective. h2. Conclusion # Using ROW type for LAST_VALUE Aggregation can improve the processing speed for retract streams, with effectiveness proportional to
[jira] [Created] (FLINK-36384) Remove dependency flink-java in flink-sql-client module
xuyang created FLINK-36384: -- Summary: Remove dependency flink-java in flink-sql-client module Key: FLINK-36384 URL: https://issues.apache.org/jira/browse/FLINK-36384 Project: Flink Issue Type: Sub-task Reporter: xuyang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36386) Remove dependency flink-java in flink-table-planner-loader and flink-table-planner-loader-bundle module
xuyang created FLINK-36386: -- Summary: Remove dependency flink-java in flink-table-planner-loader and flink-table-planner-loader-bundle module Key: FLINK-36386 URL: https://issues.apache.org/jira/browse/FLINK-36386 Project: Flink Issue Type: Sub-task Reporter: xuyang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36385) Remove dependency flink-java in flink-sql-gateway module
xuyang created FLINK-36385: -- Summary: Remove dependency flink-java in flink-sql-gateway module Key: FLINK-36385 URL: https://issues.apache.org/jira/browse/FLINK-36385 Project: Flink Issue Type: Sub-task Reporter: xuyang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36387) Remove dependency flink-java in flink-table-planner module
xuyang created FLINK-36387: -- Summary: Remove dependency flink-java in flink-table-planner module Key: FLINK-36387 URL: https://issues.apache.org/jira/browse/FLINK-36387 Project: Flink Issue Type: Sub-task Reporter: xuyang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36382) Remove dependency flink-java in flink-table-test-utils module
xuyang created FLINK-36382: -- Summary: Remove dependency flink-java in flink-table-test-utils module Key: FLINK-36382 URL: https://issues.apache.org/jira/browse/FLINK-36382 Project: Flink Issue Type: Sub-task Reporter: xuyang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36383) Remove dependency flink-java in flink-sql-jdbc-driver and flink-sql-jdbc-driver-bundle module
xuyang created FLINK-36383: -- Summary: Remove dependency flink-java in flink-sql-jdbc-driver and flink-sql-jdbc-driver-bundle module Key: FLINK-36383 URL: https://issues.apache.org/jira/browse/FLINK-36383 Project: Flink Issue Type: Sub-task Reporter: xuyang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36388) Truncate failure labels on web interface
Panagiotis Garefalakis created FLINK-36388: -- Summary: Truncate failure labels on web interface Key: FLINK-36388 URL: https://issues.apache.org/jira/browse/FLINK-36388 Project: Flink Issue Type: Bug Affects Versions: 1.19.1 Reporter: Panagiotis Garefalakis Assignee: Panagiotis Garefalakis Failure labels are plain Strings and can end we might end up bloating the UI with very length failure label boxes. We should truncate lengthy labels instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36389) Fix DelegationTokenReceiverRepository to check if Delegation Token is enabled
Archit Goyal created FLINK-36389: Summary: Fix DelegationTokenReceiverRepository to check if Delegation Token is enabled Key: FLINK-36389 URL: https://issues.apache.org/jira/browse/FLINK-36389 Project: Flink Issue Type: Bug Reporter: Archit Goyal {*}Issue{*}: During the initialization of the {{DelegationTokenReceiverRepository}} in the constructor, the {{loadReceivers()}} method is invoked without checking whether delegation tokens are enabled. This leads to the following error in the TaskManager logs: {{}} {code:java} java.lang.IllegalStateException: Delegation token receiver with service name {} has multiple implementations [hadoopfs]{code} {{}} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36396) Remove dependency flink-java in all flink-table-api-xxx module
xuyang created FLINK-36396: -- Summary: Remove dependency flink-java in all flink-table-api-xxx module Key: FLINK-36396 URL: https://issues.apache.org/jira/browse/FLINK-36396 Project: Flink Issue Type: Sub-task Reporter: xuyang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36390) Remove deprecate RichFunction#open(Configuration)
Weijie Guo created FLINK-36390: -- Summary: Remove deprecate RichFunction#open(Configuration) Key: FLINK-36390 URL: https://issues.apache.org/jira/browse/FLINK-36390 Project: Flink Issue Type: Sub-task Components: API / Core Affects Versions: 2.0-preview Reporter: Weijie Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36391) Remove dependency flink-java in flink-table-runtime module
xuyang created FLINK-36391: -- Summary: Remove dependency flink-java in flink-table-runtime module Key: FLINK-36391 URL: https://issues.apache.org/jira/browse/FLINK-36391 Project: Flink Issue Type: Sub-task Reporter: xuyang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36394) Configure Init Metaspace size by default
JieTan created FLINK-36394: -- Summary: Configure Init Metaspace size by default Key: FLINK-36394 URL: https://issues.apache.org/jira/browse/FLINK-36394 Project: Flink Issue Type: Bug Components: API / Core Affects Versions: 1.17.2, 1.16.0 Reporter: JieTan Attachments: image.png We should set the size of the Init JVM Metaspace to a sane default, like {{{}-XX:MetaspaceSize=128m{}}}. When a Flink job starts, it needs to load class metadata information, including class structure information, method information, and field information. As a result, the JVM frequently performs FullGC. Sometimes, the Metadata GC Threshold (Full GC) is 3.7s long before the Akka System is started. ``` bash 2024-07-23T20:08:40.554+0800: 1.801: [Full GC (Metadata GC Threshold) 2024-07-23T20:08:40.554+0800: 1.801: [Tenured: 0K->14029K(966656K), 0.0336205 secs] 108273K->14029K(1401664K), [Metaspace: 20594K->20594K(1067008K)], 0.0337494 secs] [Times: user=0.02 sys=0.00, real=0.04 secs] 2024-07-23T20:08:42.052+0800: 3.300: [Full GC (Metadata GC Threshold) 2024-07-23T20:08:42.052+0800: 3.300: [Tenured: 14029K->28979K(966656K), 0.0503497 secs] 184494K->28979K(1401664K), [Metaspace: 34384K->34384K(1079296K)], 0.0505108 secs] [Times: user=0.04 sys=0.00, real=0.05 secs] 2024-07-23T20:08:45.064+0800: 6.312: [Full GC (Metadata GC Threshold) 2024-07-23T20:08:45.064+0800: 6.312: [Tenured: 28979K->54550K(966656K), 0.0876744 secs] 248437K->54550K(1401664K), [Metaspace: 57015K->57015K(1099776K)], 0.0879053 secs] [Times: user=0.07 sys=0.01, real=0.09 secs] ``` The final Metaspace size takes up 100MB+. *Solution:* {*}I{*}f we configure Init Metaspace size by default. As a result, Flink job Metaspace FullGC reduced. ``` bash OpenJDK 64-Bit Server VM (25.362-b09) for linux-amd64 JRE (1.8.0_362-ByteOpenJDK-b09), built on Feb 20 2023 09:42:31 by "root" with gcc 8.3.0 Memory: 4k page, physical 4018480k(137636k free), swap 0k(0k free) CommandLine flags: -XX:CompressedClassSpaceSize=260046848 -XX:GCLogFileSize=104857600 -XX:InitialHeapSize=3368026112 -XX:MaxHeapSize=3368026112 -XX:MaxMetaspaceSize=268435456 -XX:MetaspaceSize=134217728 -XX:NumberOfGCLogFiles=5 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseGCLogFileRotation ``` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36392) Varying the Java version in the Operator CI has no effect
Sam Barker created FLINK-36392: -- Summary: Varying the Java version in the Operator CI has no effect Key: FLINK-36392 URL: https://issues.apache.org/jira/browse/FLINK-36392 Project: Flink Issue Type: Bug Components: Kubernetes Operator Reporter: Sam Barker [FLINK-33471|https://issues.apache.org/jira/browse/FLINK-33471] & [FLINK-33359|https://issues.apache.org/jira/browse/FLINK-33359] both modified the Kubernetes Operator edge to edge test suite to support to support a variety of java versions in GitHub CI. However as far as I can tell this well intentioned move hasn't actually had the desired effect. As the JVM which actually runs the maven build of the operator is executed within the context of a Docker container based on [temurin-11|https://github.com/apache/flink-kubernetes-operator/blob/d946f3f9f3a7f12098cd82db2545de7c89e220ff/Dockerfile#L19]. Therefore all the tests are actually executed by a java operator built and running under JDK/JRE 11. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36393) Flink Operator uses out of date GitHub actions.
Sam Barker created FLINK-36393: -- Summary: Flink Operator uses out of date GitHub actions. Key: FLINK-36393 URL: https://issues.apache.org/jira/browse/FLINK-36393 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Reporter: Sam Barker The operator test suite if full of warnings about using out of date github actions based on node 12 an example [picked at random|https://github.com/apache/flink-kubernetes-operator/actions/runs/10915107524] The problematic actions are: * {{actions/checkout@v2}} * {{actions/setup-java@v2}} * {{actions/cache@v3}} -- This message was sent by Atlassian Jira (v8.20.10#820010)