[jira] [Created] (FLINK-37009) Migrate PruneAggregateCallRule
Jacky Lau created FLINK-37009: - Summary: Migrate PruneAggregateCallRule Key: FLINK-37009 URL: https://issues.apache.org/jira/browse/FLINK-37009 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Affects Versions: 2.0.0 Reporter: Jacky Lau Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-36974]support overwrite flink config by command line [flink-cdc]
yuxiqian commented on code in PR #3823:
URL: https://github.com/apache/flink-cdc/pull/3823#discussion_r1903736412
##
flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/CliFrontendTest.java:
##
@@ -145,6 +146,31 @@ void testPipelineExecuting() throws Exception {
assertThat(executionInfo.getDescription()).isEqualTo("fake-description");
}
+@Test
+void testPipelineExecutingWithFlinkConfig() throws Exception {
+CliExecutor executor =
+createExecutor(
+pipelineDef(),
+"--flink-home",
+flinkHome(),
+"--global-config",
+globalPipelineConfig(),
+"--flink-conf",
+"execution.target=yarn-session",
+"--flink-conf",
+"rest.bind-port=42689",
+"-fc",
+"yarn.application.id=application_1714009558476_3563",
+"-fc",
+"rest.bind-address=10.1.140.140");
+Map configMap = executor.getFlinkConfig().toMap();
+
assertThat(configMap.get("execution.target")).isEqualTo("yarn-session");
+assertThat(configMap.get("rest.bind-port")).isEqualTo("42689");
+assertThat(configMap.get("yarn.application.id"))
+.isEqualTo("application_1714009558476_3563");
+
assertThat(configMap.get("rest.bind-address")).isEqualTo("10.1.140.140");
Review Comment:
```suggestion
assertThat(configMap)
.containsEntry("execution.target", "yarn-session")
.containsEntry("rest.bind-port", "42689")
.containsEntry("yarn.application.id",
"application_1714009558476_3563")
.containsEntry("rest.bind-address", "10.1.140.140");
```
##
flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/CliFrontendTest.java:
##
@@ -145,6 +146,31 @@ void testPipelineExecuting() throws Exception {
assertThat(executionInfo.getDescription()).isEqualTo("fake-description");
}
+@Test
+void testPipelineExecutingWithFlinkConfig() throws Exception {
Review Comment:
Please also add tests for malformed arguments (like when `=` is missing, or
`=` is part of value, etc.)
##
flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java:
##
@@ -114,6 +120,21 @@ static CliExecutor createExecutor(CommandLine commandLine)
throws Exception {
savepointSettings);
}
+private static void overrideFlinkConfiguration(
+Configuration flinkConfig, CommandLine commandLine) {
+String[] flinkConfigs = commandLine.getOptionValues(FLINK_CONFIG);
+if (flinkConfigs != null) {
+LOG.info("Find flink config items: {}", String.join(",",
flinkConfigs));
+for (String config : flinkConfigs) {
+String key = config.split("=")[0].trim();
+String value = config.split("=")[1].trim();
Review Comment:
We can split just once and verify the format first.
##
flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java:
##
@@ -114,6 +120,21 @@ static CliExecutor createExecutor(CommandLine commandLine)
throws Exception {
savepointSettings);
}
+private static void overrideFlinkConfiguration(
+Configuration flinkConfig, CommandLine commandLine) {
+String[] flinkConfigs = commandLine.getOptionValues(FLINK_CONFIG);
+if (flinkConfigs != null) {
+LOG.info("Find flink config items: {}", String.join(",",
flinkConfigs));
Review Comment:
```suggestion
LOG.info("Dynamic flink config items found: {}", flinkConfigs);
```
##
flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontendOptions.java:
##
@@ -94,6 +94,15 @@ public class CliFrontendOptions {
+ "program that was part of the program
when the savepoint was triggered.")
.build();
+public static final Option FLINK_CONFIG =
+Option.builder("fc")
Review Comment:
[SQL
Client](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sqlclient/#sql-client-startup-options)
provides `-D` to pass extra Flink options dynamically. Maybe we can follow the
same naming here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-36974) support overwrite flink config by command line
[ https://issues.apache.org/jira/browse/FLINK-36974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-36974: --- Labels: pull-request-available (was: ) > support overwrite flink config by command line > --- > > Key: FLINK-36974 > URL: https://issues.apache.org/jira/browse/FLINK-36974 > Project: Flink > Issue Type: New Feature > Components: Flink CDC >Affects Versions: cdc-3.3.0 >Reporter: hiliuxg >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.3.0 > > > Support overwrite flink config in the command line, for example: > `bin/flink-cdc.sh1732864461789.yaml --flink-conf > execution.checkpointing.interval=10min --flink-conf rest.bind-port=42689 > --flink-conf yarn.application.id=application_1714009558476_3563 --flink-conf > execution.target=yarn-session --flink-conf rest.bind-address=10.5.140.140` > The example provided is used to submit a job to a specified host's YARN > session cluster with specific Flink configurations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34123][FLINK-35068][FLINK-36903] Introduce built-in serialization support for common collection types [flink]
X-czh commented on code in PR #25797: URL: https://github.com/apache/flink/pull/25797#discussion_r1903397742 ## docs/layouts/shortcodes/generated/pipeline_configuration.html: ## @@ -20,6 +20,12 @@ Duration The interval of the automatic watermark emission. Watermarks are used throughout the streaming system to keep track of the progress of time. They are used, for example, for time based windowing. + +pipeline.built-in-collection-types +true +Boolean +If enabled, TypeExtractor will use built-in serializers for Maps, Lists, and Sets, which need to be treated by Kyro otherwise. Review Comment: Make sense, I'll update the description -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-36989][runtime] Fix scheduler benchmark regression caused by ConsumedSubpartitionContext [flink]
zhuzhurk commented on PR #25887: URL: https://github.com/apache/flink/pull/25887#issuecomment-2572026995 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34825][pipeline-connector][mongodb] Add Implementation of DataSource in MongoDB [flink-cdc]
github-actions[bot] commented on PR #3679: URL: https://github.com/apache/flink-cdc/pull/3679#issuecomment-2571801368 This pull request has been automatically marked as stale because it has not had recent activity for 60 days. It will be closed in 30 days if no further activity occurs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-37009][table] Migrate PruneAggregateCallRule to java [flink]
liuyongvs opened a new pull request, #25900: URL: https://github.com/apache/flink/pull/25900 ## What is the purpose of the change The PR migrates PruneAggregateCallRule to java ## Verifying this change This change is already covered by existing tests ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-37009) Migrate PruneAggregateCallRule
[ https://issues.apache.org/jira/browse/FLINK-37009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-37009: --- Labels: pull-request-available (was: ) > Migrate PruneAggregateCallRule > -- > > Key: FLINK-37009 > URL: https://issues.apache.org/jira/browse/FLINK-37009 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 2.0.0 >Reporter: Jacky Lau >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-37009][table] Migrate PruneAggregateCallRule to java [flink]
liuyongvs commented on code in PR #25900:
URL: https://github.com/apache/flink/pull/25900#discussion_r1903784779
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PruneAggregateCallRule.java:
##
@@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.logical;
+
+import org.apache.flink.util.Preconditions;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.Aggregate.Group;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Calc;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.RelFactories;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexProgram;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.runtime.Utilities;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.mapping.Mappings;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/** Planner rule that removes unreferenced AggregateCall from Aggregate. */
+public abstract class PruneAggregateCallRule extends
RelOptRule {
+
+public static final ProjectPruneAggregateCallRule PROJECT_ON_AGGREGATE =
+new ProjectPruneAggregateCallRule();
+public static final CalcPruneAggregateCallRule CALC_ON_AGGREGATE =
+new CalcPruneAggregateCallRule();
+
+protected PruneAggregateCallRule(Class topClass) {
+super(
+operand(topClass, operand(Aggregate.class, any())),
+RelFactories.LOGICAL_BUILDER,
+"PruneAggregateCallRule_" + topClass.getCanonicalName());
+}
+
+protected abstract ImmutableBitSet getInputRefs(T relOnAgg);
+
+@Override
+public boolean matches(RelOptRuleCall call) {
+T relOnAgg = call.rel(0);
+Aggregate agg = call.rel(1);
+if (agg.getGroupType() != Group.SIMPLE
+|| agg.getAggCallList().isEmpty()
+||
+// at least output one column
+(agg.getGroupCount() == 0 && agg.getAggCallList().size() ==
1)) {
+return false;
+}
+ImmutableBitSet inputRefs = getInputRefs(relOnAgg);
+int[] unrefAggCallIndices = getUnrefAggCallIndices(inputRefs, agg);
+return unrefAggCallIndices.length > 0;
+}
+
+private int[] getUnrefAggCallIndices(ImmutableBitSet inputRefs, Aggregate
agg) {
+int groupCount = agg.getGroupCount();
+return IntStream.range(0, agg.getAggCallList().size())
+.filter(index -> !inputRefs.get(groupCount + index))
+.toArray();
+}
+
+@Override
+public void onMatch(RelOptRuleCall call) {
+T relOnAgg = call.rel(0);
+Aggregate agg = call.rel(1);
+ImmutableBitSet inputRefs = getInputRefs(relOnAgg);
+int[] unrefAggCallIndices = getUnrefAggCallIndices(inputRefs, agg);
+Preconditions.checkArgument(unrefAggCallIndices.length > 0,
"requirement failed");
+
+List newAggCalls = new
ArrayList<>(agg.getAggCallList());
+// remove unreferenced AggCall from original aggCalls
+Arrays.stream(unrefAggCallIndices)
+.boxed()
+.sorted(Comparator.reverseOrder())
+.forEach(index -> newAggCalls.remove((int) index));
+
Review Comment:
comments: newAggCalls.remove((int) index) , we need this int cast here.
because it will remove the value instead of removing index when doesn't have
int cast
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
Re: [PR] [FLINK-22091][yarn] Make Flink on YARN honor env.java.home [flink]
X-czh commented on PR #25877: URL: https://github.com/apache/flink/pull/25877#issuecomment-2572349685 Squashed both commits into one -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-36862][table] Implement additional TO_TIMESTAMP_LTZ() functions [flink]
yiyutian1 commented on PR #25763: URL: https://github.com/apache/flink/pull/25763#issuecomment-2572428331 > Thanks for addressing feedback it looks we are approaching > > There is at least one case found showing regression > > ```sql > SELECT to_timestamp_ltz(123, CAST(NULL AS INTEGER)); > ``` > > before PR returns `null` > > after PR it fails on validation such good catch! Fixed it in the new commit. thanks for the thorough checks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-22091][yarn] Make Flink on YARN honor env.java.home [flink]
Samrat002 commented on PR #25877: URL: https://github.com/apache/flink/pull/25877#issuecomment-2572300758 NIT : Squash both the commits into one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-36576][runtime] Improving amount-based data balancing distribution algorithm for DefaultVertexParallelismAndInputInfosDecider [flink]
JunRuiLee commented on code in PR #25552:
URL: https://github.com/apache/flink/pull/25552#discussion_r1903626133
##
flink-runtime/src/main/java/org/apache/flink/streaming/api/graph/StreamEdge.java:
##
@@ -77,6 +77,10 @@ public class StreamEdge implements Serializable {
private final IntermediateDataSetID intermediateDatasetIdToProduce;
+private boolean existInterInputsKeyCorrelation;
+
+private boolean existIntraInputKeyCorrelation;
+
Review Comment:
+1 for `interInputsKeysCorrelated ` and `areInterInputsKeysCorrelated `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-36576][runtime] Improving amount-based data balancing distribution algorithm for DefaultVertexParallelismAndInputInfosDecider [flink]
JunRuiLee commented on code in PR #25552:
URL: https://github.com/apache/flink/pull/25552#discussion_r1903628396
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/BlockingInputInfo.java:
##
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler.adaptivebatch;
+
+import org.apache.flink.runtime.executiongraph.IndexRange;
+import org.apache.flink.runtime.executiongraph.ResultPartitionBytes;
+import org.apache.flink.runtime.jobgraph.IntermediateDataSetID;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+public class BlockingInputInfo implements BlockingResultInfo {
Review Comment:
> And I'm curious about the relationship between this class and
`AllToAllBlockingResultInfo` and `PointwiseBlockingResultInfo`.
>
> It seems that after this PR, `AllToAllBlockingResultInfo` and
`PointwiseBlockingResultInfo` are no longer needed.
>
> Could we just update `AllToAllBlockingResultInfo` and
`PointwiseBlockingResultInfo` or replace them with this class?
I see, result info describes the result, and input info describes the input.
Since a single result can correspond to multiple downstream input (result
reuse), it's still okay to separate them into two categories.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-37008] [runtime-web] Flink UI should show the type of checkpoint (full vs incremental) [flink]
flinkbot commented on PR #25899: URL: https://github.com/apache/flink/pull/25899#issuecomment-2572234932 ## CI report: * 1494ec245bca1ca1f43d7153421033bfa7cbc602 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-36989][runtime] Fix scheduler benchmark regression caused by ConsumedSubpartitionContext [flink]
noorall commented on PR #25887: URL: https://github.com/apache/flink/pull/25887#issuecomment-2572182080 > Thanks @noorall for fixing. LGTM. Could you post the results of the three regressed benchmark cases after the fix? I tested it on my own computer, and here are the test results: | Baseline (ms/op) | Before Fix (ms/op) | After Fix (ms/op) -- | -- | -- | -- SchedulingAndDeployingBenchmarkExecutor.startScheduling(Streaming) | 253.740 ± 38.368 | 666.177 ± 132.292 | 318.120 ± 77.141 DeployingDownstreamTasksInBatchJobBenchmarkExecutor.deployDownstreamTasks(Batch) | 40.937 ± 10.214 | 423.432 ± 103.273 | 52.442 ± 29.368 HandleGlobalFailureAndRestartAllTasksBenchmarkExecutor.handleGlobalFailureAndRestartAllTasks.STREAMING | 334.504 ± 65.792 | 615.319 ± 81.008 | 402.614 ± 73.744 DeployingTasksInStreamingJobBenchmarkExecutor.deployAllTasks.STREAMING | 81.408 ± 21.408 | 455.895 ± 146.687 | 93.725 ± 36.951 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-36989][runtime] Fix scheduler benchmark regression caused by ConsumedSubpartitionContext [flink]
zhuzhurk merged PR #25887: URL: https://github.com/apache/flink/pull/25887 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-36989) Scheduler benchmark regression since Dec.25th
[ https://issues.apache.org/jira/browse/FLINK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu closed FLINK-36989. --- Fix Version/s: 2.0.0 Resolution: Fixed 81f882bcf79770072f3cee9a026c658ed9b04219 > Scheduler benchmark regression since Dec.25th > - > > Key: FLINK-36989 > URL: https://issues.apache.org/jira/browse/FLINK-36989 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Reporter: Zakelly Lan >Assignee: Lei Yang >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > > Four regressions detected since Dec. 25th > Java 17: > [deployAllTasks.STREAMING(Java17)|http://flink-speed.xyz/timeline/#/?exe=10&ben=deployAllTasks.STREAMING&extr=on&quarts=on&equid=off&env=3&revs=200] > baseline=216.11395 current_value=1427.674875 > [deployDownstreamTasks.BATCH(Java17)|http://flink-speed.xyz/timeline/#/?exe=10&ben=deployDownstreamTasks.BATCH&extr=on&quarts=on&equid=off&env=3&revs=200] > baseline=104.424111 current_value=1336.293467 > [handleGlobalFailureAndRestartAllTasks.STREAMING(Java17)|http://flink-speed.xyz/timeline/#/?exe=10&ben=handleGlobalFailureAndRestartAllTasks.STREAMING&extr=on&quarts=on&equid=off&env=3&revs=200] > baseline=888.658946 current_value=1722.576395 > [startScheduling.STREAMING(Java17)|http://flink-speed.xyz/timeline/#/?exe=10&ben=startScheduling.STREAMING&extr=on&quarts=on&equid=off&env=3&revs=200] > baseline=760.369923 current_value=1788.658818 > > Jave 11: > [deployAllTasks.STREAMING(Java11)|http://flink-speed.xyz/timeline/#/?exe=8&ben=deployAllTasks.STREAMING&extr=on&quarts=on&equid=off&env=3&revs=200] > baseline=213.933949 current_value=1349.547905 > [deployDownstreamTasks.BATCH(Java11)|http://flink-speed.xyz/timeline/#/?exe=8&ben=deployDownstreamTasks.BATCH&extr=on&quarts=on&equid=off&env=3&revs=200] > baseline=96.236896 current_value=1192.864876 > [handleGlobalFailureAndRestartAllTasks.STREAMING(Java11)|http://flink-speed.xyz/timeline/#/?exe=8&ben=handleGlobalFailureAndRestartAllTasks.STREAMING&extr=on&quarts=on&equid=off&env=3&revs=200] > baseline=905.811447 current_value=1790.241272 > [startScheduling.STREAMING(Java11)|http://flink-speed.xyz/timeline/#/?exe=8&ben=startScheduling.STREAMING&extr=on&quarts=on&equid=off&env=3&revs=200] > baseline=824.044028 current_value=1823.469835 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-36993) FLIP-492: Support Query Modifications for Materialized Tables
[ https://issues.apache.org/jira/browse/FLINK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-36993: --- Labels: pull-request-available (was: ) > FLIP-492: Support Query Modifications for Materialized Tables > - > > Key: FLINK-36993 > URL: https://issues.apache.org/jira/browse/FLINK-36993 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API, Table SQL / Gateway >Reporter: Feng Jin >Priority: Major > Labels: pull-request-available > > Umbrella issue for > https://cwiki.apache.org/confluence/display/FLINK/FLIP-492%3A+Support+Query+Modifications+for+Materialized+Tables -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-36993][table] Support ALTER MATERIALIZED TABLE As statement [flink]
hackergin commented on code in PR #25880:
URL: https://github.com/apache/flink/pull/25880#discussion_r1903610990
##
flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/service/materializedtable/MaterializedTableManager.java:
##
@@ -804,6 +808,78 @@ protected static String getRefreshStatement(
return insertStatement.toString();
}
+private ResultFetcher callAlterMaterializedTableAsQueryOperation(
+OperationExecutor operationExecutor,
+OperationHandle handle,
+AlterMaterializedTableAsQueryOperation op) {
+ObjectIdentifier tableIdentifier = op.getTableIdentifier();
+CatalogMaterializedTable materializedTable =
+getCatalogMaterializedTable(operationExecutor,
tableIdentifier);
+
+// 1. suspend the materialized table
+if (CatalogMaterializedTable.RefreshStatus.SUSPENDED
+!= materializedTable.getRefreshStatus()) {
+if (CatalogMaterializedTable.RefreshMode.CONTINUOUS
+== materializedTable.getRefreshMode()) {
+suspendContinuousRefreshJob(
+operationExecutor, handle, tableIdentifier,
materializedTable);
+} else {
+suspendRefreshWorkflow(
+operationExecutor, handle, tableIdentifier,
materializedTable);
+}
+}
+
+// 2. replace query definition and resume the materialized table
+// alter materialized table schema
+operationExecutor.callExecutableOperation(handle, op);
+ResolvedCatalogMaterializedTable updatedMaterializedTable =
+getCatalogMaterializedTable(operationExecutor,
tableIdentifier);
+
+// 3. resume the materialized table
+if (CatalogMaterializedTable.RefreshStatus.SUSPENDED
+!= materializedTable.getRefreshStatus()) {
Review Comment:
The condition here checks the status of the original table. RESUME is needed
only if the original table’s status is not SUSPENDED.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-36993][table] Support ALTER MATERIALIZED TABLE As statement [flink]
hackergin commented on code in PR #25880:
URL: https://github.com/apache/flink/pull/25880#discussion_r1903612300
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/materializedtable/AlterMaterializedTableAsQueryOperation.java:
##
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.operations.materializedtable;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.api.internal.TableResultImpl;
+import org.apache.flink.table.api.internal.TableResultInternal;
+import org.apache.flink.table.catalog.CatalogMaterializedTable;
+import org.apache.flink.table.catalog.Column;
+import org.apache.flink.table.catalog.ObjectIdentifier;
+import org.apache.flink.table.catalog.ResolvedCatalogBaseTable;
+import org.apache.flink.table.catalog.ResolvedCatalogMaterializedTable;
+import org.apache.flink.table.catalog.ResolvedSchema;
+import org.apache.flink.table.catalog.TableChange;
+import org.apache.flink.table.operations.Operation;
+import org.apache.flink.table.operations.OperationUtils;
+import org.apache.flink.table.operations.QueryOperation;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static
org.apache.flink.table.catalog.CatalogBaseTable.TableKind.MATERIALIZED_TABLE;
+
+/** Operation to describe an ALTER MATERIALIZED TABLE AS query operation. */
+@Internal
+public class AlterMaterializedTableAsQueryOperation extends
AlterMaterializedTableOperation {
+
+protected final ObjectIdentifier tableIdentifier;
+
+private final QueryOperation queryOperation;
+
+public AlterMaterializedTableAsQueryOperation(
+ObjectIdentifier tableIdentifier, QueryOperation queryOperation) {
+super(tableIdentifier);
+this.tableIdentifier = tableIdentifier;
+this.queryOperation = queryOperation;
+}
+
+public QueryOperation getQueryOperation() {
+return queryOperation;
+}
+
+@Override
+public TableResultInternal execute(Context ctx) {
+ResolvedCatalogBaseTable resolvedCatalogBaseTable =
+
ctx.getCatalogManager().getTableOrError(tableIdentifier).getResolvedTable();
+if (MATERIALIZED_TABLE != resolvedCatalogBaseTable.getTableKind()) {
+throw new ValidationException(
+String.format(
+"Table %s is not a materialized table, does not
support materialized table related operation.",
+tableIdentifier));
+}
+
+ResolvedCatalogMaterializedTable oldResolvedMaterializedTable =
+(ResolvedCatalogMaterializedTable) resolvedCatalogBaseTable;
+
+// validate new schema and derived origin primary key and watermark
spec
+ResolvedSchema resolvedQuerySchema =
queryOperation.getResolvedSchema();
+ResolvedSchema oldResolvedSchema =
oldResolvedMaterializedTable.getResolvedSchema();
+List tableChanges =
+validateAndExtractNewColumns(oldResolvedSchema,
resolvedQuerySchema).stream()
+.map(TableChange::add)
+.collect(Collectors.toList());
+ResolvedSchema newResolvedSchema =
+new ResolvedSchema(
+resolvedQuerySchema.getColumns(),
+oldResolvedSchema.getWatermarkSpecs(),
+oldResolvedSchema.getPrimaryKey().orElse(null));
+Schema newSchema =
Schema.newBuilder().fromResolvedSchema(newResolvedSchema).build();
+
+// update schema and definition query
+String definitionQuery = queryOperation.asSerializableString();
+CatalogMaterializedTable catalogMaterializedTable =
+oldResolvedMaterializedTable.getOrigin().copy(newSchema,
definitionQuery);
+
+ResolvedCatalogMaterializedTable newResolvedMaterializedTable =
+new ResolvedCatalogMaterializedTable(catalogMaterializedTable,
ne
Re: [PR] [FLINK-36993][table] Support ALTER MATERIALIZED TABLE As statement [flink]
hackergin commented on code in PR #25880:
URL: https://github.com/apache/flink/pull/25880#discussion_r1903611482
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/operations/SqlMaterializedTableNodeToOperationConverterTest.java:
##
@@ -391,6 +392,17 @@ void testAlterMaterializedTableResume() {
.isEqualTo("ALTER MATERIALIZED TABLE builtin.default.mtbl1
RESUME WITH (k1: [v1])");
}
+@Test
+void testAlterMaterializedTableAsQuery() {
Review Comment:
Relevant unit tests have been added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-36576][runtime] Improving amount-based data balancing distribution algorithm for DefaultVertexParallelismAndInputInfosDecider [flink]
zhuzhurk commented on code in PR #25552:
URL: https://github.com/apache/flink/pull/25552#discussion_r1903612414
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultVertexParallelismAndInputInfosDecider.java:
##
@@ -128,12 +128,8 @@ public ParallelismAndInputInfos
decideParallelismAndInputInfosForVertex(
&& vertexMaxParallelism >= vertexMinParallelism);
if (consumedResults.isEmpty()) {
-// source job vertex
-int parallelism =
-vertexInitialParallelism > 0
-? vertexInitialParallelism
-: computeSourceParallelismUpperBound(jobVertexId,
vertexMaxParallelism);
-return new ParallelismAndInputInfos(parallelism,
Collections.emptyMap());
+return decideParallelismAndInputInfosForNonSource(
Review Comment:
This is the handling for sources. I mean for the handling of non-sources.
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultVertexParallelismAndInputInfosDecider.java:
##
@@ -181,18 +179,18 @@ public ParallelismAndInputInfos
decideParallelismAndInputInfosForVertex(
// we need to derive parallelism separately for each input.
//
// In the following cases, we need to reset min parallelism and max
parallelism to ensure
-// that the decide parallelism for all inputs is consistent :
+// that the decided parallelism for all inputs is consistent :
// 1. Vertex has a specified parallelism
// 2. There are edges that don't need to follow intergroup constraint
if (vertexInitialParallelism > 0 ||
inputsGroupByInterCorrelation.containsKey(false)) {
minParallelism = parallelism;
maxParallelism = parallelism;
}
-Map vertexInputInfoMap =
new HashMap<>();
+Map vertexInputInfos = new
HashMap<>();
if (inputsGroupByInterCorrelation.containsKey(true)) {
Review Comment:
It looks a bit weird to me. Because inter/intra correlations and
all-to-all/point-wise connections are not directly related.
Looks to me it is based on some fragile assumption which can be changed in
the future.
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java:
##
@@ -525,7 +523,17 @@ private void
maybeAggregateSubpartitionBytes(BlockingResultInfo resultInfo) {
&& intermediateResult.areAllConsumerVerticesCreated()
&& intermediateResult.getConsumerVertices().stream()
.map(this::getExecutionJobVertex)
-.allMatch(ExecutionJobVertex::isInitialized)) {
+.allMatch(ExecutionJobVertex::isInitialized)
+&& intermediateResult.getConsumerVertices().stream()
Review Comment:
The method comment is not updated accordingly.
##
flink-runtime/src/main/java/org/apache/flink/streaming/api/graph/StreamEdge.java:
##
@@ -77,6 +77,10 @@ public class StreamEdge implements Serializable {
private final IntermediateDataSetID intermediateDatasetIdToProduce;
+private boolean existInterInputsKeyCorrelation;
+
+private boolean existIntraInputKeyCorrelation;
+
Review Comment:
Although it's not good to have one field and one method both named as
`existInterInputsKeyCorrelation`, it's also a bit weird to directly name it as
`interInputsKeyCorrelation`, given that it is actually not a `correlation`.
Maybe `interInputsKeysCorrelated` and `areInterInputsKeysCorrelated()`?
##
flink-runtime/src/main/java/org/apache/flink/streaming/api/graph/StreamEdge.java:
##
@@ -256,19 +264,22 @@ public String getEdgeId() {
return edgeId;
}
-public boolean existInterInputsKeyCorrelation() {
-return existInterInputsKeyCorrelation;
+private void configureKeyCorrelation(StreamPartitioner partitioner) {
+this.intraInputKeyCorrelation =
+!partitioner.isPointwise() || partitioner instanceof
ForwardPartitioner;
+this.interInputsKeyCorrelation = !partitioner.isPointwise();
}
-public boolean existIntraInputKeyCorrelation() {
-return existIntraInputKeyCorrelation;
+public boolean existInterInputsKeyCorrelation() {
+return interInputsKeyCorrelation;
}
-public void setExistInterInputsKeyCorrelation(boolean
existInterInputsKeyCorrelation) {
-this.existInterInputsKeyCorrelation = existInterInputsKeyCorrelation;
+public boolean existIntraInputKeyCorrelation() {
+return intraInputKeyCorrelation;
}
-public void setExistIntraInputKeyCorrelation(boolean
existIntraInputKeyCorrelation) {
-this.existIntraInputKeyCorrelation = existIntraInputKeyCorrelation;
+public void setIntraInputKeyCorrelation(boolean intraInputK
[jira] [Created] (FLINK-37008) Flink UI should show the type of checkpoint (full vs incremental)
Ryan van Huuksloot created FLINK-37008: -- Summary: Flink UI should show the type of checkpoint (full vs incremental) Key: FLINK-37008 URL: https://issues.apache.org/jira/browse/FLINK-37008 Project: Flink Issue Type: Improvement Components: Runtime / Checkpointing, Runtime / Web Frontend Affects Versions: 2.0-preview Reporter: Ryan van Huuksloot There is no way to tell from the Flink UI if a checkpoint is full or incremental. It would be useful for folks strictly using the UI to have an easy way to see if a checkpoint is a full or incremental checkpoint. The information is available but not set in the correct places to have it exposed in the UI. At the same time we should make it available in the API so in the future it could be used in other contexts like the operator. Community Thread: [https://lists.apache.org/thread/0hnn82jrfog18337n3x56wv8n7rrw2rg] I've opened an example PR. It is lacking sophistication but hopefully it starts the conversation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-36999][cdc-source-connectors] When the source field type is de… [flink-cdc]
linjianchang commented on PR #3833: URL: https://github.com/apache/flink-cdc/pull/3833#issuecomment-2572228210 **source oracle table structure：** create table PORTAL.SYSTEM_USER ( sys_user_id NUMBER not null, staff_idNUMBER not null, sys_user_code VARCHAR2(250), passwordVARCHAR2(250), pwd_err_cnt NUMBER not null, pwd_sms_tel NUMBER ) When the pwd_sms_tel field value of the nember type is null, a null pointer exception will be reported. **Exception:** `java.lang.NullPointerException at io.debezium.data.VariableScaleDecimal.toLogical(VariableScaleDecimal.java:103) at com.ververica.cdc.debezium.event.DebeziumSchemaDataTypeInference.inferStruct(DebeziumSchemaDataTypeInference.java:190) at com.ververica.cdc.connectors.oracle.source.OracleSchemaDataTypeInference.inferStruct(OracleSchemaDataTypeInference.java:40) at com.ververica.cdc.debezium.event.DebeziumSchemaDataTypeInference.infer(DebeziumSchemaDataTypeInference.java:81) at com.ververica.cdc.debezium.event.DebeziumSchemaDataTypeInference.infer(DebeziumSchemaDataTypeInference.java:56) at com.ververica.cdc.debezium.event.DebeziumSchemaDataTypeInference.lambda$inferStruct$0(DebeziumSchemaDataTypeInference.java:199) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:546) at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:505) at com.ververica.cdc.debezium.event.DebeziumSchemaDataTypeInference.inferStruct(DebeziumSchemaDataTypeInference.java:200) at com.ververica.cdc.connectors.oracle.source.OracleSchemaDataTypeInference.inferStruct(OracleSchemaDataTypeInference.java:40) at com.ververica.cdc.debezium.event.DebeziumSchemaDataTypeInference.infer(DebeziumSchemaDataTypeInference.java:81) at com.ververica.cdc.debezium.event.DebeziumSchemaDataTypeInference.infer(DebeziumSchemaDataTypeInference.java:56) at com.ververica.cdc.debezium.event.DebeziumEventDeserializationSchema.extractDataRecord(DebeziumEventDeserializationSchema.java:157) at com.ververica.cdc.debezium.event.DebeziumEventDeserializationSchema.extractAfterDataRecord(DebeziumEventDeserializationSchema.java:153) at com.ververica.cdc.debezium.event.DebeziumEventDeserializationSchema.deserializeDataChangeRecord(DebeziumEventDeserializationSchema.java:119) at com.ververica.cdc.debezium.event.SourceRecordEventDeserializer.deserialize(SourceRecordEventDeserializer.java:47) at com.ververica.cdc.debezium.event.DebeziumEventDeserializationSchema.deserialize(DebeziumEventDeserializationSchema.java:106) at com.ververica.cdc.connectors.base.source.reader.IncrementalSourceRecordEmitter.emitElement(IncrementalSourceRecordEmitter.java:172) at com.ververica.cdc.connectors.base.source.reader.IncrementalSourceRecordEmitter.processElement(IncrementalSourceRecordEmitter.java:134) at com.ververica.cdc.connectors.oracle.source.reader.OraclePipelineRecordEmitter.processElement(OraclePipelineRecordEmitter.java:90) at com.ververica.cdc.connectors.base.source.reader.IncrementalSourceRecordEmitter.emitRecord(IncrementalSourceRecordEmitter.java:98) at com.ververica.cdc.connectors.base.source.reader.IncrementalSourceRecordEmitter.emitRecord(IncrementalSourceRecordEmitter.java:59) at org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:143) at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:389) at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68) at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:545) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:836) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:785) at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935) at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728) at org.apache.
[PR] [FLINK-37008] [runtime-web] Flink UI should show the type of checkpoint (full vs incremental) [flink]
ryanvanhuuksloot opened a new pull request, #25899: URL: https://github.com/apache/flink/pull/25899 ## What is the purpose of the change It would be useful for the UI to show if a checkpoint is full or incremental. I'm curious how others would like to expose this in the UI / API but I wanted to do a first pass to get the conversation rolling. This PR is meant to be throwaway when we decided on a path forward. There are currently no tests and likely fails CI   ## Brief change log Adds "full checkpoint flag" to checkpoints to be displayed in the UI / API ## Verifying this change N/A for this draft PR ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (kind of) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-37008) Flink UI should show the type of checkpoint (full vs incremental)
[ https://issues.apache.org/jira/browse/FLINK-37008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-37008: --- Labels: pull-request-available (was: ) > Flink UI should show the type of checkpoint (full vs incremental) > - > > Key: FLINK-37008 > URL: https://issues.apache.org/jira/browse/FLINK-37008 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / Web Frontend >Affects Versions: 2.0-preview >Reporter: Ryan van Huuksloot >Priority: Minor > Labels: pull-request-available > > There is no way to tell from the Flink UI if a checkpoint is full or > incremental. > > It would be useful for folks strictly using the UI to have an easy way to see > if a checkpoint is a full or incremental checkpoint. The information is > available but not set in the correct places to have it exposed in the UI. > > At the same time we should make it available in the API so in the future it > could be used in other contexts like the operator. > > Community Thread: > [https://lists.apache.org/thread/0hnn82jrfog18337n3x56wv8n7rrw2rg] > > I've opened an example PR. It is lacking sophistication but hopefully it > starts the conversation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-36979][rpc] Reverting pekko version bump in Flink 1.20 [flink]
He-Pin commented on PR #25866: URL: https://github.com/apache/flink/pull/25866#issuecomment-2572345937 @XComp Is there any update can share, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-37010) Unify KeyedProcessFunction and the async one
Zakelly Lan created FLINK-37010: --- Summary: Unify KeyedProcessFunction and the async one Key: FLINK-37010 URL: https://issues.apache.org/jira/browse/FLINK-37010 Project: Flink Issue Type: Sub-task Reporter: Zakelly Lan Assignee: Zakelly Lan In FLINK-36120, we introduced `AsyncKeyedProcessFunction` and `AsyncKeyedProcessOperator` to perform processing with async state. However for table runtime, all the functions will extend from `KeyedProcessFunction`, so it is better to unify the base class for function. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-36862][table] Implement additional TO_TIMESTAMP_LTZ() functions [flink]
yiyutian1 commented on code in PR #25763:
URL: https://github.com/apache/flink/pull/25763#discussion_r1903771370
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/TimeFunctionsITCase.java:
##
@@ -853,18 +848,13 @@ private Stream toTimestampLtzTestCases() {
.toInstant(),
TIMESTAMP_LTZ(3).nullable())
.testResult(
-toTimestampLtz($("f3"), literal(3)),
-"TO_TIMESTAMP_LTZ(f3, 3)",
-null,
-TIMESTAMP_LTZ(3).nullable())
-.testResult(
-toTimestampLtz($("f4"), literal(0)),
+toTimestampLtz($("f3"), literal(0)),
"TO_TIMESTAMP_LTZ(-" + Double.MAX_VALUE + ",
0)",
null,
TIMESTAMP_LTZ(3).nullable())
.testResult(
-toTimestampLtz($("f5"), literal(3)),
-"TO_TIMESTAMP_LTZ(f5, 3)",
+toTimestampLtz($("f4"), literal(3)),
Review Comment:
Moved one test case with null input to the end of the stream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [BP-1.20][FLINK-34194] Update CI to Ubuntu 22.04 (Jammy) [flink]
XComp commented on PR #25827: URL: https://github.com/apache/flink/pull/25827#issuecomment-2572493197 > Do you have an idea what could be wrong? The [CI build](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=64350&view=logs&j=d3c25614-fe65-5341-2db6-c20b16adba8d&t=837f3c6c-e20b-47da-861a-e584c6dec5c1&l=13) you are referring to does not contain the fixes of this FLINK-34194 1.20 backport PR, AFAIS (the link can be gathered from the [CI run's main page](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=64350&view=results)). The CI build refers to [this branch](https://github.com/flink-ci/flink/tree/ci_25794_85375dd705b45a3844cf411fb24b82faa96af19d). That branch has [this git history](https://github.com/flink-ci/flink/commits/ci_25794_85375dd705b45a3844cf411fb24b82faa96af19d/) which does not contain any commit of mentioning FLINK-34194 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-36825][table] Introduce FastTop1Function in Rank with Async State API [flink]
Zakelly commented on code in PR #25717:
URL: https://github.com/apache/flink/pull/25717#discussion_r1903721138
##
flink-runtime/src/main/java/org/apache/flink/streaming/api/operators/asyncprocessing/AsyncStateKeyedProcessOperator.java:
##
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.asyncprocessing;
+
+import org.apache.flink.annotation.Internal;
+import
org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.SimpleTimerService;
+import org.apache.flink.streaming.api.TimeDomain;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperator;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.OutputTag;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A {@link StreamOperator} for executing {@link KeyedProcessFunction
KeyedProcessFunctions}.
+ *
+ * This class is nearly identical with {@link KeyedProcessOperator}, but
extending from {@link
+ * AbstractAsyncStateUdfStreamOperator} to integrate with asynchronous state
access. Another
+ * difference is this class is internal.
+ */
+@Internal
+public class AsyncStateKeyedProcessOperator
+extends AbstractAsyncStateUdfStreamOperator>
+implements OneInputStreamOperator, Triggerable {
+
+private static final long serialVersionUID = 1L;
+
+private transient TimestampedCollector collector;
+
+private transient ContextImpl context;
+
+private transient OnTimerContextImpl onTimerContext;
+
+public AsyncStateKeyedProcessOperator(KeyedProcessFunction
function) {
+super(function);
+}
+
+@Override
+public void open() throws Exception {
+super.open();
+collector = new TimestampedCollector<>(output);
Review Comment:
The timestamp is set before the actual processing, which may involves
several async procedure. In the meantime, other record may reset the timestamp,
resulting in previous record may produce downstream event with wrong timestamp.
I have implement a class `TimestampedCollectorWithDeclaredVariable` to keep
the timestamp alongwith record context. The timestamp is independent between
records. Please use `AsyncKeyedProcessOperator` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-33265) Support source parallelism setting for Kafka connector
[ https://issues.apache.org/jira/browse/FLINK-33265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17910011#comment-17910011 ] Zhanghao Chen commented on FLINK-33265: --- [~yunta] [~lincoln.86xy] RocMarshal has already volunteered to review and given the green light. Could you help take another look? > Support source parallelism setting for Kafka connector > -- > > Key: FLINK-33265 > URL: https://issues.apache.org/jira/browse/FLINK-33265 > Project: Flink > Issue Type: New Feature > Components: Connectors / Kafka >Reporter: Zhanghao Chen >Assignee: SuDewei >Priority: Major > Labels: pull-request-available > > The kafka connector adaptation work for [FLIP-367: Support Setting > Parallelism for Table/SQL Sources - Apache Flink - Apache Software > Foundation|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-3154][API] Upgrade from Kryo 2.x to Kryo 5.x. Removed twitter … [flink]
kurtostfeld commented on PR #25896:
URL: https://github.com/apache/flink/pull/25896#issuecomment-2572500238
The following test failure is caused by a test file, external to the main
git repo, that I don't have access to change. It simply needs Class
changed to Class.
```
/tmp/junit13815350649750541837/CustomKryo.java:6: error: CustomKryo is not
abstract and does not override abstract method read(Kryo,Input,Class) in Serializer
public class CustomKryo extends Serializer implements Serializable {
```
This error with table seems like a flaky test that is unrelated to the
changes in this PR:
```
Jan 05 06:56:38 06:56:38.023 [ERROR]
AdaptiveJoinTest.testWithShuffleHashJoin:47 ast ==> expected: <
Jan 05 06:56:38 LogicalProject(a1=[$0], b1=[$1], c1=[$2], d1=[$3], a2=[$4],
b2=[$5], c2=[$6], d2=[$7])
Jan 05 06:56:38 +- LogicalFilter(condition=[=($0, $4)])
Jan 05 06:56:38+- LogicalJoin(condition=[true], joinType=[inner])
Jan 05 06:56:38 :- LogicalTableScan(table=[[default_catalog,
default_database, T1, source: [TestTableSource(a1, b1, c1, d1)]]])
Jan 05 06:56:38 +- LogicalTableScan(table=[[default_catalog,
default_database, T2, source: [TestTableSource(a2, b2, c2, d2)]]])
Jan 05 06:56:38 > but was: <
Jan 05 06:56:38 LogicalProject(a1=[$0], b1=[$1], c1=[$2], d1=[$3], a2=[$4],
b2=[$5], c2=[$6], d2=[$7])
Jan 05 06:56:38 +- LogicalFilter(condition=[=($0, $4)])
Jan 05 06:56:38+- LogicalJoin(condition=[true], joinType=[inner])
Jan 05 06:56:38 :- LogicalTableScan(table=[[default_catalog,
default_database, T1]])
Jan 05 06:56:38 +- LogicalTableScan(table=[[default_catalog,
default_database, T2]])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33817][flink-protobuf] Set ReadDefaultValues=False by default in proto3 for performance improvement [flink]
ikstewa commented on PR #24035: URL: https://github.com/apache/flink/pull/24035#issuecomment-2571733258 @sharath1709 I'm trying to follow up on this change to understand why we do not support optional primitive types. I believe proto3 has support for field presence on primitives when they have the `optional` keyword. Do you have more details on _why_ we specifically do not support primitives? > Additionally, We need to be careful to check for field presence only on non-primitive types if ReadDefaultValues is false and version used is Proto3. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [WIP][FLINK-36181] Use Java 17 by default [flink]
MartijnVisser opened a new pull request, #25898: URL: https://github.com/apache/flink/pull/25898 ## What is the purpose of the change This PR is a Work In Progress, and will change Flink to use Java 17 by default. ## Brief change log ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): don't know - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? docs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [WIP][FLINK-36181] Use Java 17 by default [flink]
flinkbot commented on PR #25898: URL: https://github.com/apache/flink/pull/25898#issuecomment-2571590347 ## CI report: * a0684fd6aa67d81c3e117622b82495fe6c08fe41 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-37007) FLIP-494: Add missing createTable/createView methods to TableEnvironment
Sergey Nuyanzin created FLINK-37007: --- Summary: FLIP-494: Add missing createTable/createView methods to TableEnvironment Key: FLINK-37007 URL: https://issues.apache.org/jira/browse/FLINK-37007 Project: Flink Issue Type: New Feature Components: Table SQL / API Reporter: Sergey Nuyanzin Assignee: Sergey Nuyanzin https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760466 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-30782][build] Use https for schemaLocations [flink]
snuyanzin closed pull request #21753: [FLINK-30782][build] Use https for schemaLocations URL: https://github.com/apache/flink/pull/21753 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-36593) Upgrade io.airlift:aircompressor to mitigate CVE
[ https://issues.apache.org/jira/browse/FLINK-36593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-36593: -- Fix Version/s: 1.19.2 > Upgrade io.airlift:aircompressor to mitigate CVE > > > Key: FLINK-36593 > URL: https://issues.apache.org/jira/browse/FLINK-36593 > Project: Flink > Issue Type: Improvement > Components: API / Core >Affects Versions: 2.0-preview >Reporter: Thomas Cooper >Assignee: Thomas Cooper >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0, 1.19.2, 1.20.1 > > > The current version of the aircompressor library (0.21), used in the > flink-runtime module, has a vulnerability: > [CVE-2024-36114|https://nvd.nist.gov/vuln/detail/CVE-2024-36114]. > This can be mitigated by upgrading to version 0.27 of the library. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-36593) Upgrade io.airlift:aircompressor to mitigate CVE
[
https://issues.apache.org/jira/browse/FLINK-36593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17909918#comment-17909918
]
Alexander Fedulov edited comment on FLINK-36593 at 1/5/25 3:59 PM:
---
{*}Merged to 1.19{*}: ff4243a7b4c25dd87d7d9a494f17eb358722a589
was (Author: afedulov):
Merged to 1.19: ff4243a7b4c25dd87d7d9a494f17eb358722a589
> Upgrade io.airlift:aircompressor to mitigate CVE
>
>
> Key: FLINK-36593
> URL: https://issues.apache.org/jira/browse/FLINK-36593
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 2.0-preview
>Reporter: Thomas Cooper
>Assignee: Thomas Cooper
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0, 1.19.2, 1.20.1
>
>
> The current version of the aircompressor library (0.21), used in the
> flink-runtime module, has a vulnerability:
> [CVE-2024-36114|https://nvd.nist.gov/vuln/detail/CVE-2024-36114].
> This can be mitigated by upgrading to version 0.27 of the library.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-36593) Upgrade io.airlift:aircompressor to mitigate CVE
[ https://issues.apache.org/jira/browse/FLINK-36593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17909918#comment-17909918 ] Alexander Fedulov commented on FLINK-36593: --- Merged to 1.19: ff4243a7b4c25dd87d7d9a494f17eb358722a589 > Upgrade io.airlift:aircompressor to mitigate CVE > > > Key: FLINK-36593 > URL: https://issues.apache.org/jira/browse/FLINK-36593 > Project: Flink > Issue Type: Improvement > Components: API / Core >Affects Versions: 2.0-preview >Reporter: Thomas Cooper >Assignee: Thomas Cooper >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0, 1.19.2, 1.20.1 > > > The current version of the aircompressor library (0.21), used in the > flink-runtime module, has a vulnerability: > [CVE-2024-36114|https://nvd.nist.gov/vuln/detail/CVE-2024-36114]. > This can be mitigated by upgrading to version 0.27 of the library. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-36593][runtime] Backport io.airlift:aircompressor upgrade to 0.27 [flink]
afedulov merged PR #25886: URL: https://github.com/apache/flink/pull/25886 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33571][table] Upgrade json-path from 2.7.0 to 2.9.0 [flink]
afedulov merged PR #25885: URL: https://github.com/apache/flink/pull/25885 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33571) Bump json-path from 2.7.0 to 2.9.0
[ https://issues.apache.org/jira/browse/FLINK-33571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17909920#comment-17909920 ] Alexander Fedulov commented on FLINK-33571: --- *Merged to 1.19:* 3b334bd1b0d016259cc2a2dbe97614c699954ebb > Bump json-path from 2.7.0 to 2.9.0 > -- > > Key: FLINK-33571 > URL: https://issues.apache.org/jira/browse/FLINK-33571 > Project: Flink > Issue Type: Bug >Affects Versions: 1.19.0 >Reporter: Yubin Li >Assignee: Thomas Cooper >Priority: Major > Labels: pull-request-available > Fix For: shaded-20.0, 1.20.1 > > > json-path has critical bugs in 2.7.0 used in flink project, see > [https://github.com/json-path/JsonPath/issues/906] > cve: [https://www.cve.org/CVERecord?id=CVE-2023-1370] > the current version is vulnerable to Denial of Service (DoS) due to a > StackOverflowError when parsing a deeply nested JSON array or object, and the > issue has been fixed in 2.8.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33571) Bump json-path from 2.7.0 to 2.9.0
[ https://issues.apache.org/jira/browse/FLINK-33571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-33571: -- Fix Version/s: 1.19.2 > Bump json-path from 2.7.0 to 2.9.0 > -- > > Key: FLINK-33571 > URL: https://issues.apache.org/jira/browse/FLINK-33571 > Project: Flink > Issue Type: Bug >Affects Versions: 1.19.0 >Reporter: Yubin Li >Assignee: Thomas Cooper >Priority: Major > Labels: pull-request-available > Fix For: 1.19.2, shaded-20.0, 1.20.1 > > > json-path has critical bugs in 2.7.0 used in flink project, see > [https://github.com/json-path/JsonPath/issues/906] > cve: [https://www.cve.org/CVERecord?id=CVE-2023-1370] > the current version is vulnerable to Denial of Service (DoS) due to a > StackOverflowError when parsing a deeply nested JSON array or object, and the > issue has been fixed in 2.8.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [BP-1.20][FLINK-34194] Update CI to Ubuntu 22.04 (Jammy) [flink]
afedulov commented on PR #25827: URL: https://github.com/apache/flink/pull/25827#issuecomment-2571694701 @XComp I retriggered a [follow-up PR](https://github.com/apache/flink/pull/25794) after this one has been merged: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=64350&view=logs&j=d3c25614-fe65-5341-2db6-c20b16adba8d But it still pulls the old image: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=64350&view=logs&j=d3c25614-fe65-5341-2db6-c20b16adba8d&t=837f3c6c-e20b-47da-861a-e584c6dec5c1&l=13 Do you have an idea what could be wrong? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-36976] Upgrade jackson from 2.15.3 to 2.18.2 [flink]
snuyanzin commented on code in PR #25865: URL: https://github.com/apache/flink/pull/25865#discussion_r1903312062 ## flink-kubernetes/pom.xml: ## @@ -31,7 +31,7 @@ under the License. jar - 6.9.2 + 6.13.4 Review Comment: since it also depends on jackson and there were some package rename in jackson between 2.15.x and 2.16.x -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-36976] Upgrade jackson from 2.15.3 to 2.18.2 [flink]
snuyanzin commented on code in PR #25865: URL: https://github.com/apache/flink/pull/25865#discussion_r1903311567 ## pom.xml: ## @@ -1857,7 +1857,7 @@ under the License. - com.fasterxml.jackson*:*:(,2.12.0] + com.fasterxml.jackson*:*:(,2.14.0] Review Comment: 2.13.2 has CVE so it would make sense to bump exclusions here as well -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33117) a wrong scala example in udfs page
[ https://issues.apache.org/jira/browse/FLINK-33117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17909877#comment-17909877 ] Alexander Fedulov commented on FLINK-33117: --- *Merged:* * master (2.0): fbf532e213882369494ee0f8595814a60de999bd * release-1.20: 6ce145b98ac295c78df5f21bc7b8954a5671ab5f * release-1.19: 2eda53aa54eeecf6efa4d103595aa51a4f91de65 > a wrong scala example in udfs page > -- > > Key: FLINK-33117 > URL: https://issues.apache.org/jira/browse/FLINK-33117 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 2.0.0, 1.19.0 >Reporter: cuiyanxiang >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0, 1.19.2, 1.20.1 > > Attachments: image-2023-09-19-17-58-42-102.png, > image-2023-09-19-17-59-52-174.png > > > problem：scala syntax error > !image-2023-09-19-17-58-42-102.png! > !image-2023-09-19-17-59-52-174.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-33117) a wrong scala example in udfs page
[ https://issues.apache.org/jira/browse/FLINK-33117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov resolved FLINK-33117. --- Resolution: Fixed > a wrong scala example in udfs page > -- > > Key: FLINK-33117 > URL: https://issues.apache.org/jira/browse/FLINK-33117 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 2.0.0, 1.19.0 >Reporter: cuiyanxiang >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0, 1.19.2, 1.20.1 > > Attachments: image-2023-09-19-17-58-42-102.png, > image-2023-09-19-17-59-52-174.png > > > problem：scala syntax error > !image-2023-09-19-17-58-42-102.png! > !image-2023-09-19-17-59-52-174.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [WIP][FLINK-36181] Use Java 17 by default [flink]
snuyanzin commented on PR #25898: URL: https://github.com/apache/flink/pull/25898#issuecomment-2571616830 @MartijnVisser I think you need to rebase to the latest master the failure seems to be related to the bug in master fixed within https://github.com/apache/flink/pull/25895 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-37006) Apply changes from FLINK-36277 to AdaptiveJoin
[ https://issues.apache.org/jira/browse/FLINK-37006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Nuyanzin resolved FLINK-37006. - Fix Version/s: 2.0.0 Resolution: Fixed > Apply changes from FLINK-36277 to AdaptiveJoin > -- > > Key: FLINK-37006 > URL: https://issues.apache.org/jira/browse/FLINK-37006 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Sergey Nuyanzin >Assignee: Sergey Nuyanzin >Priority: Blocker > Fix For: 2.0.0 > > > Blocker since now every master build is failing like > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=64822&view=logs&j=0c940707-2659-5648-cbe6-a1ad63045f0a&t=075c2716-8010-5565-fe08-3c4bb45824a4&l=12021 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34123][FLINK-35068][FLINK-36903] Introduce built-in serialization support for common collection types [flink]
X-czh commented on code in PR #25797:
URL: https://github.com/apache/flink/pull/25797#discussion_r1903292910
##
flink-core/src/main/java/org/apache/flink/api/common/serialization/SerializerConfigImpl.java:
##
@@ -356,6 +356,9 @@ public void configure(ReadableConfig configuration,
ClassLoader classLoader) {
configuration
.getOptional(PipelineOptions.FORCE_KRYO_AVRO)
.ifPresent(this::setForceKryoAvro);
+configuration
+.getOptional(PipelineOptions.BUILT_IN_COLLECTION_TYPES)
+.ifPresent(TypeExtractor::setBuiltInCollectionTypesEnabled);
Review Comment:
You are right, the TypeExtractor is a static util class. However, it is only
used when compiling the job, which is done on the client side for Session mode,
where only one job is executed at a time. Since migrating TypeExtractor to an
instance class is not an easy job, we might better implement it in this way
first, and try migrating it to an instance class for proper parameterization
later. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
