[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * Unknown: [CANCELED](TBD) * 38147f38be0a942edacc21b668aad7ea4e8f2c16 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * Unknown: [CANCELED](TBD) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * Unknown: [CANCELED](TBD) * 38147f38be0a942edacc21b668aad7ea4e8f2c16 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * Unknown: [CANCELED](TBD) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * Unknown: [CANCELED](TBD) * 38147f38be0a942edacc21b668aad7ea4e8f2c16 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Aitozi opened a new pull request #17846: [FLINK-24918][Runtime/State Backends]Support to specify the data dir …
Aitozi opened a new pull request #17846: URL: https://github.com/apache/flink/pull/17846 …for state benchmark ## What is the purpose of the change Support to specify the data dir for state benchmark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * Unknown: [CANCELED](TBD) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24918) Support to specify the data dir for state benchmark
[
https://issues.apache.org/jira/browse/FLINK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-24918:
---
Labels: pull-request-available (was: )
> Support to specify the data dir for state benchmark
>
>
> Key: FLINK-24918
> URL: https://issues.apache.org/jira/browse/FLINK-24918
> Project: Flink
> Issue Type: Improvement
> Components: Benchmarks, Runtime / State Backends
>Reporter: Aitozi
>Assignee: Aitozi
>Priority: Minor
> Labels: pull-request-available
>
> {{StateBackendBenchmarkUtils}} use null as the parent dir to create temp dir,
> which will finally use the /tmp as the data dir. It has two downsides:
> 1. the /tmp dir often mount with tmpfs, which may store data in memory. it
> will affect the result of rocksdb benchmark
> 2. It can not support to use benchmark to measure the performance on a new
> device.
> So I purpose to enhance the state benchmark to support specify the default
> data dir. And avoiding to use the {{/tmp}} as default.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-benchmarks] Aitozi opened a new pull request #41: [FLINK-24918][Runtime/State Backends]Support to specify the data dir for state benchmark
Aitozi opened a new pull request #41: URL: https://github.com/apache/flink-benchmarks/pull/41 Rely on the Flink [PR](https://github.com/apache/flink/pull/17846) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * Unknown: [CANCELED](TBD) * 38147f38be0a942edacc21b668aad7ea4e8f2c16 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17846: [FLINK-24918][Runtime/State Backends]Support to specify the data dir …
flinkbot commented on pull request #17846: URL: https://github.com/apache/flink/pull/17846#issuecomment-974773783 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * Unknown: [CANCELED](TBD) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17846: [FLINK-24918][Runtime/State Backends]Support to specify the data dir …
flinkbot edited a comment on pull request #17846: URL: https://github.com/apache/flink/pull/17846#issuecomment-974773793 ## CI report: * 20190bd570da9cb9e92e6237fc3c0f061b933908 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26788) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-benchmarks] Aitozi commented on pull request #41: [FLINK-24918][Runtime/State Backends]Support to specify the data dir for state benchmark
Aitozi commented on pull request #41: URL: https://github.com/apache/flink-benchmarks/pull/41#issuecomment-974774282 This build error due to it reply on the new api in [PR](https://github.com/apache/flink/pull/17846) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * Unknown: [CANCELED](TBD) * 38147f38be0a942edacc21b668aad7ea4e8f2c16 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24918) Support to specify the data dir for state benchmark
[
https://issues.apache.org/jira/browse/FLINK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17446944#comment-17446944
]
Aitozi commented on FLINK-24918:
I have create the two related pull request, please take a look when you are
free [~yunta].
> Support to specify the data dir for state benchmark
>
>
> Key: FLINK-24918
> URL: https://issues.apache.org/jira/browse/FLINK-24918
> Project: Flink
> Issue Type: Improvement
> Components: Benchmarks, Runtime / State Backends
>Reporter: Aitozi
>Assignee: Aitozi
>Priority: Minor
> Labels: pull-request-available
>
> {{StateBackendBenchmarkUtils}} use null as the parent dir to create temp dir,
> which will finally use the /tmp as the data dir. It has two downsides:
> 1. the /tmp dir often mount with tmpfs, which may store data in memory. it
> will affect the result of rocksdb benchmark
> 2. It can not support to use benchmark to measure the performance on a new
> device.
> So I purpose to enhance the state benchmark to support specify the default
> data dir. And avoiding to use the {{/tmp}} as default.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * Unknown: [CANCELED](TBD) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * Unknown: [CANCELED](TBD) * 38147f38be0a942edacc21b668aad7ea4e8f2c16 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * Unknown: [CANCELED](TBD) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * Unknown: [CANCELED](TBD) * 38147f38be0a942edacc21b668aad7ea4e8f2c16 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * Unknown: [CANCELED](TBD) * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) * c04424a0d0d66ef74ce3c4ed38719756d5b024da UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) * f5ab7477f53088b7f6794e113fdb4be6248ad230 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #30: [FLINK-24845] Add allreduce utility function in FlinkML
lindong28 commented on a change in pull request #30:
URL: https://github.com/apache/flink-ml/pull/30#discussion_r753769124
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/common/datastream/AllReduceUtils.java
##
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.datastream;
+
+import org.apache.flink.api.common.functions.RichFlatMapFunction;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Collector;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Applies all-reduce on a DataStream where each partition contains only one
double array.
+ *
+ * AllReduce is a communication primitive widely used in MPI. In this
implementation, all workers
+ * do reduce on a partition of the whole data and they all get the final
reduce result. In detail,
+ * we split each double array into chunks of fixed size buffer (4KB by
default) and let each subtask
+ * handle several chunks.
+ *
+ * There're mainly three stages:
+ * 1. All workers send their partial data to other workers for reduce.
+ * 2. All workers do reduce on all data it received and then send partial
results to others.
+ * 3. All workers merge partial results into final result.
+ */
+public class AllReduceUtils {
+
+private static final int TRANSFER_BUFFER_SIZE = 1024 * 4;
+
Review comment:
Thanks for the update. Maybe we can keep this comment open since we need
further action/comments from @gaoyunhaii :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #30: [FLINK-24845] Add allreduce utility function in FlinkML
lindong28 commented on a change in pull request #30:
URL: https://github.com/apache/flink-ml/pull/30#discussion_r753769187
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/common/datastream/AllReduceUtils.java
##
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.datastream;
+
+import org.apache.flink.api.common.functions.RichFlatMapFunction;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Collector;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Applies all-reduce on a DataStream where each partition contains only one
double array.
+ *
+ * AllReduce is a communication primitive widely used in MPI. In this
implementation, all workers
+ * do reduce on a partition of the whole data and they all get the final
reduce result. In detail,
+ * we split each double array into chunks of fixed size buffer (4KB by
default) and let each subtask
+ * handle several chunks.
+ *
+ * There're mainly three stages:
+ * 1. All workers send their partial data to other workers for reduce.
+ * 2. All workers do reduce on all data it received and then send partial
results to others.
+ * 3. All workers merge partial results into final result.
+ */
+public class AllReduceUtils {
+
+private static final int TRANSFER_BUFFER_SIZE = 1024 * 4;
+
Review comment:
Thanks for the update. Maybe we can keep this comment open since we need
further action/comments from @gaoyunhaii :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #30: [FLINK-24845] Add allreduce utility function in FlinkML
lindong28 commented on a change in pull request #30:
URL: https://github.com/apache/flink-ml/pull/30#discussion_r753769359
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/common/datastream/AllReduceUtils.java
##
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.datastream;
+
+import org.apache.flink.api.common.functions.RichFlatMapFunction;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Collector;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Applies all-reduce on a DataStream where each partition contains only one
double array.
+ *
+ * AllReduce is a communication primitive widely used in MPI. In this
implementation, all workers
+ * do reduce on a partition of the whole data and they all get the final
reduce result. In detail,
+ * we split each double array into chunks of fixed size buffer (4KB by
default) and let each subtask
+ * handle several chunks.
+ *
+ * There're mainly three stages:
+ * 1. All workers send their partial data to other workers for reduce.
+ * 2. All workers do reduce on all data it received and then send partial
results to others.
+ * 3. All workers merge partial results into final result.
+ */
+public class AllReduceUtils {
+
+private static final int TRANSFER_BUFFER_SIZE = 1024 * 4;
+
+/**
+ * Applies allReduce on the input data stream. The input data stream is
supposed to contain one
+ * double array in each partition. The result data stream has the same
parallelism as the input,
+ * where each partition contains one double array that sums all of the
double arrays in the
+ * input data stream.
+ *
+ * Note that we throw exception when one of the following two cases
happen:
+ * 1. There exists one partition that contains more than one double
array.
+ * 2. The length of double array is not consistent among all
partitions.
+ *
+ * @param input The input data stream.
+ * @return The result data stream.
+ */
+public static DataStream allReduce(DataStream input) {
+// chunkId, totalElements, partitionedArray
+DataStream> allReduceSend =
+input.flatMap(new AllReduceSend()).name("all-reduce-send");
+
+// taskId, chunkId, totalElements, partitionedArray
+DataStream> allReduceSum =
+allReduceSend
+.partitionCustom(
+(chunkId, numPartitions) -> chunkId %
numPartitions, x -> x.f0)
+.transform(
+"all-reduce-sum",
+new TupleTypeInfo<>(
+BasicTypeInfo.INT_TYPE_INFO,
+BasicTypeInfo.INT_TYPE_INFO,
+BasicTypeInfo.INT_TYPE_INFO,
+
PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO),
+new AllReduceSum())
+.name("all-reduce-sum");
+
+return allReduceSum
+.partitionCustom((taskIdx, numPartitions) -> taskIdx %
numPartitions, x -> x.f0)
+.transform(
+"all-reduce-recv", TypeInformation.of(double[].class),
new AllReduceRecv())
+.name("all-reduce-recv");
+}
+
+/**
+ * Splits each double array into multiple chunks and send each chunk to
the corresponding
+ * partition.
+ */
+private stat
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) * f5ab7477f53088b7f6794e113fdb4be6248ad230 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) * f5ab7477f53088b7f6794e113fdb4be6248ad230 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) * f5ab7477f53088b7f6794e113fdb4be6248ad230 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) * f5ab7477f53088b7f6794e113fdb4be6248ad230 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) * f5ab7477f53088b7f6794e113fdb4be6248ad230 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) * f5ab7477f53088b7f6794e113fdb4be6248ad230 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * 38147f38be0a942edacc21b668aad7ea4e8f2c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26789) * f5ab7477f53088b7f6794e113fdb4be6248ad230 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26790) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-24967) Make the IO pattern configureable in state benchmarks
Aitozi created FLINK-24967:
--
Summary: Make the IO pattern configureable in state benchmarks
Key: FLINK-24967
URL: https://issues.apache.org/jira/browse/FLINK-24967
Project: Flink
Issue Type: Improvement
Components: Benchmarks, Runtime / State Backends
Reporter: Aitozi
Currently, state benchmarks IO size are controlled by
{{StateBenchmarkConstants}}, which are not flexible to change. It's not easy to
test the performance under different IO size/pattern and different disk (which
can be solved by
[FLINK-24918|https://issues.apache.org/jira/browse/FLINK-24918]). I purpose to
make the state benchmark more configurable .
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24967) Make the IO pattern configureable in state benchmarks
[
https://issues.apache.org/jira/browse/FLINK-24967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17446952#comment-17446952
]
Aitozi commented on FLINK-24967:
cc [~yunta]
> Make the IO pattern configureable in state benchmarks
> -
>
> Key: FLINK-24967
> URL: https://issues.apache.org/jira/browse/FLINK-24967
> Project: Flink
> Issue Type: Improvement
> Components: Benchmarks, Runtime / State Backends
>Reporter: Aitozi
>Priority: Minor
>
> Currently, state benchmarks IO size are controlled by
> {{StateBenchmarkConstants}}, which are not flexible to change. It's not easy
> to test the performance under different IO size/pattern and different disk
> (which can be solved by
> [FLINK-24918|https://issues.apache.org/jira/browse/FLINK-24918]). I purpose
> to make the state benchmark more configurable .
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24967) Make the IO pattern configureable in state benchmarks
[
https://issues.apache.org/jira/browse/FLINK-24967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17446954#comment-17446954
]
Aitozi commented on FLINK-24967:
I also notice some {{TODOs}} in {{StateBenchmarkConstants}}. Do you have plan
or suggestions for this [~pnowojski] ?
> Make the IO pattern configureable in state benchmarks
> -
>
> Key: FLINK-24967
> URL: https://issues.apache.org/jira/browse/FLINK-24967
> Project: Flink
> Issue Type: Improvement
> Components: Benchmarks, Runtime / State Backends
>Reporter: Aitozi
>Priority: Minor
>
> Currently, state benchmarks IO size are controlled by
> {{StateBenchmarkConstants}}, which are not flexible to change. It's not easy
> to test the performance under different IO size/pattern and different disk
> (which can be solved by
> [FLINK-24918|https://issues.apache.org/jira/browse/FLINK-24918]). I purpose
> to make the state benchmark more configurable .
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
flinkbot edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-744252765 ## CI report: * a721a9d5f663117a203628547a19b33b23927ac2 UNKNOWN * 8f1051527a46aeba13aadc2db3c5f34d8aff78d2 UNKNOWN * f5ab7477f53088b7f6794e113fdb4be6248ad230 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26790) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #30: [FLINK-24845] Add allreduce utility function in FlinkML
zhipeng93 commented on a change in pull request #30:
URL: https://github.com/apache/flink-ml/pull/30#discussion_r753775516
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/common/datastream/AllReduceUtils.java
##
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.datastream;
+
+import org.apache.flink.api.common.functions.RichFlatMapFunction;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Collector;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Applies all-reduce on a DataStream where each partition contains only one
double array.
+ *
+ * AllReduce is a communication primitive widely used in MPI. In this
implementation, all workers
+ * do reduce on a partition of the whole data and they all get the final
reduce result. In detail,
+ * we split each double array into chunks of fixed size buffer (4KB by
default) and let each subtask
+ * handle several chunks.
+ *
+ * There're mainly three stages:
+ * 1. All workers send their partial data to other workers for reduce.
+ * 2. All workers do reduce on all data it received and then send partial
results to others.
+ * 3. All workers merge partial results into final result.
+ */
+public class AllReduceUtils {
+
+private static final int TRANSFER_BUFFER_SIZE = 1024 * 4;
+
+/**
+ * Applies allReduce on the input data stream. The input data stream is
supposed to contain one
+ * double array in each partition. The result data stream has the same
parallelism as the input,
+ * where each partition contains one double array that sums all of the
double arrays in the
+ * input data stream.
+ *
+ * Note that we throw exception when one of the following two cases
happen:
+ * 1. There exists one partition that contains more than one double
array.
+ * 2. The length of double array is not consistent among all
partitions.
+ *
+ * @param input The input data stream.
+ * @return The result data stream.
+ */
+public static DataStream allReduce(DataStream input) {
+// chunkId, totalElements, partitionedArray
+DataStream> allReduceSend =
+input.flatMap(new AllReduceSend()).name("all-reduce-send");
+
+// taskId, chunkId, totalElements, partitionedArray
+DataStream> allReduceSum =
+allReduceSend
+.partitionCustom(
+(chunkId, numPartitions) -> chunkId %
numPartitions, x -> x.f0)
+.transform(
+"all-reduce-sum",
+new TupleTypeInfo<>(
+BasicTypeInfo.INT_TYPE_INFO,
+BasicTypeInfo.INT_TYPE_INFO,
+BasicTypeInfo.INT_TYPE_INFO,
+
PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO),
+new AllReduceSum())
+.name("all-reduce-sum");
+
+return allReduceSum
+.partitionCustom((taskIdx, numPartitions) -> taskIdx %
numPartitions, x -> x.f0)
+.transform(
+"all-reduce-recv", TypeInformation.of(double[].class),
new AllReduceRecv())
+.name("all-reduce-recv");
+}
+
+/**
+ * Splits each double array into multiple chunks and send each chunk to
the corresponding
+ * partition.
+ */
+private stat
[GitHub] [flink] flinkbot edited a comment on pull request #17846: [FLINK-24918][Runtime/State Backends]Support to specify the data dir …
flinkbot edited a comment on pull request #17846: URL: https://github.com/apache/flink/pull/17846#issuecomment-974773793 ## CI report: * 20190bd570da9cb9e92e6237fc3c0f061b933908 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26788) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-14887) Provide a dedicated test class for SqlDateTimeUtils
[ https://issues.apache.org/jira/browse/FLINK-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-14887: --- Labels: auto-deprioritized-major auto-deprioritized-minor (was: auto-deprioritized-major stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Provide a dedicated test class for SqlDateTimeUtils > --- > > Key: FLINK-14887 > URL: https://issues.apache.org/jira/browse/FLINK-14887 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Runtime, Tests >Reporter: Kurt Young >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor > > Right now, SqlDateTimeUtils's logic in only covered by some code generation > cases, we should create a dedicated test class for this utility and test it > independently. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-23342) State ownership: compute shared state on recovery
[ https://issues.apache.org/jira/browse/FLINK-23342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-23342: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > State ownership: compute shared state on recovery > - > > Key: FLINK-23342 > URL: https://issues.apache.org/jira/browse/FLINK-23342 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.15.0 > > > # Introduce / refactor State object ID (consider StringBasedID, > SharedStateRegistryKey, StateHandleID; consider merging them) > # Add methods to obtain all object ids from a state handle > # Upate StateAssignmentOperation to compute shared state as a last step > # Send to TM (inject into JMTaskRestore) > See > [https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit#heading=h.uednun2snp4l] > for more details > > Code changes might look like this: > [https://github.com/rkhachatryan/flink/commit/025813c68b71e9fc357d7417d4403141998ad5f9#diff-cc692216ebbbeda497a335d79a1e568bb5164d98a45bd8efba08b9c1e9742ebdR144] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-14950) Support getKey in WindowOperator.Context
[
https://issues.apache.org/jira/browse/FLINK-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-14950:
---
Labels: auto-deprioritized-major auto-deprioritized-minor (was:
auto-deprioritized-major stale-minor)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Support getKey in WindowOperator.Context
>
>
> Key: FLINK-14950
> URL: https://issues.apache.org/jira/browse/FLINK-14950
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.9.1
>Reporter: Jiayi Liao
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor
>
> In our scenario, user needs to access the key of {{WindowOperator.Context}}
> to determine how to deal with the window.
> I think it's reasonable to support {{getKey()}} method in
> {{WindowOperator.Context}}.
> cc [~aljoscha]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-14882) Wrong null order when change flink default null order from NullCollation.LOW to NullCollation.FIRST
[ https://issues.apache.org/jira/browse/FLINK-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-14882: --- Labels: auto-deprioritized-major auto-deprioritized-minor (was: auto-deprioritized-major stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Wrong null order when change flink default null order from NullCollation.LOW > to NullCollation.FIRST > --- > > Key: FLINK-14882 > URL: https://issues.apache.org/jira/browse/FLINK-14882 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.9.1 >Reporter: Leonard Xu >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor > > Flink's default null collation keeps Nulls last for DESC, nulls first for ASC > which named > NullCollation.LOW. But when I change flink default null order from > NullCollation.LOW to NullCollation.FIRST, the null order in SQL result do not > change which not meet expected. > change file: org.apache.flink.table.planner.calcite.FlinkPlannerImpl.scala > > cc [~danny0405] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-14793) Add SPI interface to discovery state backend and establish well-known names
[
https://issues.apache.org/jira/browse/FLINK-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-14793:
---
Labels: auto-deprioritized-major auto-deprioritized-minor (was:
auto-deprioritized-major stale-minor)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Add SPI interface to discovery state backend and establish well-known names
> ---
>
> Key: FLINK-14793
> URL: https://issues.apache.org/jira/browse/FLINK-14793
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / State Backends
>Reporter: Kezhu Wang
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor
>
> Currently, {{StateBackendLoader.loadStateBackendFromConfig}} uses hybrid of
> hardcoded state backend name and class path to construct
> {{StateBackendFactory}}.
> I think we can provide a SPI interface, say \{{StateBackendProvider}}, to
> accomplish this.
> {code:title=StateBackendProvider.java}
> /**
> * SPI interface for {@link StateBackend} discovery and naming.
> *
> * @param StateBackend
> */
> @PublicEvolving
> public interface StateBackendProvider extends
> StateBackendFactory {
> /**
>* Name of provided {@link StateBackend}.
>*
>* @return name of provided {@link StateBackend}
>*/
> @Nonnull
> String name();
> /**
>* Aliases for provided {@link StateBackend}. Implementations can use
> this for deprecated names.
>*
>* @return aliases for provided {@link StateBackend}
>*/
> default Set aliases() {
> return Collections.emptySet();
> }
> }
> {code}
> With above spi, we gain several benefits over current hardcode and class path
> approach:
> * Explicit and established state backend naming and discovery instead of
> hardcoded. We can know exactly which state backends are avaiable and whether
> they are provided by user jars or not.
> * In state backend configuration, we can simply use established state
> backend name while not tied to fragile class path which is easily refactored
> out without great caution. These established state backend names could be
> flink community level, such as `jobmanager`, `filesystem` and `rocksdb`, and
> corporation level, such as `corporation-a.product.state-backend-basename`.
> Together with FLINK-14790, we can switch among state backends without
> knowledge of concrete implementation of target state backends in source code
> level.
> * More compatible with java module shipped in java 9 and above.
> Discovery of state backend can be restricted in {{StateBackendLoader}}, and
> we can fallback to class path approach if no matching state backend found in
> discovered state backends.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-14844) ConnectorCatalogTable should not patch the row type with TimeAttributes for batch mode
[ https://issues.apache.org/jira/browse/FLINK-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-14844: --- Labels: auto-deprioritized-major auto-deprioritized-minor (was: auto-deprioritized-major stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > ConnectorCatalogTable should not patch the row type with TimeAttributes for > batch mode > -- > > Key: FLINK-14844 > URL: https://issues.apache.org/jira/browse/FLINK-14844 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Affects Versions: 1.9.1 >Reporter: Danny Chen >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor > > In current code, ConnectorCatalogTable patch up the row type with time > attributes for both batch and stream mode, which is wrong, because batch mode > does not need that. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-14856) Allow setting "state.checkpoints.num-retained" for individual jobs
[
https://issues.apache.org/jira/browse/FLINK-14856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-14856:
---
Labels: auto-deprioritized-major auto-deprioritized-minor (was:
auto-deprioritized-major stale-minor)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Allow setting "state.checkpoints.num-retained" for individual jobs
> --
>
> Key: FLINK-14856
> URL: https://issues.apache.org/jira/browse/FLINK-14856
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration
>Reporter: Robert Metzger
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor
>
> In FLINK-4754 we introduced a configuration parameter
> \{{state.checkpoints.num-retained}}.
> A user recently reported a valid case of setting this configuration parameter
> for individual jobs, not globally for an entire Flink session:
> [https://lists.apache.org/thread.html/cfc868bf5aec107d51991b6cde4cf893917749583ad0fdce2d17e330@%3Cuser.flink.apache.org%3E]
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-14705) Remove nullables argument of BatchTestBase.registerCollection in blink planner
[ https://issues.apache.org/jira/browse/FLINK-14705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-14705: --- Labels: auto-deprioritized-major auto-deprioritized-minor (was: auto-deprioritized-major stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Remove nullables argument of BatchTestBase.registerCollection in blink planner > -- > > Key: FLINK-14705 > URL: https://issues.apache.org/jira/browse/FLINK-14705 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Runtime >Reporter: Danny Chen >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor > > The register collection method use TypeInformation to register collections, > the type information does not have nullability attribute, we expect to use > the new DataType instead. > So for these test cases, -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-14790) Custom state backend in job graph composition without depending on concrete state backend implementation
[
https://issues.apache.org/jira/browse/FLINK-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-14790:
---
Labels: auto-deprioritized-major auto-deprioritized-minor (was:
auto-deprioritized-major stale-minor)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Custom state backend in job graph composition without depending on concrete
> state backend implementation
>
>
> Key: FLINK-14790
> URL: https://issues.apache.org/jira/browse/FLINK-14790
> Project: Flink
> Issue Type: New Feature
> Components: API / DataStream
>Reporter: Kezhu Wang
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor
>
> In Flink 1.9 and before, if we want to custom state backend, we have to
> supply concrete state backend implementation in
> {{StreamExecutionEnvironment.setStateBackend}}. This implies that we have to
> depend on that concrete state backend implementation in job graph composition
> phase since we have to construct this concrete state backend. This may not be
> appropriate if we separate job graph composition and execution strictly, as
> job graph composition requires no state backend functions besides
> customization.
> Suppose that we are going to build a declarative streaming platform that
> building plain json/yaml like textual object, which may contain custom state
> backend configuration, with attached dependent jars to job graph and
> submitting generated job graph to Flink, we apparently should avoid ad-hoc
> state backend construction due to code reusability and indeterministic state
> backends configured.
> So I propose to support state backend customization in job graph composition
> phase to not depend on concrete state backend implementation.
> To achieve this goal, a {{Configuration}} field, say
> {{StreamExecutionEnvironment.stateBackendConfiguration}}, should be
> sufficient.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-14617) Dataset Parquet ClassCastException for SpecificRecord
[
https://issues.apache.org/jira/browse/FLINK-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-14617:
---
Labels: auto-deprioritized-major auto-deprioritized-minor (was:
auto-deprioritized-major stale-minor)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Dataset Parquet ClassCastException for SpecificRecord
> -
>
> Key: FLINK-14617
> URL: https://issues.apache.org/jira/browse/FLINK-14617
> Project: Flink
> Issue Type: Bug
> Components: API / DataSet
>Affects Versions: 1.8.0
> Environment: {code:java}
> {code}
>Reporter: Dominik Wosiński
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor
>
> The following code runs smoothly when the _executionEnvironment_ is instance
> of _StreamExecutionEnvironment_:
> {code:java}
> val filePaths = //some links to s3 files
> val job = Job.getInstance()
> AvroReadSupport.setAvroDataSupplier(job.getConfiguration,
> classOf[AvroDataSupplierWithTimestampConversion])
> val avroParquetInputFormat = new AvroParquetInputFormat[GpsPointDTO]()
> val hadoopInputFormat = new HadoopInputFormat[Void,
> GpsPointDTO](avroParquetInputFormat, classOf[Void], classOf[GpsPointDTO], job)
> FileInputFormat.addInputPaths(job, filePaths.head)
> executionEnvironment.createInput(hadoopInputFormat).map(_._2).print(){code}
> But when the _ExecutionEnvironment_ is used instead of
> _StreamExecutionEnviroment,_ then the code throws the:
> {code:java}
> Caused by: java.lang.ClassCastException: class
> org.apache.avro.generic.GenericData$Record cannot be cast to class
> com.company.GpsPointDTO (org.apache.avro.generic.GenericData$Record and
> com.company.GpsPointDTO are in unnamed module of loader 'app'){code}
> I don't think this is the expected behavior.
> The code simply reads some files from S3. It has the
> _AvroSupplierWithTimestampConversion_ which is used to add timestamp
> conversion to _DateTime_, _GpsPointDTO_ is a class generated from Avro schema.
> EDIT: It seems to work fine for Dataset API, if I remove the setting of
> _avroDataSupplier_. Obviously, in such case I needed to change Avro schema to
> use timestamp as _long_.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-14757) scala.StreamExecutionEnvironment Doesn't set return type upon adding source
[ https://issues.apache.org/jira/browse/FLINK-14757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-14757: --- Labels: auto-deprioritized-major auto-deprioritized-minor (was: auto-deprioritized-major stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > scala.StreamExecutionEnvironment Doesn't set return type upon adding source > --- > > Key: FLINK-14757 > URL: https://issues.apache.org/jira/browse/FLINK-14757 > Project: Flink > Issue Type: Bug > Components: API / Scala >Affects Versions: 1.9.0, 1.9.1 >Reporter: Tim Bess >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor > > [This > line|https://github.com/apache/flink/blob/d36ce5ff77fae2b01b8fbe8e5c15d610de8ed9f5/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/StreamExecutionEnvironment.scala#L618] > in StreamExecutionEnvironment.scala was changed to remove the `returns` call > starting in version 1.9.0. This causes subsequent transformations on the > stream to fail since the return TypeInformation isn't set anymore. > > Looks like the change was made in [this > pr|https://github.com/apache/flink/pull/8914]. Guessing it was a mistake > unless there's some reason to do this that I'm not understanding. > > I'm willing to make the PR to fix it, just need it changed so we can start > upgrading our microservices to 1.9. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-17415) Fold API-agnostic documentation into DataStream documentation (chinese)
[ https://issues.apache.org/jira/browse/FLINK-17415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-17415: --- Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Fold API-agnostic documentation into DataStream documentation (chinese) > --- > > Key: FLINK-17415 > URL: https://issues.apache.org/jira/browse/FLINK-17415 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation >Reporter: Aljoscha Krettek >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > As per > [FLIP-42|https://cwiki.apache.org/confluence/display/FLINK/FLIP-42%3A+Rework+Flink+Documentation], > we want to move most cross-API documentation to the DataStream section and > deprecate the DataSet API in the future. > We want to go from > - Project Build Setup > - Basic API Concepts > - Streaming (DataStream API) > - Batch (DataSet API) > - Table API & SQL > - Data Types & Serialization > - Managing Execution > - Libraries > - Best Practices > - API Migration Guides > To > - DataStream API > - Table API / SQL > - DataSet API -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-17437) Use StringData instead of BinaryStringData in code generation
[
https://issues.apache.org/jira/browse/FLINK-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-17437:
---
Labels: auto-deprioritized-major stale-minor (was:
auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Use StringData instead of BinaryStringData in code generation
> -
>
> Key: FLINK-17437
> URL: https://issues.apache.org/jira/browse/FLINK-17437
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Reporter: Jark Wu
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> In FLINK-16996, we force to use {{BinaryStringData}} instead of
> {{StringData}} in code generation. We hard cast StringData to
> BinaryStringData when {{RowData#getString}}. The motivation behind this is
> that this makes the code generator easily to generate opeartions based on
> string. There are too many invokings on the {{BinaryStringData}} now, if we
> use {{StringData}} in code generation, we have to refactor a lot of codes.
> However, this may be worth to do. Because using the public interface
> {{StringData}} is more straightforword.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-17177) Handle ERROR event correctly in KubernetesResourceManager#onError
[
https://issues.apache.org/jira/browse/FLINK-17177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-17177:
---
Labels: auto-deprioritized-major stale-minor (was:
auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Handle ERROR event correctly in KubernetesResourceManager#onError
> -
>
> Key: FLINK-17177
> URL: https://issues.apache.org/jira/browse/FLINK-17177
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0, 1.10.1
>Reporter: Canbin Zheng
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> Currently, once we receive an *ERROR* event that is sent from the K8s API
> server via the K8s {{Watcher}}, then {{KubernetesResourceManager#onError}}
> will handle it by calling the
> {{KubernetesResourceManager#removePodIfTerminated}}. This may be incorrect
> since the *ERROR* event may indicate an exception in the HTTP layer, which
> means the previously created {{Watcher}} may be no longer available and we'd
> better re-create the {{Watcher}} immediately.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-16937) ParquetTableSource should generate correct isFilterPushedDown
[
https://issues.apache.org/jira/browse/FLINK-16937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-16937:
---
Labels: auto-deprioritized-major stale-minor (was:
auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> ParquetTableSource should generate correct isFilterPushedDown
> -
>
> Key: FLINK-16937
> URL: https://issues.apache.org/jira/browse/FLINK-16937
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Ecosystem
>Affects Versions: 1.9.2, 1.10.0
>Reporter: Jingsong Lee
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
>
> {code:java}
> if (predicate != null) {
>this.isFilterPushedDown = true;
> }
> {code}
> If all filters can not be converted to parquet filter, the predicate will be
> null, this will lead to false isFilterPushdedDown, which is wrong.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-16780) improve Flink lookup join
[ https://issues.apache.org/jira/browse/FLINK-16780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16780: --- Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > improve Flink lookup join > -- > > Key: FLINK-16780 > URL: https://issues.apache.org/jira/browse/FLINK-16780 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Runtime >Reporter: Bowen Li >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > this is an umbrella ticket to group all the improvements related to lookup > join in Flink -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-16751) Expose bind port for Flink metric query service
[
https://issues.apache.org/jira/browse/FLINK-16751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-16751:
---
Labels: auto-deprioritized-major stale-minor (was:
auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Expose bind port for Flink metric query service
> ---
>
> Key: FLINK-16751
> URL: https://issues.apache.org/jira/browse/FLINK-16751
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Metrics
>Affects Versions: 1.11.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> With FLINK-15911 it is now possible to run Flink behind a NAT/with an
> unresolvable external address. However, due to FLINK-11127, the
> {{MetricFetcherImpl}} tries to connect to the {{TaskManagers}} instead of the
> other way around. If the TM is running behind a NAT, it would require to
> define an external port. At the moment, it is only possible to configure the
> external and bind port at the same time via
> {{metrics.internal.query-service.port}}. Hence, the external port always
> needs to be the same as the bind port if port forwarding should work.
> I think in order to properly solve this problem we either solve FLINK-11127
> or we introduce a {{metrics.internal.query-service.bind-port}} option which
> allows to configure a bind port. With such an option,
> {{metrics.internal.query-service.port}} would become the external port.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-16731) Support SHOW PARTITIONS table command in TableEnvironment and SQL Client
[ https://issues.apache.org/jira/browse/FLINK-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16731: --- Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Support SHOW PARTITIONS table command in TableEnvironment and SQL Client > - > > Key: FLINK-16731 > URL: https://issues.apache.org/jira/browse/FLINK-16731 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API, Table SQL / Client >Affects Versions: 1.10.0 >Reporter: Jun Zhang >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > Add a SHOW PARTITIONS TABLE command in sql client to support show the > partition information of the partition table -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-17360) Support custom partitioners in JDBCReadOptions
[ https://issues.apache.org/jira/browse/FLINK-17360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-17360: --- Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Support custom partitioners in JDBCReadOptions > --- > > Key: FLINK-17360 > URL: https://issues.apache.org/jira/browse/FLINK-17360 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC, Table SQL / Ecosystem >Reporter: Flavio Pompermaier >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > Suport custom ParameterValuesProvider. At the moment only > NumericBetweenParametersProvider is handled if a partition column and min/max > values are specified. > In a discussion in the mailing list some discussion about this was made. > Me ([~f.pompermaier]): > Then we can add a *scan.parametervalues.provider.class* in order to customize > the splitting of the query (we can also add a check that the query contains > at least 1 question mark). > If we introduce a custom parameters provider we need also to specify > parameters, using something like: > 'scan.parametervalues.0.name' = 'minDate', > 'scan.parametervalues.0.value'= '12/10/2019' > 'scan.parametervalues.1.name' = 'maxDate', > 'scan.parametervalues.1.value'= '01/01/2020' > [~lzljs3620320] > Maybe we need add something like "scan.parametervalues.provider.type", it can > be "bound, specify, custom": > - when *bound*, using old p*artitionLowerBound* and *partitionUpperBound*, > *numPartitions* > - when *specify*, using specify parameters like your proposal > - when *custom*, need *scan.parametervalues.provider.class* > Actually I don't know if specify and custom can be separated..but this can be > further discussed -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-17087) Use constant port for rest.port when it's set as 0 on Kubernetes
[
https://issues.apache.org/jira/browse/FLINK-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-17087:
---
Labels: auto-deprioritized-major stale-minor (was:
auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Use constant port for rest.port when it's set as 0 on Kubernetes
>
>
> Key: FLINK-17087
> URL: https://issues.apache.org/jira/browse/FLINK-17087
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Canbin Zheng
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> If people set {{rest.port}} to 0 when deploying a native K8s session cluster
> as the following command does,
> {code:java}
> ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=felix1 -Drest.port=0 ...
> {code}
> the submission client will throw an Exception as follows:
>
> {quote}org.apache.flink.client.deployment.ClusterDeploymentException: Could
> not create Kubernetes cluster felix1
> at
> org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployClusterInternal(KubernetesClusterDescriptor.java:189)
> at
> org.apache.flink.kubernetes.KubernetesClusterDescriptor.deploySessionCluster(KubernetesClusterDescriptor.java:129)
> at
> org.apache.flink.kubernetes.cli.KubernetesSessionCli.run(KubernetesSessionCli.java:108)
> at
> org.apache.flink.kubernetes.cli.KubernetesSessionCli.lambda$main$0(KubernetesSessionCli.java:185)
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> at
> org.apache.flink.kubernetes.cli.KubernetesSessionCli.main(KubernetesSessionCli.java:185)
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure
> executing: POST at:
> [https://xxx/apis/apps/v1/namespaces/default/deployments]. Message:
> Deployment.apps "felix1" is invalid:
> spec.template.spec.containers[0].ports[0].containerPort: Required value.
> Received status: Status(apiVersion=v1, code=422,
> details=StatusDetails(causes=[StatusCause(field=spec.template.spec.containers[0].ports[0].containerPort,
> message=Required value, reason=FieldValueRequired,
> additionalProperties={})], group=apps, kind=Deployment, name=felix1,
> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
> message=Deployment.apps "felix1" is invalid:
> spec.template.spec.containers[0].ports[0].containerPort: Required value,
> metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null,
> additionalProperties={}), reason=Invalid, status=Failure,
> additionalProperties={}).
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510)
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:449)
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413)
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372)
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:241)
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:798)
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:328)
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:324)
> at
> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.createJobManagerComponent(Fabric8FlinkKubeClient.java:83)
> at
> org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployClusterInternal(KubernetesClusterDescriptor.java:184)
> ... 5 more
> {quote}
>
> As we can see, the exception message is unintuitive and may confuse a variety
> of users.
> Therefore, this ticket proposes to use a fixed port instead if people set it
> as 0, like what we have done for the {{blob.server.port}} and the
> {{taskmanager.rpc.port}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-17532) Update tests to use BatchTestBase#checkTableResult
[
https://issues.apache.org/jira/browse/FLINK-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-17532:
---
Labels: auto-deprioritized-major stale-minor starter (was:
auto-deprioritized-major starter)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Update tests to use BatchTestBase#checkTableResult
> --
>
> Key: FLINK-17532
> URL: https://issues.apache.org/jira/browse/FLINK-17532
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner, Tests
>Reporter: Timo Walther
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor, starter
>
> Roughly 196 tests fail if we change the `Row.toString`. In the legacy
> planner, we will fix this quickly using some util. However, for the long-term
> Blink planner we should update those test to use the test bases and compare
> against instances instead of string.
> Similar to:
> {code}
> checkResult(
> "SELECT j, sum(k) FROM GenericTypedTable3 GROUP BY i, j",
> Seq(
> row(row(1, 1), 2),
> row(row(1, 1), 2),
> row(row(10, 1), 3)
> )
> )
> {code}
> Affected tests:
> {code}
> AggregateITCaseBase
> PartitionableSinkITCase
> CalcITCase
> JoinITCase
> SortITCase
> CorrelateITCase
> TableSinkITCase
> AggregationITCase
> GroupWindowITCase
> SetOperatorsITCase
> CalcITCase
> UnnestITCase
> AggregateRemoveITCase
> PruneAggregateCallITCase
> CalcITCase
> CorrelateITCase
> TableSinkITCase
> SetOperatorsITCase
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-16698) fllink need catalog listener to do such as preCreate/PreDrop* afterCreate/AfterDrop* things
[ https://issues.apache.org/jira/browse/FLINK-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16698: --- Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > fllink need catalog listener to do such as preCreate/PreDrop* > afterCreate/AfterDrop* things > --- > > Key: FLINK-16698 > URL: https://issues.apache.org/jira/browse/FLINK-16698 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Affects Versions: 1.10.0 >Reporter: jackylau >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > In order to support other things such as atlas or authentication, i think > flink need catalog listener to do such as preCreate/PreDrop* > afterCreate/AfterDrop* things, just like spark/hive does -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-23551) Bump ORC to 1.6.10
[ https://issues.apache.org/jira/browse/FLINK-23551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-23551: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Bump ORC to 1.6.10 > -- > > Key: FLINK-23551 > URL: https://issues.apache.org/jira/browse/FLINK-23551 > Project: Flink > Issue Type: Improvement > Components: Connectors / ORC >Affects Versions: 1.14.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available, stale-major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24198) KafkaTableITCase fail with FlinkJobNotFoundException: Could not find Flink job
[
https://issues.apache.org/jira/browse/FLINK-24198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-24198:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> KafkaTableITCase fail with FlinkJobNotFoundException: Could not find Flink job
> --
>
> Key: FLINK-24198
> URL: https://issues.apache.org/jira/browse/FLINK-24198
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.0, 1.12.5
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: stale-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24307&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5&l=6221
> {code}
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find
> Flink job (2bdb6a553c9de53f44a60bcd1d964503)
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestUtils.lambda$collectRows$0(KafkaTableTestUtils.java:60)
> ... 44 more
> Caused by: org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could
> not find Flink job (2bdb6a553c9de53f44a60bcd1d964503)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$cancelJob$8(Dispatcher.java:534)
> at java.util.Optional.orElseGet(Optional.java:267)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.cancelJob(Dispatcher.java:531)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-17378) KafkaProducerExactlyOnceITCase>KafkaProducerTestBase.testExactlyOnceCustomOperator unstable
[
https://issues.apache.org/jira/browse/FLINK-17378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-17378:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> KafkaProducerExactlyOnceITCase>KafkaProducerTestBase.testExactlyOnceCustomOperator
> unstable
> ---
>
> Key: FLINK-17378
> URL: https://issues.apache.org/jira/browse/FLINK-17378
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.11.0, 1.12.1
>Reporter: Robert Metzger
>Priority: Major
> Labels: stale-major, test-stability
>
> CI run:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=221&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=684b1416-4c17-504e-d5ab-97ee44e08a20
> {code}
> 2020-04-25T00:41:01.4191956Z 00:41:01,418 [Source: Custom Source -> Map ->
> Sink: Unnamed (1/1)] INFO
> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer
> [] - Flushing new partitions
> 2020-04-25T00:41:01.4194268Z 00:41:01,418 [FailingIdentityMapper Status
> Printer] INFO
> org.apache.flink.streaming.connectors.kafka.testutils.FailingIdentityMapper
> [] - > Failing mapper 0: count=690,
> totalCount=1000
> 2020-04-25T00:41:01.4589519Z
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 2020-04-25T00:41:01.4590089Z at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
> 2020-04-25T00:41:01.4590748Z at
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:659)
> 2020-04-25T00:41:01.4591524Z at
> org.apache.flink.streaming.util.TestStreamEnvironment.execute(TestStreamEnvironment.java:77)
> 2020-04-25T00:41:01.4592062Z at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1643)
> 2020-04-25T00:41:01.4592597Z at
> org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:35)
> 2020-04-25T00:41:01.4593092Z at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testExactlyOnce(KafkaProducerTestBase.java:370)
> 2020-04-25T00:41:01.4593680Z at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testExactlyOnceCustomOperator(KafkaProducerTestBase.java:317)
> 2020-04-25T00:41:01.4594450Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-04-25T00:41:01.4595076Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-04-25T00:41:01.4595794Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-04-25T00:41:01.4596622Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-04-25T00:41:01.4597501Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-04-25T00:41:01.4598396Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-04-25T00:41:01.460Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-04-25T00:41:01.4603082Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-04-25T00:41:01.4604023Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-04-25T00:41:01.4604590Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-04-25T00:41:01.4605225Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-04-25T00:41:01.4605902Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-04-25T00:41:01.4606591Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-04-25T00:41:01.4607468Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-04-25T00:41:01.4608577Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-04-25T00:41:01.4609030Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-04-25T00:41:01.4609460Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-04-25T00:41:01.4609842Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-04-25T00:
[jira] [Updated] (FLINK-22626) KafkaITCase.testTimestamps fails on Azure
[
https://issues.apache.org/jira/browse/FLINK-22626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22626:
---
Labels: auto-deprioritized-major test-stability (was:
auto-deprioritized-major stale-major test-stability)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> KafkaITCase.testTimestamps fails on Azure
> -
>
> Key: FLINK-22626
> URL: https://issues.apache.org/jira/browse/FLINK-22626
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.3, 1.13.1
>Reporter: Dawid Wysakowicz
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17819&view=logs&j=72d4811f-9f0d-5fd0-014a-0bc26b72b642&t=c1d93a6a-ba91-515d-3196-2ee8019fbda7&l=6708
> {code}
> Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error
> reading field 'api_keys': Error reading array of size 131096, only 50 bytes
> available
> at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:110)
> at
> org.apache.kafka.common.protocol.ApiKeys.parseResponse(ApiKeys.java:324)
> at
> org.apache.kafka.common.protocol.ApiKeys$1.parseResponse(ApiKeys.java:162)
> at
> org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:719)
> at
> org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:833)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:556)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:212)
> at
> org.apache.kafka.clients.consumer.internals.Fetcher.getTopicMetadata(Fetcher.java:368)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.partitionsFor(KafkaConsumer.java:1926)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.partitionsFor(KafkaConsumer.java:1894)
> at
> org.apache.flink.streaming.connectors.kafka.internals.KafkaPartitionDiscoverer.getAllPartitionsForTopics(KafkaPartitionDiscoverer.java:75)
> at
> org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer.discoverPartitions(AbstractPartitionDiscoverer.java:133)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.open(FlinkKafkaConsumerBase.java:577)
> at
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:428)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:545)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:93)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:535)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:575)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:758)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:573)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24935) Python module failed to compile due to "Could not create local repository"
[
https://issues.apache.org/jira/browse/FLINK-24935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17446998#comment-17446998
]
Yun Gao commented on FLINK-24935:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26775&view=logs&j=a29bcfe1-064d-50b9-354f-07802213a3c0&t=47ff6576-c9dc-5eab-9db8-183dcca3bede&l=27
> Python module failed to compile due to "Could not create local repository"
> --
>
> Key: FLINK-24935
> URL: https://issues.apache.org/jira/browse/FLINK-24935
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.12.5
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> Invoking mvn with 'mvn -Dmaven.wagon.http.pool=false --settings
> /__w/1/s/tools/ci/google-mirror-settings.xml
> -Dorg.slf4j.simpleLogger.showDateTime=true
> -Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss.SSS
> -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn
> --no-snapshot-updates -B -Dhadoop.version=2.8.3 -Dinclude_hadoop_aws
> -Dscala-2.11 clean deploy
> -DaltDeploymentRepository=validation_repository::default::file:/tmp/flink-validation-deployment
> -Dmaven.repo.local=/home/vsts/work/1/.m2/repository
> -Dflink.convergence.phase=install -Pcheck-convergence -Dflink.forkCount=2
> -Dflink.forkCountTestPackage=2 -Dmaven.javadoc.skip=true -U -DskipTests'
> [ERROR] Could not create local repository at /home/vsts/work/1/.m2/repository
> -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/LocalRepositoryNotAccessibleException
> {code}
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26625&view=logs&j=a29bcfe1-064d-50b9-354f-07802213a3c0&t=47ff6576-c9dc-5eab-9db8-183dcca3bede]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447000#comment-17447000 ] Lai Dai commented on FLINK-6573: [~monster#12] Hi, Could you tell when to open a pr for this mongodb connector > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24495) Python installdeps hangs
[
https://issues.apache.org/jira/browse/FLINK-24495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17446999#comment-17446999
]
Yun Gao commented on FLINK-24495:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26774&view=logs&j=821b528f-1eed-5598-a3b4-7f748b13f261&t=6bb545dd-772d-5d8c-f258-f5085fba3295&l=23596
> Python installdeps hangs
>
>
> Key: FLINK-24495
> URL: https://issues.apache.org/jira/browse/FLINK-24495
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.15.0
>Reporter: Xintong Song
>Assignee: Huang Xingbo
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24922&view=logs&j=821b528f-1eed-5598-a3b4-7f748b13f261&t=6bb545dd-772d-5d8c-f258-f5085fba3295&l=23587
> {code}
> Oct 10 02:30:01 py38-cython create: /__w/1/s/flink-python/.tox/py38-cython
> Oct 10 02:30:04 py38-cython installdeps: pytest, apache-beam==2.27.0,
> cython==0.29.16, grpcio>=1.29.0,<2, grpcio-tools>=1.3.5,<=1.14.2,
> apache-flink-libraries
> Oct 10 02:45:22
> ==
> Oct 10 02:45:22 Process produced no output for 900 seconds.
> Oct 10 02:45:22
> ==
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24763) ParquetFileSystemITCase.testLimitableBulkFormat failed on Azure
[
https://issues.apache.org/jira/browse/FLINK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447001#comment-17447001
]
Yun Gao commented on FLINK-24763:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26774&view=logs&j=ba53eb01-1462-56a3-8e98-0dd97fbcaab5&t=2e426bf0-b717-56bb-ab62-d63086457354&l=13847
> ParquetFileSystemITCase.testLimitableBulkFormat failed on Azure
> ---
>
> Key: FLINK-24763
> URL: https://issues.apache.org/jira/browse/FLINK-24763
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Assignee: Jingsong Lee
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
>
> The test {{ParquetFileSystemITCase.testLimitableBulkFormat}} fails with
> {code}
> 2021-11-03T22:10:11.5106075Z Nov 03 22:10:11 [ERROR]
> testLimitableBulkFormat[false] Time elapsed: 9.177 s <<< ERROR!
> 2021-11-03T22:10:11.5106643Z Nov 03 22:10:11 java.lang.RuntimeException:
> Failed to fetch next result
> 2021-11-03T22:10:11.5107213Z Nov 03 22:10:11 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:109)
> 2021-11-03T22:10:11.5111034Z Nov 03 22:10:11 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> 2021-11-03T22:10:11.5112190Z Nov 03 22:10:11 at
> org.apache.flink.table.planner.connectors.CollectDynamicSink$CloseableRowIteratorWrapper.hasNext(CollectDynamicSink.java:188)
> 2021-11-03T22:10:11.5112892Z Nov 03 22:10:11 at
> java.util.Iterator.forEachRemaining(Iterator.java:115)
> 2021-11-03T22:10:11.5113393Z Nov 03 22:10:11 at
> org.apache.flink.util.CollectionUtil.iteratorToList(CollectionUtil.java:109)
> 2021-11-03T22:10:11.5114157Z Nov 03 22:10:11 at
> org.apache.flink.formats.parquet.ParquetFileSystemITCase.testLimitableBulkFormat(ParquetFileSystemITCase.java:128)
> 2021-11-03T22:10:11.5114951Z Nov 03 22:10:11 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-11-03T22:10:11.5115568Z Nov 03 22:10:11 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-11-03T22:10:11.5116115Z Nov 03 22:10:11 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-11-03T22:10:11.5116591Z Nov 03 22:10:11 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-11-03T22:10:11.5117088Z Nov 03 22:10:11 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2021-11-03T22:10:11.5117807Z Nov 03 22:10:11 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-11-03T22:10:11.5118821Z Nov 03 22:10:11 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2021-11-03T22:10:11.5119417Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-11-03T22:10:11.5119944Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-11-03T22:10:11.5120427Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2021-11-03T22:10:11.5120919Z Nov 03 22:10:11 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-11-03T22:10:11.5121571Z Nov 03 22:10:11 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-11-03T22:10:11.5122526Z Nov 03 22:10:11 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-11-03T22:10:11.5123245Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-11-03T22:10:11.5123804Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-11-03T22:10:11.5124314Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-11-03T22:10:11.5124806Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-11-03T22:10:11.5125313Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-11-03T22:10:11.5125810Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-11-03T22:10:11.5126281Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-11-03T22:10:11.5126739Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-11-03T22:10:11.5127349Z Nov 0
[jira] [Comment Edited] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447000#comment-17447000 ] Lai Dai edited comment on FLINK-6573 at 11/21/21, 11:02 AM: [~monster#12] Hi, Could you tell when to open a pr for this mongodb connector was (Author: JIRAUSER279997): [~monster#12] Hi, Could you tell when to open a pr for this mongodb connector > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24968) "Flink over NAT end-to-end test" failed on azure
Yun Gao created FLINK-24968:
---
Summary: "Flink over NAT end-to-end test" failed on azure
Key: FLINK-24968
URL: https://issues.apache.org/jira/browse/FLINK-24968
Project: Flink
Issue Type: Bug
Components: Build System / Azure Pipelines
Affects Versions: 1.13.3
Reporter: Yun Gao
{code:java}
Nov 19 23:57:14 + dpkg --print-architecture
Nov 19 23:57:14 + wget -nv -O /usr/local/bin/gosu
https://github.com/tianon/gosu/releases/download/1.11/gosu-amd64
Nov 19 23:57:14
https://github.com/tianon/gosu/releases/download/1.11/gosu-amd64:
Nov 19 23:57:14 2021-11-19 23:57:14 ERROR 503: Service Unavailable.
The command '/bin/sh -c set -ex; wget -nv -O /usr/local/bin/gosu
"https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg
--print-architecture)"; wget -nv -O /usr/local/bin/gosu.asc "
Nov 19 23:57:14 ~/work/1/s
/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/test_nat.sh: line 64:
popd: directory stack empty
Nov 19 23:57:18 No stopped containers
sort: cannot read:
'/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-52007836763/out/docker_wc_out/*':
No such file or directory
Nov 19 23:57:18 FAIL WordCount: Output hash mismatch. Got
d41d8cd98f00b204e9800998ecf8427e, expected 72a690412be8928ba239c2da967328a5.
Nov 19 23:57:18 head hexdump of actual:
head: cannot open
'/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-52007836763/out/docker_wc_out/*'
for reading: No such file or directory
Nov 19 23:57:18 Stopping job timeout watchdog (with pid=208730)
Nov 19 23:57:18 [FAIL] Test script contains errors.
Nov 19 23:57:18 Checking of logs skipped.
Nov 19 23:57:18
Nov 19 23:57:18 [FAIL] 'Running Flink over NAT end-to-end test' failed after 0
minutes and 26 seconds! Test exited with exit code 1
Nov 19 23:57:19
23:57:19 ##[group]Environment Information
Nov 19 23:57:19 Searching for .dump, .dumpstream and related files in
'/home/vsts/work/1/s'
dmesg: read kernel buffer failed: Operation not permitted
Nov 19 23:57:20 No taskexecutor daemon to stop on host fv-az209-470.
Nov 19 23:57:20 No standalonesession daemon to stop on host fv-az209-470.
==
=== WARNING: This task took already 95% of the available time budget of 283
minutes ===
{code}
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26776&view=logs&j=91bf6583-3fb2-592f-e4d4-d79d79c3230a&t=3425d8ba-5f03-540a-c64b-51b8481bf7d6&l=3365]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24968) "Flink over NAT end-to-end test" failed on azure
[
https://issues.apache.org/jira/browse/FLINK-24968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447004#comment-17447004
]
Yun Gao commented on FLINK-24968:
-
It seems that the test first fail to download a package from the github, then
it hangs.
> "Flink over NAT end-to-end test" failed on azure
>
>
> Key: FLINK-24968
> URL: https://issues.apache.org/jira/browse/FLINK-24968
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.13.3
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> Nov 19 23:57:14 + dpkg --print-architecture
> Nov 19 23:57:14 + wget -nv -O /usr/local/bin/gosu
> https://github.com/tianon/gosu/releases/download/1.11/gosu-amd64
> Nov 19 23:57:14
> https://github.com/tianon/gosu/releases/download/1.11/gosu-amd64:
> Nov 19 23:57:14 2021-11-19 23:57:14 ERROR 503: Service Unavailable.
> The command '/bin/sh -c set -ex; wget -nv -O /usr/local/bin/gosu
> "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg
> --print-architecture)"; wget -nv -O /usr/local/bin/gosu.asc "
> Nov 19 23:57:14 ~/work/1/s
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/test_nat.sh: line 64:
> popd: directory stack empty
> Nov 19 23:57:18 No stopped containers
> sort: cannot read:
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-52007836763/out/docker_wc_out/*':
> No such file or directory
> Nov 19 23:57:18 FAIL WordCount: Output hash mismatch. Got
> d41d8cd98f00b204e9800998ecf8427e, expected 72a690412be8928ba239c2da967328a5.
> Nov 19 23:57:18 head hexdump of actual:
> head: cannot open
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-52007836763/out/docker_wc_out/*'
> for reading: No such file or directory
> Nov 19 23:57:18 Stopping job timeout watchdog (with pid=208730)
> Nov 19 23:57:18 [FAIL] Test script contains errors.
> Nov 19 23:57:18 Checking of logs skipped.
> Nov 19 23:57:18
> Nov 19 23:57:18 [FAIL] 'Running Flink over NAT end-to-end test' failed after
> 0 minutes and 26 seconds! Test exited with exit code 1
> Nov 19 23:57:19
> 23:57:19 ##[group]Environment Information
> Nov 19 23:57:19 Searching for .dump, .dumpstream and related files in
> '/home/vsts/work/1/s'
> dmesg: read kernel buffer failed: Operation not permitted
> Nov 19 23:57:20 No taskexecutor daemon to stop on host fv-az209-470.
> Nov 19 23:57:20 No standalonesession daemon to stop on host fv-az209-470.
> ==
> === WARNING: This task took already 95% of the available time budget of 283
> minutes ===
> {code}
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26776&view=logs&j=91bf6583-3fb2-592f-e4d4-d79d79c3230a&t=3425d8ba-5f03-540a-c64b-51b8481bf7d6&l=3365]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-24969) pyflink/table/tests/test_pandas_udf.py hangs on azure
Yun Gao created FLINK-24969: --- Summary: pyflink/table/tests/test_pandas_udf.py hangs on azure Key: FLINK-24969 URL: https://issues.apache.org/jira/browse/FLINK-24969 Project: Flink Issue Type: Bug Components: API / Python Affects Versions: 1.13.3 Reporter: Yun Gao -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24969) pyflink/table/tests/test_pandas_udf.py hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-24969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-24969:
Description:
{code:java}
Nov 20 05:11:02 at
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
Nov 20 05:11:02 at
java.net.SocketInputStream.read(SocketInputStream.java:171)
Nov 20 05:11:02 at
java.net.SocketInputStream.read(SocketInputStream.java:141)
Nov 20 05:11:02 at
sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
Nov 20 05:11:02 at
sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
Nov 20 05:11:02 at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
Nov 20 05:11:02 - locked <0x9379b830> (a
java.io.InputStreamReader)
Nov 20 05:11:02 at
java.io.InputStreamReader.read(InputStreamReader.java:184)
Nov 20 05:11:02 at java.io.BufferedReader.fill(BufferedReader.java:161)
Nov 20 05:11:02 at
java.io.BufferedReader.readLine(BufferedReader.java:324)
Nov 20 05:11:02 - locked <0x9379b830> (a
java.io.InputStreamReader)
Nov 20 05:11:02 at
java.io.BufferedReader.readLine(BufferedReader.java:389)
Nov 20 05:11:02 at
org.apache.flink.api.python.shaded.py4j.CallbackConnection.readBlockingResponse(CallbackConnection.java:169)
Nov 20 05:11:02 at
org.apache.flink.api.python.shaded.py4j.CallbackConnection.sendCommand(CallbackConnection.java:148)
Nov 20 05:11:02 at
org.apache.flink.api.python.shaded.py4j.CallbackClient.sendCommand(CallbackClient.java:384)
Nov 20 05:11:02 at
org.apache.flink.api.python.shaded.py4j.CallbackClient.sendCommand(CallbackClient.java:356)
Nov 20 05:11:02 at
org.apache.flink.api.python.shaded.py4j.reflection.PythonProxyHandler.invoke(PythonProxyHandler.java:106)
Nov 20 05:11:02 at com.sun.proxy.$Proxy19.ping(Unknown Source)
Nov 20 05:11:02 at
org.apache.flink.client.python.PythonGatewayServer.main(PythonGatewayServer.java:94)
Nov 20 05:11:02
Nov 20 05:11:02 "VM Thread" os_prio=0 tid=0x7f7874076000 nid=0x31da0
runnable
Nov 20 05:11:02
Nov 20 05:11:02 "GC task thread#0 (ParallelGC)" os_prio=0
tid=0x7f7874021800 nid=0x31d9e runnable
Nov 20 05:11:02
Nov 20 05:11:02 "GC task thread#1 (ParallelGC)" os_prio=0
tid=0x7f7874023000 nid=0x31d9f runnable
Nov 20 05:11:02
Nov 20 05:11:02 "VM Periodic Task Thread" os_prio=0 tid=0x7f78740c
nid=0x31da7 waiting on condition
Nov 20 05:11:02
Nov 20 05:11:02 JNI global references: 4563
Nov 20 05:11:02
Nov 20 05:11:02 Killing process with pid=636 and all descendants
/__w/1/s/tools/ci/watchdog.sh: line 113: 636 Terminated $cmd
Nov 20 05:11:02 Process exited with EXIT CODE: 143.
Nov 20 05:11:02 Trying to KILL watchdog (632).
Nov 20 05:11:02 Searching for .dump, .dumpstream and related files in '/__w/1/s'
The STDIO streams did not close within 10 seconds of the exit event from
process '/bin/bash'. This may indicate a child process inherited the STDIO
streams and has not yet exited.
##[error]Bash exited with code '143'.
Finishing: Test - python {code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26776&view=logs&j=821b528f-1eed-5598-a3b4-7f748b13f261&t=4fad9527-b9a5-5015-1b70-8356e5c91490&l=23553
Labels: test-stability (was: )
> pyflink/table/tests/test_pandas_udf.py hangs on azure
> -
>
> Key: FLINK-24969
> URL: https://issues.apache.org/jira/browse/FLINK-24969
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.13.3
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> Nov 20 05:11:02 at
> java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> Nov 20 05:11:02 at
> java.net.SocketInputStream.read(SocketInputStream.java:171)
> Nov 20 05:11:02 at
> java.net.SocketInputStream.read(SocketInputStream.java:141)
> Nov 20 05:11:02 at
> sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> Nov 20 05:11:02 at
> sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> Nov 20 05:11:02 at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> Nov 20 05:11:02 - locked <0x9379b830> (a
> java.io.InputStreamReader)
> Nov 20 05:11:02 at
> java.io.InputStreamReader.read(InputStreamReader.java:184)
> Nov 20 05:11:02 at java.io.BufferedReader.fill(BufferedReader.java:161)
> Nov 20 05:11:02 at
> java.io.BufferedReader.readLine(BufferedReader.java:324)
> Nov 20 05:11:02 - locked <0x9379b830> (a
> java.io.InputStreamReader)
> Nov 20 05:11:02 at
> java.io.BufferedReader.readLine(BufferedReader.java:389)
> Nov 20 05:11:02 at
> org.apache.flink.api.python.shaded.py4j.CallbackConn
[jira] [Commented] (FLINK-24495) Python installdeps hangs
[
https://issues.apache.org/jira/browse/FLINK-24495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447006#comment-17447006
]
Yun Gao commented on FLINK-24495:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26783&view=logs&j=821b528f-1eed-5598-a3b4-7f748b13f261&t=6bb545dd-772d-5d8c-f258-f5085fba3295&l=23596]
> Python installdeps hangs
>
>
> Key: FLINK-24495
> URL: https://issues.apache.org/jira/browse/FLINK-24495
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.15.0
>Reporter: Xintong Song
>Assignee: Huang Xingbo
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24922&view=logs&j=821b528f-1eed-5598-a3b4-7f748b13f261&t=6bb545dd-772d-5d8c-f258-f5085fba3295&l=23587
> {code}
> Oct 10 02:30:01 py38-cython create: /__w/1/s/flink-python/.tox/py38-cython
> Oct 10 02:30:04 py38-cython installdeps: pytest, apache-beam==2.27.0,
> cython==0.29.16, grpcio>=1.29.0,<2, grpcio-tools>=1.3.5,<=1.14.2,
> apache-flink-libraries
> Oct 10 02:45:22
> ==
> Oct 10 02:45:22 Process produced no output for 900 seconds.
> Oct 10 02:45:22
> ==
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24763) ParquetFileSystemITCase.testLimitableBulkFormat failed on Azure
[
https://issues.apache.org/jira/browse/FLINK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447007#comment-17447007
]
Yun Gao commented on FLINK-24763:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26783&view=logs&j=ba53eb01-1462-56a3-8e98-0dd97fbcaab5&t=2e426bf0-b717-56bb-ab62-d63086457354&l=13846]
> ParquetFileSystemITCase.testLimitableBulkFormat failed on Azure
> ---
>
> Key: FLINK-24763
> URL: https://issues.apache.org/jira/browse/FLINK-24763
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Assignee: Jingsong Lee
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
>
> The test {{ParquetFileSystemITCase.testLimitableBulkFormat}} fails with
> {code}
> 2021-11-03T22:10:11.5106075Z Nov 03 22:10:11 [ERROR]
> testLimitableBulkFormat[false] Time elapsed: 9.177 s <<< ERROR!
> 2021-11-03T22:10:11.5106643Z Nov 03 22:10:11 java.lang.RuntimeException:
> Failed to fetch next result
> 2021-11-03T22:10:11.5107213Z Nov 03 22:10:11 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:109)
> 2021-11-03T22:10:11.5111034Z Nov 03 22:10:11 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> 2021-11-03T22:10:11.5112190Z Nov 03 22:10:11 at
> org.apache.flink.table.planner.connectors.CollectDynamicSink$CloseableRowIteratorWrapper.hasNext(CollectDynamicSink.java:188)
> 2021-11-03T22:10:11.5112892Z Nov 03 22:10:11 at
> java.util.Iterator.forEachRemaining(Iterator.java:115)
> 2021-11-03T22:10:11.5113393Z Nov 03 22:10:11 at
> org.apache.flink.util.CollectionUtil.iteratorToList(CollectionUtil.java:109)
> 2021-11-03T22:10:11.5114157Z Nov 03 22:10:11 at
> org.apache.flink.formats.parquet.ParquetFileSystemITCase.testLimitableBulkFormat(ParquetFileSystemITCase.java:128)
> 2021-11-03T22:10:11.5114951Z Nov 03 22:10:11 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-11-03T22:10:11.5115568Z Nov 03 22:10:11 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-11-03T22:10:11.5116115Z Nov 03 22:10:11 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-11-03T22:10:11.5116591Z Nov 03 22:10:11 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-11-03T22:10:11.5117088Z Nov 03 22:10:11 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2021-11-03T22:10:11.5117807Z Nov 03 22:10:11 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-11-03T22:10:11.5118821Z Nov 03 22:10:11 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2021-11-03T22:10:11.5119417Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-11-03T22:10:11.5119944Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-11-03T22:10:11.5120427Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2021-11-03T22:10:11.5120919Z Nov 03 22:10:11 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-11-03T22:10:11.5121571Z Nov 03 22:10:11 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-11-03T22:10:11.5122526Z Nov 03 22:10:11 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-11-03T22:10:11.5123245Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-11-03T22:10:11.5123804Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-11-03T22:10:11.5124314Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-11-03T22:10:11.5124806Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-11-03T22:10:11.5125313Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-11-03T22:10:11.5125810Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-11-03T22:10:11.5126281Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-11-03T22:10:11.5126739Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-11-03T22:10:11.5127349Z Nov
[jira] [Created] (FLINK-24970) FlinkKafkaProducerITCase testScaleUpAfterScalingDown Timeout expired after 60000 milliseconds while awaiting InitProducerId
Yun Gao created FLINK-24970:
---
Summary: FlinkKafkaProducerITCase testScaleUpAfterScalingDown
Timeout expired after 6 milliseconds while awaiting InitProducerId
Key: FLINK-24970
URL: https://issues.apache.org/jira/browse/FLINK-24970
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.15.0
Reporter: Yun Gao
{code:java}
Nov 20 23:27:52 [ERROR] Tests run: 15, Failures: 10, Errors: 1, Skipped: 0,
Time elapsed: 341.449 s <<< FAILURE! - in
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase
Nov 20 23:27:52 [ERROR] testScaleUpAfterScalingDown Time elapsed: 75.789 s
<<< ERROR!
Nov 20 23:27:52 org.apache.kafka.common.errors.TimeoutException: Timeout
expired after 6 milliseconds while awaiting InitProducerId{code}
After that a lot of tests failed due to
{code:java}
Nov 20 23:27:52 [ERROR] testScaleUpAfterScalingDown Time elapsed: 75.789 s
<<< ERROR!
Nov 20 23:27:52 org.apache.kafka.common.errors.TimeoutException: Timeout
expired after 6 milliseconds while awaiting InitProducerId
Nov 20 23:27:52
Nov 20 23:27:52 [ERROR] testFailAndRecoverSameCheckpointTwice Time elapsed:
10.031 s <<< FAILURE!
Nov 20 23:27:52 java.lang.AssertionError: Detected producer leak. Thread name:
kafka-producer-network-thread |
producer-MockTask-002a002c-9
Nov 20 23:27:52 at org.junit.Assert.fail(Assert.java:89) {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-24971) Streaming File Sink s3 end-to-end test stalled on azure
Yun Gao created FLINK-24971:
---
Summary: Streaming File Sink s3 end-to-end test stalled on azure
Key: FLINK-24971
URL: https://issues.apache.org/jira/browse/FLINK-24971
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem
Affects Versions: 1.12.5
Reporter: Yun Gao
{code:java}
Nov 21 00:04:36 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:04:41 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:04:46 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:04:51 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:04:56 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:05:01 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:05:06 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:05:11 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:05:16 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:05:21 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:05:26 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:05:31 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:05:36 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:05:41 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:05:46 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:05:52 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:05:57 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:06:02 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:06:07 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:06:12 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:06:17 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:06:22 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:06:27 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:06:32 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:06:37 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:06:42 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:06:47 Still waiting for restarts. Expected: 1 Current: 0
Nov 21 00:06:51 Test (pid: 414853) did not finish after 900 seconds.
Nov 21 00:06:51 Printing Flink logs and killing it: {code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26784&view=logs&j=91bf6583-3fb2-592f-e4d4-d79d79c3230a&t=3425d8ba-5f03-540a-c64b-51b8481bf7d6&l=12438
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24971) Streaming File Sink s3 end-to-end test stalled on azure
[
https://issues.apache.org/jira/browse/FLINK-24971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447008#comment-17447008
]
Yun Gao commented on FLINK-24971:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26784&view=logs&j=6caf31d6-847a-526e-9624-468e053467d6&t=0b23652f-b18b-5b6e-6eb6-a11070364610&l=4329]
This is another case for new file sink test, but since the two tests shares a
lot of codes, I think they would be related.
> Streaming File Sink s3 end-to-end test stalled on azure
> ---
>
> Key: FLINK-24971
> URL: https://issues.apache.org/jira/browse/FLINK-24971
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.12.5
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> Nov 21 00:04:36 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:04:41 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:04:46 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:04:51 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:04:56 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:05:01 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:05:06 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:05:11 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:05:16 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:05:21 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:05:26 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:05:31 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:05:36 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:05:41 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:05:46 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:05:52 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:05:57 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:06:02 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:06:07 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:06:12 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:06:17 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:06:22 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:06:27 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:06:32 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:06:37 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:06:42 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:06:47 Still waiting for restarts. Expected: 1 Current: 0
> Nov 21 00:06:51 Test (pid: 414853) did not finish after 900 seconds.
> Nov 21 00:06:51 Printing Flink logs and killing it: {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26784&view=logs&j=91bf6583-3fb2-592f-e4d4-d79d79c3230a&t=3425d8ba-5f03-540a-c64b-51b8481bf7d6&l=12438
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24935) Python module failed to compile due to "Could not create local repository"
[
https://issues.apache.org/jira/browse/FLINK-24935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447009#comment-17447009
]
Yun Gao commented on FLINK-24935:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26784&view=logs&j=a29bcfe1-064d-50b9-354f-07802213a3c0&t=47ff6576-c9dc-5eab-9db8-183dcca3bede&l=28]
> Python module failed to compile due to "Could not create local repository"
> --
>
> Key: FLINK-24935
> URL: https://issues.apache.org/jira/browse/FLINK-24935
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.12.5
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> Invoking mvn with 'mvn -Dmaven.wagon.http.pool=false --settings
> /__w/1/s/tools/ci/google-mirror-settings.xml
> -Dorg.slf4j.simpleLogger.showDateTime=true
> -Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss.SSS
> -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn
> --no-snapshot-updates -B -Dhadoop.version=2.8.3 -Dinclude_hadoop_aws
> -Dscala-2.11 clean deploy
> -DaltDeploymentRepository=validation_repository::default::file:/tmp/flink-validation-deployment
> -Dmaven.repo.local=/home/vsts/work/1/.m2/repository
> -Dflink.convergence.phase=install -Pcheck-convergence -Dflink.forkCount=2
> -Dflink.forkCountTestPackage=2 -Dmaven.javadoc.skip=true -U -DskipTests'
> [ERROR] Could not create local repository at /home/vsts/work/1/.m2/repository
> -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/LocalRepositoryNotAccessibleException
> {code}
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26625&view=logs&j=a29bcfe1-064d-50b9-354f-07802213a3c0&t=47ff6576-c9dc-5eab-9db8-183dcca3bede]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-20498) SQLClientSchemaRegistryITCase.testReading test timed out after 120000 milliseconds
[
https://issues.apache.org/jira/browse/FLINK-20498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447010#comment-17447010
]
Yun Gao commented on FLINK-20498:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26785&view=logs&j=4dd4dbdd-1802-5eb7-a518-6acd9d24d0fc&t=8d6b4dd3-4ca1-5611-1743-57a7d76b395a&l=16877]
> SQLClientSchemaRegistryITCase.testReading test timed out after 12
> milliseconds
> --
>
> Key: FLINK-20498
> URL: https://issues.apache.org/jira/browse/FLINK-20498
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Client
>Affects Versions: 1.12.1, 1.13.0, 1.14.0, 1.12.3
>Reporter: Huang Xingbo
>Priority: Critical
> Labels: auto-deprioritized-major, test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10548&view=logs&j=739e6eac-8312-5d31-d437-294c4d26fced&t=a68b8d89-50e9-5977-4500-f4fde4f57f9b]
> {code:java}
> 2020-12-06T02:06:38.6416440Z Dec 06 02:06:38
> org.junit.runners.model.TestTimedOutException: test timed out after 12
> milliseconds
> 2020-12-06T02:06:38.6417052Z Dec 06 02:06:38 at java.lang.Object.wait(Native
> Method)
> 2020-12-06T02:06:38.6417586Z Dec 06 02:06:38 at
> java.lang.Thread.join(Thread.java:1252)
> 2020-12-06T02:06:38.6418170Z Dec 06 02:06:38 at
> java.lang.Thread.join(Thread.java:1326)
> 2020-12-06T02:06:38.6418788Z Dec 06 02:06:38 at
> org.apache.kafka.clients.admin.KafkaAdminClient.close(KafkaAdminClient.java:541)
> 2020-12-06T02:06:38.6419463Z Dec 06 02:06:38 at
> org.apache.kafka.clients.admin.Admin.close(Admin.java:96)
> 2020-12-06T02:06:38.6420277Z Dec 06 02:06:38 at
> org.apache.kafka.clients.admin.Admin.close(Admin.java:79)
> 2020-12-06T02:06:38.6420973Z Dec 06 02:06:38 at
> org.apache.flink.tests.util.kafka.KafkaContainerClient.createTopic(KafkaContainerClient.java:76)
> 2020-12-06T02:06:38.6421797Z Dec 06 02:06:38 at
> org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase.testReading(SQLClientSchemaRegistryITCase.java:109)
> 2020-12-06T02:06:38.6422517Z Dec 06 02:06:38 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-12-06T02:06:38.6423173Z Dec 06 02:06:38 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-12-06T02:06:38.6423990Z Dec 06 02:06:38 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-12-06T02:06:38.6424656Z Dec 06 02:06:38 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-12-06T02:06:38.6425321Z Dec 06 02:06:38 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-12-06T02:06:38.6426057Z Dec 06 02:06:38 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-12-06T02:06:38.6426766Z Dec 06 02:06:38 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-12-06T02:06:38.6427478Z Dec 06 02:06:38 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-12-06T02:06:38.6428232Z Dec 06 02:06:38 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> 2020-12-06T02:06:38.6428999Z Dec 06 02:06:38 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> 2020-12-06T02:06:38.6429707Z Dec 06 02:06:38 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2020-12-06T02:06:38.6430292Z Dec 06 02:06:38 at
> java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-20498) SQLClientSchemaRegistryITCase.testReading test timed out after 120000 milliseconds
[
https://issues.apache.org/jira/browse/FLINK-20498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447011#comment-17447011
]
Yun Gao commented on FLINK-20498:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26785&view=logs&j=6caf31d6-847a-526e-9624-468e053467d6&t=0b23652f-b18b-5b6e-6eb6-a11070364610&l=17693]
This is a case for SQLClientSchemaRegistryITCase#testWriting, but it seems to
be the same issue.
> SQLClientSchemaRegistryITCase.testReading test timed out after 12
> milliseconds
> --
>
> Key: FLINK-20498
> URL: https://issues.apache.org/jira/browse/FLINK-20498
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Client
>Affects Versions: 1.12.1, 1.13.0, 1.14.0, 1.12.3
>Reporter: Huang Xingbo
>Priority: Critical
> Labels: auto-deprioritized-major, test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10548&view=logs&j=739e6eac-8312-5d31-d437-294c4d26fced&t=a68b8d89-50e9-5977-4500-f4fde4f57f9b]
> {code:java}
> 2020-12-06T02:06:38.6416440Z Dec 06 02:06:38
> org.junit.runners.model.TestTimedOutException: test timed out after 12
> milliseconds
> 2020-12-06T02:06:38.6417052Z Dec 06 02:06:38 at java.lang.Object.wait(Native
> Method)
> 2020-12-06T02:06:38.6417586Z Dec 06 02:06:38 at
> java.lang.Thread.join(Thread.java:1252)
> 2020-12-06T02:06:38.6418170Z Dec 06 02:06:38 at
> java.lang.Thread.join(Thread.java:1326)
> 2020-12-06T02:06:38.6418788Z Dec 06 02:06:38 at
> org.apache.kafka.clients.admin.KafkaAdminClient.close(KafkaAdminClient.java:541)
> 2020-12-06T02:06:38.6419463Z Dec 06 02:06:38 at
> org.apache.kafka.clients.admin.Admin.close(Admin.java:96)
> 2020-12-06T02:06:38.6420277Z Dec 06 02:06:38 at
> org.apache.kafka.clients.admin.Admin.close(Admin.java:79)
> 2020-12-06T02:06:38.6420973Z Dec 06 02:06:38 at
> org.apache.flink.tests.util.kafka.KafkaContainerClient.createTopic(KafkaContainerClient.java:76)
> 2020-12-06T02:06:38.6421797Z Dec 06 02:06:38 at
> org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase.testReading(SQLClientSchemaRegistryITCase.java:109)
> 2020-12-06T02:06:38.6422517Z Dec 06 02:06:38 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-12-06T02:06:38.6423173Z Dec 06 02:06:38 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-12-06T02:06:38.6423990Z Dec 06 02:06:38 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-12-06T02:06:38.6424656Z Dec 06 02:06:38 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-12-06T02:06:38.6425321Z Dec 06 02:06:38 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-12-06T02:06:38.6426057Z Dec 06 02:06:38 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-12-06T02:06:38.6426766Z Dec 06 02:06:38 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-12-06T02:06:38.6427478Z Dec 06 02:06:38 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-12-06T02:06:38.6428232Z Dec 06 02:06:38 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> 2020-12-06T02:06:38.6428999Z Dec 06 02:06:38 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> 2020-12-06T02:06:38.6429707Z Dec 06 02:06:38 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2020-12-06T02:06:38.6430292Z Dec 06 02:06:38 at
> java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-22194) KafkaSourceReaderTest.testCommitOffsetsWithoutAliveFetchers fail due to commit timeout
[
https://issues.apache.org/jira/browse/FLINK-22194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447013#comment-17447013
]
Yun Gao commented on FLINK-22194:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26786&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=15a22db7-8faa-5b34-3920-d33c9f0ca23c&l=7285]
> KafkaSourceReaderTest.testCommitOffsetsWithoutAliveFetchers fail due to
> commit timeout
> --
>
> Key: FLINK-22194
> URL: https://issues.apache.org/jira/browse/FLINK-22194
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.13.0, 1.12.4
>Reporter: Guowei Ma
>Priority: Major
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16308&view=logs&j=b0097207-033c-5d9a-b48c-6d4796fbe60d&t=e8fcc430-213e-5cce-59d4-6942acf09121&l=6535
> {code:java}
> [ERROR]
> testCommitOffsetsWithoutAliveFetchers(org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest)
> Time elapsed: 60.123 s <<< ERROR!
> java.util.concurrent.TimeoutException: The offset commit did not finish
> before timeout.
> at
> org.apache.flink.core.testutils.CommonTestUtils.waitUtil(CommonTestUtils.java:210)
> at
> org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest.pollUntil(KafkaSourceReaderTest.java:285)
> at
> org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest.testCommitOffsetsWithoutAliveFetchers(KafkaSourceReaderTest.java:129)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24840) JobManagerHAProcessFailureRecoveryITCase crashed on AZP
[
https://issues.apache.org/jira/browse/FLINK-24840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-24840:
Affects Version/s: 1.14.0
> JobManagerHAProcessFailureRecoveryITCase crashed on AZP
> ---
>
> Key: FLINK-24840
> URL: https://issues.apache.org/jira/browse/FLINK-24840
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> The test {{JobManagerHAProcessFailureRecoveryITCase}} crashes on AZP.
> {code}
> 2021-11-08T23:27:32.9623564Z Nov 08 23:27:32 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test
> (integration-tests) on project flink-tests: There are test failures.
> 2021-11-08T23:27:32.9624265Z Nov 08 23:27:32 [ERROR]
> 2021-11-08T23:27:32.9624981Z Nov 08 23:27:32 [ERROR] Please refer to
> /__w/2/s/flink-tests/target/surefire-reports for the individual test results.
> 2021-11-08T23:27:32.9625718Z Nov 08 23:27:32 [ERROR] Please refer to dump
> files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
> 2021-11-08T23:27:32.9626539Z Nov 08 23:27:32 [ERROR] ExecutionException The
> forked VM terminated without properly saying goodbye. VM crash or System.exit
> called?
> 2021-11-08T23:27:32.9627945Z Nov 08 23:27:32 [ERROR] Command was /bin/sh -c
> cd /__w/2/s/flink-tests/target &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -jar
> /__w/2/s/flink-tests/target/surefire/surefirebooter6973421711355178650.jar
> /__w/2/s/flink-tests/target/surefire 2021-11-08T22-42-26_652-jvmRun2
> surefire3220623274989834473tmp surefire_1316080825708637501141tmp
> 2021-11-08T23:27:32.9628850Z Nov 08 23:27:32 [ERROR] Error occurred in
> starting fork, check output in log
> 2021-11-08T23:27:32.9629327Z Nov 08 23:27:32 [ERROR] Process Exit Code: 239
> 2021-11-08T23:27:32.9629645Z Nov 08 23:27:32 [ERROR] Crashed tests:
> 2021-11-08T23:27:32.9630025Z Nov 08 23:27:32 [ERROR]
> org.apache.flink.test.recovery.JobManagerHAProcessFailureRecoveryITCase
> 2021-11-08T23:27:32.9630643Z Nov 08 23:27:32 [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> 2021-11-08T23:27:32.9632522Z Nov 08 23:27:32 [ERROR] Command was /bin/sh -c
> cd /__w/2/s/flink-tests/target &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -jar
> /__w/2/s/flink-tests/target/surefire/surefirebooter6973421711355178650.jar
> /__w/2/s/flink-tests/target/surefire 2021-11-08T22-42-26_652-jvmRun2
> surefire3220623274989834473tmp surefire_1316080825708637501141tmp
> 2021-11-08T23:27:32.9633418Z Nov 08 23:27:32 [ERROR] Error occurred in
> starting fork, check output in log
> 2021-11-08T23:27:32.9635503Z Nov 08 23:27:32 [ERROR] Process Exit Code: 239
> 2021-11-08T23:27:32.9636287Z Nov 08 23:27:32 [ERROR] Crashed tests:
> 2021-11-08T23:27:32.9636863Z Nov 08 23:27:32 [ERROR]
> org.apache.flink.test.recovery.JobManagerHAProcessFailureRecoveryITCase
> 2021-11-08T23:27:32.9637609Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
> 2021-11-08T23:27:32.9638331Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:457)
> 2021-11-08T23:27:32.9638951Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:298)
> 2021-11-08T23:27:32.9639505Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:246)
> 2021-11-08T23:27:32.9640108Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
> 2021-11-08T23:27:32.9640916Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
> 2021-11-08T23:27:32.9641819Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
> 2021-11-08T23:27:32.9642447Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
> 2021-11-08T23:27:32.9643039Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
> 2021-11-08T23:27:32.9643582Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java
[jira] [Commented] (FLINK-24840) JobManagerHAProcessFailureRecoveryITCase crashed on AZP
[
https://issues.apache.org/jira/browse/FLINK-24840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447015#comment-17447015
]
Yun Gao commented on FLINK-24840:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26786&view=logs&j=f2b08047-82c3-520f-51ee-a30fd6254285&t=3810d23d-4df2-586c-103c-ec14ede6af00&l=12094]
> JobManagerHAProcessFailureRecoveryITCase crashed on AZP
> ---
>
> Key: FLINK-24840
> URL: https://issues.apache.org/jira/browse/FLINK-24840
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> The test {{JobManagerHAProcessFailureRecoveryITCase}} crashes on AZP.
> {code}
> 2021-11-08T23:27:32.9623564Z Nov 08 23:27:32 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test
> (integration-tests) on project flink-tests: There are test failures.
> 2021-11-08T23:27:32.9624265Z Nov 08 23:27:32 [ERROR]
> 2021-11-08T23:27:32.9624981Z Nov 08 23:27:32 [ERROR] Please refer to
> /__w/2/s/flink-tests/target/surefire-reports for the individual test results.
> 2021-11-08T23:27:32.9625718Z Nov 08 23:27:32 [ERROR] Please refer to dump
> files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
> 2021-11-08T23:27:32.9626539Z Nov 08 23:27:32 [ERROR] ExecutionException The
> forked VM terminated without properly saying goodbye. VM crash or System.exit
> called?
> 2021-11-08T23:27:32.9627945Z Nov 08 23:27:32 [ERROR] Command was /bin/sh -c
> cd /__w/2/s/flink-tests/target &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -jar
> /__w/2/s/flink-tests/target/surefire/surefirebooter6973421711355178650.jar
> /__w/2/s/flink-tests/target/surefire 2021-11-08T22-42-26_652-jvmRun2
> surefire3220623274989834473tmp surefire_1316080825708637501141tmp
> 2021-11-08T23:27:32.9628850Z Nov 08 23:27:32 [ERROR] Error occurred in
> starting fork, check output in log
> 2021-11-08T23:27:32.9629327Z Nov 08 23:27:32 [ERROR] Process Exit Code: 239
> 2021-11-08T23:27:32.9629645Z Nov 08 23:27:32 [ERROR] Crashed tests:
> 2021-11-08T23:27:32.9630025Z Nov 08 23:27:32 [ERROR]
> org.apache.flink.test.recovery.JobManagerHAProcessFailureRecoveryITCase
> 2021-11-08T23:27:32.9630643Z Nov 08 23:27:32 [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> 2021-11-08T23:27:32.9632522Z Nov 08 23:27:32 [ERROR] Command was /bin/sh -c
> cd /__w/2/s/flink-tests/target &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -jar
> /__w/2/s/flink-tests/target/surefire/surefirebooter6973421711355178650.jar
> /__w/2/s/flink-tests/target/surefire 2021-11-08T22-42-26_652-jvmRun2
> surefire3220623274989834473tmp surefire_1316080825708637501141tmp
> 2021-11-08T23:27:32.9633418Z Nov 08 23:27:32 [ERROR] Error occurred in
> starting fork, check output in log
> 2021-11-08T23:27:32.9635503Z Nov 08 23:27:32 [ERROR] Process Exit Code: 239
> 2021-11-08T23:27:32.9636287Z Nov 08 23:27:32 [ERROR] Crashed tests:
> 2021-11-08T23:27:32.9636863Z Nov 08 23:27:32 [ERROR]
> org.apache.flink.test.recovery.JobManagerHAProcessFailureRecoveryITCase
> 2021-11-08T23:27:32.9637609Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
> 2021-11-08T23:27:32.9638331Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:457)
> 2021-11-08T23:27:32.9638951Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:298)
> 2021-11-08T23:27:32.9639505Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:246)
> 2021-11-08T23:27:32.9640108Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
> 2021-11-08T23:27:32.9640916Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
> 2021-11-08T23:27:32.9641819Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
> 2021-11-08T23:27:32.9642447Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
> 2021-11-08T23:27:32.9643039Z Nov 08 23:27:32 [ERROR] at
> org.apache.maven.lif
[jira] [Created] (FLINK-24972) FlinkKafkaProducerITCase testScaleUpAfterScalingDown Timeout due to failed to complete checkpoint
Yun Gao created FLINK-24972:
---
Summary: FlinkKafkaProducerITCase testScaleUpAfterScalingDown
Timeout due to failed to complete checkpoint
Key: FLINK-24972
URL: https://issues.apache.org/jira/browse/FLINK-24972
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.14.0
Reporter: Yun Gao
{code:java}
Nov 20 23:38:33 [ERROR] Tests run: 15, Failures: 10, Errors: 1, Skipped: 0,
Time elapsed: 270.661 s <<< FAILURE! - in
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase
Nov 20 23:38:33 [ERROR] testScaleUpAfterScalingDown Time elapsed: 64.049 s
<<< ERROR!
Nov 20 23:38:33 org.apache.flink.runtime.checkpoint.CheckpointException: Could
not complete snapshot 1 for operator MockTask (2/4)#0. Failure reason:
Checkpoint was declined.
Nov 20 23:38:33 at
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:265)
{code}
After this failure the other tests failed with
{code:java}
Nov 20 23:38:33
Nov 20 23:38:33 [ERROR] testFailAndRecoverSameCheckpointTwice Time elapsed:
5.697 s <<< FAILURE!
Nov 20 23:38:33 java.lang.AssertionError: Detected producer leak. Thread name:
kafka-producer-network-thread |
producer-MockTask-002a002c-5
Nov 20 23:38:33 at org.junit.Assert.fail(Assert.java:89) {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17845: Add null check for temporal table check on SqlSnapshot
flinkbot edited a comment on pull request #17845: URL: https://github.com/apache/flink/pull/17845#issuecomment-974763929 ## CI report: * 5c35a875ef1d4c812a118d3a4c386afba1ca71d1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26787) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-24973) flink registercachedfile example no effect
anigkus created FLINK-24973:
---
Summary: flink registercachedfile example no effect
Key: FLINK-24973
URL: https://issues.apache.org/jira/browse/FLINK-24973
Project: Flink
Issue Type: Technical Debt
Components: API / Core
Affects Versions: shaded-14.0
Reporter: anigkus
{code:java}
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.myorg.quickstart;
import org.apache.commons.io.FileUtils;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.api.java.operators.DataSource;
import org.apache.flink.configuration.Configuration;
import java.io.File;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.List;
/**
* Copyright © DEEPEXI Technologies Co., Ltd. 2018-2020. All rights reserved.
*
* @Author zhangchunping
* @Date 11/21/21 12:30 PM
* @Description ???
*/
public class DistributedCache {
public static void main(String[] args) throws Exception {
ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
//1、register a file from HDFS
//env.registerCachedFile("hdfs:///path/to/your/file", "hdfsFile");
env.registerCachedFile("file:///tmp/tmp.txt", "localFile",true);
DataSource data = env.fromElements("a", "b", "c", "d");
data.map(new RichMapFunction() {
private List cache = new ArrayList();
@Override
public void open(Configuration parameters) throws Exception {
//super.open(parameters);
//2、access cached file via RuntimeContext and DistributedCache
File file =
getRuntimeContext().getDistributedCache().getFile("localFile");
List lines = FileUtils.readLines(file,"UTF-8");
for (String line : lines) {
cache.add(line);
System.out.println("line=[" + line + "]");
}
}
@Override
public String map(String value) throws Exception {
//value=["a", "b", "c", "d"];
return value;
}
}).print();
}
} {code}
#/tmp/tmp.txt–>this file existe
List lines = FileUtils.readLines(file,"UTF-8"); //lines.size()=0
Why
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24973) flink registercachedfile example no effect
[
https://issues.apache.org/jira/browse/FLINK-24973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anigkus updated FLINK-24973:
Description:
{code:java}
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.myorg.quickstart;
import org.apache.commons.io.FileUtils;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.api.java.operators.DataSource;
import org.apache.flink.configuration.Configuration;
import java.io.File;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.List;
public class DistributedCache {
public static void main(String[] args) throws Exception {
ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
//1、register a file from HDFS
//env.registerCachedFile("hdfs:///path/to/your/file", "hdfsFile");
env.registerCachedFile("file:///tmp/tmp.txt", "localFile",true);
DataSource data = env.fromElements("a", "b", "c", "d");
data.map(new RichMapFunction() {
private List cache = new ArrayList();
@Override
public void open(Configuration parameters) throws Exception {
//super.open(parameters);
//2、access cached file via RuntimeContext and DistributedCache
File file =
getRuntimeContext().getDistributedCache().getFile("localFile");
List lines = FileUtils.readLines(file,"UTF-8");
for (String line : lines) {
cache.add(line);
System.out.println("line=[" + line + "]");
}
}
@Override
public String map(String value) throws Exception {
//value=["a", "b", "c", "d"];
return value;
}
}).print();
}
} {code}
#/tmp/tmp.txt–>this file existe
List lines = FileUtils.readLines(file,"UTF-8"); //lines.size()=0
Why
was:
{code:java}
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.myorg.quickstart;
import org.apache.commons.io.FileUtils;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.api.java.operators.DataSource;
import org.apache.flink.configuration.Configuration;
import java.io.File;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.List;
/**
* Copyright © DEEPEXI Technologies Co., Ltd. 2018-2020. All rights reserved.
*
* @Author zhangchunping
* @Date 11/21/21 12:30 PM
* @Description ???
*/
public class DistributedCache {
public static void main(String[] args) throws Exception {
ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
//1、register a file from HDFS
//env.registerCachedFile("hdfs:///path/to/your/file", "hdfsFile");
env.registerCachedFile("file:///tmp/tmp.txt", "localFile",true);
DataSource data = env.fromElements("a", "b", "c", "d");
data.map(new RichMapFunction() {
private List cache = new ArrayList();
@Override
public void open(Configuration parameters) throws Exception {
//super.open(parameters);
//2、access cached file via RuntimeContext and DistributedCache
File file =
getRuntime
[jira] [Commented] (FLINK-24352) Lookup join + WITH clause throws NPE
[
https://issues.apache.org/jira/browse/FLINK-24352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447028#comment-17447028
]
Yuval Itzchakov commented on FLINK-24352:
-
[~TsReaper] It's ready: [https://github.com/apache/flink/pull/17845]
Not entirely sure that's the way we should approach this but let's get the ball
rolling.
> Lookup join + WITH clause throws NPE
>
>
> Key: FLINK-24352
> URL: https://issues.apache.org/jira/browse/FLINK-24352
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.0
>Reporter: Caizhi Weng
>Priority: Major
>
> Add the following SQL to
> {{org.apache.flink.table.api.TableEnvironmentITCase}} to reproduce this bug.
> {code:scala}
> @Test
> def myTest(): Unit = {
> tEnv.executeSql(
> """
> |CREATE TABLE T1 (
> | a INT,
> | b STRING,
> | proctime AS PROCTIME()
> |) WITH (
> | 'connector' = 'values',
> | 'bounded' = 'true'
> |)
> |""".stripMargin)
> tEnv.executeSql(
> """
> |CREATE TABLE T2 (
> | a INT,
> | b STRING
> |) WITH (
> | 'connector' = 'values',
> | 'bounded' = 'true'
> |)
> |""".stripMargin)
> tEnv.explainSql(
> """
> |WITH MyView(a, b) AS (SELECT a, b FROM T2)
> |SELECT * FROM T1 AS T
> |LEFT JOIN MyView FOR SYSTEM_TIME AS OF T.proctime AS D
> |ON T.a = D.a
> |""".stripMargin)
> }
> {code}
> The exception stack is
> {code}
> org.apache.flink.table.api.ValidationException: SQL validation failed. null
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:165)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:107)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:217)
> at
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:101)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.explainSql(TableEnvironmentImpl.java:686)
> // IDEA and Junit stacks are omitted
> Caused by: java.lang.NullPointerException
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSnapshot(SqlValidatorImpl.java:4714)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:986)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3085)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3070)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateJoin(SqlValidatorImpl.java:3133)
> at
> org.apache.flink.table.planner.calcite.FlinkCalciteSqlValidator.validateJoin(FlinkCalciteSqlValidator.java:117)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3076)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3335)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:997)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:975)
> at
> org.apache.calcite.sql.validate.WithNamespace.validateImpl(WithNamespace.java:57)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:997)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateWith(SqlValidatorImpl.java:3744)
> at org.apache.calcite.sql.SqlWith.validate(SqlWith.java:71)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:952)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:704)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:160)
> ... 43 more
> {code}
> However if we use {{CREATE VIEW}} statement to create this view it will run
> successfully.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] MonsterChenzhuo opened a new pull request #17847: [FLINK-6573][Connectors/Common ] Flink MongoDB Connector
MonsterChenzhuo opened a new pull request #17847: URL: https://github.com/apache/flink/pull/17847 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-6573: -- Labels: pull-request-available stale-assigned (was: stale-assigned) > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #17847: [FLINK-6573][Connectors/Common ] Flink MongoDB Connector
flinkbot commented on pull request #17847: URL: https://github.com/apache/flink/pull/17847#issuecomment-974817636 ## CI report: * e097b7b506b58adeb85d13eb90cfbb9e4eab3550 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
