Re: [PR] [FLINK-33704][BP 1.18][Filesytems] Update GCS filesystems to latest available versions [flink]
MartijnVisser commented on PR #23935: URL: https://github.com/apache/flink/pull/23935#issuecomment-1859729348 > Shall be #23837 Indeed, thanks for that! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33704][BP 1.18][Filesytems] Update GCS filesystems to latest available versions [flink]
MartijnVisser merged PR #23935: URL: https://github.com/apache/flink/pull/23935 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33704) Update GCS filesystems to latest available versions
[ https://issues.apache.org/jira/browse/FLINK-33704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-33704: --- Fix Version/s: 1.18.1 > Update GCS filesystems to latest available versions > --- > > Key: FLINK-33704 > URL: https://issues.apache.org/jira/browse/FLINK-33704 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / FileSystem, FileSystems >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0, 1.18.1 > > > Update GS SDK from 2.15.0 to 2.29.1 and GS Hadoop Connector from 2.2.15 to > 2.2.18 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (FLINK-33704) Update GCS filesystems to latest available versions
[ https://issues.apache.org/jira/browse/FLINK-33704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reopened FLINK-33704: > Update GCS filesystems to latest available versions > --- > > Key: FLINK-33704 > URL: https://issues.apache.org/jira/browse/FLINK-33704 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / FileSystem, FileSystems >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > Update GS SDK from 2.15.0 to 2.29.1 and GS Hadoop Connector from 2.2.15 to > 2.2.18 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33603][1.18][Filesystem] Bundle Guava for GCS connector [flink]
MartijnVisser closed pull request #23920: [FLINK-33603][1.18][Filesystem] Bundle Guava for GCS connector URL: https://github.com/apache/flink/pull/23920 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33603][1.18][Filesystem] Bundle Guava for GCS connector [flink]
MartijnVisser commented on PR #23920: URL: https://github.com/apache/flink/pull/23920#issuecomment-1859733229 Superseded by https://issues.apache.org/jira/browse/FLINK-33704 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [ FLINK-33603][FileSystems] shade guava in gs-fs filesystem [flink]
MartijnVisser commented on PR #23489: URL: https://github.com/apache/flink/pull/23489#issuecomment-1859734539 Superseded by https://issues.apache.org/jira/browse/FLINK-33704 - Thanks for the help/thinking on this @singhravidutt @singhravidutt @cnauroth Do you have any bandwidth to add a GCS connector test to Flink? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [ FLINK-33603][FileSystems] shade guava in gs-fs filesystem [flink]
MartijnVisser closed pull request #23489: [ FLINK-33603][FileSystems] shade guava in gs-fs filesystem URL: https://github.com/apache/flink/pull/23489 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25509) FLIP-208: Add RecordEvaluator to dynamically stop source based on de-serialized records
[
https://issues.apache.org/jira/browse/FLINK-25509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798044#comment-17798044
]
Martijn Visser commented on FLINK-25509:
[~ruanhang1993] [~lindong] Both the {{v3.0}} and the {{main}} branch of the
Flink Kafka connector work for Flink 1.18 (see
https://github.com/apache/flink-connector-kafka/actions/runs/7235099577 which
shows the nightly run). I'm inclined to make it only compatible with {{main}}
so that a new Flink Kafka connector v3.1 would be released. There are some
other changes already in {{main}} as well (different Kafka client)
> FLIP-208: Add RecordEvaluator to dynamically stop source based on
> de-serialized records
> ---
>
> Key: FLINK-25509
> URL: https://issues.apache.org/jira/browse/FLINK-25509
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Common, Connectors / Kafka
>Reporter: Dong Lin
>Assignee: Hang Ruan
>Priority: Major
> Labels: pull-request-available, stale-assigned
>
> This feature is needed to migrate applications which uses
> KafkaDeserializationSchema::isEndOfStream() from using FlinkKafkaConsumer to
> using KafkaSource.
> Please checkout
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-208%3A+Add+RecordEvaluator+to+dynamically+stop+source+based+on+de-serialized+records
> for the motivation and the proposed changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] Reduce getTable call for hive client and optimize graph generation time [flink]
hehuiyuan opened a new pull request, #23945: URL: https://github.com/apache/flink/pull/23945 ## What is the purpose of the change HiveCatalog.getHiveTable method wastes a lot of time when generate graph, because the number of calls is relatively high. I have an sql task with over 2000 rows, the HiveCatalog.getHiveTable method is called 4879 times , but only six hive tables were used.  The client.getTable method costs a lot of time. One call takes approximately 50 milliseconds, 4879 * 50 =243950ms = 243.95s = 4min ## Brief change log Reduce getTable call for hive client by using cache. ## Verifying this change ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-21949][table] Support ARRAY_AGG aggregate function [flink]
Jiabao-Sun commented on PR #23411: URL: https://github.com/apache/flink/pull/23411#issuecomment-1859780440 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-33871) Reduce getTable call for hive client and optimize graph generation time
hehuiyuan created FLINK-33871: - Summary: Reduce getTable call for hive client and optimize graph generation time Key: FLINK-33871 URL: https://issues.apache.org/jira/browse/FLINK-33871 Project: Flink Issue Type: Improvement Reporter: hehuiyuan HiveCatalog.getHiveTable method wastes a lot of time when generate graph, because the number of calls is relatively high. I have an sql task with over 2000 rows, the HiveCatalog.getHiveTable method is called 4879 times , but only six hive tables were used.  The client.getTable method costs a lot of time.  There is a statistic that jobmanager interacts with hive when generate graph. If One call takes approximately 50 milliseconds , How much time it spends : 4879 * 50 =243950ms = 243.95s = 4min We can cache and client.getTable method is only called six times. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33871) Reduce getTable call for hive client and optimize graph generation time
[ https://issues.apache.org/jira/browse/FLINK-33871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33871: --- Labels: pull-request-available (was: ) > Reduce getTable call for hive client and optimize graph generation time > --- > > Key: FLINK-33871 > URL: https://issues.apache.org/jira/browse/FLINK-33871 > Project: Flink > Issue Type: Improvement >Reporter: hehuiyuan >Priority: Major > Labels: pull-request-available > > HiveCatalog.getHiveTable method wastes a lot of time when generate graph, > because the number of calls is relatively high. > I have an sql task with over 2000 rows, the HiveCatalog.getHiveTable method > is called 4879 times , but only six hive tables were used. >  > The client.getTable method costs a lot of time. >  > There is a statistic that jobmanager interacts with hive when generate graph. > If One call takes approximately 50 milliseconds , > How much time it spends : 4879 * 50 =243950ms = 243.95s = 4min > We can cache and client.getTable method is only called six times. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33871][HIVE]Reduce getTable call for hive client and optimize graph generation time [flink]
flinkbot commented on PR #23945: URL: https://github.com/apache/flink/pull/23945#issuecomment-1859784457 ## CI report: * c6efa328604d58eb5ac27e39cef9f1c428b336b9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-31650][metrics][rest] Remove transient metrics for subtasks in terminal state [flink]
JunRuiLee commented on code in PR #23447:
URL: https://github.com/apache/flink/pull/23447#discussion_r1429717030
##
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/metrics/MetricStore.java:
##
@@ -126,9 +140,14 @@ public synchronized void
updateCurrentExecutionAttempts(Collection j
taskMetricStore.getSubtaskMetricStore(
subtaskIndex))
.ifPresent(
-subtaskMetricStore ->
-
subtaskMetricStore.retainAttempts(
-
attempts.getCurrentAttempts()));
+subtaskMetricStore -> {
+
subtaskMetricStore.retainAttempts(
Review Comment:
Thanks for clarification @X-czh .
I'm not quite certain I understand your concern.
In my opinion, this issue is unrelated to the web interface and more related
to the inconsistency in the MetricStore because the WebUI is also get data from
MetricStore. Specifically, the metrics in the subtaskMetricsStore are being
removed, while the metrics in the taskMetricsStore are not synchronously
removed, which could be confusing for users.
Based on your changes, you can perform the following test:
For a jobVertex that has already finished, you can use the
JobVertexMetricsHandler to retrieve subtask metrics like below:
http://localhost:8081/jobs//vertices//metrics?get=0.backPressuredTimeMsPerSecond,0.busyTimeMsPerSecond
Then, compare the results with the SubtaskMetricsHandler:
http://localhost:8081/jobs//vertices//subtasks/0/metrics?get=backPressuredTimeMsPerSecond,busyTimeMsPerSecond
The results from these two endpoints are different. In my local test, the
results are as shown in the attached image. I prefer that cleaning up should be
done simultaneously for both, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-25509) FLIP-208: Add RecordEvaluator to dynamically stop source based on de-serialized records
[ https://issues.apache.org/jira/browse/FLINK-25509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798047#comment-17798047 ] Dong Lin commented on FLINK-25509: -- [~martijnvisser] Just to make sure that I understand you correctly, do you mean it is OK to put this feature in flink-kafka-connector's main branch? This would keep its main branch compatible with Flink 1.18. But it won't be compatible with Flink 1.17. > FLIP-208: Add RecordEvaluator to dynamically stop source based on > de-serialized records > --- > > Key: FLINK-25509 > URL: https://issues.apache.org/jira/browse/FLINK-25509 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common, Connectors / Kafka >Reporter: Dong Lin >Assignee: Hang Ruan >Priority: Major > Labels: pull-request-available, stale-assigned > > This feature is needed to migrate applications which uses > KafkaDeserializationSchema::isEndOfStream() from using FlinkKafkaConsumer to > using KafkaSource. > Please checkout > https://cwiki.apache.org/confluence/display/FLINK/FLIP-208%3A+Add+RecordEvaluator+to+dynamically+stop+source+based+on+de-serialized+records > for the motivation and the proposed changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-25509) FLIP-208: Add RecordEvaluator to dynamically stop source based on de-serialized records
[
https://issues.apache.org/jira/browse/FLINK-25509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798066#comment-17798066
]
Martijn Visser commented on FLINK-25509:
[~lindong] Yes, that's what I suggest. The one thing I would like to check with
[~tzulitai] is if we want to cut {{main}} now first and create a Flink Kafka
v3.1 branch, which would (at this point) have support for both Flink 1.17 and
Flink 1.18. Then with the PR for this ticket opened, we would drop support for
Flink 1.17 and therefore release it as part of a new Flink Kafka v4.0
branch/release which only would support Flink 1.18 and newer. For users of
Flink 1.17, they would stick with the v3.1 version (until support for Flink
1.17 is stopped completely)
> FLIP-208: Add RecordEvaluator to dynamically stop source based on
> de-serialized records
> ---
>
> Key: FLINK-25509
> URL: https://issues.apache.org/jira/browse/FLINK-25509
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Common, Connectors / Kafka
>Reporter: Dong Lin
>Assignee: Hang Ruan
>Priority: Major
> Labels: pull-request-available, stale-assigned
>
> This feature is needed to migrate applications which uses
> KafkaDeserializationSchema::isEndOfStream() from using FlinkKafkaConsumer to
> using KafkaSource.
> Please checkout
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-208%3A+Add+RecordEvaluator+to+dynamically+stop+source+based+on+de-serialized+records
> for the motivation and the proposed changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-31650][metrics][rest] Remove transient metrics for subtasks in terminal state [flink]
X-czh commented on code in PR #23447:
URL: https://github.com/apache/flink/pull/23447#discussion_r1429759033
##
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/metrics/MetricStore.java:
##
@@ -126,9 +140,14 @@ public synchronized void
updateCurrentExecutionAttempts(Collection j
taskMetricStore.getSubtaskMetricStore(
subtaskIndex))
.ifPresent(
-subtaskMetricStore ->
-
subtaskMetricStore.retainAttempts(
-
attempts.getCurrentAttempts()));
+subtaskMetricStore -> {
+
subtaskMetricStore.retainAttempts(
Review Comment:
Hi @JunRuiLee, thanks for the clarification. I get your point and my concern
is actual about maintainability of the code in the future. When there're both
insertion and deletion operations, the duplication of task-level metrics
actually makes it more difficult to maintain consistency (as is the case here),
so I think it would be better to optimize the duplication issue here in the
future.
I'll clean up simultaneously for both cases first here, and create a new
issue for the optimization later. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] test request [flink]
singhravidutt closed pull request #23933: test request URL: https://github.com/apache/flink/pull/23933 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33632] Adding custom flink mutator [flink-kubernetes-operator]
AncyRominus commented on code in PR #733:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/733#discussion_r1429764133
##
flink-kubernetes-webhook/pom.xml:
##
@@ -127,6 +132,11 @@ under the License.
+
+org.apache.maven.plugins
+maven-surefire-plugin
+${surefire-plugin.version}
Review Comment:
This plugin is required for the successful execution of the test
AdmissionHandlerTest.testMutateHandler. Else it will throw <500 Internal Server
Error> while mutating the SessionJob as it’s using the deployment from the
informerManager which uses Kubernetes Client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-15571) Create a Redis Streams Connector for Flink
[ https://issues.apache.org/jira/browse/FLINK-15571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798078#comment-17798078 ] Martijn Visser commented on FLINK-15571: [~jinsuichen] If there were volunteering maintainers for this, probably yes > Create a Redis Streams Connector for Flink > -- > > Key: FLINK-15571 > URL: https://issues.apache.org/jira/browse/FLINK-15571 > Project: Flink > Issue Type: New Feature > Components: Connectors / Redis Streams >Reporter: Tugdual Grall >Assignee: ZhuoYu Chen >Priority: Minor > Labels: pull-request-available, stale-assigned > > Redis has a "log data structure" called Redis Streams, it would be nice to > integrate Redis Streams and Apache Flink as: > * Source > * Sink > See Redis Streams introduction: [https://redis.io/topics/streams-intro] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33632] Adding custom flink mutator [flink-kubernetes-operator]
AncyRominus commented on code in PR #733:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/733#discussion_r1429772463
##
flink-kubernetes-operator/pom.xml:
##
@@ -208,6 +208,18 @@ under the License.
junit-jupiter-params
test
+
+
+io.javaoperatorsdk
+kubernetes-webhooks-framework-core
+${operator.sdk.webhook-framework.version}
+
+
+*
+*
+
+
+
Review Comment:
This dependency is for the Kubernetes-webhooks-framework. This can be
removed as we don't have any strong relation with this dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-25509) FLIP-208: Add RecordEvaluator to dynamically stop source based on de-serialized records
[ https://issues.apache.org/jira/browse/FLINK-25509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798080#comment-17798080 ] Dong Lin commented on FLINK-25509: -- [~martijnvisser] Cool. I totally agree with your suggestion. > FLIP-208: Add RecordEvaluator to dynamically stop source based on > de-serialized records > --- > > Key: FLINK-25509 > URL: https://issues.apache.org/jira/browse/FLINK-25509 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common, Connectors / Kafka >Reporter: Dong Lin >Assignee: Hang Ruan >Priority: Major > Labels: pull-request-available, stale-assigned > > This feature is needed to migrate applications which uses > KafkaDeserializationSchema::isEndOfStream() from using FlinkKafkaConsumer to > using KafkaSource. > Please checkout > https://cwiki.apache.org/confluence/display/FLINK/FLIP-208%3A+Add+RecordEvaluator+to+dynamically+stop+source+based+on+de-serialized+records > for the motivation and the proposed changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33632] Adding custom flink mutator [flink-kubernetes-operator]
AncyRominus commented on PR #733: URL: https://github.com/apache/flink-kubernetes-operator/pull/733#issuecomment-1859901718 Thanks for the comments @tagarr and @gyfora !!! The modify labels are set in FlinkMutator.java and the other mutator changes (if any) be added to the default or the custom mutators. The default/custom mutator sessionJob changes are iterated and the mutateSessionJobs are processed. Yes, we can add the sessionjob mutation for Update as well as for Create operations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-31481][table] Support enhanced show databases syntax [flink]
jeyhunkarimov commented on PR #23612: URL: https://github.com/apache/flink/pull/23612#issuecomment-1859901892 Hi @jnh5y could you please check if your comments are addressed? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31049) Add support for Kafka record headers to KafkaSink
[ https://issues.apache.org/jira/browse/FLINK-31049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-31049: --- Fix Version/s: kafka-3.1.0 > Add support for Kafka record headers to KafkaSink > - > > Key: FLINK-31049 > URL: https://issues.apache.org/jira/browse/FLINK-31049 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Reporter: Alex Gout >Assignee: Alex Gout >Priority: Minor > Labels: KafkaSink, pull-request-available, stale-assigned > Fix For: kafka-3.1.0 > > Original Estimate: 48h > Remaining Estimate: 48h > > The default org.apache.flink.connector.kafka.sink.KafkaSink does not support > adding Kafka record headers. In some implementations, downstream consumers > might rely on Kafka record headers being set. > > A way to add Headers would be to create a custom > KafkaRecordSerializationSchema and inject that into the KafkaSink. > However, I'm assuming the KafkaRecordSerializationSchemaBuilder was added for > convenience and allows a more usable approach of creating a KafkaSink without > having to deal with details like the RecordProducer directly. This builder > does not support adding record headers. > This is where I think it should be added. > The code responsible for creating the Kafka record involves > org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchemaWrapper > where the RecordProducer is created. > It is relatively simple to add support for record headers by adding a > "HeaderProducer" to the KafkaRecordSerializationSchemaBuilder next to the key > and value serializers and using the appropriate RecordProducer constructor. > > The issue was discussed > [here|https://lists.apache.org/thread/shlbbcqho0q9w5shjwdlscnsywjvbfro]. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31049) Add support for Kafka record headers to KafkaSink
[ https://issues.apache.org/jira/browse/FLINK-31049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798082#comment-17798082 ] Martijn Visser commented on FLINK-31049: Merged by [~tzulitai] in apache/flink-connector-kafka:main a7785630e714af303b224c38d9a6caa89a551265 > Add support for Kafka record headers to KafkaSink > - > > Key: FLINK-31049 > URL: https://issues.apache.org/jira/browse/FLINK-31049 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Reporter: Alex Gout >Assignee: Alex Gout >Priority: Minor > Labels: KafkaSink, pull-request-available, stale-assigned > Fix For: kafka-3.1.0 > > Original Estimate: 48h > Remaining Estimate: 48h > > The default org.apache.flink.connector.kafka.sink.KafkaSink does not support > adding Kafka record headers. In some implementations, downstream consumers > might rely on Kafka record headers being set. > > A way to add Headers would be to create a custom > KafkaRecordSerializationSchema and inject that into the KafkaSink. > However, I'm assuming the KafkaRecordSerializationSchemaBuilder was added for > convenience and allows a more usable approach of creating a KafkaSink without > having to deal with details like the RecordProducer directly. This builder > does not support adding record headers. > This is where I think it should be added. > The code responsible for creating the Kafka record involves > org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchemaWrapper > where the RecordProducer is created. > It is relatively simple to add support for record headers by adding a > "HeaderProducer" to the KafkaRecordSerializationSchemaBuilder next to the key > and value serializers and using the appropriate RecordProducer constructor. > > The issue was discussed > [here|https://lists.apache.org/thread/shlbbcqho0q9w5shjwdlscnsywjvbfro]. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33819) Support setting CompressType in RocksDBStateBackend
[
https://issues.apache.org/jira/browse/FLINK-33819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798083#comment-17798083
]
Yue Ma commented on FLINK-33819:
[~masteryhx] Thanks for replying.
{quote}Linked FLINK-20684 as it has been discussed before.
{quote}
Sorry for missing the previous ticket and creating the duplicated one
{quote}Linked FLINK-11313 which talks about the LZ4 Compression which should be
more usable than Snappy.
{quote}
We did not use LZ4 in the production environment, and for small state jobs, we
directly turn off the compression. And for job with large states, we adopted a
compression algorithm based on Snappy optimization.
{quote}Do you have some test results on it ?
{quote}
Yes. we did some benchmark test for the *SnappyCompression* and *NoCompression*
And result show that and Read Performance. After turning off Compression, State
Benchmark read performance can be improved by *80% to 100%*
We also conducted end-to-end online job testing, and after turning off
Compression, {*}the CPU usage of the job decreased by 16%{*}, while the
Checkpoint Total Size increased by *4-5 times.*
It is obvious that closing Compression is not only about benefits, but also
brings some space amplification. What I want to express is that we may need to
provide such a configuration for users to balance how to exchange space for time
{quote}BTW, If we'd like to introduce such a option, it's better to guarantee
the compalibility.
{quote}
Sorry, I didn't understand the compatibility issue here. I understand that it
is compatible here. After switching the Compression Type, the newly generated
file will be compressed using the new Compress Type, and the existing file can
still be read and written with old Compress Type.
> Support setting CompressType in RocksDBStateBackend
> ---
>
> Key: FLINK-33819
> URL: https://issues.apache.org/jira/browse/FLINK-33819
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Affects Versions: 1.18.0
>Reporter: Yue Ma
>Priority: Major
> Fix For: 1.19.0
>
> Attachments: image-2023-12-14-11-32-32-968.png,
> image-2023-12-14-11-35-22-306.png
>
>
> Currently, RocksDBStateBackend does not support setting the compression
> level, and Snappy is used for compression by default. But we have some
> scenarios where compression will use a lot of CPU resources. Turning off
> compression can significantly reduce CPU overhead. So we may need to support
> a parameter for users to set the CompressType of Rocksdb.
> !image-2023-12-14-11-35-22-306.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-31049) Add support for Kafka record headers to KafkaSink
[ https://issues.apache.org/jira/browse/FLINK-31049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-31049. -- Resolution: Fixed > Add support for Kafka record headers to KafkaSink > - > > Key: FLINK-31049 > URL: https://issues.apache.org/jira/browse/FLINK-31049 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Reporter: Alex Gout >Assignee: Alex Gout >Priority: Minor > Labels: KafkaSink, pull-request-available, stale-assigned > Fix For: kafka-3.1.0 > > Original Estimate: 48h > Remaining Estimate: 48h > > The default org.apache.flink.connector.kafka.sink.KafkaSink does not support > adding Kafka record headers. In some implementations, downstream consumers > might rely on Kafka record headers being set. > > A way to add Headers would be to create a custom > KafkaRecordSerializationSchema and inject that into the KafkaSink. > However, I'm assuming the KafkaRecordSerializationSchemaBuilder was added for > convenience and allows a more usable approach of creating a KafkaSink without > having to deal with details like the RecordProducer directly. This builder > does not support adding record headers. > This is where I think it should be added. > The code responsible for creating the Kafka record involves > org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchemaWrapper > where the RecordProducer is created. > It is relatively simple to add support for record headers by adding a > "HeaderProducer" to the KafkaRecordSerializationSchemaBuilder next to the key > and value serializers and using the appropriate RecordProducer constructor. > > The issue was discussed > [here|https://lists.apache.org/thread/shlbbcqho0q9w5shjwdlscnsywjvbfro]. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33153) Kafka using latest-offset maybe missing data
[ https://issues.apache.org/jira/browse/FLINK-33153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-33153: --- Affects Version/s: (was: kafka-4.1.0) > Kafka using latest-offset maybe missing data > > > Key: FLINK-33153 > URL: https://issues.apache.org/jira/browse/FLINK-33153 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Reporter: tanjialiang >Priority: Minor > > When Kafka start with the latest-offset strategy, it does not fetch the > latest snapshot offset and specify it for consumption. Instead, it sets the > startingOffset to -1 (KafkaPartitionSplit.LATEST_OFFSET, which makes > currentOffset = -1, and call the KafkaConsumer's seekToEnd API). The > currentOffset is only set to the consumed offset + 1 when the task consumes > data, and this currentOffset is stored in the state during checkpointing. If > there are very few messages in Kafka and a partition has not consumed any > data, and I stop the task with a savepoint, then write data to that > partition, and start the task with the savepoint, the task will resume from > the saved state. Due to the startingOffset in the state being -1, it will > cause the task to miss the data that was written before the recovery point. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-31650][metrics][rest] Remove transient metrics for subtasks in terminal state [flink]
JunRuiLee commented on code in PR #23447:
URL: https://github.com/apache/flink/pull/23447#discussion_r1429785675
##
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/metrics/MetricStore.java:
##
@@ -126,9 +140,14 @@ public synchronized void
updateCurrentExecutionAttempts(Collection j
taskMetricStore.getSubtaskMetricStore(
subtaskIndex))
.ifPresent(
-subtaskMetricStore ->
-
subtaskMetricStore.retainAttempts(
-
attempts.getCurrentAttempts()));
+subtaskMetricStore -> {
+
subtaskMetricStore.retainAttempts(
Review Comment:
Clean up simultaneously for both cases first here sounds good to me, Please
feel free and go ahead~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798091#comment-17798091
]
david radley commented on FLINK-33365:
--
[~libenchao] [~Sergey Nuyanzin] Bringing back a discussion onto the issue. I
have many of the scenarios working including the Or scenario. There are
scenarios
like the one [https://lists.apache.org/thread/7r49c5ffohz1oqm3oz2g60g5lnovq0l2]
if the jdbc lookup source has rows that do not match the filter, then nulls
come out in the lookup join resultset. The same behaviour occurs with predicate
pushdown disabled and with my current patch (when I push it up).
On the discussion thread [~ruanhang1993] agreed that the nulls are not correct.
The way I am reading this is that, unless we have a reason to see these nulls,
there is an issue in lookup joins in certain scenarios. It does not appear to
relate to JDBC. Are we ok if I make this fix work the same as when predicate
pushdown is not enabled. And raise a separate issue for the lookup join which
appears not to be JDBC related.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Labels: pull-request-available
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-22765) ExceptionUtilsITCase.testIsMetaspaceOutOfMemoryError is unstable
[
https://issues.apache.org/jira/browse/FLINK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798092#comment-17798092
]
Sergey Nuyanzin commented on FLINK-22765:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55591&view=logs&j=a657ddbf-d986-5381-9649-342d9c92e7fb&t=dc085d4a-05c8-580e-06ab-21f5624dab16&l=8574
> ExceptionUtilsITCase.testIsMetaspaceOutOfMemoryError is unstable
>
>
> Key: FLINK-22765
> URL: https://issues.apache.org/jira/browse/FLINK-22765
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0, 1.13.5, 1.15.0, 1.17.2, 1.19.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Major
> Labels: pull-request-available, stale-assigned, test-stability
> Fix For: 1.14.0, 1.16.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18292&view=logs&j=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3&t=a99e99c7-21cd-5a1f-7274-585e62b72f56
> {code}
> May 25 00:56:38 java.lang.AssertionError:
> May 25 00:56:38
> May 25 00:56:38 Expected: is ""
> May 25 00:56:38 but: was "The system is out of resources.\nConsult the
> following stack trace for details."
> May 25 00:56:38 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> May 25 00:56:38 at org.junit.Assert.assertThat(Assert.java:956)
> May 25 00:56:38 at org.junit.Assert.assertThat(Assert.java:923)
> May 25 00:56:38 at
> org.apache.flink.runtime.util.ExceptionUtilsITCase.run(ExceptionUtilsITCase.java:94)
> May 25 00:56:38 at
> org.apache.flink.runtime.util.ExceptionUtilsITCase.testIsMetaspaceOutOfMemoryError(ExceptionUtilsITCase.java:70)
> May 25 00:56:38 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> May 25 00:56:38 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> May 25 00:56:38 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> May 25 00:56:38 at java.lang.reflect.Method.invoke(Method.java:498)
> May 25 00:56:38 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> May 25 00:56:38 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> May 25 00:56:38 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> May 25 00:56:38 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> May 25 00:56:38 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> May 25 00:56:38 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> May 25 00:56:38 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> May 25 00:56:38 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> May 25 00:56:38 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> May 25 00:56:38 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> May 25 00:56:38 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> May 25 00:56:38 at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> May 25 00:56:38 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> May 25 00:56:38 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> May 25 00:56:38 at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> May 25 00:56:38 at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> May 25 00:56:38 at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> May 25 00:56:38 at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> May 25 00:56:38 at
> org.apache.maven.surefire.boote
Re: [PR] [FLINK-30593][autoscaler] Improve restart time tracking [flink-kubernetes-operator]
mxm commented on code in PR #735:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/735#discussion_r1429815765
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScalerImpl.java:
##
@@ -182,22 +182,22 @@ private void runScalingLogic(Context ctx,
AutoscalerFlinkMetrics autoscalerMetri
jobTopology.getVerticesInTopologicalOrder(),
() -> lastEvaluatedMetrics.get(ctx.getJobKey()));
-if (!collectedMetrics.isFullyCollected()) {
-// We have done an upfront evaluation, but we are not ready for
scaling.
-resetRecommendedParallelism(evaluatedMetrics);
-return;
-}
-
var scalingHistory = getTrimmedScalingHistory(stateStore, ctx, now);
// A scaling tracking without an end time gets created whenever a
scaling decision is
// applied. Here, when the job transitions to RUNNING, we record the
time for it.
if (ctx.getJobStatus() == JobStatus.RUNNING) {
-if (scalingTracking.setEndTimeIfTrackedAndParallelismMatches(
+if
(scalingTracking.recordRestartDurationIfTrackedAndParallelismMatches(
now, jobTopology, scalingHistory)) {
stateStore.storeScalingTracking(ctx, scalingTracking);
}
}
Review Comment:
We don't need the RUNNING job state check. This block can be reduced to:
```java
if
(scalingTracking.recordRestartDurationIfTrackedAndParallelismMatches(
now, jobTopology, scalingHistory)) {
stateStore.storeScalingTracking(ctx, scalingTracking);
}
```
The reason is that this method only gets called when the job is in running
state (see line 99). Enforcing a RUNNING state has always been a precondition
for executing autoscaling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-30593][autoscaler] Improve restart time tracking [flink-kubernetes-operator]
mxm commented on code in PR #735:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/735#discussion_r1429815765
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScalerImpl.java:
##
@@ -182,22 +182,22 @@ private void runScalingLogic(Context ctx,
AutoscalerFlinkMetrics autoscalerMetri
jobTopology.getVerticesInTopologicalOrder(),
() -> lastEvaluatedMetrics.get(ctx.getJobKey()));
-if (!collectedMetrics.isFullyCollected()) {
-// We have done an upfront evaluation, but we are not ready for
scaling.
-resetRecommendedParallelism(evaluatedMetrics);
-return;
-}
-
var scalingHistory = getTrimmedScalingHistory(stateStore, ctx, now);
// A scaling tracking without an end time gets created whenever a
scaling decision is
// applied. Here, when the job transitions to RUNNING, we record the
time for it.
if (ctx.getJobStatus() == JobStatus.RUNNING) {
-if (scalingTracking.setEndTimeIfTrackedAndParallelismMatches(
+if
(scalingTracking.recordRestartDurationIfTrackedAndParallelismMatches(
now, jobTopology, scalingHistory)) {
stateStore.storeScalingTracking(ctx, scalingTracking);
}
}
Review Comment:
We don't need the RUNNING job state check. This block can be reduced to:
```java
if
(scalingTracking.recordRestartDurationIfTrackedAndParallelismMatches(
now, jobTopology, scalingHistory)) {
stateStore.storeScalingTracking(ctx, scalingTracking);
}
```
The reason is that this method only gets called when the job is in running
state (see line 99). Enforcing a RUNNING state has always been a precondition
for executing the autoscaling logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-30593][autoscaler] Improve restart time tracking [flink-kubernetes-operator]
mxm commented on PR #735: URL: https://github.com/apache/flink-kubernetes-operator/pull/735#issuecomment-1859964401 This needs a rebase. I'll run the tests afterwards. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33706) Build_wheels_on_macos fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-33706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798093#comment-17798093
]
Sergey Nuyanzin commented on FLINK-33706:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55593&view=logs&j=f73b5736-8355-5390-ec71-4dfdec0ce6c5&t=90f7230e-bf5a-531b-8566-ad48d3e03bbb&l=815
> Build_wheels_on_macos fails on AZP
> --
>
> Key: FLINK-33706
> URL: https://issues.apache.org/jira/browse/FLINK-33706
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Test Infrastructure
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Major
> Labels: test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55044&view=logs&j=f73b5736-8355-5390-ec71-4dfdec0ce6c5&t=90f7230e-bf5a-531b-8566-ad48d3e03bbb&l=427
> fails on AZP as
> {noformat}
>note: This error originates from a subprocess, and is likely not a
> problem with pip.
> ERROR: Failed cleaning build dir for crcmod
> Building wheel for dill (setup.py): started
> Building wheel for dill (setup.py): finished with status 'done'
> Created wheel for dill: filename=dill-0.0.0-py3-none-any.whl size=899
> sha256=39d0b4b66ce11f42313482f4ad825029e861fd6dab87a743a95d75a44a1fedd6
> Stored in directory:
> /Users/runner/Library/Caches/pip/wheels/07/35/78/e9004fa30578734db7f10e7a211605f3f0778d2bdde38a239d
> Building wheel for hdfs (setup.py): started
> Building wheel for hdfs (setup.py): finished with status 'done'
> Created wheel for hdfs: filename=UNKNOWN-0.0.0-py3-none-any.whl
> size=928
> sha256=cb3fd7d8c71b52bbc27cfb75842f9d4d9c6f3b847f3f4fe50323c945a0e38ccc
> Stored in directory:
> /Users/runner/Library/Caches/pip/wheels/68/dd/29/c1a590238f9ebbe4f7ee9b3583f5185d0b9577e23f05c990eb
> WARNING: Built wheel for hdfs is invalid: Wheel has unexpected file
> name: expected 'hdfs', got 'UNKNOWN'
> Building wheel for pymongo (pyproject.toml): started
> Building wheel for pymongo (pyproject.toml): finished with status
> 'done'
> Created wheel for pymongo:
> filename=pymongo-4.6.1-cp38-cp38-macosx_10_9_x86_64.whl size=478012
> sha256=5dfc6fdb6a8a399f8f9da44e28bae19be244b15c8000cd3b2d7d6ff513cc6277
> Stored in directory:
> /Users/runner/Library/Caches/pip/wheels/54/d8/0e/2a61e90bb3872d903b15eb3c94cb70f438fb8792a28fee7bb1
> Building wheel for docopt (setup.py): started
> Building wheel for docopt (setup.py): finished with status 'done'
> Created wheel for docopt: filename=UNKNOWN-0.0.0-py3-none-any.whl
> size=920
> sha256=612c56cd1a6344b8def6c4d3c3c1c8bb10e1f2b0d978fee0fc8b9281026e8288
> Stored in directory:
> /Users/runner/Library/Caches/pip/wheels/56/ea/58/ead137b087d9e326852a851351d1debf4ada529b6ac0ec4e8c
> WARNING: Built wheel for docopt is invalid: Wheel has unexpected file
> name: expected 'docopt', got 'UNKNOWN'
> Successfully built dill pymongo
> Failed to build fastavro crcmod hdfs docopt
> ERROR: Could not build wheels for fastavro, which is required to
> install pyproject.toml-based projects
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31472) AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
[
https://issues.apache.org/jira/browse/FLINK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798094#comment-17798094
]
Sergey Nuyanzin commented on FLINK-31472:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55601&view=logs&j=1c002d28-a73d-5309-26ee-10036d8476b4&t=d1c117a6-8f13-5466-55f0-d48dbb767fcd&l=10380
> AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
>
>
> Key: FLINK-31472
> URL: https://issues.apache.org/jira/browse/FLINK-31472
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.17.0, 1.16.1, 1.19.0
>Reporter: Ran Tao
>Priority: Major
> Labels: test-stability
>
> when run mvn clean test, this case failed occasionally.
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.955
> s <<< FAILURE! - in
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest
> [ERROR]
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
> Time elapsed: 0.492 s <<< ERROR!
> java.lang.IllegalStateException: Illegal thread detected. This method must be
> called from inside the mailbox thread!
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
> at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
> at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(Engi
[jira] [Updated] (FLINK-31472) AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
[
https://issues.apache.org/jira/browse/FLINK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin updated FLINK-31472:
Description:
when run mvn clean test, this case failed occasionally.
{noformat}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.955 s
<<< FAILURE! - in
org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest
[ERROR]
org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
Time elapsed: 0.492 s <<< ERROR!
java.lang.IllegalStateException: Illegal thread detected. This method must be
called from inside the mailbox thread!
at
org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
at
org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
at
org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
at
org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
at
org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
at
org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
at
org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at
org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
at
org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
at
org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
at
org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
at
org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
[jira] [Commented] (FLINK-22765) ExceptionUtilsITCase.testIsMetaspaceOutOfMemoryError is unstable
[
https://issues.apache.org/jira/browse/FLINK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798095#comment-17798095
]
Sergey Nuyanzin commented on FLINK-22765:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55601&view=logs&j=a657ddbf-d986-5381-9649-342d9c92e7fb&t=dc085d4a-05c8-580e-06ab-21f5624dab16&l=8573
> ExceptionUtilsITCase.testIsMetaspaceOutOfMemoryError is unstable
>
>
> Key: FLINK-22765
> URL: https://issues.apache.org/jira/browse/FLINK-22765
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0, 1.13.5, 1.15.0, 1.17.2, 1.19.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Major
> Labels: pull-request-available, stale-assigned, test-stability
> Fix For: 1.14.0, 1.16.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18292&view=logs&j=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3&t=a99e99c7-21cd-5a1f-7274-585e62b72f56
> {code}
> May 25 00:56:38 java.lang.AssertionError:
> May 25 00:56:38
> May 25 00:56:38 Expected: is ""
> May 25 00:56:38 but: was "The system is out of resources.\nConsult the
> following stack trace for details."
> May 25 00:56:38 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> May 25 00:56:38 at org.junit.Assert.assertThat(Assert.java:956)
> May 25 00:56:38 at org.junit.Assert.assertThat(Assert.java:923)
> May 25 00:56:38 at
> org.apache.flink.runtime.util.ExceptionUtilsITCase.run(ExceptionUtilsITCase.java:94)
> May 25 00:56:38 at
> org.apache.flink.runtime.util.ExceptionUtilsITCase.testIsMetaspaceOutOfMemoryError(ExceptionUtilsITCase.java:70)
> May 25 00:56:38 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> May 25 00:56:38 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> May 25 00:56:38 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> May 25 00:56:38 at java.lang.reflect.Method.invoke(Method.java:498)
> May 25 00:56:38 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> May 25 00:56:38 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> May 25 00:56:38 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> May 25 00:56:38 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> May 25 00:56:38 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> May 25 00:56:38 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> May 25 00:56:38 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> May 25 00:56:38 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> May 25 00:56:38 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> May 25 00:56:38 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> May 25 00:56:38 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> May 25 00:56:38 at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> May 25 00:56:38 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> May 25 00:56:38 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> May 25 00:56:38 at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> May 25 00:56:38 at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> May 25 00:56:38 at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> May 25 00:56:38 at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> May 25 00:56:38 at
> org.apache.maven.surefire.boote
[jira] [Commented] (FLINK-31472) AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
[
https://issues.apache.org/jira/browse/FLINK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798096#comment-17798096
]
Sergey Nuyanzin commented on FLINK-31472:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55602&view=logs&j=1c002d28-a73d-5309-26ee-10036d8476b4&t=d1c117a6-8f13-5466-55f0-d48dbb767fcd&l=10634
> AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
>
>
> Key: FLINK-31472
> URL: https://issues.apache.org/jira/browse/FLINK-31472
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.17.0, 1.16.1, 1.19.0
>Reporter: Ran Tao
>Priority: Major
> Labels: test-stability
>
> when run mvn clean test, this case failed occasionally.
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.955
> s <<< FAILURE! - in
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest
> [ERROR]
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
> Time elapsed: 0.492 s <<< ERROR!
> java.lang.IllegalStateException: Illegal thread detected. This method must be
> called from inside the mailbox thread!
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
> at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
> at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(Engi
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
mxm commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1426756716
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/utils/AutoScalerUtils.java:
##
@@ -94,4 +99,106 @@ public static boolean excludeVerticesFromScaling(
conf.set(AutoScalerOptions.VERTEX_EXCLUDE_IDS, new
ArrayList<>(excludedIds));
return anyAdded;
}
+
+/** Quartz doesn't have the invertTimeRange flag so rewrite this method. */
+static boolean isTimeIncluded(CronCalendar cron, long timeInMillis) {
+if (cron.getBaseCalendar() != null
+&& !cron.getBaseCalendar().isTimeIncluded(timeInMillis)) {
+return false;
+} else {
+return cron.getCronExpression().isSatisfiedBy(new
Date(timeInMillis));
+}
+}
+
+static Optional interpretAsDaily(String subExpression) {
+String[] splits = subExpression.split("-");
+if (splits.length != 2) {
+return Optional.empty();
+}
+try {
+DailyCalendar daily = new DailyCalendar(splits[0], splits[1]);
+daily.setInvertTimeRange(true);
+return Optional.of(daily);
+} catch (Exception e) {
+return Optional.empty();
+}
+}
+
+static Optional interpretAsCron(String subExpression) {
+try {
+return Optional.of(new CronCalendar(subExpression));
+} catch (Exception e) {
+return Optional.empty();
Review Comment:
I think we should log this exception. Otherwise it is going to be hard to
figure out what is wrong with the cron string. This also applies to all other
methods in this class which have this pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-32523) NotifyCheckpointAbortedITCase.testNotifyCheckpointAborted fails with timeout on AZP
[
https://issues.apache.org/jira/browse/FLINK-32523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798098#comment-17798098
]
Sergey Nuyanzin commented on FLINK-32523:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55602&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=8330
> NotifyCheckpointAbortedITCase.testNotifyCheckpointAborted fails with timeout
> on AZP
> ---
>
> Key: FLINK-32523
> URL: https://issues.apache.org/jira/browse/FLINK-32523
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.2, 1.18.0, 1.17.1, 1.19.0
>Reporter: Sergey Nuyanzin
>Assignee: Hangxiang Yu
>Priority: Critical
> Labels: pull-request-available, stale-assigned, test-stability
> Attachments: failure.log
>
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50795&view=logs&j=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3&t=0c010d0c-3dec-5bf1-d408-7b18988b1b2b&l=8638
> fails with timeout
> {noformat}
> Jul 03 01:26:35 org.junit.runners.model.TestTimedOutException: test timed out
> after 10 milliseconds
> Jul 03 01:26:35 at java.lang.Object.wait(Native Method)
> Jul 03 01:26:35 at java.lang.Object.wait(Object.java:502)
> Jul 03 01:26:35 at
> org.apache.flink.core.testutils.OneShotLatch.await(OneShotLatch.java:61)
> Jul 03 01:26:35 at
> org.apache.flink.test.checkpointing.NotifyCheckpointAbortedITCase.verifyAllOperatorsNotifyAborted(NotifyCheckpointAbortedITCase.java:198)
> Jul 03 01:26:35 at
> org.apache.flink.test.checkpointing.NotifyCheckpointAbortedITCase.testNotifyCheckpointAborted(NotifyCheckpointAbortedITCase.java:189)
> Jul 03 01:26:35 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jul 03 01:26:35 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jul 03 01:26:35 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jul 03 01:26:35 at java.lang.reflect.Method.invoke(Method.java:498)
> Jul 03 01:26:35 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Jul 03 01:26:35 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jul 03 01:26:35 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Jul 03 01:26:35 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jul 03 01:26:35 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> Jul 03 01:26:35 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> Jul 03 01:26:35 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Jul 03 01:26:35 at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-32523) NotifyCheckpointAbortedITCase.testNotifyCheckpointAborted fails with timeout on AZP
[
https://issues.apache.org/jira/browse/FLINK-32523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798098#comment-17798098
]
Sergey Nuyanzin edited comment on FLINK-32523 at 12/18/23 9:59 AM:
---
1.18:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55602&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=8330
was (Author: sergey nuyanzin):
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55602&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=8330
> NotifyCheckpointAbortedITCase.testNotifyCheckpointAborted fails with timeout
> on AZP
> ---
>
> Key: FLINK-32523
> URL: https://issues.apache.org/jira/browse/FLINK-32523
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.2, 1.18.0, 1.17.1, 1.19.0
>Reporter: Sergey Nuyanzin
>Assignee: Hangxiang Yu
>Priority: Critical
> Labels: pull-request-available, stale-assigned, test-stability
> Attachments: failure.log
>
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50795&view=logs&j=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3&t=0c010d0c-3dec-5bf1-d408-7b18988b1b2b&l=8638
> fails with timeout
> {noformat}
> Jul 03 01:26:35 org.junit.runners.model.TestTimedOutException: test timed out
> after 10 milliseconds
> Jul 03 01:26:35 at java.lang.Object.wait(Native Method)
> Jul 03 01:26:35 at java.lang.Object.wait(Object.java:502)
> Jul 03 01:26:35 at
> org.apache.flink.core.testutils.OneShotLatch.await(OneShotLatch.java:61)
> Jul 03 01:26:35 at
> org.apache.flink.test.checkpointing.NotifyCheckpointAbortedITCase.verifyAllOperatorsNotifyAborted(NotifyCheckpointAbortedITCase.java:198)
> Jul 03 01:26:35 at
> org.apache.flink.test.checkpointing.NotifyCheckpointAbortedITCase.testNotifyCheckpointAborted(NotifyCheckpointAbortedITCase.java:189)
> Jul 03 01:26:35 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jul 03 01:26:35 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jul 03 01:26:35 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jul 03 01:26:35 at java.lang.reflect.Method.invoke(Method.java:498)
> Jul 03 01:26:35 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Jul 03 01:26:35 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jul 03 01:26:35 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Jul 03 01:26:35 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jul 03 01:26:35 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> Jul 03 01:26:35 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> Jul 03 01:26:35 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Jul 03 01:26:35 at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
mxm commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1429838068
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingMetricCollector.java:
##
@@ -143,7 +145,11 @@ public CollectedMetricHistory updateMetrics(
if (now.isBefore(windowFullTime)) {
LOG.info("Metric window not full until {}", windowFullTime);
} else {
-collectedMetrics.setFullyCollected(true);
+if (isExcluded) {
+LOG.info("autoscaling now in excluded period");
Review Comment:
```suggestion
LOG.info("Autoscaling on halt based on exclusion rule {}",
conf.get(AutoScalerOptions. EXCLUDED_PERIODS));
```
I think this could be helpful for debugging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
mxm commented on PR #728: URL: https://github.com/apache/flink-kubernetes-operator/pull/728#issuecomment-1859986239 Thanks @flashJd! Changes look great. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32877][Filesystem]add HTTP options to gcs-cloud-storage client [flink]
singhravidutt commented on PR #23226: URL: https://github.com/apache/flink/pull/23226#issuecomment-1859988955 > Also @singhravidutt what's your Jira user name, so I can assign this ticket to you? I don't have a Jira account. @cnauroth can you take the ownership of jira ticket? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33449) Support array_contains_seq function
[ https://issues.apache.org/jira/browse/FLINK-33449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798100#comment-17798100 ] leo cheng commented on FLINK-33449: --- Hi [~xuyangzhong] Thank you Pls do it. > Support array_contains_seq function > --- > > Key: FLINK-33449 > URL: https://issues.apache.org/jira/browse/FLINK-33449 > Project: Flink > Issue Type: New Feature >Reporter: leo cheng >Priority: Minor > Labels: pull-request-available > > support function array_contains_seq like trino contains_sequence > trino: > https://trino.io/docs/current/functions/array.html?highlight=contains_sequence#contains_sequence -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31472) AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
[
https://issues.apache.org/jira/browse/FLINK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798101#comment-17798101
]
Ahmed Hamdy commented on FLINK-31472:
-
hi [~Sergey Nuyanzin] could you assign me the ticket, I will take a look ASAP.
> AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
>
>
> Key: FLINK-31472
> URL: https://issues.apache.org/jira/browse/FLINK-31472
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.17.0, 1.16.1, 1.19.0
>Reporter: Ran Tao
>Priority: Major
> Labels: test-stability
>
> when run mvn clean test, this case failed occasionally.
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.955
> s <<< FAILURE! - in
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest
> [ERROR]
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
> Time elapsed: 0.492 s <<< ERROR!
> java.lang.IllegalStateException: Illegal thread detected. This method must be
> called from inside the mailbox thread!
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
> at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
> at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
> at
> org.junit.platform.launcher.core.EngineExecutionOrche
[jira] [Commented] (FLINK-33858) CI fails with No space left on device
[ https://issues.apache.org/jira/browse/FLINK-33858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798102#comment-17798102 ] Jing Ge commented on FLINK-33858: - Does anyone know if we could just delete /home/agent03/myagent/_work/_temp/debug_files/mvn-1.log? Or we must archive i.e. zip it? Thanks! > CI fails with No space left on device > - > > Key: FLINK-33858 > URL: https://issues.apache.org/jira/browse/FLINK-33858 > Project: Flink > Issue Type: Bug > Components: Build System / CI >Reporter: Sergey Nuyanzin >Priority: Blocker > > AlibabaCI003-agent01 > AlibabaCI003-agent03 > AlibabaCI003-agent05 > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8&view=logs&j=0c940707-2659-5648-cbe6-a1ad63045f0a&t=075c2716-8010-5565-fe08-3c4bb45824a4&l=9765] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798105#comment-17798105
]
Sergey Nuyanzin edited comment on FLINK-33365 at 12/18/23 10:25 AM:
Since
1. predicate push down from one side brings so many issues and probably there
is something else which is not known yet.
2. IIRC this issue was a blocker before jdbc connector release
I tend to think
1. to revert/disable push down commit
((https://github.com/apache/flink-connector-jdbc/commit/3e3b40e8cfadfc16a8ab74d4ef6a3ab3ceafa57b)
2. make a release
3. continue working on pushdown
what do you think?
was (Author: sergey nuyanzin):
Since
1. predicate push down from one side brings so many issues and probably there
is something else which is not known yet.
2. IIRC this issue was a blocker before jdbc connector release
I tend to think
1. to revert push down commit
((https://github.com/apache/flink-connector-jdbc/commit/3e3b40e8cfadfc16a8ab74d4ef6a3ab3ceafa57b)
2. make a release
3. continue working on pushdown
what do you think?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Labels: pull-request-available
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798105#comment-17798105
]
Sergey Nuyanzin commented on FLINK-33365:
-
Since
1. predicate push down from one side brings so many issues and probably there
is something else which is not known yet.
2. IIRC this issue was a blocker before jdbc connector release
I tend to think
1. to revert push down commit
((https://github.com/apache/flink-connector-jdbc/commit/3e3b40e8cfadfc16a8ab74d4ef6a3ab3ceafa57b)
2. make a release
3. continue working on pushdown
what do you think?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Labels: pull-request-available
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33858) CI fails with No space left on device
[ https://issues.apache.org/jira/browse/FLINK-33858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798108#comment-17798108 ] Sergey Nuyanzin commented on FLINK-33858: - zip would be the safest otherwise if it relates to successful build then it could be removed if it relates to failure then probably would make sense probably to attach here zip version for all mvn-*, not only mvn-1.log in case the size is suitable > CI fails with No space left on device > - > > Key: FLINK-33858 > URL: https://issues.apache.org/jira/browse/FLINK-33858 > Project: Flink > Issue Type: Bug > Components: Build System / CI >Reporter: Sergey Nuyanzin >Priority: Blocker > > AlibabaCI003-agent01 > AlibabaCI003-agent03 > AlibabaCI003-agent05 > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8&view=logs&j=0c940707-2659-5648-cbe6-a1ad63045f0a&t=075c2716-8010-5565-fe08-3c4bb45824a4&l=9765] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33704) Update GCS filesystems to latest available versions
[ https://issues.apache.org/jira/browse/FLINK-33704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798109#comment-17798109 ] Jing Ge commented on FLINK-33704: - Thanks [~martijnvisser] for fixing it! Can we resolve the ticket now? > Update GCS filesystems to latest available versions > --- > > Key: FLINK-33704 > URL: https://issues.apache.org/jira/browse/FLINK-33704 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / FileSystem, FileSystems >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0, 1.18.1 > > > Update GS SDK from 2.15.0 to 2.29.1 and GS Hadoop Connector from 2.2.15 to > 2.2.18 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-31472) AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
[
https://issues.apache.org/jira/browse/FLINK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin reassigned FLINK-31472:
---
Assignee: Ahmed Hamdy
> AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
>
>
> Key: FLINK-31472
> URL: https://issues.apache.org/jira/browse/FLINK-31472
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.17.0, 1.16.1, 1.19.0
>Reporter: Ran Tao
>Assignee: Ahmed Hamdy
>Priority: Major
> Labels: test-stability
>
> when run mvn clean test, this case failed occasionally.
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.955
> s <<< FAILURE! - in
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest
> [ERROR]
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
> Time elapsed: 0.492 s <<< ERROR!
> java.lang.IllegalStateException: Illegal thread detected. This method must be
> called from inside the mailbox thread!
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
> at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
> at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
[jira] [Commented] (FLINK-31472) AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
[
https://issues.apache.org/jira/browse/FLINK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798112#comment-17798112
]
Sergey Nuyanzin commented on FLINK-31472:
-
[~chalixar], thanks for volunteering, done
> AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
>
>
> Key: FLINK-31472
> URL: https://issues.apache.org/jira/browse/FLINK-31472
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.17.0, 1.16.1, 1.19.0
>Reporter: Ran Tao
>Assignee: Ahmed Hamdy
>Priority: Major
> Labels: test-stability
>
> when run mvn clean test, this case failed occasionally.
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.955
> s <<< FAILURE! - in
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest
> [ERROR]
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
> Time elapsed: 0.492 s <<< ERROR!
> java.lang.IllegalStateException: Illegal thread detected. This method must be
> called from inside the mailbox thread!
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
> at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
> at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
> at
> org.junit.platform.launcher.core.EngineExecut
Re: [PR] [FLINK-33588][Flink-Runtime] Fix NullArgumentException of getQuantile method in DescriptiveStatisticsHistogramStatistics [flink]
JingGe merged PR #23931: URL: https://github.com/apache/flink/pull/23931 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33588) Fix Flink Checkpointing Statistics Bug
[ https://issues.apache.org/jira/browse/FLINK-33588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798114#comment-17798114 ] Jing Ge commented on FLINK-33588: - master: 5e1deeea19c2e73f3bc6f1f9881a778d13abf6a4 > Fix Flink Checkpointing Statistics Bug > -- > > Key: FLINK-33588 > URL: https://issues.apache.org/jira/browse/FLINK-33588 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.14.5, 1.16.0, 1.17.0, 1.15.2, 1.14.6, 1.18.0, 1.17.1 >Reporter: Tongtong Zhu >Assignee: Tongtong Zhu >Priority: Critical > Labels: pull-request-available > Fix For: 1.19.0, 1.18.1 > > Attachments: FLINK-33588.patch, image-2023-12-11-17-35-23-391.png, > image-2023-12-13-11-35-43-780.png, image-2023-12-15-13-59-28-201.png, > newCommit-FLINK-33688.patch > > > When the Flink task is first started, the checkpoint data is null due to the > lack of data, and Percentile throws a null pointer exception when calculating > the percentage. After multiple tests, I found that it is necessary to set an > initial value for the statistical data value of the checkpoint when the > checkpoint data is null (i.e. at the beginning of the task) to solve this > problem. > The following is an abnormal description of the bug： > 2023-09-13 15:02:54,608 ERROR > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler > [] - Unhandled exception. > org.apache.commons.math3.exception.NullArgumentException: input array > at > org.apache.commons.math3.util.MathArrays.verifyValues(MathArrays.java:1650) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.commons.math3.stat.descriptive.AbstractUnivariateStatistic.test(AbstractUnivariateStatistic.java:158) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.commons.math3.stat.descriptive.rank.Percentile.evaluate(Percentile.java:272) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.commons.math3.stat.descriptive.rank.Percentile.evaluate(Percentile.java:241) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogramStatistics$CommonMetricsSnapshot.getPercentile(DescriptiveStatisticsHistogramStatistics.java:159) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogramStatistics.getQuantile(DescriptiveStatisticsHistogramStatistics.java:53) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.checkpoint.StatsSummarySnapshot.getQuantile(StatsSummarySnapshot.java:108) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.messages.checkpoints.StatsSummaryDto.valueOf(StatsSummaryDto.java:81) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.createCheckpointingStatistics(CheckpointingStatisticsHandler.java:129) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleRequest(CheckpointingStatisticsHandler.java:84) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleRequest(CheckpointingStatisticsHandler.java:58) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.AbstractAccessExecutionGraphHandler.handleRequest(AbstractAccessExecutionGraphHandler.java:68) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.lambda$handleRequest$0(AbstractExecutionGraphHandler.java:87) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) > [?:1.8.0_151] > at > java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) > [?:1.8.0_151] > at > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) > [?:1.8.0_151] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [?:1.8.0_151] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_151] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [?:1.8.0_151] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [?:1.8.0_151] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [?:1.8.0_151] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [?:1.8.0_151] > at java.lan
[jira] [Updated] (FLINK-31472) AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
[
https://issues.apache.org/jira/browse/FLINK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-31472:
---
Labels: pull-request-available test-stability (was: test-stability)
> AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
>
>
> Key: FLINK-31472
> URL: https://issues.apache.org/jira/browse/FLINK-31472
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.17.0, 1.16.1, 1.19.0
>Reporter: Ran Tao
>Assignee: Ahmed Hamdy
>Priority: Major
> Labels: pull-request-available, test-stability
>
> when run mvn clean test, this case failed occasionally.
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.955
> s <<< FAILURE! - in
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest
> [ERROR]
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
> Time elapsed: 0.492 s <<< ERROR!
> java.lang.IllegalStateException: Illegal thread detected. This method must be
> called from inside the mailbox thread!
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
> at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
> at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
> at
> org.junit.platform.launcher.core.EngineExecutionOrch
[PR] [FLINK-31472][Connectors/Base] Update AsyncSink Throttling test to use concurrent mailbox [flink]
vahmed-hamdy opened a new pull request, #23946: URL: https://github.com/apache/flink/pull/23946 ## What is the purpose of the change This pull request updates `AsyncSinkThrottlingTest` to use concurrent `TestSinkInitContextAnyThreadMailbox` due the concurrent nature of the test and to address CI stability issues. ## Brief change log - Updated `AsyncSinkThrottlingTest` to use concurrent `TestSinkInitContextAnyThreadMailbox` ## Verifying this change This change is a trivial rework for test suite, changes are verified by rerunning tests in debug mode. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive):no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33588) Fix Flink Checkpointing Statistics Bug
[ https://issues.apache.org/jira/browse/FLINK-33588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798115#comment-17798115 ] Jing Ge commented on FLINK-33588: - [~zhutong66] could you please cp the change to 1.18? > Fix Flink Checkpointing Statistics Bug > -- > > Key: FLINK-33588 > URL: https://issues.apache.org/jira/browse/FLINK-33588 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.14.5, 1.16.0, 1.17.0, 1.15.2, 1.14.6, 1.18.0, 1.17.1 >Reporter: Tongtong Zhu >Assignee: Tongtong Zhu >Priority: Critical > Labels: pull-request-available > Fix For: 1.19.0, 1.18.1 > > Attachments: FLINK-33588.patch, image-2023-12-11-17-35-23-391.png, > image-2023-12-13-11-35-43-780.png, image-2023-12-15-13-59-28-201.png, > newCommit-FLINK-33688.patch > > > When the Flink task is first started, the checkpoint data is null due to the > lack of data, and Percentile throws a null pointer exception when calculating > the percentage. After multiple tests, I found that it is necessary to set an > initial value for the statistical data value of the checkpoint when the > checkpoint data is null (i.e. at the beginning of the task) to solve this > problem. > The following is an abnormal description of the bug： > 2023-09-13 15:02:54,608 ERROR > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler > [] - Unhandled exception. > org.apache.commons.math3.exception.NullArgumentException: input array > at > org.apache.commons.math3.util.MathArrays.verifyValues(MathArrays.java:1650) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.commons.math3.stat.descriptive.AbstractUnivariateStatistic.test(AbstractUnivariateStatistic.java:158) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.commons.math3.stat.descriptive.rank.Percentile.evaluate(Percentile.java:272) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.commons.math3.stat.descriptive.rank.Percentile.evaluate(Percentile.java:241) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogramStatistics$CommonMetricsSnapshot.getPercentile(DescriptiveStatisticsHistogramStatistics.java:159) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogramStatistics.getQuantile(DescriptiveStatisticsHistogramStatistics.java:53) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.checkpoint.StatsSummarySnapshot.getQuantile(StatsSummarySnapshot.java:108) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.messages.checkpoints.StatsSummaryDto.valueOf(StatsSummaryDto.java:81) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.createCheckpointingStatistics(CheckpointingStatisticsHandler.java:129) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleRequest(CheckpointingStatisticsHandler.java:84) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleRequest(CheckpointingStatisticsHandler.java:58) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.AbstractAccessExecutionGraphHandler.handleRequest(AbstractAccessExecutionGraphHandler.java:68) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.lambda$handleRequest$0(AbstractExecutionGraphHandler.java:87) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) > [?:1.8.0_151] > at > java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) > [?:1.8.0_151] > at > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) > [?:1.8.0_151] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [?:1.8.0_151] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_151] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [?:1.8.0_151] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [?:1.8.0_151] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [?:1.8.0_151] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [?:1.8.0_151] > at java
Re: [PR] [FLINK-31472][Connectors/Base] Update AsyncSink Throttling test to use concurrent mailbox [flink]
flinkbot commented on PR #23946: URL: https://github.com/apache/flink/pull/23946#issuecomment-1860072502 ## CI report: * 4f0d2bc42eef189b04854b445eea2e7c7470933d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33704) Update GCS filesystems to latest available versions
[ https://issues.apache.org/jira/browse/FLINK-33704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798124#comment-17798124 ] Martijn Visser commented on FLINK-33704: [~jingge] I'm trying to confirm if this issue occurs in 1.17 or not. > Update GCS filesystems to latest available versions > --- > > Key: FLINK-33704 > URL: https://issues.apache.org/jira/browse/FLINK-33704 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / FileSystem, FileSystems >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0, 1.18.1 > > > Update GS SDK from 2.15.0 to 2.29.1 and GS Hadoop Connector from 2.2.15 to > 2.2.18 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33704) Update GCS filesystems to latest available versions
[ https://issues.apache.org/jira/browse/FLINK-33704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798126#comment-17798126 ] Martijn Visser commented on FLINK-33704: [~jingge] Closing this issue, it doesn't appear in 1.17.2 !screenshot-1.png! > Update GCS filesystems to latest available versions > --- > > Key: FLINK-33704 > URL: https://issues.apache.org/jira/browse/FLINK-33704 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / FileSystem, FileSystems >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0, 1.18.1 > > Attachments: screenshot-1.png > > > Update GS SDK from 2.15.0 to 2.29.1 and GS Hadoop Connector from 2.2.15 to > 2.2.18 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33704) Update GCS filesystems to latest available versions
[ https://issues.apache.org/jira/browse/FLINK-33704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-33704: --- Attachment: screenshot-1.png > Update GCS filesystems to latest available versions > --- > > Key: FLINK-33704 > URL: https://issues.apache.org/jira/browse/FLINK-33704 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / FileSystem, FileSystems >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0, 1.18.1 > > Attachments: screenshot-1.png > > > Update GS SDK from 2.15.0 to 2.29.1 and GS Hadoop Connector from 2.2.15 to > 2.2.18 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33603) Fix guava shading for GCS connector
[
https://issues.apache.org/jira/browse/FLINK-33603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser closed FLINK-33603.
--
Resolution: Resolved
FLINK-33704
> Fix guava shading for GCS connector
> ---
>
> Key: FLINK-33603
> URL: https://issues.apache.org/jira/browse/FLINK-33603
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.17.1
>Reporter: Jayadeep Jayaraman
>Priority: Major
> Labels: pull-request-available
>
> GCS connector has guava shading issue, This change introduced dependency on
> guava version({{{}31.1-jre{}}}) required by {{{}google-cloud-storage{}}}.
> Upgrade of {{google-cloud-storage}} lead to runtime failure because of new
> functionalities added in {{{}31.1-jre{}}}.
> This change pins guava version to the one required by storage client
> specifically in {{{}flink-gs-fs-hadoop{}}}, leaving all other filesystem
> implementation untouched.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33704) Update GCS filesystems to latest available versions
[ https://issues.apache.org/jira/browse/FLINK-33704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798126#comment-17798126 ] Martijn Visser edited comment on FLINK-33704 at 12/18/23 10:48 AM: --- [~jingge] Closing this issue, it doesn't appear in 1.17.2 !screenshot-1.png|width=623,height=294! was (Author: martijnvisser): [~jingge] Closing this issue, it doesn't appear in 1.17.2 !screenshot-1.png! > Update GCS filesystems to latest available versions > --- > > Key: FLINK-33704 > URL: https://issues.apache.org/jira/browse/FLINK-33704 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / FileSystem, FileSystems >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0, 1.18.1 > > Attachments: screenshot-1.png > > > Update GS SDK from 2.15.0 to 2.29.1 and GS Hadoop Connector from 2.2.15 to > 2.2.18 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33704) Update GCS filesystems to latest available versions
[ https://issues.apache.org/jira/browse/FLINK-33704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-33704. -- Resolution: Fixed > Update GCS filesystems to latest available versions > --- > > Key: FLINK-33704 > URL: https://issues.apache.org/jira/browse/FLINK-33704 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / FileSystem, FileSystems >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0, 1.18.1 > > Attachments: screenshot-1.png > > > Update GS SDK from 2.15.0 to 2.29.1 and GS Hadoop Connector from 2.2.15 to > 2.2.18 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33704) Update GCS filesystems to latest available versions
[ https://issues.apache.org/jira/browse/FLINK-33704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798130#comment-17798130 ] Jing Ge commented on FLINK-33704: - [~martijnvisser] thanks for letting us know. > Update GCS filesystems to latest available versions > --- > > Key: FLINK-33704 > URL: https://issues.apache.org/jira/browse/FLINK-33704 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / FileSystem, FileSystems >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0, 1.18.1 > > Attachments: screenshot-1.png > > > Update GS SDK from 2.15.0 to 2.29.1 and GS Hadoop Connector from 2.2.15 to > 2.2.18 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33793) java.lang.NoSuchMethodError when checkpointing in Google Cloud Storage
[
https://issues.apache.org/jira/browse/FLINK-33793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser closed FLINK-33793.
--
Resolution: Fixed
Fixed via FLINK-33704
> java.lang.NoSuchMethodError when checkpointing in Google Cloud Storage
> --
>
> Key: FLINK-33793
> URL: https://issues.apache.org/jira/browse/FLINK-33793
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.18.0
> Environment: Flink 1.18
>Reporter: ChangZhuo Chen (陳昌倬)
>Priority: Critical
>
> We have the following exception when checkpointing in Flink 1.18 + Google
> Cloud Storage. The same code works well in Flink 1.17:
>
> {{2023-12-11 07:45:28,861 ERROR
> org.apache.flink.util.FatalExitExceptionHandler [] - FATAL:
> Thread 'jobmanager-io-thread-5' produced an uncaught exception. Stopping the
> process...}}
> {{java.lang.NoSuchMethodError: 'com.google.common.collect.ImmutableMap
> com.google.common.collect.ImmutableMap$Builder.buildOrThrow()'}}
> {{ at
> com.google.cloud.storage.UnifiedOpts$Opts.getRpcOptions(UnifiedOpts.java:2096)
> ~[?:?]}}
> {{ at
> com.google.cloud.storage.StorageImpl.writer(StorageImpl.java:624) ~[?:?]}}
> {{ at com.google.cloud.storage.StorageImpl.writer(StorageImpl.java:90)
> ~[?:?]}}
> {{ at
> org.apache.flink.fs.gs.storage.GSBlobStorageImpl.writeBlob(GSBlobStorageImpl.java:64)
> ~[?:?]}}
> {{ at
> org.apache.flink.fs.gs.writer.GSRecoverableFsDataOutputStream.createWriteChannel(GSRecoverableFsDataOutputStream.java:229)
> ~[?:?]}}
> {{ at
> org.apache.flink.fs.gs.writer.GSRecoverableFsDataOutputStream.write(GSRecoverableFsDataOutputStream.java:152)
> ~[?:?]}}
> {{ at
> org.apache.flink.fs.gs.writer.GSRecoverableFsDataOutputStream.write(GSRecoverableFsDataOutputStream.java:135)
> ~[?:?]}}
> {{ at
> org.apache.flink.fs.gs.writer.GSRecoverableFsDataOutputStream.write(GSRecoverableFsDataOutputStream.java:128)
> ~[?:?]}}
> {{ at
> org.apache.flink.runtime.state.filesystem.FsCheckpointMetadataOutputStream.write(FsCheckpointMetadataOutputStream.java:73)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at java.io.DataOutputStream.writeInt(Unknown Source) ~[?:?]}}
> {{ at
> org.apache.flink.runtime.checkpoint.Checkpoints.storeCheckpointMetadata(Checkpoints.java:98)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at
> org.apache.flink.runtime.checkpoint.Checkpoints.storeCheckpointMetadata(Checkpoints.java:88)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at
> org.apache.flink.runtime.checkpoint.Checkpoints.storeCheckpointMetadata(Checkpoints.java:83)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at
> org.apache.flink.runtime.checkpoint.PendingCheckpoint.finalizeCheckpoint(PendingCheckpoint.java:335)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.finalizeCheckpoint(CheckpointCoordinator.java:1404)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1303)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:1195)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at
> org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$acknowledgeCheckpoint$1(ExecutionGraphHandler.java:89)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at
> org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$processCheckpointCoordinatorMessage$3(ExecutionGraphHandler.java:119)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source) ~[?:?]}}
> {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source) ~[?:?]}}
> {{ at java.lang.Thread.run(Unknown Source) [?:?]}}
>
> The issue has been reported in GitHub
> [https://github.com/apache/flink/pull/22281#issuecomment-1728553794.]
> However, it is still not fixed yet in 1.18.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33603) Fix guava shading for GCS connector
[
https://issues.apache.org/jira/browse/FLINK-33603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-33603:
---
Affects Version/s: 1.18.0
(was: 1.17.1)
> Fix guava shading for GCS connector
> ---
>
> Key: FLINK-33603
> URL: https://issues.apache.org/jira/browse/FLINK-33603
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.18.0
>Reporter: Jayadeep Jayaraman
>Priority: Major
> Labels: pull-request-available
>
> GCS connector has guava shading issue, This change introduced dependency on
> guava version({{{}31.1-jre{}}}) required by {{{}google-cloud-storage{}}}.
> Upgrade of {{google-cloud-storage}} lead to runtime failure because of new
> functionalities added in {{{}31.1-jre{}}}.
> This change pins guava version to the one required by storage client
> specifically in {{{}flink-gs-fs-hadoop{}}}, leaving all other filesystem
> implementation untouched.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33588) Fix Flink Checkpointing Statistics Bug
[ https://issues.apache.org/jira/browse/FLINK-33588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798115#comment-17798115 ] Jing Ge edited comment on FLINK-33588 at 12/18/23 10:54 AM: [~zhutong66] could you please backport the change to 1.18? was (Author: jingge): [~zhutong66] could you please cp the change to 1.18? > Fix Flink Checkpointing Statistics Bug > -- > > Key: FLINK-33588 > URL: https://issues.apache.org/jira/browse/FLINK-33588 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.14.5, 1.16.0, 1.17.0, 1.15.2, 1.14.6, 1.18.0, 1.17.1 >Reporter: Tongtong Zhu >Assignee: Tongtong Zhu >Priority: Critical > Labels: pull-request-available > Fix For: 1.19.0, 1.18.1 > > Attachments: FLINK-33588.patch, image-2023-12-11-17-35-23-391.png, > image-2023-12-13-11-35-43-780.png, image-2023-12-15-13-59-28-201.png, > newCommit-FLINK-33688.patch > > > When the Flink task is first started, the checkpoint data is null due to the > lack of data, and Percentile throws a null pointer exception when calculating > the percentage. After multiple tests, I found that it is necessary to set an > initial value for the statistical data value of the checkpoint when the > checkpoint data is null (i.e. at the beginning of the task) to solve this > problem. > The following is an abnormal description of the bug： > 2023-09-13 15:02:54,608 ERROR > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler > [] - Unhandled exception. > org.apache.commons.math3.exception.NullArgumentException: input array > at > org.apache.commons.math3.util.MathArrays.verifyValues(MathArrays.java:1650) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.commons.math3.stat.descriptive.AbstractUnivariateStatistic.test(AbstractUnivariateStatistic.java:158) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.commons.math3.stat.descriptive.rank.Percentile.evaluate(Percentile.java:272) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.commons.math3.stat.descriptive.rank.Percentile.evaluate(Percentile.java:241) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogramStatistics$CommonMetricsSnapshot.getPercentile(DescriptiveStatisticsHistogramStatistics.java:159) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogramStatistics.getQuantile(DescriptiveStatisticsHistogramStatistics.java:53) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.checkpoint.StatsSummarySnapshot.getQuantile(StatsSummarySnapshot.java:108) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.messages.checkpoints.StatsSummaryDto.valueOf(StatsSummaryDto.java:81) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.createCheckpointingStatistics(CheckpointingStatisticsHandler.java:129) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleRequest(CheckpointingStatisticsHandler.java:84) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleRequest(CheckpointingStatisticsHandler.java:58) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.AbstractAccessExecutionGraphHandler.handleRequest(AbstractAccessExecutionGraphHandler.java:68) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.lambda$handleRequest$0(AbstractExecutionGraphHandler.java:87) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) > [?:1.8.0_151] > at > java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) > [?:1.8.0_151] > at > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) > [?:1.8.0_151] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [?:1.8.0_151] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_151] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [?:1.8.0_151] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [?:1.8.0_151] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798131#comment-17798131
]
Benchao Li commented on FLINK-33365:
[~davidradl] I didn't see a problem in the example you gave in this thread:
https://lists.apache.org/thread/7r49c5ffohz1oqm3oz2g60g5lnovq0l2.
The result is correct since you are using 'LEFT JOIN', for the records from
left table which does not have matching rows, it should emit the line, and pad
nulls for fields in right table. Am I missing something?
The behavior should be the same when pushdown is enabled or not as you stated
in previously.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Labels: pull-request-available
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-27876][table-planner] Choose the right side as build side when using default shuffle hash strategy if left size is equal with right [flink]
xuyangzhong commented on PR #19866: URL: https://github.com/apache/flink/pull/19866#issuecomment-1860163668 > @xuyangzhong Do you think we should pick this fix to release-1.18? I feel that cp is not needed because it changes the plan and will cause compatibility issues. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33560) Externalize Kinesis Python connector code
[ https://issues.apache.org/jira/browse/FLINK-33560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer updated FLINK-33560: -- Fix Version/s: aws-connector-4.3.0 > Externalize Kinesis Python connector code > - > > Key: FLINK-33560 > URL: https://issues.apache.org/jira/browse/FLINK-33560 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Connectors / Kinesis >Affects Versions: 1.18.0 >Reporter: Márton Balassi >Assignee: Danny Cranmer >Priority: Major > Fix For: 1.19.0, aws-connector-4.3.0 > > > See description of parent ticket for context. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33560) Externalize Kinesis Python connector code
[ https://issues.apache.org/jira/browse/FLINK-33560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer updated FLINK-33560: -- Affects Version/s: aws-connector-4.2.0 > Externalize Kinesis Python connector code > - > > Key: FLINK-33560 > URL: https://issues.apache.org/jira/browse/FLINK-33560 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Connectors / Kinesis >Affects Versions: 1.18.0, aws-connector-4.2.0 >Reporter: Márton Balassi >Assignee: Danny Cranmer >Priority: Major > Fix For: 1.19.0, aws-connector-4.3.0 > > > See description of parent ticket for context. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33449) Support array_contains_seq function
[ https://issues.apache.org/jira/browse/FLINK-33449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798135#comment-17798135 ] Martijn Visser commented on FLINK-33449: I think a Dev ML discussion makes sense, I would probably vote for ARRAY_CONTAINS_SEQUENCE > Support array_contains_seq function > --- > > Key: FLINK-33449 > URL: https://issues.apache.org/jira/browse/FLINK-33449 > Project: Flink > Issue Type: New Feature >Reporter: leo cheng >Priority: Minor > Labels: pull-request-available > > support function array_contains_seq like trino contains_sequence > trino: > https://trino.io/docs/current/functions/array.html?highlight=contains_sequence#contains_sequence -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-32877][Filesystem]add HTTP options to gcs-cloud-storage client [flink]
MartijnVisser commented on PR #23226: URL: https://github.com/apache/flink/pull/23226#issuecomment-1860175028 > I don't have a Jira account. @cnauroth can you take the ownership of jira ticket? @singhravidutt Please request one via https://selfserve.apache.org/jira-account.html Also, please rebase your PRs (merge comitts can't be merged) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Fix NullArgumentException of getQuantile method in DescriptiveStatisticsHistogramStatistics [flink]
JingGe opened a new pull request, #23947: URL: https://github.com/apache/flink/pull/23947 BP of #23931 with no change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33804) Add Option to disable showing metrics in JobMananger UI
[ https://issues.apache.org/jira/browse/FLINK-33804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798136#comment-17798136 ] Martijn Visser commented on FLINK-33804: [~qqibrow] Please open a discussion with a proposal on the Dev ML for this > Add Option to disable showing metrics in JobMananger UI > --- > > Key: FLINK-33804 > URL: https://issues.apache.org/jira/browse/FLINK-33804 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Reporter: Lu Niu >Priority: Major > > Flink allows users to view metric in JobMananger UI. However there are 2 > problems we found: > # The JobManager is required to aggregate metrics from all task managers. > When the metric cardinality is quite high, this process can trigger a > JobManager Full GC and slow response time. > # Flink user cases in prod usually have their own dashboard to view metrics. > so this feature sometimes is not useful. > In light of this, we propose to add option to disable this feature. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798131#comment-17798131
]
Benchao Li edited comment on FLINK-33365 at 12/18/23 11:09 AM:
---
[~davidradl] I didn't see a problem in the example you gave in this thread:
https://lists.apache.org/thread/7r49c5ffohz1oqm3oz2g60g5lnovq0l2.
The result is correct since you are using 'LEFT JOIN', for the records from
left table which does not have matching rows, it should emit the line, and pad
nulls for fields in right table. Am I missing something?
The behavior should be the same when pushdown is enabled or not as you stated
previously.
was (Author: libenchao):
[~davidradl] I didn't see a problem in the example you gave in this thread:
https://lists.apache.org/thread/7r49c5ffohz1oqm3oz2g60g5lnovq0l2.
The result is correct since you are using 'LEFT JOIN', for the records from
left table which does not have matching rows, it should emit the line, and pad
nulls for fields in right table. Am I missing something?
The behavior should be the same when pushdown is enabled or not as you stated
in previously.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Labels: pull-request-available
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32007) Implement Python Wrappers for DynamoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-32007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer updated FLINK-32007: -- Fix Version/s: aws-connector-4.3.0 (was: 1.19.0) > Implement Python Wrappers for DynamoDB Connector > > > Key: FLINK-32007 > URL: https://issues.apache.org/jira/browse/FLINK-32007 > Project: Flink > Issue Type: New Feature > Components: API / Python, Connectors / DynamoDB >Reporter: Ahmed Hamdy >Assignee: Khanh Vu >Priority: Minor > Fix For: aws-connector-4.3.0 > > > Implement Python API Wrappers for DynamoDB Sink -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] Fix NullArgumentException of getQuantile method in DescriptiveStatisticsHistogramStatistics [flink]
JingGe commented on PR #23947: URL: https://github.com/apache/flink/pull/23947#issuecomment-1860197970 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix NullArgumentException of getQuantile method in DescriptiveStatisticsHistogramStatistics [flink]
flinkbot commented on PR #23947: URL: https://github.com/apache/flink/pull/23947#issuecomment-1860198985 ## CI report: * 28314d4fed2721dbabac426b41a416c79df772f0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-33823] Make PlannerQueryOperation SQL serializable [flink]
dawidwys opened a new pull request, #23948: URL: https://github.com/apache/flink/pull/23948 ## What is the purpose of the change Make PlannerQueryOperation SQL serializable ## Verifying this change Added tests in: * org.apache.flink.table.api.QueryOperationSqlSerializationTest * org.apache.flink.table.api.QueryOperationSqlExecutionTest ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33823) Serialize PlannerQueryOperation into SQL
[ https://issues.apache.org/jira/browse/FLINK-33823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33823: --- Labels: pull-request-available (was: ) > Serialize PlannerQueryOperation into SQL > > > Key: FLINK-33823 > URL: https://issues.apache.org/jira/browse/FLINK-33823 > Project: Flink > Issue Type: Sub-task >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33588) Fix Flink Checkpointing Statistics Bug
[ https://issues.apache.org/jira/browse/FLINK-33588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798115#comment-17798115 ] Jing Ge edited comment on FLINK-33588 at 12/18/23 11:22 AM: [~zhutong66] bp to 1.18: https://github.com/apache/flink/pull/23947 was (Author: jingge): [~zhutong66] could you please backport the change to 1.18? > Fix Flink Checkpointing Statistics Bug > -- > > Key: FLINK-33588 > URL: https://issues.apache.org/jira/browse/FLINK-33588 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.14.5, 1.16.0, 1.17.0, 1.15.2, 1.14.6, 1.18.0, 1.17.1 >Reporter: Tongtong Zhu >Assignee: Tongtong Zhu >Priority: Critical > Labels: pull-request-available > Fix For: 1.19.0, 1.18.1 > > Attachments: FLINK-33588.patch, image-2023-12-11-17-35-23-391.png, > image-2023-12-13-11-35-43-780.png, image-2023-12-15-13-59-28-201.png, > newCommit-FLINK-33688.patch > > > When the Flink task is first started, the checkpoint data is null due to the > lack of data, and Percentile throws a null pointer exception when calculating > the percentage. After multiple tests, I found that it is necessary to set an > initial value for the statistical data value of the checkpoint when the > checkpoint data is null (i.e. at the beginning of the task) to solve this > problem. > The following is an abnormal description of the bug： > 2023-09-13 15:02:54,608 ERROR > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler > [] - Unhandled exception. > org.apache.commons.math3.exception.NullArgumentException: input array > at > org.apache.commons.math3.util.MathArrays.verifyValues(MathArrays.java:1650) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.commons.math3.stat.descriptive.AbstractUnivariateStatistic.test(AbstractUnivariateStatistic.java:158) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.commons.math3.stat.descriptive.rank.Percentile.evaluate(Percentile.java:272) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.commons.math3.stat.descriptive.rank.Percentile.evaluate(Percentile.java:241) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogramStatistics$CommonMetricsSnapshot.getPercentile(DescriptiveStatisticsHistogramStatistics.java:159) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogramStatistics.getQuantile(DescriptiveStatisticsHistogramStatistics.java:53) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.checkpoint.StatsSummarySnapshot.getQuantile(StatsSummarySnapshot.java:108) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.messages.checkpoints.StatsSummaryDto.valueOf(StatsSummaryDto.java:81) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.createCheckpointingStatistics(CheckpointingStatisticsHandler.java:129) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleRequest(CheckpointingStatisticsHandler.java:84) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleRequest(CheckpointingStatisticsHandler.java:58) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.AbstractAccessExecutionGraphHandler.handleRequest(AbstractAccessExecutionGraphHandler.java:68) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.lambda$handleRequest$0(AbstractExecutionGraphHandler.java:87) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) > [?:1.8.0_151] > at > java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) > [?:1.8.0_151] > at > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) > [?:1.8.0_151] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [?:1.8.0_151] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_151] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [?:1.8.0_151] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [?:1.8.0_151] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
[jira] [Commented] (FLINK-32346) JdbcNumericBetweenParametersProvider Sharding key boundaries large storage long integer overflow, use BigDecimal instead Long
[ https://issues.apache.org/jira/browse/FLINK-32346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798139#comment-17798139 ] Benchao Li commented on FLINK-32346: [~sunnny] I agree the partition columns can/should consider more data types. Currently the documentation and the code already diverge. The document says "The scan.partition.column must be a numeric, date, or timestamp column from the table in question", however the code only supports numeric (not bigger than long). > JdbcNumericBetweenParametersProvider Sharding key boundaries large storage > long integer overflow, use BigDecimal instead Long > -- > > Key: FLINK-32346 > URL: https://issues.apache.org/jira/browse/FLINK-32346 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Reporter: zhilinli >Priority: Major > Attachments: image-2023-06-15-16-42-16-773.png, > image-2023-06-15-16-46-13-188.png > > > *JdbcNumericBetweenParametersProvider.class* > Sharding key boundaries large storage long integer overflow, use BigDecimal > instead Long, so that length types such as DecimalType(30,0) are compatible > and LONG cannot be stored Can be assigned to me and I want to complete it > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33823] Make PlannerQueryOperation SQL serializable [flink]
flinkbot commented on PR #23948: URL: https://github.com/apache/flink/pull/23948#issuecomment-1860215435 ## CI report: * 675da5c03660ff590f1ceda33c14452181d0c2c6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33865) exponential-delay.attempts-before-reset-backoff doesn't work when it's set in Job Configuration
[ https://issues.apache.org/jira/browse/FLINK-33865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-33865: Description: exponential-delay.attempts-before-reset-backoff doesn't work when it's set in Job Configuration. h2. Reason: When exponential-delay.attempts-before-reset-backoff is set by job Configuration instead of cluster configuration. ExecutionConfig#configure will call RestartStrategies#parseConfiguration to create the ExponentialDelayRestartStrategyConfiguration. And then RestartBackoffTimeStrategyFactoryLoader#getJobRestartStrategyFactory will create the ExponentialDelayRestartBackoffTimeStrategyFactory by the ExponentialDelayRestartStrategyConfiguration. Since 1.19, RestartStrategies and RestartStrategyConfiguration are depreated, so ExponentialDelayRestartStrategyConfiguration doesn't support exponential-delay.attempts-before-reset-backoff. So if we set exponential-delay.attempts-before-reset-backoff at job level, it won't be supported. h2. Solution If we use the ExponentialDelayRestartStrategyConfiguration to save restartStrategy related options in the ExecutionConfig, all new options are set at job level will be missed. So we can use the Configuration to save the restartStrategy options inside of ExecutionConfig. !image-2023-12-17-17-56-59-138.png|width=1212,height=256! was: exponential-delay.attempts-before-reset-backoff doesn't work when it's set in Job Configuration. h2. Reason: when exponential-delay.attempts-before-reset-backoff is set by job Configuration instead of cluster configuration. ExecutionConfig#configure will call RestartStrategies#parseConfiguration to create the ExponentialDelayRestartStrategyConfiguration. And then RestartBackoffTimeStrategyFactoryLoader#getJobRestartStrategyFactory will create the ExponentialDelayRestartBackoffTimeStrategyFactory by the ExponentialDelayRestartStrategyConfiguration. Since 1.19, RestartStrategies and RestartStrategyConfiguration are depreated, so ExponentialDelayRestartStrategyConfiguration doesn't support exponential-delay.attempts-before-reset-backoff. So if we set exponential-delay.attempts-before-reset-backoff at job level, it won't be supported. h2. Solution If we use the ExponentialDelayRestartStrategyConfiguration to save restartStrategy related options in the ExecutionConfig, all new options are set at job level will be missed. So we can use the Configuration to save the restartStrategy options inside of ExecutionConfig. !image-2023-12-17-17-56-59-138.png! > exponential-delay.attempts-before-reset-backoff doesn't work when it's set in > Job Configuration > --- > > Key: FLINK-33865 > URL: https://issues.apache.org/jira/browse/FLINK-33865 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > Labels: pull-request-available > Attachments: image-2023-12-17-17-56-59-138.png > > > exponential-delay.attempts-before-reset-backoff doesn't work when it's set in > Job Configuration. > h2. Reason: > When exponential-delay.attempts-before-reset-backoff is set by job > Configuration instead of cluster configuration. ExecutionConfig#configure > will call RestartStrategies#parseConfiguration to create the > ExponentialDelayRestartStrategyConfiguration. And then > RestartBackoffTimeStrategyFactoryLoader#getJobRestartStrategyFactory will > create the ExponentialDelayRestartBackoffTimeStrategyFactory by the > ExponentialDelayRestartStrategyConfiguration. > Since 1.19, RestartStrategies and RestartStrategyConfiguration are depreated, > so ExponentialDelayRestartStrategyConfiguration doesn't support > exponential-delay.attempts-before-reset-backoff. So if we set > exponential-delay.attempts-before-reset-backoff at job level, it won't be > supported. > h2. Solution > If we use the ExponentialDelayRestartStrategyConfiguration to save > restartStrategy related options in the ExecutionConfig, all new options are > set at job level will be missed. > So we can use the Configuration to save the restartStrategy options inside of > ExecutionConfig. > !image-2023-12-17-17-56-59-138.png|width=1212,height=256! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33862) Flink Unit Test Failures on 1.18.0
[
https://issues.apache.org/jira/browse/FLINK-33862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798143#comment-17798143
]
Prabhu Joseph commented on FLINK-33862:
---
[FLINK-28203|https://issues.apache.org/jira/browse/FLINK-28203] has marked
bundled dependecies as optional. Setting the flag flink.markBundledAsOptional
to false has fixed the unit test failures.
> Flink Unit Test Failures on 1.18.0
> --
>
> Key: FLINK-33862
> URL: https://issues.apache.org/jira/browse/FLINK-33862
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.18.0, 1.19.0
>Reporter: Prabhu Joseph
>Priority: Major
>
> Flink Unit Test Failures on 1.18.0. There are 100+ unit test cases failing
> due to below common issues.
> *Issue 1*
> {code:java}
> ./mvnw -DfailIfNoTests=false -Dmaven.test.failure.ignore=true
> -Dtest=ExecutionPlanAfterExecutionTest test
> [INFO] Running org.apache.flink.client.program.ExecutionPlanAfterExecutionTest
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:141)
> at
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
> at
> org.apache.flink.runtime.rpc.pekko.PekkoInvocationHandler.lambda$invokeRpc$1(PekkoInvocationHandler.java:268)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
> at
> org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1267)
> at
> org.apache.flink.runtime.concurrent.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$1(ClassLoadingUtils.java:93)
> at
> org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
> at
> org.apache.flink.runtime.concurrent.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
> at
> org.apache.flink.runtime.concurrent.pekko.ScalaFutureUtils$1.onComplete(ScalaFutureUtils.java:47)
> at org.apache.pekko.dispatch.OnComplete.internal(Future.scala:310)
> at org.apache.pekko.dispatch.OnComplete.internal(Future.scala:307)
> at org.apache.pekko.dispatch.japi$CallbackBridge.apply(Future.scala:234)
> at org.apache.pekko.dispatch.japi$CallbackBridge.apply(Future.scala:231)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
> at
> org.apache.flink.runtime.concurrent.pekko.ScalaFutureUtils$DirectExecutionContext.execute(ScalaFutureUtils.java:65)
> at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
> at
> scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288)
> at
> scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288)
> at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288)
> at org.apache.pekko.pattern.PromiseActorRef.$bang(AskSupport.scala:629)
> at
> org.apache.pekko.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:34)
> at
> org.apache.pekko.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:33)
> at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:536)
> at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
> at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
> at
> org.apache.pekko.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:73)
> at
> org.apac
[jira] [Commented] (FLINK-33793) java.lang.NoSuchMethodError when checkpointing in Google Cloud Storage
[
https://issues.apache.org/jira/browse/FLINK-33793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798144#comment-17798144
]
Jun Qin commented on FLINK-33793:
-
Could we update the Fix versions field such that everyone knows?
> java.lang.NoSuchMethodError when checkpointing in Google Cloud Storage
> --
>
> Key: FLINK-33793
> URL: https://issues.apache.org/jira/browse/FLINK-33793
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.18.0
> Environment: Flink 1.18
>Reporter: ChangZhuo Chen (陳昌倬)
>Priority: Critical
>
> We have the following exception when checkpointing in Flink 1.18 + Google
> Cloud Storage. The same code works well in Flink 1.17:
>
> {{2023-12-11 07:45:28,861 ERROR
> org.apache.flink.util.FatalExitExceptionHandler [] - FATAL:
> Thread 'jobmanager-io-thread-5' produced an uncaught exception. Stopping the
> process...}}
> {{java.lang.NoSuchMethodError: 'com.google.common.collect.ImmutableMap
> com.google.common.collect.ImmutableMap$Builder.buildOrThrow()'}}
> {{ at
> com.google.cloud.storage.UnifiedOpts$Opts.getRpcOptions(UnifiedOpts.java:2096)
> ~[?:?]}}
> {{ at
> com.google.cloud.storage.StorageImpl.writer(StorageImpl.java:624) ~[?:?]}}
> {{ at com.google.cloud.storage.StorageImpl.writer(StorageImpl.java:90)
> ~[?:?]}}
> {{ at
> org.apache.flink.fs.gs.storage.GSBlobStorageImpl.writeBlob(GSBlobStorageImpl.java:64)
> ~[?:?]}}
> {{ at
> org.apache.flink.fs.gs.writer.GSRecoverableFsDataOutputStream.createWriteChannel(GSRecoverableFsDataOutputStream.java:229)
> ~[?:?]}}
> {{ at
> org.apache.flink.fs.gs.writer.GSRecoverableFsDataOutputStream.write(GSRecoverableFsDataOutputStream.java:152)
> ~[?:?]}}
> {{ at
> org.apache.flink.fs.gs.writer.GSRecoverableFsDataOutputStream.write(GSRecoverableFsDataOutputStream.java:135)
> ~[?:?]}}
> {{ at
> org.apache.flink.fs.gs.writer.GSRecoverableFsDataOutputStream.write(GSRecoverableFsDataOutputStream.java:128)
> ~[?:?]}}
> {{ at
> org.apache.flink.runtime.state.filesystem.FsCheckpointMetadataOutputStream.write(FsCheckpointMetadataOutputStream.java:73)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at java.io.DataOutputStream.writeInt(Unknown Source) ~[?:?]}}
> {{ at
> org.apache.flink.runtime.checkpoint.Checkpoints.storeCheckpointMetadata(Checkpoints.java:98)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at
> org.apache.flink.runtime.checkpoint.Checkpoints.storeCheckpointMetadata(Checkpoints.java:88)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at
> org.apache.flink.runtime.checkpoint.Checkpoints.storeCheckpointMetadata(Checkpoints.java:83)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at
> org.apache.flink.runtime.checkpoint.PendingCheckpoint.finalizeCheckpoint(PendingCheckpoint.java:335)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.finalizeCheckpoint(CheckpointCoordinator.java:1404)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1303)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:1195)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at
> org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$acknowledgeCheckpoint$1(ExecutionGraphHandler.java:89)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at
> org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$processCheckpointCoordinatorMessage$3(ExecutionGraphHandler.java:119)
> ~[flink-dist-1.18.0.jar:1.18.0]}}
> {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source) ~[?:?]}}
> {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source) ~[?:?]}}
> {{ at java.lang.Thread.run(Unknown Source) [?:?]}}
>
> The issue has been reported in GitHub
> [https://github.com/apache/flink/pull/22281#issuecomment-1728553794.]
> However, it is still not fixed yet in 1.18.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33872) Checkpoint history does not display for completed jobs
[ https://issues.apache.org/jira/browse/FLINK-33872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Liang Teoh updated FLINK-33872: Description: Prior to https://issues.apache.org/jira/browse/FLINK-32469, we see checkpoint history for completed jobs (CANCELED, FAILED, FINISHED). After https://issues.apache.org/jira/browse/FLINK-32469, the checkpoint history does not show up for completed jobs. *Reproduction steps:* # Start a Flink cluster. # Submit a job with checkpointing enabled. # Wait until at least 1 checkpoint completes. # Cancel job. # Open the Flink dashboard > Job > Checkpoints > History. We will see log line in JobManager saying "FlinkJobNotFoundException: Could not find Flink job ( )" *Snapshot of failure:* When job is running, we can see checkpoints. !image-2023-12-18-11-37-11-914.png|width=1023,height=350! When job has been CANCELLED, we no longer see checkpoints data. !image-2023-12-18-11-37-29-596.png|width=1023,height=307! was: Prior to https://issues.apache.org/jira/browse/FLINK-32469, we see checkpoint history for completed jobs (CANCELED, FAILED, FINISHED). After https://issues.apache.org/jira/browse/FLINK-32469, the checkpoint history does not show up for completed jobs. *Reproduction steps:* # Start a Flink cluster. # Submit a job with checkpointing enabled. # Wait until at least 1 checkpoint completes. # Cancel job. # Open the Flink dashboard > Job > Checkpoints > History. We will see log line in JobManager saying "FlinkJobNotFoundException: Could not find Flink job ( )" *Snapshot of failure:* When job is running, we can see checkpoints. !image-2023-12-18-11-37-11-914.png! When job has been CANCELLED, we no longer see checkpoints data. !image-2023-12-18-11-37-29-596.png! > Checkpoint history does not display for completed jobs > -- > > Key: FLINK-33872 > URL: https://issues.apache.org/jira/browse/FLINK-33872 > Project: Flink > Issue Type: Bug > Components: Runtime / REST >Affects Versions: 1.18.0 >Reporter: Hong Liang Teoh >Priority: Major > Fix For: 1.19.0, 1.18.2 > > Attachments: image-2023-12-18-11-37-11-914.png, > image-2023-12-18-11-37-29-596.png > > > Prior to https://issues.apache.org/jira/browse/FLINK-32469, we see checkpoint > history for completed jobs (CANCELED, FAILED, FINISHED). > After https://issues.apache.org/jira/browse/FLINK-32469, the checkpoint > history does not show up for completed jobs. > *Reproduction steps:* > # Start a Flink cluster. > # Submit a job with checkpointing enabled. > # Wait until at least 1 checkpoint completes. > # Cancel job. > # Open the Flink dashboard > Job > Checkpoints > History. > We will see log line in JobManager saying "FlinkJobNotFoundException: Could > not find Flink job ( )" > *Snapshot of failure:* > When job is running, we can see checkpoints. > !image-2023-12-18-11-37-11-914.png|width=1023,height=350! > When job has been CANCELLED, we no longer see checkpoints data. > !image-2023-12-18-11-37-29-596.png|width=1023,height=307! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33872) Checkpoint history does not display for completed jobs
[ https://issues.apache.org/jira/browse/FLINK-33872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Liang Teoh updated FLINK-33872: Description: Prior to https://issues.apache.org/jira/browse/FLINK-32469, we see checkpoint history for completed jobs (CANCELED, FAILED, FINISHED). After https://issues.apache.org/jira/browse/FLINK-32469, the checkpoint history does not show up for completed jobs. *Reproduction steps:* # Start a Flink cluster. # Submit a job with checkpointing enabled. # Wait until at least 1 checkpoint completes. # Cancel job. # Open the Flink dashboard > Job > Checkpoints > History. We will see log line in JobManager saying "FlinkJobNotFoundException: Could not find Flink job ( )" *Snapshot of failure:* When job is running, we can see checkpoints. !image-2023-12-18-11-37-11-914.png|width=862,height=295! When job has been CANCELLED, we no longer see checkpoints data. !image-2023-12-18-11-37-29-596.png|width=860,height=258! was: Prior to https://issues.apache.org/jira/browse/FLINK-32469, we see checkpoint history for completed jobs (CANCELED, FAILED, FINISHED). After https://issues.apache.org/jira/browse/FLINK-32469, the checkpoint history does not show up for completed jobs. *Reproduction steps:* # Start a Flink cluster. # Submit a job with checkpointing enabled. # Wait until at least 1 checkpoint completes. # Cancel job. # Open the Flink dashboard > Job > Checkpoints > History. We will see log line in JobManager saying "FlinkJobNotFoundException: Could not find Flink job ( )" *Snapshot of failure:* When job is running, we can see checkpoints. !image-2023-12-18-11-37-11-914.png|width=1023,height=350! When job has been CANCELLED, we no longer see checkpoints data. !image-2023-12-18-11-37-29-596.png|width=1023,height=307! > Checkpoint history does not display for completed jobs > -- > > Key: FLINK-33872 > URL: https://issues.apache.org/jira/browse/FLINK-33872 > Project: Flink > Issue Type: Bug > Components: Runtime / REST >Affects Versions: 1.18.0 >Reporter: Hong Liang Teoh >Priority: Major > Fix For: 1.19.0, 1.18.2 > > Attachments: image-2023-12-18-11-37-11-914.png, > image-2023-12-18-11-37-29-596.png > > > Prior to https://issues.apache.org/jira/browse/FLINK-32469, we see checkpoint > history for completed jobs (CANCELED, FAILED, FINISHED). > After https://issues.apache.org/jira/browse/FLINK-32469, the checkpoint > history does not show up for completed jobs. > *Reproduction steps:* > # Start a Flink cluster. > # Submit a job with checkpointing enabled. > # Wait until at least 1 checkpoint completes. > # Cancel job. > # Open the Flink dashboard > Job > Checkpoints > History. > We will see log line in JobManager saying "FlinkJobNotFoundException: Could > not find Flink job ( )" > *Snapshot of failure:* > When job is running, we can see checkpoints. > !image-2023-12-18-11-37-11-914.png|width=862,height=295! > When job has been CANCELLED, we no longer see checkpoints data. > !image-2023-12-18-11-37-29-596.png|width=860,height=258! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33872) Checkpoint history does not display for completed jobs
Hong Liang Teoh created FLINK-33872: --- Summary: Checkpoint history does not display for completed jobs Key: FLINK-33872 URL: https://issues.apache.org/jira/browse/FLINK-33872 Project: Flink Issue Type: Bug Components: Runtime / REST Affects Versions: 1.18.0 Reporter: Hong Liang Teoh Fix For: 1.19.0, 1.18.2 Attachments: image-2023-12-18-11-37-11-914.png, image-2023-12-18-11-37-29-596.png Prior to https://issues.apache.org/jira/browse/FLINK-32469, we see checkpoint history for completed jobs (CANCELED, FAILED, FINISHED). After https://issues.apache.org/jira/browse/FLINK-32469, the checkpoint history does not show up for completed jobs. *Reproduction steps:* # Start a Flink cluster. # Submit a job with checkpointing enabled. # Wait until at least 1 checkpoint completes. # Cancel job. # Open the Flink dashboard > Job > Checkpoints > History. We will see log line in JobManager saying "FlinkJobNotFoundException: Could not find Flink job ( )" *Snapshot of failure:* When job is running, we can see checkpoints. !image-2023-12-18-11-37-11-914.png! When job has been CANCELLED, we no longer see checkpoints data. !image-2023-12-18-11-37-29-596.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33778][table] Cleanup usage of deprecated TableConfig#setIdleS… [flink]
liuyongvs commented on PR #23894: URL: https://github.com/apache/flink/pull/23894#issuecomment-1860241060 > LTGM. BTW, should we also update this function in pyflink? cc @HuangXingBo Further more, please modify the description of this pr. the python code of has no use, we do not need modify ``` def set_idle_state_retention_time(self, min_time: datetime.timedelta, max_time: datetime.timedelta): ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-33872) Checkpoint history does not display for completed jobs
[ https://issues.apache.org/jira/browse/FLINK-33872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Liang Teoh reassigned FLINK-33872: --- Assignee: Hong Liang Teoh > Checkpoint history does not display for completed jobs > -- > > Key: FLINK-33872 > URL: https://issues.apache.org/jira/browse/FLINK-33872 > Project: Flink > Issue Type: Bug > Components: Runtime / REST >Affects Versions: 1.18.0 >Reporter: Hong Liang Teoh >Assignee: Hong Liang Teoh >Priority: Major > Fix For: 1.19.0, 1.18.2 > > Attachments: image-2023-12-18-11-37-11-914.png, > image-2023-12-18-11-37-29-596.png > > > Prior to https://issues.apache.org/jira/browse/FLINK-32469, we see checkpoint > history for completed jobs (CANCELED, FAILED, FINISHED). > After https://issues.apache.org/jira/browse/FLINK-32469, the checkpoint > history does not show up for completed jobs. > *Reproduction steps:* > # Start a Flink cluster. > # Submit a job with checkpointing enabled. > # Wait until at least 1 checkpoint completes. > # Cancel job. > # Open the Flink dashboard > Job > Checkpoints > History. > We will see log line in JobManager saying "FlinkJobNotFoundException: Could > not find Flink job ( )" > *Snapshot of failure:* > When job is running, we can see checkpoints. > !image-2023-12-18-11-37-11-914.png|width=862,height=295! > When job has been CANCELLED, we no longer see checkpoints data. > !image-2023-12-18-11-37-29-596.png|width=860,height=258! -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33778][table] Cleanup usage of deprecated TableConfig#setIdleS… [flink]
HuangXingBo merged PR #23894: URL: https://github.com/apache/flink/pull/23894 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33778) Cleanup usage of deprecated TableConfig#setIdleStateRetentionTime(Time, Time)
[ https://issues.apache.org/jira/browse/FLINK-33778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798149#comment-17798149 ] Jacky Lau commented on FLINK-33778: --- merged 3c86dcadf5366fa0026125051d69b0a8913d5e61 > Cleanup usage of deprecated TableConfig#setIdleStateRetentionTime(Time, Time) > - > > Key: FLINK-33778 > URL: https://issues.apache.org/jira/browse/FLINK-33778 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.19.0 >Reporter: Jacky Lau >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33778) Cleanup usage of deprecated TableConfig#setIdleStateRetentionTime(Time, Time)
[ https://issues.apache.org/jira/browse/FLINK-33778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Lau closed FLINK-33778. - Resolution: Fixed > Cleanup usage of deprecated TableConfig#setIdleStateRetentionTime(Time, Time) > - > > Key: FLINK-33778 > URL: https://issues.apache.org/jira/browse/FLINK-33778 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.19.0 >Reporter: Jacky Lau >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
