[jira] [Updated] (FLINK-15983) add native reader for Hive parquet files
[ https://issues.apache.org/jira/browse/FLINK-15983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15983: --- Fix Version/s: (was: 1.11.0) > add native reader for Hive parquet files > > > Key: FLINK-15983 > URL: https://issues.apache.org/jira/browse/FLINK-15983 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Reporter: Bowen Li >Assignee: Jingsong Lee >Priority: Critical > > add native reader for Hive parquet files, as well as benchmark that show how > faster the native reader is compared to generic hive reader, and publish a > blog about it > > cc [~ykt836] [~lirui] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15984) support hive stream table sink
[ https://issues.apache.org/jira/browse/FLINK-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15984: --- Fix Version/s: (was: 1.11.0) > support hive stream table sink > -- > > Key: FLINK-15984 > URL: https://issues.apache.org/jira/browse/FLINK-15984 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Reporter: Bowen Li >Priority: Major > > support hive stream table sink for stream processing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-13890) HiveCatalogUseBlinkITCase failed to get metastore connection
[
https://issues.apache.org/jira/browse/FLINK-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee resolved FLINK-13890.
--
Resolution: Duplicate
Feel free to re-open if the problem is re-produced.
> HiveCatalogUseBlinkITCase failed to get metastore connection
>
>
> Key: FLINK-13890
> URL: https://issues.apache.org/jira/browse/FLINK-13890
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Tests
>Affects Versions: 1.10.0
>Reporter: Piotr Nowojski
>Priority: Critical
>
> Failed on master with
> {code:java}
> Could not connect to meta store using any of the URIs provided. Most recent
> failure: org.apache.thrift.transport.TTransportException:
> java.net.ConnectException: Connection refused (Connection refused)
> {code}
> https://api.travis-ci.org/v3/job/578208116/log.txt
> https://travis-ci.org/apache/flink/builds/578208104
> CC [~lzljs3620320] [~ykt836]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] hequn8128 commented on a change in pull request #11444: [FLINK-16608][python] Support primitive data types for vectorized Python UDF
hequn8128 commented on a change in pull request #11444: [FLINK-16608][python]
Support primitive data types for vectorized Python UDF
URL: https://github.com/apache/flink/pull/11444#discussion_r396243233
##
File path:
flink-python/src/main/java/org/apache/flink/table/runtime/arrow/writers/DateWriter.java
##
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.arrow.writers;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.runtime.functions.SqlDateTimeUtils;
+import org.apache.flink.types.Row;
+
+import org.apache.arrow.vector.DateDayVector;
+
+import java.sql.Date;
+
+/**
+ * {@link ArrowFieldWriter} for Date.
+ */
+@Internal
+public final class DateWriter extends ArrowFieldWriter {
+
+ public DateWriter(DateDayVector dateDayVector) {
+ super(dateDayVector);
+ }
+
+ @Override
+ public void doWrite(Row value, int ordinal) {
+ if (value.getField(ordinal) == null) {
+ ((DateDayVector) getValueVector()).setNull(getCount());
+ } else {
+ ((DateDayVector) getValueVector()).setSafe(
+ getCount(),
SqlDateTimeUtils.dateToInternal(((Date) value.getField(ordinal;
Review comment:
`DateWriter` is used for the old planner. It seems we can't use the blink
planner class directly here, otherwise, Pyflink will depend on blink planner
even it is configured to use the old planner.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] hequn8128 commented on a change in pull request #11444: [FLINK-16608][python] Support primitive data types for vectorized Python UDF
hequn8128 commented on a change in pull request #11444: [FLINK-16608][python]
Support primitive data types for vectorized Python UDF
URL: https://github.com/apache/flink/pull/11444#discussion_r396237402
##
File path:
flink-python/src/main/java/org/apache/flink/table/runtime/arrow/writers/BaseRowDecimalWriter.java
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.arrow.writers;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.dataformat.BaseRow;
+
+import org.apache.arrow.vector.DecimalVector;
+
+/**
+ * {@link ArrowFieldWriter} for Decimal.
+ */
+@Internal
+public final class BaseRowDecimalWriter extends ArrowFieldWriter {
+
+ private final int precision;
+ private final int scale;
+
+ public BaseRowDecimalWriter(DecimalVector decimalVector, int precision,
int scale) {
+ super(decimalVector);
+ this.precision = precision;
+ this.scale = scale;
+ }
+
+ @Override
+ public void doWrite(BaseRow row, int ordinal) {
+ if (row.isNullAt(ordinal)) {
+ ((DecimalVector) getValueVector()).setNull(getCount());
+ } else {
+ ((DecimalVector) getValueVector()).setSafe(
Review comment:
Should we also adjust the precision and scale here, like it in
`DecimalWriter`? If we pass a decimal with different precision and scale, an
exception will be thrown.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] hequn8128 commented on a change in pull request #11444: [FLINK-16608][python] Support primitive data types for vectorized Python UDF
hequn8128 commented on a change in pull request #11444: [FLINK-16608][python]
Support primitive data types for vectorized Python UDF
URL: https://github.com/apache/flink/pull/11444#discussion_r396237588
##
File path:
flink-python/src/main/java/org/apache/flink/table/runtime/arrow/writers/BaseRowDecimalWriter.java
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.arrow.writers;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.dataformat.BaseRow;
+
+import org.apache.arrow.vector.DecimalVector;
+
+/**
+ * {@link ArrowFieldWriter} for Decimal.
+ */
+@Internal
+public final class BaseRowDecimalWriter extends ArrowFieldWriter {
+
+ private final int precision;
+ private final int scale;
+
+ public BaseRowDecimalWriter(DecimalVector decimalVector, int precision,
int scale) {
+ super(decimalVector);
+ this.precision = precision;
+ this.scale = scale;
+ }
+
+ @Override
+ public void doWrite(BaseRow row, int ordinal) {
+ if (row.isNullAt(ordinal)) {
+ ((DecimalVector) getValueVector()).setNull(getCount());
+ } else {
+ ((DecimalVector) getValueVector()).setSafe(
Review comment:
Should we also adjust the precision and scale here, like it in
`DecimalWriter`? If we pass a decimal with different precision and scale for
the BaseRowDecimalWriter, an exception will be thrown.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-16708) When a JDBC connection has been closed, the retry policy of the JDBCUpsertOutputFormat cannot take effect and may result in data loss
[
https://issues.apache.org/jira/browse/FLINK-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064580#comment-17064580
]
Weike Dong commented on FLINK-16708:
Hi [~lzljs3620320] , do you have time to look into this issue? Thanks
> When a JDBC connection has been closed, the retry policy of the
> JDBCUpsertOutputFormat cannot take effect and may result in data loss
> -
>
> Key: FLINK-16708
> URL: https://issues.apache.org/jira/browse/FLINK-16708
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.10.0
>Reporter: tangshangwen
>Assignee: tangshangwen
>Priority: Major
>
> In our test environment, I used the tcpkill command to simulate a scenario
> where the postgresql connection was closed. I found that the retry strategy
> of the flush method did not take effect, and when it retried the second time,
> it could not recognize that the connection had been closed because Before the
> first check whether the connection is closed, the batchStatements of
> PgStatement have been cleared, which causes the second execution to check
> that the batchStatements are empty and return normally.
> {code:java}
> 2020-03-20 21:16:18.246 [jdbc-upsert-output-format-thread-1] ERROR
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat - JDBC executeBatch
> error, retry times = 1
> org.postgresql.util.PSQLException: This connection has been closed.
> at
> org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:857)
> at
> org.postgresql.jdbc.PgConnection.getAutoCommit(PgConnection.java:817)
> at
> org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:813)
> at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:873)
> at
> org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1569)
> at
> org.apache.flink.api.java.io.jdbc.writer.AppendOnlyWriter.executeBatch(AppendOnlyWriter.java:62)
> at
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat.flush(JDBCUpsertOutputFormat.java:159)
> at
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat.lambda$open$0(JDBCUpsertOutputFormat.java:124)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-03-20 21:16:21.247 [jdbc-upsert-output-format-thread-1] ERROR
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat - JDBC executeBatch
> error, retry times = 1
> org.postgresql.util.PSQLException: This connection has been closed.
> at
> org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:857)
> at
> org.postgresql.jdbc.PgConnection.getAutoCommit(PgConnection.java:817)
> at
> org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:813)
> at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:873)
> at
> org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1569)
> at
> org.apache.flink.api.java.io.jdbc.writer.AppendOnlyWriter.executeBatch(AppendOnlyWriter.java:62)
> at
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat.flush(JDBCUpsertOutputFormat.java:159)
> at
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat.lambda$open$0(JDBCUpsertOutputFormat.java:124)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-13420) Supported hive versions are hard coded
[ https://issues.apache.org/jira/browse/FLINK-13420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-13420. Resolution: Invalid > Supported hive versions are hard coded > -- > > Key: FLINK-13420 > URL: https://issues.apache.org/jira/browse/FLINK-13420 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Affects Versions: 1.9.0 >Reporter: Jeff Zhang >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently, the supported hive versions are hardcoded in > [HiveShimLoader|[https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/client/HiveShimLoader.java]] > That makes me unable to run hive connector for 1.2.2, I believe we only need > to check the major.minor version number, but ignore the bugfix version number. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] TisonKun closed pull request #11415: [FLINK-15667][k8s] Support to mount custom Hadoop configurations
TisonKun closed pull request #11415: [FLINK-15667][k8s] Support to mount custom Hadoop configurations URL: https://github.com/apache/flink/pull/11415 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15667) Support to mount Hadoop Configurations
[ https://issues.apache.org/jira/browse/FLINK-15667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064581#comment-17064581 ] Zili Chen commented on FLINK-15667: --- master via f56a075aa029832b9cbacab51649c900498e18e9 > Support to mount Hadoop Configurations > -- > > Key: FLINK-15667 > URL: https://issues.apache.org/jira/browse/FLINK-15667 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Canbin Zheng >Assignee: Canbin Zheng >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Mounts the Hadoop configurations, either as a pre-defined config map, or a > local configuration directory on the Pod. > We plan to provide two options for the users to mount custom Hadoop > configurations: > * option 1: specify an existing ConfigMap that contains custom Hadoop > Configurations, one can share a single ConfigMap for more than one Flink > clusters. > * option 2: create a dedicated ConfigMap containing Hadoop Configurations > loaded from the local directory specified by the *HADOOP_CONF_DIR* or > *HADOOP_HOME* environment, then bind that ConfigMap to the lifecycle of the > new Flink cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15667) Support to mount Hadoop Configurations
[ https://issues.apache.org/jira/browse/FLINK-15667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-15667. - Resolution: Fixed > Support to mount Hadoop Configurations > -- > > Key: FLINK-15667 > URL: https://issues.apache.org/jira/browse/FLINK-15667 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Canbin Zheng >Assignee: Canbin Zheng >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Mounts the Hadoop configurations, either as a pre-defined config map, or a > local configuration directory on the Pod. > We plan to provide two options for the users to mount custom Hadoop > configurations: > * option 1: specify an existing ConfigMap that contains custom Hadoop > Configurations, one can share a single ConfigMap for more than one Flink > clusters. > * option 2: create a dedicated ConfigMap containing Hadoop Configurations > loaded from the local directory specified by the *HADOOP_CONF_DIR* or > *HADOOP_HOME* environment, then bind that ConfigMap to the lifecycle of the > new Flink cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10973: [FLINK-15647][k8s] Support user-specified annotations for JM/TM Pods
flinkbot edited a comment on issue #10973: [FLINK-15647][k8s] Support user-specified annotations for JM/TM Pods URL: https://github.com/apache/flink/pull/10973#issuecomment-580223229 ## CI report: * 33776e6aa098656ddb2820bbac39b4b08f9fa3f8 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/154537476) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6505) * 3bfbfa984bf69d1abf17594aabd4b0e3ec1f782b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15932) Add download url to hive dependencies
[ https://issues.apache.org/jira/browse/FLINK-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee updated FLINK-15932: - Fix Version/s: (was: 1.11.0) > Add download url to hive dependencies > - > > Key: FLINK-15932 > URL: https://issues.apache.org/jira/browse/FLINK-15932 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Documentation >Reporter: Jingsong Lee >Priority: Major > > Now in > [https://ci.apache.org/projects/flink/flink-docs-master/dev/table/hive/#dependencies] > We list all dependencies for all supported hive versions. But no download > url, it is not so handy. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16405) Fix hive-metastore dependency description in connecting to hive docs
[ https://issues.apache.org/jira/browse/FLINK-16405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee updated FLINK-16405: - Fix Version/s: (was: 1.10.1) (was: 1.11.0) > Fix hive-metastore dependency description in connecting to hive docs > > > Key: FLINK-16405 > URL: https://issues.apache.org/jira/browse/FLINK-16405 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Documentation >Reporter: Leonard Xu >Priority: Major > > we need hive-metastore dependency when connect to hive(version 3.1.x) if > using hive-metastore in embedded mode. but looks like we missed it: > [https://ci.apache.org/projects/flink/flink-docs-master/dev/table/hive/#connecting-to-hive] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13278) add Chinese documentation for Hive source/sink
[
https://issues.apache.org/jira/browse/FLINK-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee updated FLINK-13278:
-
Parent: (was: FLINK-10729)
Issue Type: Task (was: Sub-task)
> add Chinese documentation for Hive source/sink
> --
>
> Key: FLINK-13278
> URL: https://issues.apache.org/jira/browse/FLINK-13278
> Project: Flink
> Issue Type: Task
> Components: Connectors / Hive, Documentation
>Reporter: Bowen Li
>Assignee: Zijie Lu
>Priority: Major
> Fix For: 1.9.3
>
>
> add Chinese documentation of Hive source/sink in {{batch/connector_zh.md}}
> it's corresponding English one is FLINK-13277
> cc [~xuefuz] [~lirui] [~Terry1897]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16475) PulsarConsumerSource messageReceiveTimeoutMs parameter should be configurable
[ https://issues.apache.org/jira/browse/FLINK-16475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064583#comment-17064583 ] Robert Metzger commented on FLINK-16475: No worries, and thanks for referring to the new ticket! > PulsarConsumerSource messageReceiveTimeoutMs parameter should be configurable > - > > Key: FLINK-16475 > URL: https://issues.apache.org/jira/browse/FLINK-16475 > Project: Flink > Issue Type: Bug >Affects Versions: 1.9.2 > Environment: Not releveant >Reporter: Jason Kania >Priority: Major > > The messageReceiveTimeoutMs value in the PulsarConsumerSource class is > hardcoded to 100ms but should be configurable to accommodate different > hardware setups such as developer labs where the performance level may not be > as critical. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16713) Flink can not support the type of source table to query data from es
jackray wang created FLINK-16713: Summary: Flink can not support the type of source table to query data from es Key: FLINK-16713 URL: https://issues.apache.org/jira/browse/FLINK-16713 Project: Flink Issue Type: Improvement Components: Table SQL / Client Affects Versions: 1.10.0 Reporter: jackray wang [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#elasticsearch-connector] For append-only queries, the connector can also operate in [append mode|https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#update-modes] for exchanging only INSERT messages with the external system. If no key is defined by the query, a key is automatically generated by Elasticsearch. I want to know ,why the connector of flink with ES just support sink but doesn't support source .Which version could add this feature to ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-4407) Add DSL for specifying Window Triggers
[ https://issues.apache.org/jira/browse/FLINK-4407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kostas Kloudas closed FLINK-4407. - Resolution: Won't Fix > Add DSL for specifying Window Triggers > -- > > Key: FLINK-4407 > URL: https://issues.apache.org/jira/browse/FLINK-4407 > Project: Flink > Issue Type: New Feature > Components: API / DataStream >Reporter: Kostas Kloudas >Assignee: Kostas Kloudas >Priority: Major > Labels: pull-request-available > > This issue refers to the implementation of the trigger DSL. > The specification of the DSL has an open FLIP here: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-9%3A+Trigger+DSL > And is currently under discussion in the dev@ mailing list here: > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-9-Trigger-DSL-td13065.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT.
flinkbot edited a comment on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT. URL: https://github.com/apache/flink/pull/11284#issuecomment-593407935 ## CI report: * 0be3fd11e11d708b254036e9e91fc71f1fd02181 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/154513428) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6492) * ff670f742762f45214ca32df966e02f4dbe6f758 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/154538899) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6507) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-10729) Create a Hive connector for Hive data access in Flink
[
https://issues.apache.org/jira/browse/FLINK-10729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-10729.
Resolution: Fixed
> Create a Hive connector for Hive data access in Flink
> -
>
> Key: FLINK-10729
> URL: https://issues.apache.org/jira/browse/FLINK-10729
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Hive, Table SQL / Ecosystem
>Affects Versions: 1.6.2
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>Priority: Major
>
> As part of Flink-Hive integration effort, it's important for Flink to access
> (read/write) Hive data, which is the responsibility of Hive connector. While
> there is a HCatalog data connector in the code base, it's not complete (i.e.
> missing all connector related classes such as validators, etc.). Further,
> HCatalog interface has many limitations such as accessing a subset of Hive
> data, supporting a subset of Hive data types, etc. In addition, it's not
> actively maintained. In fact, it's now only a sub-project in Hive.
> Therefore, here we propose a complete connector set for Hive tables, not via
> HCatalog, but via direct Hive interface. HCatalog connector will be
> deprecated.
> Please note that connector on Hive metadata is already covered in other
> JIRAs, as {{HiveExternalCatalog}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-6215) Make the StatefulSequenceSource scalable.
[
https://issues.apache.org/jira/browse/FLINK-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kostas Kloudas reassigned FLINK-6215:
-
Assignee: (was: Kostas Kloudas)
> Make the StatefulSequenceSource scalable.
> -
>
> Key: FLINK-6215
> URL: https://issues.apache.org/jira/browse/FLINK-6215
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.3.0
>Reporter: Kostas Kloudas
>Priority: Major
> Labels: pull-request-available
>
> Currently the {{StatefulSequenceSource}} instantiates all the elements to
> emit first and keeps them in memory. This is not scalable as for large
> sequences of elements this can lead to out of memory exceptions.
> To solve this, we can pre-partition the sequence of elements based on the
> {{maxParallelism}} parameter, and just keep state (to checkpoint) per such
> partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-10729) Create a Hive connector for Hive data access in Flink
[
https://issues.apache.org/jira/browse/FLINK-10729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064584#comment-17064584
]
Jingsong Lee commented on FLINK-10729:
--
Close, Thanks for the great work [~xuefuz]
> Create a Hive connector for Hive data access in Flink
> -
>
> Key: FLINK-10729
> URL: https://issues.apache.org/jira/browse/FLINK-10729
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Hive, Table SQL / Ecosystem
>Affects Versions: 1.6.2
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>Priority: Major
>
> As part of Flink-Hive integration effort, it's important for Flink to access
> (read/write) Hive data, which is the responsibility of Hive connector. While
> there is a HCatalog data connector in the code base, it's not complete (i.e.
> missing all connector related classes such as validators, etc.). Further,
> HCatalog interface has many limitations such as accessing a subset of Hive
> data, supporting a subset of Hive data types, etc. In addition, it's not
> actively maintained. In fact, it's now only a sub-project in Hive.
> Therefore, here we propose a complete connector set for Hive tables, not via
> HCatalog, but via direct Hive interface. HCatalog connector will be
> deprecated.
> Please note that connector on Hive metadata is already covered in other
> JIRAs, as {{HiveExternalCatalog}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-7771) Make the operator state queryable
[ https://issues.apache.org/jira/browse/FLINK-7771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kostas Kloudas reassigned FLINK-7771: - Assignee: (was: Kostas Kloudas) > Make the operator state queryable > - > > Key: FLINK-7771 > URL: https://issues.apache.org/jira/browse/FLINK-7771 > Project: Flink > Issue Type: Improvement > Components: Runtime / Queryable State >Affects Versions: 1.4.0 >Reporter: Kostas Kloudas >Priority: Major > > There seem to be some requests for making the operator (non-keyed) state > queryable. This means that the user will specify the *uuid* of the operator > and the *taskId*, and he will be able to access the state that corresponds to > that operator and for that specific task. > This issue will serve to document the discussion on the topic, so that > everybody can participate. > I also link [~till.rohrmann] and [~skonto] as he also mentioned that this > feature could be helpful. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11481: [FLINK-13691][Table]Remove deprecated query config
flinkbot edited a comment on issue #11481: [FLINK-13691][Table]Remove deprecated query config URL: https://github.com/apache/flink/pull/11481#issuecomment-602174011 ## CI report: * cb9eb2458536cf858d110b30c6481b3d6597c9bc Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/154532502) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6502) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11471: [FLINK-16692] flink joblistener can register from config
flinkbot edited a comment on issue #11471: [FLINK-16692] flink joblistener can register from config URL: https://github.com/apache/flink/pull/11471#issuecomment-602012274 ## CI report: * ad9049971efa640a2b22ee8f258b0a0d7f64cc47 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/154513464) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6493) * 7f0c7e868fce22437c1b4b7a02fece52afb70f4a UNKNOWN * 57f1b99ac6db6626e6c78cb7b39e2026b59d1282 UNKNOWN * 84606e23725fc6c688ce72da42d06b8f30ac4910 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/154537643) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6506) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11448: [FLINK-16666][python][table] Support new Python dependency configuration options in flink-java, flink-streaming-java and flink-table.
flinkbot edited a comment on issue #11448: [FLINK-1][python][table] Support new Python dependency configuration options in flink-java, flink-streaming-java and flink-table. URL: https://github.com/apache/flink/pull/11448#issuecomment-601042255 ## CI report: * d595c9fddd3ac097d6861e53305ea7f7a558 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/154041174) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6386) * 2821e4a2542bffceeef1b742f43a09cae359e8ba UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16692) flink joblistener can register from config
[
https://issues.apache.org/jira/browse/FLINK-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064587#comment-17064587
]
Kostas Kloudas commented on FLINK-16692:
Hi, I think this is a nice feature to have. I have 2 comments on the design
here:
1) I believe that the option should go in the {{DeploymentOptions}} rather than
the {{ExecutionOptions}}
2) the name has to be aligned with the names of the other parameters, so it has
to be something like: {{execution.job-listeners}}.
> flink joblistener can register from config
> --
>
> Key: FLINK-16692
> URL: https://issues.apache.org/jira/browse/FLINK-16692
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.10.0
>Reporter: jackylau
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> we should do as spark does ,which can register listener from conf such as
> "spark.extraListeners"。 And it will be convinient for users when users just
> want to set hook
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] azagrebin commented on issue #11445: [FLINK-16615] Introduce data structures and utilities to calculate Job Manager memory components
azagrebin commented on issue #11445: [FLINK-16615] Introduce data structures and utilities to calculate Job Manager memory components URL: https://github.com/apache/flink/pull/11445#issuecomment-602426159 Thanks for the review @xintongsong I refactored more the class structure. The final composition is a bit different but goes into a similar direction. This should also allow better test separation (probably later) comparing to the inheritance approach which is also in one of commits. I also addressed smaller comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16708) When a JDBC connection has been closed, the retry policy of the JDBCUpsertOutputFormat cannot take effect and may result in data loss
[
https://issues.apache.org/jira/browse/FLINK-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064588#comment-17064588
]
Jingsong Lee commented on FLINK-16708:
--
Hi, Thanks for reporting. Can you try FLINK-16281 ?
> When a JDBC connection has been closed, the retry policy of the
> JDBCUpsertOutputFormat cannot take effect and may result in data loss
> -
>
> Key: FLINK-16708
> URL: https://issues.apache.org/jira/browse/FLINK-16708
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.10.0
>Reporter: tangshangwen
>Assignee: tangshangwen
>Priority: Major
>
> In our test environment, I used the tcpkill command to simulate a scenario
> where the postgresql connection was closed. I found that the retry strategy
> of the flush method did not take effect, and when it retried the second time,
> it could not recognize that the connection had been closed because Before the
> first check whether the connection is closed, the batchStatements of
> PgStatement have been cleared, which causes the second execution to check
> that the batchStatements are empty and return normally.
> {code:java}
> 2020-03-20 21:16:18.246 [jdbc-upsert-output-format-thread-1] ERROR
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat - JDBC executeBatch
> error, retry times = 1
> org.postgresql.util.PSQLException: This connection has been closed.
> at
> org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:857)
> at
> org.postgresql.jdbc.PgConnection.getAutoCommit(PgConnection.java:817)
> at
> org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:813)
> at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:873)
> at
> org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1569)
> at
> org.apache.flink.api.java.io.jdbc.writer.AppendOnlyWriter.executeBatch(AppendOnlyWriter.java:62)
> at
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat.flush(JDBCUpsertOutputFormat.java:159)
> at
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat.lambda$open$0(JDBCUpsertOutputFormat.java:124)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-03-20 21:16:21.247 [jdbc-upsert-output-format-thread-1] ERROR
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat - JDBC executeBatch
> error, retry times = 1
> org.postgresql.util.PSQLException: This connection has been closed.
> at
> org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:857)
> at
> org.postgresql.jdbc.PgConnection.getAutoCommit(PgConnection.java:817)
> at
> org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:813)
> at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:873)
> at
> org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1569)
> at
> org.apache.flink.api.java.io.jdbc.writer.AppendOnlyWriter.executeBatch(AppendOnlyWriter.java:62)
> at
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat.flush(JDBCUpsertOutputFormat.java:159)
> at
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat.lambda$open$0(JDBCUpsertOutputFormat.java:124)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13483) PrestoS3FileSystemITCase.testDirectoryListing fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-13483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-13483:
---
Affects Version/s: 1.10.0
> PrestoS3FileSystemITCase.testDirectoryListing fails on Travis
> -
>
> Key: FLINK-13483
> URL: https://issues.apache.org/jira/browse/FLINK-13483
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
>Affects Versions: 1.10.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Lu Niu
>Priority: Critical
>
> https://api.travis-ci.org/v3/job/564894421/log.txt
> {code}
> PrestoS3FileSystemITCase>AbstractHadoopFileSystemITTest.testDirectoryListing:144->AbstractHadoopFileSystemITTest.checkPathExistence:61
> expected: but was:
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13483) PrestoS3FileSystemITCase.testDirectoryListing fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-13483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-13483:
---
Fix Version/s: 1.11.0
> PrestoS3FileSystemITCase.testDirectoryListing fails on Travis
> -
>
> Key: FLINK-13483
> URL: https://issues.apache.org/jira/browse/FLINK-13483
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
>Affects Versions: 1.10.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Lu Niu
>Priority: Critical
> Fix For: 1.11.0
>
>
> https://api.travis-ci.org/v3/job/564894421/log.txt
> {code}
> PrestoS3FileSystemITCase>AbstractHadoopFileSystemITTest.testDirectoryListing:144->AbstractHadoopFileSystemITTest.checkPathExistence:61
> expected: but was:
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13483) PrestoS3FileSystemITCase.testDirectoryListing fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-13483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-13483:
---
Labels: test-stability (was: )
> PrestoS3FileSystemITCase.testDirectoryListing fails on Travis
> -
>
> Key: FLINK-13483
> URL: https://issues.apache.org/jira/browse/FLINK-13483
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
>Affects Versions: 1.10.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Lu Niu
>Priority: Critical
> Labels: test-stability
> Fix For: 1.11.0
>
>
> https://api.travis-ci.org/v3/job/564894421/log.txt
> {code}
> PrestoS3FileSystemITCase>AbstractHadoopFileSystemITTest.testDirectoryListing:144->AbstractHadoopFileSystemITTest.checkPathExistence:61
> expected: but was:
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16713) Support source mode of elasticsearch connector
[ https://issues.apache.org/jira/browse/FLINK-16713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee updated FLINK-16713: - Component/s: (was: Table SQL / Client) Connectors / ElasticSearch > Support source mode of elasticsearch connector > -- > > Key: FLINK-16713 > URL: https://issues.apache.org/jira/browse/FLINK-16713 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.10.0 >Reporter: jackray wang >Priority: Major > > [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#elasticsearch-connector] > For append-only queries, the connector can also operate in [append > mode|https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#update-modes] > for exchanging only INSERT messages with the external system. If no key is > defined by the query, a key is automatically generated by Elasticsearch. > I want to know ,why the connector of flink with ES just support sink but > doesn't support source .Which version could add this feature to ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16713) Support source mode of elasticsearch connector
[ https://issues.apache.org/jira/browse/FLINK-16713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee updated FLINK-16713: - Summary: Support source mode of elasticsearch connector (was: Flink can not support the type of source table to query data from es) > Support source mode of elasticsearch connector > -- > > Key: FLINK-16713 > URL: https://issues.apache.org/jira/browse/FLINK-16713 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Client >Affects Versions: 1.10.0 >Reporter: jackray wang >Priority: Major > > [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#elasticsearch-connector] > For append-only queries, the connector can also operate in [append > mode|https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#update-modes] > for exchanging only INSERT messages with the external system. If no key is > defined by the query, a key is automatically generated by Elasticsearch. > I want to know ,why the connector of flink with ES just support sink but > doesn't support source .Which version could add this feature to ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16713) Support source mode of elasticsearch connector
[ https://issues.apache.org/jira/browse/FLINK-16713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064590#comment-17064590 ] Jingsong Lee commented on FLINK-16713: -- Hi [~jackray], unfortunately, as far as I know, there is no plan... Welcome to contribute. > Support source mode of elasticsearch connector > -- > > Key: FLINK-16713 > URL: https://issues.apache.org/jira/browse/FLINK-16713 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.10.0 >Reporter: jackray wang >Priority: Major > > [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#elasticsearch-connector] > For append-only queries, the connector can also operate in [append > mode|https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#update-modes] > for exchanging only INSERT messages with the external system. If no key is > defined by the query, a key is automatically generated by Elasticsearch. > I want to know ,why the connector of flink with ES just support sink but > doesn't support source .Which version could add this feature to ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11482: [FLINK-16581][table] Minibatch deduplication lack state TTL bug fix
flinkbot edited a comment on issue #11482: [FLINK-16581][table] Minibatch deduplication lack state TTL bug fix URL: https://github.com/apache/flink/pull/11482#issuecomment-60726 ## CI report: * 95c74ce398cdc779ac895edb7cf3433579f1 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/154529184) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6499) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] TisonKun commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
TisonKun commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity. URL: https://github.com/apache/flink/pull/11320#issuecomment-602429506 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11448: [FLINK-16666][python][table] Support new Python dependency configuration options in flink-java, flink-streaming-java and flink-table.
flinkbot edited a comment on issue #11448: [FLINK-1][python][table] Support new Python dependency configuration options in flink-java, flink-streaming-java and flink-table. URL: https://github.com/apache/flink/pull/11448#issuecomment-601042255 ## CI report: * d595c9fddd3ac097d6861e53305ea7f7a558 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/154041174) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6386) * 2821e4a2542bffceeef1b742f43a09cae359e8ba Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/154540657) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6510) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #11448: [FLINK-16666][python][table] Support new Python dependency configuration options in flink-java, flink-streaming-java and flink
WeiZhong94 commented on a change in pull request #11448:
[FLINK-1][python][table] Support new Python dependency configuration
options in flink-java, flink-streaming-java and flink-table.
URL: https://github.com/apache/flink/pull/11448#discussion_r396254822
##
File path:
flink-java/src/main/java/org/apache/flink/api/java/utils/PythonDependencyManager.java
##
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.java.utils;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.cache.DistributedCache;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.PythonOptions;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.Preconditions;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+import static
org.apache.flink.configuration.PythonOptions.PYTHON_CLIENT_EXECUTABLE;
+import static org.apache.flink.configuration.PythonOptions.PYTHON_EXECUTABLE;
+
+/**
+ * Utility class for Python dependency management. The dependencies will be
registered at the distributed
+ * cache.
+ */
+@Internal
+public class PythonDependencyManager {
+
+ public static final String PYTHON_FILE_PREFIX = "python_file";
+ public static final String PYTHON_REQUIREMENTS_FILE_PREFIX =
"python_requirements_file";
+ public static final String PYTHON_REQUIREMENTS_CACHE_PREFIX =
"python_requirements_cache";
+ public static final String PYTHON_ARCHIVE_PREFIX = "python_archive";
+ public static final String CLUSTER = "cluster";
+ public static final String FILE = "file";
+ public static final String CACHE = "cache";
+ public static final String FILE_DELIMITER = ",";
+ public static final String PARAM_DELIMITER = "#";
+
+ // Metadata of Python dependencies.
+
+ public static final ConfigOption> PYTHON_FILES =
+
ConfigOptions.key("python.internal.files-key-map").mapType().noDefaultValue();
+ public static final ConfigOption>
PYTHON_REQUIREMENTS_FILE =
+
ConfigOptions.key("python.internal.requirements-file-key").mapType().noDefaultValue();
+ public static final ConfigOption> PYTHON_ARCHIVES =
+
ConfigOptions.key("python.internal.archives-key-map").mapType().noDefaultValue();
+
+ private final Configuration config;
+ private final List> cachedFiles;
+ private int counter = -1;
+
+ public PythonDependencyManager(
+ Configuration config,
+ List>
cachedFiles) {
+ this.config = Preconditions.checkNotNull(config);
+ this.cachedFiles = cachedFiles;
+ }
+
+ /**
+* Adds a Python dependency which could be .py files, Python
packages(.zip, .egg etc.) or
+* local directories. The dependencies will be added to the PYTHONPATH
of the Python UDF worker
+* and the local Py4J python client.
+*
+* @param filePath The path of the Python dependency.
+*/
+ public void addPythonFile(String filePath) {
Review comment:
In this PR this method is only public for test purpose. But in following PR
the "add_python_file" method of Python TableEnvironment will point to this
method so I think the annotation is unnecessary as it will be removed soon.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] kl0u closed pull request #11472: [FLINK-16547][yarn] Respect the config option of FileJobGraphRetriever#JOB_GRAPH_FILE_PATH
kl0u closed pull request #11472: [FLINK-16547][yarn] Respect the config option of FileJobGraphRetriever#JOB_GRAPH_FILE_PATH URL: https://github.com/apache/flink/pull/11472 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-16547) Respect the config option of FileJobGraphRetriever#JOB_GRAPH_FILE_PATH
[
https://issues.apache.org/jira/browse/FLINK-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-16547:
---
Labels: pull-request-available (was: )
> Respect the config option of FileJobGraphRetriever#JOB_GRAPH_FILE_PATH
> --
>
> Key: FLINK-16547
> URL: https://issues.apache.org/jira/browse/FLINK-16547
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Affects Versions: 1.10.0
>Reporter: Canbin Zheng
>Assignee: Canbin Zheng
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> Currently, in {{YarnClusterDescriptor#startAppMaster}}, we first write out
> and upload the Flink Configuration file, then start to write out the JobGraph
> file and set its name into the Flink Configuration object, the afterward
> setting is not written into the Flink Configuration file so that it does not
> take effect in the cluster side.
> Since in the client-side we name the JobGraph file with the default value of
> FileJobGraphRetriever.JOB_GRAPH_FILE_PATH option, the cluster side could
> succeed in retrieving that file.
> This ticket proposes to write out the JobGraph file before the Configuration
> file to ensure that the setting of FileJobGraphRetriever.JOB_GRAPH_FILE_PATH
> is delivered to the cluster side.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] WeiZhong94 commented on a change in pull request #11448: [FLINK-16666][python][table] Support new Python dependency configuration options in flink-java, flink-streaming-java and flink
WeiZhong94 commented on a change in pull request #11448:
[FLINK-1][python][table] Support new Python dependency configuration
options in flink-java, flink-streaming-java and flink-table.
URL: https://github.com/apache/flink/pull/11448#discussion_r396257250
##
File path:
flink-java/src/main/java/org/apache/flink/api/java/utils/PythonDependencyManager.java
##
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.java.utils;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.cache.DistributedCache;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.PythonOptions;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.Preconditions;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+import static
org.apache.flink.configuration.PythonOptions.PYTHON_CLIENT_EXECUTABLE;
+import static org.apache.flink.configuration.PythonOptions.PYTHON_EXECUTABLE;
+
+/**
+ * Utility class for Python dependency management. The dependencies will be
registered at the distributed
+ * cache.
+ */
+@Internal
+public class PythonDependencyManager {
+
+ public static final String PYTHON_FILE_PREFIX = "python_file";
+ public static final String PYTHON_REQUIREMENTS_FILE_PREFIX =
"python_requirements_file";
+ public static final String PYTHON_REQUIREMENTS_CACHE_PREFIX =
"python_requirements_cache";
+ public static final String PYTHON_ARCHIVE_PREFIX = "python_archive";
+ public static final String CLUSTER = "cluster";
+ public static final String FILE = "file";
+ public static final String CACHE = "cache";
+ public static final String FILE_DELIMITER = ",";
+ public static final String PARAM_DELIMITER = "#";
+
+ // Metadata of Python dependencies.
+
+ public static final ConfigOption> PYTHON_FILES =
+
ConfigOptions.key("python.internal.files-key-map").mapType().noDefaultValue();
+ public static final ConfigOption>
PYTHON_REQUIREMENTS_FILE =
+
ConfigOptions.key("python.internal.requirements-file-key").mapType().noDefaultValue();
+ public static final ConfigOption> PYTHON_ARCHIVES =
+
ConfigOptions.key("python.internal.archives-key-map").mapType().noDefaultValue();
+
+ private final Configuration config;
+ private final List> cachedFiles;
+ private int counter = -1;
+
+ public PythonDependencyManager(
+ Configuration config,
+ List>
cachedFiles) {
+ this.config = Preconditions.checkNotNull(config);
+ this.cachedFiles = cachedFiles;
+ }
+
+ /**
+* Adds a Python dependency which could be .py files, Python
packages(.zip, .egg etc.) or
+* local directories. The dependencies will be added to the PYTHONPATH
of the Python UDF worker
+* and the local Py4J python client.
+*
+* @param filePath The path of the Python dependency.
+*/
+ public void addPythonFile(String filePath) {
+ Preconditions.checkNotNull(filePath);
+ if (fileAlreadyRegistered(filePath)) {
Review comment:
This case is expected because the `configure()` method will be called when
the
`translate()` method of the Table API planners is called. As the
`translate()` can be called many times in interactive programing, a python file
will be registered multiple times.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #11448: [FLINK-16666][python][table] Support new Python dependency configuration options in flink-java, flink-streaming-java and flink
WeiZhong94 commented on a change in pull request #11448:
[FLINK-1][python][table] Support new Python dependency configuration
options in flink-java, flink-streaming-java and flink-table.
URL: https://github.com/apache/flink/pull/11448#discussion_r396257250
##
File path:
flink-java/src/main/java/org/apache/flink/api/java/utils/PythonDependencyManager.java
##
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.java.utils;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.cache.DistributedCache;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.PythonOptions;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.Preconditions;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+import static
org.apache.flink.configuration.PythonOptions.PYTHON_CLIENT_EXECUTABLE;
+import static org.apache.flink.configuration.PythonOptions.PYTHON_EXECUTABLE;
+
+/**
+ * Utility class for Python dependency management. The dependencies will be
registered at the distributed
+ * cache.
+ */
+@Internal
+public class PythonDependencyManager {
+
+ public static final String PYTHON_FILE_PREFIX = "python_file";
+ public static final String PYTHON_REQUIREMENTS_FILE_PREFIX =
"python_requirements_file";
+ public static final String PYTHON_REQUIREMENTS_CACHE_PREFIX =
"python_requirements_cache";
+ public static final String PYTHON_ARCHIVE_PREFIX = "python_archive";
+ public static final String CLUSTER = "cluster";
+ public static final String FILE = "file";
+ public static final String CACHE = "cache";
+ public static final String FILE_DELIMITER = ",";
+ public static final String PARAM_DELIMITER = "#";
+
+ // Metadata of Python dependencies.
+
+ public static final ConfigOption> PYTHON_FILES =
+
ConfigOptions.key("python.internal.files-key-map").mapType().noDefaultValue();
+ public static final ConfigOption>
PYTHON_REQUIREMENTS_FILE =
+
ConfigOptions.key("python.internal.requirements-file-key").mapType().noDefaultValue();
+ public static final ConfigOption> PYTHON_ARCHIVES =
+
ConfigOptions.key("python.internal.archives-key-map").mapType().noDefaultValue();
+
+ private final Configuration config;
+ private final List> cachedFiles;
+ private int counter = -1;
+
+ public PythonDependencyManager(
+ Configuration config,
+ List>
cachedFiles) {
+ this.config = Preconditions.checkNotNull(config);
+ this.cachedFiles = cachedFiles;
+ }
+
+ /**
+* Adds a Python dependency which could be .py files, Python
packages(.zip, .egg etc.) or
+* local directories. The dependencies will be added to the PYTHONPATH
of the Python UDF worker
+* and the local Py4J python client.
+*
+* @param filePath The path of the Python dependency.
+*/
+ public void addPythonFile(String filePath) {
+ Preconditions.checkNotNull(filePath);
+ if (fileAlreadyRegistered(filePath)) {
Review comment:
This case is expected because the `configure()` method will be called when
the
`translate()` method of the Table API planners is called. As the
`translate()` can be called many times in interactive programing, a python file
may be registered multiple times.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (FLINK-16714) Kafka committedOffsets value <0
pingle wang created FLINK-16714: --- Summary: Kafka committedOffsets value <0 Key: FLINK-16714 URL: https://issues.apache.org/jira/browse/FLINK-16714 Project: Flink Issue Type: Improvement Components: Runtime / Metrics Affects Versions: 1.6.3 Reporter: pingle wang Attachments: image-2020-03-23-15-36-16-487.png, image-2020-03-23-15-36-27-910.png dear all: use rest api get kafka offsets sometimes produces errors value like < 0, so use this value monitoring Kafka's lag size will produce false positives. This should be the problem of flink metrics collection? thanks! !image-2020-03-23-15-36-27-910.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-16547) Respect the config option of FileJobGraphRetriever#JOB_GRAPH_FILE_PATH
[
https://issues.apache.org/jira/browse/FLINK-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kostas Kloudas closed FLINK-16547.
--
Resolution: Fixed
Merged on master with 6049b8307482a1d2d67d9642675345fc5093c197
> Respect the config option of FileJobGraphRetriever#JOB_GRAPH_FILE_PATH
> --
>
> Key: FLINK-16547
> URL: https://issues.apache.org/jira/browse/FLINK-16547
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Affects Versions: 1.10.0
>Reporter: Canbin Zheng
>Assignee: Canbin Zheng
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, in {{YarnClusterDescriptor#startAppMaster}}, we first write out
> and upload the Flink Configuration file, then start to write out the JobGraph
> file and set its name into the Flink Configuration object, the afterward
> setting is not written into the Flink Configuration file so that it does not
> take effect in the cluster side.
> Since in the client-side we name the JobGraph file with the default value of
> FileJobGraphRetriever.JOB_GRAPH_FILE_PATH option, the cluster side could
> succeed in retrieving that file.
> This ticket proposes to write out the JobGraph file before the Configuration
> file to ensure that the setting of FileJobGraphRetriever.JOB_GRAPH_FILE_PATH
> is delivered to the cluster side.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wangyang0918 commented on a change in pull request #11284: [FLINK-15911][runtime] Make Flink work with NAT.
wangyang0918 commented on a change in pull request #11284:
[FLINK-15911][runtime] Make Flink work with NAT.
URL: https://github.com/apache/flink/pull/11284#discussion_r396259803
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskManagerRunner.java
##
@@ -351,21 +351,21 @@ public static TaskExecutor startTaskManager(
LOG.info("Starting TaskManager with ResourceID: {}",
resourceID);
- InetAddress remoteAddress =
InetAddress.getByName(rpcService.getAddress());
+ InetAddress externalAddress =
InetAddress.getByName(rpcService.getAddress());
Review comment:
@tillrohrmann @xintongsong
FYI
I think we do not have the requirement that other `TMs` need to be able to
resolve a `TM's ` address. For current standalone Flink cluster on K8s, all the
`TMs` could not resolve other's hostname. However, it works pretty well. This
is the case for current master code without external address.
When introducing the external address, i think it is same that we do not
require other `TMs` need to be able to resolve a `TM's ` address.
Correct me if i am wrong somewhere.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhengcanbin commented on issue #11472: [FLINK-16547][yarn] Respect the config option of FileJobGraphRetriever#JOB_GRAPH_FILE_PATH
zhengcanbin commented on issue #11472: [FLINK-16547][yarn] Respect the config option of FileJobGraphRetriever#JOB_GRAPH_FILE_PATH URL: https://github.com/apache/flink/pull/11472#issuecomment-602436628 > Thanks for the work @zhengcanbin and for the review @TisonKun. I will merge this. > > Side note for a potential future JIRA: in the `startAppMaster()` method we are using sometime the `configuration` argument to the method to get/set config options, and sometime the `flinkConfiguration` which is a class member. Shouldn't we always use the `configuration` argument? This will make the method more self-contained. Thanks for the review @kl0u @TisonKun! I am in favor of always using the `configuration` argument to make the method more self-contained. I would like to create a tracking JIRA. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] kl0u commented on issue #11471: [FLINK-16692] flink joblistener can register from config
kl0u commented on issue #11471: [FLINK-16692] flink joblistener can register from config URL: https://github.com/apache/flink/pull/11471#issuecomment-602438160 Hi @liuyongvs . Thanks for the work, I had some comments on the JIRA and I will have a better look at the PR now. In the meantime, could you make your changes as a single commit? Although not forced yet, we do not allow merge commits in the repo. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-16407) Remove "FAQ" section from Flink website
[ https://issues.apache.org/jira/browse/FLINK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-16407: --- Labels: pull-request-available (was: ) > Remove "FAQ" section from Flink website > --- > > Key: FLINK-16407 > URL: https://issues.apache.org/jira/browse/FLINK-16407 > Project: Flink > Issue Type: Task > Components: Project Website >Reporter: Robert Metzger >Priority: Major > Labels: pull-request-available > > I propose to remove the [FAQ|https://flink.apache.org/faq.html] page from the > Flink website for the following reasons: > - the information on there is not very up to date, nor helpful or extensive > (its a small selection of what somebody a few years ago considered frequent > questions) > - StackOverflow lists a different set of questions as most frequent: > https://stackoverflow.com/questions/tagged/apache-flink?tab=Votes > - The page is only on position 39 in Google Analytics (in the last ~10 months) > I'm happy to hear opinions on this! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] kl0u edited a comment on issue #11471: [FLINK-16692] flink joblistener can register from config
kl0u edited a comment on issue #11471: [FLINK-16692] flink joblistener can register from config URL: https://github.com/apache/flink/pull/11471#issuecomment-602438160 Hi @liuyongvs . Thanks for the work, I had some comments on the JIRA and I will have a better look at the PR now. In the meantime, could you make your changes as a single commit and rename your branch? Although not forced yet, we do not allow merge commits in the repo. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11351: [FLINK-16404][runtime] Solve the potential deadlock problem when reducing exclusive buffers to zero
zhijiangW commented on a change in pull request #11351: [FLINK-16404][runtime] Solve the potential deadlock problem when reducing exclusive buffers to zero URL: https://github.com/apache/flink/pull/11351#discussion_r396264764 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java ## @@ -140,6 +141,8 @@ /** Channels, which notified this input gate about available data. */ private final ArrayDeque inputChannelsWithData = new ArrayDeque<>(); + private final HashMap channelsBlockedByCheckpoint = new HashMap<>(); Review comment: It is better to reuse the blocked states from `CheckpointBarrierAligner` if possible to avoid managing it duplicated in another place and causing potential inconsistency. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT.
flinkbot edited a comment on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT. URL: https://github.com/apache/flink/pull/11284#issuecomment-593407935 ## CI report: * ff670f742762f45214ca32df966e02f4dbe6f758 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/154538899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6507) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16421) Changing default catalog to hive without changing default database fails
[
https://issues.apache.org/jira/browse/FLINK-16421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064611#comment-17064611
]
Gyula Fora commented on FLINK-16421:
my point is that defualt_database should switch to the catalogs default
> Changing default catalog to hive without changing default database fails
>
>
> Key: FLINK-16421
> URL: https://issues.apache.org/jira/browse/FLINK-16421
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: Gyula Fora
>Priority: Major
>
> The default database in Hive is caled "default" not "default_database". This
> causes an error when starting the SQL CLI with hive set as default catalog:
> {code:java}
> Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: A
> database with name [default_database] does not exist in the catalog:
> [hive].Caused by: org.apache.flink.table.catalog.exceptions.CatalogException:
> A database with name [default_database] does not exist in the catalog:
> [hive]. at
> org.apache.flink.table.catalog.CatalogManager.setCurrentDatabase(CatalogManager.java:174)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.useDatabase(TableEnvironmentImpl.java:631)
> at java.util.Optional.ifPresent(Optional.java:159) at
> org.apache.flink.table.client.gateway.local.ExecutionContext.initializeCatalogs(ExecutionContext.java:561)
> at
> org.apache.flink.table.client.gateway.local.ExecutionContext.initializeTableEnvironment(ExecutionContext.java:494)
> at
> org.apache.flink.table.client.gateway.local.ExecutionContext.(ExecutionContext.java:159)
> at
> org.apache.flink.table.client.gateway.local.ExecutionContext.(ExecutionContext.java:118)
> at
> org.apache.flink.table.client.gateway.local.ExecutionContext$Builder.build(ExecutionContext.java:744)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16715) Always use the configuration argument in YarnClusterDescriptor#startAppMaster to make it more self-contained
Canbin Zheng created FLINK-16715:
Summary: Always use the configuration argument in
YarnClusterDescriptor#startAppMaster to make it more self-contained
Key: FLINK-16715
URL: https://issues.apache.org/jira/browse/FLINK-16715
Project: Flink
Issue Type: Improvement
Components: Deployment / YARN
Reporter: Canbin Zheng
Fix For: 1.11.0
In the YarnClusterDescriptor#{{startAppMaster()}} we are using some time the
configuration argument to the method to get/set config options, and
sometimes the flinkConfiguration which is a class member. This ticket
proposes to always use the configuration argument to make the method
more self-contained.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16715) Always use the configuration argument in YarnClusterDescriptor#startAppMaster to make it more self-contained
[
https://issues.apache.org/jira/browse/FLINK-16715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Canbin Zheng updated FLINK-16715:
-
Description: In the YarnClusterDescriptor#{{startAppMaster()}} we are using
some time the {{configuration}} argument to the method to get/set config
options, and sometimes the {{flinkConfiguration}} which is a class member. This
ticket proposes to always use the {{configuration}} argument to make the method
more self-contained. (was: In the YarnClusterDescriptor#{{startAppMaster()}}
we are using some time the configuration argument to the method to
get/set config options, and sometimes the flinkConfiguration which is a
class member. This ticket proposes to always use the configuration
argument to make the method more self-contained.)
> Always use the configuration argument in YarnClusterDescriptor#startAppMaster
> to make it more self-contained
>
>
> Key: FLINK-16715
> URL: https://issues.apache.org/jira/browse/FLINK-16715
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Canbin Zheng
>Priority: Trivial
> Fix For: 1.11.0
>
>
> In the YarnClusterDescriptor#{{startAppMaster()}} we are using some time the
> {{configuration}} argument to the method to get/set config options, and
> sometimes the {{flinkConfiguration}} which is a class member. This ticket
> proposes to always use the {{configuration}} argument to make the method more
> self-contained.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16715) Always use the configuration argument in YarnClusterDescriptor#startAppMaster to make it more self-contained
[
https://issues.apache.org/jira/browse/FLINK-16715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Canbin Zheng updated FLINK-16715:
-
Priority: Trivial (was: Minor)
> Always use the configuration argument in YarnClusterDescriptor#startAppMaster
> to make it more self-contained
>
>
> Key: FLINK-16715
> URL: https://issues.apache.org/jira/browse/FLINK-16715
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Canbin Zheng
>Priority: Trivial
> Fix For: 1.11.0
>
>
> In the YarnClusterDescriptor#{{startAppMaster()}} we are using some time the
> configuration argument to the method to get/set config options, and
> sometimes the flinkConfiguration which is a class member. This ticket
> proposes to always use the configuration argument to make the method
> more self-contained.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] liuzhixing1006 commented on issue #11455: [FLINK-16098] [chinese-translation, Documentation] Translate "Overvie…
liuzhixing1006 commented on issue #11455: [FLINK-16098] [chinese-translation, Documentation] Translate "Overvie… URL: https://github.com/apache/flink/pull/11455#issuecomment-602444612 Hi @JingsongLi @lirui-apache I have finished the modification, Is there anything else I need to do? Thanks~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11351: [FLINK-16404][runtime] Solve the potential deadlock problem when reducing exclusive buffers to zero
zhijiangW commented on a change in pull request #11351: [FLINK-16404][runtime]
Solve the potential deadlock problem when reducing exclusive buffers to zero
URL: https://github.com/apache/flink/pull/11351#discussion_r396269643
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java
##
@@ -416,7 +452,18 @@ public int getUnannouncedCredit() {
* @return Credit which was not announced to the sender yet.
*/
public int getAndResetUnannouncedCredit() {
- return unannouncedCredit.getAndSet(0);
+ synchronized (bufferQueue) {
+ int credit = unannouncedCredit;
+ unannouncedCredit = 0;
+ if (credit == 0) {
Review comment:
It is tricky to make zero credit has a special meaning, and we also break
the previous assumption without negative credits.
I think it might be better to define a separate message to describe the
semantic of barrier alignment to unblock upstream side. Then the credit is
always positive to be consistent as before, and we only define another blocked
state on upstream side to control the buffer transport besides with positive
credits.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-16713) Support source mode of elasticsearch connector
[ https://issues.apache.org/jira/browse/FLINK-16713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064613#comment-17064613 ] jackray wang commented on FLINK-16713: -- Hi [~lzljs3620320] ,this feature is essential for our company，so if nobody to develop this feature ，i will take it. > Support source mode of elasticsearch connector > -- > > Key: FLINK-16713 > URL: https://issues.apache.org/jira/browse/FLINK-16713 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.10.0 >Reporter: jackray wang >Priority: Major > > [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#elasticsearch-connector] > For append-only queries, the connector can also operate in [append > mode|https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#update-modes] > for exchanging only INSERT messages with the external system. If no key is > defined by the query, a key is automatically generated by Elasticsearch. > I want to know ,why the connector of flink with ES just support sink but > doesn't support source .Which version could add this feature to ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-16540) Fully specify bugfix version of Docker containers in Flink Playground docker-compose.yaml files
[ https://issues.apache.org/jira/browse/FLINK-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske closed FLINK-16540. - Fix Version/s: 1.9.0 1.10.0 1.11.0 Resolution: Fixed > Fully specify bugfix version of Docker containers in Flink Playground > docker-compose.yaml files > --- > > Key: FLINK-16540 > URL: https://issues.apache.org/jira/browse/FLINK-16540 > Project: Flink > Issue Type: Improvement > Components: Examples >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0, 1.10.0, 1.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > I recently noticed that we do not guarantee client-jobmanager compatibility > among different bugfix versions of the same minor version (see > https://github.com/ververica/sql-training/issues/8 for details). > In this case, a job submitted via a Flink 1.9.0 client could not be executed > on a Flink 1.9.2 cluster. > For the playgrounds this can easily happen, because we build a custom Docker > image (with a fixed bugfix version) and load the latest Flink images for the > latest bugfix version of the same minor version. > We can avoid such problems by fixing the bugfix version of the Flink images > that we run in the playground. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-16713) Support source mode of elasticsearch connector
[ https://issues.apache.org/jira/browse/FLINK-16713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064613#comment-17064613 ] jackray wang edited comment on FLINK-16713 at 3/23/20, 8:15 AM: Hi [~lzljs3620320] ,this feature is essential for our company，so if nobody to develop this feature ，I will take it. was (Author: jackray): Hi [~lzljs3620320] ,this feature is essential for our company，so if nobody to develop this feature ，i will take it. > Support source mode of elasticsearch connector > -- > > Key: FLINK-16713 > URL: https://issues.apache.org/jira/browse/FLINK-16713 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.10.0 >Reporter: jackray wang >Priority: Major > > [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#elasticsearch-connector] > For append-only queries, the connector can also operate in [append > mode|https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#update-modes] > for exchanging only INSERT messages with the external system. If no key is > defined by the query, a key is automatically generated by Elasticsearch. > I want to know ,why the connector of flink with ES just support sink but > doesn't support source .Which version could add this feature to ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11481: [FLINK-13691][Table]Remove deprecated query config
flinkbot edited a comment on issue #11481: [FLINK-13691][Table]Remove deprecated query config URL: https://github.com/apache/flink/pull/11481#issuecomment-602174011 ## CI report: * cb9eb2458536cf858d110b30c6481b3d6597c9bc Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/154532502) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6502) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11448: [FLINK-16666][python][table] Support new Python dependency configuration options in flink-java, flink-streaming-java and flink-table.
flinkbot edited a comment on issue #11448: [FLINK-1][python][table] Support new Python dependency configuration options in flink-java, flink-streaming-java and flink-table. URL: https://github.com/apache/flink/pull/11448#issuecomment-601042255 ## CI report: * 2821e4a2542bffceeef1b742f43a09cae359e8ba Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/154540657) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6510) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11471: [FLINK-16692] flink joblistener can register from config
flinkbot edited a comment on issue #11471: [FLINK-16692] flink joblistener can register from config URL: https://github.com/apache/flink/pull/11471#issuecomment-602012274 ## CI report: * 7f0c7e868fce22437c1b4b7a02fece52afb70f4a UNKNOWN * 57f1b99ac6db6626e6c78cb7b39e2026b59d1282 UNKNOWN * 84606e23725fc6c688ce72da42d06b8f30ac4910 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/154537643) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6506) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16156) TableAggregateITCase.testGroupByFlatAggregate failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-16156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064619#comment-17064619
]
Kurt Young commented on FLINK-16156:
FunctionITCase.testDynamicTableFunction
has been fixed in FLINK-16118
> TableAggregateITCase.testGroupByFlatAggregate failed on Travis
> --
>
> Key: FLINK-16156
> URL: https://issues.apache.org/jira/browse/FLINK-16156
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.10.1, 1.11.0
>
>
> The {{TableAggregateITCase.testGroupByFlatAggregate}} failed because the
> order of the result did not match:
> {code}
> 11:50:11.239 [ERROR] Failures:
> 11:50:11.239 [ERROR] FunctionITCase.testDynamicTableFunction:611
> Expected: <[Test is a string, 42, null]>
> but: was <[42, null, Test is a string]>
> 11:50:11.239 [ERROR] Errors:
> 11:50:11.239 [ERROR] TableAggregateITCase.testGroupByFlatAggregate:59 »
> JobExecution Job execution ...
> 11:50:11.239 [INFO]
> {code}
> https://api.travis-ci.com/v3/job/288392903/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16156) TableAggregateITCase.testGroupByFlatAggregate failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-16156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064620#comment-17064620
]
Kurt Young commented on FLINK-16156:
TableAggregateITCase.testGroupByFlatAggregate seems to be a random submission
failure
> TableAggregateITCase.testGroupByFlatAggregate failed on Travis
> --
>
> Key: FLINK-16156
> URL: https://issues.apache.org/jira/browse/FLINK-16156
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.10.1, 1.11.0
>
>
> The {{TableAggregateITCase.testGroupByFlatAggregate}} failed because the
> order of the result did not match:
> {code}
> 11:50:11.239 [ERROR] Failures:
> 11:50:11.239 [ERROR] FunctionITCase.testDynamicTableFunction:611
> Expected: <[Test is a string, 42, null]>
> but: was <[42, null, Test is a string]>
> 11:50:11.239 [ERROR] Errors:
> 11:50:11.239 [ERROR] TableAggregateITCase.testGroupByFlatAggregate:59 »
> JobExecution Job execution ...
> 11:50:11.239 [INFO]
> {code}
> https://api.travis-ci.com/v3/job/288392903/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16692) flink joblistener can register from config
[ https://issues.apache.org/jira/browse/FLINK-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064621#comment-17064621 ] jackylau commented on FLINK-16692: -- HI [~kkl0u], 1) i will move the arguments to DeploymentOptions 2) you can review my code, i use "execution.job-listeners" ,which is same with you. > flink joblistener can register from config > -- > > Key: FLINK-16692 > URL: https://issues.apache.org/jira/browse/FLINK-16692 > Project: Flink > Issue Type: Improvement > Components: API / Core >Affects Versions: 1.10.0 >Reporter: jackylau >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 10m > Remaining Estimate: 0h > > we should do as spark does ,which can register listener from conf such as > "spark.extraListeners"。 And it will be convinient for users when users just > want to set hook -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16175) Add config option to switch case sensitive for column names in SQL
[ https://issues.apache.org/jira/browse/FLINK-16175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064624#comment-17064624 ] Dawid Wysakowicz commented on FLINK-16175: -- I agree with the principle of the ticket to support case sensitivity. I think though we should solve it in a uniform way for all aspects at once. Otherwise I think the behaviour would be inconsistent, e.g. case sensitive for column names but case insensitive for table names, which I am sure will surprise many users. > Add config option to switch case sensitive for column names in SQL > -- > > Key: FLINK-16175 > URL: https://issues.apache.org/jira/browse/FLINK-16175 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API, Table SQL / Planner >Affects Versions: 1.11.0 >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: usability > Fix For: 1.11.0 > > > Flink SQL is default CaseSensitive and have no option to config. This issue > aims to support > a configOption so that user can set CaseSensitive for their SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] TisonKun commented on issue #11458: [FLINK-16625][k8s] Extract BootstrapTools#getEnvironmentVariables to a general utility in ConfigurationUtils
TisonKun commented on issue #11458: [FLINK-16625][k8s] Extract BootstrapTools#getEnvironmentVariables to a general utility in ConfigurationUtils URL: https://github.com/apache/flink/pull/11458#issuecomment-602454643 @zhengcanbin please rebase on master. I think this PR itself is good to go. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] tzulitai closed pull request #62: [FLINK-16685] Add a k8s example for the Python SDK
tzulitai closed pull request #62: [FLINK-16685] Add a k8s example for the Python SDK URL: https://github.com/apache/flink-statefun/pull/62 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11471: [FLINK-16692] flink joblistener can register from config
flinkbot edited a comment on issue #11471: [FLINK-16692] flink joblistener can register from config URL: https://github.com/apache/flink/pull/11471#issuecomment-602012274 ## CI report: * 7f0c7e868fce22437c1b4b7a02fece52afb70f4a UNKNOWN * 57f1b99ac6db6626e6c78cb7b39e2026b59d1282 UNKNOWN * 84606e23725fc6c688ce72da42d06b8f30ac4910 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/154537643) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6506) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11486: [FLINK-16712][task] Refactor StreamTask to construct final fields
flinkbot edited a comment on issue #11486: [FLINK-16712][task] Refactor StreamTask to construct final fields URL: https://github.com/apache/flink/pull/11486#issuecomment-602377929 ## CI report: * 53b97eb753986785848b55a6626e01f20e070626 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/154529257) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6500) * 9535112ff0782fd584600cc3fdbefe0457b1bcfb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-16685) Add an example of how to deploy a multi-language function to Kubernetes.
[ https://issues.apache.org/jira/browse/FLINK-16685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai closed FLINK-16685. --- Fix Version/s: statefun-1.1 Resolution: Fixed Merged to master via b9752c895487c188b5e687b6445d06b388d02239 > Add an example of how to deploy a multi-language function to Kubernetes. > > > Key: FLINK-16685 > URL: https://issues.apache.org/jira/browse/FLINK-16685 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Reporter: Igal Shilman >Assignee: Igal Shilman >Priority: Major > Labels: pull-request-available > Fix For: statefun-1.1 > > Time Spent: 20m > Remaining Estimate: 0h > > Add an example of how to deploy a multi-language function to Kubernetes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-16706) Update Stateful Functions master branch version to 2.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-16706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai closed FLINK-16706.
---
Fix Version/s: statefun-1.1
Resolution: Fixed
Merged via 2c9074d443866048c49ec07784ec384cd3054602
> Update Stateful Functions master branch version to 2.0-SNAPSHOT
> ---
>
> Key: FLINK-16706
> URL: https://issues.apache.org/jira/browse/FLINK-16706
> Project: Flink
> Issue Type: Task
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: statefun-1.1
>
>
> As discussed in
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Update-on-Flink-Stateful-Functions-amp-what-are-the-next-steps-td38646.html,
> the community wants the next (and first Apache-endorsed) Stateful Functions
> release number to be 2.0.
> We should update the master branch snapshot version, as well as the {{Affects
> Version}} and {{Fixed Version}} fields of all Stateful Functions JIRAs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16716) Update Roadmap after Flink 1.10 release
Fabian Hueske created FLINK-16716: - Summary: Update Roadmap after Flink 1.10 release Key: FLINK-16716 URL: https://issues.apache.org/jira/browse/FLINK-16716 Project: Flink Issue Type: Bug Components: Project Website Reporter: Fabian Hueske The roadmap on the Flink website needs to be updated to reflect the new features of Flink 1.10 and the planned features and improvements of future releases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] TisonKun commented on issue #11472: [FLINK-16547][yarn] Respect the config option of FileJobGraphRetriever#JOB_GRAPH_FILE_PATH
TisonKun commented on issue #11472: [FLINK-16547][yarn] Respect the config option of FileJobGraphRetriever#JOB_GRAPH_FILE_PATH URL: https://github.com/apache/flink/pull/11472#issuecomment-602458510 > Thanks for the work @zhengcanbin and for the review @TisonKun. I will merge this. > > Side note for a potential future JIRA: in the `startAppMaster()` method we are using sometime the `configuration` argument to the method to get/set config options, and sometime the `flinkConfiguration` which is a class member. Shouldn't we always use the `configuration` argument? This will make the method more self-contained. @kl0u yes I think I get some cycle of that problem already. However, I'd prefer refer to the field `flinkConfiguration`; otherwise user(previous me) might ask what's the difference between *this configuration object* and *the one I've seen as field*? It is one of Java best practices that we don't pass field as parameter in the same class. Besides, I've heard developers complain with the tight couplings between `YarnClusterDescriptor` and `Utils`. There are several huge & context-aware function. If YARN deployment is still under active development, we might think of refactor these codes gradually. cc @wangyang0918 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] TisonKun edited a comment on issue #11472: [FLINK-16547][yarn] Respect the config option of FileJobGraphRetriever#JOB_GRAPH_FILE_PATH
TisonKun edited a comment on issue #11472: [FLINK-16547][yarn] Respect the config option of FileJobGraphRetriever#JOB_GRAPH_FILE_PATH URL: https://github.com/apache/flink/pull/11472#issuecomment-602458510 > Thanks for the work @zhengcanbin and for the review @TisonKun. I will merge this. > > Side note for a potential future JIRA: in the `startAppMaster()` method we are using sometime the `configuration` argument to the method to get/set config options, and sometime the `flinkConfiguration` which is a class member. Shouldn't we always use the `configuration` argument? This will make the method more self-contained. @kl0u yes I think I get some cycle of that problem already. However, I'd prefer refer to the field `flinkConfiguration`; otherwise user(previous me) might ask what's the difference between *this configuration object* and *the one I've seen as field*? It is one of Java best practices that we don't pass field as parameter of this class's method. Besides, I've heard developers complain with the tight couplings between `YarnClusterDescriptor` and `Utils`. There are several huge & context-aware function. If YARN deployment is still under active development, we might think of refactor these codes gradually. cc @wangyang0918 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] kl0u commented on a change in pull request #11471: [FLINK-16692] flink joblistener can register from config
kl0u commented on a change in pull request #11471: [FLINK-16692] flink
joblistener can register from config
URL: https://github.com/apache/flink/pull/11471#discussion_r396289410
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##
@@ -761,6 +768,20 @@ public void configure(ReadableConfig configuration,
ClassLoader classLoader) {
});
config.configure(configuration, classLoader);
checkpointCfg.configure(configuration);
+ configuration.getOptional(ExecutionOptions.JOB_LISTENERS)
+ .ifPresent(l -> {
+ for (String listener : l) {
+ try {
Review comment:
You can use the method `InstantiationUtil#instantiate(final String
className, final Class targetType, final ClassLoader classLoader)`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] kl0u commented on a change in pull request #11471: [FLINK-16692] flink joblistener can register from config
kl0u commented on a change in pull request #11471: [FLINK-16692] flink
joblistener can register from config
URL: https://github.com/apache/flink/pull/11471#discussion_r396281963
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##
@@ -237,6 +237,13 @@ public ExecutionConfig getConfig() {
return cacheFile;
}
+ /**
+* Gets the config JobListeners.
+*/
Review comment:
Annotate as `@PublicEvolving`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-16620) Add attempt information in logging
[
https://issues.apache.org/jira/browse/FLINK-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064631#comment-17064631
]
Roman Khachatryan commented on FLINK-16620:
---
I think this way we'll have in too many places, but probably it's fine.
And something shorter would be better IMO: "$taskName (4/5, attempt 1)".
Should we increment attempt number by 1 too?
> Add attempt information in logging
> --
>
> Key: FLINK-16620
> URL: https://issues.apache.org/jira/browse/FLINK-16620
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
>Affects Versions: 1.10.0
>Reporter: Jiayi Liao
>Assignee: Jiayi Liao
>Priority: Minor
>
> Currently logging in places such as {{Task}} and {{StreamTask}} , is using
> {{taskNameWithSubtasks}} in {{TaskInfo}} to represent an execution. I think
> it'll be more user-friendly if we can add attempt information into the
> logging.
> The format can be consitent with logging information in {{Execution}} :
> {code:java}
> MySink (3/10) - execution #0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] kl0u commented on a change in pull request #11471: [FLINK-16692] flink joblistener can register from config
kl0u commented on a change in pull request #11471: [FLINK-16692] flink joblistener can register from config URL: https://github.com/apache/flink/pull/11471#discussion_r396291783 ## File path: flink-core/src/main/java/org/apache/flink/configuration/ExecutionOptions.java ## @@ -55,4 +56,11 @@ "throughput") ) .build()); + + public static final ConfigOption> JOB_LISTENERS = Review comment: I would suggest to move it to the `DeploymentOptions`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] kl0u commented on a change in pull request #11471: [FLINK-16692] flink joblistener can register from config
kl0u commented on a change in pull request #11471: [FLINK-16692] flink
joblistener can register from config
URL: https://github.com/apache/flink/pull/11471#discussion_r396289877
##
File path:
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironmentComplexConfigurationTest.java
##
@@ -97,4 +105,47 @@ public void
testNotOverridingCachedFilesFromConfiguration() {
Tuple2.of("file3", new
DistributedCache.DistributedCacheEntry("/tmp3", true))
)));
}
+
+ @Test
+ public void testLoadingListenersFromConfiguration() {
+ StreamExecutionEnvironment envFromConfiguration =
StreamExecutionEnvironment.getExecutionEnvironment();
+ List listenersClass =
Arrays.asList(BasicJobSubmittedCounter.class, BasicJobExecutedCounter.class);
+
+ Configuration configuration = new Configuration();
+ configuration.setString("execution.job-listeners",
listenersClass.stream().map(l -> l.getName()).collect(Collectors.joining(";")));
+
Review comment:
You can use the `ConfigUtils#encodeCollectionToConfig()`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] kl0u commented on a change in pull request #11471: [FLINK-16692] flink joblistener can register from config
kl0u commented on a change in pull request #11471: [FLINK-16692] flink
joblistener can register from config
URL: https://github.com/apache/flink/pull/11471#discussion_r396281963
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##
@@ -237,6 +237,13 @@ public ExecutionConfig getConfig() {
return cacheFile;
}
+ /**
+* Gets the config JobListeners.
+*/
Review comment:
Annotate as `@PublicEvolving`. And I would prefer if we did not even add a
new method in the `env`, if possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-16715) Always use the configuration argument in YarnClusterDescriptor#startAppMaster to make it more self-contained
[
https://issues.apache.org/jira/browse/FLINK-16715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064634#comment-17064634
]
Xintong Song commented on FLINK-16715:
--
I tend to think this differently.
Instead of always using {{configuration}} argument, I would suggest to always
use the class field {{flinkConfiguration}}, for the following reasons.
* As a private non-static method, we don't really need {{startAppMaster}} to be
self-contained.
* Even we change it to always use the argument, it does guarantee that in
future, because the class field is visible to the method anyway. On the other
hand, if we change it to always use the class filed, we can remove the method
argument to make sure there is only one configuration visible to the method.
* It helps reducing the number of arguments needed for the {{startAppMaster}}
method.
Actually, I think we should do the same thing for {{yarnClient}}, which is also
a class field.
> Always use the configuration argument in YarnClusterDescriptor#startAppMaster
> to make it more self-contained
>
>
> Key: FLINK-16715
> URL: https://issues.apache.org/jira/browse/FLINK-16715
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Canbin Zheng
>Priority: Trivial
> Fix For: 1.11.0
>
>
> In the YarnClusterDescriptor#{{startAppMaster()}} we are using some time the
> {{configuration}} argument to the method to get/set config options, and
> sometimes the {{flinkConfiguration}} which is a class member. This ticket
> proposes to always use the {{configuration}} argument to make the method more
> self-contained.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16717) Use headless service for rpc and blob port when flink on K8S
vanderliang created FLINK-16717: --- Summary: Use headless service for rpc and blob port when flink on K8S Key: FLINK-16717 URL: https://issues.apache.org/jira/browse/FLINK-16717 Project: Flink Issue Type: Improvement Components: Deployment / Kubernetes Affects Versions: 1.10.0 Environment: flink 1.10 Reporter: vanderliang Current, when submitting a flink job cluster on kubernetes, the [https://github.com/apache/flink/blob/release-1.10/flink-container/kubernetes/job-cluster-service.yaml] will create Node Port type service for all ports. First, the RPC and blob only need a headless service which could avoid iptables or ipvs forwarding. Second，serverless K8S(like AWS EKS) in public cloud does not support Node Port，which cause some problem when running flink On serverless K8S. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11448: [FLINK-16666][python][table] Support new Python dependency configuration options in flink-java, flink-streaming-java and flink-table.
flinkbot edited a comment on issue #11448: [FLINK-1][python][table] Support new Python dependency configuration options in flink-java, flink-streaming-java and flink-table. URL: https://github.com/apache/flink/pull/11448#issuecomment-601042255 ## CI report: * 2821e4a2542bffceeef1b742f43a09cae359e8ba Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/154540657) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6510) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT.
flinkbot edited a comment on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT. URL: https://github.com/apache/flink/pull/11284#issuecomment-593407935 ## CI report: * ff670f742762f45214ca32df966e02f4dbe6f758 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/154538899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6507) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-16718) KvStateServerHandlerTest leaks Netty ByteBufs
Gary Yao created FLINK-16718:
Summary: KvStateServerHandlerTest leaks Netty ByteBufs
Key: FLINK-16718
URL: https://issues.apache.org/jira/browse/FLINK-16718
Project: Flink
Issue Type: Bug
Components: Runtime / Queryable State, Tests
Affects Versions: 1.10.0, 1.11.0
Reporter: Gary Yao
Assignee: Gary Yao
Fix For: 1.11.0
The {{KvStateServerHandlerTest}} leaks Netty {{ByteBuf}}s.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11486: [FLINK-16712][task] Refactor StreamTask to construct final fields
flinkbot edited a comment on issue #11486: [FLINK-16712][task] Refactor StreamTask to construct final fields URL: https://github.com/apache/flink/pull/11486#issuecomment-602377929 ## CI report: * 9535112ff0782fd584600cc3fdbefe0457b1bcfb Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/154549644) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6512) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-16718) KvStateServerHandlerTest leaks Netty ByteBufs
[
https://issues.apache.org/jira/browse/FLINK-16718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-16718:
-
Description: The {{KvStateServerHandlerTest}} leaks Netty {{ByteBuf}}
instances. (was: The {{KvStateServerHandlerTest}} leaks Netty {{ByteBuf}}s.)
> KvStateServerHandlerTest leaks Netty ByteBufs
> -
>
> Key: FLINK-16718
> URL: https://issues.apache.org/jira/browse/FLINK-16718
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Queryable State, Tests
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Major
> Fix For: 1.11.0
>
>
> The {{KvStateServerHandlerTest}} leaks Netty {{ByteBuf}} instances.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16717) Use headless service for rpc and blob port when flink on K8S
[ https://issues.apache.org/jira/browse/FLINK-16717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064649#comment-17064649 ] Canbin Zheng commented on FLINK-16717: -- Is it a duplicate of FLINK-16602? > Use headless service for rpc and blob port when flink on K8S > > > Key: FLINK-16717 > URL: https://issues.apache.org/jira/browse/FLINK-16717 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Affects Versions: 1.10.0 > Environment: flink 1.10 >Reporter: vanderliang >Priority: Major > > Current, when submitting a flink job cluster on kubernetes, the > [https://github.com/apache/flink/blob/release-1.10/flink-container/kubernetes/job-cluster-service.yaml] > will create Node Port type service for all ports. > First, the RPC and blob only need a headless service which could avoid > iptables or ipvs forwarding. > Second，serverless K8S(like AWS EKS) in public cloud does not support Node > Port，which cause some problem when running flink On serverless K8S. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16638) Flink checkStateMappingCompleteness doesn't include UserDefinedOperatorIDs
[
https://issues.apache.org/jira/browse/FLINK-16638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064651#comment-17064651
]
Roman Khachatryan commented on FLINK-16638:
---
Unfortunately, I can't follow the stacktrace (e.g. LegacyScheduler).
Are you sure your version is 1.10?
> Flink checkStateMappingCompleteness doesn't include UserDefinedOperatorIDs
> --
>
> Key: FLINK-16638
> URL: https://issues.apache.org/jira/browse/FLINK-16638
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.10.0
>Reporter: Bashar Abdul Jawad
>Priority: Critical
>
> [StateAssignmentOperation.checkStateMappingCompleteness|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java#L555]
> doesn't check for UserDefinedOperatorIDs (specified using setUidHash),
> causing the exception:
> {code}
> java.lang.IllegalStateException: There is no operator for the state {}
> {code}
> to be thrown when a savepoint can't be mapped to an ExecutionJobVertex, even
> when the operator hash is explicitly specified.
> I believe this logic should be extended to also include
> UserDefinedOperatorIDs as so:
> {code:java}
> for (ExecutionJobVertex executionJobVertex : tasks) {
> allOperatorIDs.addAll(executionJobVertex.getOperatorIDs());
> allOperatorIDs.addAll(executionJobVertex.getUserDefinedOperatorIDs());
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] liuyongvs commented on issue #11471: [FLINK-16692] flink joblistener can register from config
liuyongvs commented on issue #11471: [FLINK-16692] flink joblistener can register from config URL: https://github.com/apache/flink/pull/11471#issuecomment-602476909 Thanks you so much. Becase my personal PC doen't run fast. so i just not run on my PC and commits directly. i will commit with one commits next time. > Hi @liuyongvs . Thanks for the work, I had some comments on the JIRA and I will have a better look at the PR now. > > In the meantime, could you make your changes as a single commit and rename your branch? Although not forced yet, we do not allow merge commits in the repo. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] GJL closed pull request #11453: [FLINK-16718][tests] Fix ByteBuf leak in KvStateServerHandlerTest
GJL closed pull request #11453: [FLINK-16718][tests] Fix ByteBuf leak in KvStateServerHandlerTest URL: https://github.com/apache/flink/pull/11453 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-16718) KvStateServerHandlerTest leaks Netty ByteBufs
[
https://issues.apache.org/jira/browse/FLINK-16718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-16718:
---
Labels: pull-request-available (was: )
> KvStateServerHandlerTest leaks Netty ByteBufs
> -
>
> Key: FLINK-16718
> URL: https://issues.apache.org/jira/browse/FLINK-16718
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Queryable State, Tests
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> The {{KvStateServerHandlerTest}} leaks Netty {{ByteBuf}} instances.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16718) KvStateServerHandlerTest leaks Netty ByteBufs
[
https://issues.apache.org/jira/browse/FLINK-16718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-16718:
-
Fix Version/s: 1.10.1
> KvStateServerHandlerTest leaks Netty ByteBufs
> -
>
> Key: FLINK-16718
> URL: https://issues.apache.org/jira/browse/FLINK-16718
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Queryable State, Tests
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.1, 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{KvStateServerHandlerTest}} leaks Netty {{ByteBuf}} instances.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11347: [FLINK-14971][checkpointing] Make all the non-IO operations in CheckpointCoordinator single-threaded
flinkbot edited a comment on issue #11347: [FLINK-14971][checkpointing] Make all the non-IO operations in CheckpointCoordinator single-threaded URL: https://github.com/apache/flink/pull/11347#issuecomment-596323413 ## CI report: * 655adff5f2d9298b34b167b5a355f4b536613246 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/153558707) * 9f59db6be464318410e6ce165cb599233980b534 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] ifndef-SleePy commented on a change in pull request #11347: [FLINK-14971][checkpointing] Make all the non-IO operations in CheckpointCoordinator single-threaded
ifndef-SleePy commented on a change in pull request #11347:
[FLINK-14971][checkpointing] Make all the non-IO operations in
CheckpointCoordinator single-threaded
URL: https://github.com/apache/flink/pull/11347#discussion_r396315368
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/PendingCheckpoint.java
##
@@ -311,25 +315,32 @@ public CheckpointException getFailureCause() {
try (CheckpointMetadataOutputStream out =
targetLocation.createMetadataOutputStream()) {
Checkpoints.storeCheckpointMetadata(savepoint, out);
finalizedLocation =
out.closeAndFinalizeCheckpoint();
+ }
+ CompletedCheckpoint completed = new
CompletedCheckpoint(
+ jobId,
+ checkpointId,
+ checkpointTimestamp,
+ System.currentTimeMillis(),
+ operatorStates,
+ masterStates,
+ props,
+ finalizedLocation);
+
+ try {
+
completedCheckpointStore.addCheckpoint(completed);
+ } catch (Throwable t) {
+ completed.discardOnFailedStoring();
Review comment:
It should be.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-16715) Always use the configuration argument in YarnClusterDescriptor#startAppMaster to make it more self-contained
[
https://issues.apache.org/jira/browse/FLINK-16715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064658#comment-17064658
]
Kostas Kloudas commented on FLINK-16715:
[~xintongsong] I see your point. In general the whole {{YarnClusterDescriptor}}
could use some refactoring (e.g. the {{startAppMaster()}} has ~400 lines of
code). I do not have strong feelings about using the one or the other
({{configuration}} or {{flinkConfiguration}) and maybe using the class member
makes it clearer that these changes will be visible to all methods in the
class.
> Always use the configuration argument in YarnClusterDescriptor#startAppMaster
> to make it more self-contained
>
>
> Key: FLINK-16715
> URL: https://issues.apache.org/jira/browse/FLINK-16715
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Canbin Zheng
>Priority: Trivial
> Fix For: 1.11.0
>
>
> In the YarnClusterDescriptor#{{startAppMaster()}} we are using some time the
> {{configuration}} argument to the method to get/set config options, and
> sometimes the {{flinkConfiguration}} which is a class member. This ticket
> proposes to always use the {{configuration}} argument to make the method more
> self-contained.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-16715) Always use the configuration argument in YarnClusterDescriptor#startAppMaster to make it more self-contained
[
https://issues.apache.org/jira/browse/FLINK-16715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064658#comment-17064658
]
Kostas Kloudas edited comment on FLINK-16715 at 3/23/20, 9:33 AM:
--
[~xintongsong] I see your point. In general the whole {{YarnClusterDescriptor}}
could use some refactoring (e.g. the {{startAppMaster()}} has ~400 lines of
code). I do not have strong feelings about using the one or the other
({{configuration}} or {{flinkConfiguration}}) and maybe using the class member
makes it clearer that these changes will be visible to all methods in the
class.
was (Author: kkl0u):
[~xintongsong] I see your point. In general the whole {{YarnClusterDescriptor}}
could use some refactoring (e.g. the {{startAppMaster()}} has ~400 lines of
code). I do not have strong feelings about using the one or the other
({{configuration}} or {{flinkConfiguration}) and maybe using the class member
makes it clearer that these changes will be visible to all methods in the
class.
> Always use the configuration argument in YarnClusterDescriptor#startAppMaster
> to make it more self-contained
>
>
> Key: FLINK-16715
> URL: https://issues.apache.org/jira/browse/FLINK-16715
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Canbin Zheng
>Priority: Trivial
> Fix For: 1.11.0
>
>
> In the YarnClusterDescriptor#{{startAppMaster()}} we are using some time the
> {{configuration}} argument to the method to get/set config options, and
> sometimes the {{flinkConfiguration}} which is a class member. This ticket
> proposes to always use the {{configuration}} argument to make the method more
> self-contained.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
