[GitHub] [flink] flinkbot edited a comment on pull request #14068: [FLINK-200137][python] Emit timestamps of current records to downstream in PythonFunctionOperator.

[GitBox]

flinkbot edited a comment on pull request #14068:
URL: https://github.com/apache/flink/pull/14068#issuecomment-726851708


   
   ## CI report:
   
   * ba91f7f17ba4306ae7e78909029228f114a8b66e Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9727)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9699)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Closed] (FLINK-19816) Flink restored from a wrong checkpoint (a very old one and not the last completed one)

[Till Rohrmann (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-19816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-19816.
-
Resolution: Fixed

Fixed via

1.12.0: cb850fdda2b40866f3b0782e038ae4bce35c9eb0
1.11.3: a2925a0d2e894bf28aaced2993ec453589d143de

> Flink restored from a wrong checkpoint (a very old one and not the last 
> completed one)
> --
>
> Key: FLINK-19816
> URL: https://issues.apache.org/jira/browse/FLINK-19816
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Coordination
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Steven Zhen Wu
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.12.0, 1.11.3
>
>
> h2. Summary
> Upon failure, it seems that Flink didn't restore from the last completed 
> checkpoint. Instead, it restored from a very old checkpoint. As a result, 
> Kafka offsets are invalid and caused the job to replay from the beginning as 
> Kafka consumer "auto.offset.reset" was set to "EARLIEST".
> This is an embarrassingly parallel stateless job. Parallelism is over 1,000. 
> I have the full log file from jobmanager at INFO level available upon request.
> h2. Sequence of events from the logs
> Just before the failure, checkpoint *210768* completed.
> {code}
> 2020-10-25 02:35:05,970 INFO 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator 
> [jobmanager-future-thread-5] - Completed checkpoint 210768 for job 
> 233b4938179c06974e4535ac8a868675 (4623776 bytes in 120402 ms).
> {code}
> During restart, somehow it decided to restore from a very old checkpoint 
> *203531*.
> {code:java}
> 2020-10-25 02:36:03,301 INFO  
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess 
> [cluster-io-thread-3]  - Start SessionDispatcherLeaderProcess.
> 2020-10-25 02:36:03,302 INFO  
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess 
> [cluster-io-thread-5]  - Recover all persisted job graphs.
> 2020-10-25 02:36:03,304 INFO  com.netflix.bdp.s3fs.BdpS3FileSystem
>  [cluster-io-thread-25]  - Deleting path: 
> s3:///checkpoints/XM3B/clapp_avro-clapp_avro_nontvui/1593/233b4938179c06974e4535ac8a868675/chk-210758/c31aec1e-07a7-4193-aa00-3fbe83f9e2e6
> 2020-10-25 02:36:03,307 INFO  
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess 
> [cluster-io-thread-5]  - Trying to recover job with job id 
> 233b4938179c06974e4535ac8a868675.
> 2020-10-25 02:36:03,381 INFO  com.netflix.bdp.s3fs.BdpS3FileSystem
>  [cluster-io-thread-25]  - Deleting path: 
> s3:///checkpoints/Hh86/clapp_avro-clapp_avro_nontvui/1593/233b4938179c06974e4535ac8a868675/chk-210758/4ab92f70-dfcd-4212-9b7f-bdbecb9257fd
> ...
> 2020-10-25 02:36:03,427 INFO  
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore 
> [flink-akka.actor.default-dispatcher-82003]  - Recovering checkpoints from 
> ZooKeeper.
> 2020-10-25 02:36:03,432 INFO  
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore 
> [flink-akka.actor.default-dispatcher-82003]  - Found 0 checkpoints in 
> ZooKeeper.
> 2020-10-25 02:36:03,432 INFO  
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore 
> [flink-akka.actor.default-dispatcher-82003]  - Trying to fetch 0 checkpoints 
> from storage.
> 2020-10-25 02:36:03,432 INFO  
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator
> [flink-akka.actor.default-dispatcher-82003]  - Starting job 
> 233b4938179c06974e4535ac8a868675 from savepoint 
> s3:///checkpoints/metadata/clapp_avro-clapp_avro_nontvui/1113/47e2a25a8d0b696c7d0d423722bb6f54/chk-203531/_metadata
>  ()
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] tillrohrmann closed pull request #14100: [BP-1.11][FLINK-19816] Make job state cleanup dependent on final job result

[GitBox]

tillrohrmann closed pull request #14100:
URL: https://github.com/apache/flink/pull/14100


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] tillrohrmann commented on pull request #14100: [BP-1.11][FLINK-19816] Make job state cleanup dependent on final job result

[GitBox]

tillrohrmann commented on pull request #14100:
URL: https://github.com/apache/flink/pull/14100#issuecomment-729506536


   Manually merged via a2925a0d2e894bf28aaced2993ec453589d143de



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14108: [FLINK-20183][python] Fix the default PYTHONPATH is overwritten in client side

[GitBox]

flinkbot edited a comment on pull request #14108:
URL: https://github.com/apache/flink/pull/14108#issuecomment-729368144


   
   ## CI report:
   
   * 90d6ba76b1efb64f7dd9949f5bbbe027185636bf Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9728)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14112: [FLINK-20207][python] improve the error message printed when submitting the pyflink jobs via 'flink run'.

[GitBox]

flinkbot edited a comment on pull request #14112:
URL: https://github.com/apache/flink/pull/14112#issuecomment-729496292


   
   ## CI report:
   
   * 61e1b28fc13ad5ddc3b7898f4f79d1905c6e145b Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9738)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #14113: [FLINK-20202][python] Add the Check of Unsupported Result Type in Pandas UDAF

[GitBox]

flinkbot commented on pull request #14113:
URL: https://github.com/apache/flink/pull/14113#issuecomment-729508037


   
   ## CI report:
   
   * 9bd25bd7728f8097a45b11b0a871a13277c92c9f UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-6984) Do not recover CompletedCheckpointStore on every restore

[Jiayi Liao (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiayi Liao updated FLINK-6984:
--
Fix Version/s: 1.12.0

> Do not recover CompletedCheckpointStore on every restore
> 
>
> Key: FLINK-6984
> URL: https://issues.apache.org/jira/browse/FLINK-6984
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / State Backends
>Affects Versions: 1.2.1, 1.3.0, 1.4.0
>Reporter: Stefan Richter
>Assignee: Jiayi Liao
>Priority: Major
> Fix For: 1.12.0
>
>
> Currently, restoring the latest checkpoint always recovers the 
> {{CompletedCheckpointStore}} in 
> {{CheckpointCoordinator::restoreLatestCheckpointedState(...)}} by calling 
> {{completedCheckpointStore.recover(sharedStateRegistry)}}. I think this is 
> not required on every restore, but can happen only once when the JM becomes 
> leader, e.g. when the {{CheckpointCoordinator}} is created. This can avoid 
> unnecessary work and simplify the logic, e.g. for populating the 
> {{SharedStateRegistry}}, and we could probably also move the ownership for 
> {{SharedStateRegistry}} into some {{AbstractCompletedCheckpontStore}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (FLINK-6984) Do not recover CompletedCheckpointStore on every restore

[Jiayi Liao (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiayi Liao closed FLINK-6984.
-
Resolution: Duplicate

Resoloved in FLINK-19596.

> Do not recover CompletedCheckpointStore on every restore
> 
>
> Key: FLINK-6984
> URL: https://issues.apache.org/jira/browse/FLINK-6984
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / State Backends
>Affects Versions: 1.2.1, 1.3.0, 1.4.0
>Reporter: Stefan Richter
>Assignee: Jiayi Liao
>Priority: Major
>
> Currently, restoring the latest checkpoint always recovers the 
> {{CompletedCheckpointStore}} in 
> {{CheckpointCoordinator::restoreLatestCheckpointedState(...)}} by calling 
> {{completedCheckpointStore.recover(sharedStateRegistry)}}. I think this is 
> not required on every restore, but can happen only once when the JM becomes 
> leader, e.g. when the {{CheckpointCoordinator}} is created. This can avoid 
> unnecessary work and simplify the logic, e.g. for populating the 
> {{SharedStateRegistry}}, and we could probably also move the ownership for 
> {{SharedStateRegistry}} into some {{AbstractCompletedCheckpontStore}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] JingsongLi merged pull request #14097: [FLINK-20028][table-planner-blink] FileCompactionITCase is unstable

[GitBox]

JingsongLi merged pull request #14097:
URL: https://github.com/apache/flink/pull/14097


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-20028) FileCompactionITCase is unstable

[Jingsong Lee (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-20028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234352#comment-17234352
 ] 

Jingsong Lee commented on FLINK-20028:
--

master (1.12): 2465b7150301a114c36ff07b098e78c744f9bf62

Feel free to re-open this if there is a recurrence.

> FileCompactionITCase is unstable
> 
>
> Key: FLINK-20028
> URL: https://issues.apache.org/jira/browse/FLINK-20028
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / FileSystem, Table SQL / Ecosystem
>Affects Versions: 1.12.0
>Reporter: Stephan Ewen
>Assignee: Jingsong Lee
>Priority: Critical
>  Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> The {{ParquetFileCompactionITCase}} hangs and times out.
> Log: 
> https://dev.azure.com/sewen0794/Flink/_build/results?buildId=178&view=logs&j=66592496-52df-56bb-d03e-37509e1d9d0f&t=ae0269db-6796-5583-2e5f-d84757d711aa
> Exception:
> {code}
> org.junit.runners.model.TestTimedOutException: test timed out after 60 seconds
>   at sun.misc.Unsafe.park(Native Method)
>   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>   at 
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
>   at 
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
>   at 
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
>   at 
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
>   at 
> org.apache.flink.table.api.internal.TableResultImpl.awaitInternal(TableResultImpl.java:119)
>   at 
> org.apache.flink.table.pi.internal.TableResultImpl.await(TableResultImpl.java:86)
>   at 
> org.apache.flink.table.planner.runtime.stream.sql.FileCompactionITCaseBase.testNonPartition(FileCompactionITCaseBase.java:91)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (FLINK-20028) FileCompactionITCase is unstable

[Jingsong Lee (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingsong Lee closed FLINK-20028.

Resolution: Fixed

> FileCompactionITCase is unstable
> 
>
> Key: FLINK-20028
> URL: https://issues.apache.org/jira/browse/FLINK-20028
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / FileSystem, Table SQL / Ecosystem
>Affects Versions: 1.12.0
>Reporter: Stephan Ewen
>Assignee: Jingsong Lee
>Priority: Critical
>  Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> The {{ParquetFileCompactionITCase}} hangs and times out.
> Log: 
> https://dev.azure.com/sewen0794/Flink/_build/results?buildId=178&view=logs&j=66592496-52df-56bb-d03e-37509e1d9d0f&t=ae0269db-6796-5583-2e5f-d84757d711aa
> Exception:
> {code}
> org.junit.runners.model.TestTimedOutException: test timed out after 60 seconds
>   at sun.misc.Unsafe.park(Native Method)
>   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>   at 
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
>   at 
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
>   at 
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
>   at 
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
>   at 
> org.apache.flink.table.api.internal.TableResultImpl.awaitInternal(TableResultImpl.java:119)
>   at 
> org.apache.flink.table.pi.internal.TableResultImpl.await(TableResultImpl.java:86)
>   at 
> org.apache.flink.table.planner.runtime.stream.sql.FileCompactionITCaseBase.testNonPartition(FileCompactionITCaseBase.java:91)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] dianfu closed pull request #14108: [FLINK-20183][python] Fix the default PYTHONPATH is overwritten in client side

[GitBox]

dianfu closed pull request #14108:
URL: https://github.com/apache/flink/pull/14108


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] wuchong commented on pull request #12998: [FLINK-18731][table-planner-blink] Fix monotonicity logic of UNIX_TIMESTAMP & UUID functions

[GitBox]

wuchong commented on pull request #12998:
URL: https://github.com/apache/flink/pull/12998#issuecomment-729519905


   @flinkbot run azure



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #12998: [FLINK-18731][table-planner-blink] Fix monotonicity logic of UNIX_TIMESTAMP & UUID functions

[GitBox]

flinkbot edited a comment on pull request #12998:
URL: https://github.com/apache/flink/pull/12998#issuecomment-664380377


   
   ## CI report:
   
   * 9bca4ec897eeca4b204350866c564139b970603e Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9724)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9686)
 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9747)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] godfreyhe commented on pull request #12998: [FLINK-18731][table-planner-blink] Fix monotonicity logic of UNIX_TIMESTAMP & UUID functions

[GitBox]

godfreyhe commented on pull request #12998:
URL: https://github.com/apache/flink/pull/12998#issuecomment-729521417


   my private azure is green 
https://dev.azure.com/godfreyhe/godfreyhe-flink/_build/results?buildId=74&view=results



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] wuchong commented on a change in pull request #13933: [FLINK-19946][Connectors / HBase]Support sink parallelism configuration for Hbase connector

[GitBox]

wuchong commented on a change in pull request #13933:
URL: https://github.com/apache/flink/pull/13933#discussion_r525896588



##
File path: docs/dev/table/connectors/hbase.md
##
@@ -162,6 +162,13 @@ Connector Options
   can be set to '0' with the flush interval set allowing for 
complete async processing of buffered actions.
   
 
+
+  sink.parallelism
+  optional
+  (none)
+  Integer
+  Defines the parallelism of the Hbase sink operator. By default, the 
parallelism is determined by the framework using the same parallelism of the 
upstream chained operator.

Review comment:
   ```suggestion
 Defines the parallelism of the HBase sink operator. By default, 
the parallelism is determined by the framework using the same parallelism of 
the upstream chained operator.
   ```

##
File path: docs/dev/table/connectors/hbase.zh.md
##
@@ -155,6 +155,13 @@ ON myTopic.key = hTable.rowkey;
   写入的参数选项。刷写缓存行的间隔。它能提升写入 HBase 数据库的性能，但是也可能增加延迟。设置为 "0" 
关闭此选项。注意："sink.buffer-flush.max-size" 和 "sink.buffer-flush.max-rows" 同时设置为 
"0"，刷写选项整个异步处理缓存行为。
   
 
+
+  sink.parallelism
+  可选
+  (none)
+  Integer
+  为 Hbase sink operator 定义并行度。默认情况下，并行度由框架决定，和链在一起的上游 operator 一样。

Review comment:
   ```suggestion
 为 HBase sink operator 定义并行度。默认情况下，并行度由框架决定，和链在一起的上游 operator 
一样。
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Assigned] (FLINK-20197) FileSink fails with UnsupportedOperationException when using default values

[Aljoscha Krettek (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned FLINK-20197:


Assignee: Yun Gao

> FileSink fails with UnsupportedOperationException when using default values
> ---
>
> Key: FLINK-20197
> URL: https://issues.apache.org/jira/browse/FLINK-20197
> Project: Flink
>  Issue Type: Bug
>  Components: API / DataStream, Connectors / FileSystem
>Affects Versions: 1.12.0
>Reporter: Till Rohrmann
>Assignee: Yun Gao
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: 1.12.0
>
>
> When using the default values for the new {{FileSink}}, it fails with 
> {code}
> Caused by: java.lang.UnsupportedOperationException: not supported
>   at 
> org.apache.flink.connector.file.sink.writer.FileWriter$BucketerContext.currentProcessingTime(FileWriter.java:296)
>  ~[flink-connector-files-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.DateTimeBucketAssigner.getBucketId(DateTimeBucketAssigner.java:111)
>  ~[flink-file-sink-common-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.DateTimeBucketAssigner.getBucketId(DateTimeBucketAssigner.java:55)
>  ~[flink-file-sink-common-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.connector.file.sink.writer.FileWriter.write(FileWriter.java:189)
>  ~[flink-connector-files-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.runtime.operators.sink.AbstractSinkWriterOperator.processElement(AbstractSinkWriterOperator.java:80)
>  ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71)
>  ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46)
>  ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26)
>  ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
>  ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
>  ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.api.operators.StreamGroupedReduceOperator.processElement(StreamGroupedReduceOperator.java:68)
>  ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:193)
>  ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:179)
>  ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:152)
>  ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
>  ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:372)
>  ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:186)
>  ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:574)
>  ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:538)
>  ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722) 
> ~[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547) 
> ~[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
>   at java.lang.Thread.run(Thread.java:834) ~[?:?]
> {code}
> The problem seems that the 
> {{FileWriter.BucketerContext.currentProcessingTime}} throws an 
> {{UnsupportedOperationException}}. This feels not correct. A job for 
> reproducing the problem can be fo

[GitHub] [flink] wuchong closed pull request #12998: [FLINK-18731][table-planner-blink] Fix monotonicity logic of UNIX_TIMESTAMP & UUID functions

[GitBox]

wuchong closed pull request #12998:
URL: https://github.com/apache/flink/pull/12998


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Closed] (FLINK-18731) The monotonicity of UNIX_TIMESTAMP function is not correct

[Jark Wu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-18731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu closed FLINK-18731.
---
Resolution: Fixed

Fixed in master (1.12.0):
 - 8c301ba850d2ddc11a59802e4b916a9dcd8b42d9
 - 60906fa9c3217c4f36af2a9295ed9c19d5073606

> The monotonicity of UNIX_TIMESTAMP function is not correct
> --
>
> Key: FLINK-18731
> URL: https://issues.apache.org/jira/browse/FLINK-18731
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / Planner
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Currently, the monotonicity of {{UNIX_TIMESTAMP}} function is always 
> {{INCREASING}}, actually, when it has empty function arguments 
> ({{UNIX_TIMESTAMP()}}, is equivalent to {{NOW()}}), its monotonicity is 
> INCREASING. otherwise its monotonicity should be NOT_MONOTONIC. (e.g. 
> UNIX_TIMESTAMP(string))



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] twalthr commented on pull request #14106: [FLINK-17424][e2e] Adopt download strategy of Kafka tests in Elasticsearch tests

[GitBox]

twalthr commented on pull request #14106:
URL: https://github.com/apache/flink/pull/14106#issuecomment-729523679


   Thanks @leonardBang. I removed the version as a parameter. I will merge this 
if there are no objections.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Assigned] (FLINK-20183) Fix the default PYTHONPATH is overwritten in client side

[Dian Fu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dian Fu reassigned FLINK-20183:
---

Assignee: Huang Xingbo

> Fix the default PYTHONPATH is overwritten in client side
> 
>
> Key: FLINK-20183
> URL: https://issues.apache.org/jira/browse/FLINK-20183
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.12.0, 1.11.2
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
>  Labels: pull-request-available
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-20183) Fix the default PYTHONPATH is overwritten in client side

[Dian Fu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dian Fu updated FLINK-20183:

Affects Version/s: 1.10.0

> Fix the default PYTHONPATH is overwritten in client side
> 
>
> Key: FLINK-20183
> URL: https://issues.apache.org/jira/browse/FLINK-20183
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.10.0, 1.12.0, 1.11.2
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.12.0, 1.10.3, 1.11.3
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (FLINK-20183) Fix the default PYTHONPATH is overwritten in client side

[Dian Fu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dian Fu closed FLINK-20183.
---
Fix Version/s: 1.11.3
   1.10.3
   1.12.0
   Resolution: Fixed

Fixed in
- master(1.12.0) via e7caf34a799bfbb4d7e634c1f8216920082d68b2
- release-1.11 via 61f4148524931e47761c52c39e571e5f7bad7885
- release-1.10 via 2eaba7499dbdd3447425e54bea024fc09142a517

> Fix the default PYTHONPATH is overwritten in client side
> 
>
> Key: FLINK-20183
> URL: https://issues.apache.org/jira/browse/FLINK-20183
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.12.0, 1.11.2
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.12.0, 1.10.3, 1.11.3
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #14096: [FLINK-17159] Use testcontainers for Elasticsearch ITCases

[GitBox]

flinkbot edited a comment on pull request #14096:
URL: https://github.com/apache/flink/pull/14096#issuecomment-728845125


   
   ## CI report:
   
   * 870375c22611f70c998db671632c6484552b0488 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9709)
 
   * 75fa5bf8cfb74c1d018d5e8e0f9890c9ec6887d5 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14113: [FLINK-20202][python] Add the Check of Unsupported Result Type in Pandas UDAF

[GitBox]

flinkbot edited a comment on pull request #14113:
URL: https://github.com/apache/flink/pull/14113#issuecomment-729508037


   
   ## CI report:
   
   * 9bd25bd7728f8097a45b11b0a871a13277c92c9f Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9742)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-19979) Sanity check after bash e2e tests for no leftover processes

[Robert Metzger (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234365#comment-17234365
 ] 

Robert Metzger commented on FLINK-19979:


The common pattern seems that this action returns a non-zero exit code:
{{Waiting till process is stopped: pid = 7813 pattern = 'kafka'}}.

> Sanity check after bash e2e tests for no leftover processes
> ---
>
> Key: FLINK-19979
> URL: https://issues.apache.org/jira/browse/FLINK-19979
> Project: Flink
>  Issue Type: New Feature
>  Components: Test Infrastructure
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.12.0
>
>
> As seen in FLINK-19974, if an e2e test is not cleaning up properly, other e2e 
> tests might fail with difficult to diagnose issues.
> I propose to check that no leftover processes (including docker containers) 
> are running after each bash e2e test.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] wuchong commented on pull request #14080: [FLINK-19996][e2e] Add end-to-end IT case for Debezium + Kafka + temporal join

[GitBox]

wuchong commented on pull request #14080:
URL: https://github.com/apache/flink/pull/14080#issuecomment-729527918


   As discussed offline, let's move this tests to IT cases. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] wuchong merged pull request #14102: [FLINK-19635][hbase] Fix HBaseConnectorITCase.testTableSourceSinkWithDDL is unstable with a result mismatch

[GitBox]

wuchong merged pull request #14102:
URL: https://github.com/apache/flink/pull/14102


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-18712) Flink RocksDB statebackend memory leak issue

[Robert Metzger (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-18712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-18712:
---
Fix Version/s: (was: 1.12.0)

> Flink RocksDB statebackend memory leak issue 
> -
>
> Key: FLINK-18712
> URL: https://issues.apache.org/jira/browse/FLINK-18712
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Affects Versions: 1.10.0
>Reporter: Farnight
>Assignee: Yun Tang
>Priority: Critical
>  Labels: usability
>
> When using RocksDB as our statebackend, we found it will lead to memory leak 
> when restarting job (manually or in recovery case).
>  
> How to reproduce:
>  # increase RocksDB blockcache size(e.g. 1G), it is easier to monitor and 
> reproduce.
>  # start a job using RocksDB statebackend.
>  # when the RocksDB blockcache reachs maximum size, restart the job. and 
> monitor the memory usage (k8s pod working set) of the TM.
>  # go through step 2-3 few more times. and memory will keep raising.
>  
> Any solution or suggestion for this? Thanks!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (FLINK-19635) HBaseConnectorITCase.testTableSourceSinkWithDDL is unstable with a result mismatch

[Jark Wu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-19635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu closed FLINK-19635.
---
Resolution: Fixed

Fixed in master (1.12.0): 7ae2be5167b4b143d3aaef2b9bc58faadc2d3a26

> HBaseConnectorITCase.testTableSourceSinkWithDDL is unstable with a result 
> mismatch
> --
>
> Key: FLINK-19635
> URL: https://issues.apache.org/jira/browse/FLINK-19635
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / HBase
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Leonard Xu
>Priority: Critical
>  Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7562&view=logs&j=d44f43ce-542c-597d-bf94-b0718c71e5e8&t=03dca39c-73e8-5aaf-601d-328ae5c35f20
> {code}
> 2020-10-14T04:35:36.9268975Z testTableSourceSinkWithDDL[planner = 
> BLINK_PLANNER, legacy = 
> false](org.apache.flink.connector.hbase2.HBaseConnectorITCase)  Time elapsed: 
> 3.131 sec  <<< FAILURE!
> 2020-10-14T04:35:36.9276246Z java.lang.AssertionError: 
> expected:<[1,10,Hello-1,100,1.01,false,Welt-1,2019-08-18T19:00,2019-08-18,19:00,12345678.0001,
>  
> 2,20,Hello-2,200,2.02,true,Welt-2,2019-08-18T19:01,2019-08-18,19:01,12345678.0002,
>  
> 3,30,Hello-3,300,3.03,false,Welt-3,2019-08-18T19:02,2019-08-18,19:02,12345678.0003,
>  
> 4,40,null,400,4.04,true,Welt-4,2019-08-18T19:03,2019-08-18,19:03,12345678.0004,
>  
> 5,50,Hello-5,500,5.05,false,Welt-5,2019-08-19T19:10,2019-08-19,19:10,12345678.0005,
>  
> 6,60,Hello-6,600,6.06,true,Welt-6,2019-08-19T19:20,2019-08-19,19:20,12345678.0006,
>  
> 7,70,Hello-7,700,7.07,false,Welt-7,2019-08-19T19:30,2019-08-19,19:30,12345678.0007,
>  
> 8,80,null,800,8.08,true,Welt-8,2019-08-19T19:40,2019-08-19,19:40,12345678.0008]>
>  but 
> was:<[1,10,Hello-1,100,1.01,false,Welt-1,2019-08-18T19:00,2019-08-18,19:00,12345678.0001,
>  
> 2,20,Hello-2,200,2.02,true,Welt-2,2019-08-18T19:01,2019-08-18,19:01,12345678.0002,
>  
> 3,30,Hello-3,300,3.03,false,Welt-3,2019-08-18T19:02,2019-08-18,19:02,12345678.0003]>
> 2020-10-14T04:35:36.9281340Z  at org.junit.Assert.fail(Assert.java:88)
> 2020-10-14T04:35:36.9282023Z  at 
> org.junit.Assert.failNotEquals(Assert.java:834)
> 2020-10-14T04:35:36.9328385Z  at 
> org.junit.Assert.assertEquals(Assert.java:118)
> 2020-10-14T04:35:36.9338939Z  at 
> org.junit.Assert.assertEquals(Assert.java:144)
> 2020-10-14T04:35:36.9339880Z  at 
> org.apache.flink.connector.hbase2.HBaseConnectorITCase.testTableSourceSinkWithDDL(HBaseConnectorITCase.java:449)
> 2020-10-14T04:35:36.9341003Z  at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-20170) json deserialize decimal loses precision

[Mulan (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mulan updated FLINK-20170:
--
Attachment: image-2020-11-18-16-51-35-317.png

> json deserialize decimal loses precision
> 
>
> Key: FLINK-20170
> URL: https://issues.apache.org/jira/browse/FLINK-20170
> Project: Flink
>  Issue Type: Improvement
>  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table 
> SQL / API
>Reporter: Mulan
>Priority: Major
> Attachments: image-2020-11-18-16-51-35-317.png
>
>
> {code:java}
> CREATE TABLE ods (
> id BIGINT,
> factor DECIMAL(38, 18)
> ) WITH (
> 'connector.type' = 'kafka',
> 'connector.version' = 'universal',
> 'connector.topic' = '_foo',
> 'connector.topic?' = '_foo',
> 'connector.properties.bootstrap.servers' = 'localhost:9092',
> 'connector.properties.group.id' = 'g',
> 'format.type' = 'json',
> 'update-mode' = 'append'
> );
> {code}
> this following is input data.
> {code:json}
> {"id": 1, "factor": 799.929496989092949698}
> {code}
> this following is output data and loses precision.
> {code:json}
> 1, 799.92949698909300
> {code}
> This following code call readTree() method. This method make value loses 
> precision.
> {code:java}
>   public Row deserialize(byte[] message) throws IOException {
>   try {
>   final JsonNode root = objectMapper.readTree(message);
>   return (Row) runtimeConverter.convert(objectMapper, 
> root);
>   } catch (Throwable t) {
>   throw new IOException("Failed to deserialize JSON 
> object.", t);
>   }
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20210) "RequestFactory" interface in flink elasticsearch connector doesn't support "routing" param

[Oneway Chen (Jira)]

Oneway Chen created FLINK-20210:
---

 Summary: "RequestFactory" interface in flink elasticsearch 
connector doesn't support "routing" param
 Key: FLINK-20210
 URL: https://issues.apache.org/jira/browse/FLINK-20210
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / ElasticSearch
Reporter: Oneway Chen


_org.apache.flink.streaming.connectors.elasticsearch.table.RequestFactory_

This interface doesn't support the "routing" param in its three functions. When 
the "routing" param is obligatory, such as Parent-Child Mapping, it will be 
quite inconvenient to use the connector directly. So is it possible to extend 
this interface?

accordingly, this classes may be changed:

_org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch6DynamicSink_

_org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch7DynamicSink_

_org.apache.flink.streaming.connectors.elasticsearch.table.RowElasticsearchSinkFunction_

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20209) Add missing checkpoint configuration to Flink UI

[Peidian Li (Jira)]

Peidian Li created FLINK-20209:
--

 Summary: Add missing checkpoint configuration to Flink UI
 Key: FLINK-20209
 URL: https://issues.apache.org/jira/browse/FLINK-20209
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Web Frontend
Affects Versions: 1.11.2
Reporter: Peidian Li


Some of the [checkpointing 
configurations|https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#checkpointing]
 are not shown in the Flink UI，Can we consider adding these configurations to 
the checkpoints configuration tab to make it easier for users to view 
checkpoint configurations.

These configurations need to be added：
{code:java}
execution.checkpointing.prefer-checkpoint-for-recovery

execution.checkpointing.tolerable-failed-checkpoints

execution.checkpointing.unaligned
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-20170) json deserialize decimal loses precision

[Mulan (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-20170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234378#comment-17234378
 ] 

Mulan commented on FLINK-20170:
---

I try out this in 1.11 version using new field 'connector' = kafka . This 
happened the same problem.

This is jackson problem. What should I do.

 

!image-2020-11-18-16-51-35-317.png!

> json deserialize decimal loses precision
> 
>
> Key: FLINK-20170
> URL: https://issues.apache.org/jira/browse/FLINK-20170
> Project: Flink
>  Issue Type: Improvement
>  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table 
> SQL / API
>Reporter: Mulan
>Priority: Major
> Attachments: image-2020-11-18-16-51-35-317.png
>
>
> {code:java}
> CREATE TABLE ods (
> id BIGINT,
> factor DECIMAL(38, 18)
> ) WITH (
> 'connector.type' = 'kafka',
> 'connector.version' = 'universal',
> 'connector.topic' = '_foo',
> 'connector.topic?' = '_foo',
> 'connector.properties.bootstrap.servers' = 'localhost:9092',
> 'connector.properties.group.id' = 'g',
> 'format.type' = 'json',
> 'update-mode' = 'append'
> );
> {code}
> this following is input data.
> {code:json}
> {"id": 1, "factor": 799.929496989092949698}
> {code}
> this following is output data and loses precision.
> {code:json}
> 1, 799.92949698909300
> {code}
> This following code call readTree() method. This method make value loses 
> precision.
> {code:java}
>   public Row deserialize(byte[] message) throws IOException {
>   try {
>   final JsonNode root = objectMapper.readTree(message);
>   return (Row) runtimeConverter.convert(objectMapper, 
> root);
>   } catch (Throwable t) {
>   throw new IOException("Failed to deserialize JSON 
> object.", t);
>   }
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-20209) Add missing checkpoint configuration to Flink UI

[Peidian Li (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peidian Li updated FLINK-20209:
---
Description: 
Some of the [checkpointing 
configurations|https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#checkpointing]
 are not shown in the Flink UI，Can we consider adding these configurations to 
the checkpoints configuration tab to make it easier for users to view 
checkpoint configurations.

These configurations need to be added：
{code:java}
execution.checkpointing.prefer-checkpoint-for-recovery

execution.checkpointing.tolerable-failed-checkpoints

execution.checkpointing.unaligned
{code}
 

!image-2020-11-18-16-54-31-638.png|width=915,height=311!

  was:
Some of the [checkpointing 
configurations|https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#checkpointing]
 are not shown in the Flink UI，Can we consider adding these configurations to 
the checkpoints configuration tab to make it easier for users to view 
checkpoint configurations.

These configurations need to be added：
{code:java}
execution.checkpointing.prefer-checkpoint-for-recovery

execution.checkpointing.tolerable-failed-checkpoints

execution.checkpointing.unaligned
{code}


> Add missing checkpoint configuration to Flink UI
> 
>
> Key: FLINK-20209
> URL: https://issues.apache.org/jira/browse/FLINK-20209
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Web Frontend
>Affects Versions: 1.11.2
>Reporter: Peidian Li
>Priority: Major
> Attachments: image-2020-11-18-16-54-31-638.png
>
>
> Some of the [checkpointing 
> configurations|https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#checkpointing]
>  are not shown in the Flink UI，Can we consider adding these configurations to 
> the checkpoints configuration tab to make it easier for users to view 
> checkpoint configurations.
> These configurations need to be added：
> {code:java}
> execution.checkpointing.prefer-checkpoint-for-recovery
> execution.checkpointing.tolerable-failed-checkpoints
> execution.checkpointing.unaligned
> {code}
>  
> !image-2020-11-18-16-54-31-638.png|width=915,height=311!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #13933: [FLINK-19946][Connectors / HBase]Support sink parallelism configuration for Hbase connector

[GitBox]

flinkbot edited a comment on pull request #13933:
URL: https://github.com/apache/flink/pull/13933#issuecomment-722212503


   
   ## CI report:
   
   * eed3ec1b04aa25fcaafe0ac2003d22e5ff202941 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9055)
 
   * 7bd218f99c49a08cc83bb39f8fbf199947ed117a UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-20209) Add missing checkpoint configuration to Flink UI

[Peidian Li (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peidian Li updated FLINK-20209:
---
Attachment: image-2020-11-18-16-54-31-638.png

> Add missing checkpoint configuration to Flink UI
> 
>
> Key: FLINK-20209
> URL: https://issues.apache.org/jira/browse/FLINK-20209
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Web Frontend
>Affects Versions: 1.11.2
>Reporter: Peidian Li
>Priority: Major
> Attachments: image-2020-11-18-16-54-31-638.png
>
>
> Some of the [checkpointing 
> configurations|https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#checkpointing]
>  are not shown in the Flink UI，Can we consider adding these configurations to 
> the checkpoints configuration tab to make it easier for users to view 
> checkpoint configurations.
> These configurations need to be added：
> {code:java}
> execution.checkpointing.prefer-checkpoint-for-recovery
> execution.checkpointing.tolerable-failed-checkpoints
> execution.checkpointing.unaligned
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #14096: [FLINK-17159] Use testcontainers for Elasticsearch ITCases

[GitBox]

flinkbot edited a comment on pull request #14096:
URL: https://github.com/apache/flink/pull/14096#issuecomment-728845125


   
   ## CI report:
   
   * 870375c22611f70c998db671632c6484552b0488 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9709)
 
   * 75fa5bf8cfb74c1d018d5e8e0f9890c9ec6887d5 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9749)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14106: [FLINK-17424][e2e] Adopt download strategy of Kafka tests in Elasticsearch tests

[GitBox]

flinkbot edited a comment on pull request #14106:
URL: https://github.com/apache/flink/pull/14106#issuecomment-729111646


   
   ## CI report:
   
   * f1a25002df7fcce8e7a9edcf1ad0064b8de018e5 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9718)
 
   * 0c28ec6e3b331f9fc367b910a63a3c9dc3edbd02 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14111: [FLINK-20191][document] Add documents for FLIP-95 ability interfaces

[GitBox]

flinkbot edited a comment on pull request #14111:
URL: https://github.com/apache/flink/pull/14111#issuecomment-729488072


   
   ## CI report:
   
   * 94088ed1b38fec47e5e43d7d399021003a1b7482 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9735)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] dianfu commented on a change in pull request #14113: [FLINK-20202][python] Add the Check of Unsupported Result Type in Pandas UDAF

[GitBox]

dianfu commented on a change in pull request #14113:
URL: https://github.com/apache/flink/pull/14113#discussion_r525912705



##
File path: docs/dev/python/table-api-users-guide/udfs/vectorized_python_udfs.md
##
@@ -66,6 +66,7 @@ table_env.sql_query("SELECT add(bigint, bigint) FROM MyTable")
 ## Vectorized Aggregate Functions
 
 Vectorized Python aggregate functions takes one or more `pandas.Series` as the 
inputs and return one scalar value as output.
+NoteNow the return type does not support 
`RowType` and `MapType`

Review comment:
   ```suggestion
   Note Now the return type does not 
support `RowType` and `MapType`
   ```

##
File path: docs/dev/python/table-api-users-guide/udfs/vectorized_python_udfs.md
##
@@ -66,6 +66,7 @@ table_env.sql_query("SELECT add(bigint, bigint) FROM MyTable")
 ## Vectorized Aggregate Functions
 
 Vectorized Python aggregate functions takes one or more `pandas.Series` as the 
inputs and return one scalar value as output.

Review comment:
   Should add one empty line before "Note". Otherwise, it will appear in 
the same line of the previous section.

##
File path: 
docs/dev/python/table-api-users-guide/udfs/vectorized_python_udfs.zh.md
##
@@ -64,6 +64,7 @@ table_env.sql_query("SELECT add(bigint, bigint) FROM MyTable")
 ## 向量化聚合函数
 
 向量化Python聚合函数以一个或多个`pandas.Series`类型的参数作为输入，并返回一个标量值作为输出。
+注意现在返回类型还不支持 `RowType` 和 `MapType` 。

Review comment:
   ```suggestion
   注意 现在返回类型还不支持 `RowType` 和 `MapType` 。
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Assigned] (FLINK-20202) Add the Check of Unsupported Result Type in Pandas UDAF

[Dian Fu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dian Fu reassigned FLINK-20202:
---

Assignee: Huang Xingbo

> Add the Check of Unsupported Result Type in Pandas UDAF
> ---
>
> Key: FLINK-20202
> URL: https://issues.apache.org/jira/browse/FLINK-20202
> Project: Flink
>  Issue Type: Improvement
>  Components: API / Python
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Currently the return type of Pandas UDAF should be a primitive data type, and 
> the returned scalar can be either a python primitive type, e.g., {{int}} or 
> {{float}} or a numpy data type, e.g., {{numpy.int64}} or {{numpy.float64}}. 
> {{Any}} should ideally be a specific scalar type accordingly. We will add 
> related DataType check and throw a more readable exception for unsupported 
> DataTypes. What's more, we will add related notes in docs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-20202) Add check of unsupported result type in Pandas UDAF

[Dian Fu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dian Fu updated FLINK-20202:

Summary: Add check of unsupported result type in Pandas UDAF  (was: Add the 
Check of Unsupported Result Type in Pandas UDAF)

> Add check of unsupported result type in Pandas UDAF
> ---
>
> Key: FLINK-20202
> URL: https://issues.apache.org/jira/browse/FLINK-20202
> Project: Flink
>  Issue Type: Improvement
>  Components: API / Python
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Currently the return type of Pandas UDAF should be a primitive data type, and 
> the returned scalar can be either a python primitive type, e.g., {{int}} or 
> {{float}} or a numpy data type, e.g., {{numpy.int64}} or {{numpy.float64}}. 
> {{Any}} should ideally be a specific scalar type accordingly. We will add 
> related DataType check and throw a more readable exception for unsupported 
> DataTypes. What's more, we will add related notes in docs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-17196) Improve the implementation of Fabric8FlinkKubeClient#getRestEndpoint

[Yang Wang (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-17196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234402#comment-17234402
 ] 

Yang Wang commented on FLINK-17196:
---

I have to revive this ticket again. Because currently if we deploy a native 
Flink cluster on K8s with {{kubernetes.rest-service.exposed.type: 
LoadBalancer}}, the Flink client always prints an unreachable address 
"http://API_ADDRESS:8081";. It is really confusing to the users. For the 
{{NodePort}}, it works well.

 

Hi [~felixzheng], if you have no capacity on this ticket, maybe I could take 
over and improve the user experience.

> Improve the implementation of Fabric8FlinkKubeClient#getRestEndpoint
> 
>
> Key: FLINK-17196
> URL: https://issues.apache.org/jira/browse/FLINK-17196
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / Kubernetes
>Affects Versions: 1.10.0, 1.10.1
>Reporter: Canbin Zheng
>Priority: Major
> Fix For: 1.12.0
>
>
> Currently there are some bugs in the 
> {{Fabric8FlinkKubeClient#getRestEndpoint}} and serveral implicit 
> fallback/toleration behaviors when retrieving the Endpoint.
> This ticket proposes to fix the bugs and improve the implementation by 
> deprecating some implicit fallback/toleration behaviors.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20211) Can not get the JobManager web ip according to the document

[Guowei Ma (Jira)]

Guowei Ma created FLINK-20211:
-

 Summary: Can not get the JobManager web ip according to the 
document
 Key: FLINK-20211
 URL: https://issues.apache.org/jira/browse/FLINK-20211
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Kubernetes
Affects Versions: 1.12.0
Reporter: Guowei Ma


According to 
[https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/native_kubernetes.html#accessing-job-manager-ui]
 's LoadBalancer section I use the following cmd

kubectl get services/cluster-id

But I could not get the EXTERNAL-IP it always non

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-20211) Can not get the JobManager web ip according to the document

[Guowei Ma (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guowei Ma updated FLINK-20211:
--
Description: 
According to 
[https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/native_kubernetes.html#accessing-job-manager-ui]
 's LoadBalancer section I use the following cmd

kubectl get services/cluster-id

But I could not get the EXTERNAL-IP it is always non

 

  was:
According to 
[https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/native_kubernetes.html#accessing-job-manager-ui]
 's LoadBalancer section I use the following cmd

kubectl get services/cluster-id

But I could not get the EXTERNAL-IP it always non

 


> Can not get the JobManager web ip according to the document
> ---
>
> Key: FLINK-20211
> URL: https://issues.apache.org/jira/browse/FLINK-20211
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / Kubernetes
>Affects Versions: 1.12.0
>Reporter: Guowei Ma
>Priority: Critical
>
> According to 
> [https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/native_kubernetes.html#accessing-job-manager-ui]
>  's LoadBalancer section I use the following cmd
> kubectl get services/cluster-id
> But I could not get the EXTERNAL-IP it is always non
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] aljoscha opened a new pull request #14114: [FLINK-20153] Add documentation for BATCH execution mode

[GitBox]

aljoscha opened a new pull request #14114:
URL: https://github.com/apache/flink/pull/14114


   This adds documentation for the new `BATCH` execution mode. We also explain 
`STREAMING` execution mode because there is no central page that explains the 
basic behavior, so far.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): no
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
 - The serializers: no
 - The runtime per-record code paths (performance sensitive): no
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
 - The S3 file system connector: no
   
   ## Documentation
   
   Documentation only.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-10954) Hardlink from files of previous local stored state might cross devices

[Yun Tang (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yun Tang updated FLINK-10954:
-
Fix Version/s: (was: 1.12.0)
   1.13.0

> Hardlink from files of previous local stored state might cross devices
> --
>
> Key: FLINK-10954
> URL: https://issues.apache.org/jira/browse/FLINK-10954
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Affects Versions: 1.6.2
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Critical
> Fix For: 1.13.0
>
>
> Currently, local recovery's base directories is initialized from 
> '{{io.tmp.dirs}}' if parameter '{{taskmanager.state.local.root-dirs}}' is not 
> set. For Yarn environment, the tmp dirs is replaced by its '{{LOCAL_DIRS}}', 
> which might consist of directories from different devices, such as 
> /dump/1/nm-local-dir, /dump/2/nm-local-dir. The local directory for RocksDB 
> is initialized from IOManager's spillingDirectories, which might located in 
> different device from local recovery's folder. However, hard-link between 
> different devices is not allowed, it will throw exception below:
> {code:java}
> java.nio.file.FileSystemException: target -> souce: Invalid cross-device link
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-20153) Add documentation for BATCH execution mode

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-20153:
---
Labels: pull-request-available  (was: )

> Add documentation for BATCH execution mode
> --
>
> Key: FLINK-20153
> URL: https://issues.apache.org/jira/browse/FLINK-20153
> Project: Flink
>  Issue Type: Sub-task
>  Components: API / DataStream
>Reporter: Aljoscha Krettek
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.12.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (FLINK-18121) Support creating Docker image from local Flink distribution

[Guowei Ma (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-18121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234342#comment-17234342
 ] 

Guowei Ma edited comment on FLINK-18121 at 11/18/20, 9:13 AM:
--

+1 from the local file system.

When I test https://issues.apache.org/jira/browse/FLINK-20113 I find I have to 
connect docker hub or build a http server myself. For me it is a little 
inconvenient. For example sometime the connection would be lost or timeout.


was (Author: maguowei):
+1 from the local file system.

When I test https://issues.apache.org/jira/browse/FLINK-20206 I find I have to 
connect docker hub or build a http server myself. For me it is a little 
inconvenient. For example sometime the connection would be lost or timeout.

> Support creating Docker image from local Flink distribution
> ---
>
> Key: FLINK-18121
> URL: https://issues.apache.org/jira/browse/FLINK-18121
> Project: Flink
>  Issue Type: Improvement
>  Components: flink-docker
>Affects Versions: docker-1.11.0.0
>Reporter: Till Rohrmann
>Priority: Major
>
> Currently, 
> https://github.com/apache/flink-docker/blob/dev-master/Dockerfile-debian.template
>  only supports to create a Docker image from a Flink distribution which is 
> hosted on a web server. I think it would be helpful if we could also create a 
> Docker image from a Flink distribution which is stored on one's local file 
> system. That way, one would not have to upload the file or start a web server 
> for serving it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] wuchong closed pull request #14074: [FLINK-20129][table ecosystem] Create a zh download page for all optional sql client components

[GitBox]

wuchong closed pull request #14074:
URL: https://github.com/apache/flink/pull/14074


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Closed] (FLINK-20129) Create a zh download page for all optional sql client components

[Jark Wu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu closed FLINK-20129.
---
Fix Version/s: 1.12.0
   Resolution: Fixed

Fixed in master (1.12.0): d950bbd898caf39e87bab200633cfe4ab889f270

> Create a zh download page for all optional sql client components
> 
>
> Key: FLINK-20129
> URL: https://issues.apache.org/jira/browse/FLINK-20129
> Project: Flink
>  Issue Type: Improvement
>  Components: chinese-translation
>Reporter: Dawid Wysakowicz
>Assignee: hailong wang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.12.0
>
>
> The task is to mirror changes introduced in FLINK-20093 in zh docs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot commented on pull request #14114: [FLINK-20153] Add documentation for BATCH execution mode

[GitBox]

flinkbot commented on pull request #14114:
URL: https://github.com/apache/flink/pull/14114#issuecomment-729545405


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit 70975eba381f72d59bfefd5d5b344e79121e09db (Wed Nov 18 
09:16:21 UTC 2020)
   
   **Warnings:**
* Documentation files were touched, but no `.zh.md` files: Update Chinese 
documentation or file Jira ticket.
* **This pull request references an unassigned [Jira 
ticket](https://issues.apache.org/jira/browse/FLINK-20153).** According to the 
[code contribution 
guide](https://flink.apache.org/contributing/contribute-code.html), tickets 
need to be assigned before starting with the implementation work.
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #13884: [FLINK-19921][runtime] Remove legacy restart strategy

[GitBox]

flinkbot edited a comment on pull request #13884:
URL: https://github.com/apache/flink/pull/13884#issuecomment-720451080


   
   ## CI report:
   
   * b8aa0f9c13f6df568244db5edfbfa56df51c0e33 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8825)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8814)
 
   * 76d8fc44bd6f73200fd57f76cc3ebd64f176eb63 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #13933: [FLINK-19946][Connectors / HBase]Support sink parallelism configuration for Hbase connector

[GitBox]

flinkbot edited a comment on pull request #13933:
URL: https://github.com/apache/flink/pull/13933#issuecomment-722212503


   
   ## CI report:
   
   * eed3ec1b04aa25fcaafe0ac2003d22e5ff202941 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9055)
 
   * 7bd218f99c49a08cc83bb39f8fbf199947ed117a Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9751)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14106: [FLINK-17424][e2e] Adopt download strategy of Kafka tests in Elasticsearch tests

[GitBox]

flinkbot edited a comment on pull request #14106:
URL: https://github.com/apache/flink/pull/14106#issuecomment-729111646


   
   ## CI report:
   
   * f1a25002df7fcce8e7a9edcf1ad0064b8de018e5 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9718)
 
   * 0c28ec6e3b331f9fc367b910a63a3c9dc3edbd02 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9752)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Assigned] (FLINK-20207) Improve the error message printed when submitting the pyflink jobs via 'flink run'

[Dian Fu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dian Fu reassigned FLINK-20207:
---

Assignee: Wei Zhong

> Improve the error message printed when submitting the pyflink jobs via 'flink 
> run'
> --
>
> Key: FLINK-20207
> URL: https://issues.apache.org/jira/browse/FLINK-20207
> Project: Flink
>  Issue Type: Improvement
>  Components: API / Python
>Reporter: Wei Zhong
>Assignee: Wei Zhong
>Priority: Major
>  Labels: pull-request-available
>
> Sometimes the Java stack traces were swallowed when submitting the pyflink 
> jobs via "flink run", e.g.:
> File "/home/cdh272705/poc/T24_parse.py", line 179, in 
> from_kafka_to_oracle_demo
>    
> main_table.execute_insert("sink")#.get_job_client().get_job_execution_result().result()
>  File 
> "/home/cdh272705/.local/lib/python3.6/site-packages/pyflink/table/table.py", 
> line 783, in execute_insert
>    return TableResult(self._j_table.executeInsert(table_path, overwrite))
>  File 
> "/home/cdh272705/.local/lib/python3.6/site-packages/py4j/java_gateway.py", 
> line 1286, in __call__
>    answer, self.gateway_client, self.target_id, self.name)
>  File 
> "/home/cdh272705/.local/lib/python3.6/site-packages/pyflink/util/exceptions.py",
>  line 154, in deco
>    raise exception_mapping[exception](s.split(': ', 1)[1], stack_trace)
> pyflink.util.exceptions.TableException: 'Failed to execute sql'
>  
> The Java stack traces under the TableException were swallowed, which makes 
> the troubleshooting difficult.
> We need to improve the error reporting logic.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-19461) yarn-sesson.sh -jm -tm arguments have no effect

[Till Rohrmann (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-19461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234412#comment-17234412
 ] 

Till Rohrmann commented on FLINK-19461:
---

Should we maybe fail if we see an unrecognized option [~rmetzger]? This could 
have made the problem with {{ytm}} clearer.

> yarn-sesson.sh -jm -tm arguments have no effect
> ---
>
> Key: FLINK-19461
> URL: https://issues.apache.org/jira/browse/FLINK-19461
> Project: Flink
>  Issue Type: Bug
>  Components: Command Line Client, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Major
>  Labels: usability
> Fix For: 1.12.0
>
> Attachments: jobmanager.log, yarn-debug.log
>
>
> It seems that I can set arbitrary values for the documented {{-jm}} and 
> {{-tm}} arguments, not leading to any effects.
> Example: {{./bin/yarn-session -jm 512m -tm 512m}} should fail in my opinion, 
> but it starts with the default memory configuration (1280mb / 1200mb? or so).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] kl0u closed pull request #14082: [FLINK-20143][yarn] Support non-qualified path for Yarn shared lib

[GitBox]

kl0u closed pull request #14082:
URL: https://github.com/apache/flink/pull/14082


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-20211) Can not get the JobManager web ip according to the document

[Yang Wang (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-20211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234413#comment-17234413
 ] 

Yang Wang commented on FLINK-20211:
---

Nice catch. I think it should be {{kubectl get services/-rest}}, not 
{{kubectl get services/}}. I will attach a fix for the documentation.

> Can not get the JobManager web ip according to the document
> ---
>
> Key: FLINK-20211
> URL: https://issues.apache.org/jira/browse/FLINK-20211
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / Kubernetes
>Affects Versions: 1.12.0
>Reporter: Guowei Ma
>Priority: Critical
>
> According to 
> [https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/native_kubernetes.html#accessing-job-manager-ui]
>  's LoadBalancer section I use the following cmd
> kubectl get services/cluster-id
> But I could not get the EXTERNAL-IP it is always non
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink-web] wuchong merged pull request #393: [FLINK-20129] Link to zh SQL components download page

[GitBox]

wuchong merged pull request #393:
URL: https://github.com/apache/flink-web/pull/393


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Comment Edited] (FLINK-20129) Create a zh download page for all optional sql client components

[Jark Wu (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-20129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234410#comment-17234410
 ] 

Jark Wu edited comment on FLINK-20129 at 11/18/20, 9:27 AM:


Fixed in master (1.12.0): d950bbd898caf39e87bab200633cfe4ab889f270
Fixed in flink-web: 0ce9ceb6f07646bf65ce2a3199eefd0fd2151140


was (Author: jark):
Fixed in master (1.12.0): d950bbd898caf39e87bab200633cfe4ab889f270

> Create a zh download page for all optional sql client components
> 
>
> Key: FLINK-20129
> URL: https://issues.apache.org/jira/browse/FLINK-20129
> Project: Flink
>  Issue Type: Improvement
>  Components: chinese-translation
>Reporter: Dawid Wysakowicz
>Assignee: hailong wang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.12.0
>
>
> The task is to mirror changes introduced in FLINK-20093 in zh docs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-19461) yarn-sesson.sh -jm -tm arguments have no effect

[Robert Metzger (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-19461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234415#comment-17234415
 ] 

Robert Metzger commented on FLINK-19461:


Yes, I believe this is a helpful improvement. I very vaguely remember that it 
was difficult set this up with the command line arguments parsing library, but 
that's a long time ago .. we probably use something else today.

> yarn-sesson.sh -jm -tm arguments have no effect
> ---
>
> Key: FLINK-19461
> URL: https://issues.apache.org/jira/browse/FLINK-19461
> Project: Flink
>  Issue Type: Bug
>  Components: Command Line Client, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Major
>  Labels: usability
> Fix For: 1.12.0
>
> Attachments: jobmanager.log, yarn-debug.log
>
>
> It seems that I can set arbitrary values for the documented {{-jm}} and 
> {{-tm}} arguments, not leading to any effects.
> Example: {{./bin/yarn-session -jm 512m -tm 512m}} should fail in my opinion, 
> but it starts with the default memory configuration (1280mb / 1200mb? or so).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-18785) flink goes into dead lock leader election when restoring from a do-not-exist checkpoint/savepoint path

[Till Rohrmann (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-18785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234416#comment-17234416
 ] 

Till Rohrmann commented on FLINK-18785:
---

I think there is not much we can do about it now. My recommendation would be to 
use a failure rate restart strategy so that Yarn will eventually fail the 
application.

> flink goes into dead lock leader election when restoring from a do-not-exist 
> checkpoint/savepoint path
> --
>
> Key: FLINK-18785
> URL: https://issues.apache.org/jira/browse/FLINK-18785
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / YARN, Runtime / Checkpointing, Runtime / 
> Coordination
>Affects Versions: 1.10.0, 1.10.1
> Environment: flink on yarn
> flink-1.10.x
> jdk8
> flink-conf.yaml yarn.application-attempts: 2 (or just delete this config)
> yarn-2.7.2
>Reporter: Kai Chen
>Priority: Major
> Fix For: 1.12.0, 1.10.3, 1.11.4
>
> Attachments: flink_savepoint_path_do_not_exits.jpg, 
> image-2020-07-31-19-04-19-241.png, jobmanager.log.attemp1, 
> jobmanager.log.attemp2-13
>
>
> flink goes into dead lock leader election when restoring from a do-not-exist 
> checkpoint/savepoint path.
> I just run this cmd:
>  bin/flink run -m yarn-cluster  -s "hdfs:///do/not/exist/path" 
> examples/streaming/
>  WindowJoin.jar
>  when i visit UI，i meet this:
>  !image-2020-07-31-19-04-19-241.png!
>  in flink-1.9.3, the program just exits. But in 1.10.x, it stucks in leader 
> election
>  
> Here is the stack trace in `jobmanager.err`:
> ERROR ConnectionState Authentication failed
>  ERROR ClusterEntrypoint Fatal error occurred in the cluster entrypoint.
>  org.apache.flink.runtime.dispatcher.DispatcherException: Could not start 
> recovered job 94b0911af12b61d3ee905baf1.
>  at 
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$handleRecoveredJobStartError$0(Dispatcher.java:218)
>  at 
> org.apache.flink.runtime.dispatcher.Dispatcher$$Lambda$128/130098676.apply(Unknown
>  Source)
>  at 
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
>  at 
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
>  at 
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>  at 
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561)
>  at 
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:739)
>  at 
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
>  at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402)
>  at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195)
>  at 
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
>  at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
>  at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$60/278409878.apply(Unknown
>  Source)
>  at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
>  at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
>  at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
>  at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
>  at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
>  at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
>  at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
>  at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
>  at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
>  at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
>  at akka.actor.ActorCell.invoke(ActorCell.scala:561)
>  at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
>  at akka.dispatch.Mailbox.run(Mailbox.scala:225)
>  at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
>  at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>  at 
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>  at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>  at 
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>  Caused by: java.util.concurrent.CompletionException: 
> java.lang.RuntimeException: 
> org.apache.flink.runtime.client.JobExecutionException: Could not set up 
> JobManager
>  at 
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
>  at 
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
>  at 
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:158

[jira] [Closed] (FLINK-18785) flink goes into dead lock leader election when restoring from a do-not-exist checkpoint/savepoint path

[Till Rohrmann (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-18785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-18785.
-
Fix Version/s: (was: 1.11.4)
   (was: 1.10.3)
   (was: 1.12.0)
   Resolution: Information Provided

> flink goes into dead lock leader election when restoring from a do-not-exist 
> checkpoint/savepoint path
> --
>
> Key: FLINK-18785
> URL: https://issues.apache.org/jira/browse/FLINK-18785
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / YARN, Runtime / Checkpointing, Runtime / 
> Coordination
>Affects Versions: 1.10.0, 1.10.1
> Environment: flink on yarn
> flink-1.10.x
> jdk8
> flink-conf.yaml yarn.application-attempts: 2 (or just delete this config)
> yarn-2.7.2
>Reporter: Kai Chen
>Priority: Major
> Attachments: flink_savepoint_path_do_not_exits.jpg, 
> image-2020-07-31-19-04-19-241.png, jobmanager.log.attemp1, 
> jobmanager.log.attemp2-13
>
>
> flink goes into dead lock leader election when restoring from a do-not-exist 
> checkpoint/savepoint path.
> I just run this cmd:
>  bin/flink run -m yarn-cluster  -s "hdfs:///do/not/exist/path" 
> examples/streaming/
>  WindowJoin.jar
>  when i visit UI，i meet this:
>  !image-2020-07-31-19-04-19-241.png!
>  in flink-1.9.3, the program just exits. But in 1.10.x, it stucks in leader 
> election
>  
> Here is the stack trace in `jobmanager.err`:
> ERROR ConnectionState Authentication failed
>  ERROR ClusterEntrypoint Fatal error occurred in the cluster entrypoint.
>  org.apache.flink.runtime.dispatcher.DispatcherException: Could not start 
> recovered job 94b0911af12b61d3ee905baf1.
>  at 
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$handleRecoveredJobStartError$0(Dispatcher.java:218)
>  at 
> org.apache.flink.runtime.dispatcher.Dispatcher$$Lambda$128/130098676.apply(Unknown
>  Source)
>  at 
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
>  at 
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
>  at 
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>  at 
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561)
>  at 
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:739)
>  at 
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
>  at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402)
>  at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195)
>  at 
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
>  at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
>  at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$60/278409878.apply(Unknown
>  Source)
>  at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
>  at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
>  at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
>  at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
>  at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
>  at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
>  at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
>  at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
>  at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
>  at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
>  at akka.actor.ActorCell.invoke(ActorCell.scala:561)
>  at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
>  at akka.dispatch.Mailbox.run(Mailbox.scala:225)
>  at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
>  at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>  at 
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>  at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>  at 
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>  Caused by: java.util.concurrent.CompletionException: 
> java.lang.RuntimeException: 
> org.apache.flink.runtime.client.JobExecutionException: Could not set up 
> JobManager
>  at 
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
>  at 
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
>  at 
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1584)
>  at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
>  at 
> akka.dispatch.ForkJoinExecutorConfigurator$Akk

[jira] [Commented] (FLINK-19461) yarn-sesson.sh -jm -tm arguments have no effect

[Till Rohrmann (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-19461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234418#comment-17234418
 ] 

Till Rohrmann commented on FLINK-19461:
---

I think we are still using the same library from back in the days.

> yarn-sesson.sh -jm -tm arguments have no effect
> ---
>
> Key: FLINK-19461
> URL: https://issues.apache.org/jira/browse/FLINK-19461
> Project: Flink
>  Issue Type: Bug
>  Components: Command Line Client, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Major
>  Labels: usability
> Fix For: 1.12.0
>
> Attachments: jobmanager.log, yarn-debug.log
>
>
> It seems that I can set arbitrary values for the documented {{-jm}} and 
> {{-tm}} arguments, not leading to any effects.
> Example: {{./bin/yarn-session -jm 512m -tm 512m}} should fail in my opinion, 
> but it starts with the default memory configuration (1280mb / 1200mb? or so).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] rkhachatryan commented on pull request #14051: [FLINK-20107][network] Make InputChannel.statePersister immutable

[GitBox]

rkhachatryan commented on pull request #14051:
URL: https://github.com/apache/flink/pull/14051#issuecomment-729554352


   Thanks for merging!
   
   > Known channels are fully initialized including state persister in task 
thread before being passed to network thread
   
   Before this PR, the fields in known channels weren't `final` (and there is 
no explicit fence). So even being fully initialized, the network thread might 
see them un-initialized



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-19461) yarn-sesson.sh -jm -tm arguments have no effect

[Till Rohrmann (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-19461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234420#comment-17234420
 ] 

Till Rohrmann commented on FLINK-19461:
---

Do you wanna create a ticket for this problem [~rmetzger]?

> yarn-sesson.sh -jm -tm arguments have no effect
> ---
>
> Key: FLINK-19461
> URL: https://issues.apache.org/jira/browse/FLINK-19461
> Project: Flink
>  Issue Type: Bug
>  Components: Command Line Client, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Major
>  Labels: usability
> Fix For: 1.12.0
>
> Attachments: jobmanager.log, yarn-debug.log
>
>
> It seems that I can set arbitrary values for the documented {{-jm}} and 
> {{-tm}} arguments, not leading to any effects.
> Example: {{./bin/yarn-session -jm 512m -tm 512m}} should fail in my opinion, 
> but it starts with the default memory configuration (1280mb / 1200mb? or so).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] alpinegizmo commented on a change in pull request #14114: [FLINK-20153] Add documentation for BATCH execution mode

[GitBox]

alpinegizmo commented on a change in pull request #14114:
URL: https://github.com/apache/flink/pull/14114#discussion_r525933950



##
File path: docs/dev/datastream_execution_mode.md
##
@@ -0,0 +1,242 @@
+---
+title: "Execution Mode (Batch/Streaming)"
+nav-id: datastream_execution_mode
+nav-parent_id: streaming
+nav-pos: 1
+---
+
+
+The DataStream API supports different runtime execution modes from which you
+can choose depending on the requirements of your use case and the
+characteristics of your job.
+
+There is the "classic" execution behavior of the DataStream API which we call
+`STREAMING` execution mode. This should be used for unbounded jobs that require
+continuous incremental processing and are expected to stay online indefinitely.
+
+Additionally, there is a batch-style execution mode that we call `BATCH`
+execution mode. This executes jobs in a way that is more reminiscent of batch
+processing frameworks such as MapReduce. This should be used for bounded jobs
+for which you have a known fixed input and which do not run continuously.
+
+We have these different execution modes because `BATCH` execution allows some
+additional optimizations that we can only do when we now that our input is

Review comment:
   ```suggestion
   additional optimizations that we can only do when we know that our input is
   ```

##
File path: docs/dev/datastream_execution_mode.md
##
@@ -0,0 +1,242 @@
+---
+title: "Execution Mode (Batch/Streaming)"
+nav-id: datastream_execution_mode
+nav-parent_id: streaming
+nav-pos: 1
+---
+
+
+The DataStream API supports different runtime execution modes from which you
+can choose depending on the requirements of your use case and the
+characteristics of your job.
+
+There is the "classic" execution behavior of the DataStream API which we call
+`STREAMING` execution mode. This should be used for unbounded jobs that require
+continuous incremental processing and are expected to stay online indefinitely.
+
+Additionally, there is a batch-style execution mode that we call `BATCH`
+execution mode. This executes jobs in a way that is more reminiscent of batch
+processing frameworks such as MapReduce. This should be used for bounded jobs
+for which you have a known fixed input and which do not run continuously.
+
+We have these different execution modes because `BATCH` execution allows some
+additional optimizations that we can only do when we now that our input is
+bounded. For example, different join/aggregation strategies can be used, in
+additional to a different shuffle implementation that allows more efficient
+failure recovery behavior. We will go into some of the details of the execution
+behavior below.
+
+* This will be replaced by the TOC
+{:toc}
+
+## When can/should I use BATCH execution mode?
+
+The BATCH execution mode can only be used for Jobs/Flink Programs that are
+_bounded_. Boundedness is a property of a data source that tells us whether all
+the input coming from that source is known before execution or whether new data
+will show up, potentially for forever. A job, in turn, is bounded if all its

Review comment:
   ```suggestion
   will show up, potentially indefinitely. A job, in turn, is bounded if all its
   ```

##
File path: docs/dev/datastream_execution_mode.md
##
@@ -0,0 +1,242 @@
+---
+title: "Execution Mode (Batch/Streaming)"
+nav-id: datastream_execution_mode
+nav-parent_id: streaming
+nav-pos: 1
+---
+
+
+The DataStream API supports different runtime execution modes from which you
+can choose depending on the requirements of your use case and the
+characteristics of your job.
+
+There is the "classic" execution behavior of the DataStream API which we call
+`STREAMING` execution mode. This should be used for unbounded jobs that require
+continuous incremental processing and are expected to stay online indefinitely.
+
+Additionally, there is a batch-style execution mode that we call `BATCH`
+execution mode. This executes jobs in a way that is more reminiscent of batch
+processing frameworks such as MapReduce. This should be used for bounded jobs
+for which you have a known fixed input and which do not run continuously.
+
+We have these different execution modes because `BATCH` execution allows some
+additional optimizations that we can only do when we now that our input is
+bounded. For example, different join/aggregation strategies can be used, in
+additional to a different shuffle implementation that allows more efficient
+failure recovery behavior. We will go into some of the details of the execution
+behavior below.
+
+* This will be replaced by the TOC
+{:toc}
+
+## When can/should I use BATCH execution mode?
+
+The BATCH execution mode can only be used for Jobs/Flink Programs that are
+_bounded_. Boundedness is a property of a data source that tells us whether all
+the input coming from that source is known before execution or whether new data
+will show up, potentially for forever. A job, in t

[jira] [Created] (FLINK-20212) Fail command line interfaces when provided with an unrecognized option

[Robert Metzger (Jira)]

Robert Metzger created FLINK-20212:
--

 Summary: Fail command line interfaces when provided with an 
unrecognized option 
 Key: FLINK-20212
 URL: https://issues.apache.org/jira/browse/FLINK-20212
 Project: Flink
  Issue Type: Bug
  Components: Command Line Client, Deployment / YARN
Affects Versions: 1.12.0
Reporter: Robert Metzger


Based on the discussion in FLINK-19461, we can improve the usability of Flink 
by failing if a user provided an unrecognized option.
This will help users debug typos, or misunderstandings reading the 
documentation.





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] leitsiang opened a new pull request #14115: [hotfix][docs] fix table systemFunction lag description.

[GitBox]

leitsiang opened a new pull request #14115:
URL: https://github.com/apache/flink/pull/14115


   
   
   ## What is the purpose of the change
   
   In systemFunctions.md and systemFunctions.zh.md function lag's description 
should "Returns the value of expression at the offsetth row **before** the 
current row in the window" not ~~after the current row~~
   
   
   ## Brief change log
   
   fix wrong description of function lag
   
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (yes / **no**)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / **no**)
 - The serializers: (yes / **no** / don't know)
 - The runtime per-record code paths (performance sensitive): (yes / **no** 
/ don't know)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / 
don't know)
 - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (yes / **no**)
 - If yes, how is the feature documented? (**not applicable** / docs / 
JavaDocs / not documented)
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-docker] tillrohrmann commented on a change in pull request #43: [FLINK-19125] Adopt Jemalloc as default memory allocator to avoid memory leak

[GitBox]

tillrohrmann commented on a change in pull request #43:
URL: https://github.com/apache/flink-docker/pull/43#discussion_r525944915



##
File path: docker-entrypoint.sh
##
@@ -93,21 +94,41 @@ prepare_job_manager_start() {
 envsubst < "${CONF_FILE}" > "${CONF_FILE}.tmp" && mv "${CONF_FILE}.tmp" 
"${CONF_FILE}"
 }
 
+disable_jemalloc_env() {
+  local -n _args=$1
+  if [ "${_args[0]}" = ${COMMAND_DISABLE_JEMALLOC} ]; then
+  echo "Disable Jemalloc as the memory allocator"
+  _args=("${_args[@]:1}")

Review comment:
   I am not a bash expert, hence, could you explain how this change will be 
visible from the outside?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-docker] tillrohrmann commented on a change in pull request #43: [FLINK-19125] Adopt Jemalloc as default memory allocator to avoid memory leak

[GitBox]

tillrohrmann commented on a change in pull request #43:
URL: https://github.com/apache/flink-docker/pull/43#discussion_r525945766



##
File path: docker-entrypoint.sh
##
@@ -93,21 +94,41 @@ prepare_job_manager_start() {
 envsubst < "${CONF_FILE}" > "${CONF_FILE}.tmp" && mv "${CONF_FILE}.tmp" 
"${CONF_FILE}"
 }
 
+disable_jemalloc_env() {
+  local -n _args=$1
+  if [ "${_args[0]}" = ${COMMAND_DISABLE_JEMALLOC} ]; then
+  echo "Disable Jemalloc as the memory allocator"
+  _args=("${_args[@]:1}")

Review comment:
   isn't `_args` a local variable which should only be valid within the 
function?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-19461) yarn-sesson.sh -jm -tm arguments have no effect

[Robert Metzger (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-19461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234425#comment-17234425
 ] 

Robert Metzger commented on FLINK-19461:


I filed FLINK-20212 for it. Since it's not a regression from previous releases, 
I didn't set the fixfor version for the upcoming 1.12.0 release. 

> yarn-sesson.sh -jm -tm arguments have no effect
> ---
>
> Key: FLINK-19461
> URL: https://issues.apache.org/jira/browse/FLINK-19461
> Project: Flink
>  Issue Type: Bug
>  Components: Command Line Client, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Major
>  Labels: usability
> Fix For: 1.12.0
>
> Attachments: jobmanager.log, yarn-debug.log
>
>
> It seems that I can set arbitrary values for the documented {{-jm}} and 
> {{-tm}} arguments, not leading to any effects.
> Example: {{./bin/yarn-session -jm 512m -tm 512m}} should fail in my opinion, 
> but it starts with the default memory configuration (1280mb / 1200mb? or so).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot commented on pull request #14115: [hotfix][docs] fix table systemFunction lag description.

[GitBox]

flinkbot commented on pull request #14115:
URL: https://github.com/apache/flink/pull/14115#issuecomment-729560791


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit 46a1adc285805e0b6254598dbe7f3f629211166e (Wed Nov 18 
09:44:30 UTC 2020)
   
✅no warnings
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-20213) Partition commit is delayed when records keep coming

[Jingsong Lee (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingsong Lee updated FLINK-20213:
-
Fix Version/s: 1.11.3

> Partition commit is delayed when records keep coming
> 
>
> Key: FLINK-20213
> URL: https://issues.apache.org/jira/browse/FLINK-20213
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / FileSystem
>Affects Versions: 1.11.2
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
>Priority: Major
> Fix For: 1.12.0, 1.11.3
>
>
> When set partition-commit.delay=0, Users expect partitions to be committed 
> immediately.
> However, if the record of this partition continues to flow in, the bucket for 
> the partition will be activated, and no inactive bucket will appear.
> We need to consider listening to bucket created.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20213) Partition commit is delayed when records keep coming

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-20213:


 Summary: Partition commit is delayed when records keep coming
 Key: FLINK-20213
 URL: https://issues.apache.org/jira/browse/FLINK-20213
 Project: Flink
  Issue Type: Bug
  Components: Connectors / FileSystem
Affects Versions: 1.11.2
Reporter: Jingsong Lee
Assignee: Jingsong Lee
 Fix For: 1.12.0


When set partition-commit.delay=0, Users expect partitions to be committed 
immediately.

However, if the record of this partition continues to flow in, the bucket for 
the partition will be activated, and no inactive bucket will appear.

We need to consider listening to bucket created.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #13884: [FLINK-19921][runtime] Remove legacy restart strategy

[GitBox]

flinkbot edited a comment on pull request #13884:
URL: https://github.com/apache/flink/pull/13884#issuecomment-720451080


   
   ## CI report:
   
   * b8aa0f9c13f6df568244db5edfbfa56df51c0e33 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8825)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8814)
 
   * 76d8fc44bd6f73200fd57f76cc3ebd64f176eb63 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9754)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-20209) Add missing checkpoint configuration to Flink UI

[Yun Tang (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-20209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234435#comment-17234435
 ] 

Yun Tang commented on FLINK-20209:
--

The information of 'unalignedCheckpoints' has been added to web UI in 
https://issues.apache.org/jira/browse/FLINK-18694
However, {{prefer-checkpoint-for-recovery}} and 
{{tolerable-failed-checkpoints}} are still not. [~lipeidian], would you like to 
take this ticket? And I could assign this ticket to you if agreed.

> Add missing checkpoint configuration to Flink UI
> 
>
> Key: FLINK-20209
> URL: https://issues.apache.org/jira/browse/FLINK-20209
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Web Frontend
>Affects Versions: 1.11.2
>Reporter: Peidian Li
>Priority: Major
> Attachments: image-2020-11-18-16-54-31-638.png
>
>
> Some of the [checkpointing 
> configurations|https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#checkpointing]
>  are not shown in the Flink UI，Can we consider adding these configurations to 
> the checkpoints configuration tab to make it easier for users to view 
> checkpoint configurations.
> These configurations need to be added：
> {code:java}
> execution.checkpointing.prefer-checkpoint-for-recovery
> execution.checkpointing.tolerable-failed-checkpoints
> execution.checkpointing.unaligned
> {code}
>  
> !image-2020-11-18-16-54-31-638.png|width=915,height=311!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #14068: [FLINK-200137][python] Emit timestamps of current records to downstream in PythonFunctionOperator.

[GitBox]

flinkbot edited a comment on pull request #14068:
URL: https://github.com/apache/flink/pull/14068#issuecomment-726851708


   
   ## CI report:
   
   * ba91f7f17ba4306ae7e78909029228f114a8b66e Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9727)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9699)
 
   * deefe94f7e88be11ad13c0a54dc526cc01fd9783 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (FLINK-20214) Unnecessary warning log when starting a k8s session cluster

[Guowei Ma (Jira)]

Guowei Ma created FLINK-20214:
-

 Summary: Unnecessary warning log when starting a k8s session 
cluster
 Key: FLINK-20214
 URL: https://issues.apache.org/jira/browse/FLINK-20214
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Kubernetes
Affects Versions: 1.12.0
Reporter: Guowei Ma


2020-11-18 17:46:36,727 WARN 
org.apache.flink.kubernetes.kubeclient.decorators.HadoopConfMountDecorator [] - 
Found 0 files in directory null/etc/hadoop, skip to mount the Hadoop 
Configuration ConfigMap.
2020-11-18 17:46:36,727 WARN 
org.apache.flink.kubernetes.kubeclient.decorators.HadoopConfMountDecorator [] - 
Found 0 files in directory null/etc/hadoop, skip to create the Hadoop 
Configuration ConfigMap.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot commented on pull request #14114: [FLINK-20153] Add documentation for BATCH execution mode

[GitBox]

flinkbot commented on pull request #14114:
URL: https://github.com/apache/flink/pull/14114#issuecomment-729567248


   
   ## CI report:
   
   * 70975eba381f72d59bfefd5d5b344e79121e09db UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-20213) Partition commit is delayed when records keep coming

[Jark Wu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu updated FLINK-20213:

Component/s: Table SQL / Ecosystem

> Partition commit is delayed when records keep coming
> 
>
> Key: FLINK-20213
> URL: https://issues.apache.org/jira/browse/FLINK-20213
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / FileSystem, Table SQL / Ecosystem
>Affects Versions: 1.11.2
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
>Priority: Major
> Fix For: 1.12.0, 1.11.3
>
>
> When set partition-commit.delay=0, Users expect partitions to be committed 
> immediately.
> However, if the record of this partition continues to flow in, the bucket for 
> the partition will be activated, and no inactive bucket will appear.
> We need to consider listening to bucket created.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-16057) Performance regression in ContinuousFileReaderOperator

[Roman Khachatryan (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Khachatryan updated FLINK-16057:
--
Fix Version/s: 1.12.0

> Performance regression in ContinuousFileReaderOperator
> --
>
> Key: FLINK-16057
> URL: https://issues.apache.org/jira/browse/FLINK-16057
> Project: Flink
>  Issue Type: Bug
>  Components: API / DataStream, Runtime / Task
>Affects Versions: 1.11.0
>Reporter: Roman Khachatryan
>Assignee: Roman Khachatryan
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: 1.11.0, 1.12.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> After switching CFRO to a single-threaded execution model performance 
> regression was expected to be about 15-20% (benchmarked in November).
> But after merging to master it turned out to be about 50%.
>   
> One reason is that the chaining strategy isn't set by default in CFRO factory.
> Without that even reading and outputting all records of a split in a single 
> mail action doesn't reverse the regression (only about half).
> However,  with strategy set AND batching enabled fixes the regression 
> (starting from batch size 6).
> Though batching can't be used in practice because it can significantly delay 
> checkpointing.
>  
> Another approach would be to process one record and the repeat until 
> defaultMailboxActionAvailable OR haveNewMail.
> This reverses regression and even improves the performance by about 50% 
> compared to the old version.
>  
> The final solution could also be FLIP-27.
>  
> Other things tried (didn't help):
>  * CFRO rework without subsequent commits (removing checkpoint lock)
>  * different batch sizes, including the whole split, without chaining 
> strategy fixed - partial improvement only
>  * disabling close
>  * disabling checkpointing
>  * disabling output (serialization)
>  * using LinkedList instead of PriorityQueue
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-18202) Introduce Protobuf format

[Suhan Mao (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-18202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234445#comment-17234445
 ] 

Suhan Mao commented on FLINK-18202:
---

[~twalthr] Sure. It will be an honor to contribute this feature. I need to 
adjust the code to adapt to current DynamicTableFactory interface. I'll show 
you the code in these tow days.

 

> Introduce Protobuf format
> -
>
> Key: FLINK-18202
> URL: https://issues.apache.org/jira/browse/FLINK-18202
> Project: Flink
>  Issue Type: New Feature
>  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table 
> SQL / API
>Reporter: Benchao Li
>Priority: Major
> Attachments: image-2020-06-15-17-18-03-182.png
>
>
> PB[1] is a very famous and wildly used (de)serialization framework. The ML[2] 
> also has some discussions about this. It's a useful feature.
> This issue maybe needs some designs, or a FLIP.
> [1] [https://developers.google.com/protocol-buffers]
> [2] [http://apache-flink.147419.n8.nabble.com/Flink-SQL-UDF-td3725.html]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode

[Kostas Kloudas (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostas Kloudas closed FLINK-20143.
--
Fix Version/s: 1.11.3
   1.12.0
   Resolution: Fixed

master: 424d41d124871b0a82d514f1ce15bc87b52169c6
1.11: f6f6271e8c17f7873ccb4cfe649d0ad35dfde445

> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0, 1.11.2
>Reporter: zhisheng
>Assignee: Yang Wang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.12.0, 1.11.3
>
> Attachments: image-2020-11-13-17-21-47-751.png, 
> image-2020-11-13-17-22-06-111.png, image-2020-11-13-18-43-55-188.png
>
>
> use follow command deploy flink job to yarn failed 
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD 
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD 
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib    
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster 
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD 
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib    
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains 
> multiple SLF4J bindings.SLF4J: Found binding in 
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
>  Found binding in 
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
>  Found binding in 
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
>  See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.SLF4J: Actual binding is of type 
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO  
> org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic 
> Property set: 
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13 
> 16:14:30,347 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli              
>   [] - Dynamic Property set: 
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with 
> built-in data generator: StateMachineExample [--error-rate 
> ] [--sleep ]Usage 
> with Kafka: StateMachineExample --kafka-topic  [--brokers 
> ]Options for both the above setups: [--backend ] 
> [--checkpoint-dir ] [--async-checkpoints ] 
> [--incremental-checkpoints ] [--output  OR null for 
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN  
> org.apache.flink.yarn.configuration.YarnLogConfigUtil        [] - The 
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already 
> contains a LOG4J config file.If you want to use logback, then please delete 
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO  
> org.apache.hadoop.yarn.client.AHSProxy                       [] - Connecting 
> to Application History server at 
> FAT-hadoopuat-69117.vm.dc01.tech/10.69.1.17:102002020-11-13 16:14:30,958 INFO 
>  org.apache.flink.yarn.YarnClusterDescriptor                  [] - No path 
> for the flink jar passed. Using the location of class 
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13 
> 16:14:31,065 INFO  
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing 
> over to rm22020-11-13 16:14:31,130 INFO  
> org.apache.flink.yarn.YarnClusterDescriptor                  [] - The 
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make 
> up an integer multiple of its minimum allocation memory (2048 MB, configured 
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be 
> used by Flink.2020-11-13 16:14:31,130 INFO  
> org.apache.flink.yarn.YarnClusterDescriptor                  [] - The 
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make 
> up an integer multiple of its minimum allocation memory (2048 MB, configured 
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be 
> used by Flink.2020-11-13 16:14:31,130 INFO  
> org.apache.flink.yarn.YarnClusterDescriptor                  [] - Cluster 
> specification: ClusterSpecificat

[GitHub] [flink] flinkbot edited a comment on pull request #13641: [FLINK-17760][tests] Rework tests to not rely on legacy scheduling codes in ExecutionGraph components

[GitBox]

flinkbot edited a comment on pull request #13641:
URL: https://github.com/apache/flink/pull/13641#issuecomment-708569491


   
   ## CI report:
   
   * 09d8deb89416f53dfe8b5c16fb9d723cbd98612c UNKNOWN
   * 517b53a68f3e9c8c0897cd7afba90b8a9befaa4f Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8807)
 
   * 021cac170ea26cddfd8af0a2bec5fea4e6a76b69 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #13882: [FLINK-19919][runtime] Remove legacy scheduling in ExecutionGraph components

[GitBox]

flinkbot edited a comment on pull request #13882:
URL: https://github.com/apache/flink/pull/13882#issuecomment-720449193


   
   ## CI report:
   
   * cdeecd5d3e5fca4d232d8c9f8c4f9fc04307b72f Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8812)
 
   * b15be6a32e44adf7989dbf763821bb2ea7d7ad9a UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #13883: [FLINK-19920][runtime] Remove legacy failover strategy

[GitBox]

flinkbot edited a comment on pull request #13883:
URL: https://github.com/apache/flink/pull/13883#issuecomment-720449982


   
   ## CI report:
   
   * 598917cee8c1482777d6c3f513d9a09d5acadacf Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8813)
 
   * 7a31bb52f16f4da8af82b71aa903f00b70aaf590 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14068: [FLINK-200137][python] Emit timestamps of current records to downstream in PythonFunctionOperator.

[GitBox]

flinkbot edited a comment on pull request #14068:
URL: https://github.com/apache/flink/pull/14068#issuecomment-726851708


   
   ## CI report:
   
   * ba91f7f17ba4306ae7e78909029228f114a8b66e Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9727)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9699)
 
   * deefe94f7e88be11ad13c0a54dc526cc01fd9783 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9757)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14114: [FLINK-20153] Add documentation for BATCH execution mode

[GitBox]

flinkbot edited a comment on pull request #14114:
URL: https://github.com/apache/flink/pull/14114#issuecomment-729567248


   
   ## CI report:
   
   * 70975eba381f72d59bfefd5d5b344e79121e09db Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9758)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-20170) json deserialize decimal loses precision

[CaoZhen (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-20170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234455#comment-17234455
 ] 

CaoZhen commented on FLINK-20170:
-

hi all, I've had this problem too.

I think the problem comes from when "com.fasterxml.Jackson" deserializes JSON. 
[1]  

As mulan said, we can fix it by code in JsonRowDeserializationSchema and 
JsonRowDataDeserializationSchema:
objectMapper.enable(DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS);
So whether to consider opening the Jackson configuration to users？

[1] 
https://github.com/FasterXML/jackson-databind/wiki/Deserialization-Features#type-conversions

 

> json deserialize decimal loses precision
> 
>
> Key: FLINK-20170
> URL: https://issues.apache.org/jira/browse/FLINK-20170
> Project: Flink
>  Issue Type: Improvement
>  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table 
> SQL / API
>Reporter: Mulan
>Priority: Major
> Attachments: image-2020-11-18-16-51-35-317.png
>
>
> {code:java}
> CREATE TABLE ods (
> id BIGINT,
> factor DECIMAL(38, 18)
> ) WITH (
> 'connector.type' = 'kafka',
> 'connector.version' = 'universal',
> 'connector.topic' = '_foo',
> 'connector.topic?' = '_foo',
> 'connector.properties.bootstrap.servers' = 'localhost:9092',
> 'connector.properties.group.id' = 'g',
> 'format.type' = 'json',
> 'update-mode' = 'append'
> );
> {code}
> this following is input data.
> {code:json}
> {"id": 1, "factor": 799.929496989092949698}
> {code}
> this following is output data and loses precision.
> {code:json}
> 1, 799.92949698909300
> {code}
> This following code call readTree() method. This method make value loses 
> precision.
> {code:java}
>   public Row deserialize(byte[] message) throws IOException {
>   try {
>   final JsonNode root = objectMapper.readTree(message);
>   return (Row) runtimeConverter.convert(objectMapper, 
> root);
>   } catch (Throwable t) {
>   throw new IOException("Failed to deserialize JSON 
> object.", t);
>   }
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (FLINK-20170) json deserialize decimal loses precision

[CaoZhen (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-20170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234455#comment-17234455
 ] 

CaoZhen edited comment on FLINK-20170 at 11/18/20, 10:16 AM:
-

hi all, I've had this problem too.

I think the problem comes from when "com.fasterxml.Jackson" deserializes JSON. 
[1]  

As mulan said, we can fix it by code in JsonRowDeserializationSchema and 
JsonRowDataDeserializationSchema:
{code:java}
objectMapper.enable(DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS);
{code}
So whether to consider opening the Jackson configuration to users？  

[1] 
[https://github.com/FasterXML/jackson-databind/wiki/Deserialization-Features#type-conversions]

 


was (Author: caozhen1937):
hi all, I've had this problem too.

I think the problem comes from when "com.fasterxml.Jackson" deserializes JSON. 
[1]  

As mulan said, we can fix it by code in JsonRowDeserializationSchema and 
JsonRowDataDeserializationSchema:
objectMapper.enable(DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS);
So whether to consider opening the Jackson configuration to users？

[1] 
https://github.com/FasterXML/jackson-databind/wiki/Deserialization-Features#type-conversions

 

> json deserialize decimal loses precision
> 
>
> Key: FLINK-20170
> URL: https://issues.apache.org/jira/browse/FLINK-20170
> Project: Flink
>  Issue Type: Improvement
>  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table 
> SQL / API
>Reporter: Mulan
>Priority: Major
> Attachments: image-2020-11-18-16-51-35-317.png
>
>
> {code:java}
> CREATE TABLE ods (
> id BIGINT,
> factor DECIMAL(38, 18)
> ) WITH (
> 'connector.type' = 'kafka',
> 'connector.version' = 'universal',
> 'connector.topic' = '_foo',
> 'connector.topic?' = '_foo',
> 'connector.properties.bootstrap.servers' = 'localhost:9092',
> 'connector.properties.group.id' = 'g',
> 'format.type' = 'json',
> 'update-mode' = 'append'
> );
> {code}
> this following is input data.
> {code:json}
> {"id": 1, "factor": 799.929496989092949698}
> {code}
> this following is output data and loses precision.
> {code:json}
> 1, 799.92949698909300
> {code}
> This following code call readTree() method. This method make value loses 
> precision.
> {code:java}
>   public Row deserialize(byte[] message) throws IOException {
>   try {
>   final JsonNode root = objectMapper.readTree(message);
>   return (Row) runtimeConverter.convert(objectMapper, 
> root);
>   } catch (Throwable t) {
>   throw new IOException("Failed to deserialize JSON 
> object.", t);
>   }
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink-docker] Myasuka commented on a change in pull request #43: [FLINK-19125] Adopt Jemalloc as default memory allocator to avoid memory leak

[GitBox]

Myasuka commented on a change in pull request #43:
URL: https://github.com/apache/flink-docker/pull/43#discussion_r525968923



##
File path: docker-entrypoint.sh
##
@@ -93,21 +94,41 @@ prepare_job_manager_start() {
 envsubst < "${CONF_FILE}" > "${CONF_FILE}.tmp" && mv "${CONF_FILE}.tmp" 
"${CONF_FILE}"
 }
 
+disable_jemalloc_env() {
+  local -n _args=$1
+  if [ "${_args[0]}" = ${COMMAND_DISABLE_JEMALLOC} ]; then
+  echo "Disable Jemalloc as the memory allocator"
+  _args=("${_args[@]:1}")

Review comment:
   `local -n` is the same as `declare -n` which serves as a name 
[reference](https://www.gnu.org/software/bash/manual/bash.html#index-declare) 
and I got this from [this answer](https://unix.stackexchange.com/a/449537).
   This feature is introduced in 
[bash-4.3](https://github.com/bminor/bash/blob/ac50fbac377e32b98d2de396f016ea81e8ee9961/CHANGES#L761)
 and luckily our base image installed `bash-5.0.3` to include this.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #14115: [hotfix][docs] fix table systemFunction lag description.

[GitBox]

flinkbot commented on pull request #14115:
URL: https://github.com/apache/flink/pull/14115#issuecomment-729579954


   
   ## CI report:
   
   * 46a1adc285805e0b6254598dbe7f3f629211166e UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] dawidwys commented on a change in pull request #14114: [FLINK-20153] Add documentation for BATCH execution mode

[GitBox]

dawidwys commented on a change in pull request #14114:
URL: https://github.com/apache/flink/pull/14114#discussion_r525965548



##
File path: docs/dev/datastream_execution_mode.md
##
@@ -0,0 +1,242 @@
+---
+title: "Execution Mode (Batch/Streaming)"
+nav-id: datastream_execution_mode
+nav-parent_id: streaming
+nav-pos: 1
+---
+
+
+The DataStream API supports different runtime execution modes from which you
+can choose depending on the requirements of your use case and the
+characteristics of your job.
+
+There is the "classic" execution behavior of the DataStream API which we call
+`STREAMING` execution mode. This should be used for unbounded jobs that require
+continuous incremental processing and are expected to stay online indefinitely.
+
+Additionally, there is a batch-style execution mode that we call `BATCH`
+execution mode. This executes jobs in a way that is more reminiscent of batch
+processing frameworks such as MapReduce. This should be used for bounded jobs
+for which you have a known fixed input and which do not run continuously.
+
+We have these different execution modes because `BATCH` execution allows some
+additional optimizations that we can only do when we now that our input is
+bounded. For example, different join/aggregation strategies can be used, in
+additional to a different shuffle implementation that allows more efficient
+failure recovery behavior. We will go into some of the details of the execution
+behavior below.
+
+* This will be replaced by the TOC
+{:toc}
+
+## When can/should I use BATCH execution mode?
+
+The BATCH execution mode can only be used for Jobs/Flink Programs that are
+_bounded_. Boundedness is a property of a data source that tells us whether all
+the input coming from that source is known before execution or whether new data
+will show up, potentially for forever. A job, in turn, is bounded if all its
+sources are bounded and unbounded otherwise.
+
+STREAMING execution mode, on the other hand, can be used for both bounded and
+unbounded jobs.
+
+Another term that is typically used for bounded sources is _batch source_. Or,
+we can say that we are working with a batch data set or a batch of data. Some
+typical other terms for an unbounded source are _continuous source_, _streaming
+source_, _stream_, or _infinite stream_.
+
+As a rule of thumb, you should be using BATCH execution mode when your program
+is bounded because this will be more efficient. You have to use STREAMING
+execution mode when your program is unbounded because only this mode is general
+enough to be able to deal with continuous data streams.
+
+TODO: Should we even go into this?
+
+One obvious outlier case is when you want to use a bounded job to bootstrap
+some job state that you then want to use in an unbounded job. For example, by
+running a bounded job using STREAMING mode, taking a savepoint and then
+restoring that savepoint on an unbounded job. This is a very specific use case
+and one that might soon become obsolete when we allow producing a savepoint as
+additional output of a BATCH execution job.
+
+## Configuring BATCH execution mode
+
+The execution mode can be configured via the `execution.runtime-mode` setting.
+There are three possible values:
+
+ - `STREAMING`: The classic DataStream execution mode
+ - `BATCH`: Batch-style execution on the DataStream API
+ - `AUTOMATIC`: Let the system decide based on the boundedness of the sources
+
+ This can be configured either in the `flink-conf.yaml`, via command line
+ parameters of `bin/flink run ...`, or programmatically when
+ creating/configuring the `StreamExecutionEnvironment`.
+
+ Here's how you can configure the execution mode via the command line:
+
+ ```bash
+ $ bin/flink run -Dexecution.runtime-mode=BATCH 
examples/streaming/WordCount.jar
+ ```
+
+ This example shows how you can configure the execution mode in code:
+
+ ```java
+Configuration config = new Configuration();
+config.set(ExecutionOptions.RUNTIME_MODE, RuntimeExecutionMode.BATCH);
+StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment(config);
+ ```
+
+## Execution Behavior
+
+This section provides an overview of the execution behavior of BATCH execution
+mode and contrasts it with STREAMING execution mode. For more details, please
+refer to the FLIPs that introduced this feature:
+[FLIP-134](https://cwiki.apache.org/confluence/x/4i94CQ) and
+[FLIP-140](https://cwiki.apache.org/confluence/x/kDh4CQ). As well as the
+documentation about [task scheduling (TODO)]().
+
+### Task Scheduling And Network Shuffle
+
+Flink jobs consist of different operations that are connected together in a
+dataflow graph. The system decides how to schedule the execution of these
+operations on different processes/machines (TaskManager) and how data is
+shuffled (sent) betwixt them.

Review comment:
   I appreciate learning a new word `betwixt`, but I think it is nicer 
towards our readers, which are very often not native speakers, to

[GitHub] [flink] wuchong commented on pull request #14110: [FLINK-20200][table-planner] SQL Hints are not supported in CREATE VI…

[GitBox]

wuchong commented on pull request #14110:
URL: https://github.com/apache/flink/pull/14110#issuecomment-729583905


   @danny0405 could you help to open a pull request for release-1.11 branch? I 
think this problem also exists in 1.11. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] wuchong commented on pull request #14110: [FLINK-20200][table-planner] SQL Hints are not supported in CREATE VI…

[GitBox]

wuchong commented on pull request #14110:
URL: https://github.com/apache/flink/pull/14110#issuecomment-729584307


   Build is passed: 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=9732&view=results



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-20203) Could not find any document about how to build a Flink image from local build.

[Guowei Ma (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guowei Ma updated FLINK-20203:
--
Component/s: Deployment / Kubernetes

> Could not find any document about how to build a Flink image from local build.
> --
>
> Key: FLINK-20203
> URL: https://issues.apache.org/jira/browse/FLINK-20203
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / Kubernetes
>Affects Versions: 1.12.0
>Reporter: Guowei Ma
>Priority: Critical
>
> If user wants to use or try some feature that does not include in the 
> "official" Flink image the user might need to build a docker image based on 
> his local build. But there is such 
> document([https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/docker.html)]
> So I would like to propose that we might need to introduce some documentation 
> about how to build the image from local build.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)