[PR] update flink architecture url for text [Flinks's Architecture] [flink-web]
shashikumarec088 opened a new pull request, #724: URL: https://github.com/apache/flink-web/pull/724 updated the url which was not found before, i feel even this fix is not permanent as version 1.18 might change in the future -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31663) Add ARRAY_EXCEPT supported in SQL & Table API
[
https://issues.apache.org/jira/browse/FLINK-31663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825176#comment-17825176
]
Sergey Nuyanzin commented on FLINK-31663:
-
{quote}
Since the behavior of array_union is already aligned with Spark's(without
duplicates ), if we don't align here, the logic of the entire function would
seem inconsistent. should we change the behavior if array_union. but if we
change it ,it will cause version compatibility problem
{quote}
I don't think we should copy everything that present in Spark.
there is Snowflake, ClickHouse, PostgreSQL, DuckDB and etc.
{{ARRAY_EXCEPT}} keeps duplicates (as in Snowflake) and it allows to cover some
cases not covered by the version eliminating duplicates. In cae there is a need
to eliminate duplicates there is {{ARRAY_DISTINCT}}.
And Flink follows this way
Yep there is {{ARRAY_UNION}} which eliminates duplicates
However there is also {{ARRAY_CONCAT}} which concatenates arrays without
duplicates elimination, moreover it can concatenate more than 2 arrays at once
(like in BigQuery, ClickHouse, DuckDB)
> Add ARRAY_EXCEPT supported in SQL & Table API
> -
>
> Key: FLINK-31663
> URL: https://issues.apache.org/jira/browse/FLINK-31663
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: luoyuxia
>Assignee: Hanyu Zheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31663) Add ARRAY_EXCEPT supported in SQL & Table API
[
https://issues.apache.org/jira/browse/FLINK-31663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825176#comment-17825176
]
Sergey Nuyanzin edited comment on FLINK-31663 at 3/11/24 7:25 AM:
--
{quote}
Since the behavior of array_union is already aligned with Spark's(without
duplicates ), if we don't align here, the logic of the entire function would
seem inconsistent. should we change the behavior if array_union. but if we
change it ,it will cause version compatibility problem
{quote}
I don't think we should copy everything that is present in Spark.
there is Snowflake, ClickHouse, PostgreSQL, DuckDB and etc.
{{ARRAY_EXCEPT}} keeps duplicates (as in Snowflake) and it allows to cover some
cases not covered by the version eliminating duplicates. In case there is a
need to eliminate duplicates there is {{ARRAY_DISTINCT}}.
And Flink follows this way
Yep there is {{ARRAY_UNION}} which eliminates duplicates
However there is also {{ARRAY_CONCAT}} which concatenates arrays without
duplicates elimination, moreover it can concatenate more than 2 arrays at once
(like in BigQuery, ClickHouse, DuckDB)
was (Author: sergey nuyanzin):
{quote}
Since the behavior of array_union is already aligned with Spark's(without
duplicates ), if we don't align here, the logic of the entire function would
seem inconsistent. should we change the behavior if array_union. but if we
change it ,it will cause version compatibility problem
{quote}
I don't think we should copy everything that present in Spark.
there is Snowflake, ClickHouse, PostgreSQL, DuckDB and etc.
{{ARRAY_EXCEPT}} keeps duplicates (as in Snowflake) and it allows to cover some
cases not covered by the version eliminating duplicates. In cae there is a need
to eliminate duplicates there is {{ARRAY_DISTINCT}}.
And Flink follows this way
Yep there is {{ARRAY_UNION}} which eliminates duplicates
However there is also {{ARRAY_CONCAT}} which concatenates arrays without
duplicates elimination, moreover it can concatenate more than 2 arrays at once
(like in BigQuery, ClickHouse, DuckDB)
> Add ARRAY_EXCEPT supported in SQL & Table API
> -
>
> Key: FLINK-31663
> URL: https://issues.apache.org/jira/browse/FLINK-31663
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: luoyuxia
>Assignee: Hanyu Zheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32483) RescaleCheckpointManuallyITCase.testCheckpointRescalingOutKeyedState fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-32483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-32483:
--
Affects Version/s: 1.20.0
> RescaleCheckpointManuallyITCase.testCheckpointRescalingOutKeyedState fails on
> AZP
> -
>
> Key: FLINK-32483
> URL: https://issues.apache.org/jira/browse/FLINK-32483
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Tests
>Affects Versions: 1.17.2, 1.20.0
>Reporter: Sergey Nuyanzin
>Priority: Major
> Labels: auto-deprioritized-critical, test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50397&view=logs&j=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3&t=0c010d0c-3dec-5bf1-d408-7b18988b1b2b&l=7495
> fails with
> {noformat}
> Jun 26 06:08:57 [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 21.041 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase
> Jun 26 06:08:57 [ERROR]
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingOutKeyedState
> Time elapsed: 6.435 s <<< FAILURE!
> Jun 26 06:08:57 java.lang.AssertionError: expected:<[(0,24000), (2,58500),
> (0,34500), (0,45000), (3,43500), (2,18000), (1,6000), (1,16500), (0,28500),
> (0,52500), (3,27000), (1,51000), (2,25500), (0,1500), (0,49500), (3,0),
> (3,48000), (0,36000), (2,22500), (1,10500), (0,46500), (2,33000), (1,21000),
> (0,9000), (0,57000), (3,31500), (2,19500), (1,7500), (1,55500), (3,42000),
> (2,3), (0,54000), (2,40500), (1,4500), (3,15000), (2,3000), (1,39000),
> (2,13500), (0,37500), (0,61500), (3,12000), (3,6)]> but was:<[(2,58500),
> (0,34500), (0,45000), (3,43500), (2,18000), (1,16500), (0,52500), (3,27000),
> (2,25500), (0,49500), (3,0), (3,48000), (0,36000), (2,22500), (1,21000),
> (0,9000), (0,57000), (3,31500), (1,7500), (2,3), (0,54000), (2,40500),
> (1,4500), (2,3000), (1,39000), (2,13500), (0,61500), (3,12000)]>
> Jun 26 06:08:57 at org.junit.Assert.fail(Assert.java:89)
> Jun 26 06:08:57 at org.junit.Assert.failNotEquals(Assert.java:835)
> Jun 26 06:08:57 at org.junit.Assert.assertEquals(Assert.java:120)
> Jun 26 06:08:57 at org.junit.Assert.assertEquals(Assert.java:146)
> Jun 26 06:08:57 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.restoreAndAssert(RescaleCheckpointManuallyITCase.java:219)
> Jun 26 06:08:57 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingKeyedState(RescaleCheckpointManuallyITCase.java:138)
> Jun 26 06:08:57 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingOutKeyedState(RescaleCheckpointManuallyITCase.java:116)
> Jun 26 06:08:57 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34571][test] Fix flaky test `SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError` [flink]
reswqa merged PR #24469: URL: https://github.com/apache/flink/pull/24469 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34571][test] Fix flaky test `SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError` [flink]
reswqa commented on PR #24469: URL: https://github.com/apache/flink/pull/24469#issuecomment-1987838887 Thanks for the quick review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34227) Job doesn't disconnect from ResourceManager
[
https://issues.apache.org/jira/browse/FLINK-34227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825197#comment-17825197
]
Matthias Pohl commented on FLINK-34227:
---
Same issue was observed for {{GroupWindowTableAggregateITCase}}:
https://github.com/apache/flink/actions/runs/8181761395/job/22372319416#step:10:12561
> Job doesn't disconnect from ResourceManager
> ---
>
> Key: FLINK-34227
> URL: https://issues.apache.org/jira/browse/FLINK-34227
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.18.1
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Critical
> Labels: github-actions, test-stability
> Attachments: FLINK-34227.7e7d69daebb438b8d03b7392c9c55115.log,
> FLINK-34227.log
>
>
> https://github.com/XComp/flink/actions/runs/7634987973/job/20800205972#step:10:14557
> {code}
> [...]
> "main" #1 prio=5 os_prio=0 tid=0x7f4b7000 nid=0x24ec0 waiting on
> condition [0x7fccce1eb000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xbdd52618> (a
> java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2131)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2099)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2077)
> at
> org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:876)
> at
> org.apache.flink.table.planner.runtime.stream.sql.WindowDistinctAggregateITCase.testHopWindow_Cube(WindowDistinctAggregateITCase.scala:550)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34571) SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError failed due an assertion
[
https://issues.apache.org/jira/browse/FLINK-34571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825198#comment-17825198
]
Weijie Guo commented on FLINK-34571:
master(1.20) via fcf61b9108942a892aa93765178a81f2c0e787c5.
> SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError failed
> due an assertion
> --
>
> Key: FLINK-34571
> URL: https://issues.apache.org/jira/browse/FLINK-34571
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.19.0, 1.20.0
>Reporter: Matthias Pohl
>Assignee: Weijie Guo
>Priority: Critical
> Labels: github-actions, pull-request-available, test-stability
>
> https://github.com/apache/flink/actions/runs/8134965216/job/8875618#step:10:8586
> {code}
> Error: 02:39:36 02:39:36.688 [ERROR] Tests run: 9, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 13.68 s <<< FAILURE! -- in
> org.apache.flink.runtime.io.network.partition.SortMergeResultPartitionReadSchedulerTest
> Error: 02:39:36 02:39:36.689 [ERROR]
> org.apache.flink.runtime.io.network.partition.SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError
> -- Time elapsed: 0.174 s <<< FAILURE!
> Mar 04 02:39:36 org.opentest4j.AssertionFailedError:
> Mar 04 02:39:36
> Mar 04 02:39:36 Expecting value to be true but was false
> Mar 04 02:39:36 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> Mar 04 02:39:36 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Mar 04 02:39:36 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Mar 04 02:39:36 at
> org.apache.flink.runtime.io.network.partition.SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError(SortMergeResultPartitionReadSchedulerTest.java:225)
> Mar 04 02:39:36 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 04 02:39:36 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [BP-1.19][FLINK-34571][test] Fix flaky test SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError [flink]
reswqa opened a new pull request, #24479: URL: https://github.com/apache/flink/pull/24479 Backport FLINK-34571 to release-1.19 branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34571) SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError failed due an assertion
[
https://issues.apache.org/jira/browse/FLINK-34571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo updated FLINK-34571:
---
Fix Version/s: 1.20.0
> SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError failed
> due an assertion
> --
>
> Key: FLINK-34571
> URL: https://issues.apache.org/jira/browse/FLINK-34571
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.19.0, 1.20.0
>Reporter: Matthias Pohl
>Assignee: Weijie Guo
>Priority: Critical
> Labels: github-actions, pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> https://github.com/apache/flink/actions/runs/8134965216/job/8875618#step:10:8586
> {code}
> Error: 02:39:36 02:39:36.688 [ERROR] Tests run: 9, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 13.68 s <<< FAILURE! -- in
> org.apache.flink.runtime.io.network.partition.SortMergeResultPartitionReadSchedulerTest
> Error: 02:39:36 02:39:36.689 [ERROR]
> org.apache.flink.runtime.io.network.partition.SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError
> -- Time elapsed: 0.174 s <<< FAILURE!
> Mar 04 02:39:36 org.opentest4j.AssertionFailedError:
> Mar 04 02:39:36
> Mar 04 02:39:36 Expecting value to be true but was false
> Mar 04 02:39:36 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> Mar 04 02:39:36 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Mar 04 02:39:36 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Mar 04 02:39:36 at
> org.apache.flink.runtime.io.network.partition.SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError(SortMergeResultPartitionReadSchedulerTest.java:225)
> Mar 04 02:39:36 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 04 02:39:36 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34571) SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError failed due an assertion
[
https://issues.apache.org/jira/browse/FLINK-34571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825198#comment-17825198
]
Weijie Guo edited comment on FLINK-34571 at 3/11/24 8:21 AM:
-
master(1.20) via fcf61b9108942a892aa93765178a81f2c0e787c5.
Will backport this to release-1.19 after it released.
was (Author: weijie guo):
master(1.20) via fcf61b9108942a892aa93765178a81f2c0e787c5.
> SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError failed
> due an assertion
> --
>
> Key: FLINK-34571
> URL: https://issues.apache.org/jira/browse/FLINK-34571
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.19.0, 1.20.0
>Reporter: Matthias Pohl
>Assignee: Weijie Guo
>Priority: Critical
> Labels: github-actions, pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> https://github.com/apache/flink/actions/runs/8134965216/job/8875618#step:10:8586
> {code}
> Error: 02:39:36 02:39:36.688 [ERROR] Tests run: 9, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 13.68 s <<< FAILURE! -- in
> org.apache.flink.runtime.io.network.partition.SortMergeResultPartitionReadSchedulerTest
> Error: 02:39:36 02:39:36.689 [ERROR]
> org.apache.flink.runtime.io.network.partition.SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError
> -- Time elapsed: 0.174 s <<< FAILURE!
> Mar 04 02:39:36 org.opentest4j.AssertionFailedError:
> Mar 04 02:39:36
> Mar 04 02:39:36 Expecting value to be true but was false
> Mar 04 02:39:36 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> Mar 04 02:39:36 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Mar 04 02:39:36 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Mar 04 02:39:36 at
> org.apache.flink.runtime.io.network.partition.SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError(SortMergeResultPartitionReadSchedulerTest.java:225)
> Mar 04 02:39:36 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 04 02:39:36 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31472) AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
[
https://issues.apache.org/jira/browse/FLINK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825202#comment-17825202
]
Ahmed Hamdy commented on FLINK-31472:
-
[~mapohl] Sorry I was out of office, Since I am back now I am happy to take a
look and not disable the test. I will post updates on the ticket.
> AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
>
>
> Key: FLINK-31472
> URL: https://issues.apache.org/jira/browse/FLINK-31472
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.17.0, 1.16.1, 1.18.0, 1.19.0, 1.20.0
>Reporter: Ran Tao
>Assignee: Ahmed Hamdy
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0
>
>
> when run mvn clean test, this case failed occasionally.
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.955
> s <<< FAILURE! - in
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest
> [ERROR]
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
> Time elapsed: 0.492 s <<< ERROR!
> java.lang.IllegalStateException: Illegal thread detected. This method must be
> called from inside the mailbox thread!
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
> at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
> at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
> at
>
Re: [PR] [BP-1.19][FLINK-34571][test] Fix flaky test SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError [flink]
flinkbot commented on PR #24479: URL: https://github.com/apache/flink/pull/24479#issuecomment-1987859634 ## CI report: * 7207cdb3f47f5470ce49c343b4b7c302a75ef1e7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Reopened] (FLINK-34629) Pulsar source lost topic subscribe
[ https://issues.apache.org/jira/browse/FLINK-34629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Pohl reopened FLINK-34629: --- Assignee: WangMinChao I re-opened that one in favor of FLINK-34627 because it appears to be the one that is used to fix the issue. [~wangminchao] Thanks for reporting issues. But please be mindful when doing this. It appears that you created the same issue 5 times > Pulsar source lost topic subscribe > -- > > Key: FLINK-34629 > URL: https://issues.apache.org/jira/browse/FLINK-34629 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: pulsar-3.0.1 >Reporter: WangMinChao >Assignee: WangMinChao >Priority: Major > Labels: pull-request-available > > The non-partition pulsar topic partition id is `-1`, using multiples of the > non-partition topics > in Pulsar source maybe lose topic subscribe. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-34627) Pulsar source lost topic subscribe
[ https://issues.apache.org/jira/browse/FLINK-34627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Pohl closed FLINK-34627. - Resolution: Duplicate I'm closing this one as well in favor of the reopened duplicate FLINK-34629. It looks like the contributor decided to create the contributions under FLINK-34629. > Pulsar source lost topic subscribe > -- > > Key: FLINK-34627 > URL: https://issues.apache.org/jira/browse/FLINK-34627 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: pulsar-3.0.1 >Reporter: WangMinChao >Priority: Major > > The non-partition pulsar topic partition id is `-1`, using multiples of the > non-partition topics > in Pulsar source maybe lose topic subscribe. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-31472] Disable Intermittently failing throttling test [flink]
XComp closed pull request #24175: [FLINK-31472] Disable Intermittently failing throttling test URL: https://github.com/apache/flink/pull/24175 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-31472] Disable Intermittently failing throttling test [flink]
XComp commented on PR #24175: URL: https://github.com/apache/flink/pull/24175#issuecomment-1987866921 According to [this comment in FLINK-31472](https://issues.apache.org/jira/browse/FLINK-31472?focusedCommentId=17825202&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17825202) we're gonna go ahead trying to fix the issue without disabling the test for now. I'm gonna close this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34152][autoscaler] Refactor the MemoryTuning to avoid the huge method [flink-kubernetes-operator]
1996fanrui commented on code in PR #795:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/795#discussion_r1519353312
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/tuning/TuningSimpleMemorySpec.java:
##
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.autoscaler.tuning;
+
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.util.config.memory.CommonProcessMemorySpec;
+import
org.apache.flink.runtime.util.config.memory.taskmanager.TaskExecutorFlinkMemory;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+
+/** The task manager memory spec for memory tuning. */
+@AllArgsConstructor
+@Getter
+public class TuningSimpleMemorySpec {
Review Comment:
I'm not sure whether the current class name is good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-34152) Tune TaskManager memory
[ https://issues.apache.org/jira/browse/FLINK-34152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825218#comment-17825218 ] Rui Fan commented on FLINK-34152: - Merged to main(1.8.0) via: * 48692b7243086a2ccf2cb4c64a6b00306fa1d65f * 3ead906f33a4b3790fa5c12f2018e09db9443a09 * d526174d9ab4c3ccf92548c821d9f44acbd3f247 > Tune TaskManager memory > --- > > Key: FLINK-34152 > URL: https://issues.apache.org/jira/browse/FLINK-34152 > Project: Flink > Issue Type: Sub-task > Components: Autoscaler, Kubernetes Operator >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.8.0 > > > The current autoscaling algorithm adjusts the parallelism of the job task > vertices according to the processing needs. By adjusting the parallelism, we > systematically scale the amount of CPU for a task. At the same time, we also > indirectly change the amount of memory tasks have at their dispense. However, > there are some problems with this. > # Memory is overprovisioned: On scale up we may add more memory than we > actually need. Even on scale down, the memory / cpu ratio can still be off > and too much memory is used. > # Memory is underprovisioned: For stateful jobs, we risk running into > OutOfMemoryErrors on scale down. Even before running out of memory, too > little memory can have a negative impact on the effectiveness of the scaling. > We lack the capability to tune memory proportionally to the processing needs. > In the same way that we measure CPU usage and size the tasks accordingly, we > need to evaluate memory usage and adjust the heap memory size. > https://docs.google.com/document/d/19GXHGL_FvN6WBgFvLeXpDABog2H_qqkw1_wrpamkFSc/edit > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34639) Flink CDC: Support DebeziumDeserializationSchema in OceanBase source connector
He Wang created FLINK-34639: --- Summary: Flink CDC: Support DebeziumDeserializationSchema in OceanBase source connector Key: FLINK-34639 URL: https://issues.apache.org/jira/browse/FLINK-34639 Project: Flink Issue Type: Improvement Components: Flink CDC Reporter: He Wang Support DebeziumDeserializationSchema in OceanBase source connector. Related issue: [https://github.com/apache/flink-cdc/issues/2450] -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34632][runtime/checkpointing] Log checkpoint Id when logging delay [flink]
masteryhx closed pull request #24474: [FLINK-34632][runtime/checkpointing] Log checkpoint Id when logging delay URL: https://github.com/apache/flink/pull/24474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-34632) Log checkpoint Id when logging checkpoint processing delay
[ https://issues.apache.org/jira/browse/FLINK-34632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hangxiang Yu closed FLINK-34632. Fix Version/s: 1.20.0 Assignee: Mingliang Liu Resolution: Fixed merged 22309fdc into master > Log checkpoint Id when logging checkpoint processing delay > -- > > Key: FLINK-34632 > URL: https://issues.apache.org/jira/browse/FLINK-34632 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.18.1 >Reporter: Mingliang Liu >Assignee: Mingliang Liu >Priority: Minor > Labels: pull-request-available > Fix For: 1.20.0 > > > Currently we log a warning message when the checkpoint barrier takes too long > to start processing. It has the delay and would be easier for debugging > respective checkpoint if the id is also logged. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [hotfix] A couple of hotfixes related to e2e tests [flink]
pnowojski merged PR #24472: URL: https://github.com/apache/flink/pull/24472 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add announcement blog post for Flink 1.19 [flink-web]
xintongsong commented on code in PR #721:
URL: https://github.com/apache/flink-web/pull/721#discussion_r1519286951
##
docs/content/posts/2024-03-xx-release-1.19.0.md:
##
@@ -0,0 +1,456 @@
+---
+authors:
+- LincolnLee:
+ name: "Lincoln Lee"
+ twitter: lincoln_86xy
+
+date: "2024-03-xxT22:00:00Z"
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.19
+aliases:
+- /news/2024/03/xx/release-1.19.0.html
+---
+
+The Apache Flink PMC is pleased to announce the release of Apache Flink
1.19.0. As usual, we are
+looking at a packed release with a wide variety of improvements and new
features. Overall, 162
+people contributed to this release completing 33 FLIPS and 600+ issues. Thank
you!
Review Comment:
```suggestion
people contributed to this release completing 33 FLIPs and 600+ issues.
Thank you!
```
##
docs/content/posts/2024-03-xx-release-1.19.0.md:
##
@@ -0,0 +1,456 @@
+---
+authors:
+- LincolnLee:
+ name: "Lincoln Lee"
+ twitter: lincoln_86xy
+
+date: "2024-03-xxT22:00:00Z"
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.19
+aliases:
+- /news/2024/03/xx/release-1.19.0.html
+---
+
+The Apache Flink PMC is pleased to announce the release of Apache Flink
1.19.0. As usual, we are
+looking at a packed release with a wide variety of improvements and new
features. Overall, 162
+people contributed to this release completing 33 FLIPS and 600+ issues. Thank
you!
+
+Let's dive into the highlights.
+
+# Flink SQL Improvements
+
+## Custom Parallelism for Table/SQL Sources
+
+Now in Flink 1.19, you can set a custom parallelism for performance tuning via
the `scan.parallelism`
+option. The first available connector is DataGen (Kafka connector is on the
way). Here is an example
+using SQL Client:
+
+```sql
+-- set parallelism within the ddl
+CREATE TABLE Orders (
+order_number BIGINT,
+priceDECIMAL(32,2),
+buyerROW,
+order_time TIMESTAMP(3)
+) WITH (
+'connector' = 'datagen',
+'scan.parallelism' = '4'
+);
+
+-- or set parallelism via dynamic table option
+SELECT * FROM Orders /*+ OPTIONS('scan.parallelism'='4') */;
Review Comment:
I'd suggest to keep the announcement concise and move all the sample codes
to detailed documentations.
##
docs/content/posts/2024-03-xx-release-1.19.0.md:
##
@@ -0,0 +1,456 @@
+---
+authors:
+- LincolnLee:
+ name: "Lincoln Lee"
+ twitter: lincoln_86xy
+
+date: "2024-03-xxT22:00:00Z"
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.19
+aliases:
+- /news/2024/03/xx/release-1.19.0.html
+---
+
+The Apache Flink PMC is pleased to announce the release of Apache Flink
1.19.0. As usual, we are
+looking at a packed release with a wide variety of improvements and new
features. Overall, 162
+people contributed to this release completing 33 FLIPS and 600+ issues. Thank
you!
+
+Let's dive into the highlights.
+
+# Flink SQL Improvements
+
+## Custom Parallelism for Table/SQL Sources
+
+Now in Flink 1.19, you can set a custom parallelism for performance tuning via
the `scan.parallelism`
+option. The first available connector is DataGen (Kafka connector is on the
way). Here is an example
+using SQL Client:
+
+```sql
+-- set parallelism within the ddl
+CREATE TABLE Orders (
+order_number BIGINT,
+priceDECIMAL(32,2),
+buyerROW,
+order_time TIMESTAMP(3)
+) WITH (
+'connector' = 'datagen',
+'scan.parallelism' = '4'
+);
+
+-- or set parallelism via dynamic table option
+SELECT * FROM Orders /*+ OPTIONS('scan.parallelism'='4') */;
+```
+
+**More Information**
+*
[Documentation](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sourcessinks/#scan-table-source)
+* [FLIP-367: Support Setting Parallelism for Table/SQL
Sources](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150)
+
+
+## Configurable SQL Gateway Java Options
+
+A new option `env.java.opts.sql-gateway` for specifying the Java options, so
you can fine-tune the
+memory settings, garbage collection behavior, and other relevant Java
parameters for SQL Gateway.
+
+**More Information**
+* [FLINK-33203](https://issues.apache.org/jira/browse/FLINK-33203)
+
+
+## Configure Different State TTLs using SQL Hint
+
+Starting from Flink 1.18, Table API and SQL users can set state time-to-live
(TTL) individually for
+stateful operators via the SQL compiled plan. In Flink 1.19, users have a more
flexible way to
+specify custom TTL values for regular joins and group aggregations directly
within their queries by [utilizing the STATE_TTL
hint](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/queries/hints/#state-ttl-hints).
+This improvement means that you no longer need to alter your compiled plan to
set specific TTLs for
+these frequently used operators. With the introduction of `STATE_TTL` hints,
you can streamline your workflow and
+dynamically adjust the TTL based on your oper
Re: [PR] Add announcement blog post for Flink 1.19 [flink-web]
MartijnVisser commented on code in PR #721: URL: https://github.com/apache/flink-web/pull/721#discussion_r1519461611 ## docs/content/posts/2024-03-xx-release-1.19.0.md: ## @@ -0,0 +1,456 @@ +--- Review Comment: @lincoln-lil I think we're missing something on the changes to the Sink API, from both https://issues.apache.org/jira/browse/FLINK-25857 and https://issues.apache.org/jira/browse/FLINK-33972. Same for https://cwiki.apache.org/confluence/display/FLINK/FLIP-400%3A+AsyncScalarFunction+for+asynchronous+scalar+function+support. LGTM otherwise -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add announcement blog post for Flink 1.19 [flink-web]
Sxnan commented on code in PR #721:
URL: https://github.com/apache/flink-web/pull/721#discussion_r1519464357
##
docs/content/posts/2024-03-xx-release-1.19.0.md:
##
@@ -0,0 +1,456 @@
+---
+authors:
+- LincolnLee:
+ name: "Lincoln Lee"
+ twitter: lincoln_86xy
+
+date: "2024-03-xxT22:00:00Z"
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.19
+aliases:
+- /news/2024/03/xx/release-1.19.0.html
+---
+
+The Apache Flink PMC is pleased to announce the release of Apache Flink
1.19.0. As usual, we are
+looking at a packed release with a wide variety of improvements and new
features. Overall, 162
+people contributed to this release completing 33 FLIPS and 600+ issues. Thank
you!
+
+Let's dive into the highlights.
+
+# Flink SQL Improvements
+
+## Custom Parallelism for Table/SQL Sources
+
+Now in Flink 1.19, you can set a custom parallelism for performance tuning via
the `scan.parallelism`
+option. The first available connector is DataGen (Kafka connector is on the
way). Here is an example
+using SQL Client:
+
+```sql
+-- set parallelism within the ddl
+CREATE TABLE Orders (
+order_number BIGINT,
+priceDECIMAL(32,2),
+buyerROW,
+order_time TIMESTAMP(3)
+) WITH (
+'connector' = 'datagen',
+'scan.parallelism' = '4'
+);
+
+-- or set parallelism via dynamic table option
+SELECT * FROM Orders /*+ OPTIONS('scan.parallelism'='4') */;
+```
+
+**More Information**
+*
[Documentation](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sourcessinks/#scan-table-source)
+* [FLIP-367: Support Setting Parallelism for Table/SQL
Sources](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150)
+
+
+## Configurable SQL Gateway Java Options
+
+A new option `env.java.opts.sql-gateway` for specifying the Java options, so
you can fine-tune the
+memory settings, garbage collection behavior, and other relevant Java
parameters for SQL Gateway.
+
+**More Information**
+* [FLINK-33203](https://issues.apache.org/jira/browse/FLINK-33203)
+
+
+## Configure Different State TTLs using SQL Hint
+
+Starting from Flink 1.18, Table API and SQL users can set state time-to-live
(TTL) individually for
+stateful operators via the SQL compiled plan. In Flink 1.19, users have a more
flexible way to
+specify custom TTL values for regular joins and group aggregations directly
within their queries by [utilizing the STATE_TTL
hint](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/queries/hints/#state-ttl-hints).
+This improvement means that you no longer need to alter your compiled plan to
set specific TTLs for
+these frequently used operators. With the introduction of `STATE_TTL` hints,
you can streamline your workflow and
+dynamically adjust the TTL based on your operational requirements.
+
+Here is an example:
+```sql
+-- set state ttl for join
+SELECT /*+ STATE_TTL('Orders'= '1d', 'Customers' = '20d') */ *
+FROM Orders LEFT OUTER JOIN Customers
+ON Orders.o_custkey = Customers.c_custkey;
+
+-- set state ttl for aggregation
+SELECT /*+ STATE_TTL('o' = '1d') */ o_orderkey, SUM(o_totalprice) AS revenue
+FROM Orders AS o
+GROUP BY o_orderkey;
+```
+
+**More Information**
+*
[Documentation](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/queries/hints/#state-ttl-hints)
+* [FLIP-373: Support Configuring Different State TTLs using SQL
Hint](https://cwiki.apache.org/confluence/display/FLINK/FLIP-373%3A+Support+Configuring+Different+State+TTLs+using+SQL+Hint)
+
+
+## Named Parameters for Functions and Procedures
+
+Named parameters now can be used when calling a function or stored procedure.
With named parameters,
+users do not need to strictly specify the parameter position, just specify the
parameter name and its
+corresponding value. At the same time, if non-essential parameters are not
specified, they will default to being filled with null.
+
+Here's an example of defining a function with one mandatory parameter and two
optional parameters using named parameters:
+```java
+public static class NamedArgumentsTableFunction extends TableFunction {
+
+ @FunctionHint(
+ output = @DataTypeHint("STRING"),
+ arguments = {
+ @ArgumentHint(name = "in1", isOptional
= false, type = @DataTypeHint("STRING")),
+ @ArgumentHint(name = "in2", isOptional
= true, type = @DataTypeHint("STRING")),
+ @ArgumentHint(name = "in3", isOptional
= true, type = @DataTypeHint("STRING"))})
+ public void eval(String arg1, String arg2, String arg3) {
+ collect(arg1 + ", " + arg2 + "," + arg3);
+ }
+
+}
+```
+When calling the function in SQL, parameters can be specified by name, for
example:
+```sql
+SELECT * FROM TABLE(myFunction(in1 => 'v1', in3 => 'v3', in2 => 'v2'))
+```
+Also the optional parameters can b
[jira] [Commented] (FLINK-31663) Add ARRAY_EXCEPT supported in SQL & Table API
[ https://issues.apache.org/jira/browse/FLINK-31663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825244#comment-17825244 ] Jacky Lau commented on FLINK-31663: --- [~Sergey Nuyanzin] in Flink, array_contains returns true if the array contains null values. However, if the element is null, array_position returns null, which is also a different inconsistent behavior. because the two functions handle null values inconsistently. and can also be confusing for users. should we change the behavior ? > Add ARRAY_EXCEPT supported in SQL & Table API > - > > Key: FLINK-31663 > URL: https://issues.apache.org/jira/browse/FLINK-31663 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: luoyuxia >Assignee: Hanyu Zheng >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34543][datastream]Support Full Partition Processing On Non-keyed DataStream [flink]
reswqa commented on code in PR #24398:
URL: https://github.com/apache/flink/pull/24398#discussion_r1519327670
##
flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/KeyedPartitionWindowedStreamITCase.java:
##
@@ -0,0 +1,521 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.streaming.runtime;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.MapPartitionFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.common.operators.Order;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.datastream.KeyedPartitionWindowedStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.util.CloseableIterator;
+import org.apache.flink.util.Collector;
+
+import org.apache.flink.shaded.guava31.com.google.common.collect.Lists;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/** Integration tests for {@link KeyedPartitionWindowedStream}. */
+public class KeyedPartitionWindowedStreamITCase {
+
+private static final int EVENT_NUMBER = 100;
+
+private static final String TEST_EVENT = "Test";
+
+@Test
+public void testMapPartition() throws Exception {
Review Comment:
```suggestion
void testMapPartition() throws Exception {
```
Please let it package-private for junit5 backed testing class and suite. We
need to review all newly introduced tests.
##
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/MapPartitionOperator.java:
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.MapPartitionFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.ExceptionUtils;
+
+/**
+ * The {@link MapPartitionOperator} is used to process all records in each
partition on non-keyed
+ * stream. Each partition contains all records of a subtask.
+ */
+@Internal
+public class MapPartitionOperator
+extends AbstractUdfStreamOperator>
+implements OneInputStreamOperator, BoundedOneInput {
+
+private final MapPartitionFunction function;
+
+private MapPartitionIterator iterator;
+
+public MapPartitionOperator(MapPartitionFunction function) {
+super(function);
+this.function = function;
+// This operator is set to be non-chained as it doesn't use task main
thread to write
+// records to output, which may introduce risks to downstream chained
operators.
+this.chainingStrategy = ChainingStrategy.NEVER;
Review Comment:
IIUC, this prevents this operator from being chained with the previous
operator, we do not want `mapPartition` to introduce shuffle stage here. If
this is the case, we may need a new chaining strategy(Not thinking it through).
##
flink-streamin
[jira] [Commented] (FLINK-31663) Add ARRAY_EXCEPT supported in SQL & Table API
[
https://issues.apache.org/jira/browse/FLINK-31663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825251#comment-17825251
]
Sergey Nuyanzin commented on FLINK-31663:
-
let's no mix everything in one place
This is a jira issue for {{ARRAY_EXCEPT}}
if you feel there is a some other problem, feel free to file a jira issue with
detailed description what and why is wrong
> Add ARRAY_EXCEPT supported in SQL & Table API
> -
>
> Key: FLINK-31663
> URL: https://issues.apache.org/jira/browse/FLINK-31663
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: luoyuxia
>Assignee: Hanyu Zheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31663) Add ARRAY_EXCEPT supported in SQL & Table API
[
https://issues.apache.org/jira/browse/FLINK-31663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825251#comment-17825251
]
Sergey Nuyanzin edited comment on FLINK-31663 at 3/11/24 10:36 AM:
---
let's no mix everything in one place
This is a jira issue for {{ARRAY_EXCEPT}}
if you feel there is some other problem, feel free to file a jira issue with
detailed description what and why is wrong
was (Author: sergey nuyanzin):
let's no mix everything in one place
This is a jira issue for {{ARRAY_EXCEPT}}
if you feel there is a some other problem, feel free to file a jira issue with
detailed description what and why is wrong
> Add ARRAY_EXCEPT supported in SQL & Table API
> -
>
> Key: FLINK-31663
> URL: https://issues.apache.org/jira/browse/FLINK-31663
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: luoyuxia
>Assignee: Hanyu Zheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31663) Add ARRAY_EXCEPT supported in SQL & Table API
[
https://issues.apache.org/jira/browse/FLINK-31663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825251#comment-17825251
]
Sergey Nuyanzin edited comment on FLINK-31663 at 3/11/24 10:41 AM:
---
let's not mix everything in one place please.
This is a jira issue for {{ARRAY_EXCEPT}}.
if you feel there is some other problem, feel free to file a jira issue with
detailed description what and why is wrong.
was (Author: sergey nuyanzin):
let's no mix everything in one place
This is a jira issue for {{ARRAY_EXCEPT}}
if you feel there is some other problem, feel free to file a jira issue with
detailed description what and why is wrong
> Add ARRAY_EXCEPT supported in SQL & Table API
> -
>
> Key: FLINK-31663
> URL: https://issues.apache.org/jira/browse/FLINK-31663
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: luoyuxia
>Assignee: Hanyu Zheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] Add announcement blog post for Flink 1.19 [flink-web]
JunRuiLee commented on code in PR #721:
URL: https://github.com/apache/flink-web/pull/721#discussion_r1519522297
##
docs/content/posts/2024-03-xx-release-1.19.0.md:
##
@@ -0,0 +1,456 @@
+---
+authors:
+- LincolnLee:
+ name: "Lincoln Lee"
+ twitter: lincoln_86xy
+
+date: "2024-03-xxT22:00:00Z"
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.19
+aliases:
+- /news/2024/03/xx/release-1.19.0.html
+---
+
+The Apache Flink PMC is pleased to announce the release of Apache Flink
1.19.0. As usual, we are
+looking at a packed release with a wide variety of improvements and new
features. Overall, 162
+people contributed to this release completing 33 FLIPS and 600+ issues. Thank
you!
+
+Let's dive into the highlights.
+
+# Flink SQL Improvements
+
+## Custom Parallelism for Table/SQL Sources
+
+Now in Flink 1.19, you can set a custom parallelism for performance tuning via
the `scan.parallelism`
+option. The first available connector is DataGen (Kafka connector is on the
way). Here is an example
+using SQL Client:
+
+```sql
+-- set parallelism within the ddl
+CREATE TABLE Orders (
+order_number BIGINT,
+priceDECIMAL(32,2),
+buyerROW,
+order_time TIMESTAMP(3)
+) WITH (
+'connector' = 'datagen',
+'scan.parallelism' = '4'
+);
+
+-- or set parallelism via dynamic table option
+SELECT * FROM Orders /*+ OPTIONS('scan.parallelism'='4') */;
+```
+
+**More Information**
+*
[Documentation](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sourcessinks/#scan-table-source)
+* [FLIP-367: Support Setting Parallelism for Table/SQL
Sources](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150)
+
+
+## Configurable SQL Gateway Java Options
+
+A new option `env.java.opts.sql-gateway` for specifying the Java options, so
you can fine-tune the
+memory settings, garbage collection behavior, and other relevant Java
parameters for SQL Gateway.
+
+**More Information**
+* [FLINK-33203](https://issues.apache.org/jira/browse/FLINK-33203)
+
+
+## Configure Different State TTLs using SQL Hint
+
+Starting from Flink 1.18, Table API and SQL users can set state time-to-live
(TTL) individually for
+stateful operators via the SQL compiled plan. In Flink 1.19, users have a more
flexible way to
+specify custom TTL values for regular joins and group aggregations directly
within their queries by [utilizing the STATE_TTL
hint](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/queries/hints/#state-ttl-hints).
+This improvement means that you no longer need to alter your compiled plan to
set specific TTLs for
+these frequently used operators. With the introduction of `STATE_TTL` hints,
you can streamline your workflow and
+dynamically adjust the TTL based on your operational requirements.
+
+Here is an example:
+```sql
+-- set state ttl for join
+SELECT /*+ STATE_TTL('Orders'= '1d', 'Customers' = '20d') */ *
+FROM Orders LEFT OUTER JOIN Customers
+ON Orders.o_custkey = Customers.c_custkey;
+
+-- set state ttl for aggregation
+SELECT /*+ STATE_TTL('o' = '1d') */ o_orderkey, SUM(o_totalprice) AS revenue
+FROM Orders AS o
+GROUP BY o_orderkey;
+```
+
+**More Information**
+*
[Documentation](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/queries/hints/#state-ttl-hints)
+* [FLIP-373: Support Configuring Different State TTLs using SQL
Hint](https://cwiki.apache.org/confluence/display/FLINK/FLIP-373%3A+Support+Configuring+Different+State+TTLs+using+SQL+Hint)
+
+
+## Named Parameters for Functions and Procedures
+
+Named parameters now can be used when calling a function or stored procedure.
With named parameters,
+users do not need to strictly specify the parameter position, just specify the
parameter name and its
+corresponding value. At the same time, if non-essential parameters are not
specified, they will default to being filled with null.
+
+Here's an example of defining a function with one mandatory parameter and two
optional parameters using named parameters:
+```java
+public static class NamedArgumentsTableFunction extends TableFunction {
+
+ @FunctionHint(
+ output = @DataTypeHint("STRING"),
+ arguments = {
+ @ArgumentHint(name = "in1", isOptional
= false, type = @DataTypeHint("STRING")),
+ @ArgumentHint(name = "in2", isOptional
= true, type = @DataTypeHint("STRING")),
+ @ArgumentHint(name = "in3", isOptional
= true, type = @DataTypeHint("STRING"))})
+ public void eval(String arg1, String arg2, String arg3) {
+ collect(arg1 + ", " + arg2 + "," + arg3);
+ }
+
+}
+```
+When calling the function in SQL, parameters can be specified by name, for
example:
+```sql
+SELECT * FROM TABLE(myFunction(in1 => 'v1', in3 => 'v3', in2 => 'v2'))
+```
+Also the optional parameters c
Re: [PR] Add announcement blog post for Flink 1.19 [flink-web]
JunRuiLee commented on code in PR #721:
URL: https://github.com/apache/flink-web/pull/721#discussion_r1519522297
##
docs/content/posts/2024-03-xx-release-1.19.0.md:
##
@@ -0,0 +1,456 @@
+---
+authors:
+- LincolnLee:
+ name: "Lincoln Lee"
+ twitter: lincoln_86xy
+
+date: "2024-03-xxT22:00:00Z"
+subtitle: ""
+title: Announcing the Release of Apache Flink 1.19
+aliases:
+- /news/2024/03/xx/release-1.19.0.html
+---
+
+The Apache Flink PMC is pleased to announce the release of Apache Flink
1.19.0. As usual, we are
+looking at a packed release with a wide variety of improvements and new
features. Overall, 162
+people contributed to this release completing 33 FLIPS and 600+ issues. Thank
you!
+
+Let's dive into the highlights.
+
+# Flink SQL Improvements
+
+## Custom Parallelism for Table/SQL Sources
+
+Now in Flink 1.19, you can set a custom parallelism for performance tuning via
the `scan.parallelism`
+option. The first available connector is DataGen (Kafka connector is on the
way). Here is an example
+using SQL Client:
+
+```sql
+-- set parallelism within the ddl
+CREATE TABLE Orders (
+order_number BIGINT,
+priceDECIMAL(32,2),
+buyerROW,
+order_time TIMESTAMP(3)
+) WITH (
+'connector' = 'datagen',
+'scan.parallelism' = '4'
+);
+
+-- or set parallelism via dynamic table option
+SELECT * FROM Orders /*+ OPTIONS('scan.parallelism'='4') */;
+```
+
+**More Information**
+*
[Documentation](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sourcessinks/#scan-table-source)
+* [FLIP-367: Support Setting Parallelism for Table/SQL
Sources](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150)
+
+
+## Configurable SQL Gateway Java Options
+
+A new option `env.java.opts.sql-gateway` for specifying the Java options, so
you can fine-tune the
+memory settings, garbage collection behavior, and other relevant Java
parameters for SQL Gateway.
+
+**More Information**
+* [FLINK-33203](https://issues.apache.org/jira/browse/FLINK-33203)
+
+
+## Configure Different State TTLs using SQL Hint
+
+Starting from Flink 1.18, Table API and SQL users can set state time-to-live
(TTL) individually for
+stateful operators via the SQL compiled plan. In Flink 1.19, users have a more
flexible way to
+specify custom TTL values for regular joins and group aggregations directly
within their queries by [utilizing the STATE_TTL
hint](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/queries/hints/#state-ttl-hints).
+This improvement means that you no longer need to alter your compiled plan to
set specific TTLs for
+these frequently used operators. With the introduction of `STATE_TTL` hints,
you can streamline your workflow and
+dynamically adjust the TTL based on your operational requirements.
+
+Here is an example:
+```sql
+-- set state ttl for join
+SELECT /*+ STATE_TTL('Orders'= '1d', 'Customers' = '20d') */ *
+FROM Orders LEFT OUTER JOIN Customers
+ON Orders.o_custkey = Customers.c_custkey;
+
+-- set state ttl for aggregation
+SELECT /*+ STATE_TTL('o' = '1d') */ o_orderkey, SUM(o_totalprice) AS revenue
+FROM Orders AS o
+GROUP BY o_orderkey;
+```
+
+**More Information**
+*
[Documentation](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/queries/hints/#state-ttl-hints)
+* [FLIP-373: Support Configuring Different State TTLs using SQL
Hint](https://cwiki.apache.org/confluence/display/FLINK/FLIP-373%3A+Support+Configuring+Different+State+TTLs+using+SQL+Hint)
+
+
+## Named Parameters for Functions and Procedures
+
+Named parameters now can be used when calling a function or stored procedure.
With named parameters,
+users do not need to strictly specify the parameter position, just specify the
parameter name and its
+corresponding value. At the same time, if non-essential parameters are not
specified, they will default to being filled with null.
+
+Here's an example of defining a function with one mandatory parameter and two
optional parameters using named parameters:
+```java
+public static class NamedArgumentsTableFunction extends TableFunction {
+
+ @FunctionHint(
+ output = @DataTypeHint("STRING"),
+ arguments = {
+ @ArgumentHint(name = "in1", isOptional
= false, type = @DataTypeHint("STRING")),
+ @ArgumentHint(name = "in2", isOptional
= true, type = @DataTypeHint("STRING")),
+ @ArgumentHint(name = "in3", isOptional
= true, type = @DataTypeHint("STRING"))})
+ public void eval(String arg1, String arg2, String arg3) {
+ collect(arg1 + ", " + arg2 + "," + arg3);
+ }
+
+}
+```
+When calling the function in SQL, parameters can be specified by name, for
example:
+```sql
+SELECT * FROM TABLE(myFunction(in1 => 'v1', in3 => 'v3', in2 => 'v2'))
+```
+Also the optional parameters c
[jira] [Created] (FLINK-34640) Replace DummyMetricGroup usage with UnregisteredMetricsGroup
Chesnay Schepler created FLINK-34640:
Summary: Replace DummyMetricGroup usage with
UnregisteredMetricsGroup
Key: FLINK-34640
URL: https://issues.apache.org/jira/browse/FLINK-34640
Project: Flink
Issue Type: Technical Debt
Components: Runtime / Metrics, Tests
Reporter: Chesnay Schepler
Fix For: 1.20.0
The {{DummyMetricGroup}} is terrible because it is decidedly unsafe to use. Use
the {{UnregisteredMetricsGroup}} instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-29114][connector][filesystem] Fix issue of file overwriting caused by multiple writes to the same sink table and shared staging directory [flink]
XComp commented on code in PR #24390:
URL: https://github.com/apache/flink/pull/24390#discussion_r1519417563
##
flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/table/FileSystemOutputFormatTest.java:
##
@@ -83,153 +103,175 @@ void after() {
@Test
void testClosingWithoutInput() throws Exception {
try (OneInputStreamOperatorTestHarness testHarness =
-createSink(false, false, false, new LinkedHashMap<>(), new
AtomicReference<>())) {
+createTestHarness(
+false, false, false, new LinkedHashMap<>(), new
AtomicReference<>())) {
testHarness.setup();
testHarness.open();
}
}
@Test
void testNonPartition() throws Exception {
-AtomicReference> ref = new
AtomicReference<>();
-try (OneInputStreamOperatorTestHarness testHarness =
-createSink(false, false, false, new LinkedHashMap<>(), ref)) {
-writeUnorderedRecords(testHarness);
-assertThat(getFileContentByPath(tmpPath)).hasSize(1);
-}
-
-ref.get().finalizeGlobal(finalizationContext);
-Map content = getFileContentByPath(outputPath);
-assertThat(content.values())
-.containsExactly("a1,1,p1\n" + "a2,2,p1\n" + "a2,2,p2\n" +
"a3,3,p1\n");
-}
-
-private void writeUnorderedRecords(OneInputStreamOperatorTestHarness testHarness)
-throws Exception {
-testHarness.setup();
-testHarness.open();
-
-testHarness.processElement(new StreamRecord<>(Row.of("a1", 1, "p1"),
1L));
-testHarness.processElement(new StreamRecord<>(Row.of("a2", 2, "p1"),
1L));
-testHarness.processElement(new StreamRecord<>(Row.of("a2", 2, "p2"),
1L));
-testHarness.processElement(new StreamRecord<>(Row.of("a3", 3, "p1"),
1L));
+checkWriteAndCommit(
+false,
+false,
+false,
+new LinkedHashMap<>(),
+DEFAULT_INPUT_SUPPLIER,
+DEFAULT_OUTPUT_SUPPLIER);
}
@Test
void testOverrideNonPartition() throws Exception {
testNonPartition();
-
-AtomicReference> ref = new
AtomicReference<>();
-try (OneInputStreamOperatorTestHarness testHarness =
-createSink(true, false, false, new LinkedHashMap<>(), ref)) {
-writeUnorderedRecords(testHarness);
-assertThat(getFileContentByPath(tmpPath)).hasSize(1);
-}
-
-ref.get().finalizeGlobal(finalizationContext);
-Map content = getFileContentByPath(outputPath);
-assertThat(content).hasSize(1);
-assertThat(content.values())
-.containsExactly("a1,1,p1\n" + "a2,2,p1\n" + "a2,2,p2\n" +
"a3,3,p1\n");
-assertThat(new File(tmpPath.toUri())).doesNotExist();
+checkWriteAndCommit(
+true,
+false,
+false,
+new LinkedHashMap<>(),
+DEFAULT_INPUT_SUPPLIER,
+DEFAULT_OUTPUT_SUPPLIER);
}
@Test
void testStaticPartition() throws Exception {
-AtomicReference> ref = new
AtomicReference<>();
LinkedHashMap staticParts = new LinkedHashMap<>();
staticParts.put("c", "p1");
-try (OneInputStreamOperatorTestHarness testHarness =
-createSink(false, true, false, staticParts, ref)) {
-testHarness.setup();
-testHarness.open();
-
-testHarness.processElement(new StreamRecord<>(Row.of("a1", 1),
1L));
-testHarness.processElement(new StreamRecord<>(Row.of("a2", 2),
1L));
-testHarness.processElement(new StreamRecord<>(Row.of("a2", 2),
1L));
-testHarness.processElement(new StreamRecord<>(Row.of("a3", 3),
1L));
-assertThat(getFileContentByPath(tmpPath)).hasSize(1);
-}
-ref.get().finalizeGlobal(finalizationContext);
-Map content = getFileContentByPath(outputPath);
-assertThat(content).hasSize(1);
-
assertThat(content.keySet().iterator().next().getParentFile().getName()).isEqualTo("c=p1");
-assertThat(content.values()).containsExactly("a1,1\n" + "a2,2\n" +
"a2,2\n" + "a3,3\n");
-assertThat(new File(tmpPath.toUri())).doesNotExist();
+checkWriteAndCommit(
+false,
+true,
+false,
+staticParts,
+() ->
+Arrays.asList(
+new StreamRecord<>(Row.of("a1", 1), 1L),
+new StreamRecord<>(Row.of("a2", 2), 1L),
+new StreamRecord<>(Row.of("a2", 2), 1L),
+new StreamRecord<>(Row.of("a3", 3), 1L)),
+() ->
+Collections.singletonList(
+"c=p1:
Re: [PR] [FLINK-34334][state] Add sub-task level RocksDB file count metrics [flink]
Zakelly commented on code in PR #24322:
URL: https://github.com/apache/flink/pull/24322#discussion_r1519549762
##
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBProperty.java:
##
@@ -31,55 +31,92 @@
*/
@Internal
public enum RocksDBProperty {
-NumImmutableMemTable("num-immutable-mem-table"),
-MemTableFlushPending("mem-table-flush-pending"),
-CompactionPending("compaction-pending"),
-BackgroundErrors("background-errors"),
-CurSizeActiveMemTable("cur-size-active-mem-table"),
-CurSizeAllMemTables("cur-size-all-mem-tables"),
-SizeAllMemTables("size-all-mem-tables"),
-NumEntriesActiveMemTable("num-entries-active-mem-table"),
-NumEntriesImmMemTables("num-entries-imm-mem-tables"),
-NumDeletesActiveMemTable("num-deletes-active-mem-table"),
-NumDeletesImmMemTables("num-deletes-imm-mem-tables"),
-EstimateNumKeys("estimate-num-keys"),
-EstimateTableReadersMem("estimate-table-readers-mem"),
-NumSnapshots("num-snapshots"),
-NumLiveVersions("num-live-versions"),
-EstimateLiveDataSize("estimate-live-data-size"),
-TotalSstFilesSize("total-sst-files-size"),
-LiveSstFilesSize("live-sst-files-size"),
-EstimatePendingCompactionBytes("estimate-pending-compaction-bytes"),
-NumRunningCompactions("num-running-compactions"),
-NumRunningFlushes("num-running-flushes"),
-ActualDelayedWriteRate("actual-delayed-write-rate"),
-IsWriteStopped("is-write-stopped"),
-BlockCacheCapacity("block-cache-capacity"),
-BlockCacheUsage("block-cache-usage"),
-BlockCachePinnedUsage("block-cache-pinned-usage");
+NumImmutableMemTable("num-immutable-mem-table", PropertyType.NUMBER),
+MemTableFlushPending("mem-table-flush-pending", PropertyType.NUMBER),
+CompactionPending("compaction-pending", PropertyType.NUMBER),
+BackgroundErrors("background-errors", PropertyType.NUMBER),
+CurSizeActiveMemTable("cur-size-active-mem-table", PropertyType.NUMBER),
+CurSizeAllMemTables("cur-size-all-mem-tables", PropertyType.NUMBER),
+SizeAllMemTables("size-all-mem-tables", PropertyType.NUMBER),
+NumEntriesActiveMemTable("num-entries-active-mem-table",
PropertyType.NUMBER),
+NumEntriesImmMemTables("num-entries-imm-mem-tables", PropertyType.NUMBER),
+NumDeletesActiveMemTable("num-deletes-active-mem-table",
PropertyType.NUMBER),
+NumDeletesImmMemTables("num-deletes-imm-mem-tables", PropertyType.NUMBER),
+EstimateNumKeys("estimate-num-keys", PropertyType.NUMBER),
+EstimateTableReadersMem("estimate-table-readers-mem", PropertyType.NUMBER),
+NumSnapshots("num-snapshots", PropertyType.NUMBER),
+NumLiveVersions("num-live-versions", PropertyType.NUMBER),
+EstimateLiveDataSize("estimate-live-data-size", PropertyType.NUMBER),
+TotalSstFilesSize("total-sst-files-size", PropertyType.NUMBER),
+LiveSstFilesSize("live-sst-files-size", PropertyType.NUMBER),
+EstimatePendingCompactionBytes("estimate-pending-compaction-bytes",
PropertyType.NUMBER),
+NumRunningCompactions("num-running-compactions", PropertyType.NUMBER),
+NumRunningFlushes("num-running-flushes", PropertyType.NUMBER),
+ActualDelayedWriteRate("actual-delayed-write-rate", PropertyType.NUMBER),
+IsWriteStopped("is-write-stopped", PropertyType.NUMBER),
+BlockCacheCapacity("block-cache-capacity", PropertyType.NUMBER),
+BlockCacheUsage("block-cache-usage", PropertyType.NUMBER),
+BlockCachePinnedUsage("block-cache-pinned-usage", PropertyType.NUMBER),
+NumFilesAtLevel("num-files-at-level", PropertyType.STRING);
private static final String ROCKS_DB_PROPERTY_FORMAT = "rocksdb.%s";
private static final String CONFIG_KEY_FORMAT =
"state.backend.rocksdb.metrics.%s";
-private final String property;
+private final String propertyName;
-RocksDBProperty(String property) {
-this.property = property;
+private final PropertyType type;
+
+/** Property type. */
+public enum PropertyType {
+NUMBER,
+STRING
}
-/**
- * @return property string that can be used to query {@link
- * RocksDB#getLongProperty(ColumnFamilyHandle, String)}.
- */
-public String getRocksDBProperty() {
-return String.format(ROCKS_DB_PROPERTY_FORMAT, property);
+RocksDBProperty(String propertyName, PropertyType type) {
+this.propertyName = propertyName;
+this.type = type;
+}
+
+public String getPropertyName() {
+return this.propertyName;
+}
+
+public static RocksDBProperty getRocksDBProperty(final String property) {
+// NumFilesAtLevel controls multiple levels of file count monitoring,
each a separate metric
+if (property.startsWith(NumFilesAtLevel.getPropertyName())) {
+return NumFilesAtLevel;
+}
+for (final RocksDBProperty rocksDBProperty : RocksDBProperty.values())
{
Review Comment:
Is
[jira] [Commented] (FLINK-31663) Add ARRAY_EXCEPT supported in SQL & Table API
[ https://issues.apache.org/jira/browse/FLINK-31663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825260#comment-17825260 ] Jacky Lau commented on FLINK-31663: --- [~Sergey Nuyanzin] ok. if the array_except and array_concat have duplicates while array_union do not have. we should add description in the docs > Add ARRAY_EXCEPT supported in SQL & Table API > - > > Key: FLINK-31663 > URL: https://issues.apache.org/jira/browse/FLINK-31663 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: luoyuxia >Assignee: Hanyu Zheng >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34588) FineGrainedSlotManager checks whether resources need to reconcile but doesn't act on the result
[
https://issues.apache.org/jira/browse/FLINK-34588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825261#comment-17825261
]
Matthias Pohl commented on FLINK-34588:
---
Ok, thanks for clarification. I might add this information as comments to my
FLINK-34427 PR. (y)
> FineGrainedSlotManager checks whether resources need to reconcile but doesn't
> act on the result
> ---
>
> Key: FLINK-34588
> URL: https://issues.apache.org/jira/browse/FLINK-34588
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.19.0, 1.18.1, 1.20.0
>Reporter: Matthias Pohl
>Priority: Major
>
> There are a few locations in {{FineGrainedSlotManager}} where we check
> whether resources can/need to be reconciled but don't care about the result
> and just trigger the resource update (e.g. in
> [FineGrainedSlotManager:626|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/FineGrainedSlotManager.java#L626]
> and
> [FineGrainedSlotManager:682|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/FineGrainedSlotManager.java#L682]).
> Looks like we could reduce the calls to the backend here.
> It's not having a major impact because this feature is only used in the
> {{ActiveResourceManager}} which triggers
> [checkResourceDeclarations|https://github.com/apache/flink/blob/c678244a3890273145a786b9e1bf1a4f96f6dcfd/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java#L331]
> and reevaluates the {{resourceDeclarations}}. Not sure whether I missed
> something here and there's actually a bigger issue with it. But considering
> that nobody complained about it in the past, I'd assume that it's not a
> severe issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] Add announcement blog post for Flink 1.19 [flink-web]
masteryhx commented on code in PR #721: URL: https://github.com/apache/flink-web/pull/721#discussion_r1519492535 ## docs/data/release_archive.yml: ## @@ -1,6 +1,10 @@ release_archive: flink: -- version_short: "1.18" +- version_short: "1.19" + version_long: 1.18.0 Review Comment: ```suggestion version_long: 1.19.0 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-34641) Possibility to add Python Files from http/https sources in PythonDriver
Aleksei Perminov created FLINK-34641:
Summary: Possibility to add Python Files from http/https sources
in PythonDriver
Key: FLINK-34641
URL: https://issues.apache.org/jira/browse/FLINK-34641
Project: Flink
Issue Type: New Feature
Components: flink-contrib
Affects Versions: 1.18.1, 1.18.0, 1.18.2
Reporter: Aleksei Perminov
Problem: Now it is impossible to add python files from the external http
sources, only fs sources are available. It is not very convenient to add Python
packages to the Flink Docker image or external fs every time the new version of
the Python job is built, especially when using the Session mode.
Proposal: Add possibility to handle the http/https protocols in the
PythonDriver so that it is possible to add packages from the external Python
repositories (e.g. PyPi).
Example of the solution in the PythonEnvUtils.java
{code:java}
private static void addToPythonPath(PythonEnvironment env, List
pythonFiles) {
...
if (isValidURL(pythonFile.toString()) && new
URL(pythonFile.toString()).getProtocol().startsWith("http")) {
getFileFromHTTP(pythonFile, targetPath);
} else if (!pythonFile.getFileSystem().isDistributedFS()) {
...
}{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [hotfix] Minor autoscaler event and metric improvements [flink-kubernetes-operator]
gyfora merged PR #794: URL: https://github.com/apache/flink-kubernetes-operator/pull/794 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34334][state] Add sub-task level RocksDB file count metrics [flink]
masteryhx commented on code in PR #24322:
URL: https://github.com/apache/flink/pull/24322#discussion_r1519609456
##
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBProperty.java:
##
@@ -31,55 +31,92 @@
*/
@Internal
public enum RocksDBProperty {
-NumImmutableMemTable("num-immutable-mem-table"),
-MemTableFlushPending("mem-table-flush-pending"),
-CompactionPending("compaction-pending"),
-BackgroundErrors("background-errors"),
-CurSizeActiveMemTable("cur-size-active-mem-table"),
-CurSizeAllMemTables("cur-size-all-mem-tables"),
-SizeAllMemTables("size-all-mem-tables"),
-NumEntriesActiveMemTable("num-entries-active-mem-table"),
-NumEntriesImmMemTables("num-entries-imm-mem-tables"),
-NumDeletesActiveMemTable("num-deletes-active-mem-table"),
-NumDeletesImmMemTables("num-deletes-imm-mem-tables"),
-EstimateNumKeys("estimate-num-keys"),
-EstimateTableReadersMem("estimate-table-readers-mem"),
-NumSnapshots("num-snapshots"),
-NumLiveVersions("num-live-versions"),
-EstimateLiveDataSize("estimate-live-data-size"),
-TotalSstFilesSize("total-sst-files-size"),
-LiveSstFilesSize("live-sst-files-size"),
-EstimatePendingCompactionBytes("estimate-pending-compaction-bytes"),
-NumRunningCompactions("num-running-compactions"),
-NumRunningFlushes("num-running-flushes"),
-ActualDelayedWriteRate("actual-delayed-write-rate"),
-IsWriteStopped("is-write-stopped"),
-BlockCacheCapacity("block-cache-capacity"),
-BlockCacheUsage("block-cache-usage"),
-BlockCachePinnedUsage("block-cache-pinned-usage");
+NumImmutableMemTable("num-immutable-mem-table", PropertyType.NUMBER),
+MemTableFlushPending("mem-table-flush-pending", PropertyType.NUMBER),
+CompactionPending("compaction-pending", PropertyType.NUMBER),
+BackgroundErrors("background-errors", PropertyType.NUMBER),
+CurSizeActiveMemTable("cur-size-active-mem-table", PropertyType.NUMBER),
+CurSizeAllMemTables("cur-size-all-mem-tables", PropertyType.NUMBER),
+SizeAllMemTables("size-all-mem-tables", PropertyType.NUMBER),
+NumEntriesActiveMemTable("num-entries-active-mem-table",
PropertyType.NUMBER),
+NumEntriesImmMemTables("num-entries-imm-mem-tables", PropertyType.NUMBER),
+NumDeletesActiveMemTable("num-deletes-active-mem-table",
PropertyType.NUMBER),
+NumDeletesImmMemTables("num-deletes-imm-mem-tables", PropertyType.NUMBER),
+EstimateNumKeys("estimate-num-keys", PropertyType.NUMBER),
+EstimateTableReadersMem("estimate-table-readers-mem", PropertyType.NUMBER),
+NumSnapshots("num-snapshots", PropertyType.NUMBER),
+NumLiveVersions("num-live-versions", PropertyType.NUMBER),
+EstimateLiveDataSize("estimate-live-data-size", PropertyType.NUMBER),
+TotalSstFilesSize("total-sst-files-size", PropertyType.NUMBER),
+LiveSstFilesSize("live-sst-files-size", PropertyType.NUMBER),
+EstimatePendingCompactionBytes("estimate-pending-compaction-bytes",
PropertyType.NUMBER),
+NumRunningCompactions("num-running-compactions", PropertyType.NUMBER),
+NumRunningFlushes("num-running-flushes", PropertyType.NUMBER),
+ActualDelayedWriteRate("actual-delayed-write-rate", PropertyType.NUMBER),
+IsWriteStopped("is-write-stopped", PropertyType.NUMBER),
+BlockCacheCapacity("block-cache-capacity", PropertyType.NUMBER),
+BlockCacheUsage("block-cache-usage", PropertyType.NUMBER),
+BlockCachePinnedUsage("block-cache-pinned-usage", PropertyType.NUMBER),
+NumFilesAtLevel("num-files-at-level", PropertyType.STRING);
private static final String ROCKS_DB_PROPERTY_FORMAT = "rocksdb.%s";
private static final String CONFIG_KEY_FORMAT =
"state.backend.rocksdb.metrics.%s";
-private final String property;
+private final String propertyName;
-RocksDBProperty(String property) {
-this.property = property;
+private final PropertyType type;
+
+/** Property type. */
+public enum PropertyType {
+NUMBER,
+STRING
}
-/**
- * @return property string that can be used to query {@link
- * RocksDB#getLongProperty(ColumnFamilyHandle, String)}.
- */
-public String getRocksDBProperty() {
-return String.format(ROCKS_DB_PROPERTY_FORMAT, property);
+RocksDBProperty(String propertyName, PropertyType type) {
+this.propertyName = propertyName;
+this.type = type;
+}
+
+public String getPropertyName() {
+return this.propertyName;
+}
+
+public static RocksDBProperty getRocksDBProperty(final String property) {
+// NumFilesAtLevel controls multiple levels of file count monitoring,
each a separate metric
+if (property.startsWith(NumFilesAtLevel.getPropertyName())) {
+return NumFilesAtLevel;
+}
+for (final RocksDBProperty rocksDBProperty : RocksDBProperty.values())
{
Review Comment:
A
[jira] [Updated] (FLINK-27593) Parquet/Orc format reader optimization
[ https://issues.apache.org/jira/browse/FLINK-27593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-27593: Fix Version/s: 1.20.0 > Parquet/Orc format reader optimization > -- > > Key: FLINK-27593 > URL: https://issues.apache.org/jira/browse/FLINK-27593 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Runtime >Reporter: dalongliu >Priority: Major > Fix For: 1.19.0, 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29941) Support SQL Client connects to SQL Gateway
[ https://issues.apache.org/jira/browse/FLINK-29941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-29941: Fix Version/s: 1.20.0 > Support SQL Client connects to SQL Gateway > -- > > Key: FLINK-29941 > URL: https://issues.apache.org/jira/browse/FLINK-29941 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Client, Table SQL / Gateway >Reporter: Jark Wu >Priority: Major > Fix For: 1.19.0, 1.20.0 > > > This is an umbrella issue to track the effort to support SQL Client connects > to SQL Gateway, including SQL Client refactoring, end-to-end tests, and > documentation. > Design Docs: > https://docs.google.com/document/d/14cS4VBSamMUnlM_PZuK6QKLfriUuQU51iqET5oiYy_c/edit#heading=h.xfpxlv5v61pg -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29355) Sql parse failed because of Desc catalog.database.table is incorrectly parsed to desc catalog
[ https://issues.apache.org/jira/browse/FLINK-29355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-29355: Fix Version/s: 1.20.0 > Sql parse failed because of Desc catalog.database.table is incorrectly parsed > to desc catalog > -- > > Key: FLINK-29355 > URL: https://issues.apache.org/jira/browse/FLINK-29355 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.17.0 >Reporter: Yunhong Zheng >Priority: Major > Fix For: 1.19.0, 1.20.0 > > > If user names the CATALOG he uses as ‘catalog’, and he tries to desc table > using syntax 'describe catalog.testDatabase.testTable'. This statement will > be incorrectly parsed to 'DESC CATALOG' instead of 'DESC TABLE' . > !image-2022-09-20-20-00-19-478.png|width=592,height=187! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-22418) The currently activated tab in the same page isn't consistent
[ https://issues.apache.org/jira/browse/FLINK-22418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-22418: Fix Version/s: 1.20.0 > The currently activated tab in the same page isn't consistent > - > > Key: FLINK-22418 > URL: https://issues.apache.org/jira/browse/FLINK-22418 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.13.0 >Reporter: Dian Fu >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.19.0, 1.20.0 > > Attachments: image-2021-04-23-10-41-06-577.png > > > Currently, the activated tab isn't always the same in the same page after > click "Java/Scala/Python" a couple of times: > !image-2021-04-23-10-41-06-577.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-19034) Remove deprecated StreamExecutionEnvironment#set/getNumberOfExecutionRetries
[
https://issues.apache.org/jira/browse/FLINK-19034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-19034:
Fix Version/s: 1.20.0
> Remove deprecated StreamExecutionEnvironment#set/getNumberOfExecutionRetries
>
>
> Key: FLINK-19034
> URL: https://issues.apache.org/jira/browse/FLINK-19034
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataStream
>Reporter: Dawid Wysakowicz
>Assignee: Daisy Tsang
>Priority: Major
> Labels: auto-unassigned, stale-assigned
> Fix For: 1.19.0, 1.20.0
>
>
> Remove deprecated
> {code}
> StreamExecutionEnvironment#setNumberOfExecutionRetries/getNumberOfExecutionRetries
> {code}
> The corresponding settings in {{ExecutionConfig}} will be removed in a
> separate issue, as they are {{Public}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30959) Improve the documentation of UNIX_TIMESTAMP for different argument format
[
https://issues.apache.org/jira/browse/FLINK-30959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-30959:
Fix Version/s: 1.20.0
> Improve the documentation of UNIX_TIMESTAMP for different argument format
> -
>
> Key: FLINK-30959
> URL: https://issues.apache.org/jira/browse/FLINK-30959
> Project: Flink
> Issue Type: Improvement
> Components: Documentation, Table SQL / API
>Affects Versions: 1.16.1
>Reporter: Yunfeng Zhou
>Assignee: Hang Ruan
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.19.0, 1.20.0
>
>
> When running the following pyflink program
>
> {code:python}
> import pandas as pd
> from pyflink.datastream import StreamExecutionEnvironment, HashMapStateBackend
> from pyflink.table import StreamTableEnvironment
> if __name__ == "__main__":
> input_data = pd.DataFrame(
> [
> ["Alex", 100.0, "2022-01-01 08:00:00.001 +0800"],
> ["Emma", 400.0, "2022-01-01 00:00:00.003 +"],
> ["Alex", 200.0, "2022-01-01 08:00:00.005 +0800"],
> ["Emma", 300.0, "2022-01-01 00:00:00.007 +"],
> ["Jack", 500.0, "2022-01-01 08:00:00.009 +0800"],
> ["Alex", 450.0, "2022-01-01 00:00:00.011 +"],
> ],
> columns=["name", "avg_cost", "time"],
> )
> env = StreamExecutionEnvironment.get_execution_environment()
> env.set_state_backend(HashMapStateBackend())
> t_env = StreamTableEnvironment.create(env)
> input_table = t_env.from_pandas(input_data)
> t_env.create_temporary_view("input_table", input_table)
> time_format = "-MM-dd HH:mm:ss.SSS X"
> output_table = t_env.sql_query(
> f"SELECT *, UNIX_TIMESTAMP(`time`, '{time_format}') AS unix_time FROM
> input_table"
> )
> output_table.execute().print()
> {code}
> The actual output is
> {code}
> +++++--+
> | op | name | avg_cost |
> time |unix_time |
> +++++--+
> | +I | Alex | 100.0 |
> 2022-01-01 08:00:00.001 +0800 | 1640995200 |
> | +I | Emma | 400.0 |
> 2022-01-01 00:00:00.003 + | 1640995200 |
> | +I | Alex | 200.0 |
> 2022-01-01 08:00:00.005 +0800 | 1640995200 |
> | +I | Emma | 300.0 |
> 2022-01-01 00:00:00.007 + | 1640995200 |
> | +I | Jack | 500.0 |
> 2022-01-01 08:00:00.009 +0800 | 1640995200 |
> | +I | Alex | 450.0 |
> 2022-01-01 00:00:00.011 + | 1640995200 |
> +++++--+
> {code}
> While the expected result is
> {code:java}
> +++++--+
> | op | name | avg_cost |
> time |unix_time |
> +++++--+
> | +I | Alex | 100.0 |
> 2022-01-01 08:00:00.001 +0800 | 1640995200 |
> | +I | Emma | 400.0 |
> 2022-01-01 00:00:00.003 + | 1640966400 |
> | +I | Alex | 200.0 |
> 2022-01-01 08:00:00.005 +0800 | 1640995200 |
> | +I | Emma | 300.0 |
> 2022-01-01 00:00:00.007 + | 1640966400 |
> | +I | Jack | 500.0 |
> 2022-01-01 08:00:00.009 +0800 | 1640995200 |
> | +I | Alex | 450.0 |
> 2022-01-01 00:00:00.011 + | 1640966400 |
> +++++--+
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30292) Better support for conversion between DataType and TypeInformation
[ https://issues.apache.org/jira/browse/FLINK-30292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-30292: Fix Version/s: 1.20.0 > Better support for conversion between DataType and TypeInformation > -- > > Key: FLINK-30292 > URL: https://issues.apache.org/jira/browse/FLINK-30292 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Affects Versions: 1.15.3 >Reporter: Yunfeng Zhou >Priority: Major > Fix For: 1.19.0, 1.20.0 > > > In Flink 1.15, we have the following ways to convert a DataType to a > TypeInformation. Each of them has some disadvantages. > * `TypeConversions.fromDataTypeToLegacyInfo` > It might lead to precision losses in face of some data types like timestamp. > It has been deprecated. > * `ExternalTypeInfo.of` > It cannot be used to get detailed type information like `RowTypeInfo` > It might bring some serialization overhead. > Given that the ways mentioned above are both not perfect, Flink SQL should > provide a better API to support DataType-TypeInformation conversions, and > thus better support Table-DataStream conversions. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-19432) Whether to capture the updates which don't change any monitored columns
[ https://issues.apache.org/jira/browse/FLINK-19432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-19432: Fix Version/s: 1.20.0 > Whether to capture the updates which don't change any monitored columns > --- > > Key: FLINK-19432 > URL: https://issues.apache.org/jira/browse/FLINK-19432 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.11.1 >Reporter: shizhengchao >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.19.0, 1.20.0 > > > with `debezium-json` and `canal-json`: > Whether to capture the updates which don't change any monitored columns. This > may happen if the monitored columns (columns defined in Flink SQL DDL) is a > subset of the columns in database table. We can provide an optional option， > default 'true', which means all the updates will be captured. You can set to > 'false' to only capture changed updates -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30970) Fix SQL Client doc still uses old style output
[ https://issues.apache.org/jira/browse/FLINK-30970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-30970: Fix Version/s: 1.20.0 > Fix SQL Client doc still uses old style output > -- > > Key: FLINK-30970 > URL: https://issues.apache.org/jira/browse/FLINK-30970 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.17.0 >Reporter: Shengkai Fang >Assignee: Shengkai Fang >Priority: Major > Fix For: 1.19.0, 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-17784) Better detection for parquet and orc in hive
[ https://issues.apache.org/jira/browse/FLINK-17784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-17784: Fix Version/s: 1.20.0 > Better detection for parquet and orc in hive > > > Key: FLINK-17784 > URL: https://issues.apache.org/jira/browse/FLINK-17784 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Reporter: Jingsong Lee >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.19.0, 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-26129) Using fail method instead of kill method when yarn application deployment error occures
[ https://issues.apache.org/jira/browse/FLINK-26129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-26129: Fix Version/s: 1.20.0 > Using fail method instead of kill method when yarn application deployment > error occures > --- > > Key: FLINK-26129 > URL: https://issues.apache.org/jira/browse/FLINK-26129 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.14.3 >Reporter: zlzhang0122 >Priority: Major > Fix For: 1.19.0, 1.20.0 > > > When deploy an application using yarn, it add a DeploymentFailureHook and if > the deployment duration exceed the applicationSubmitTimeout(this situation > always occures when the resource is not sufficient,etc.). > Now it uses yarnClient.killApplication() to kill fail this deployment, I > think maybe we should using yarnClient.failApplicationAttempt() to fail the > deployment. IMO, using fail method instead of kill method can trigger the > retry action and is more reasonable. > The change is very simple and only affect the yarn application deployment. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-20853) Add reader schema null check for AvroDeserializationSchema when recordClazz is GenericRecord
[ https://issues.apache.org/jira/browse/FLINK-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-20853: Fix Version/s: 1.20.0 > Add reader schema null check for AvroDeserializationSchema when recordClazz > is GenericRecord > - > > Key: FLINK-20853 > URL: https://issues.apache.org/jira/browse/FLINK-20853 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.12.0 >Reporter: hailong wang >Priority: Minor > Fix For: 1.19.0, 1.20.0 > > > Reader schema can not be null when recordClazz is GenericRecord. > Although its constructor is default, this will cause NPE when reader schema > is null and recordClazz is GenericRecord for the class extends it, such as > RegistryAvroDeserializationSchema. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-23860) Conversion to relational algebra failed to preserve datatypes
[
https://issues.apache.org/jira/browse/FLINK-23860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-23860:
Fix Version/s: 1.20.0
> Conversion to relational algebra failed to preserve datatypes
> -
>
> Key: FLINK-23860
> URL: https://issues.apache.org/jira/browse/FLINK-23860
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.13.1, 1.13.2
>Reporter: lixu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0, 1.20.0
>
>
> {code:java}
> //代码占位符
> StreamExecutionEnvironment streamExecutionEnvironment =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnvironment =
> StreamTableEnvironment.create(streamExecutionEnvironment);
> tableEnvironment.executeSql("CREATE TABLE datagen (\n" +
> " f_sequence INT,\n" +
> " f_random INT,\n" +
> " f_random_str STRING,\n" +
> " ts AS localtimestamp,\n" +
> " WATERMARK FOR ts AS ts\n" +
> ") WITH (\n" +
> " 'connector' = 'datagen',\n" +
> " 'rows-per-second'='5',\n" +
> " 'fields.f_sequence.kind'='sequence',\n" +
> " 'fields.f_sequence.start'='1',\n" +
> " 'fields.f_sequence.end'='1000',\n" +
> " 'fields.f_random.min'='1',\n" +
> " 'fields.f_random.max'='1000',\n" +
> " 'fields.f_random_str.length'='10'\n" +
> ")");
> Table table = tableEnvironment.sqlQuery("select row(f_sequence, f_random) as
> c from datagen");
> Table table1 = tableEnvironment.sqlQuery("select * from " + table);
> table1.execute().print();
> {code}
> {code:java}
> // exception
> Exception in thread "main" java.lang.AssertionError: Conversion to relational
> algebra failed to preserve datatypes:Exception in thread "main"
> java.lang.AssertionError: Conversion to relational algebra failed to preserve
> datatypes:validated type:RecordType(RecordType:peek_no_expand(INTEGER EXPR$0,
> INTEGER EXPR$1) NOT NULL c) NOT NULLconverted
> type:RecordType(RecordType(INTEGER EXPR$0, INTEGER EXPR$1) NOT NULL c) NOT
> NULLrel:LogicalProject(c=[ROW($0, $1)])
> LogicalWatermarkAssigner(rowtime=[ts], watermark=[$3])
> LogicalProject(f_sequence=[$0], f_random=[$1], f_random_str=[$2],
> ts=[LOCALTIMESTAMP]) LogicalTableScan(table=[[default_catalog,
> default_database, datagen]]) at
> org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:467)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:582)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:169)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:161)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:989)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:958)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:283)
> at
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:101)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:704)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-25345) FLIP-196: Source API stability guarantees
[ https://issues.apache.org/jira/browse/FLINK-25345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-25345: Fix Version/s: 1.20.0 > FLIP-196: Source API stability guarantees > - > > Key: FLINK-25345 > URL: https://issues.apache.org/jira/browse/FLINK-25345 > Project: Flink > Issue Type: Improvement > Components: API / DataStream, Documentation >Affects Versions: 1.15.0 >Reporter: Till Rohrmann >Priority: Major > Fix For: 1.19.0, 1.20.0 > > > This ticket is an umbrella ticket for the work on > [FLIP-196|https://cwiki.apache.org/confluence/x/IJeqCw] which will properly > document and guard the agreed upon source API stability guarantees. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-17535) Treat min/max as part of the hierarchy of config option
[ https://issues.apache.org/jira/browse/FLINK-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-17535: Fix Version/s: 1.20.0 > Treat min/max as part of the hierarchy of config option > --- > > Key: FLINK-17535 > URL: https://issues.apache.org/jira/browse/FLINK-17535 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration >Reporter: Yangze Guo >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.19.0, 1.20.0 > > > As discussed in > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Should-max-min-be-part-of-the-hierarchy-of-config-option-td40578.html. > We decide to treat min/max as part of the hierarchy of config option. This > ticket is an umbrella of all tasks related to it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-24226) Improve error message when trying Presto S3 FS with recoverable writer
[
https://issues.apache.org/jira/browse/FLINK-24226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-24226:
Fix Version/s: 1.20.0
> Improve error message when trying Presto S3 FS with recoverable writer
> --
>
> Key: FLINK-24226
> URL: https://issues.apache.org/jira/browse/FLINK-24226
> Project: Flink
> Issue Type: Technical Debt
> Components: FileSystems
>Reporter: Chesnay Schepler
>Priority: Major
> Fix For: 1.19.0, 1.20.0
>
>
> When you use the Presto S3 FileSystem with recoverable writers you get this
> exception:
> {code}
> java.lang.UnsupportedOperationException: This s3 file system implementation
> does not support recoverable writers.
> at
> org.apache.flink.fs.s3.common.FlinkS3FileSystem.createRecoverableWriter(FlinkS3FileSystem.java:136)
> at
> org.apache.flink.core.fs.PluginFileSystemFactory$ClassLoaderFixingFileSystem.createRecoverableWriter(PluginFileSystemFactory.java:134)
> at
> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69)
> at
> org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:260)
> at
> org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:396)
> {code}
> While this is technically _correct_ it's not really helpful to a user. It
> does not propose a course of action (e.g., "Use the Hadoop S3 FileSystem
> instead"), which is particularly important because this limitation isn't
> documented.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31682) map_from_arrays should take whether allow duplicate keys and null key into consideration
[ https://issues.apache.org/jira/browse/FLINK-31682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-31682: Fix Version/s: 1.20.0 > map_from_arrays should take whether allow duplicate keys and null key into > consideration > > > Key: FLINK-31682 > URL: https://issues.apache.org/jira/browse/FLINK-31682 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.18.0 >Reporter: Jacky Lau >Priority: Major > Fix For: 1.19.0, 1.20.0 > > > after research the spark/presto/maxcompute about > map_from_arrays/map_from_entries, there all support duplicate keys and null > key for the most part > > spark https://github.com/apache/spark/pull/21258/files > maxcompute > [https://www.alibabacloud.com/help/en/maxcompute/latest/complex-type-functions#section-7ue-e91-m0s] > presto https://prestodb.io/docs/current/functions/map.html -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29646) SQL Gateway should return a simpler error message
[
https://issues.apache.org/jira/browse/FLINK-29646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-29646:
Fix Version/s: 1.20.0
> SQL Gateway should return a simpler error message
> -
>
> Key: FLINK-29646
> URL: https://issues.apache.org/jira/browse/FLINK-29646
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Gateway
>Reporter: yuanfenghu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0, 1.20.0
>
>
> sql gateway should return simpler exception information
> for example:
> If i execute a sql statement through sql gateway but my statement has
> syntax error ：[ inset into tablea select * from tableb ]
> When I get exception information. The abnormal information returned by the
> server is too redundant to quickly find the Key Information.
> {code:java}
> org.apache.flink.table.gateway.api.utils.SqlGatewayException:
> org.apache.flink.table.gateway.api.utils.SqlGatewayException: Failed to
> fetchResults.
> at
> org.apache.flink.table.gateway.rest.handler.statement.FetchResultsHandler.handleRequest(FetchResultsHandler.java:77)
> at
> org.apache.flink.table.gateway.rest.handler.AbstractSqlGatewayRestHandler.respondToRequest(AbstractSqlGatewayRestHandler.java:84)
> at
> org.apache.flink.table.gateway.rest.handler.AbstractSqlGatewayRestHandler.respondToRequest(AbstractSqlGatewayRestHandler.java:52)
> at
> org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:196)
> at
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83)
> at java.util.Optional.ifPresent(Optional.java:159)
> at
> org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45)
> at
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80)
> at
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115)
> at
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94)
> at
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:210)
> at
> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:69)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.Abstract
[jira] [Updated] (FLINK-12692) Support disk spilling in HeapKeyedStateBackend
[
https://issues.apache.org/jira/browse/FLINK-12692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-12692:
Fix Version/s: 1.20.0
> Support disk spilling in HeapKeyedStateBackend
> --
>
> Key: FLINK-12692
> URL: https://issues.apache.org/jira/browse/FLINK-12692
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / State Backends
>Reporter: Yu Li
>Priority: Major
> Labels: auto-unassigned, pull-request-available
> Fix For: 1.19.0, 1.20.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{HeapKeyedStateBackend}} is one of the two {{KeyedStateBackends}} in Flink,
> since state lives as Java objects on the heap and the de/serialization only
> happens during state snapshot and restore, it outperforms
> {{RocksDBKeyedStateBackend}} when all data could reside in memory.
> However, along with the advantage, {{HeapKeyedStateBackend}} also has its
> shortcomings, and the most painful one is the difficulty to estimate the
> maximum heap size (Xmx) to set, and we will suffer from GC impact once the
> heap memory is not enough to hold all state data. There’re several
> (inevitable) causes for such scenario, including (but not limited to):
> * Memory overhead of Java object representation (tens of times of the
> serialized data size).
> * Data flood caused by burst traffic.
> * Data accumulation caused by source malfunction.
> To resolve this problem, we propose a solution to support spilling state data
> to disk before heap memory is exhausted. We will monitor the heap usage and
> choose the coldest data to spill, and reload them when heap memory is
> regained after data removing or TTL expiration, automatically.
> More details please refer to the design doc and mailing list discussion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28901) CassandraSinkBaseTest.testTimeoutExceptionOnInvoke failed with TestTimedOutException
[
https://issues.apache.org/jira/browse/FLINK-28901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-28901:
Fix Version/s: 1.20.0
> CassandraSinkBaseTest.testTimeoutExceptionOnInvoke failed with
> TestTimedOutException
>
>
> Key: FLINK-28901
> URL: https://issues.apache.org/jira/browse/FLINK-28901
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Cassandra
>Affects Versions: 1.16.0
>Reporter: Xingbo Huang
>Priority: Major
> Labels: test-stability
> Fix For: 1.19.0, 1.20.0
>
>
> {code:java}
> 2022-08-10T03:39:22.6587394Z Aug 10 03:39:22 [ERROR]
> org.apache.flink.streaming.connectors.cassandra.CassandraSinkBaseTest.testTimeoutExceptionOnInvoke
> Time elapsed: 5.113 s <<< ERROR!
> 2022-08-10T03:39:22.6588579Z Aug 10 03:39:22
> org.junit.runners.model.TestTimedOutException: test timed out after 5000
> milliseconds
> 2022-08-10T03:39:22.6589463Z Aug 10 03:39:22 at
> java.util.zip.ZipFile.read(Native Method)
> 2022-08-10T03:39:22.6590286Z Aug 10 03:39:22 at
> java.util.zip.ZipFile.access$1400(ZipFile.java:60)
> 2022-08-10T03:39:22.6591287Z Aug 10 03:39:22 at
> java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:734)
> 2022-08-10T03:39:22.6592323Z Aug 10 03:39:22 at
> java.util.zip.ZipFile$ZipFileInflaterInputStream.fill(ZipFile.java:434)
> 2022-08-10T03:39:22.6593673Z Aug 10 03:39:22 at
> java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
> 2022-08-10T03:39:22.6594638Z Aug 10 03:39:22 at
> sun.misc.Resource.getBytes(Resource.java:124)
> 2022-08-10T03:39:22.6595535Z Aug 10 03:39:22 at
> java.net.URLClassLoader.defineClass(URLClassLoader.java:463)
> 2022-08-10T03:39:22.6596506Z Aug 10 03:39:22 at
> java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> 2022-08-10T03:39:22.6597477Z Aug 10 03:39:22 at
> java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> 2022-08-10T03:39:22.6598393Z Aug 10 03:39:22 at
> java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> 2022-08-10T03:39:22.6599286Z Aug 10 03:39:22 at
> java.security.AccessController.doPrivileged(Native Method)
> 2022-08-10T03:39:22.6600209Z Aug 10 03:39:22 at
> java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> 2022-08-10T03:39:22.6601141Z Aug 10 03:39:22 at
> java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> 2022-08-10T03:39:22.6602070Z Aug 10 03:39:22 at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> 2022-08-10T03:39:22.6603090Z Aug 10 03:39:22 at
> java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> 2022-08-10T03:39:22.6604199Z Aug 10 03:39:22 at
> net.bytebuddy.NamingStrategy$SuffixingRandom.(NamingStrategy.java:153)
> 2022-08-10T03:39:22.6605188Z Aug 10 03:39:22 at
> net.bytebuddy.ByteBuddy.(ByteBuddy.java:196)
> 2022-08-10T03:39:22.6606063Z Aug 10 03:39:22 at
> net.bytebuddy.ByteBuddy.(ByteBuddy.java:187)
> 2022-08-10T03:39:22.6607141Z Aug 10 03:39:22 at
> org.mockito.internal.creation.bytebuddy.InlineBytecodeGenerator.(InlineBytecodeGenerator.java:81)
> 2022-08-10T03:39:22.6608423Z Aug 10 03:39:22 at
> org.mockito.internal.creation.bytebuddy.InlineByteBuddyMockMaker.(InlineByteBuddyMockMaker.java:224)
> 2022-08-10T03:39:22.6609844Z Aug 10 03:39:22 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 2022-08-10T03:39:22.6610990Z Aug 10 03:39:22 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> 2022-08-10T03:39:22.6612246Z Aug 10 03:39:22 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 2022-08-10T03:39:22.6613551Z Aug 10 03:39:22 at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> 2022-08-10T03:39:22.6614482Z Aug 10 03:39:22 at
> java.lang.Class.newInstance(Class.java:442)
> 2022-08-10T03:39:22.6615527Z Aug 10 03:39:22 at
> org.mockito.internal.configuration.plugins.PluginInitializer.loadImpl(PluginInitializer.java:50)
> 2022-08-10T03:39:22.6616718Z Aug 10 03:39:22 at
> org.mockito.internal.configuration.plugins.PluginLoader.loadPlugin(PluginLoader.java:63)
> 2022-08-10T03:39:22.6617900Z Aug 10 03:39:22 at
> org.mockito.internal.configuration.plugins.PluginLoader.loadPlugin(PluginLoader.java:48)
> 2022-08-10T03:39:22.6619079Z Aug 10 03:39:22 at
> org.mockito.internal.configuration.plugins.PluginRegistry.(PluginRegistry.java:23)
> 2022-08-10T03:39:22.6620206Z Aug 10 03:39:22 at
> org.mockito.internal.configuration.plugins.Plugins.(Plugins.java:19)
> 2022-08-10T03:39:22.6621435Z Aug 10 03:39:22 at
> org.mockito.internal.util.MockUtil.(MockUtil.java:24)
> 2022-08-10T03:39:22.6622523Z Aug 10 03:39:22 at
> org.mockito.internal.util.MockCreationValidator.validateType(
[jira] [Updated] (FLINK-15532) Enable strict capacity limit for memory usage for RocksDB
[ https://issues.apache.org/jira/browse/FLINK-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-15532: Fix Version/s: 1.20.0 > Enable strict capacity limit for memory usage for RocksDB > - > > Key: FLINK-15532 > URL: https://issues.apache.org/jira/browse/FLINK-15532 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.19.0, 1.20.0 > > Attachments: image-2020-10-23-14-39-45-997.png, > image-2020-10-23-14-41-10-584.png, image-2020-10-23-14-43-18-739.png, > image-2020-10-23-14-55-08-120.png > > > Currently, due to the limitation of RocksDB (see > [issue-6247|https://github.com/facebook/rocksdb/issues/6247]), we cannot > create a strict-capacity-limit LRUCache which shared among rocksDB > instance(s). > This issue tracks this problem and offer the ability of strict mode once we > could enable this feature. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32535) CheckpointingStatisticsHandler periodically returns NullArgumentException after job restarts
[
https://issues.apache.org/jira/browse/FLINK-32535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-32535:
Fix Version/s: 1.20.0
> CheckpointingStatisticsHandler periodically returns NullArgumentException
> after job restarts
>
>
> Key: FLINK-32535
> URL: https://issues.apache.org/jira/browse/FLINK-32535
> Project: Flink
> Issue Type: Bug
> Components: Runtime / REST
>Affects Versions: 1.16.2, 1.17.1
>Reporter: Hong Liang Teoh
>Priority: Major
> Fix For: 1.19.0, 1.20.0
>
>
> *What*
> When making requests to /checkpoints REST API after a job restart, we see 500
> for a short period of time. We should handle this gracefully in the
> CheckpointingStatisticsHandler.
>
> *How to replicate*
> * Checkpointing interval 1s
> * Job is constantly restarting
> * Make constant requests to /checkpoints REST API.
> See [here|https://github.com/apache/flink/pull/22901#issuecomment-1617830035]
> for more info
>
> Stack trace:
> {{org.apache.commons.math3.exception.NullArgumentException: input array}}
> {{ at
> org.apache.commons.math3.util.MathArrays.verifyValues(MathArrays.java:1753)}}
> {{ at
> org.apache.commons.math3.stat.descriptive.AbstractUnivariateStatistic.test(AbstractUnivariateStatistic.java:158)}}
> {{ at
> org.apache.commons.math3.stat.descriptive.rank.Percentile.evaluate(Percentile.java:272)}}
> {{ at
> org.apache.commons.math3.stat.descriptive.rank.Percentile.evaluate(Percentile.java:241)}}
> {{ at
> org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogramStatistics$CommonMetricsSnapshot.getPercentile(DescriptiveStatisticsHistogramStatistics.java:159)}}
> {{ at
> org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogramStatistics.getQuantile(DescriptiveStatisticsHistogramStatistics.java:53)}}
> {{ at
> org.apache.flink.runtime.checkpoint.StatsSummarySnapshot.getQuantile(StatsSummarySnapshot.java:108)}}
> {{ at
> org.apache.flink.runtime.rest.messages.checkpoints.StatsSummaryDto.valueOf(StatsSummaryDto.java:81)}}
> {{ at
> org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.createCheckpointingStatistics(CheckpointingStatisticsHandler.java:133)}}
> {{ at
> org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:85)}}
> {{ at
> org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:59)}}
> {{ at
> org.apache.flink.runtime.rest.handler.job.checkpoints.AbstractCheckpointStatsHandler.lambda$handleRequest$1(AbstractCheckpointStatsHandler.java:62)}}
> {{ at
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)}}
> {{ at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)}}
> {{ at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)}}
> {{ at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)}}
> {{ at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)}}
> {{ at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)}}
> {{ at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)}}
> {{ at java.base/java.lang.Thread.run(Thread.java:829)\n}}
>
> See graphs here for tests. The dips in the green line correspond to the
> failures immediately after a job restart.
> !https://user-images.githubusercontent.com/35062175/250529297-908a6714-ea15-4aac-a7fc-332589da2582.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28097) Remove bundling of formats-common in json/csv format
[ https://issues.apache.org/jira/browse/FLINK-28097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-28097: Fix Version/s: 1.20.0 > Remove bundling of formats-common in json/csv format > > > Key: FLINK-28097 > URL: https://issues.apache.org/jira/browse/FLINK-28097 > Project: Flink > Issue Type: Technical Debt > Components: Build System, Formats (JSON, Avro, Parquet, ORC, > SequenceFile) >Reporter: Chesnay Schepler >Priority: Major > Fix For: 1.19.0, 1.20.0 > > > For some reason the normal json/csv format jars bundle flink-formats-common. > This seems unnecessary; this bundling can either be relegated to the sql-jar > (Which is already done!) or the user-jar packaging. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28398) CheckpointCoordinatorTriggeringTest.discardingTriggeringCheckpointWillExecuteNextCheckpointRequest( gets stuck
[
https://issues.apache.org/jira/browse/FLINK-28398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-28398:
Fix Version/s: 1.20.0
> CheckpointCoordinatorTriggeringTest.discardingTriggeringCheckpointWillExecuteNextCheckpointRequest(
> gets stuck
> --
>
> Key: FLINK-28398
> URL: https://issues.apache.org/jira/browse/FLINK-28398
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.0
>Reporter: Martijn Visser
>Priority: Major
> Labels: stale-assigned, test-stability
> Fix For: 1.19.0, 1.20.0
>
>
> {code:java}
> Jul 01 02:16:55 "main" #1 prio=5 os_prio=0 tid=0x7fe41000b800 nid=0x5ca2
> in Object.wait() [0x7fe41a429000]
> Jul 01 02:16:55java.lang.Thread.State: WAITING (on object monitor)
> Jul 01 02:16:55 at java.lang.Object.wait(Native Method)
> Jul 01 02:16:55 at java.lang.Object.wait(Object.java:502)
> Jul 01 02:16:55 at
> org.apache.flink.core.testutils.OneShotLatch.await(OneShotLatch.java:61)
> Jul 01 02:16:55 - locked <0xf096ab58> (a java.lang.Object)
> Jul 01 02:16:55 at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinatorTriggeringTest.discardingTriggeringCheckpointWillExecuteNextCheckpointRequest(CheckpointCoordinatorTriggeringTest.java:731)
> Jul 01 02:16:55 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jul 01 02:16:55 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jul 01 02:16:55 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jul 01 02:16:55 at java.lang.reflect.Method.invoke(Method.java:498)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37433&view=logs&j=d89de3df-4600-5585-dadc-9bbc9a5e661c&t=be5a4b15-4b23-56b1-7582-795f58a645a2&l=15207
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-22371) Chinese grammatical error
[
https://issues.apache.org/jira/browse/FLINK-22371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-22371:
Fix Version/s: 1.20.0
> Chinese grammatical error
> -
>
> Key: FLINK-22371
> URL: https://issues.apache.org/jira/browse/FLINK-22371
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.12.2
>Reporter: simenliuxing
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.19.0, 1.20.0
>
>
> Chinese grammatical error in
> [https://ci.apache.org/projects/flink/flink-docs-release-1.12/zh/dev/table/common.html]
> . text content "下文讨论的 {{DataSet}} API 只与旧计划起有关。" Should be replaced with
> "下文讨论的 {{DataSet}} API 只与旧计划有关。"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-25350) Verify stability guarantees of annotated classes
[ https://issues.apache.org/jira/browse/FLINK-25350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-25350: Fix Version/s: 1.20.0 > Verify stability guarantees of annotated classes > > > Key: FLINK-25350 > URL: https://issues.apache.org/jira/browse/FLINK-25350 > Project: Flink > Issue Type: Sub-task > Components: Tests >Affects Versions: 1.15.0 >Reporter: Till Rohrmann >Priority: Major > Fix For: 1.19.0, 1.20.0 > > > In order to give API stability guarantees we should add a test which ensures > that an API class can at most give a stability guarantee of the weakest > annotated method a user needs to implement for this class. This will ensure > that we can extend API classes with methods that have weaker guarantees but > that must have default implementations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31267) Fine-Grained Resource Management supports table and sql levels
[ https://issues.apache.org/jira/browse/FLINK-31267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-31267: Fix Version/s: 1.20.0 > Fine-Grained Resource Management supports table and sql levels > -- > > Key: FLINK-31267 > URL: https://issues.apache.org/jira/browse/FLINK-31267 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Affects Versions: 1.16.1 >Reporter: waywtdcc >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0, 1.20.0 > > > Fine-Grained Resource Management supports table and sql levels. Now > Fine-Grained Resource can only be used at the datastream api level, and does > not support table and sql level settings. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28104) Drop the unused order parameter in FirstValueFunction/LastValueFunction
[ https://issues.apache.org/jira/browse/FLINK-28104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-28104: Fix Version/s: 1.20.0 > Drop the unused order parameter in FirstValueFunction/LastValueFunction > --- > > Key: FLINK-28104 > URL: https://issues.apache.org/jira/browse/FLINK-28104 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API, Table SQL / Planner >Reporter: luoyuxia >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0, 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-20091) Introduce avro.ignore-parse-errors for AvroRowDataDeserializationSchema
[ https://issues.apache.org/jira/browse/FLINK-20091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-20091: Fix Version/s: 1.20.0 > Introduce avro.ignore-parse-errors for AvroRowDataDeserializationSchema > > > Key: FLINK-20091 > URL: https://issues.apache.org/jira/browse/FLINK-20091 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.12.0 >Reporter: hailong wang >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > pull-request-available > Fix For: 1.19.0, 1.20.0 > > > Introduce avro.ignore-parse-errors to allow users to skip rows with parsing > errors instead of failing when deserializing avro format data. > This is useful when there are dirty data, for without this option, users can > not skip the dirty row. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34315) Temporarily force disabling window join, window rank and window deduplicate optimization when using session window tvf
[ https://issues.apache.org/jira/browse/FLINK-34315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-34315: Fix Version/s: 1.20.0 > Temporarily force disabling window join, window rank and window deduplicate > optimization when using session window tvf > -- > > Key: FLINK-34315 > URL: https://issues.apache.org/jira/browse/FLINK-34315 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.19.0 >Reporter: xuyang >Priority: Major > Fix For: 1.19.0, 1.20.0 > > > Currently session window tvf is first introduced after > https://issues.apache.org/jira/browse/FLINK-24024 . However after > https://issues.apache.org/jira/browse/FLINK-34100 the session window tvf node > can exist independently of window aggregation, but it is not ready for window > join, window rank and window deduplicate. So we need to temporarily disable > window join, window rank and window deduplicate optimization when using > session window tvf. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-18981) Support column comment for Hive tables
[ https://issues.apache.org/jira/browse/FLINK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-18981: Fix Version/s: 1.20.0 > Support column comment for Hive tables > -- > > Key: FLINK-18981 > URL: https://issues.apache.org/jira/browse/FLINK-18981 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Reporter: Rui Li >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.19.0, 1.20.0 > > > Start working on this once FLINK-18958 is done -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-24094) Support DROP PARTITION by partial partition specification
[
https://issues.apache.org/jira/browse/FLINK-24094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-24094:
Fix Version/s: 1.20.0
> Support DROP PARTITION by partial partition specification
> -
>
> Key: FLINK-24094
> URL: https://issues.apache.org/jira/browse/FLINK-24094
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: lixu
>Assignee: luoyuxia
>Priority: Major
> Fix For: 1.19.0, 1.20.0
>
>
> [https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/hive/hive_dialect/#drop-partitions]
> {code:java}
> //代码占位符
> Caused by:
> org.apache.flink.table.catalog.exceptions.PartitionSpecInvalidException:
> PartitionSpec CatalogPartitionSpec{{dt=2021-08-31}} does not match partition
> keys [dt, xtlx, sblx] of table test_flink.test_partition in catalog
> check_rule_base_hive_catalog.at
> org.apache.flink.table.catalog.hive.HiveCatalog.getOrderedFullPartitionValues(HiveCatalog.java:1189)
> ~[flink-sql-connector-hive-2.3.6_2.11-1.13.1.jar:1.13.1]at
> org.apache.flink.table.catalog.hive.HiveCatalog.dropPartition(HiveCatalog.java:899)
> ~[flink-sql-connector-hive-2.3.6_2.11-1.13.1.jar:1.13.1] at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:982)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:730)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-24376) Operator name in OperatorCoordinator should not use chained name
[
https://issues.apache.org/jira/browse/FLINK-24376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-24376:
Fix Version/s: 1.20.0
> Operator name in OperatorCoordinator should not use chained name
>
>
> Key: FLINK-24376
> URL: https://issues.apache.org/jira/browse/FLINK-24376
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.14.0, 1.12.5, 1.13.2
>Reporter: Qingsheng Ren
>Assignee: Qingsheng Ren
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.19.0, 1.20.0
>
>
> Currently the operator name passed to
> {{CoordinatedOperatorFactory#getCoordinatorProvider}} is a chained operator
> name (e.g. Source -> Map) instead of the name of coordinating operator, which
> might be misleading.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32607) Kafka table source and json format support projection pushdown
[ https://issues.apache.org/jira/browse/FLINK-32607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-32607: Fix Version/s: 1.20.0 > Kafka table source and json format support projection pushdown > -- > > Key: FLINK-32607 > URL: https://issues.apache.org/jira/browse/FLINK-32607 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka, Table SQL / Planner >Affects Versions: 1.18.0 >Reporter: dalongliu >Priority: Major > Fix For: kafka-4.0.0, 1.19.0, 1.20.0 > > > ProjectionPushDown has a huge performance impact and is not currently > implemented in Kafka Source, so we can support it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34264) Prometheuspushgateway's hosturl can use basicAuth certification login
[ https://issues.apache.org/jira/browse/FLINK-34264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-34264: Fix Version/s: 1.20.0 > Prometheuspushgateway's hosturl can use basicAuth certification login > - > > Key: FLINK-34264 > URL: https://issues.apache.org/jira/browse/FLINK-34264 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Affects Versions: 1.19.0 >Reporter: Yangzhou Huang >Priority: Minor > Labels: pull-request-available > Fix For: 1.19.0, 1.20.0 > > > The scene is as follows: > Pushgateway uses Basicauth to verify, while the current code does not > implement permissions on Basicauth. > hostUrl eg: > [https://username:password@localhost:9091|https://username:password@localhost:9091/] > If this proposal is approved, I will propose a PR improvement. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31249) Checkpoint timeout mechanism fails when finalizeCheckpoint is stuck
[
https://issues.apache.org/jira/browse/FLINK-31249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-31249:
Fix Version/s: 1.20.0
> Checkpoint timeout mechanism fails when finalizeCheckpoint is stuck
> ---
>
> Key: FLINK-31249
> URL: https://issues.apache.org/jira/browse/FLINK-31249
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.11.6, 1.16.0
>Reporter: Renxiang Zhou
>Priority: Major
> Fix For: 1.19.0, 1.20.0
>
> Attachments: image-2023-02-28-11-25-03-637.png,
> image-2023-02-28-12-04-35-178.png, image-2023-02-28-12-17-19-607.png
>
>
> When jobmanager receives all ACKs of tasks, it will finalize the pending
> checkpoint to a completed checkpoint. Currently JM finalizes the pending
> checkpoint with holding the checkpoint coordinator lock.
> When a DFS failure occurs, the {{jobmanager-future}} thread may be blocked at
> finalizing the pending checkpoint.
> !image-2023-02-28-12-17-19-607.png|width=1010,height=244!
> And then the next checkpoint is triggered, the {{Checkpoint Timer}} thread
> waits for the lock to be released.
> !image-2023-02-28-11-25-03-637.png|width=1144,height=248!
> If the previous checkpoint times out, the {{Checkpoint Timer}} will not
> execute the timeout event since it is blocked at waiting for the lock. As a
> result, the previous checkpoint cannot be cancelled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-20952) Changelog json formats should support inherit options from JSON format
[ https://issues.apache.org/jira/browse/FLINK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-20952: Fix Version/s: 1.20.0 > Changelog json formats should support inherit options from JSON format > -- > > Key: FLINK-20952 > URL: https://issues.apache.org/jira/browse/FLINK-20952 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Jark Wu >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > pull-request-available > Fix For: 1.19.0, 1.20.0 > > > Recently, we introduced several config options for json format, e.g. > FLINK-20861. It reveals a potential problem that adding a small config option > into json may need touch debezium-json, canal-json, maxwell-json formats. > This is verbose and error-prone. We need an abstract machanism support > reuable options. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29656) Modify all join-related sql tests to make having statistics as the default choice
[ https://issues.apache.org/jira/browse/FLINK-29656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-29656: Fix Version/s: 1.20.0 > Modify all join-related sql tests to make having statistics as the default > choice > - > > Key: FLINK-29656 > URL: https://issues.apache.org/jira/browse/FLINK-29656 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.17.0 >Reporter: Yunhong Zheng >Priority: Major > Fix For: 1.19.0, 1.20.0 > > > Modify all join-related sql tests to make having statistics as the default > choice. This issue is related to FLINK-29559 to make tests stable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-20190) A New Window Trigger that can trigger window operation both by event time interval、event count for DataStream API
[
https://issues.apache.org/jira/browse/FLINK-20190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-20190:
Fix Version/s: 1.20.0
> A New Window Trigger that can trigger window operation both by event time
> interval、event count for DataStream API
> -
>
> Key: FLINK-20190
> URL: https://issues.apache.org/jira/browse/FLINK-20190
> Project: Flink
> Issue Type: New Feature
> Components: API / DataStream
>Reporter: GaryGao
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.19.0, 1.20.0
>
>
> In production environment, when we are do some window operation, such as
> window aggregation, using data stream api, developers are always asked to not
> only trigger the window operation when the watermark pass the max timestamp
> of window, but also trigger it both by fixed event time interval and fixed
> count of event.The reason why we want to do this is we are looking forward to
> get the frequently updated window operation result, other than waiting for a
> long time until the watermark pass the max timestamp of window.This is very
> useful in reporting and other BI applications.
> For now the default triggers provided by flink can not close this
> requirement, so I developed a New Trigger, so called
> CountAndContinuousEventTimeTrigger, combine ContinuousEventTimeTrigger with
> CountTrigger to do the above thing.
>
> To use CountAndContinuousEventTimeTrigger, you should specify two parameters
> as revealed in it constructor:
> {code:java}
> private CountAndContinuousEventTimeTrigger(Time interval, long
> maxCount);{code}
> * Time interval, it means this trigger will continuously fires based on a
> given time interval, the same as ContinuousEventTimeTrigger.
> * long maxCount, it means this trigger will fires once the count of elements
> in a pane reaches the given count, the same as CountTrigger.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29459) Sink v2 has bugs in supporting legacy v1 implementations with global committer
[ https://issues.apache.org/jira/browse/FLINK-29459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-29459: Fix Version/s: 1.20.0 > Sink v2 has bugs in supporting legacy v1 implementations with global committer > -- > > Key: FLINK-29459 > URL: https://issues.apache.org/jira/browse/FLINK-29459 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Affects Versions: 1.16.0, 1.17.0, 1.15.3 >Reporter: Yun Gao >Assignee: Yun Gao >Priority: Major > Fix For: 1.19.0, 1.20.0 > > > Currently when supporting Sink implementation using version 1 interface, > there are issues after restoring from a checkpoint after failover: > # In global committer operator, when restoring SubtaskCommittableManager, > the subtask id is replaced with the one in the current operator. This means > that the id originally is the id of the sender task (0 ~ N - 1), but after > restoring it has to be 0. This would cause Duplication Key exception during > restoring. > # For Committer operator, the subtaskId of CheckpointCommittableManagerImpl > is always restored to 0 after failover for all the subtasks. This makes the > summary sent to the Global Committer is attached with wrong subtask id. > # For Committer operator, the checkpoint id of SubtaskCommittableManager is > always restored to 1 after failover, this make the following committable sent > to the global committer is attached with wrong checkpoint id. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-21613) Parse Compute Column with `IN` expression throws NPE
[
https://issues.apache.org/jira/browse/FLINK-21613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-21613:
Fix Version/s: 1.20.0
> Parse Compute Column with `IN` expression throws NPE
>
>
> Key: FLINK-21613
> URL: https://issues.apache.org/jira/browse/FLINK-21613
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.13.0
>Reporter: Shuo Cheng
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
> Fix For: 1.19.0, 1.20.0
>
>
> Considering the following given sql:
> {code:sql}
> CREATE TABLE MyInputFormatTable (
> `a` INT,
> `b` BIGINT,
> `c` STRING,
> `d` as `c` IN ('Hi', 'Hello')
> ) WITH (
> 'connector' = 'values',
> 'data-id' = '$dataId',
> 'runtime-source' = 'InputFormat'
> )
> {code}
> NPE will be thrown during parsing the sql:
> `select * from MyInputFormatTable`
> It seems it's the commit "[hotfix][table-planner-blink] Simplify SQL
> expression to RexNode conversion" which introduces this problem. This hotfix
> uses a method `SqlToRelConverter#convertExpression` and this method does not
> has any tests and is not used in Calcite anywhere, which is unsafe. Maybe
> reverting the hotfix is a good choice.
> CC [~twalthr]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28764) Support more than 64 distinct aggregate function calls in one aggregate SQL query
[
https://issues.apache.org/jira/browse/FLINK-28764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-28764:
Fix Version/s: 1.20.0
> Support more than 64 distinct aggregate function calls in one aggregate SQL
> query
> -
>
> Key: FLINK-28764
> URL: https://issues.apache.org/jira/browse/FLINK-28764
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.13.6, 1.14.5, 1.15.1
>Reporter: Wei Zhong
>Assignee: Wei Zhong
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.19.0, 1.20.0
>
>
> Currently Flink SQL does not support more than 64 distinct aggregate function
> calls in one aggregate SQL query. We encountered this problem while migrating
> batch jobs from spark to flink. The spark job has 79 distinct aggregate
> function calls in one aggregate SQL query.
> Reproduce code:
> {code:java}
> public class Test64Distinct {
> public static void main(String[] args) {
> TableEnvironment tableEnv =
> TableEnvironment.create(EnvironmentSettings.inBatchMode());
> tableEnv.executeSql("create table datagen_source(id BIGINT, val
> BIGINT) with " +
> "('connector'='datagen', 'number-of-rows'='1000')");
> tableEnv.executeSql("select " +
> "count(distinct val * 1), " +
> "count(distinct val * 2), " +
> "count(distinct val * 3), " +
> "count(distinct val * 4), " +
> "count(distinct val * 5), " +
> "count(distinct val * 6), " +
> "count(distinct val * 7), " +
> "count(distinct val * 8), " +
> "count(distinct val * 9), " +
> "count(distinct val * 10), " +
> "count(distinct val * 11), " +
> "count(distinct val * 12), " +
> "count(distinct val * 13), " +
> "count(distinct val * 14), " +
> "count(distinct val * 15), " +
> "count(distinct val * 16), " +
> "count(distinct val * 17), " +
> "count(distinct val * 18), " +
> "count(distinct val * 19), " +
> "count(distinct val * 20), " +
> "count(distinct val * 21), " +
> "count(distinct val * 22), " +
> "count(distinct val * 23), " +
> "count(distinct val * 24), " +
> "count(distinct val * 25), " +
> "count(distinct val * 26), " +
> "count(distinct val * 27), " +
> "count(distinct val * 28), " +
> "count(distinct val * 29), " +
> "count(distinct val * 30), " +
> "count(distinct val * 31), " +
> "count(distinct val * 32), " +
> "count(distinct val * 33), " +
> "count(distinct val * 34), " +
> "count(distinct val * 35), " +
> "count(distinct val * 36), " +
> "count(distinct val * 37), " +
> "count(distinct val * 38), " +
> "count(distinct val * 39), " +
> "count(distinct val * 40), " +
> "count(distinct val * 41), " +
> "count(distinct val * 42), " +
> "count(distinct val * 43), " +
> "count(distinct val * 44), " +
> "count(distinct val * 45), " +
> "count(distinct val * 46), " +
> "count(distinct val * 47), " +
> "count(distinct val * 48), " +
> "count(distinct val * 49), " +
> "count(distinct val * 50), " +
> "count(distinct val * 51), " +
> "count(distinct val * 52), " +
> "count(distinct val * 53), " +
> "count(distinct val * 54), " +
> "count(distinct val * 55), " +
> "count(distinct val * 56), " +
> "count(distinct val * 57), " +
> "count(distinct val * 58), " +
> "count(distinct val * 59), " +
> "count(distinct val * 60), " +
> "count(distinct val * 61), " +
> "count(distinct val * 62), " +
> "count(distinct val * 63), " +
> "count(distinct val * 64), " +
> "count(distinct val * 65) from datagen_source").print();
> }
> } {code}
> Exception:
> {code:java}
> Exception in thread "main" org.apache.flink.table.api.TableException: Sql
> optimization: Cannot generate a valid execution plan for the given query:
> LogicalSink(table=[*anonymous_collect$1*], fie
[jira] [Updated] (FLINK-26951) Add HASH supported in SQL & Table API
[
https://issues.apache.org/jira/browse/FLINK-26951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-26951:
Fix Version/s: 1.20.0
> Add HASH supported in SQL & Table API
> -
>
> Key: FLINK-26951
> URL: https://issues.apache.org/jira/browse/FLINK-26951
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: dalongliu
>Priority: Major
> Fix For: 1.19.0, 1.20.0
>
>
> Returns a hash value of the arguments.
> Syntax:
> {code:java}
> hash(expr1, ...) {code}
> Arguments:
> * {{{}exprN{}}}: An expression of any type.
> Returns:
> An INTEGER.
> Examples:
> {code:java}
> > SELECT hash('Flink', array(123), 2);
> -1321691492 {code}
> See more:
> * [Spark|https://spark.apache.org/docs/latest/api/sql/index.html#hash]
> * [Hive|https://cwiki.apache.org/confluence/display/hive/languagemanual+udf]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29863) Properly handle NaN/Infinity in OpenAPI spec
[ https://issues.apache.org/jira/browse/FLINK-29863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-29863: Fix Version/s: 1.20.0 > Properly handle NaN/Infinity in OpenAPI spec > > > Key: FLINK-29863 > URL: https://issues.apache.org/jira/browse/FLINK-29863 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Runtime / REST >Affects Versions: 1.15.0 >Reporter: Chesnay Schepler >Priority: Major > Labels: 2.0-related > Fix For: 1.19.0, 1.20.0 > > > Our OpenAPI spec maps all float/double fields to float64, but we at times > also return NaN/infinity which can't be represented as such since the JSON > spec doesn't support it. > One alternative could be to document it as an either type, returning either a > float64 or a string. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28838) Avoid to notify the elementQueue consumer when the fetch result is empty
[
https://issues.apache.org/jira/browse/FLINK-28838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-28838:
Fix Version/s: 1.20.0
> Avoid to notify the elementQueue consumer when the fetch result is empty
>
>
> Key: FLINK-28838
> URL: https://issues.apache.org/jira/browse/FLINK-28838
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Affects Versions: 1.15.0, 1.15.1
>Reporter: WenJun Min
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0, 1.20.0
>
> Attachments: 20220805165441.jpg
>
>
> When using the new source api, I found that if the source has no data, it
> still brings high cpu usage.
> The reason behind this is that it will always return the
> {{RecordsWithSplitIds}} from the {{splitReader.fetch}} in FetchTask and it
> will be added to the elementQueue. It will make the consumer be notified to
> wake up frequently.
> This causes the thread to keep busy to run and wake up, which leads to the
> high sys and user cpu usage.
> I think not all the SplitReader#fetch will block until there is data, if it
> returns immediately when there is no data, then this problem will happen
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29723) Supports hive first_value function by native implementation
[ https://issues.apache.org/jira/browse/FLINK-29723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-29723: Fix Version/s: 1.20.0 > Supports hive first_value function by native implementation > --- > > Key: FLINK-29723 > URL: https://issues.apache.org/jira/browse/FLINK-29723 > Project: Flink > Issue Type: Sub-task >Reporter: dalongliu >Priority: Major > Fix For: 1.19.0, 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27954) JobVertexFlameGraphHandler does not work on standby Dispatcher
[
https://issues.apache.org/jira/browse/FLINK-27954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-27954:
Fix Version/s: 1.20.0
> JobVertexFlameGraphHandler does not work on standby Dispatcher
> --
>
> Key: FLINK-27954
> URL: https://issues.apache.org/jira/browse/FLINK-27954
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Runtime / REST
>Affects Versions: 1.15.0
>Reporter: Chesnay Schepler
>Assignee: Weijie Guo
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.19.0, 1.20.0
>
>
> The {{JobVertexFlameGraphHandler}} relies internally on the
> {{JobVertexThreadInfoTracker}} which calls
> {{ResourceManagerGateway#requestTaskExecutorThreadInfoGateway}} to get a
> gateway for requesting the thread info from the task executors. Since this
> gateway is not serializable it would categorically fail if called from a
> standby dispatcher.
> Instead this should follow the logic of the {{MetricFetcherImpl}}, which
> requests addresses instead and manually connects to the task executors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31691) Add MAP_FROM_ENTRIES supported in SQL & Table API
[
https://issues.apache.org/jira/browse/FLINK-31691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-31691:
Fix Version/s: 1.20.0
> Add MAP_FROM_ENTRIES supported in SQL & Table API
> -
>
> Key: FLINK-31691
> URL: https://issues.apache.org/jira/browse/FLINK-31691
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.18.0
>Reporter: Jacky Lau
>Assignee: Jacky Lau
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.19.0, 1.20.0
>
>
> map_from_entries(map) - Returns a map created from an arrays of row with two
> fields. Note that the number of fields in a row array should be 2 and the key
> of a row array should not be null.
> Syntax:
> map_from_entries(array_of_rows)
> Arguments:
> array_of_rows: an arrays of row with two fields.
> Returns:
> Returns a map created from an arrays of row with two fields. Note that the
> number of fields in a row array should be 2 and the key of a row array should
> not be null.
> Returns null if the argument is null
> {code:sql}
> > SELECT map_from_entries(map[1, 'a', 2, 'b']);
> [(1,"a"),(2,"b")]{code}
> See also
> presto [https://prestodb.io/docs/current/functions/map.html]
> spark https://spark.apache.org/docs/latest/api/sql/index.html#map_from_entries
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28830) new stack udtf doesn't support atomic type
[
https://issues.apache.org/jira/browse/FLINK-28830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-28830:
Fix Version/s: 1.20.0
> new stack udtf doesn't support atomic type
> ---
>
> Key: FLINK-28830
> URL: https://issues.apache.org/jira/browse/FLINK-28830
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0
>Reporter: Jacky Lau
>Priority: Major
> Fix For: 1.19.0, 1.20.0
>
>
> {code:java}
> // code placeholder
> public class GenerateSeriesFunction extends BuiltInTableFunction {
> private static final long serialVersionUID = 1L;
> public GenerateSeriesFunction(SpecializedContext specializedContext) {
> super(BuiltInFunctionDefinitions.GENERATE_SERIES, specializedContext);
> }
> public void eval(long start, long stop) {
> eval(start, stop, 1);
> }
>
> public void eval(long start, long stop, long step) {
> long s = start;
> while (s <= stop) {
> collect(s);
> s += step;
> }
> }
> }
> public static final BuiltInFunctionDefinition GENERATE_SERIES =
> BuiltInFunctionDefinition.newBuilder()
> .name("GENERATE_SERIES")
> .kind(TABLE)
> .inputTypeStrategy(
> or(
> sequence(
> logical(LogicalTypeFamily.NUMERIC),
> logical(LogicalTypeFamily.NUMERIC)),
> sequence(
> logical(LogicalTypeFamily.NUMERIC),
> logical(LogicalTypeFamily.NUMERIC),
> logical(LogicalTypeFamily.NUMERIC
> .outputTypeStrategy(explicit(DataTypes.BIGINT()))
> .runtimeClass(
>
> "org.apache.flink.table.runtime.functions.table.GenerateSeriesFunction")
> .build(); {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-11937) Resolve small file problem in RocksDB incremental checkpoint
[ https://issues.apache.org/jira/browse/FLINK-11937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-11937: Fix Version/s: 1.20.0 > Resolve small file problem in RocksDB incremental checkpoint > > > Key: FLINK-11937 > URL: https://issues.apache.org/jira/browse/FLINK-11937 > Project: Flink > Issue Type: New Feature > Components: Runtime / Checkpointing >Reporter: Congxian Qiu >Priority: Major > Labels: auto-unassigned, pull-request-available > Fix For: 1.19.0, 1.20.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently when incremental checkpoint is enabled in RocksDBStateBackend a > separate file will be generated on DFS for each sst file. This may cause > “file flood” when running intensive workload (many jobs with high > parallelism) in big cluster. According to our observation in Alibaba > production, such file flood introduces at lease two drawbacks when using HDFS > as the checkpoint storage FileSystem: 1) huge number of RPC request issued to > NN which may burst its response queue; 2) huge number of files causes big > pressure on NN’s on-heap memory. > In Flink we ever noticed similar small file flood problem and tried to > resolved it by introducing ByteStreamStateHandle(FLINK-2808), but this > solution has its limitation that if we configure the threshold too low there > will still be too many small files, while if too high the JM will finally > OOM, thus could hardly resolve the issue in case of using RocksDBStateBackend > with incremental snapshot strategy. > We propose a new OutputStream called > FileSegmentCheckpointStateOutputStream(FSCSOS) to fix the problem. FSCSOS > will reuse the same underlying distributed file until its size exceeds a > preset threshold. We > plan to complete the work in 3 steps: firstly introduce FSCSOS, secondly > resolve the specific storage amplification issue on FSCSOS, and lastly add an > option to reuse FSCSOS across multiple checkpoints to further reduce the DFS > file number. > More details please refer to the attached design doc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27268) build sql query error in JdbcDynamicTableSource
[
https://issues.apache.org/jira/browse/FLINK-27268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-27268:
Fix Version/s: 1.20.0
> build sql query error in JdbcDynamicTableSource
> ---
>
> Key: FLINK-27268
> URL: https://issues.apache.org/jira/browse/FLINK-27268
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.16.0
>Reporter: chouc
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0, 1.20.0
>
>
> h1. # Condidtion
> build sql query error in JdbcDynamicTableSource
>
> {code:java}
> //代码占位符
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tenv = StreamTableEnvironment.create(env);
> String createMysqlTableMapping = "CREATE TABLE table(\n" +
> "id int \n" +
> " ) WITH (\n" +
> " 'connector' = 'jdbc',\n" +
> " 'url' = 'jdbc:mysql://s1:3306/db',\n" +
> " 'username' = '',\n" +
> " 'password' = '',\n" +
> " 'table-name' = 'table_name'" +
> ")\n"; String countSql = "select count(1) from
> t_ds_task_instance";
> tenv.executeSql(createMysqlTableMapping).print();
> tenv.executeSql(countSql).print(); {code}
> h1. ERROR
> {code:java}
> //代码占位符
> Caused by: java.lang.IllegalArgumentException: open() failed.You have an
> error in your SQL syntax; check the manual that corresponds to your MySQL
> server version for the right syntax to use near 'FROM `table`' at line 1
> at
> org.apache.flink.connector.jdbc.table.JdbcRowDataInputFormat.open(JdbcRowDataInputFormat.java:207)
> at
> org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:84)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:67)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:332)
> Caused by: java.sql.SQLSyntaxErrorException: You have an error in your SQL
> syntax; check the manual that corresponds to your MySQL server version for
> the right syntax to use near 'FROM `table`' at line 1
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120)
> at
> com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
> at
> com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
> at
> com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1009)
> at
> org.apache.flink.connector.jdbc.table.JdbcRowDataInputFormat.open(JdbcRowDataInputFormat.java:204)
> ... 4 more {code}
>
> h1. Reason
> because constants cann't be push to jdbc sql as columns, when user query
> single constants in a table,and result to build sql state error
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29967) Optimize determinism requirements from sink node with considering that SinkUpsertMaterializer already supports upsertKey
[ https://issues.apache.org/jira/browse/FLINK-29967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-29967: Fix Version/s: 1.20.0 > Optimize determinism requirements from sink node with considering that > SinkUpsertMaterializer already supports upsertKey > - > > Key: FLINK-29967 > URL: https://issues.apache.org/jira/browse/FLINK-29967 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.16.0 >Reporter: lincoln lee >Priority: Major > Fix For: 1.19.0, 1.20.0 > > > This is followup optimization for FLINK-28569 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27492) Flink table scala example does not including the scala-api jars
[ https://issues.apache.org/jira/browse/FLINK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-27492: Fix Version/s: 1.20.0 > Flink table scala example does not including the scala-api jars > --- > > Key: FLINK-27492 > URL: https://issues.apache.org/jira/browse/FLINK-27492 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.15.0, 1.16.0 >Reporter: Yun Gao >Assignee: Timo Walther >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.19.0, 1.20.0 > > > Currently it seems the flink-scala-api, flink-table-api-scala-bridge is not > including from the binary package[1]. However, currently the scala table > examples seems not include the scala-api classes in the generated jar, If we > start a standalone cluster from the binary distribution package and then > submit a table / sql job in scala, it would fail due to not found the > StreamTableEnvironment class. > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/advanced/#anatomy-of-table-dependencies -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-20865) Prevent potential resource deadlock in fine-grained resource management
[ https://issues.apache.org/jira/browse/FLINK-20865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-20865: Fix Version/s: 1.20.0 > Prevent potential resource deadlock in fine-grained resource management > --- > > Key: FLINK-20865 > URL: https://issues.apache.org/jira/browse/FLINK-20865 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Yangze Guo >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.19.0, 1.20.0 > > Attachments: 屏幕快照 2021-01-06 下午2.32.57.png > > > !屏幕快照 2021-01-06 下午2.32.57.png|width=954,height=288! > The above figure demonstrates a potential case of deadlock due to scheduling > dependency. For the given topology, initially the scheduler will request 4 > slots, for A, B, C and D. Assuming only 2 slots are available, if both slots > are assigned to Pipeline Region 0 (as shown on the left), A and B will first > finish execution, then C and D will be executed, and finally E will be > executed. However, if in the beginning the 2 slots are assigned to A and C > (as shown on the right), then neither of A and C can finish execution due to > missing B and D consuming the data they produced. > Currently, with coarse-grained resource management, the scheduler guarantees > to always finish fulfilling requirements of one pipeline region before > starting to fulfill requirements of another. That means the deadlock case > shown on the right of the above figure can never happen. > However, there’s no such guarantee in fine-grained resource management. Since > resource requirements for SSGs can be different, there’s no control on which > requirements will be fulfilled first, when there’s not enough resources to > fulfill all the requirements. Therefore, it’s not always possible to fulfill > one pipeline region prior to another. > To solve this problem, we can make the scheduler defer requesting slots for > other SSGs before requirements of the current SSG are fulfilled, for > fine-grained resource management, at the price of more scheduling time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-20154) Improve error messages when using CLI with wrong target
[
https://issues.apache.org/jira/browse/FLINK-20154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-20154:
Fix Version/s: 1.20.0
> Improve error messages when using CLI with wrong target
> ---
>
> Key: FLINK-20154
> URL: https://issues.apache.org/jira/browse/FLINK-20154
> Project: Flink
> Issue Type: Improvement
> Components: Command Line Client, Documentation
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major, usability
> Fix For: 1.19.0, 1.20.0
>
>
> According to the [CLI
> documentation|https://ci.apache.org/projects/flink/flink-docs-stable/ops/cli.html#job-submission-examples]
> one can use the CLI with the following {{--target}} values: "remote",
> "local", "kubernetes-session", "yarn-per-job", "yarn-session",
> "yarn-application" and "kubernetes-application". However, when running the
> following commands:
> {{bin/flink run -t yarn-session -p 1 examples/streaming/WindowJoin.jar}} I
> get the following exception:
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:330)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:743)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:242)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:971)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1047)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1047)
> Caused by: java.lang.IllegalStateException
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:182)
> at
> org.apache.flink.client.deployment.executors.AbstractSessionClusterExecutor.execute(AbstractSessionClusterExecutor.java:63)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1917)
> at
> org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:128)
> at
> org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1799)
> at
> org.apache.flink.streaming.examples.join.WindowJoin.main(WindowJoin.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:316)
> ... 11 more
> {code}
> Similarly when running the command {{bin/flink run -t yarn-application -p 1
> examples/streaming/WindowJoin.jar}} I get the exception:
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: No ExecutorFactory found to execute the application.
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:330)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:743)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:242)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:971)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1047)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
> at
> org.apache.flink.runtim
[jira] [Updated] (FLINK-26831) Move JarHandlers to flink-runtime
[ https://issues.apache.org/jira/browse/FLINK-26831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-26831: Fix Version/s: 1.20.0 > Move JarHandlers to flink-runtime > - > > Key: FLINK-26831 > URL: https://issues.apache.org/jira/browse/FLINK-26831 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / REST >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.19.0, 1.20.0 > > > Once upon a time the JarHandlers were intended to only be used by the WebUI > and hence placed in flink-runtime-web. > As these handlers have seen widespread use independent of the UI we should > move them to flink-runtime and get rid of the whole WebSubmissionExtension > logic. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-14148) Investigate pushing predicate/projection to underlying Hive input format
[ https://issues.apache.org/jira/browse/FLINK-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-14148: Fix Version/s: 1.20.0 > Investigate pushing predicate/projection to underlying Hive input format > > > Key: FLINK-14148 > URL: https://issues.apache.org/jira/browse/FLINK-14148 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Rui Li >Priority: Major > Fix For: 1.19.0, 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-18567) Add Support for Azure Cognitive Search Table & SQL Connector
[ https://issues.apache.org/jira/browse/FLINK-18567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-18567: Fix Version/s: 1.20.0 > Add Support for Azure Cognitive Search Table & SQL Connector > > > Key: FLINK-18567 > URL: https://issues.apache.org/jira/browse/FLINK-18567 > Project: Flink > Issue Type: Improvement > Components: API / DataStream, Connectors / Common >Affects Versions: 1.12.0 >Reporter: Israel Ekpo >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.19.0, 1.20.0 > > > The objective of this improvement is to add Azure Cognitive Search [2] as an > output sink for the Table & SQL connectors [1] > [1] > [https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connectors/] > [2] > [https://docs.microsoft.com/en-us/azure/search/search-what-is-azure-search] -- This message was sent by Atlassian Jira (v8.20.10#820010)
