[jira] [Created] (FLINK-34108) Add URL_ENCODE and URL_DECODE function

2024-01-16 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-34108: -- Summary: Add URL_ENCODE and URL_DECODE function Key: FLINK-34108 URL: https://issues.apache.org/jira/browse/FLINK-34108 Project: Flink Issue Type: New Fe

[jira] [Created] (FLINK-34109) FileSystem sink connector restore job from historical checkpoint failure

2024-01-16 Thread Sergey Paryshev (Jira)
Sergey Paryshev created FLINK-34109: --- Summary: FileSystem sink connector restore job from historical checkpoint failure Key: FLINK-34109 URL: https://issues.apache.org/jira/browse/FLINK-34109 Projec

[jira] [Created] (FLINK-34110) Bump janino to 3.1.11

2024-01-16 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-34110: --- Summary: Bump janino to 3.1.11 Key: FLINK-34110 URL: https://issues.apache.org/jira/browse/FLINK-34110 Project: Flink Issue Type: Technical Debt

[jira] [Created] (FLINK-34111) Add JSON_QUOTE and JSON_UNQUOTE function

2024-01-16 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-34111: -- Summary: Add JSON_QUOTE and JSON_UNQUOTE function Key: FLINK-34111 URL: https://issues.apache.org/jira/browse/FLINK-34111 Project: Flink Issue Type: Sub-

[jira] [Created] (FLINK-34112) JavaCodeSplitter OOM

2024-01-16 Thread Junning Liang (Jira)
Junning Liang created FLINK-34112: - Summary: JavaCodeSplitter OOM Key: FLINK-34112 URL: https://issues.apache.org/jira/browse/FLINK-34112 Project: Flink Issue Type: Bug Reporter:

Re: [VOTE] FLIP-377: Support fine-grained configuration to control filter push down for Table/SQL Sources

2024-01-16 Thread Hang Ruan
+1 (non-binding) Best, Hang Jiabao Sun 于2024年1月9日周二 19:39写道: > Hi Devs, > > I'd like to start a vote on FLIP-377: Support fine-grained configuration > to control filter push down for Table/SQL Sources[1] > which has been discussed in this thread[2]. > > The vote will be open for at least 72 hou

[jira] [Created] (FLINK-34113) Update flink-connector-elasticsearch to be compatible with updated SinkV2 interfaces

2024-01-16 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-34113: -- Summary: Update flink-connector-elasticsearch to be compatible with updated SinkV2 interfaces Key: FLINK-34113 URL: https://issues.apache.org/jira/browse/FLINK-34113

Re: [VOTE] FLIP-377: Support fine-grained configuration to control filter push down for Table/SQL Sources

2024-01-16 Thread Leonard Xu
+1 (binding) Best, Leonard > 2024年1月16日 下午5:40,Hang Ruan 写道: > > +1 (non-binding) > > Best, > Hang > > Jiabao Sun 于2024年1月9日周二 19:39写道: > >> Hi Devs, >> >> I'd like to start a vote on FLIP-377: Support fine-grained configuration >> to control filter push down for Table/SQL Sources[1] >> wh

Re:Re: [VOTE] FLIP-377: Support fine-grained configuration to control filter push down for Table/SQL Sources

2024-01-16 Thread Xuyang
+1 (non-binding) -- Best! Xuyang 在 2024-01-16 17:52:38,"Leonard Xu" 写道: >+1 (binding) > >Best, >Leonard > >> 2024年1月16日 下午5:40,Hang Ruan 写道: >> >> +1 (non-binding) >> >> Best, >> Hang >> >> Jiabao Sun 于2024年1月9日周二 19:39写道: >> >>> Hi Devs, >>> >>> I'd like to start a vote on

Re: [DISCUSS] FLIP-415: Introduce a new join operator to support minibatch

2024-01-16 Thread Benchao Li
shuai, Thanks for the explanations, I understand the scenario you described now. IIUC, this will be a rather rare case that need to disable "compaction" when mini-batch is enabled, so I won't be against introducing it. However, I would suggest to enable the "compaction" by default (if mini-batch e

Re: [DISCUSS] Support detecting table schema from external files.

2024-01-16 Thread Benchao Li
Thanks Yisha for bringing up this discussion. Schema inferring is a very interesting and useful feature, especially when it comes to formats with well defined schemas such as Protobuf/Parquet. I'm looking forward to the FLIP. Yisha Zhou 于2024年1月15日周一 16:29写道: > > Hi dev, > > Currently, we are us

[jira] [Created] (FLINK-34114) Parse error while $internal.application.program-args contains '#' in Yarn/K8s Application Mode

2024-01-16 Thread jonasjc (Jira)
jonasjc created FLINK-34114: --- Summary: Parse error while $internal.application.program-args contains '#' in Yarn/K8s Application Mode Key: FLINK-34114 URL: https://issues.apache.org/jira/browse/FLINK-34114

Re: [DISCUSS] FLIP-418: Show data skew score on Flink Dashboard

2024-01-16 Thread Kartoglu, Emre
Hi Rui, Thanks for the feedback. Please find my response below: > The number_of_records_received_by_each_subtask is the total received records, > right? No it's not the total. I understand why this is confusing. I had initially wanted to name it "the list of number of records received by each

Re: Re:[DISCUSS] FLIP-418: Show data skew score on Flink Dashboard

2024-01-16 Thread Kartoglu, Emre
Hi Xuyang, Thanks for the feedback! Please find my response below. > 1. How will the colors of vertics with high data skew scores be unified with > existing backpressure and high busyness colors on the UI? Users should be able to distinguish at a glance which vertics in the entire job graph is

Flink SQL ANTI JOIN results with windowing in DataStream

2024-01-16 Thread Péter Váry
Hi Team, I am working on Iceberg in process compaction, and trying to use SQL window join to compare 2 streams like this: *Table fsFiles = tEnv.sqlQuery("SELECT runId, location, window_start, window_end " +"FROM TABLE(" +"* *TUMBLE(" +"TABLE " + fileSystemFilesTable +

Re: Flink SQL ANTI JOIN results with windowing in DataStream

2024-01-16 Thread Péter Váry
Turns out the issue was that the windows were not triggered for the joins. For one of the tasks the watermark was advanced, so the fdFiles and the tableFiles windows were triggered, but the global watermark was not advanced (parallelism was 4). Since the global watermark was not advanced, the join

[jira] [Created] (FLINK-34115) TableAggregateITCase.testFlagAggregateWithOrWithoutIncrementalUpdate fails

2024-01-16 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-34115: - Summary: TableAggregateITCase.testFlagAggregateWithOrWithoutIncrementalUpdate fails Key: FLINK-34115 URL: https://issues.apache.org/jira/browse/FLINK-34115 Project:

[jira] [Created] (FLINK-34116) GlobalConfigurationTest.testInvalidStandardYamlFile fails

2024-01-16 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-34116: - Summary: GlobalConfigurationTest.testInvalidStandardYamlFile fails Key: FLINK-34116 URL: https://issues.apache.org/jira/browse/FLINK-34116 Project: Flink I

[DISCUSS] FLIP-388: Support Dynamic Logger Level Adjustment

2024-01-16 Thread Yuepeng Pan
Hi all, I created the FLIP-388[1] to support dynamic logger level adjustment. Comprehensive and detailed system logs(like debug, trace, all, etc.) could contribute to improved visibility of internal system execution information and also enhance the efficiency of progra

Re: [DISCUSS] FLIP-388: Support Dynamic Logger Level Adjustment

2024-01-16 Thread David Morávek
Hi Yuepeng, Thanks for the FLIP! There was already quite a discussion on FLIP-210 [1][2], that has proposed more or less the same thing. FLIP was marked as out of scope for Flink because underlying technologies already address it. Are you aware of the effort? If yes, what has changed since then?

Re: [VOTE] Accept Flink CDC into Apache Flink

2024-01-16 Thread Gunnar Morling
+1 (non-binding) Best, --Gunnar Am Mo., 15. Jan. 2024 um 18:24 Uhr schrieb Fabian Hueske : > +1 (binding) > > Cheers, Fabian > > > Timo Walther schrieb am Mo., 15. Jan. 2024, 16:00: > > > +1 (binding) > > > > Cheers, > > Timo > > > > > > On 09.01.24 10:58, xiangyu feng wrote: > > > +1 (non-bi

[jira] [Created] (FLINK-34117) CompactCoordinator for table file sink loses data upon job termination

2024-01-16 Thread Alexander Fedulov (Jira)
Alexander Fedulov created FLINK-34117: - Summary: CompactCoordinator for table file sink loses data upon job termination Key: FLINK-34117 URL: https://issues.apache.org/jira/browse/FLINK-34117 Proj

[DISCUSS] FLIP-419: Optimize multi-sink query plan generation

2024-01-16 Thread Jeyhun Karimov
Hi devs, I’d like to start a discussion on FLIP-419: Optimize multi-sink query plan generation [1]. Currently, the optimization process of multi-sink query plans are suboptimal: 1) it requires to go through the optimization process several times and 2) as a result of this some low-level code com

Re: [DISCUSS] FLIP-419: Optimize multi-sink query plan generation

2024-01-16 Thread Jim Hughes
Hi Jeyhun, Generally, I like the idea of speeding up the optimizer in the case of multiple queries! I am new to the optimizer, but I have a few comments / questions. 1. StreamOptimizeContext may still be needed to pass the fact that we are optimizing a streaming query. I don't think t

Re: [DISCUSS] FLIP-417: Expose JobManagerOperatorMetrics via REST API

2024-01-16 Thread Mason Chen
Hi Xuyang and Hang, Thanks for your support and feedback! See my responses below: 1. IIRC, in a sense, operator ID and vertex ID are the same thing. The > operator ID can > be converted from the vertex ID[1]. Therefore, it is somewhat strange to > have both vertex > ID and operator ID in a single

[jira] [Created] (FLINK-34118) Implement restore tests for Sort node

2024-01-16 Thread Bonnie Varghese (Jira)
Bonnie Varghese created FLINK-34118: --- Summary: Implement restore tests for Sort node Key: FLINK-34118 URL: https://issues.apache.org/jira/browse/FLINK-34118 Project: Flink Issue Type: Sub-t

Re: [DISCUSS] FLIP-417: Expose JobManagerOperatorMetrics via REST API

2024-01-16 Thread Hang Ruan
Hi, Mason. The field `operatorName` in JobManagerOperatorQueryScopeInfo refers to the fields in OperatorQueryScopeInfo and chooses the operatorName instead of OperatorID. It is fine by my side to change from opertorName to operatorID in this FLIP. Best, Hang Mason Chen 于2024年1月17日周三 09:39写道: >

[jira] [Created] (FLINK-34119) Improve description about changelog in document

2024-01-16 Thread Hangxiang Yu (Jira)
Hangxiang Yu created FLINK-34119: Summary: Improve description about changelog in document Key: FLINK-34119 URL: https://issues.apache.org/jira/browse/FLINK-34119 Project: Flink Issue Type: I

[jira] [Created] (FLINK-34120) Introduce unified serialization config option for all Kryo, POJO and customized serializers

2024-01-16 Thread Zhanghao Chen (Jira)
Zhanghao Chen created FLINK-34120: - Summary: Introduce unified serialization config option for all Kryo, POJO and customized serializers Key: FLINK-34120 URL: https://issues.apache.org/jira/browse/FLINK-34120

[jira] [Created] (FLINK-34121) Introduce pipeline.force-kryo-avro to control whether to force registration of Avro serializer with Kryo

2024-01-16 Thread Zhanghao Chen (Jira)
Zhanghao Chen created FLINK-34121: - Summary: Introduce pipeline.force-kryo-avro to control whether to force registration of Avro serializer with Kryo Key: FLINK-34121 URL: https://issues.apache.org/jira/browse/FLI

[jira] [Created] (FLINK-34122) Deprecate old serialization config methods and options

2024-01-16 Thread Zhanghao Chen (Jira)
Zhanghao Chen created FLINK-34122: - Summary: Deprecate old serialization config methods and options Key: FLINK-34122 URL: https://issues.apache.org/jira/browse/FLINK-34122 Project: Flink Issu

Re: [DISCUSS] FLIP-403: High Availability Services for OLAP Scenarios

2024-01-16 Thread Yangze Guo
Thanks for the comments, Zhu. > Did you look into which part takes most of the time? Jar uploading, Jar > downloading, JobInformation shipping, TDD shipping, or others? In our scenario, the key factor should be the JobInformation shipping, as the jobs are completed within 1 second. This can have

[jira] [Created] (FLINK-34123) Introduce built-in serializers for common composited data types

2024-01-16 Thread Zhanghao Chen (Jira)
Zhanghao Chen created FLINK-34123: - Summary: Introduce built-in serializers for common composited data types Key: FLINK-34123 URL: https://issues.apache.org/jira/browse/FLINK-34123 Project: Flink

[jira] [Created] (FLINK-34124) Flink 2.0: Disable Kyro by default

2024-01-16 Thread Zhanghao Chen (Jira)
Zhanghao Chen created FLINK-34124: - Summary: Flink 2.0: Disable Kyro by default Key: FLINK-34124 URL: https://issues.apache.org/jira/browse/FLINK-34124 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-34125) Flink 2.0: Remove deprecated serialization config methods and options

2024-01-16 Thread Zhanghao Chen (Jira)
Zhanghao Chen created FLINK-34125: - Summary: Flink 2.0: Remove deprecated serialization config methods and options Key: FLINK-34125 URL: https://issues.apache.org/jira/browse/FLINK-34125 Project: Flin

Re: [DISCUSS] FLIP-403: High Availability Services for OLAP Scenarios

2024-01-16 Thread Yang Wang
I am completely in favor of splitting the LeaderServices and PersistenceServices while sharing the same concern that MaterialProvider is not very easy to understand. It just feels like we do the separation but not thoroughly. If you have a clear plan for the subsequent improvements, I am fine that

[jira] [Created] (FLINK-34126) Correct the description of jobmanager.scheduler

2024-01-16 Thread Junrui Li (Jira)
Junrui Li created FLINK-34126: - Summary: Correct the description of jobmanager.scheduler Key: FLINK-34126 URL: https://issues.apache.org/jira/browse/FLINK-34126 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-34127) Kafka connector repo runs a duplicate of `IntegrationTests` framework tests

2024-01-16 Thread Mason Chen (Jira)
Mason Chen created FLINK-34127: -- Summary: Kafka connector repo runs a duplicate of `IntegrationTests` framework tests Key: FLINK-34127 URL: https://issues.apache.org/jira/browse/FLINK-34127 Project: Flin

[jira] [Created] (FLINK-34128) oracle jdbc connector bug. When the oracle table field is of type float, the type obtained by jdbc is bigdecimal. error is 'java.lang.ClassCastException: java.math.BigD

2024-01-16 Thread blackpighe (Jira)
blackpighe created FLINK-34128: -- Summary: oracle jdbc connector bug. When the oracle table field is of type float, the type obtained by jdbc is bigdecimal. error is 'java.lang.ClassCastException: java.math.BigDecimal cannot be cast to java.lang.Fl