[jira] [Created] (FLINK-25604) Remove useless aggregate function

2022-01-10 Thread Jing Zhang (Jira)
Jing Zhang created FLINK-25604: -- Summary: Remove useless aggregate function Key: FLINK-25604 URL: https://issues.apache.org/jira/browse/FLINK-25604 Project: Flink Issue Type: Sub-task

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

2022-01-10 Thread Jingsong Li
Thanks everyone for your voting. If there are no objections, I'll close this vote and send a vote result mail: - create a sub project named `flink-table-store`. Best, Jingsong On Tue, Jan 11, 2022 at 2:51 PM Jingsong Li wrote: > > Hi Fabian, > > Thanks for your information. > > If gradle is mat

[VOTE] FLIP-199: Change some default config values of blocking shuffle for better usability

2022-01-10 Thread Yingjie Cao
Hi all, I'd like to start a vote on FLIP-199: Change some default config values of blocking shuffle for better usability [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1] https://cwiki.apache.org/con

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

2022-01-10 Thread Jingsong Li
Hi Fabian, Thanks for your information. If gradle is mature later, it should not be too difficult to migrate from maven, we can consider it later. Best, Jingsong On Tue, Jan 11, 2022 at 12:00 PM 刘首维 wrote: > > Thanks for driving this, Jingsong. > +1 (non-binding) for separate repository. > > >

[jira] [Created] (FLINK-25602) make the BlobServer use aws s3

2022-01-10 Thread lupan (Jira)
lupan created FLINK-25602: - Summary: make the BlobServer use aws s3 Key: FLINK-25602 URL: https://issues.apache.org/jira/browse/FLINK-25602 Project: Flink Issue Type: Improvement Components

[jira] [Created] (FLINK-25603) make the BlobServer use aws s3

2022-01-10 Thread lupan (Jira)
lupan created FLINK-25603: - Summary: make the BlobServer use aws s3 Key: FLINK-25603 URL: https://issues.apache.org/jira/browse/FLINK-25603 Project: Flink Issue Type: Improvement Components

Re: [DISCUSS] Releasing Flink 1.14.3

2022-01-10 Thread Thomas Weise
Thank you Xingbo. I meanwhile also got my Azure pipeline working and was able to build the artifacts. Although in general it would be nice if not every release volunteer had to set up their separate Azure environment. Martijn, The release is staged, except for the website PR: https://issues.apac

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

2022-01-10 Thread ??????
Thanks for driving this, Jingsong. +1 (non-binding) for separate repository. Best Regards, Shouwei --  -- ??: "dev"

[jira] [Created] (FLINK-25601) Update 'state.backend' in flink-conf.yaml

2022-01-10 Thread Ada Wong (Jira)
Ada Wong created FLINK-25601: Summary: Update 'state.backend' in flink-conf.yaml Key: FLINK-25601 URL: https://issues.apache.org/jira/browse/FLINK-25601 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-205: Support cache in DataStream for Batch Processing

2022-01-10 Thread Xuannan Su
Hi Gen, Thanks for your feedback. I think you are talking about how we are going to store the caching data. The first option is to write the data with a sink to an external file system, much like the file store of the Dynamic Table. If I understand correctly, it requires a distributed file system

[jira] [Created] (FLINK-25600) Support new statement set syntax in sql client and update docs

2022-01-10 Thread Wenlong Lyu (Jira)
Wenlong Lyu created FLINK-25600: --- Summary: Support new statement set syntax in sql client and update docs Key: FLINK-25600 URL: https://issues.apache.org/jira/browse/FLINK-25600 Project: Flink

[jira] [Created] (FLINK-25599) The description of taskmanager.memory.task.heap.size in the official document is incorrect

2022-01-10 Thread Zhuang Liu (Jira)
Zhuang Liu created FLINK-25599: --- Summary: The description of taskmanager.memory.task.heap.size in the official document is incorrect Key: FLINK-25599 URL: https://issues.apache.org/jira/browse/FLINK-25599

Re: [DISCUSS] FLIP-208: Update KafkaSource to detect EOF based on de-serialized record

2022-01-10 Thread Dong Lin
Hi Fabian, Thanks for the comments! By "add a source mixin interface", are you suggesting to update the org.apache.flink.api.connector.source.Source interface to add the API "RecordEvaluator getRecordEvaluator()"? If so, it seems to add more public API and thus more complexity than the solution i

[jira] [Created] (FLINK-25598) Changelog materialized state discarded on failure

2022-01-10 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-25598: - Summary: Changelog materialized state discarded on failure Key: FLINK-25598 URL: https://issues.apache.org/jira/browse/FLINK-25598 Project: Flink I

[jira] [Created] (FLINK-25597) Document which URI based config options works with local / general filesystem

2022-01-10 Thread Jira
David Morávek created FLINK-25597: - Summary: Document which URI based config options works with local / general filesystem Key: FLINK-25597 URL: https://issues.apache.org/jira/browse/FLINK-25597 Proje

Re: [DISCUSS] FLIP-210: Change logging level dynamically at runtime

2022-01-10 Thread Konstantin Knauf
Thank you for starting the discussion. Being able to change the logging level at runtime is very valuable in my experience. Instead of introducing our own API (and eventually even persistence), could we just periodically reload the log4j or logback configuration from the environment/filesystem? I

[DISCUSS] FLIP-210: Change logging level dynamically at runtime

2022-01-10 Thread Wenhao Ji
Hi everyone, Hope you enjoyed the Holiday Season. I would like to start the discussion on the improvement purpose FLIP-210 [1] which aims to provide a way to change log levels at runtime to simplify issues and bugs detection as reported in the ticket FLINK-16478 [2]. Firstly, thanks Xingxing Di a

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

2022-01-10 Thread godfrey he
+1 for the separate repository, and the name "flink-table-store". Best, Godfrey Becket Qin 于2022年1月10日周一 22:22写道: > > Thanks for the FLIP, Jingsong. > > +1 (binding) > > Naming wise, I am also slightly leaning towards calling it > "flink-table-store". > > Thanks, > > Jiangjie (Becket) Qin > > On

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

2022-01-10 Thread Becket Qin
Thanks for the FLIP, Jingsong. +1 (binding) Naming wise, I am also slightly leaning towards calling it "flink-table-store". Thanks, Jiangjie (Becket) Qin On Mon, Jan 10, 2022 at 7:39 PM Fabian Paul wrote: > Hi all, > > I just wanted to give my two cents for the build system discussion. In >

[jira] [Created] (FLINK-25596) Specify hash/sortmerge join in SQL hint

2022-01-10 Thread Jing Zhang (Jira)
Jing Zhang created FLINK-25596: -- Summary: Specify hash/sortmerge join in SQL hint Key: FLINK-25596 URL: https://issues.apache.org/jira/browse/FLINK-25596 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-25595) Specify hash/sort aggregate strategy in SQL hint

2022-01-10 Thread Jing Zhang (Jira)
Jing Zhang created FLINK-25595: -- Summary: Specify hash/sort aggregate strategy in SQL hint Key: FLINK-25595 URL: https://issues.apache.org/jira/browse/FLINK-25595 Project: Flink Issue Type: Sub-

[jira] [Created] (FLINK-25594) Take parquet metadata into consideration when source is parquet files

2022-01-10 Thread Jing Zhang (Jira)
Jing Zhang created FLINK-25594: -- Summary: Take parquet metadata into consideration when source is parquet files Key: FLINK-25594 URL: https://issues.apache.org/jira/browse/FLINK-25594 Project: Flink

[jira] [Created] (FLINK-25593) A redundant scan could be skipped if it is an input of join and the other input is empty after partition prune

2022-01-10 Thread Jing Zhang (Jira)
Jing Zhang created FLINK-25593: -- Summary: A redundant scan could be skipped if it is an input of join and the other input is empty after partition prune Key: FLINK-25593 URL: https://issues.apache.org/jira/browse/FLI

Re: [VOTE] FLIP-201: Persist local state in working directory

2022-01-10 Thread Till Rohrmann
+1 (binding) Cheers, Till On Mon, Jan 10, 2022 at 2:30 PM Etienne Chauchot wrote: > +1 > > Best > > Etienne Chauchot > > Le 10/01/2022 à 10:22, Till Rohrmann a écrit : > > Hi everyone, > > > > I'd like to start a vote on FLIP-201: Persist local state in working > > directory [1] which has been

Re: is task reassignment possible

2022-01-10 Thread DONG, Weike
Hi Qin, Thanks for bringing up this issue. AFAIK, there is no such mechanism in Flink for dynamic task re-assignment at runtime, as states need to be correctly re-distributed across the nodes, which is highly error-prone and not well-suited for the current computation model. However, if the data-

[jira] [Created] (FLINK-25592) Improvement of parser, optimizer and execution for Flink Batch SQL

2022-01-10 Thread Jing Zhang (Jira)
Jing Zhang created FLINK-25592: -- Summary: Improvement of parser, optimizer and execution for Flink Batch SQL Key: FLINK-25592 URL: https://issues.apache.org/jira/browse/FLINK-25592 Project: Flink

Re: [VOTE] FLIP-201: Persist local state in working directory

2022-01-10 Thread Etienne Chauchot
+1 Best Etienne Chauchot Le 10/01/2022 à 10:22, Till Rohrmann a écrit : Hi everyone, I'd like to start a vote on FLIP-201: Persist local state in working directory [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not

[jira] [Created] (FLINK-25591) Use FileSource for StreamExecutionEnvironment.readFiles

2022-01-10 Thread Fabian Paul (Jira)
Fabian Paul created FLINK-25591: --- Summary: Use FileSource for StreamExecutionEnvironment.readFiles Key: FLINK-25591 URL: https://issues.apache.org/jira/browse/FLINK-25591 Project: Flink Issue T

Re: [DISCUSS] FLIP-208: Update KafkaSource to detect EOF based on de-serialized record

2022-01-10 Thread Fabian Paul
Hi Dong, Thank you for updating the FLIP and making it applicable for all sources. I am a bit unsure about the implementation part. I would propose to add a source mixin interface that implements `getRecordEvaluator` and sources that want to allow dynamically stopping implement that interface. An

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

2022-01-10 Thread Fabian Paul
Hi all, I just wanted to give my two cents for the build system discussion. In general, I agree with David's opinion to start new projects with Gradle but during the development of the external connector repository, we found some difficulties that still need to be solved. I do not want to force an

Re: Regarding Connector Options - value.deserializer

2022-01-10 Thread Hang Ruan
Hi, Ronak, We can not set specific 'value.deserializer' in table option. 'key.deserializer' and 'value.deserializer' is always set to 'org.apache.kafka.common.serialization.ByteArrayDeserializer'. If you want to implement a format, you could take a look at the code JsonFormatFactory.java in flink

[jira] [Created] (FLINK-25590) Logging warning of insufficient memory for all configured buffers

2022-01-10 Thread Anton Kalashnikov (Jira)
Anton Kalashnikov created FLINK-25590: - Summary: Logging warning of insufficient memory for all configured buffers Key: FLINK-25590 URL: https://issues.apache.org/jira/browse/FLINK-25590 Project:

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

2022-01-10 Thread Jark Wu
I'm also in favour of "flink-table-store". Best, Jark On Mon, 10 Jan 2022 at 16:18, David Morávek wrote: > Hi Jingsong, > > the connector repository prototype I've seen is being built on top of > Gradle [1], that's why I was referring to it (I think one idea was also to > migrate the main repos

[jira] [Created] (FLINK-25589) Update Chinese version of Elasticsearch connector docs

2022-01-10 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-25589: Summary: Update Chinese version of Elasticsearch connector docs Key: FLINK-25589 URL: https://issues.apache.org/jira/browse/FLINK-25589 Project: Flink

Re: [DISCUSS] Slimmed down docker images.

2022-01-10 Thread Chesnay Schepler
There could be a problem with the fat image actually depending on the slim version; I'm not sure if the official-images repo supports that. We should however be able to generate 2 separate standalone dockerfiles. On 23/12/2021 11:16, Till Rohrmann wrote: Hi David, Thanks for starting this disc

RE: Regarding Connector Options - value.deserializer

2022-01-10 Thread Ronak Beejawat (rbeejawa)
Hi Hang, My question is can we use specific ‘value.deserializer’ in table option via kafka connector is there any way or not ? I have already kept 'value.format' in below code snippet so is that enough and handle deserializer by itself internally? How to create custom format can you please sha

Re: [DISCUSS] Creating an external connector repository

2022-01-10 Thread Chesnay Schepler
What CI resources do you actually intend use? Asking since the ASF GHA resources are afaik quite overloaded. On 05/01/2022 11:48, Martijn Visser wrote: Hi everyone, I wanted to summarise the email thread and see if there are any open items that still need to be discussed, before we can finalis

[jira] [Created] (FLINK-25588) Add jdk8 and datetime module to jackson shaded

2022-01-10 Thread Francesco Guardiani (Jira)
Francesco Guardiani created FLINK-25588: --- Summary: Add jdk8 and datetime module to jackson shaded Key: FLINK-25588 URL: https://issues.apache.org/jira/browse/FLINK-25588 Project: Flink

Re: Regarding Connector Options - value.deserializer

2022-01-10 Thread Hang Ruan
Hi, Ronak, I think you should implement a custom format by yourself instead of overriding. The 'value.format' is a required table option. Best, Hang Ronak Beejawat (rbeejawa) 于2022年1月10日周一 17:09写道: > Hi Team, > > Is there any way we use value.deserializer in Connector Options from kafka > via

Re: [VOTE] Release flink-shaded 15.0, release candidate #1

2022-01-10 Thread Matthias Pohl
Correction: My previous vote in this thread is actually non-binding. 8) On Wed, Jan 5, 2022 at 2:57 PM Matthias Pohl wrote: > +1 (binding) > > - Verified the checksums > - Checked the website PR > - Diff'd the NOTICE files comparing it to 14.0 to check for anything > suspicious > - build Flink s

Re: Flink SQL Kafka connector

2022-01-10 Thread Hang Ruan
Hi, There is already an on-going issue about it. ( https://issues.apache.org/jira/browse/FLINK-24456) Best, hang 聂荧屏 于2022年1月10日周一 10:06写道: > hello > > > Is there any plan to develop batch mode of Flink SQL Kafka connector? > > I would like to use kafka connector for daily/hourly/minute-by-min

[VOTE] FLIP-201: Persist local state in working directory

2022-01-10 Thread Till Rohrmann
Hi everyone, I'd like to start a vote on FLIP-201: Persist local state in working directory [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1] https://cwiki.apache.org/confluence/x/wJuqCw [2] https://

Re: [DISCUSS] FLIP-201: Persist local state in working directory

2022-01-10 Thread Till Rohrmann
Hi Yun, I assume that most people will use this feature with k8s like deployment environments. But in theory it works everywhere where you can establish a stable relationship between volumes and Flink processes. If Flink processes are restarted on different nodes, then of course you need volumes t

Regarding Connector Options - value.deserializer

2022-01-10 Thread Ronak Beejawat (rbeejawa)
Hi Team, Is there any way we use value.deserializer in Connector Options from kafka via sql api? PFB below code snippt : tableEnv.executeSql("CREATE TABLE cmrTable (\r\n" + " org_id STRING\r\n" + " ,cluster_id STRING\r\n" + " ,globalcallid_callmanage

Re: [ANNOUNCE] Apache Flink ML 2.0.0 released

2022-01-10 Thread Till Rohrmann
This is really great news. Thanks a lot for all the work Dong, Yun, Zhipeng and others! Cheers, Till On Fri, Jan 7, 2022 at 2:36 PM David Morávek wrote: > Great job! <3 Thanks Dong and Yun for managing the release and big thanks > to everyone who has contributed! > > Best, > D. > > On Fri, Jan

Re: [DISCUSS] FLIP-201: Persist local state in working directory

2022-01-10 Thread Yun Tang
I think this feature could indeed help recovery faster on the case of node failure. It seems this feature could only work well with k8s-like deployment environment? Best, Yun Tang From: David Morávek Sent: Wednesday, January 5, 2022 19:51 To: dev Subject: Re:

[jira] [Created] (FLINK-25587) HiveCatalogITCase crashed on Azure with exit code 239

2022-01-10 Thread Yun Gao (Jira)
Yun Gao created FLINK-25587: --- Summary: HiveCatalogITCase crashed on Azure with exit code 239 Key: FLINK-25587 URL: https://issues.apache.org/jira/browse/FLINK-25587 Project: Flink Issue Type: Bug

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

2022-01-10 Thread David Morávek
Hi Jingsong, the connector repository prototype I've seen is being built on top of Gradle [1], that's why I was referring to it (I think one idea was also to migrate the main repository to Gradle eventually). I think Martijn / Fabian may be bit more familiar with the connectors repository effort a