Re: Issue in seeing job manager and task manager logs in history sever

2022-05-25 Thread Chesnay Schepler
This is currently not supported, no. There are some conceptual issues. For example, session mode can't isolate the logs between jobs, sou you'd end up with gargantuan logs being archived for every job. Similarly, and form of log rotation would reduce the value of such a mechanism. I would rec

Re: [DISCUSS] FLIP-235: Hybrid Shuffle Mode

2022-05-25 Thread Chesnay Schepler
Will this also allow spilling everything to disk while also forwarding data to the next task? This would allow us to improve fine-grained recovery by no longer being constrained to pipelined regions. On 25/05/2022 05:55, weijie guo wrote: Hi All, Thank you for your attention and feedback. Do

[jira] [Created] (FLINK-27765) TaskManagerProcessFailureBatchRecoveryITCase.testTaskManagerProcessFailure failed with FlinkJobNotFoundException

2022-05-25 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-27765: Summary: TaskManagerProcessFailureBatchRecoveryITCase.testTaskManagerProcessFailure failed with FlinkJobNotFoundException Key: FLINK-27765 URL: https://issues.apache.org/jira/bro

Re: Issue in seeing job manager and task manager logs in history sever

2022-05-25 Thread Sriram Ganesh
Thank you for the clarification. Sure will use your inputs. On Wed, May 25, 2022, 12:34 Chesnay Schepler wrote: > This is currently not supported, no. > > There are some conceptual issues. > For example, session mode can't isolate the logs between jobs, sou you'd > end up with gargantuan logs be

[GitHub] [flink-connector-redis] chayim commented on pull request #2: [FLINK-15571][connector][WIP] Redis Stream connector for Flink

2022-05-25 Thread GitBox
chayim commented on PR #2: URL: https://github.com/apache/flink-connector-redis/pull/2#issuecomment-1136864263 @MartijnVisser Pinging here - as this superceeds https://github.com/apache/flink/pull/15487 -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

2022-05-25 Thread Lincoln Lee
Hi Jark, Thanks for your reply! Currently 'lookup.async' just lies in HBase connector, I have no idea whether or when to remove it (we can discuss it in another issue for the HBase connector after FLINK-27625 is done), just not add it into a common option now. Best, Lincoln Lee Jark Wu 于2022年

[RESULT][VOTE] FLIP-91: Support SQL Gateway

2022-05-25 Thread Shengkai Fang
Hi, dev. FLIP-91: Support SQL Gateway[1] has been accepted. There are 4 binding votes, 5 non-binding votes[2]. - Jark Wu(binding) - Paul Lam(non-binding) - Yufei Zhang(non-binding) - Shqiprim Bunjaku(non-binding) - Jingsong Li(binding) - Nicholas Jiang(non-binding) - Luning Wang(non-binding) - go

Re: [DISCUSS] FLIP-224: Blacklist Mechanism

2022-05-25 Thread Lijie Wang
Hi everyone, I've updated the FLIP according to Chesnay's feedback, changes as follows: 1. Change the GET result to a map. 2. Only left *endTimestamp* in ADD operation, and change the rest (from POST) to PUT 3. Introduce a new slot pool implementation(BlocklistSlotPool) to encapsulate blocklist re

[jira] [Created] (FLINK-27767) Introduce Endpoint API and utils

2022-05-25 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-27767: - Summary: Introduce Endpoint API and utils Key: FLINK-27767 URL: https://issues.apache.org/jira/browse/FLINK-27767 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-27766) Introduce the framework of the GatewayService

2022-05-25 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-27766: - Summary: Introduce the framework of the GatewayService Key: FLINK-27766 URL: https://issues.apache.org/jira/browse/FLINK-27766 Project: Flink Issue Type: S

[jira] [Created] (FLINK-27769) Introduce the REST endpoint framework

2022-05-25 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-27769: - Summary: Introduce the REST endpoint framework Key: FLINK-27769 URL: https://issues.apache.org/jira/browse/FLINK-27769 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-27770) Introduce the script to start/stop/stop-all gateway

2022-05-25 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-27770: - Summary: Introduce the script to start/stop/stop-all gateway Key: FLINK-27770 URL: https://issues.apache.org/jira/browse/FLINK-27770 Project: Flink Issue T

[jira] [Created] (FLINK-27771) Introduce the listAPIs in the SqlGatewayService

2022-05-25 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-27771: - Summary: Introduce the listAPIs in the SqlGatewayService Key: FLINK-27771 URL: https://issues.apache.org/jira/browse/FLINK-27771 Project: Flink Issue Type:

[jira] [Created] (FLINK-27772) Expose Operation-level logs in the SqlGatewayService

2022-05-25 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-27772: - Summary: Expose Operation-level logs in the SqlGatewayService Key: FLINK-27772 URL: https://issues.apache.org/jira/browse/FLINK-27772 Project: Flink Issue

[jira] [Created] (FLINK-27773) Introduce the E2E tests for SQL Gateway

2022-05-25 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-27773: - Summary: Introduce the E2E tests for SQL Gateway Key: FLINK-27773 URL: https://issues.apache.org/jira/browse/FLINK-27773 Project: Flink Issue Type: Sub-ta

[jira] [Created] (FLINK-27768) Allow to execute sql for the SqlGatewayService

2022-05-25 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-27768: - Summary: Allow to execute sql for the SqlGatewayService Key: FLINK-27768 URL: https://issues.apache.org/jira/browse/FLINK-27768 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

2022-05-25 Thread Qingsheng Ren
Hi Lincoln and Jark, Thanks for the comments! If the community reaches a consensus that we use SQL hint instead of table options to decide whether to use sync or async mode, it’s indeed not necessary to introduce the “lookup.async” option. I think it’s a good idea to let the decision of async

Fwd: Flink Kinesis Connector GetRecords Frequency

2022-05-25 Thread Aqib Mehmood
Dear Support Team, We were setting up a Kinesis-Flink pipeline on our cluster and we seem to be running into an issue. It seems that a Kinesis Data Stream GetRecords limit is 5 requests/second and we were wondering if Flink surpasses this limit by default. Is there any metric that we can use to

[jira] [Created] (FLINK-27774) Window Agg doesn't work when used in Batch Table API

2022-05-25 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-27774: Summary: Window Agg doesn't work when used in Batch Table API Key: FLINK-27774 URL: https://issues.apache.org/jira/browse/FLINK-27774 Project: Flink Issue Ty

Re: [DISCUSS] FLIP-232: Add Retry Support For Async I/O In DataStream API

2022-05-25 Thread Jark Wu
Hi Lincoln, Thanks for proposing this retry feature for the async operator, this would be very helpful for FLIP-234. It's glad to see the vivid discussion, and the following are my thoughts: 1) +1 w/o retry state. It's very tricky and hard to implement a semantic exact state for retry (currentAtt

[DISCUSS] Publish flink-end-to-end-tests-common to Maven

2022-05-25 Thread Alexander Fedulov
Hi everyone, As part of the efforts to externalize the connectors, it became evident that some test utilities that reside in Flink will be required in the external connector repositories. For instance, FlinkContainerTestEnvironment and its dependencies reside in the flink-end-to-end-tests-common m

Re: [DISCUSS] Publish flink-end-to-end-tests-common to Maven

2022-05-25 Thread Alexander Preuß
Hi Alex, Thank you for driving this effort! I think it's a good idea to publish flink-end-to-end-tests-common to maven, to help with the development of the external components. Best, On Wed, May 25, 2022 at 11:56 AM Alexander Fedulov wrote: > Hi everyone, > > As part of the efforts to external

Re: Flink Kinesis Connector GetRecords Frequency

2022-05-25 Thread Ahmed Hamdy
Hi Aqib Flink uses FlinkKinesisConsumer as the source connector to read from kinesis sources. You can refer to this part of the documentation on how to configure Flink

Re: Flink Kinesis Connector GetRecords Frequency

2022-05-25 Thread Teoh, Hong
Hi Aqib, To add to Ahmed's response: * The specific GetRecords Configurations you can control for the KinesisConsumer can be seen here

Re: [DISCUSS] FLIP-228: Support Within between events in CEP Pattern

2022-05-25 Thread Martijn Visser
Hi Nicholas, Thanks for creating the FLIP, I can imagine that there will be many use cases who can be created using this new feature. The FLIP doesn't mention anything with regards to SQL, could this feature also be supported when using MATCH_RECOGNIZE? Best regards, Martijn https://twitter.com

Re: [DISCUSS] FLIP-231: Introduce SupportStatisticReport to support reporting statistics from source connectors

2022-05-25 Thread Martijn Visser
Hi Godfrey, Thanks for creating the FLIP. I have no comments. Best regards, Martijn On Tue, 17 May 2022 at 04:52, Jingsong Li wrote: > Hi Godfrey, > > Thanks for your reply. > > Sounds good to me. > > > I think we should also introduce a config option > > We can add this option to the FLIP.

[GitHub] [flink-connector-redis] MartijnVisser commented on pull request #2: [FLINK-15571][connector][WIP] Redis Stream connector for Flink

2022-05-25 Thread GitBox
MartijnVisser commented on PR #2: URL: https://github.com/apache/flink-connector-redis/pull/2#issuecomment-1137146071 @chayim Thanks for that! I'm off for a little bit but I'll see if we can find anyone who can help with a review of this! -- This is an automated message from the Apache G

[jira] [Created] (FLINK-27775) FlinkKafkaProducer VS KafkaSink

2022-05-25 Thread Jiangfei Liu (Jira)
Jiangfei Liu created FLINK-27775: Summary: FlinkKafkaProducer VS KafkaSink Key: FLINK-27775 URL: https://issues.apache.org/jira/browse/FLINK-27775 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-27776) Throws exception when udaf used in sliding window does not implement merge method in PyFlink

2022-05-25 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-27776: Summary: Throws exception when udaf used in sliding window does not implement merge method in PyFlink Key: FLINK-27776 URL: https://issues.apache.org/jira/browse/FLINK-27776

[GitHub] [flink-connector-redis] sazzad16 commented on pull request #2: [FLINK-15571][connector][WIP] Redis Stream connector for Flink

2022-05-25 Thread GitBox
sazzad16 commented on PR #2: URL: https://github.com/apache/flink-connector-redis/pull/2#issuecomment-1137164171 @MartijnVisser Thank you :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [DISCUSS] FLIP-232: Add Retry Support For Async I/O In DataStream API

2022-05-25 Thread Lincoln Lee
Hi Jark, Thanks for your feedback! for 2) good advice for the generic type naming, use OUT instead of T for the async scenario can be better. for 3) your concern makes sense to me, we should make the change more explicitly to users, especially the api itself (although the documentation is necess

Re: [DISCUSS] FLIP-235: Hybrid Shuffle Mode

2022-05-25 Thread Xintong Song
> > Will this also allow spilling everything to disk while also forwarding > data to the next task? > Yes, as long as the downstream task is started, this always forward the data, even while spilling everything. This would allow us to improve fine-grained recovery by no longer being > constrained

Re: [DISCUSS] FLIP-235: Hybrid Shuffle Mode

2022-05-25 Thread Chesnay Schepler
Yes; but that's also a limitation of the current fine-grained recovery. My suggestion was primarily aimed at jobs that have no/few blocking exchanges, where users would currently have to explicitly configure additional blocking exchanges to really get something out of fine-grained recovery (at

[jira] [Created] (FLINK-27777) Can not get the parquet.compression when using native parquet&orc writer to sink hive

2022-05-25 Thread Small Wong (Jira)
Small Wong created FLINK-2: -- Summary: Can not get the parquet.compression when using native parquet&orc writer to sink hive Key: FLINK-2 URL: https://issues.apache.org/jira/browse/FLINK-2 Pro

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.0.0, release candidate #1

2022-05-25 Thread Yang Wang
I have verified this release candidate with the following and do not find other issues. * Install via helm chart * Create a Flink application and * Trigger savepoint manually * Status of finished job could be synced to CR status * Kill JobManager of finished job and do not trigger a fresh res

Re: [DISCUSS] FLIP-235: Hybrid Shuffle Mode

2022-05-25 Thread Xintong Song
In general, I agree with you about aiming jobs with no/few blocking exchanges for fine-grained recovery. The only problem is, correct me if I'm wrong, users currently cannot control the data exchanging mode of a specific edge. I'm not aware of such APIs. As a first step, I'd prefer excluding this

[jira] [Created] (FLINK-27778) table-planner should explicitly depend on table-code-splitter

2022-05-25 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-27778: Summary: table-planner should explicitly depend on table-code-splitter Key: FLINK-27778 URL: https://issues.apache.org/jira/browse/FLINK-27778 Project: Flink

Re: [DISCUSS] FLIP-235: Hybrid Shuffle Mode

2022-05-25 Thread Chesnay Schepler
You can influence it to some extent via ExecutionConfig#setExecutionMode. You can for example for all shuffles to use blocking exchanges. I'm not proposing an API that would allow this to be set per edge. On 25/05/2022 15:23, Xintong Song wrote: In general, I agree with you about aiming jobs wit

Re: [DISCUSS] Publish flink-end-to-end-tests-common to Maven

2022-05-25 Thread Chesnay Schepler
I'd be more inclined to move the FlinkContainerTestEnvironment to flink-(connector-)test-utils for the time being, because the vast majority of stuff in flink-end-to-end-tests-common isn't really in a state where we want to share it. On 25/05/2022 11:55, Alexander Fedulov wrote: Hi everyone,

Re: [DISCUSS] FLIP-235: Hybrid Shuffle Mode

2022-05-25 Thread Xintong Song
Ok, I think we are on the same page. I'm aware of ExecutionConfig#setExecutionMode, which sets the data exchanging mode at the job scope. Best, Xintong On Wed, May 25, 2022 at 9:50 PM Chesnay Schepler wrote: > You can influence it to some extent via ExecutionConfig#setExecutionMode. > You ca

[jira] [Created] (FLINK-27779) Connectors should not depend on `flink-table-planner`

2022-05-25 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-27779: Summary: Connectors should not depend on `flink-table-planner` Key: FLINK-27779 URL: https://issues.apache.org/jira/browse/FLINK-27779 Project: Flink

[jira] [Created] (FLINK-27780) Connector-cassandra should not depend on `flink-table-planner`

2022-05-25 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-27780: Summary: Connector-cassandra should not depend on `flink-table-planner` Key: FLINK-27780 URL: https://issues.apache.org/jira/browse/FLINK-27780 Project: Flink

[jira] [Created] (FLINK-27781) Connector-kafka should not depend on `flink-table-planner`

2022-05-25 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-27781: Summary: Connector-kafka should not depend on `flink-table-planner` Key: FLINK-27781 URL: https://issues.apache.org/jira/browse/FLINK-27781 Project: Flink

[jira] [Created] (FLINK-27782) Connector-kinesis should not depend on `flink-table-planner`

2022-05-25 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-27782: Summary: Connector-kinesis should not depend on `flink-table-planner` Key: FLINK-27782 URL: https://issues.apache.org/jira/browse/FLINK-27782 Project: Flink

[jira] [Created] (FLINK-27783) Connector-aws-kinesis should not depend on `flink-table-planner`

2022-05-25 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-27783: Summary: Connector-aws-kinesis should not depend on `flink-table-planner` Key: FLINK-27783 URL: https://issues.apache.org/jira/browse/FLINK-27783 Project: Fli

[jira] [Created] (FLINK-27785) Connector-hbase should not depend on `flink-table-planner`

2022-05-25 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-27785: Summary: Connector-hbase should not depend on `flink-table-planner` Key: FLINK-27785 URL: https://issues.apache.org/jira/browse/FLINK-27785 Project: Flink

[jira] [Created] (FLINK-27784) Connector-jdbc should not depend on `flink-table-planner`

2022-05-25 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-27784: Summary: Connector-jdbc should not depend on `flink-table-planner` Key: FLINK-27784 URL: https://issues.apache.org/jira/browse/FLINK-27784 Project: Flink

[jira] [Created] (FLINK-27786) Connector-hive should not depend on `flink-table-planner`

2022-05-25 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-27786: Summary: Connector-hive should not depend on `flink-table-planner` Key: FLINK-27786 URL: https://issues.apache.org/jira/browse/FLINK-27786 Project: Flink

Re: [DISCUSS] Publish flink-end-to-end-tests-common to Maven

2022-05-25 Thread Alexander Fedulov
I see. A couple of considerations: 1. I would assume FlinkContainerTestEnvironment could be used for more things than solely connectors, hence it should ideally not be placed into flink-connector-test-utils 2. If we put it and its dependencies into flink-test-utils, what would be the purpose of t

Re: [DISCUSS] Publish flink-end-to-end-tests-common to Maven

2022-05-25 Thread Chesnay Schepler
1) That's a fair point. So far everything that came out of the connector e2e framework was unfortunately somehow coupled to connectors conceptually. I wouldn't mind changing that, because as you said this could also be useful for other things, but it needs a more holistic view over the entire s

Re: About Native Deployment's Autoscaling implementation

2022-05-25 Thread Talat Uyarer
Hi Gyula, I did some investigation. Kubernetes developers suggest not using the kubernetes metric system for application specific metrics. [1] Currently only possible workflow is over prometheus. Prometheus is used widely on Kubernetes deployments. Kubernetes metrics provides Custom Metrics API, e

Re: About Native Deployment's Autoscaling implementation

2022-05-25 Thread Talat Uyarer
Hi Yang, I thought we could enable Adaptive Scheduler, so adding or removing a task manager is the same as restarting a job when we use an adaptive scheduler. Do I miss anything ? Thanks On Tue, May 24, 2022 at 8:16 PM Yang Wang wrote: > Thanks for the interesting discussion. > > Compared with

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.0.0, release candidate #1

2022-05-25 Thread Jim Busche
Hi Yang, Thank you for the RC1. I couldn’t build from source due to the .git copy issue, but cloning and building from the release-1.0 branch looked good. * Twistlock vulnerability scan was clean of all known issues * Helm install with this release-1.0 source image ran fine on OpenShift

[jira] [Created] (FLINK-27787) New tasks think they've been restored from savepoint (even when they weren't present in that savepoint)

2022-05-25 Thread Stephen Patel (Jira)
Stephen Patel created FLINK-27787: - Summary: New tasks think they've been restored from savepoint (even when they weren't present in that savepoint) Key: FLINK-27787 URL: https://issues.apache.org/jira/browse/FLIN

[jira] [Created] (FLINK-27788) Adding annotation to k8 operator Pod

2022-05-25 Thread Jaganathan Asokan (Jira)
Jaganathan Asokan created FLINK-27788: - Summary: Adding annotation to k8 operator Pod Key: FLINK-27788 URL: https://issues.apache.org/jira/browse/FLINK-27788 Project: Flink Issue Type: Im

[jira] [Created] (FLINK-27790) Move ADD/Remove Jar syntax code implementation from SqlClient to TableEnvironment side

2022-05-25 Thread dalongliu (Jira)
dalongliu created FLINK-27790: - Summary: Move ADD/Remove Jar syntax code implementation from SqlClient to TableEnvironment side Key: FLINK-27790 URL: https://issues.apache.org/jira/browse/FLINK-27790 Proj

[jira] [Created] (FLINK-27789) LegacySource compatible with overdraft buffer

2022-05-25 Thread fanrui (Jira)
fanrui created FLINK-27789: -- Summary: LegacySource compatible with overdraft buffer Key: FLINK-27789 URL: https://issues.apache.org/jira/browse/FLINK-27789 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-27791) SlotCountExceedingParallelismTest tests failed with NoResourceAvailableException

2022-05-25 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-27791: Summary: SlotCountExceedingParallelismTest tests failed with NoResourceAvailableException Key: FLINK-27791 URL: https://issues.apache.org/jira/browse/FLINK-27791 Proj

[jira] [Created] (FLINK-27792) WindowDistinctAggregateITCase. testTumbleWindow_Rollup failed with InterruptedException

2022-05-25 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-27792: Summary: WindowDistinctAggregateITCase. testTumbleWindow_Rollup failed with InterruptedException Key: FLINK-27792 URL: https://issues.apache.org/jira/browse/FLINK-27792

Re: [DISCUSS] FLIP-227: Support overdraft buffer

2022-05-25 Thread rui fan
Hi everyone, Offline Confirmed with Anton. He has replied in an earlier email: "I vote for 5 as 'max-overdraft-buffers-per-gate'." So as we understand everybody agrees this. This FLIP-227 discussion is over, I've updated FLIP-227. It will be split into 3 tickets to complete. Thanks for all the d

[jira] [Created] (FLINK-27793) After upgrading to flink-1.15.0, flink jobmanager cannot shutdown

2022-05-25 Thread jinshuangxian (Jira)
jinshuangxian created FLINK-27793: - Summary: After upgrading to flink-1.15.0, flink jobmanager cannot shutdown Key: FLINK-27793 URL: https://issues.apache.org/jira/browse/FLINK-27793 Project: Flink

[jira] [Created] (FLINK-27794) The primary key obtained from MySQL is incorrect by using MysqlCatalog

2022-05-25 Thread dusukang (Jira)
dusukang created FLINK-27794: Summary: The primary key obtained from MySQL is incorrect by using MysqlCatalog Key: FLINK-27794 URL: https://issues.apache.org/jira/browse/FLINK-27794 Project: Flink

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

2022-05-25 Thread Alexander Smirnov
Hi Qingsheng and devs, I like the new concept of partial / full caching. Thanks for your efforts, Qingsheng! > The schedule-with-delay idea looks reasonable to me, but I think we need to > redesign the builder API of full caching to make it more descriptive > for > developers. I've got the acc

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

2022-05-25 Thread Becket Qin
Hi Qingsheng, Thanks for updating the FLIP. A few comments / questions below: 1. Is there a reason that we have both "XXXFactory" and "XXXProvider". What is the difference between them? If they are the same, can we just use XXXFactory everywhere? 2. Regarding the FullCachingLookupProvider, shoul

[VOTE] FLIP-227: Support overdraft buffer

2022-05-25 Thread rui fan
Hi, everyone, Thanks for your feedback for FLIP-227: Support overdraft buffer[1] on the discussion thread[2]. I'd like to start a vote for it. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-

[jira] [Created] (FLINK-27795) SqlValidatorException when insert null value

2022-05-25 Thread linqichen (Jira)
linqichen created FLINK-27795: - Summary: SqlValidatorException when insert null value Key: FLINK-27795 URL: https://issues.apache.org/jira/browse/FLINK-27795 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-235: Hybrid Shuffle Mode

2022-05-25 Thread weijie guo
Hi All, Thanks for all the feedback about this FLIP. Since there are no other concerns, this FLIP-235 discussion is over. I will open a vote today. Best regards, Weijie Xintong Song 于2022年5月25日周三 22:17写道: > Ok, I think we are on the same page. I'm aware of > ExecutionConfig#setExecutionMod

Re: [VOTE] FLIP-224: Blocklist Mechanism

2022-05-25 Thread Xintong Song
Thanks for driving this effort, Lijie. I think a nice addition would be to make this feature accessible directly from webui. However, there's no reason to block this FLIP on it. So +1 (binding) from my side. Best, Xintong On Fri, May 20, 2022 at 12:57 PM Lijie Wang wrote: > Hi everyone, >

[jira] [Created] (FLINK-27796) Document approaches to updating existing instances of stateful functions

2022-05-25 Thread Zixuan Rao (Jira)
Zixuan Rao created FLINK-27796: -- Summary: Document approaches to updating existing instances of stateful functions Key: FLINK-27796 URL: https://issues.apache.org/jira/browse/FLINK-27796 Project: Flink

Performance benchmarking for flink.

2022-05-25 Thread Sriram Ganesh
Hi Everyone, I am looking for industry-standard performance benchmarking for flink for both stateless and stateful data processing for streaming. I came across the yahoo performance benchmarking tool. And flink-benchmarks repo (https://github.com/apache/flink-benchmarks). Can anyone suggest to m

flinksql关联hive维表java.lang.UnsupportedOperationException错误

2022-05-25 Thread hdxg1101300...@163.com
你好: 我在使用flink1.12.4版本和hive1.1.0版本时遇到下面的错误: 场景时使用hive的catalog管理元数据,用flinksql把Kafka的数据注册成输入表,关联hive维表做数据拉宽;提交任务到yarn时遇到如下错误; sql如下: create view if not exists dwm_ai_robot_contact_view as select CALLER,CALLEE,CUST_LEVEL,PRO_ID,REASON_CODE,INTENT_NAME,GEN_CENTER,BUS_PRO_ID from realt

Re: flinksql关联hive维表java.lang.UnsupportedOperationException错误

2022-05-25 Thread Jingsong Li
如果flink-table-legacy包没用到,把它删掉即可修复你的问题 Best, Jingsong On Thu, May 26, 2022 at 2:16 PM hdxg1101300...@163.com < hdxg1101300...@163.com> wrote: > 你好: > 我在使用flink1.12.4版本和hive1.1.0版本时遇到下面的错误: > > > 场景时使用hive的catalog管理元数据,用flinksql把Kafka的数据注册成输入表,关联hive维表做数据拉宽;提交任务到yarn时遇到如下错误; >sql如下: >

Re: Performance benchmarking for flink.

2022-05-25 Thread Jingsong Li
You can take a look to https://github.com/nexmark/nexmark Best, Jingsong On Thu, May 26, 2022 at 2:04 PM Sriram Ganesh wrote: > Hi Everyone, > > I am looking for industry-standard performance benchmarking for flink for > both stateless and stateful data processing for streaming. > > I came acro

Re: Performance benchmarking for flink.

2022-05-25 Thread Sriram Ganesh
Thanks a lot Jingsong Li. Will try it out. If you can share information like did you it in Alibaba that would be great. On Thu, May 26, 2022 at 11:52 AM Jingsong Li wrote: > You can take a look to https://github.com/nexmark/nexmark > > Best, > Jingsong > > On Thu, May 26, 2022 at 2:04 PM Sriram

Re: Re: flinksql关联hive维表java.lang.UnsupportedOperationException错误

2022-05-25 Thread hdxg1101300...@163.com
flink-table-legacy 这个模块应该没用,但是我也没找到他在那,是一个独立的模块吗? hdxg1101300...@163.com 发件人: Jingsong Li 发送时间: 2022-05-26 14:20 收件人: dev 主题: Re: flinksql关联hive维表java.lang.UnsupportedOperationException错误 如果flink-table-legacy包没用到,把它删掉即可修复你的问题 Best, Jingsong On Thu, May 26, 2022 at 2:16 PM hdxg1101300...@16

Re: Re: flinksql关联hive维表java.lang.UnsupportedOperationException错误

2022-05-25 Thread Jingsong Li
Please don't use Chinese on the dev mailing list to discuss issues, I've replied on user...@flink.apache.org. Best, Jingsong On Thu, May 26, 2022 at 2:43 PM hdxg1101300...@163.com < hdxg1101300...@163.com> wrote: > flink-table-legacy 这个模块应该没用,但是我也没找到他在那,是一个独立的模块吗? > > ---