Re: [DISCUSS] Deprecate SourceFunction APIs

Hi everyone, very interesting thread. The proposal for deprecation seems to have sparked a very important discussion. Do we what users struggle with specifically? Speaking for myself, when I upgrade flink-faker to the new Source API an unbounded version of the NumberSequenceSource would have been

[jira] [Created] (FLINK-27901) support TableEnvironment.create(configuration) in PyFlink

Juntao Hu created FLINK-27901: - Summary: support TableEnvironment.create(configuration) in PyFlink Key: FLINK-27901 URL: https://issues.apache.org/jira/browse/FLINK-27901 Project: Flink Issue Typ

Re: [DISCUSS] Bi-Weekly Flink Community Sync Meeting

Hi Robert, Hi everyone, I am still skeptical about this. Generally, orgs of any kind tend to alternate between "can we do a meeting for this?" and "why don't we do this async?". For anything that goes towards a status update or information sharing, asynchronous, written communication is just more

[jira] [Created] (FLINK-27902) Refactor ResultPartitionType to decouple scheduling and partition release logic

Weijie Guo created FLINK-27902: -- Summary: Refactor ResultPartitionType to decouple scheduling and partition release logic Key: FLINK-27902 URL: https://issues.apache.org/jira/browse/FLINK-27902 Project:

[jira] [Created] (FLINK-27903) Introduce and support HYBRID resultPartitionType

Weijie Guo created FLINK-27903: -- Summary: Introduce and support HYBRID resultPartitionType Key: FLINK-27903 URL: https://issues.apache.org/jira/browse/FLINK-27903 Project: Flink Issue Type: Sub-

[jira] [Created] (FLINK-27904) Introduce HsDataBuffer

Weijie Guo created FLINK-27904: -- Summary: Introduce HsDataBuffer Key: FLINK-27904 URL: https://issues.apache.org/jira/browse/FLINK-27904 Project: Flink Issue Type: Sub-task Reporter:

[jira] [Created] (FLINK-27905) Introduce HsSpillStrategy and Implement SelectiveSpillStrategy

Weijie Guo created FLINK-27905: -- Summary: Introduce HsSpillStrategy and Implement SelectiveSpillStrategy Key: FLINK-27905 URL: https://issues.apache.org/jira/browse/FLINK-27905 Project: Flink I

[jira] [Created] (FLINK-27907) Abstract and implement disk read and write logic for hybrid shuffle

Weijie Guo created FLINK-27907: -- Summary: Abstract and implement disk read and write logic for hybrid shuffle Key: FLINK-27907 URL: https://issues.apache.org/jira/browse/FLINK-27907 Project: Flink

[jira] [Created] (FLINK-27906) Introduce HsDataIndex

Weijie Guo created FLINK-27906: -- Summary: Introduce HsDataIndex Key: FLINK-27906 URL: https://issues.apache.org/jira/browse/FLINK-27906 Project: Flink Issue Type: Sub-task Reporter:

[jira] [Created] (FLINK-27908) Introduce HsResultPartition and HsSubpartitionView

Weijie Guo created FLINK-27908: -- Summary: Introduce HsResultPartition and HsSubpartitionView Key: FLINK-27908 URL: https://issues.apache.org/jira/browse/FLINK-27908 Project: Flink Issue Type: Su

[jira] [Created] (FLINK-27909) Add document and tests for hybrid shuffle mode

Weijie Guo created FLINK-27909: -- Summary: Add document and tests for hybrid shuffle mode Key: FLINK-27909 URL: https://issues.apache.org/jira/browse/FLINK-27909 Project: Flink Issue Type: Sub-ta

Re: [DISCUSS] FLIP-228: Support Within between events in CEP Pattern

Hi Nicholas, Thanks a lot for the update. Regarding the pattern API, should we also introduce APIs such as Pattern.times(int from, int to, Time windowTime) to indicate the time interval between events matched in the loop? Regarding the naming of the classes, does it make sense to rename `WithinT

[DISCUSS] Api Since Version Annotation

Hi everyone, I'm a bit curious why we don't use the api since annotation. I found this annotation very useful when I am using other dependencies, it tells me very directly the version of the API and which API I used would break the compatibility of which version. For example, String in JDK. ```

Re: [DISCUSS] Api Since Version Annotation

+1 I think @since annotation would be super helpful when upgrading Flink versions. BTW, I proposed to add @since annotation to configuration options as well [1]. [1] https://issues.apache.org/jira/browse/FLINK-23700 Best, Paul Lam > 2022年6月6日 15:49，Jingsong Li 写道： > > Hi everyone, > > I'm

Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint

Hi. Godfrey. Nice to hear the comments from you. > Could you give a whole architecture about the Ecosystem of HiveServers > and the SqlGateway, such as JDBC driver, Beeline, etc. > Which is more clear for users. Yes. I have updated the FLIP and added the architecture of the Gateway with the Hive

[jira] [Created] (FLINK-27910) FileSink not registered the timer to enforce rolling policy if started from scratch

Yun Gao created FLINK-27910: --- Summary: FileSink not registered the timer to enforce rolling policy if started from scratch Key: FLINK-27910 URL: https://issues.apache.org/jira/browse/FLINK-27910 Project: Fl

[jira] [Created] (FLINK-27911) Set HDFS LEASE_TIMEOUT as user-configurable

zoucao created FLINK-27911: -- Summary: Set HDFS LEASE_TIMEOUT as user-configurable Key: FLINK-27911 URL: https://issues.apache.org/jira/browse/FLINK-27911 Project: Flink Issue Type: Improvement A

Re: [DISCUSS] FLIP-231: Introduce SupportStatisticReport to support reporting statistics from source connectors

Hi, everyone. Thanks for all the inputs. Since there is no feedback any more, I will start the vote tomorrow. Best, Godfrey godfrey he 于2022年6月1日周三 13:30写道： > > Hi, Jing. > > Thanks for the suggestion, I have updated the doc and > will continue to optimize the code in subsequent PR. > > Best, >

Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint

Hi, Shengkai. Thanks for the update, LGTM now. Best, Godfrey Shengkai Fang 于2022年6月6日周一 16:47写道： > > Hi. Godfrey. > > Nice to hear the comments from you. > > > Could you give a whole architecture about the Ecosystem of HiveServers > > and the SqlGateway, such as JDBC driver, Beeline, etc. > >

[jira] [Created] (FLINK-27912) Improve operator config names

Gyula Fora created FLINK-27912: -- Summary: Improve operator config names Key: FLINK-27912 URL: https://issues.apache.org/jira/browse/FLINK-27912 Project: Flink Issue Type: Improvement C

[jira] [Created] (FLINK-27913) Remove savepointHistoryMaxCount and savepointHistoryMaxAge from FlinkOperatorConfiguration

Gyula Fora created FLINK-27913: -- Summary: Remove savepointHistoryMaxCount and savepointHistoryMaxAge from FlinkOperatorConfiguration Key: FLINK-27913 URL: https://issues.apache.org/jira/browse/FLINK-27913

[jira] [Created] (FLINK-27914) Integrate JOSDK metrics with Flink Metrics reporter

Gyula Fora created FLINK-27914: -- Summary: Integrate JOSDK metrics with Flink Metrics reporter Key: FLINK-27914 URL: https://issues.apache.org/jira/browse/FLINK-27914 Project: Flink Issue Type: N

Re: 邮件退订

邮件退订 张立志 于2022年6月5日周日 15:14写道： > 邮件退订 > > > > | | > zh_ha...@163.com > | > | > 邮箱：zh_ha...@163.com > | > > > > > 回复的原邮件 > | 发件人 | 米子日匀 | > | 日期 | 2022年06月02日 22:05 | > | 收件人 | dev@flink.apache.org | > | 抄送至 | | > | 主题 | 邮件退订 | > > > 邮件退订 > > > > > > > > > >

How to schedule stateful function in flink

I have a use case in which I have to schedule a stateful function for a certain frequency in a day , so that I can perform business functions on all the data stored in state till data time. Thanks.

[jira] [Created] (FLINK-27915) WindowJoinITCase.testInnerJoinOnWTFWithOffset failed with InterruptedException

Huang Xingbo created FLINK-27915: Summary: WindowJoinITCase.testInnerJoinOnWTFWithOffset failed with InterruptedException Key: FLINK-27915 URL: https://issues.apache.org/jira/browse/FLINK-27915 Projec

(无主题)

(无主题)

[jira] [Created] (FLINK-27916) HybridSourceReaderTest.testReader failed with AssertionError

Huang Xingbo created FLINK-27916: Summary: HybridSourceReaderTest.testReader failed with AssertionError Key: FLINK-27916 URL: https://issues.apache.org/jira/browse/FLINK-27916 Project: Flink

[jira] [Created] (FLINK-27917) PulsarUnorderedPartitionSplitReaderTest.consumeMessageCreatedBeforeHandleSplitsChangesAndResetToEarliestPosition failed with AssertionError

Huang Xingbo created FLINK-27917: Summary: PulsarUnorderedPartitionSplitReaderTest.consumeMessageCreatedBeforeHandleSplitsChangesAndResetToEarliestPosition failed with AssertionError Key: FLINK-27917 URL: https:/

[jira] [Created] (FLINK-27918) SavepointITCase.testStopWithSavepointFailingAfterSnapshotCreation failed with Expected RuntimeException after snapshot creation

Huang Xingbo created FLINK-27918: Summary: SavepointITCase.testStopWithSavepointFailingAfterSnapshotCreation failed with Expected RuntimeException after snapshot creation Key: FLINK-27918 URL: https://issues.apac

[jira] [Created] (FLINK-27919) Add FLIP-27-based Data Generator Source

Alexander Fedulov created FLINK-27919: - Summary: Add FLIP-27-based Data Generator Source Key: FLINK-27919 URL: https://issues.apache.org/jira/browse/FLINK-27919 Project: Flink Issue Type:

[jira] [Created] (FLINK-27920) Documented enums constant support ExcludeFromDocumentation annotation

Weijie Guo created FLINK-27920: -- Summary: Documented enums constant support ExcludeFromDocumentation annotation Key: FLINK-27920 URL: https://issues.apache.org/jira/browse/FLINK-27920 Project: Flink

[jira] [Created] (FLINK-27921) Introduce the checkResourceRequirementsWithDelay in DeclarativeSlotManager

Aitozi created FLINK-27921: -- Summary: Introduce the checkResourceRequirementsWithDelay in DeclarativeSlotManager Key: FLINK-27921 URL: https://issues.apache.org/jira/browse/FLINK-27921 Project: Flink

Re: [DISCUSS] Deprecate SourceFunction APIs

> > David, can you elaborate why you need watermark generation in the source > for your data generators? The training exercises should strive to provide examples of best practices. If the exercises and their solutions use env.fromSource(source, WatermarkStrategy.noWatermarks(), "name-of-source")

[jira] [Created] (FLINK-27922) Flink SQL unique key lost when set table.exec.mini-batch.enabled=true

zhangbin created FLINK-27922: Summary: Flink SQL unique key lost when set table.exec.mini-batch.enabled=true Key: FLINK-27922 URL: https://issues.apache.org/jira/browse/FLINK-27922 Project: Flink

Re: [DISCUSS] Deprecate SourceFunction APIs

One question from my side: As SourceFunction a @Public interface, we cannot remove it before doing a major version bump (Flink 2.0). Of course it's not a blocker to make such deprecation and let the new interface step in. My question is whether we have a plan to finally remove the deprecated inte

Re: [DISCUSS] Deprecate SourceFunction APIs

The starting point of the version bump and removal question is that downstream projects may experience a tough time to adapt new interfaces while Flink keeps in 1.x versions so that users may expect it as an easy task. From my experience, it's really challenge to maintain compatibility between mult

Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint

> > Hi, everyone. > > Thanks for all the input. Hope my feedback can address most of the > questions. > > Since the discussion has lasted for a long time, I think we have reached a > consensus about the > design. If the discussion doesn't get more responses, I think we can start > the vote tomorrow

Re: [DISCUSS] Deprecate SourceFunction APIs

Well. It's a bit off-topic. For deprecating SourceFunction as FLIP-27 series works go ahead, +1 from my side. It's a significant work towards the unification of batch and streaming effort :) Best, tison. tison 于2022年6月6日周一 21:54写道： > The starting point of the version bump and removal question

[jira] [Created] (FLINK-27923) Typo fix for release-1.0.0 quick-start.md

James Busche created FLINK-27923: Summary: Typo fix for release-1.0.0 quick-start.md Key: FLINK-27923 URL: https://issues.apache.org/jira/browse/FLINK-27923 Project: Flink Issue Type: Bug

Re: [DISCUSS] Deprecate SourceFunction APIs

Thanks for the nice discussion all. I was recently trying to implement a very simple polling source and would've loved a higher-level base to work from. I'm wondering if in addition to the data generator use cases, it would be good to support a simple non-parallel polling abstraction to make it ea

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

Thank you for all your efforts! Thanks John On Sun, Jun 5, 2022 at 10:33 PM Aitozi wrote: > Thanks Yang and Nice to see it happen. > > Best, > Aitozi. > > Yang Wang 于2022年6月5日周日 16:14写道： > >> The Apache Flink community is very happy to announce the release of >> Apache Flink Kubernetes Operato

[jira] [Created] (FLINK-27924) Add pulsar comments in the datastream package

LuNng Wang created FLINK-27924: -- Summary: Add pulsar comments in the datastream package Key: FLINK-27924 URL: https://issues.apache.org/jira/browse/FLINK-27924 Project: Flink Issue Type: Improve

[jira] [Created] (FLINK-27925) Avoid to create watcher without the resourceVersion

Aitozi created FLINK-27925: -- Summary: Avoid to create watcher without the resourceVersion Key: FLINK-27925 URL: https://issues.apache.org/jira/browse/FLINK-27925 Project: Flink Issue Type: Improveme

[VOTE] FLIP-231: Introduce SupportStatisticReport to support reporting statistics from source connectors

Hi everyone, Thanks for all the feedback so far. Based on the discussion[1] we seem to have consensus, so I would like to start a vote on FLIP-231 for which the FLIP has now also been updated[2]. The vote will last for at least 72 hours (Jun 10th 12:00 GMT) unless there is an objection or insuffi

Re: [DISCUSS] FLIP-224: Blacklist Mechanism

Hi Chesnay, Would you please take a look at the FLIP and discussion to see if all your concerns have been addressed? Thanks, Zhu Zhu Zhu 于2022年5月28日周六 13:26写道： > > Regarding the concern of the SlotManager, my two cents: > 1. it is necessary for the SlotManager to host blocked slots, in 2 cases:

[jira] [Created] (FLINK-27926) Translate ```upgrading.md``` into chinese

Zixuan Rao created FLINK-27926: -- Summary: Translate ```upgrading.md``` into chinese Key: FLINK-27926 URL: https://issues.apache.org/jira/browse/FLINK-27926 Project: Flink Issue Type: Improvement

[VOTE] FLIP-234: Support Retryable Lookup Join To Solve Delayed Updates Issue In External Systems

Dear Flink developers, Thanks for all your feedback for FLIP-234: Support Retryable Lookup Join To Solve Delayed Updates Issue In External Systems[1] on the discussion thread[2]. I'd like to start a vote for it. The vote will be open for at least 72 hours unless there is an objection or not enoug

Re: [DISCUSS] FLIP-234: Support Retryable Lookup Join To Solve Delayed Updates Issue In External Systems

Hi everyone, I started a vote for this FLIP [1], please vote there or ask additional questions here. [2] [1] https://lists.apache.org/thread/bb0kqjs8co3hhmtklmwptws4fc4rz810 [2] https://lists.apache.org/thread/9k1sl2519kh2n3yttwqc00p07xdfns3h Best, Lincoln Lee Lincoln Lee 于2022年6月2日周四 15:51写

[DISCUSS] FLIP-237: Thrift Format Support in Flink

Hi there, I want to kick off the first round of FLIP-237 : Thrift Format Support discussion. Notice for the area marked as WIP, we are looking for more feedback from folks, those areas would either stay in the sco

[jira] [Created] (FLINK-27927) Adjust table store connector common apis

Jingsong Lee created FLINK-27927: Summary: Adjust table store connector common apis Key: FLINK-27927 URL: https://issues.apache.org/jira/browse/FLINK-27927 Project: Flink Issue Type: Improvem

[VOTE] FLIP-223: Support HiveServer2 Endpoint

Hi, everyone. Thanks for all feedback for FLIP-223: Support HiveServer2 Endpoint[1] on the discussion thread[2]. I'd like to start a vote for it. The vote will be open for at least 72 hours unless there is an objection or not enough votes. Best, Shengkai [1] https://cwiki.apache.org/confluence/

Re: [VOTE] FLIP-231: Introduce SupportStatisticReport to support reporting statistics from source connectors

Hi Godfrey, +1 (non-binding) Best regards, Jing On Tue, Jun 7, 2022 at 4:42 AM godfrey he wrote: > Hi everyone, > > Thanks for all the feedback so far. Based on the discussion[1] we seem > to have consensus, so I would like to start a vote on FLIP-231 for > which the FLIP has now also been up

[jira] [Created] (FLINK-27928) External Resource Framework: 'external-resources' config delimiter is not working as expected

James Cho created FLINK-27928: - Summary: External Resource Framework: 'external-resources' config delimiter is not working as expected Key: FLINK-27928 URL: https://issues.apache.org/jira/browse/FLINK-27928

Re: [DISCUSS] FLIP-237: Thrift Format Support in Flink

Hi Chen, Thanks for driving this! Afaik, the community has the consensus to *Start a [DISCUSS] thread on the Apache mailing list*[1]. I just walked through some existing FLIPs and didn't find any that have been using google doc as the discussion thread. Would you like to follow the current process

[jira] [Created] (FLINK-27929) Drop support for python 3.6

LuNng Wang created FLINK-27929: -- Summary: Drop support for python 3.6 Key: FLINK-27929 URL: https://issues.apache.org/jira/browse/FLINK-27929 Project: Flink Issue Type: Improvement Com