RE: [DISCUSS] FLIP-212: Introduce Flink Kubernetes Operator

Hi, regarding the construction of k8s Flink Operator, I have already completed some functions. I hope to contribute this part of the functions and discuss with the community how to improve it. How should I start? So far I have seen that the component has no operation persistence. Should we persist

Re: question about StatefunContext in golang Statefun SDK

Hmm ... a downside to my proposal is that Go contexts are supposed to be immutable, i.e. when adding a custom value to a context, a new context is created with the new value and the old context isn't changed. Changing the context.Context associated with the statefun.Context sort of goes against the

[jira] [Created] (FLINK-26257) Document metrics configuration for Prometheus

Matyas Orhidi created FLINK-26257: - Summary: Document metrics configuration for Prometheus Key: FLINK-26257 URL: https://issues.apache.org/jira/browse/FLINK-26257 Project: Flink Issue Type: S

Re: question about StatefunContext in golang Statefun SDK

An example of passing it around would be: func (f *MyFunc) Invoke(ctx statefun.Context, message statefun.Message) error { logger := NewLogger() ctx.SetContext(ctxzap.ToContext(ctx, logger)) return SomeOtherFunc(ctx) } func SomeOtherFunc(ctx context.Context) error { logger := ct

Re: question about StatefunContext in golang Statefun SDK

Ha, our emails keep passing. I've been playing around with options locally, and the SetContext option seems to be the most flexible (and non-breaking), imo. The implementation would be trivial, just add: SetContext(ctx context.Context) ... to the statefun.Context interface, which is implemented

Re: question about StatefunContext in golang Statefun SDK

Austin -- what would you think about an option to add SetContext(ctx context.Context) as a method to the statefun.Context interface? Then one could do something like this: func (f *MyFunc) Invoke(ctx statefun.Context, message statefun.Message) error { logger := NewLogger() ctx.SetContext(

Re: question about StatefunContext in golang Statefun SDK

It would be helpful to have a small example though, if you have on Galen, to see how you're passing it around. On Fri, Feb 18, 2022 at 11:10 AM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Looking through the statefun Context interface, it indeed doesn't give > access to the underlyi

Re: question about StatefunContext in golang Statefun SDK

Looking through the statefun Context interface, it indeed doesn't give access to the underlying context.Context and the only implementation is package-private [1]. I don't think there would be a way to update the statfun.Context interface without introducing breaking changes, but if we were to make

Re: question about StatefunContext in golang Statefun SDK

Sorry Austin, I didn't see your response before I replied. Yes, we're saying the same thing. On Fri, Feb 18, 2022 at 10:56 AM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hey all, jumping in. This makes sense to me – for instance to attach a > logger with some common metadata, e.g tr

Re: question about StatefunContext in golang Statefun SDK

Sure, you could do that. But it's a common pattern in Go to attach things like loggers, request ids, etc. to the context so that they don't have to be explicitly passed around. It would be convenient to be able to follow this pattern with Statefun. On Fri, Feb 18, 2022 at 10:53 AM Till Rohrmann w

Re: question about StatefunContext in golang Statefun SDK

Hey all, jumping in. This makes sense to me – for instance to attach a logger with some common metadata, e.g trace ID for the request? This is common in go to add arbitrary items without updating the method signatures, similar to thread local storage in Java. On Fri, Feb 18, 2022 at 10:53 AM Till

Re: question about StatefunContext in golang Statefun SDK

Thanks for the clarification Galen. If you call the other Go functions, then you could also pass the other values as separate arguments to these functions, can't you? Cheers, Till On Fri, Feb 18, 2022 at 3:31 PM Galen Warren wrote: > The former. > > I think there's potential for confusion here

Re: [ANNOUNCE] Apache Flink 1.13.6 released

Thanks Konstantin for being our release manager and also thanks a lot to everyone who contributed to the release! Cheers, Till On Fri, Feb 18, 2022 at 2:54 PM Johannes Moser wrote: > Thanks. Great job. > > > On 18.02.2022, at 14:36, Martijn Visser wrote: > > > > Hi Konstantin, > > > > Thanks f

[jira] [Created] (FLINK-26256) AWS SDK Async Event Loop Group Classloader Issue

Danny Cranmer created FLINK-26256: - Summary: AWS SDK Async Event Loop Group Classloader Issue Key: FLINK-26256 URL: https://issues.apache.org/jira/browse/FLINK-26256 Project: Flink Issue Type

Re: [DISCUSS] Plan to externalize connectors and versioning

Why do you want to immediately do a release for the elasticsearch connector? What does that provide us? I'd rather first have a fully working setup and integrated documentation before thinking about releasing anything. Once we have that we may be able to externalize all connectors within 1 rel

[jira] [Created] (FLINK-26255) SplitAggregateITCase.testAggWithJoin failed on azure

Roman Khachatryan created FLINK-26255: - Summary: SplitAggregateITCase.testAggWithJoin failed on azure Key: FLINK-26255 URL: https://issues.apache.org/jira/browse/FLINK-26255 Project: Flink

Re: question about StatefunContext in golang Statefun SDK

The former. I think there's potential for confusion here because we're using the word *function *in a couple of senses. One sense is a *stateful function*; another sense is a *Go function*. What I'm looking to do is to put values in the Context so that downstream Go functions that receive the con

Re: [DISCUSS] Plan to externalize connectors and versioning

+1 (non-binding) > On 18.02.2022, at 15:12, Konstantin Knauf wrote: > > +1 to the approach. > > I expect that we will encounter more questions and challenges as we go, > but these are best discussed and addressed in the context of a specific > connector like ElasticSearch. > > On Fri, Feb 18,

[jira] [Created] (FLINK-26254) KafkaSink might violate order of sequence numbers and risk exactly-once processing

Fabian Paul created FLINK-26254: --- Summary: KafkaSink might violate order of sequence numbers and risk exactly-once processing Key: FLINK-26254 URL: https://issues.apache.org/jira/browse/FLINK-26254 Proj

Re: [DISCUSS] Plan to externalize connectors and versioning

+1 to the approach. I expect that we will encounter more questions and challenges as we go, but these are best discussed and addressed in the context of a specific connector like ElasticSearch. On Fri, Feb 18, 2022 at 2:54 PM Martijn Visser wrote: > Hi everyone, > > As a follow-up to earlier d

Re: Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang

Congratulations! Best, Jing Zhang Benchao Li 于2022年2月18日周五 20:15写道： > Congratulations! > > Jing Ge 于2022年2月18日周五 17:03写道： > > > Congrats! Very well deserved! > > > > Best regards, > > Jing > > > > On Fri, Feb 18, 2022 at 9:16 AM Yu Li wrote: > > > > > Congratulations! > > > > > > Best Regards

Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao

Congratulations! Best, Jing Zhang Benchao Li 于2022年2月18日周五 20:22写道： > Congratulations! > > Yu Li 于2022年2月18日周五 19:34写道： > > > Congratulations! > > > > Best Regards, > > Yu > > > > > > On Fri, 18 Feb 2022 at 14:19, godfrey he wrote: > > > > > Congratulations! > > > > > > Best, > > > Godfrey >

[jira] [Created] (FLINK-26253) Speed up detection fo dropped message in MessageSerializationTest

Chesnay Schepler created FLINK-26253: Summary: Speed up detection fo dropped message in MessageSerializationTest Key: FLINK-26253 URL: https://issues.apache.org/jira/browse/FLINK-26253 Project: Fl

[DISCUSS] Plan to externalize connectors and versioning

Hi everyone, As a follow-up to earlier discussions [1] [2] to externalize the connectors from the Flink repository, I would like to propose a plan to externalize these connectors. The goal of this plan is to start with moving connectors to its own repositories without introducing regressions for c

Re: [ANNOUNCE] Apache Flink 1.13.6 released

Thanks. Great job. > On 18.02.2022, at 14:36, Martijn Visser wrote: > > Hi Konstantin, > > Thanks for driving the release. Thanks to everyone that has contributed > with bug fixes and other improvements! > > Best regards, > > Martijn Visser > https://twitter.com/MartijnVisser82 > > > On Fri

[jira] [Created] (FLINK-26252) Refactor MiniClusterExtension

Francesco Guardiani created FLINK-26252: --- Summary: Refactor MiniClusterExtension Key: FLINK-26252 URL: https://issues.apache.org/jira/browse/FLINK-26252 Project: Flink Issue Type: Bug

Re: [ANNOUNCE] Apache Flink 1.13.6 released

Hi Konstantin, Thanks for driving the release. Thanks to everyone that has contributed with bug fixes and other improvements! Best regards, Martijn Visser https://twitter.com/MartijnVisser82 On Fri, 18 Feb 2022 at 14:32, Konstantin Knauf wrote: > The Apache Flink community is very happy to a

[ANNOUNCE] Apache Flink 1.13.6 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.13.6, which is the fifth bugfix release for the Apache Flink 1.13 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming a

Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao

Congratulations! Yu Li 于2022年2月18日周五 19:34写道： > Congratulations! > > Best Regards, > Yu > > > On Fri, 18 Feb 2022 at 14:19, godfrey he wrote: > > > Congratulations! > > > > Best, > > Godfrey > > > > Lincoln Lee 于2022年2月18日周五 14:07写道： > > > > > > Congratulations! > > > > > > Best, > > > Lincoln

Re: Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang

Congratulations! Jing Ge 于2022年2月18日周五 17:03写道： > Congrats! Very well deserved! > > Best regards, > Jing > > On Fri, Feb 18, 2022 at 9:16 AM Yu Li wrote: > > > Congratulations! > > > > Best Regards, > > Yu > > > > > > On Fri, 18 Feb 2022 at 14:19, godfrey he wrote: > > > > > Congratulations! >

Re: question about StatefunContext in golang Statefun SDK

Hi Galen, Am I understanding it correctly, that you would like to set some values in the Context of function A that is then accessible in a downstream call of function B? Or would you like to set a value that is accessible once function A is called again (w/ or w/o the same id)? Cheers, Till On

[jira] [Created] (FLINK-26251) [JUnit5 Migration] Module: flink-rpc-akka

Chesnay Schepler created FLINK-26251: Summary: [JUnit5 Migration] Module: flink-rpc-akka Key: FLINK-26251 URL: https://issues.apache.org/jira/browse/FLINK-26251 Project: Flink Issue Type:

Re: [ANNOUNCE] Flink 1.15 Feature Freeze

Thanks for letting us know Wenlong. Since the SQL client testing hasn't really started yet, I think it is ok to merge this PR. Cheers, Till On Thu, Feb 17, 2022 at 2:32 PM wenlong.lwl wrote: > Hi,all, I want to merge a pr (https://github.com/apache/flink/pull/18363) > belonging to FLIP-190, whi

Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao

Congratulations! Best Regards, Yu On Fri, 18 Feb 2022 at 14:19, godfrey he wrote: > Congratulations! > > Best, > Godfrey > > Lincoln Lee 于2022年2月18日周五 14:07写道： > > > > Congratulations! > > > > Best, > > Lincoln Lee > > > > > > Jingsong Li 于2022年2月18日周五 12:42写道： > > > > > Congratulations! > >

[jira] [Created] (FLINK-26250) Document the upgrade story

Timo Walther created FLINK-26250: Summary: Document the upgrade story Key: FLINK-26250 URL: https://issues.apache.org/jira/browse/FLINK-26250 Project: Flink Issue Type: Sub-task Com

[jira] [Created] (FLINK-26249) Run BuiltInFunctionITCase tests in parallel

Francesco Guardiani created FLINK-26249: --- Summary: Run BuiltInFunctionITCase tests in parallel Key: FLINK-26249 URL: https://issues.apache.org/jira/browse/FLINK-26249 Project: Flink Iss

Re: [NOTICE] Pulsar tests disabled

The Pulsar tests are too weak now and lead to Flink CI instabilities. There’re 10+ bug issues[1] for Pulsar connector with various error causes. [1] https://issues.apache.org/jira/issues/?filter=12351366 > 2022年2月18日 下午4:34，Yufei Zhan

Re: [NOTICE] Pulsar tests disabled

Hi, Here are some updates regarding the ongoing pulsar test instabilities. The following issues are believed to be caused by PulsarEmbeddedRuntime or mockRuntime, and currently, we have a PR https://github.com/apache/flink/pull/18838/commits to

[jira] [Created] (FLINK-26248) flink job not recover from latest checkpoint on native k8s

zhisheng created FLINK-26248: Summary: flink job not recover from latest checkpoint on native k8s Key: FLINK-26248 URL: https://issues.apache.org/jira/browse/FLINK-26248 Project: Flink Issue Type

Re: Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang

Congrats! Very well deserved! Best regards, Jing On Fri, Feb 18, 2022 at 9:16 AM Yu Li wrote: > Congratulations! > > Best Regards, > Yu > > > On Fri, 18 Feb 2022 at 14:19, godfrey he wrote: > > > Congratulations! > > > > Best, > > Godfrey > > > > Lincoln Lee 于2022年2月18日周五 14:05写道： > > > > > >

[jira] [Created] (FLINK-26247) Introduce a better expire strategy

Jingsong Lee created FLINK-26247: Summary: Introduce a better expire strategy Key: FLINK-26247 URL: https://issues.apache.org/jira/browse/FLINK-26247 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-26246) Setup JUnit 5 infrastructure for parallel test execution

Chesnay Schepler created FLINK-26246: Summary: Setup JUnit 5 infrastructure for parallel test execution Key: FLINK-26246 URL: https://issues.apache.org/jira/browse/FLINK-26246 Project: Flink

[jira] [Created] (FLINK-26245) flink-runtime-web submit showPlan not working

Junhan Yang created FLINK-26245: --- Summary: flink-runtime-web submit showPlan not working Key: FLINK-26245 URL: https://issues.apache.org/jira/browse/FLINK-26245 Project: Flink Issue Type: Bug

Re: [NOTICE] Pulsar tests disabled

Hi, Sorry for any inconvenience and late reply, after some investigation, the test instabilities are believed to be caused by the newly introduced PulsarEmbedRuntime, and an alternative to it is to use the PulsarContainerRuntime which runs in testcontainer. A pr is being submitted to resolve the

Re: Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang

Congratulations! Best Regards, Yu On Fri, 18 Feb 2022 at 14:19, godfrey he wrote: > Congratulations! > > Best, > Godfrey > > Lincoln Lee 于2022年2月18日周五 14:05写道： > > > > Congratulations Feng and Zhipeng! > > > > Best, > > Lincoln Lee > > > > > > Yuepeng Pan 于2022年2月18日周五 12:45写道： > > > > > > >

[jira] [Created] (FLINK-26244) [JUnit5 Migration] Module: flink-runtime-web

Chesnay Schepler created FLINK-26244: Summary: [JUnit5 Migration] Module: flink-runtime-web Key: FLINK-26244 URL: https://issues.apache.org/jira/browse/FLINK-26244 Project: Flink Issue Ty

Re: [NOTICE] Pulsar tests disabled

Given these instabilities we might also want to consider excluding it from the 1.15 release, because it is clearly not in a reasonable stable state before the feature freeze. On 18/02/2022 08:58, Chesnay Schepler wrote: FYI: I just disabled all Pulsar tests on master in 296c3be7576940d6bd76f4d