Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)

2023-04-27 Thread Thomas Weise
Is the intention to bump the Flink major version and only support Java 17+? If so, can Scala not be upgraded at the same time? Thanks, Thomas On Thu, Apr 27, 2023 at 4:53 PM Martijn Visser wrote: > Scala 2.12.7 doesn't compile on Java 17, see > https://issues.apache.org/jira/browse/FLINK-25000

Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

2022-06-30 Thread Thomas Weise
Hi Mason, I added mason6345 to the Flink confluence space, you should be able to add a FLIP now. Looking forward to the contribution! Thomas On Thu, Jun 30, 2022 at 9:25 AM Martijn Visser wrote: > > Hi Mason, > > I'm sure there's a PMC (*hint*) out there who can grant you access to > create a

Re: Issue with HybridSource recovering from Savepoint

2022-05-08 Thread Thomas Weise
One more question: Are you changing the parallelism when resuming from savepoint? On Sun, May 8, 2022 at 4:05 PM Thomas Weise wrote: > > Hi Kevin, > > Unfortunately I did not find a way to test the savepoint scenario with > the MiniCluster. Savepoints are not supported in th

Re: Issue with HybridSource recovering from Savepoint

2022-05-08 Thread Thomas Weise
cible consistently? Can you capture the debug log of the enumerator? Thanks, Thomas On Wed, May 4, 2022 at 10:05 AM Martijn Visser wrote: > > Hi Kevin, > > I'm hoping that @Thomas Weise could help with the issue regarding the > recovery from the savepoint. > > Best regards, >

Re: Flink 1.14.4 HybridSource consumes lots of CPU resources

2022-05-03 Thread Thomas Weise
Thank you for reporting the issue. Mason has already identified the root cause and the JIRA is now assigned to him: https://issues.apache.org/jira/browse/FLINK-27479 Thomas On Tue, May 3, 2022 at 4:02 AM Martijn Visser wrote: > I'm looping in @Thomas Weise since he has expertise

[ANNOUNCE] Apache Flink 1.14.3 released

2022-01-19 Thread Thomas Weise
The Apache Flink community is very happy to announce the release of Apache Flink 1.14.3, which is the second bugfix release for the Apache Flink 1.14 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming

Re: [DISCUSS] Future of Per-Job Mode

2022-01-13 Thread Thomas Weise
Regarding session mode: ## Session Mode * main() method executed in client Session mode also supports execution of the main method on Jobmanager with submission through REST API. That's how Flinkk k8s operators like [1] work. It's actually an important capability because it allows for allocation

Re: Flink native k8s integration vs. operator

2022-01-12 Thread Thomas Weise
;> integration advantages come from the active mode: being able to better > >>> understand the job's resource requirements and adjust the deployment > >>> resource accordingly. Both fine-grained resource management (customizing > >>> TM > >>> r

Re: Flink native k8s integration vs. operator

2022-01-05 Thread Thomas Weise
out-of the box > experience with standalone mode on k8s, that makes the experience as close to > running / operating any other application as possible. > > I'd really appreciate to hear your thoughts on this topic. > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-18

Flink native k8s integration vs. operator

2022-01-03 Thread Thomas Weise
Hi, I was recently looking at the Flink native Kubernetes integration [1] to get an idea how it relates to existing operator based solutions [2], [3]. Part of the native integration's motivations was simplicity (no extra component to install), but arguably that is also a shortcoming. The k8s oper

Re: Re: Dependency injection for TypeSerializer?

2021-11-12 Thread Thomas Weise
like a file and embed the writer schema in the > state, so there is no need for an external registry. That should also work > with dynamic queries. Only when you output messages into an event queue, > you'd need the schema registry but that's orthogonal. > > On W

Re: Re: Dependency injection for TypeSerializer?

2021-11-10 Thread Thomas Weise
>> 2. The important thing here was that all your classes have to be > >> "deployed" on every Task Manager/Job Manager in order to load them through > >> DI. > >> We achieved this by using what is called "Job session" cluster. Where our > >> custom Fli

Dependency injection for TypeSerializer?

2021-11-08 Thread Thomas Weise
Hi, I was looking into a problem that requires a configurable type serializer for communication with a schema registry. The service endpoint can change, so I would not want to make it part of the serializer snapshot but rather resolve it at graph construction time (similar to how a Kafka bootstrap

Re: [ANNOUNCE] Apache Flink 1.13.3 released

2021-10-21 Thread Thomas Weise
Thanks for making the release happen! On Thu, Oct 21, 2021 at 5:54 AM Leonard Xu wrote: > > Thanks to Chesnay & Martijn and everyone who made this release happen. > > > > 在 2021年10月21日,20:08,Martijn Visser 写道: > > > > Thank you Chesnay, Leonard and all contributors! > > > > On Thu, 21 Oct 2021 a

Re: [DISCUSS] Creating an external connector repository

2021-10-20 Thread Thomas Weise
gt; releases > >> > > of connectors, which are not bound to the release cycle of Flink > >> itself. > >> > I > >> > > agree that in order to get there, we need to have stable interfaces > >> which > >> > > are trustwo

Re: [DISCUSS] Creating an external connector repository

2021-10-18 Thread Thomas Weise
Thanks for initiating this discussion. There are definitely a few things that are not optimal with our current management of connectors. I would not necessarily characterize it as a "mess" though. As the points raised so far show, it isn't easy to find a solution that balances competing requiremen

Re: [DISCUSS] FLIP-111: Docker image unification

2020-03-17 Thread Thomas Weise
docker plugin sounds good. I'll take a look at it. > > > > Regarding supporting JAVA 11: > > - Not sure if it is necessary to ship JAVA. Maybe we could just change > > the base image from openjdk:8-jre to openjdk:11-jre in template docker > > file[1]. Correct me i

Re: [DISCUSS] FLIP-111: Docker image unification

2020-03-10 Thread Thomas Weise
Thanks for working on improvements to the Flink Docker container images. This will be important as more and more users are looking to adopt Kubernetes and other deployment tooling that relies on Docker images. A generic, dynamic configuration mechanism based on environment variables is essential a

Re: [DISCUSS] Support configure remote flink jar

2019-11-18 Thread Thomas Weise
There is a related use case (not specific to HDFS) that I came across: It would be nice if the jar upload endpoint could accept the URL of a jar file as alternative to the jar file itself. Such URL could point to an artifactory or distributed file system. Thomas On Mon, Nov 18, 2019 at 7:40 PM

Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2

2019-11-04 Thread Thomas Weise
from end to > end by this Sunday. Then I'll update FLIP-27 wiki page. > > Thanks, > > Jiangjie (Becket) Qin > > > On Sat, Nov 2, 2019 at 8:18 AM Thomas Weise wrote: > >> Is there any activity on FLIP-27 that would make it relevant for 1.10 >> release? &

Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2

2019-11-01 Thread Thomas Weise
Is there any activity on FLIP-27 that would make it relevant for 1.10 release? Thanks Gary for the update, it provides excellent visibility on current activity and what we can expect with the release. On Fri, Nov 1, 2019 at 1:52 PM Steven Wu wrote: > Gary, FLIP-27 seems to get omitted in the

Re: [ANNOUNCE] New Flink PMC member Thomas Weise

2019-02-16 Thread Thomas Weise
13, 2019 at 10:02 AM Shaoxuan Wang > wrote: > > > Congratulations, Thomas! > > > > On Tue, Feb 12, 2019 at 5:59 PM Fabian Hueske > wrote: > > > >> Hi everyone, > >> > >> On behalf of the Flink PMC I am happy to announce Thomas Weise as a new

Re: [DISCUSS] Towards a leaner flink-dist

2019-01-23 Thread Thomas Weise
+1 for trimming the size by default and offering the fat distribution as alternative download On Wed, Jan 23, 2019 at 8:35 AM Till Rohrmann wrote: > Ufuk's proposal (having a lean default release and a user convenience > tarball) sounds good to me. That way advanced users won't be bothered by a

[ANNOUNCE] Apache Flink 1.5.6 released

2018-12-26 Thread Thomas Weise
The Apache Flink community is very happy to announce the release of Apache Flink 1.5.6, which is the final bugfix release for the Apache Flink 1.5 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming app