Looking for reviewer: FLINK-13127

Hi, I've prepared a small patch related to Yarn deployment. Would anyone please have a time to take a look at it? Thanks https://github.com/apache/flink/pull/9022 Regards, D.

[DISCUSS] CPU flame graph for a job vertex in web UI.

Hello, While looking into Flink internals, I've noticed that there is already a mechanism for stack-trace sampling of a particular job vertex. I think it may be really useful to allow user to easily render a cpu flamegraph in a new UI for a selected

Re: Looking for reviewer: FLINK-13127

On Mon, Jul 22, 2019 at 8:37 PM Zili Chen wrote: > > > > > Hi David, > > > > > > Just reviewed and left several comments. Thanks for your contribution! > > > > > > Best, > > > tison. > > > > > > > > > David Morávek 于2

Re: [DISCUSS] CPU flame graph for a job vertex in web UI.

d multiple requests for the > same operators. That way we would decrease a bit the RPC load. > > Apart from that, I think the next steps would be to find a committer who > could shepherd this effort and help you with merging it. > > Cheers, > Till > > On Wed, Jul 31,

Re: [DISCUSS] CPU flame graph for a job vertex in web UI.

1 for > this. > > And a minor question: do we plan to support multiple kinds of flame > graphs? It would be great if we have both on-cpu and off-cpu flame graphs. > > Best, > Paul Lam > > > 在 2019年8月2日，04:24，David Morávek 写道： > > > > Hi Till, thanks for th

Re: [DISCUSS] CPU flame graph for a job vertex in web UI.

I've created FLINK-13550 <https://issues.apache.org/jira/browse/FLINK-13550> to track the issue. Is there any committer who'd be willing to "shepherd this effort"? :) Thanks, D. On Fri, Aug 2, 2019 at 10:22 AM David Morávek wrote: > Hi Paul, for now I only plan

Re: [DISCUSS] Repository split

+1 for the motivation, -1 for the solution as all of the problems mention above can be addressed with the mono-repo as well. Multiple repositories: 1) This creates a big pain in case of change that targets code base in multiple repositories. Change needs to be split in multiple PRs, that need to b

Re: [VOTE] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

+1 On Wed, Sep 4, 2019 at 1:38 PM Till Rohrmann wrote: > +1 (binding) > > On Wed, Sep 4, 2019 at 12:43 PM Chesnay Schepler > wrote: > > > +1 (binding) > > > > On 04/09/2019 11:18, JingsongLee wrote: > > > +1 (non-binding) > > > > > > default 0 is really not user production friendly. > > > > > >

Fine grained batch recovery vs. native libraries

Hi, we're testing the newly released batch recovery and are running into class loading related issues. 1) We have a per-job flink cluster 2) We use BATCH execution mode + region failover strategy Point 1) should imply single user code class loader per task manager (because there is only single p

Re: Fine grained batch recovery vs. native libraries

Hi Chesnay, I've created FLINK-13958 <https://issues.apache.org/jira/browse/FLINK-13958> to track the issue. Thanks, D. On Wed, Sep 4, 2019 at 1:56 PM Chesnay Schepler wrote: > This sounds like a serious bug, please open a JIRA ticket. > > On 04/09/2019 13:41, David M

[DISCUSS] Supporting multiple Flink versions vs. tech debt

Hello, we currently have an opened PR for Flink 1.9 , which greatly improves the runner for batch use-case. In case the PR gets merged, we would be supporting 5 latest major versions of Flink, which obviously come with high maintenance price and makes futu

Re: [DISCUSS] Supporting multiple Flink versions vs. tech debt

sorry, wrong mailing list, I wanted to send this to the beam one. Sorry for the confusion. D. On Sat, Sep 7, 2019 at 12:32 PM David Morávek wrote: > Hello, > > we currently have an opened PR for Flink 1.9 > <https://github.com/apache/beam/pull/9296>, which greatly improve

[DISCUSS] Preventing Mockito usage for the new code with Checkstyle

Hi Everyone, A long time ago, the community decided not to use Mockito-based tests because those are hard to maintain. This is already baked in our Code Style and Quality Guide [1]. Because we still have Mockito imported into the code base, it's very easy for newcomers to unconsciously introduce

Re: [DISCUSS] FLINK-31873: Add setMaxParallelism to the DataStreamSink Class

Hi Eric, this sounds reasonable, there are definitely cases where you need to limit sink parallelism for example not to overload the storage or limit the number of output files +1 Best, D. On Sun, Apr 23, 2023 at 1:09 PM Weihua Hu wrote: > Hi, Eric > > Thanks for bringing this discussion. > I

Re: [DISCUSS] Planning Flink 2.0

Hi, Great to see this topic moving forward; I agree it's long overdue. I keep thinking about 2.0 as a chance to eliminate things that didn't work, make the feature set denser, and fix rough edges and APIs that hold us back. Some items in the doc (Key Features section) don't tick these boxes for

Re: [DISCUSS] Preventing Mockito usage for the new code with Checkstyle

the proposal. > > > >>>>> > > > >>>>> Can we also prevent Junit4 usage in new code by this way？Because > > > >>>> currently > > > >>>>> we are aiming to migrate our codebase to JUnit 5. > > > >

Call for help on the Web UI (In-Place Rescaling)

Hi Everyone, In FLINK-31471, we've introduced new "in-place rescaling features" to the Web UI that show up when the scheduler supports FLIP-291 REST endpoints. I expect this to be a significant feature for user education (they have an easy way to try out how rescaling behaves, especially in combi

Re: Call for help on the Web UI (In-Place Rescaling)

s as they are in the video! > > > > Couple other things to consider: > > > > > > * Confirming new parallelism before actually doing it, e.g. having a > > “Deploy/Commit/Save” button > > * Allow users to enter parallelism without having to >

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

Hi, Thanks for the FLIP! Data lineage is an important problem to tackle. Can you please expand on how this is planned to be wired into the JobManager? As I understand, the listeners will be configured globally (per cluster), so this won't introduce a new code path for running per-job / per-sessio

Re: [DISCUSS] Flink REST API improvements

Hi Hong, Thanks for starting the discussion. seems to be using the cached version of the entire Execution graph (stale > data), when it could just use the CheckpointStatsCache directly CheckpointStatsCache is also populated using the "cached execution graph," so there is nothing to gain from th

Re: [VOTE] FLIP-322 Cooldown period for adaptive scheduler

> > The vote closes within 6 hours and, as for now, there was no vote. This > is a very short FLIP, that takes a few minutes to read. > Maybe because there should have been a dedicated voting thread (marked as [VOTE]), this one was hidden and hard to notice. We should restart the vote with proper

Re: [VOTE] FLIP-322 Cooldown period for adaptive scheduler

8:49 AM David Morávek wrote: > The vote closes within 6 hours and, as for now, there was no vote. This >> is a very short FLIP, that takes a few minutes to read. >> > > Maybe because there should have been a dedicated voting thread (marked as > [VOTE]), this one was h

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

The FLIP reads sane to me. I'm unsure about the default values, though; 5 minutes of wait time between rescales feels rather strict, and we should rethink it to provide a better out-of-the-box experience. I'd focus on newcomers trying AS / Reactive Mode out. They will struggle if they add new reso

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

4, 2023 at 9:11 AM David Morávek wrote: > The FLIP reads sane to me. I'm unsure about the default values, though; 5 > minutes of wait time between rescales feels rather strict, and we should > rethink it to provide a better out-of-the-box experience. > > I'd focus on n

Re: Working improving the REST API

I've left some comments on the PR. On Tue, Jul 4, 2023 at 9:20 AM Martijn Visser wrote: > Hi Hong, > > Given that this changes the REST API, which is a public interface, I'm > wondering if this shouldn't first have had a (small) FLIP if I follow the > guidelines from the overview page [1]. > > B

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

Rescale, yes, but you shouldn't do that in the first place. That sounds great; let's go ahead and outline this in the FLIP. Best, D. On Tue, Jul 4, 2023 at 12:30 PM Etienne Chauchot wrote: > Hi all, > > Thanks David for your feedback. My comments are inline > > Le 04/07/2

Re: [DISCUSS][2.0] Deprecating Accumulator in favor of MetricsGroup

AFAIK Apache Beam also used acummulators for metric collection, which is indeed a major use case. I’m not convinced that MetricGroup is fuĺly replacing what acummulators have to offer though; OperatorCoordinators might be able to rplace remaining capabilities, but this need bit more thoughts, the

Re: Proposal for Implementing Keyed Watermarks in Apache Flink

Hi Tawfik, It's exciting to see any ongoing research that tries to push Flink forward! The get the discussion started, can you please your paper with the community? Assessing the proposal without further context is tough. Best, D. On Mon, Sep 4, 2023 at 4:42 PM Tawfek Yasser Tawfek wrote: > D

Re: Re: [DISCUSS] FLIP-357: Deprecate Iteration API of DataStream

+1 since there is an alternative, more complete implementation available Best, D. On Sat, Sep 2, 2023 at 12:07 AM David Anderson wrote: > +1 > > Keeping the legacy implementation in place is confusing and encourages > adoption of something that really shouldn't be used. > > Thanks for driving t

Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling

Hello Yuepeng, The FLIP reads sane; nice work! To re-phrase my understanding: The problem you're trying to solve only exists in complex graphs with different per-vertex parallelism. If the parallelism is set globally (assuming the pipeline has roughly even data skew), the algorithm could make thi

Re: [Discuss] FLIP-362: Support minimum resource limitation

H Xiangyui, The sentiment of the FLIP makes sense, but I keep wondering whether this is the best way to think about the problem. I assume that "interactive session cluster" users always want to keep some spare resources around (up to a configured threshold) to reduce cold start instead of statical

Re: Support AWS SDK V2 for Flink's S3 FileSystem

Hi Maomao, I wonder whether it would make sense to take a stab at consolidating the S3 filesystems instead and introduce a native one. The whole Hadoop wrapper around the S3 client exists for legacy reasons, and it adds complexity and probably an unnecessary performance penalty. If you take a loo

Re: [Discuss] FLIP-362: Support minimum resource limitation

r large min-reserved resource > for user cases submitting lots of short-lived jobs concurrently, but we > don't want to configure a large spare resource since this might double the > total resource usage and lead to resource waste. > > Looking forward to hearing from you. > &g

Re: Support AWS SDK V2 for Flink's S3 FileSystem

as a follow-up. > > > > [1] > https://github.com/apache/flink/blob/d78d52b27af2550f50b44349d3ec6dc84b966a8a/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java#L695 > > [2] > https://github.com/apache/flink/blob/d78d52b27af2550f50b44349d3ec6dc84b966a8a/flink

Re: [DISCUSS] REST API behaviour when user main method throws error

Hi Danny, > My current proposal is that the REST API should not leave the Flink cluster in an inconsistent state. Regarding consistency, Flink only cares about individual jobs, but I can see your point. For streaming, this is probably something we could address by book-keeping jobs submitted by

Re: Proposal for Implementing Keyed Watermarks in Apache Flink

The paper looks interesting, but it might not manifest the described benefit for practical reasons: 1. It forces you to remember all keys in the broadcasted (partitioned is impossible without timeouts, etc.) operator state. Forever. This itself is a blocker for a bunch of pipelines. The primary mo

Re: [VOTE] FLIP-384: Introduce TraceReporter and use it to create checkpointing and recovery traces

+1 (binding) Best, D. On Wed, Nov 22, 2023 at 11:21 AM Roman Khachatryan wrote: > +1 (binding) > > Regards, > Roman > > On Wed, Nov 22, 2023, 7:08 AM Zakelly Lan wrote: > > > +1(non-binding) > > > > Best, > > Zakelly > > > > On Wed, Nov 22, 2023 at 3:04 PM Hangxiang Yu > wrote: > > > > > +1 (

Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for improved state compatibility on parallelism change

Hi Zhanghao, Thanks for the FLIP. What you're proposing makes a lot of sense +1 Have you thought about how this works with unaligned checkpoints in case you go from unchained to chained? I think it should be fine because this scenario should only apply to forward/rebalance scenarios where we, as

Re: FW: [ANNOUNCE] New Apache Flink Committer - Alexander Fedulov

Congrats, Alex! Best, D. On Fri, Jan 5, 2024 at 7:25 AM Sergey Nuyanzin wrote: > Congratulations, Alex! > > On Fri, Jan 5, 2024, 05:12 Lincoln Lee wrote: > > > Congratulations, Alex! > > > > Best, > > Lincoln Lee > > > > > > Alexander Fedulov 于2024年1月4日周四 19:08写道： > > > > > Thanks, everyone!

Multiple rebalances are incorrectly ignored in some cases.

Hello Flinkers, we have run into unexpected behaviour with chained Reshuffles in Apache Beam's Flink runner (batch). In flink optimizer, when we `.rebalance()` dataset, is output channel is marked as `FORCED_REBALANCED`. When we chain this with another `.rebalance()`, the latter is ignored becaus

Re: Multiple rebalances are incorrectly ignored in some cases.

properties/RequestedGlobalProperties.java#L301 D. On Mon, Apr 27, 2020 at 11:24 AM Aljoscha Krettek wrote: > On 27.04.20 09:34, David Morávek wrote: > > > When we include `flatMap` in between rebalances -> > > `.rebalance().flatMap(...).rebalance()`, we need to reshuffle again, > > because data

connection timeout during shuffle initialization

Hello community, I'm currently struggling with an Apache Beam batch pipeline on top of Flink. The pipeline runs smoothly in smaller environments, but in production it always ends up with `connection timeout` in one of the last shuffle phases. org.apache.flink.runtime.io.network.partition.consumer

Re: connection timeout during shuffle initialization

.type > > And it’s default value will be `file`. > > We would appreciate if you could get back to us with the results! > > Piotrek > > > On 28 Jan 2020, at 11:03, Till Rohrmann wrote: > > > > Hi David, > > > > I'm unfortunately not familiar

Re: connection timeout during shuffle initialization

>> command from some representative Task Manager? If indeed disks are > > >> overloaded, changing Flink’s config option > > >> > > >>taskmanager.network.bounded-blocking-subpartition-type > > >> > > >> From default

Re: connection timeout during shuffle initialization

non blocking, or at least much > less blocking. > > Piotrek > > > On 29 Jan 2020, at 14:05, Stephan Ewen wrote: > > > > /CC Piotr and Zhijiang > > > > Sounds reasonable at first glance. Would like to hear Piotr's and > Zhijiang's take, though, t

Re: connection timeout during shuffle initialization

Just to clarify, these are bare metal nodes (128G ram, 16 cpus + hyperthreading, 4xHDDS, 10g network), which run yarn, hdfs and hbase. D. On Wed, Jan 29, 2020 at 5:03 PM David Morávek wrote: > Hi Piotr, > > removal of buffer prefetch in BoundedBlockingSubpartitionReader did not >

Re: [DISCUSS] Adopting a Code Style and Quality Guide

I love this kind of unification being pushed forward, the benefit it has on code reviews is enormous. Just my two cents, did you guys think about adopting an automatic code formatter for java, the same way as Apache Beam did? It is super easy to use for contributors as they don't need to keep any

Re: [DISCUSS] Releasing Apache Flink 1.12.1

Hi, I think https://issues.apache.org/jira/browse/FLINK-20648 should be addressed, as Kubernetes HA was one of the main selling points of this release. WDYT? D. Sent from my iPhone > On 17. 12. 2020, at 13:54, Yun Tang wrote: > > Thanks for driving this quick-fix release. > +1 for fixing th

Re: [DISCUSS] Cleaning up HighAvailabilityServices interface to reflect the per-JM-process LeaderElection

Hi Dong, > Adding regarding the effort to add back the per-component election capability: given that the implementation already follows per-process election, and given that there will likely be a lot of extra design/implementation/test effort needed to achieve the use-cases described above, maybe

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

Hi Shammon, I'm starting to see what you're trying to achieve, and it's really exciting. I share Piotr's concerns about e2e latency and disability to use unaligned checkpoints. I have a couple of questions that are not clear to me from going over the FLIP: 1) Global Checkpoint Commit Are you pl

Re: Reworking the Rescale API

Hi Gyula, > can you please explain why the AdaptiveScheduler is not the default > scheduler? There are still some smaller bits missing. As far as I know, the missing parts are: 1) Local recovery (reusing the already downloaded state files after restart / rescale) 2) Support for fine-grained re

Re: Reworking the Rescale API

it was very much meant as a better version of the default scheduler for > streaming jobs. > > On 26/01/2023 19:06, David Morávek wrote: > > Hi Gyula, > > > > > >> can you please explain why the AdaptiveScheduler is not the default > >> scheduler?

Re: Reworking the Rescale API

It makes sense to give the whole "scheduler ecosystem," not just the adaptive scheduler, a little bit more structure in the docs. We already have 4 different schedulers (Default, Adaptive, AdaptiveBatch, AdaptiveBatchSpeculative), and it becomes quite confusing since the details are scattered aroun

[DISCUSS] FLIP-291: Externalized Declarative Resource Management

Hi everyone, This FLIP [1] introduces a new REST API for declaring resource requirements for the Adaptive Scheduler. There seems to be a clear need for this API based on the discussion on the "Reworking the Rescale API" [2] thread. Before we get started, this work is heavily based on the prototyp

Re: [VOTE] FLIP-285: Refactoring LeaderElection to make Flink support multi-component leader election out-of-the-box

Thanks for the detailed FLIP, Matthias; this will simplify the HA code-base significantly. +1 (binding) Best, D. On Tue, Jan 31, 2023 at 5:22 AM Yang Wang wrote: > +1 (Binding) > > Best, > Yang > > ConradJam 于2023年1月31日周二 12:09写道： > > > +1 non-binding > > > > Matthias Pohl 于2023年1月25日周三 17:3

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

; Thank you for drive this flip, which helps less flink shutdown time > > > > > > > > for this flip, I would like to make a few idea on share > > > > > > > > > > > >- when the number of "slots" is insufficient, can we can stop >

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

y should update the > config > > file and restart flink job, which may take a long time. > > 2. After providing the REST API, users can just send a request to the job > > via REST API quickly after updating the config file. > > The configuration in the running job and confi

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/ > > > [2] https://lists.apache.org/thread/2f7dgr88xtbmsohtr0f6wmsvw8sw04f5 > > > > > > On Mon, Feb 13, 2023 at 1:16 PM feng xiangyu > wrote: > > > > > > > > Hi David, > >

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

zation when running in application mode > >> would be to drop as many task managers during the restart such that > >> NUM_REQUIRED_SLOTS >= NUM_AVAILABLE_SLOTS stays true. We can look into > >> this independently of the FLIP. > >> > >> Feel free to star

Re: [ANNOUNCE] New Apache Flink Committer - Anton Kalashnikov

Congratulations Anton, well deserved! Best, d. On Wed, Feb 22, 2023 at 5:01 AM Jane Chan wrote: > Congratulations, Anton! > > Best regards, > Jane > > On Wed, Feb 22, 2023 at 11:22 AM Yun Tang wrote: > > > Congratulations, Anton! > > > > Best > > Yun Tang > > >

Re: [DISCUSS] Deprecating GlobalAggregateManager

I think this makes sense, +1 from my side; as I wrote on the ticket, I'm not aware of any other usages apart from the kinesis connector, and we already have more feature complete API that can replace the functionality there. Best, D. On Mon, Feb 27, 2023 at 2:44 PM Zhanghao Chen wrote: > Hi dev

Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager

Hi Weihua, I still need to dig into the details, but the overall sentiment of this change sounds reasonable. Best, D. On Mon, Feb 27, 2023 at 2:26 PM Zhanghao Chen wrote: > Thanks for driving this topic. I think this FLIP could help clean up the > codebase to make it easier to maintain. +1 on i

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

ll transition into the executing state. We're still planning to dig deeper in this direction with other efforts, but this is already good enough and should allow us to move the FLIP forward. WDYT? Unless there are any objectives against the above, I'd like to proceed to a vote. Bes

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

esource >> limits reached. >> >> [1] However, there might be costs involved with executing the >> rescaling, e.g. for using external storage like s3, especially without >> local recovery. >> [2] >> https://github.com/dmvk/flink/commit/5e7edcb77d8522c367b

[VOTE] FLIP-291: Externalized Declarative Resource Management

Hi Everyone, I want to start the vote on FLIP-291: Externalized Declarative Resource Management [1]. The FLIP was discussed in this thread [2]. The goal of the FLIP is to enable external declaration of the resource requirements of a running job. The vote will last for at least 72 hours (Friday,

Re: [VOTE] FLIP-291: Externalized Declarative Resource Management

Thanks, everyone, I'm closing this vote now. I'll follow up with the result in a separate email. On Wed, Mar 1, 2023 at 10:01 AM Shammon FY wrote: > +1 (non-binding) > > Best, > Shammon > > > On Wed, Mar 1, 2023 at 4:51 PM Roman Khachatryan wrote: > > > +1 (binding) > > > > Thanks David, and ev

[RESULT][VOTE] FLIP-291: Externalized Declarative Resource Management

I'm happy to announce that we have unanimously approved this FLIP. There are 8 approving votes, 3 of which are binding: * John Roesler * Konstantin Knauf (binding) * Zhanghao Chen * ConradJam * Feng Xiangyu * Gyula Fóra (binding) * Roman Khachatryan (binding) * Shammon FY There are no disapprovi

Re: Regarding new command to download jars in flink cluster

Hi Surendra, Since you're mentioning docker, I assume you're deploying your application to k8s. Is that correct? For handcrafted Kubernetes deployments, you can simply download the jar into the user lib folder in an init container [1]. You can usually reuse existing docker images to download the

Re: [Vote] FLIP-298: Unifying the Implementation of SlotManager

+1 (binding) Best, D. On Fri, Mar 10, 2023 at 4:49 AM Yuxin Tan wrote: > Thanks, Weihua! > +1 (non-binding) > > Best, > Yuxin > > > weijie guo 于2023年3月10日周五 11:29写道： > > > +1 (binding) > > > > Best regards, > > > > Weijie > > > > > > Shammon FY 于2023年3月10日周五 11:02写道： > > > > > Thanks weihua,

Re: Re: [DISCUSS] Extract core autoscaling algorithm as new SubModule in flink-kubernetes-operator

> Although YARN serves as the platform for Flink, does YARN also operate on K8s? YARN is an alternative to k8s and Flink should make no assumptions about how it's deployed, even though some companies might deploy it as an overlay RM on top of k8s (I doubt that, but I guess they might do it for leg

Re: [DISCUSS] FLIP-304: Pluggable failure handling for Apache Flink

Hi Panagiotis, This is an excellent proposal and something everyone trying to provide "Flink as a service" needs to solve at some point. I have a couple of questions: If I understand the proposal correctly, this is just about adding tags to the Throwable by running a tuple of (Throwable, FailureC

Re: [DISCUSS] FLIP-304: Pluggable failure handling for Apache Flink

plugin manager) > however listeners can use previous state (tags/labels) to make decisions: > e.g., wont assign *UNKNOWN* failureType when we have already seen *USER *or > the other way around -- when we have seen *UNKNOWN* remove in favor of > *USER* > > > Cheers, > Pan

Re: [DISCUSS] FLIP-304: Pluggable failure handling for Apache Flink

t; > inconsistent because they imply specific implementation > details, > > > > and > > > > > > not > > > > > > > > all FailureListeners need to handle them, we shouldn't put > them > > > in > > > > > th

Re: [DISCUSS] FLIP-304: Pluggable failure handling for Apache Flink

se. The use case Piotr described (where you want to reuse > some > > > other > > > >> classifier) is the only one I can think of where we actually need > > > >> classifiers depending on each other. Supporting such a use case >

Re: [Discussion] - Take findify/flink-scala-api under Flink umbrella

cc dev@f.a.o On Sun, Apr 16, 2023 at 11:42 AM David Morávek wrote: > Hi Alexey, > > I'm a bit skeptical because, looking at the project, I see a couple of red > flags: > > - The project is inactive. The last release and commit are both from the > last May. > - The

Re: [VOTE] FLIP-304: Pluggable Failure Enrichers

Hi Panos, It seems that most recent discussions (e.g. changing the semantics of the config option) are not reflected in the FLIP. Can you please double-check that this is the correct version? Best, D. On Mon, Apr 17, 2023 at 9:24 AM Panagiotis Garefalakis wrote: > Hello everyone, > > I want t

Re: [VOTE] FLIP-304: Pluggable Failure Enrichers

: > > > +1 to what David wrote. I think we need to update the FLIP and extend the > > voting? > > > > Piotrek > > > > śr., 19 kwi 2023 o 09:06 David Morávek napisał(a): > > > >> Hi Panos, > >> > >> It seems that most recent dis

Re: [ANNOUNCE] New Apache Flink PMC Member - Leonard Xu

Congratulations, Leonard, well deserved! Best, D. On Fri 21. 4. 2023 at 16:40, Feng Jin wrote: > Congratulations, Leonard > > > > Best, > Feng Jin > > On Fri, Apr 21, 2023 at 8:38 PM Mang Zhang wrote: > > > Congratulations, Leonard. > > > > > > -- > > > > Best regards, > > Mang Zhang > >

Re: [ANNOUNCE] New Apache Flink PMC Member - Qingsheng Ren

Congratulations, Qingsheng, well deserved! Best, D. On Fri 21. 4. 2023 at 16:41, Feng Jin wrote: > Congratulations, Qingsheng > > > > Best, > Feng Jin > > On Fri, Apr 21, 2023 at 8:39 PM Mang Zhang wrote: > > > Congratulations, Qingsheng. > > > > > > > > > > > > -- > > > > Best regards, >

Re: [DISCUSS] FLIP-388: Support Dynamic Logger Level Adjustment

Hi Yuepeng, Thanks for the FLIP! There was already quite a discussion on FLIP-210 [1][2], that has proposed more or less the same thing. FLIP was marked as out of scope for Flink because underlying technologies already address it. Are you aware of the effort? If yes, what has changed since then?

Re: [VOTE] FLIP-437: Support ML Models in Flink SQL

+1 (binding) My only suggestion would be to move Catalog changes into a separate interface to allow us to begin with lower stability guarantees. Existing Catalogs would be able to opt-in by implementing it. It's a minor thing though, overall the FLIP is solid and the direction is pretty exciting.

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

Thanks for the FLIP Matthias, I think it looks pretty solid! I also don't see a relation to unaligned checkpoints. From the AS perspective, the checkpoint time doesn't matter. Is it possible a change event observed right after a complete checkpoint > (or within a specific short time after a check

Re: [VOTE] FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler

+1 (binding) On Mon, Jun 17, 2024 at 10:24 AM Matthias Pohl wrote: > Hi everyone, > the discussion in [1] about FLIP-461 [2] is kind of concluded. I am > starting a vote on this one here. > > The vote will be open for at least 72 hours (i.e. until June 20, 2024; > 8:30am UTC) unless there are an

Re: [DISCUSS] FLIP-220: Temporal State

Hi David, I really like the proposal. This has so much potential for various optimizations, especially for temporal joins. My only concern is that the interfaces seems unnecessarily complicated. My feeling would be that we only need a single, simple interface that would fit it all (the same way a

Re: [DISCUSS] FLIP-220: Temporal State

Hi David, It seems to me that at least with the heap-based state backend, readRange > is going to have to do a lot of unnecessary work to implement this > isEmpty() operation, since it have will to consider the entire range from > MIN_VALUE to MAX_VALUE. (Maybe we should add an explicit isEmpty me

Re: [DISCUSS] FLIP-220: Temporal State

at 1:27 PM David Morávek wrote: > Hi David, > > It seems to me that at least with the heap-based state backend, readRange >> is going to have to do a lot of unnecessary work to implement this >> isEmpty() operation, since it have will to consider the entire range from >&

Re: [DISCUSS] FLIP-220: Temporal State

atural > order if comparing the bytes. Thus, we might need to avoid to write the > length in the serialized bytes. > On the other hand, changelog logger would record operation per key one by > one in the logs. We need to consider how to distinguish each key in the > combined seriali

Re: [DISCUSS] FLIP-220: Temporal State

tly handle binary keys... > > > 2) Duplicates > > Isn't allowing a TemporalValueState just a special case of b.III? So if a > user > of the state wants that, then they can leverage a simple API vs. if you > want > fancier duplicate handling, you'd just go with Temp

Re: [DISCUSS] FLIP-220: Temporal State

;> Isn't allowing a TemporalValueState just a special case of b.III? So if a >> user >> of the state wants that, then they can leverage a simple API vs. if you >> want >> fancier duplicate handling, you'd just go with TemporalListState and >> implement &g

Re: [DISCUSS] Planning Flink 1.16

Thanks Konstantin and Chesnay for starting the discussion and volunteering. The timeline proposal sounds reasonable :+1: Best, D. On Tue, Apr 26, 2022 at 1:37 PM Martijn Visser wrote: > Hi everyone, > > Thanks for starting this discussion. I would also volunteer to help out as > a release manag

Re: [ANNOUNCE] New Flink PMC member: Yang Wang

Nice! Congrats Yang, well deserved! ;) On Fri 6. 5. 2022 at 17:53, Peter Huang wrote: > Congrats, Yang! > > > > Best Regards > Peter Huang > > On Fri, May 6, 2022 at 8:46 AM Yu Li wrote: > > > Congrats and welcome, Yang! > > > > Best Regards, > > Yu > > > > > > On Fri, 6 May 2022 at 14:48, Paul

Re: About Native Deployment's Autoscaling implementation

Hi Talat, This is definitely an interesting and rather complex topic. Few unstructured thoughts / notes / questions: - The main struggle has always been that it's hard to come up with a generic one-size-fits-it-all metrics for autoscaling. - Flink doesn't have knowledge of the external environ

Re: Flink UI in Application Mode

Hi Zain, you can find a link to web-ui either in the CLI output after the job submission or in the YARN ResourceManager web ui [1]. With YARN Flink needs to choose the application master port at random (could be somehow controlled by setting _yarn.application-master.port_) as there might be multip

Re: [VOTE] FLIP-227: Support overdraft buffer

+1 (binding) On Thu, May 26, 2022 at 11:55 AM Dawid Wysakowicz wrote: > +1 (binding) > > On Thu, 26 May 2022, 11:21 Piotr Nowojski, wrote: > > > Yes, it will be a good improvement :) > > > > +1 (binding) > > > > Piotrek > > > > czw., 26 maj 2022 o 10:26 Anton Kalashnikov > > napisał(a): > > >

Re: [DISCUSS] Deprecate Java 8 support

Thank you Chesnay for starting the discussion! This will generate bit of a work for some users, but it's a good thing to keep moving the project forward. Big +1 for this. Jingsong: Receiving this signal, the user may be unhappy because his application > may be all on Java 8. Upgrading is a big jo

Re: [DISCUSS] Releasing Flink 1.14.1

Hi Martijn, * https://issues.apache.org/jira/browse/FLINK-24543 - Zookeeper connection > issue causes inconsistent state in Flink -> I think this depends on the > outcome of dropping Zookeeper 3.4 as was proposed on the Dev mailing list > We already have an approved patch for master, I'll be prep

Re: [DISCUSS] FLIP-194: Introduce the JobResultStore

>>> > > >>> > Thanks, > >>> > Zhu > >>> > > >>> > Till Rohrmann 于2021年11月27日周六 上午1:29写道： > >>> > > >>> > > Thanks for creating this FLIP Matthias, Mika and David. > >>> > > > >>&g

Re: FLink Accessing two hdfs cluster

LINK > client . (The YARN node contains all nodes of the two HDFS . ) > > >There are more details in JIRA FLINK-25099 > <https://issues.apache.org/jira/browse/FLINK-25099> > > > > At 2021-11-30 21:50:08, "David Morávek" wrote: > > Hi chenqiz

Re: [ANNOUNCE] New Apache Flink Committer - Ingo Bürk

Congrats Ingo, well deserved ;) Best, D. On Thu, Dec 2, 2021 at 5:17 PM Dawid Wysakowicz wrote: > Congratulations Ingo! Happy to have you onboard as a committer! > > Best, > > Dawid > > On 02/12/2021 17:14, Francesco Guardiani wrote: > > Congrats Ingo! > > > > On Thu, Dec 2, 2021 at 4:58 PM Nic

Re: [ANNOUNCE] New Apache Flink Committer - Matthias Pohl

Congrats Matthias, well deserved ;) Best, D. On Thu, Dec 2, 2021 at 5:17 PM Dawid Wysakowicz wrote: > Congratulations Matthias! Really well deserved! > > Best, > > Dawid > > On 02/12/2021 16:53, Nicolaus Weidner wrote: > > Congrats Matthias, well deserved! > > > > Best, > > Nico > > > > On Thu,

  1   2   3   >