FW: Re: flink-sql-connector-jdbc new release

2023-12-08 Thread David Radley
the blocking issue; it has had some reviews and one approval; please could someone either merge or provide feedback as to what needs to be changed so it can be merged, Kind regards, David. From: David Radley Date: Friday, 27 October 2023 at 16:22 To: dev@flink.apache.org Subject: Re

Building Flink JDBC connector locally on M1 Mac

2023-12-08 Thread David Radley
running rancher, so do have a docker environment. Any pointers? Kind regards, David. Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Question on lookup joins

2023-12-15 Thread David Radley
Hi , I am working on FLINK-33365 which related to JDBC predicate pushdown. I want to ensure that the same results occur with predicate pushdown as without. So I am asking this question outside the pr / issue. I notice the following behaviour for lookup joins without predicate pushdown. I was no

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

2023-12-19 Thread David Radley
Hi, I had a 3 of comments: - the name of the config option is "scan.filter-push-down.enabled". This implies it is only for scan sources and not lookups. I suggest removing the scan. prefix. - there is a talk of having a numeric option, as the filter pushdown might result in a full table scan. W

Re: [DISCUSS] FLIP-400: AsyncScalarFunction for asynchronous scalar function support

2023-12-20 Thread David Anderson
I'm delighted to see the progress on this. This is going to be a major enabler for some important use cases. The proposed simplifications (global config and ordered mode) for V1 make a lot of sense to me. +1 David On Wed, Dec 20, 2023 at 12:31 PM Alan Sheinberg wrote: > Thanks

Re: [ANNOUNCE] New Apache Flink Committer - Alexander Fedulov

2024-01-02 Thread David Anderson
That's great news. Congratulations, Alex! David On Tue, Jan 2, 2024 at 9:00 AM Ryan Skraba wrote: > Awesome news for the community -- congratulations Alex (and Happy New > Year everyone!) > > Ryan > > On Tue, Jan 2, 2024 at 2:55 PM Yun Tang wrote: > > > > C

Re: [ANNOUNCE] New Apache Flink Committer - Alexander Fedulov

2024-01-03 Thread David Radley
Many Congratulations David . From: Maximilian Michels Date: Tuesday, 2 January 2024 at 12:16 To: dev Cc: Alexander Fedulov Subject: [EXTERNAL] [ANNOUNCE] New Apache Flink Committer - Alexander Fedulov Happy New Year everyone, I'd like to start the year off by announcing Alexander Fedulov

FW: [ANNOUNCE] New Apache Flink Committer - Alexander Fedulov

2024-01-03 Thread David Radley
Sorry for my typo. Many congratulations Alex! From: David Radley Date: Wednesday, 3 January 2024 at 10:23 To: David Anderson Cc: dev@flink.apache.org Subject: Re: [EXTERNAL] [ANNOUNCE] New Apache Flink Committer - Alexander Fedulov Many Congratulations David . From: Maximilian Michels Date

RE: [VOTE] Release flink-connector-jdbc, release candidate #1

2024-01-07 Thread David Radley
connector. I can then work on fixing the remaining FLINK-33365 query cases, which should not take too long, but I am out until Thursday this week so will be looking at it then, Kind regards, David. From: Martijn Visser Date: Friday, 5 January 2024 at 14:24 To: dev@flink.apache.org Subject

Re: [DISCUSS] FLIP-413: Enable unaligned checkpoints by default

2024-01-07 Thread David Anderson
link/flink-docs-release-1.18/docs/ops/state/checkpoints_vs_savepoints/#capabilities-and-limitations David On Fri, Jan 5, 2024 at 5:42 AM Piotr Nowojski wrote: > Ops, fixing the topic. > > Hi! > > > > I would like to propose by default to enable unaligned checkpoints and >

Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for improved state compatibility on parallelism change

2024-01-09 Thread David Morávek
Hi Zhanghao, Thanks for the FLIP. What you're proposing makes a lot of sense +1 Have you thought about how this works with unaligned checkpoints in case you go from unchained to chained? I think it should be fine because this scenario should only apply to forward/rebalance scenarios where we, as

Re: FW: [ANNOUNCE] New Apache Flink Committer - Alexander Fedulov

2024-01-09 Thread David Morávek
t;> > > > > >> Best, > > > > >> Junrui > > > > >> > > > > >> weijie guo 于2024年1月4日周四 09:57写道: > > > > >> > > > > >>> Congratulations, Alex! > > > > >>> &g

Re: [DISCUSS] Extend (or maintain) "shell" script support for Windows

2020-02-20 Thread David Anderson
based environment once they are done testing in the IDE, be that via WSL, Docker, a local VM, or a staging server. David On Thu, Feb 20, 2020 at 11:09 AM Stephan Ewen wrote: > I think this question should go on the user mailing list, not on dev. > > You can also use Twitter to collect so

Re: [DISCUSS] Improvements on FLIP Process

2020-02-20 Thread David Anderson
ted to Released, if they would add a section at the end outlining any ways in which the implementation being released differs from what is described in the FLIP. David On Thu, Feb 20, 2020 at 1:05 PM Hequn Cheng wrote: > Hi everyone, > > I appreciate all the feedback! Let me try to

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-11 Thread David Anderson
+1 I like where this is headed. One question: during restore, it could happen that a new task manager is configured with fewer or smaller buffers than was previously the case. How will this be handled? David On Wed, Mar 11, 2020 at 8:31 AM Arvid Heise wrote: > Hi Thomas, > > it&#

Re: [DISCUSS] Drop Bucketing Sink

2020-03-12 Thread David Anderson
46 [1]. Removing the bucketing sink will go down better, in my opinion, if it's coupled with progress on some of the open StreamingFileSink tickets. Best, David [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs On Thu, Mar 12, 2020 at 4:27 PM Zhiji

[PROPOSAL] Contribute training materials to Apache Flink

2020-04-09 Thread David Anderson
would be best to integrate the website content into flink.apache.org, and to add the exercises to flink-playgrounds -- but these points can be discussed separately once we've established that the community wants this content. Looking forward to hearing what you think! Best regards, David

Re: [PROPOSAL] Contribute training materials to Apache Flink

2020-04-10 Thread David Anderson
g one version (for the latest Flink release), and I don't really see any added value to the community in keeping older versions of the training online. On Fri, Apr 10, 2020 at 8:46 PM Eduardo Winpenny Tejedor < eduardo.winpe...@gmail.com> wrote: > +1 to the idea, David regarding your

Re: [PROPOSAL] Contribute training materials to Apache Flink

2020-04-15 Thread David Anderson
Till, Yun, etal, Now that we've established the community's interest in engaging with this content, I've started a new thread on the dev list for discussion of the details. I've said a bit there already regarding ongoing maintenance, and CI for the exercises. Best, David

[DISCUSS] Integration of training materials into Apache Flink

2020-04-15 Thread David Anderson
exercises are focused on the most basic parts of the API, that hasn’t been difficult. As for content from the training website that would move into the docs, this content is much more stable, and has only needed a gentle revision every year or two. Regards, David

Re: [DISCUSS] Integration of training materials into Apache Flink

2020-04-16 Thread David Anderson
g there. But yes, there is room for improvement in this part of the docs, so I'm expecting to be able to help with that. David On Wed, Apr 15, 2020 at 9:20 PM Seth Wiesman wrote: > Hi David, > > I am happy to get the repo created for you. > > If we go with the document

Re: [DISCUSS] Integration of training materials into Apache Flink

2020-04-17 Thread David Anderson
@Robert, do you think we could create a new flink-training component in Jira for tracking tickets related to the flink-training content? David On Thu, Apr 16, 2020 at 10:54 AM David Anderson wrote: > > I am happy to get the repo created for you. > > Thank you, @seth. I think we ar

Re: [DISCUSS] Integration of training materials into Apache Flink

2020-04-19 Thread David Anderson
> > How about calling the component "Documentation / Training"? Sounds good. Thanks! *David Anderson* | Training Coordinator Follow us @VervericaData -- Join Flink Forward - The Apache Flink Conference Stream Processing | Event Driven | Real Time On Sun, Apr 19, 2020 at

Multiple rebalances are incorrectly ignored in some cases.

2020-04-27 Thread David Morávek
Hello Flinkers, we have run into unexpected behaviour with chained Reshuffles in Apache Beam's Flink runner (batch). In flink optimizer, when we `.rebalance()` dataset, is output channel is marked as `FORCED_REBALANCED`. When we chain this with another `.rebalance()`, the latter is ignored becaus

Re: Multiple rebalances are incorrectly ignored in some cases.

2020-04-27 Thread David Morávek
properties/RequestedGlobalProperties.java#L301 D. On Mon, Apr 27, 2020 at 11:24 AM Aljoscha Krettek wrote: > On 27.04.20 09:34, David Morávek wrote: > > > When we include `flatMap` in between rebalances -> > > `.rebalance().flatMap(...).rebalance()`, we need to reshuffle again, > > because data

Re: [DISCUSS] FLIP-126: Unify (and separate) Watermark Assigners

2020-04-27 Thread David Anderson
-orderness, and uses those statistics to implement a variable delay. David On Mon, Apr 27, 2020 at 11:44 AM Kostas Kloudas wrote: > Hi Aljoscha, > > Thanks for opening the discussion! > > I have two comments on the FLIP: > 1) we could add lifecycle methods to the Generator, i.

Re: [DISCUSS] Move docker development into versioned branches

2020-04-27 Thread David Anderson
Makes sense to me. I think this would align well enough with user expectations, and be more straightforward. David On Thu, Apr 23, 2020 at 1:23 PM Chesnay Schepler wrote: > Hello everyone, > > Currently, all development in docker-flink occurs on the master branch, > for all rele

Re: [DISCUSS] Rename the SQL ANY type to OPAQUE type

2019-10-22 Thread David Anderson
e: >> > >> >> Hi everyone, >> >> >> >> Stephan pointed out that our naming of a generic/blackbox/opaque type in >> >> SQL might be not intuitive for users. As the term ANY rather describes a >> >> "super-class of all types"

Re: [DISCUSS] Introduce a location-oriented two-stage query mechanism toimprove the queryable state.

2019-10-25 Thread David Anderson
During recovery the state can regress, in the sense that it reflects an earlier point in time than what may've been previously fetched. (4) The state that is wanted (e.g., window state, or operator state) isn't queryable. Best, David On Fri, Oct 25, 2019 at 9:51 AM vino yang wrote: >

connection timeout during shuffle initialization

2020-01-27 Thread David Morávek
ralelism remains the same due to beam nature (400). Does this sound familiar to anyone? Do you have any suggestions how to solve this? Thanks for help, David

Re: connection timeout during shuffle initialization

2020-01-28 Thread David Morávek
ecking TM's io stats. D. On Tue, Jan 28, 2020 at 11:45 AM Piotr Nowojski wrote: > Hi David, > > The usual cause for connection time out is long deployment. For example if > your Job's jar is large and takes long time to distribute across the > cluster. I’m not sure if larg

Re: connection timeout during shuffle initialization

2020-01-29 Thread David Morávek
. > > > > > Also there shouldn't be any state involved > > > (unless Beam IO's use it internally). > > > > My bad. Instead of > > > > > - data size per single job run ranging from 100G to 1TB > > > > I read state size 100G to 1TB. > >

Re: connection timeout during shuffle initialization

2020-01-29 Thread David Morávek
On Wed, Jan 29, 2020 at 4:45 PM Piotr Nowojski wrote: > Hi David, > > > with high load and io waits > > How high values are talking about? > > Could you attach a CPU profiler and post the results somehow? Which > threads are busy on what call trees? > > > &g

Re: connection timeout during shuffle initialization

2020-01-29 Thread David Morávek
Just to clarify, these are bare metal nodes (128G ram, 16 cpus + hyperthreading, 4xHDDS, 10g network), which run yarn, hdfs and hbase. D. On Wed, Jan 29, 2020 at 5:03 PM David Morávek wrote: > Hi Piotr, > > removal of buffer prefetch in BoundedBlockingSubpartitionReader did not >

Re: [VOTE] Release 1.10.0, release candidate #1

2020-01-31 Thread David Anderson
removed ExternalCatalog interface. I found the documentation for the new Catalog API insufficient for figuring out how to adapt. David On Fri, Jan 31, 2020 at 10:00 AM Aljoscha Krettek wrote: > +1 (binding) > > - I verified the checksum > - I verified the signatures > - I eyeballed

Re: [DISCUSS] Feature freeze for Flink 1.7

2018-10-01 Thread David Anderson
tic, too early, too late) and > >> which features/issues do you think should get priority? > >> > >> Since every release needs a release manager, I would also volunteer for > it. > >> > >> Cheers, > >> Till > >> > > -- *David Anderson* | Training Coordinator | data Artisans -- Join Flink Forward - The Apache Flink Conference Stream Processing | Event Driven | Real Time

Re: [DISCUSS] Clean up and reorganize the JIRA components

2019-02-11 Thread David Anderson
> > We can also try to assign documentation components to both "Documentation" > and the affected component, such as "Runtime / Metrics". This is critical for anyone trying to keep an eye on the documentation as a whole -- e.g., ensuring that it remains readable, is well-organized, and is being t

Re: [DISCUSS] Adopting a Code Style and Quality Guide

2019-06-23 Thread David Morávek
self. The one Beam uses is https://github.com/diffplug/spotless with GoogleJavaFormat, it may be worth to look into. Best, David On Fri, Jun 21, 2019 at 4:40 PM Stephan Ewen wrote: > Thanks, everyone, for the positive feedback :-) > > @Robert - It probably makes sense to break this down

Exception but successfully build

2016-07-04 Thread David Herzog
what could cause this problem? And do you have any tips to save time for building Flink? Regards, David Herzog

TaskManager Transfer buffers

2016-08-09 Thread David Herzog
from other Tasks are handled and how the ResultPartition actually answers those request. Thank you in advance :) Regards David Herzog

[DISCUSS] Proposal for Asynchronous I/O in FLINK

2016-09-13 Thread Jeffery David
for the FLINK users to implement their async i/o codes in FLINK job. Here is the link to Google Doc: https://docs.google.com/document/d/1Lr9UYXEz6s6R_3PWg3bZQLF3upGaNEkc0rQCFSzaYDI/edit Any feedback is appreciated. Thanks, David

[DISCUSS] Proposal for Asynchronous I/O in FLINK

2016-09-13 Thread Jeffery David
the FLINK users to implement their async i/o codes in FLINK job. Here is the link to Google Doc: https://docs.google.com/ document/d/1Lr9UYXEz6s6R_3PWg3bZQLF3upGaNEkc0rQCFSzaYDI/edit Any feedback is appreciated. Thanks, David

Re: [DISCUSS] Proposal for Asynchronous I/O in FLINK

2016-09-18 Thread David Wang
Hi Henry, Here is the FLIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673 Thanks, David 2016-09-15 12:26 GMT+08:00 Henry Saputra : > HI David, > > Thanks so much for the interest to contribute to Apache Flink. > > To help review and DISCUSS for new f

Re: [DISCUSS] Proposal for Asynchronous I/O in FLINK

2016-09-19 Thread David Wang
is Async Client's reponsibility to handle timeout issue, but not FLINK. In AsyncFunction, user can configure connection timeout while setting up the function. AsyncWaitOperator provides a way to propogate timeout errors from async operation.* Thanks, David 2016-09-20 5:06 GMT+08:00 Fabian H

Re: [DISCUSS] Proposal for Asynchronous I/O in FLINK

2016-09-21 Thread David Wang
will make a note in the FLIP about how to share resources ;D Thanks, David 2016-09-22 1:46 GMT+08:00 Stephan Ewen : > @Shannon: One could have a "static" broker to share the same netty across > slots in the same JVM. Implicitly, Flink does the same with broadcast > variables.

Re: FLIP-6 and FLIP-12

2016-10-26 Thread David Wang
+1 sounds great! Thanks, David 2016-10-26 20:07 GMT+08:00 Till Rohrmann : > +1 for the changes and thanks for updating the Flips. > > Cheers, > Till > > On Wed, Oct 26, 2016 at 1:05 PM, Stephan Ewen wrote: > > > Hi all! > > > > I updated the wiki with th

Re: [jira] [Created] (FLINK-6167) Consider removing ArchivedExecutionGraph classes

2017-03-27 Thread David Wang
Hi Chesnay, If Acrhived* versions of ExecutionGraph classes have been removed, is there any way to retrieve data in ExecutionGraph from JobMaster via RPC? Some of our projects rely on the Archived* classes from Job Masters running in the cluster. Thanks, David 2017-03-22 22:11 GMT+08:00 Chesnay

Re: [DISCUSS] Releasing Flink 1.11.2

2020-09-02 Thread David Anderson
I think it's worth considering whether we can get this bugfix included in 1.11.2: - FLINK-19109 Split Reader eats chained periodic watermarks There is a PR, but it's still a work in progress. Cc'ing Roman, who has been working on this. Regards, David On Wed, Sep 2, 2020 at

Re: [VOTE] FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API)

2020-09-03 Thread David Anderson
+1 On Thu, Sep 3, 2020 at 3:06 PM Piotr Nowojski wrote: > +1 > > czw., 3 wrz 2020 o 10:08 Guowei Ma napisał(a): > > > +1 > > Looking forward to having a unified datastream api. > > Best, > > Guowei > > > > > > On Thu, Sep 3, 2020 at 3:46 PM Dawid Wysakowicz > > wrote: > > > > > +1 > > > > > >

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

2020-09-10 Thread David Anderson
ere users have explicitly specified something that isn’t going to happen. (I would also like to see a warning given for any job that uses event time timers without having a watermark strategy, though that's unrelated to the topic at hand.) David On Thu, Sep 10, 2020 at 2:57 PM Dawid Wysako

Re: [VOTE] Release 1.11.2, release candidate #1

2020-09-14 Thread David Anderson
+1 Checks: - Verified that the fix for FLINK-19109 solves the problem I reported, running against the maven artifacts On Thu, Sep 10, 2020 at 9:04 AM Zhu Zhu wrote: > Hi everyone, > > Please review and vote on the release candidate #1 for the version 1.11.2, > as follows: > [ ] +1, Approve th

Re: [ANNOUNCE] New Apache Flink Committer - Niels Basjes

2020-09-14 Thread David Anderson
Congratulations! --David On Mon, Sep 14, 2020 at 8:24 PM Arvid Heise wrote: > Congrats Niels! > > On Mon, Sep 14, 2020 at 4:04 PM Benchao Li wrote: > > > Congratulations! > > > > Xingbo Huang 于2020年9月14日周一 下午9:36写道: > > > > > Congratulations! &g

Re: [ANNOUNCE] New Apache Flink Committer - Igal Shilman

2020-09-15 Thread David Anderson
Congrats, Igal! Well deserved. David On Tue, Sep 15, 2020 at 10:14 AM Tzu-Li (Gordon) Tai wrote: > Hi all, > > It's great seeing many new Flink committers recently, and to add to that > I'd like to announce one more new committer: Igal Shilman! > > Igal has be

Re: [ANNOUNCE] New Apache Flink Committer - Arvid Heise

2020-09-15 Thread David Anderson
Congratulations, Arvid! Well deserved. Best, David On Tue, Sep 15, 2020 at 10:23 AM Paul Lam wrote: > Congrats, Arvid! > > Best, > Paul Lam > > > 2020年9月15日 15:29,Jingsong Li 写道: > > > > Congratulations Arvid ! > > > > Best, > > Jin

Re: [VOTE] FLIP-142: Disentangle StateBackends from Checkpointing

2020-09-15 Thread David Anderson
+1 (binding) David On Tue, Sep 15, 2020 at 10:25 AM Tzu-Li (Gordon) Tai wrote: > +1 (binding) > > On Tue, Sep 15, 2020 at 3:26 PM Aljoscha Krettek > wrote: > > > +1 (binding) > > > > Aljoscha > > >

Re: KeyedCoProcessFunction, processElement1, processElement2, onTimer timeout

2020-09-16 Thread David Anderson
Mazen, questions like this are better suited to the user mailing list. FYI, this is also being discussed on stackoverflow: https://stackoverflow.com/questions/63902457/flink-timeout-using-keyedcoprocessfunction-and-order-of-reading-for-flinkkafkaco Regards, David On Wed, Sep 16, 2020 at 9:41 AM

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

2020-09-16 Thread David Anderson
Aljoscha, Thanks for the thorough response. I'm still wanting to think about and discuss the Trigger topic some more, but I'm content with where you've left it for now. Everything else seems good. David On Fri, Sep 11, 2020 at 2:08 PM Aljoscha Krettek wrote: > Thanks

Re: Re: [VOTE] FLIP-134: Batch execution for the DataStream API

2020-09-16 Thread David Anderson
+1 Thanks for the discussion. Looking forward to seeing this all come to pass! David On Wed, Sep 16, 2020 at 9:28 AM Guowei Ma wrote: > +1 (non-binding) > > Best, > Guowei > > > On Wed, Sep 16, 2020 at 3:03 PM Yun Gao > wrote: > > > +1 (non-binding) > >

Re: Need help in setting up flink 1.10

2020-09-29 Thread David Anderson
ience with GCS, but I hope this helps. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/internals/filesystems.html#implementations [2] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/filesystems/#pluggable-file-systems Regards, David On Tue, Sep 29, 2020 at 3:04 AM

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-13 Thread David Anderson
I think the pertinent question is whether there are interesting cases where the BucketingSink is still a better choice. One case I'm not sure about is the situation described in docs for the StreamingFileSink under Important Note 2 [1]: ... upon normal termination of a job, the last in-progres

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-13 Thread David Anderson
tdown Handling -- and in fact, the StreamingFileSink is mentioned in that FLIP as a motivating use case. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs On Tue, Oct 13, 2020 at 1:01 PM Aljoscha Krettek wrote: > On 13.10.20 11:18, David Anderson wrote:

Re: [DISCUSS] Support KeyedSortedMapState in DataStream API

2020-10-14 Thread David Anderson
with Flink Forward preparations at the moment. Best, David On Wed, Oct 14, 2020 at 9:30 AM Sean Z wrote: > Hi devs, > > Current DataStream API doesn't have SortedMapState supported. There are > lots of use cases based on sorted time-series data like range-query or > higher/low

Re: [DISCUSS] Make SQL docs Blink only

2020-12-08 Thread David Anderson
I agree -- I think separating out the legacy planner info should make things clearer for everyone, and then some day we can simply drop it. Plus, doing it now will make it easier to make improvements to the docs going forward. David On Tue, Dec 8, 2020 at 4:38 PM Timo Walther wrote: > Hi S

Re: [DISCUSS] Releasing Apache Flink 1.12.1

2020-12-17 Thread David Morávek
Hi, I think https://issues.apache.org/jira/browse/FLINK-20648 should be addressed, as Kubernetes HA was one of the main selling points of this release. WDYT? D. Sent from my iPhone > On 17. 12. 2020, at 13:54, Yun Tang wrote: > > Thanks for driving this quick-fix release. > +1 for fixing th

Re: [DISCUSS] Cleaning up HighAvailabilityServices interface to reflect the per-JM-process LeaderElection

2022-12-10 Thread David Morávek
Hi Dong, > Adding regarding the effort to add back the per-component election capability: given that the implementation already follows per-process election, and given that there will likely be a lot of extra design/implementation/test effort needed to achieve the use-cases described above, maybe

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

2022-12-12 Thread David Morávek
Hi Shammon, I'm starting to see what you're trying to achieve, and it's really exciting. I share Piotr's concerns about e2e latency and disability to use unaligned checkpoints. I have a couple of questions that are not clear to me from going over the FLIP: 1) Global Checkpoint Commit Are you pl

Re: [DISCUSS] Release Flink 1.16.1

2022-12-15 Thread David Anderson
Martijn, Thank you for bringing this up. From my (admittedly narrow) perspective, I'd like to see a release sooner rather than later, as there's an already merged bug fix I'd like to get released. David On Thu, Dec 15, 2022 at 1:53 PM Martijn Visser wrote: > Hi everyone, &

Re: [VOTE] Update Flink's Scala 2.12 support from 2.12.7 to 2.12.16

2022-12-17 Thread David Anderson
+1 (binding) On Fri, Dec 16, 2022 at 12:22 PM Martijn Visser wrote: > Hi all, > > I'm bumping this old vote thread once more. > > If we want to add Java 17 support at some point, we will need to update > our Scala 2.12 version (see > https://issues.apache.org/jira/browse/FLINK-25000). As expla

Re: [DISCUSS] Extending the feature freezing date of Flink 1.17

2023-01-02 Thread David Anderson
I'm also in favor of extending the feature freeze to Jan 31st. David On Thu, Dec 29, 2022 at 9:01 AM Leonard Xu wrote: > Thanks Qingsheng for the proposal, the pandemic has really impacted > development schedules. > > Jan 31st makes sense to me. > > > Best, > Leonard > >

Re: Reworking the Rescale API

2023-01-26 Thread David Morávek
Hi Gyula, > can you please explain why the AdaptiveScheduler is not the default > scheduler? There are still some smaller bits missing. As far as I know, the missing parts are: 1) Local recovery (reusing the already downloaded state files after restart / rescale) 2) Support for fine-grained re

Re: Reworking the Rescale API

2023-01-27 Thread David Morávek
it was very much meant as a better version of the default scheduler for > streaming jobs. > > On 26/01/2023 19:06, David Morávek wrote: > > Hi Gyula, > > > > > >> can you please explain why the AdaptiveScheduler is not the default > >> scheduler?

Re: Reworking the Rescale API

2023-02-01 Thread David Morávek
, it's already prepared and we should be able to publish it until Friday. Best, D. On Wed, Feb 1, 2023 at 9:56 AM Gyula Fóra wrote: > Chesnay, David: > > Thank you guys for the extra information. We were clearly missing some > context here around the scheduler related efforts an

[DISCUSS] FLIP-291: Externalized Declarative Resource Management

2023-02-03 Thread David Morávek
Hi everyone, This FLIP [1] introduces a new REST API for declaring resource requirements for the Adaptive Scheduler. There seems to be a clear need for this API based on the discussion on the "Reworking the Rescale API" [2] thread. Before we get started, this work is heavily based on the prototyp

Re: [VOTE] FLIP-285: Refactoring LeaderElection to make Flink support multi-component leader election out-of-the-box

2023-02-03 Thread David Morávek
Thanks for the detailed FLIP, Matthias; this will simplify the HA code-base significantly. +1 (binding) Best, D. On Tue, Jan 31, 2023 at 5:22 AM Yang Wang wrote: > +1 (Binding) > > Best, > Yang > > ConradJam 于2023年1月31日周二 12:09写道: > > > +1 non-binding > > > > Matthias Pohl 于2023年1月25日周三 17:3

Re: [DISCUSS] Incorporate performance regression monitoring into routine process

2023-02-03 Thread David Anderson
+1 I don't have anything substantive to add, but I want to express how pleased I am to see this conversation happening. David On Thu, Feb 2, 2023 at 5:09 AM Martijn Visser wrote: > Hi all, > > +1 for the overall proposal. My feedback matches with what Matthias > has already

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

2023-02-07 Thread David Morávek
e* What is the default value here (based on what configuration), or just > infinite? > Currently, for the lower bound, it's always one, and for the upper bound, it's either parallelism (if defined) or the maxParallelism of the vertex in JobGraph. This question might be another signal

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

2023-02-12 Thread David Morávek
writing the whole payload by hand. WDYT? Best, D. On Fri, Feb 10, 2023 at 11:21 AM feng xiangyu wrote: > Hi David, > > Thanks for creating this flip. I think this work it is very useful, > especially in autoscaling scenario. I would like to share some questions > from my view. &

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

2023-02-20 Thread David Morávek
requirements have changed. This wouldn't achieve what we had in mind, > > i.e. sticking to the old resource requirements until enough slots are > > available to fulfil the new resource requirements. So this may not be > > 100% what we need but it could be extended to do

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

2023-02-23 Thread David Morávek
t, D. On Tue, Feb 21, 2023 at 12:24 AM John Roesler wrote: > Thanks for the FLIP, David! > > I just had one small question. IIUC, the REST API PUT request will go > through the new DispatcherGateway method to be handled. Then, after > validation, the dispatcher would call the new Job

Re: [ANNOUNCE] New Apache Flink Committer - Anton Kalashnikov

2023-02-23 Thread David Morávek
Congratulations Anton, well deserved! Best, d. On Wed, Feb 22, 2023 at 5:01 AM Jane Chan wrote: > Congratulations, Anton! > > Best regards, > Jane > > On Wed, Feb 22, 2023 at 11:22 AM Yun Tang wrote: > > > Congratulations, Anton! > > > > Best > > Yun Tang > > >

Re: [DISCUSS] Deprecating GlobalAggregateManager

2023-02-27 Thread David Morávek
I think this makes sense, +1 from my side; as I wrote on the ticket, I'm not aware of any other usages apart from the kinesis connector, and we already have more feature complete API that can replace the functionality there. Best, D. On Mon, Feb 27, 2023 at 2:44 PM Zhanghao Chen wrote: > Hi dev

Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager

2023-02-27 Thread David Morávek
Hi Weihua, I still need to dig into the details, but the overall sentiment of this change sounds reasonable. Best, D. On Mon, Feb 27, 2023 at 2:26 PM Zhanghao Chen wrote: > Thanks for driving this topic. I think this FLIP could help clean up the > codebase to make it easier to maintain. +1 on i

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

2023-02-28 Thread David Morávek
ll transition into the executing state. We're still planning to dig deeper in this direction with other efforts, but this is already good enough and should allow us to move the FLIP forward. WDYT? Unless there are any objectives against the above, I'd like to proceed to a vote. Bes

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

2023-02-28 Thread David Morávek
lism might be useful in > environments with multiple jobs: Scheduler will then have an option to stop > the less suitable job. > In other setups, where the job should not be stopped at all, the user can > always set it to 0. > > Regards, > Roman > > > On Tue, Feb 28,

[VOTE] FLIP-291: Externalized Declarative Resource Management

2023-02-28 Thread David Morávek
Hi Everyone, I want to start the vote on FLIP-291: Externalized Declarative Resource Management [1]. The FLIP was discussed in this thread [2]. The goal of the FLIP is to enable external declaration of the resource requirements of a running job. The vote will last for at least 72 hours (Friday,

Re: [VOTE] FLIP-291: Externalized Declarative Resource Management

2023-03-03 Thread David Morávek
Thanks, everyone, I'm closing this vote now. I'll follow up with the result in a separate email. On Wed, Mar 1, 2023 at 10:01 AM Shammon FY wrote: > +1 (non-binding) > > Best, > Shammon > > > On Wed, Mar 1, 2023 at 4:51 PM Roman Khachatryan wrote: > > >

[RESULT][VOTE] FLIP-291: Externalized Declarative Resource Management

2023-03-03 Thread David Morávek
I'm happy to announce that we have unanimously approved this FLIP. There are 8 approving votes, 3 of which are binding: * John Roesler * Konstantin Knauf (binding) * Zhanghao Chen * ConradJam * Feng Xiangyu * Gyula Fóra (binding) * Roman Khachatryan (binding) * Shammon FY There are no disapprovi

Re: Regarding new command to download jars in flink cluster

2023-03-04 Thread David Morávek
Hi Surendra, Since you're mentioning docker, I assume you're deploying your application to k8s. Is that correct? For handcrafted Kubernetes deployments, you can simply download the jar into the user lib folder in an init container [1]. You can usually reuse existing docker images to download the

Re: [Vote] FLIP-298: Unifying the Implementation of SlotManager

2023-03-09 Thread David Morávek
+1 (binding) Best, D. On Fri, Mar 10, 2023 at 4:49 AM Yuxin Tan wrote: > Thanks, Weihua! > +1 (non-binding) > > Best, > Yuxin > > > weijie guo 于2023年3月10日周五 11:29写道: > > > +1 (binding) > > > > Best regards, > > > > Weijie > > > > > > Shammon FY 于2023年3月10日周五 11:02写道: > > > > > Thanks weihua,

Re: Re: [DISCUSS] Extract core autoscaling algorithm as new SubModule in flink-kubernetes-operator

2023-03-13 Thread David Morávek
> Although YARN serves as the platform for Flink, does YARN also operate on K8s? YARN is an alternative to k8s and Flink should make no assumptions about how it's deployed, even though some companies might deploy it as an overlay RM on top of k8s (I doubt that, but I guess they might do it for leg

Re: [DISCUSS] FLIP-304: Pluggable failure handling for Apache Flink

2023-03-19 Thread David Morávek
Hi Panagiotis, This is an excellent proposal and something everyone trying to provide "Flink as a service" needs to solve at some point. I have a couple of questions: If I understand the proposal correctly, this is just about adding tags to the Throwable by running a tuple of (Throwable, FailureC

Re: [DISCUSS] FLIP-304: Pluggable failure handling for Apache Flink

2023-03-20 Thread David Morávek
Mon, Mar 20, 2023 at 7:23 AM Panagiotis Garefalakis wrote: > Hey David, Shammon, > > Thanks for the valuable comments! > I am glad you find this proposal useful, some thoughts: > > @Shammon > > 1. How about adding more job information in FailureListenerContext? For > >

Re: [DISCUSS] FLIP-304: Pluggable failure handling for Apache Flink

2023-03-21 Thread David Morávek
recoveried more quickly. > > > > Regards, > > Dian > > > > > > > > On Tue, Mar 21, 2023 at 4:36 PM Gen Luo wrote: > > > > > Hi Panagiotis, > > > > > > Thanks for the proposal. > > > > > > It's useful to enric

Re: [DISCUSS] FLIP-304: Pluggable failure handling for Apache Flink

2023-03-23 Thread David Morávek
dd my view on some of the items: > > > >> > > > >> - independent execution vs ordered execution: I prefer the listeners > > > being > > > >> processed independently from each other because it adds less > > complexity > > > >> code-wi

Re: Unit Testing onTimer() event function with TestHarness - onTimer() not being called

2023-03-25 Thread David Anderson
(testHarness.numEventTimeTimers(), is(1)); // should cause the fire timer to fire testHarness.processWatermark(new Watermark(20L)); assertThat(testHarness.numEventTimeTimers(), is(0)); // verify the results assertThat(testHarness.getOutput(), containsInExactlyThisOrder(6L)); Best, David On Wed, Mar 22, 2023

what happened to the images in FLIP-24: SQL Client?

2023-04-04 Thread David Anderson
Does anyone know what happened to the diagrams that used to be in FLIP-24: SQL Client? The last time I looked at this FLIP -- a few weeks ago -- there were architecture diagrams for Gateway Mode and Embedded Mode, but now those images are missing. David

Re: [Discussion] - Take findify/flink-scala-api under Flink umbrella

2023-04-16 Thread David Morávek
cc dev@f.a.o On Sun, Apr 16, 2023 at 11:42 AM David Morávek wrote: > Hi Alexey, > > I'm a bit skeptical because, looking at the project, I see a couple of red > flags: > > - The project is inactive. The last release and commit are both from the > last May. > - The

Re: [VOTE] FLIP-304: Pluggable Failure Enrichers

2023-04-19 Thread David Morávek
Hi Panos, It seems that most recent discussions (e.g. changing the semantics of the config option) are not reflected in the FLIP. Can you please double-check that this is the correct version? Best, D. On Mon, Apr 17, 2023 at 9:24 AM Panagiotis Garefalakis wrote: > Hello everyone, > > I want t

Re: [VOTE] FLIP-304: Pluggable Failure Enrichers

2023-04-20 Thread David Morávek
Thanks for the update! +1 (binding) Best, D. On Thu, Apr 20, 2023 at 9:50 AM Piotr Nowojski wrote: > Hi, > > I see that the FLIP has been updated, thanks Panos! > > +1 (binding) > > Best, > Piotrek > > śr., 19 kwi 2023 o 13:49 Piotr Nowojski > napisał(a)

Re: [ANNOUNCE] New Apache Flink PMC Member - Leonard Xu

2023-04-21 Thread David Morávek
Congratulations, Leonard, well deserved! Best, D. On Fri 21. 4. 2023 at 16:40, Feng Jin wrote: > Congratulations, Leonard > > > > Best, > Feng Jin > > On Fri, Apr 21, 2023 at 8:38 PM Mang Zhang wrote: > > > Congratulations, Leonard. > > > > > > -- > > > > Best regards, > > Mang Zhang > >

<    1   2   3   4   5   6   7   8   >