Re: [ANNOUNCE] Jiangjie (Becket) Qin has been added as a committer to the Flink project

Congrats! On Thu, Jul 18, 2019 at 9:58 AM Richard Deurwaarder wrote: > Congrats Becket! :) > > Richard > > On Thu, Jul 18, 2019 at 5:52 PM Xuefu Z wrote: > > > Congratulation, Becket! At least you're able to assign JIRAs now! > > > > On Thu, Jul 18, 2019 at 8:22 AM Rong Rong wrote: > > > > >

REST API / JarRunHandler: More flexibility for launching jobs

Hi, While considering different options to launch Beam jobs through the Flink REST API, I noticed that the implementation of JarRunHandler places quite a few restrictions on how the entry point shall construct a Flink job, by extracting and manipulating the job graph. That's normally not a proble

Fine Grained Recovery / FLIP-1

Hi, We are using Flink for streaming and find the "stop-the-world" recovery behavior of Flink prohibitive for use cases that prioritize availability. Partial recovery as outlined in FLIP-1 would probably alleviate these concerns. https://cwiki.apache.org/confluence/display/FLINK/FLIP-1+%3A+Fine+G

Re: REST API / JarRunHandler: More flexibility for launching jobs

JarRunHandler? If another process > is building the JobGraph, then one could use the JobSubmitHandler which > expects a JobGraph and then starts executing it. > > Cheers, > Till > > On Thu, Jul 25, 2019 at 7:45 PM Thomas Weise wrote: > > > Hi, > > > > While co

Re: Fine Grained Recovery / FLIP-1

We have already done many works about the new shuffle in the old Flink > > shuffle architectures because many of our customers have the concern. We > > have a plan to move the work to the new Flink pluggable shuffle > > architecture. > > > > Best, > > Guowei

Re: REST API / JarRunHandler: More flexibility for launching jobs

; interface, even though it is a bit outdated and might get removed in the >>> future, offers a `getPlan` method which is called in order to generate >>> the >>> `JobGraph`. In the client refactoring discussion thread it is currently >>> being discussed what to do wi

Re: REST API / JarRunHandler: More flexibility for launching jobs

> it seems also unable to customize... > > Best, > tison. > > > Thomas Weise 于2019年8月3日周六 上午4:09写道： > > > Thanks for looking into this. > > > > I see the "Jar run handler" as function that takes few parameters and > > returns a job ID. I think i

Checkpointing under backpressure

Hi, One of the major operational difficulties we observe with Flink are checkpoint timeouts under backpressure. I'm looking for both confirmation of my understanding of the current behavior as well as pointers for future improvement work: Prior to introduction of credit based flow control in the

Re: [DISCUSS] Flink Python User-Defined Function for Table API

Hi Jincheng, Thanks for putting this together. The proposal is very detailed, thorough and for me as a Beam Flink runner contributor easy to understand :) One thing that you should probably detail more is the bundle processing. It is critically important for performance that multiple elements are

Re: [DISCUSS] Merging new features post-feature-freeze

Hi, I agree with others in this thread that everyone should be focused on making the release happen and not push their own agenda. It is generally difficult to manage releases with so many interests and moving parts. Many feel the urgency to complete what they were originally hoping to complete. B

Re: Checkpointing under backpressure

Great discussion! I'm excited that this is already under consideration! Are there any JIRAs or other traces of discussion to follow? Paris, if I understand correctly, then the proposal is to not block any input channel at all, but only log data from the backpressured channel (and make it part of t

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

Congrats! On Wed, Aug 14, 2019, 7:12 AM Robert Metzger wrote: > Congratulations! Very happy to have you onboard :) > > On Wed, Aug 14, 2019 at 4:06 PM Kostas Kloudas wrote: > > > Congratulations Andrey! > > Well deserved! > > > > Kostas > > > > On Wed, Aug 14, 2019 at 4:04 PM Yun Tang wrote:

Re: Watermarks not propagated to WebUI?

I have also noticed this issue (Flink 1.5, Flink 1.8), and it appears with higher parallelism. This can be confusing to the user when watermarks actually work and can be observed using the metrics. On Wed, Aug 14, 2019 at 7:36 AM Jan Lukavský wrote: > Hi, > > is it possible, that watermarks are

Re: Checkpointing under backpressure

;>>>>> IV Marker Bypassing > > >>>>>> --- > > >>>>>> > > >>>>>> 1. (correctness) This leads to equivalent in-flight snapshots so > > with > > >>>>> some quick thi

Cwiki edit access

Hi, I would like to be able to edit pages in the Confluence Flink space. Can someone give me access please? Thanks

Re: [VOTE] Flink Project Bylaws

+0 (binding) I don't think committers should be allowed to approve their own changes. I would prefer if non-committer contributors can approve committer PRs as that would encourage more participation in code review and ability to contribute. On Fri, Aug 16, 2019 at 9:02 PM Shaoxuan Wang wrote:

Re: Cwiki edit access

Thanks! On Mon, Aug 19, 2019 at 1:19 AM Till Rohrmann wrote: > Hi Thomas, > > I've given you access. You should be able to access it now with your Apache > account. Please let me know if something is not working. > > Cheers, > Till > > On Mon, Aug 19, 201

Re: [DISCUSS] Upgrade kinesis connector to Apache 2.0 License and include it in official release

+1 for KCL 1.x changes only. I also think it would make sense to align FLIP-27 work and KCL 2.x related changes, since these will require a hardening cycle with extensive testing that is probably not practical to repeat. On Tue, Aug 20, 2019 at 10:57 AM Bowen Li wrote: > @Stephan @Becket kine

Re: Customize StreamingFileSink: Enable extending StreamingFileSink class

Hi Kailash, Any update on this? Thanks, Thomas On Wed, Aug 7, 2019 at 7:54 AM Fabian Hueske wrote: > Hi Kailash, > > Yes, I think creating another Jira and PR would be the right thing to do. > > Thank you, > Fabian > > Am Fr., 2. Aug. 2019 um 10:29 Uhr schrieb Kailash Dayanand < > kailash...@

Re: [DISCUSS] Best practice to run flink on kubernetes

Till had already summed it up, but I want to emphasize that Flink as project only needs to provide #1 (reactive mode) and #3 (active mode, which necessarily is tied to the cluster manager of choice). The latter would be needed for Flink jobs to be elastic (in the future), although we may want to di

Re: Flink operators for Kubernetes

In case anyone comes across this thread in the archives, the FlinkK8sOperator is now available here: https://github.com/lyft/flinkk8soperator The community is invited to check it out, provide feedback (use github issues) or even better, join and contribute to it. Thomas On Fri, Nov 2, 2018 at

Re: [DISCUSS] Releasing Flink 1.8.2

+1 for the 1.8.2 release I marked https://issues.apache.org/jira/browse/FLINK-13586 for this release. It would be good to compensate for the backward incompatible change to ClosureCleaner that was introduced in 1.8.1, which affects downstream dependencies. Thanks, Thomas On Sun, Sep 1, 2019 at

Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC

Congratulations! On Fri, Sep 6, 2019 at 9:22 AM zhijiang wrote: > Congratulations Klou! > > Best, > Zhijiang > -- > From:Zhu Zhu > Send Time:2019年9月6日(星期五) 17:19 > To:dev > Subject:Re: [ANNOUNCE] Kostas Kloudas joins the Flink PM

Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)

Is the intention to bump the Flink major version and only support Java 17+? If so, can Scala not be upgraded at the same time? Thanks, Thomas On Thu, Apr 27, 2023 at 4:53 PM Martijn Visser wrote: > Scala 2.12.7 doesn't compile on Java 17, see > https://issues.apache.org/jira/browse/FLINK-25000

Re: Kubernetes Operator 1.5.0 release planning

+1 and thanks for volunteering! On Mon, May 1, 2023 at 5:04 PM Peter Huang wrote: > +1, thanks. > > On Mon, May 1, 2023 at 7:58 AM Márton Balassi > wrote: > > > +1, thanks. > > > > On Mon, May 1, 2023 at 4:23 PM Őrhidi Mátyás > > wrote: > > > > > +1 SGTM. > > > > > > Cheers, > > > Matyas > >

Re: [DISCUSS] SqlClient gateway mode: support for URLs

Hi Alex, Thanks for the investigation and for the ideas on how to improve the SQL gateway REST client. I think the solution could come in 2 parts. Near term the existing client can be patched to support URL mapping and HTTPS based on a standard URL. If I understand your proposal correctly, that c

Re: [NOTICE] Flink master branch now uses Maven 3.8.6

Thanks a lot Chesnay for getting this squared away! It was quite painful to have to keep an outdated maven around specifically for building Flink but many had probably given up on this by now ;-) On Sun, May 14, 2023 at 9:33 PM yuxia wrote: > Thanks Chesnay for the efforts. Happy to see we can

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

Hi Mason, Thanks for the iterations on the FLIP, I think this is in a very good shape now. Small correction for the MultiClusterKafkaSourceEnumerator section: "This reader is responsible for discovering and assigning splits from 1+ cluster" Regarding the user facing name of the connector: I agre

Re: [VOTE] FLIP-246: Dynamic Kafka Source (originally Multi Cluster Kafka Source)

+1 (binding) On Mon, Jun 19, 2023 at 8:09 AM Ryan van Huuksloot wrote: > +1 (non-binding) > > +1 for DynamicKafkaSource > > Ryan van Huuksloot > Sr. Production Engineer | Streaming Platform > [image: Shopify] > > > > On M

Re: [DISSCUSS] Kubernetes Operator Flink Version Support Policy

+1, thanks for the proposal On Tue, Sep 5, 2023 at 8:13 AM Gyula Fóra wrote: > Hi All! > > @Maximilian Michels has raised the question of Flink > version support in the operator before the last release. I would like to > open this discussion publicly so we can finalize this before the next > re

Re: [Discuss] CRD for flink sql gateway in the flink k8s operator

It is already possible to bring up a SQL Gateway as a sidecar utilizing the pod templates - I tend to also see this more of a documentation/example issue rather than something that calls for a separate CRD or other dedicated operator support. Thanks, Thomas On Tue, Sep 19, 2023 at 3:41 PM Gyula

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.6.1, release candidate #1

+1 (binding) - Verified checksums, signatures, source release content - Run unit tests Side note: mvn clean verifyfails with Java 17 compiler. While the build target version may be 11, preferably a higher JDK version can be used to build the source. Caused by: java.lang.IllegalAccessError

Re: [VOTE] Release 1.10.0, release candidate #1

ttps://issues.apache.org/jira/browse/FLINK-15917 > [2] https://issues.apache.org/jira/browse/FLINK-15918 > [3] https://issues.apache.org/jira/browse/FLINK-8949 > > On Wed, Feb 5, 2020 at 7:04 AM Thomas Weise wrote: > >> Hi Gary, >> >> Thanks for the reply. >>

Re: [VOTE] Release 1.10.0, release candidate #1

is a bit unfortunate that these things have been raised so late. > > > Having these rather open ended tickets (how to re-define the existing > > > metrics in the new scheduler/failover handling) now as release blockers > > > would mean that 1.10 is indefinitely delayed. >

Re: [VOTE] Release 1.10.0, release candidate #2

I deployed commit 81cf2f9e59259389a6549b07dcf822ec63c899a4 and can confirm that the dataformat-cbor and checkpoint alignment metric issues are resolved. On Wed, Feb 5, 2020 at 11:26 AM Gary Yao wrote: > Note that there is currently an ongoing discussion about whether > FLINK-15917 > and FLINK-1

Re: [VOTE] Release 1.10.0, release candidate #3

+1 (binding) - verified signatures and hashes - rebased internal deploy to rc3 and verified that previously reported issues are resolved Thank you for addressing these issues promptly! Thomas On Fri, Feb 7, 2020 at 1:54 PM Gary Yao wrote: > Hi everyone, > Please review and vote on the release

Re: [DISCUSS] FLIP-111: Docker image unification

Thanks for working on improvements to the Flink Docker container images. This will be important as more and more users are looking to adopt Kubernetes and other deployment tooling that relies on Docker images. A generic, dynamic configuration mechanism based on environment variables is essential a

Re: [VOTE] [FLIP-76] Unaligned checkpoints

+1 Thanks for putting this together, looking forward to the experimental support in the next release. One clarification: since the MVP won't support rescaling, does it imply that savepoints will always use aligned checkpointing? If so, this would still block the user from taking a savepoint and r

Re: [DISCUSS] FLIP-111: Docker image unification

docker plugin sounds good. I'll take a look at it. > > > > Regarding supporting JAVA 11: > > - Not sure if it is necessary to ship JAVA. Maybe we could just change > > the base image from openjdk:8-jre to openjdk:11-jre in template docker > > file[1]. Correct me i

Re: [ANNOUNCE] New Committers and PMC member

Congratulations! On Wed, Apr 1, 2020 at 9:31 AM Fabian Hueske wrote: > Congrats everyone! > > Cheers, Fabian > > Am Mi., 1. Apr. 2020 um 18:26 Uhr schrieb Yun Tang : > > > Congratulations to all of you! > > > > Best > > Yun Tang > > > > From: Yang Wang > > Sent

Re: [DISCUSS] FLIP-73: Introducing Executors for job submission

Thanks for the proposal. These changes will make it significantly easier to programmatically use Flink in downstream frameworks. Should the new Executor execute method be defined as asynchronous? It could return a job handle to interact with the job and the legacy environments can still block to r

Re: [DISCUSS] FLIP-73: Introducing Executors for job submission

s of JobClient(as FLIP-74). Will start a separated > discussion > thread on that interface as soon as I finish an early version. > > Best, > tison. > > > Thomas Weise 于2019年9月25日周三 上午1:17写道： > > > Thanks for the proposal. These changes will make it significantly

Re: REST API / JarRunHandler: More flexibility for launching jobs

his > is > > not officially guaranteed. > > > > Cheers, > > Till > > > > On Tue, Aug 6, 2019 at 3:56 AM Zili Chen wrote: > > > >> It sounds like a request to change the interface Program into > >> > >> public interface Program { >

Re: [COMMITTER] repo locked due to synchronization issues

+1 for recommendation to use the github repo Thanks, Thomas On Fri, Sep 27, 2019 at 9:29 AM Rong Rong wrote: > +1 on to state with one recommendation method in the wiki. > I haven't encountered this often, so I do not have a preference regarding > which way to go (Gitbox or Github). However, I'

Re: [DISCUSS] FLIP-74: Flink JobClient API

I did not realize there was a plan to deprecate anything in the REST API? The REST API is super important for tooling written in non JVM languages, that does not include a Flink client (like FlinkK8sOperator). The REST API should continue to support all job management operations, including job sub

Re: REST API / JarRunHandler: More flexibility for launching jobs

ipeline) interface that > is > > the common interface of StreamGraph and Plan. Maybe we could reintroduce > > something like Program that returns a Pipeline for this purpose. > > > > Best, > > Aljoscha > > > > > On 26. Sep 2019, at 19:30, Thomas Weise

Re: [DISCUSS] FLIP-27: Refactor Source Interface

Thanks for putting together this proposal! I see that the "Per Split Event Time" and "Event Time Alignment" sections are still TBD. It would probably be good to flesh those out a bit before proceeding too far as the event time alignment will probably influence the interaction with the split reade

Re: [VOTE] FLIP-73: Introducing Executors for job submission

+1 On Fri, Oct 4, 2019 at 8:56 AM Zili Chen wrote: > Thanks for your works Kostas! > > +1 for FLIP-73 > > Best, > tison > > > Kostas Kloudas 于2019年10月4日周五 下午11:40写道： > > > Hi all, > > > > I would like to start the vote for FLIP-73 [1], which is discussed and > > reached a consensus in the disc

Re: [DISCUSS] FLIP-73: Introducing Executors for job submission

It might be useful to mention on FLIP-73 that the intention for Executor.execute is to be an asynchronous API once it becomes public and also refer to FLIP-74 as such. On Fri, Oct 4, 2019 at 2:52 AM Aljoscha Krettek wrote: > Hi Tison, > > I agree, for now the async Executor.execute() is an inte

Re: [VOTE] FLIP-78: Flink Python UDF Environment and Dependency Management

Sorry for joining the discussion late. The Beam environment already supports artifact staging, it works out of the box with the Docker environment. I think it would be helpful to explain in the FLIP how this proposal relates to what Beam offers / how it would be integrated. Thanks, Thomas On Mo

Re: [ANNOUNCE] Apache Flink 1.9.1 released

Thanks Jark and everyone who contributed to the release! I took 1.9.1 for a test drive, upgrading an existing 1.8.x deployment. This brought in the shiny new UI :) Everything looks good so far, except an error message that is flickering on the job overview page: "Server response message: unable t

Re: [ANNOUNCE] Apache Flink 1.9.1 released

Filed as https://issues.apache.org/jira/browse/FLINK-14470 On Sun, Oct 20, 2019 at 7:09 PM Yadong Xie wrote: > Hi Thomas Weise > > We have an issue related to this problem > https://issues.apache.org/jira/browse/FLINK-14147, but I think you could > open another JIRA > The wa

Re: Watermarks not propagated to WebUI?

gt; is small enough. > > > > Not sure whether we ever fixed that. > > > > On 15/08/2019 12:01, Jan Lukavský wrote: > > > Hi, > > > > > > Thomas, thanks for confirming this. I have noticed, that in 1.9 the > > > WebUI

Re: [VOTE] Accept Stateful Functions into Apache Flink

+1 (binding) On Mon, Oct 21, 2019 at 7:10 AM Timo Walther wrote: > +1 (binding) > > Thanks, > Timo > > > On 21.10.19 15:59, Till Rohrmann wrote: > > +1 (binding) > > > > Cheers, > > Till > > > > On Mon, Oct 21, 2019 at 12:13 PM Robert Metzger > wrote: > > > >> +1 (binding) > >> > >> On Mon, Oc

Re: [VOTE] FLIP-78: Flink Python UDF Environment and Dependency Management

gt; As you can see from the section "Goals" that this FLIP focuses on the > > dependency management in process mode. However, the APIs and design > > proposed in this FLIP also applies for the docker mode. So it makes sense > > to me to also describe how this design is i

Re: [VOTE] FLIP-78: Flink Python UDF Environment and Dependency Management

endency Management for python, so I recommend > voting in the current ML for Flink 1.10, Beam artifact staging improvements > are discussed in a separate Beam dev@. > > What do you think? > > Best, > Jincheng > > Thomas Weise 于2019年10月21日周一 下午10:25写道： > > > Beam arti

Re: [ANNOUNCE] Becket Qin joins the Flink PMC

Congrats! -- sent from mobile On Mon, Oct 28, 2019, 12:56 PM Shuyi Chen wrote: > Congratulations! > > On Mon, Oct 28, 2019 at 11:18 AM Xingcan Cui wrote: > > > Congratulations, Becket! > > > > Best, > > Xingcan > > > > > On Oct 28, 2019, at 1:23 PM, Xuefu Z wrote: > > > > > > Congratulations,

Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2

Is there any activity on FLIP-27 that would make it relevant for 1.10 release? Thanks Gary for the update, it provides excellent visibility on current activity and what we can expect with the release. On Fri, Nov 1, 2019 at 1:52 PM Steven Wu wrote: > Gary, FLIP-27 seems to get omitted in the

Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2

from end to > end by this Sunday. Then I'll update FLIP-27 wiki page. > > Thanks, > > Jiangjie (Becket) Qin > > > On Sat, Nov 2, 2019 at 8:18 AM Thomas Weise wrote: > >> Is there any activity on FLIP-27 that would make it relevant for 1.10 >> release? &

Re: [ANNOUNCE] Jark Wu is now part of the Flink PMC

Congratulations! On Fri, Nov 8, 2019 at 5:30 AM Forward Xu wrote: > Congratulations, Jark! > > > Best, > > forward > > Congxian Qiu 于2019年11月8日周五 下午9:27写道： > > > Congratulations, Jark! > > > > Best, > > Congxian > > > > > > Leonard Xu 于2019年11月8日周五 下午8:40写道： > > > > > Congratulations, Jark. >

Re: [DISCUSS] Support configure remote flink jar

There is a related use case (not specific to HDFS) that I came across: It would be nice if the jar upload endpoint could accept the URL of a jar file as alternative to the jar file itself. Such URL could point to an artifactory or distributed file system. Thomas On Mon, Nov 18, 2019 at 7:40 PM

Re: Checkpointing under backpressure

;> > > the > >> > > >>>>> explanation! > >> > > >>>>>> There are a few intermixed concepts here, some how to do with > >> > > >>>>> correctness some with performance. > >

Re: [DISCUSS] Drop Kafka 0.8/0.9

+1 On Fri, Dec 6, 2019 at 3:46 AM Benchao Li wrote: > +1 for dropping. > > Zhenghua Gao 于2019年12月5日周四 下午4:05写道： > > > +1 for dropping. > > > > *Best Regards,* > > *Zhenghua Gao* > > > > > > On Thu, Dec 5, 2019 at 11:08 AM Dian Fu wrote: > > > > > +1 for dropping them. > > > > > > Just FYI: th

Re: [VOTE] Release 1.10.0, release candidate #1

As part of testing the RC, I run into the following issue with a test case that runs a job from a packaged jar on a MiniCluster. This test had to be modified due to the client-side API changes in 1.10. The issue is that the jar file that also contains the entry point isn't part of the user classpa

Re: [VOTE] Release 1.10.0, release candidate #1

The above issue was resolved by adding RemoteEnvironmentConfigUtils.setJarURLsToConfig(new String[] {JAR_PATH}, config); It might be helpful to provide migration instructions to users as part of this release. On Fri, Jan 31, 2020 at 9:20 PM Thomas Weise wrote: > As part of testing the RC

Re: [VOTE] Release 1.10.0, release candidate #1

I found another issue with the Kinesis connector: https://issues.apache.org/jira/browse/FLINK-15868 On Mon, Feb 3, 2020 at 3:35 AM Gary Yao wrote: > Hi everyone, > > I am hereby canceling the vote due to: > > FLINK-15837 > FLINK-15840 > > Another RC will be created later today. > > Bes

Re: [VOTE] Release 1.10.0, release candidate #1

1) Is the low watermark display in the UI still broken? 2) Was there a change in how job recovery reflects in the uptime metric? Didn't uptime previously reset to 0 on recovery (now it just keeps increasing)? Thanks, Thomas On Mon, Feb 3, 2020 at 10:55 AM Thomas Weise wrote: > I found

Re: [VOTE] Release 1.10.0, release candidate #1

t.) Another problem with this 1.10 RC is that the checkpointAlignmentTime metric is missing. (I have not been able to investigate this further yet.) > > Best, > Gary > > [1] https://issues.apache.org/jira/browse/FLINK-14651 > > On Tue, Feb 4, 2020 at 2:56 AM Thomas Weise wrot

Re: [ANNOUNCE] Two new committers: Xingcan Cui and Nico Kruber

Congrats! On Wed, May 9, 2018 at 10:14 AM, Bowen Li wrote: > Congratulations! > > On Tue, May 8, 2018 at 11:06 PM, Zhijiang(wangzhijiang999) < > wangzhijiang...@aliyun.com> wrote: > > > Congratulations, Xingcan and Nico ! > > Nico is a good PR reviewer and I gained a lot from him. > > :)---

Re: [VOTE] Release 1.5.0, release candidate #2

Hi, Regarding the Kinesis consumer, fwiw we backported most of the changes to 1.4.x in our fork and running our pipelines with those. Thanks, Thomas On Tue, May 15, 2018 at 5:50 AM, Till Rohrmann wrote: > Thanks for the feedback. I actually have to cancel this RC because of the > following is

Re: Closing (automatically?) inactive pull requests

I like Till's proposal to notify the participants on the PR to PTAL. But I would also suggest to auto-close when no action is taken, with a friendly reminder that PRs can be reopened anytime. The current situation with 350 open PRs may send a signal to contributors that it may actually be too much

Re: [DISCUSS] GitBox

+1 On Wed, May 16, 2018 at 8:31 AM, Stefan Richter wrote: > +1 > > > Am 16.05.2018 um 12:40 schrieb Chesnay Schepler : > > > > Hello, > > > > during the discussion about how to better manage pull requests [1] the > topic of GitBox integration came up again. > > > > This seems like a good opport

Re: [VOTE] Release 1.5.0, release candidate #4

+1 - verified signatures and hashes - built Kinesis connector from source archive - run Kinesis consumer against AWS - PASSED - run Kinesis consumer against Kinesalite - FAILED https://issues.apache.org/jira/browse/FLINK-9402 Thomas On Thu, May 17, 2018 at 5:50 PM, Till Rohrmann wrote: > Hi

Re: [VOTE] Enable GitBox integration (#2)

+1 On Tue, May 22, 2018 at 2:37 AM, Timo Walther wrote: > +1 > > Am 22.05.18 um 10:49 schrieb Ted Yu: > > +1 >> Original message From: Chesnay Schepler < >> ches...@apache.org> Date: 5/22/18 1:12 AM (GMT-08:00) To: >> dev@flink.apache.org Subject: [VOTE] Enable GitBox integra

Re: [VOTE] Release 1.5.0, release candidate #4

. This bug lets certain batch > jobs > > fail. > > > > [1] https://issues.apache.org/jira/browse/FLINK-9406 > > > > Cheers, > > Till > > > > On Sat, May 19, 2018 at 9:17 AM, Thomas Weise wrote: > > > >> +1 > >> > >> - v

Re: [VOTE] Release 1.5.0, release candidate #6

+1 (non-binding) - checked signatures and hashes - run Kinesis consumer test Regarding the rate of issues being found: I would credit some of it to increased coverage and due diligence and some to to the nature of changes in the release. It is probably expected that a 1.5.1 release will follow s

Re: [DISCUSS] Flink 1.4 and below STOPS writing to Kinesis after June 12th.

--> On Wed, Jun 6, 2018 at 1:06 AM, Bowen Li wrote: > Hi, > > I think the following email thread might have gone lost. > > Dyana brought up the attention that AWS has informed users that KPL > versions > < 0.12.6 will *stop working* starting from the 12th of June. Flink 1.4 is > using KPL 0.12.5

Re: [DISCUSS] Flink 1.4 and below STOPS writing to Kinesis after June 12th.

on > > > On 11 June 2018 at 12:29:10 PM, Till Rohrmann (trohrm...@apache.org) > wrote: > > I agree that it would be nice to offer a backport for all 1.4 users. > Thomas > would it be possible to contribute the backport back to the community? > > Cheers, > Till > &g

Flink Docker 1.5.x local mode

I tried to run the 1.5.0 and 1.5.1 docker containers with local mode, following the instructions at https://docs.docker.com/samples/library/flink/#run-a-flink-local-cluster docker run --name flink_local -p 8081:8081 -t flink local Starts the cluster, but it does not start a task manager. That's

Re: Flink Docker 1.5.x local mode

; > > Docker documentation of flink : > > https://docs.docker.com/samples/library/flink/ > > > > [1]: > > > > > https://github.com/docker-flink/docker-flink/blob/27cd2b49486a31fa449d8551be58e23c576bf44c/1.5/scala_2.11-debian/docker-entrypoint.sh#L56 > > > >

Backwards compatibility for REST API

Hi, Are there any compatibility guarantees for the Flink REST API or plans to establish the same? I'm asking since I noticed that the 1.5.1 release has an incompatible change that prevents a 1.5.0 client to submit a job to 1.5.0 server (FLINK-9280, removed endpoint /blobserver/port). Such change

Re: Backwards compatibility for REST API

link CLI). > Note we do not provide any guarantee that Flink components interact > properly if the versions don't exactly match, so you should always upgrade > the clients anyway. > > On 24.07.2018 16:43, Thomas Weise wrote: > > Hi, > > > > Are there any compatibilit

Re: [VOTE] Release 1.6.0, release candidate #1

Is there something that should be done regarding Chesnay's comment WRT the REST API: https://lists.apache.org/thread.html/513a67e20abbb7e265d976ee9c5824ad3297a89ec275bcd93c08a03c@%3Cdev.flink.apache.org%3E Thanks, Thomas On Thu, Jul 26, 2018 at 7:51 AM Renjie Liu wrote: > This bug is also int

Re: [VOTE] Release 1.5.2, release candidate #2

+1 but see below Following shasum verification failed: shasum: /mnt/c/Users/Zento/Documents/GitHub/flink/tools/releasing/release/flink-1.5.2-src.tgz: /mnt/c/Users/Zento/Documents/GitHub/flink/tools/releasing/release/flink-1.5.2-src.tgz: FAILED open or read The problem can be corrected by fixing

Re: Consuming data from dynamoDB streams to flink

The user is yxu-lyft, Ying had commented on that JIRA as well. https://issues.apache.org/jira/browse/FLINK-4582 On Mon, Jul 30, 2018 at 1:25 AM Fabian Hueske wrote: > Hi Ying, > > Thanks for considering to contribute the connector! > > In general, you don't need special permissions to contribu

Re: [RESULT][VOTE] Release 1.5.2, release candidate #2

Thanks to Chesnay for running the release and getting it completed quickly! On Tue, Jul 31, 2018 at 2:02 AM Fabian Hueske wrote: > Thanks Chesnay! > > 2018-07-31 10:59 GMT+02:00 vino yang : > > > Thanks for releasing Flink 1.5.2, Chesnay! > > > > Thanks. > > Vino. > > > > 2018-07-31 16:49 GMT+0

Re: Setting an allowable number of checkpoint failures

Hi, What we are looking for is that the job does *not* restart on transient checkpoint failures and we would like to cap the number of allowable subsequent failures until a restart occurs. The reason is that every restart is a service interruption that is potentially very expensive. Thanks, Thom

Re: Setting an allowable number of checkpoint failures

vino. > > 2018-08-06 21:22 GMT+08:00 Thomas Weise : > > > Hi, > > > > What we are looking for is that the job does *not* restart on transient > > checkpoint failures and we would like to cap the number of allowable > > subsequent failures until a restart

Re: Flink 1.7 Development Priorities

Any plans to advance the "splittable source" idea? It probably won't be ready to implement, but important to get on the radar. Thanks, Thomas On Thu, Aug 23, 2018 at 1:12 AM Aljoscha Krettek wrote: > Hi Everyone, > > After the recent Flink 1.6 release the people working on Flink at data > Artis

Re: [ANNOUNCE] New committer Gary Yao

Congrats, Gary! On Fri, Sep 7, 2018 at 4:17 PM Dawid Wysakowicz wrote: > Congratulations Gary! Well deserved! > > On 07/09/18 16:00, zhangmingleihe wrote: > > Congrats Gary! > > > > Cheers > > Minglei > > > >> 在 2018年9月7日，下午9:59，Andrey Zagrebin 写道： > >> > >> Congratulations Gary! > >> > >>> On

Re: [PROPOSAL] [community] A more structured approach to reviews and contributions

Follow-up regarding the PR template that pops up when opening a PR: I think what we have now is a fairly big blob of text that jumps up a bit unexpectedly for a first time contributor and is also cumbersome to deal with in the small PR description window. Perhaps we can improve it a bit: * Instea

Re: [DISCUSS] [Contributing] (2) - Review Steps

I think that all discussion/coordination related to a contribution / PR should be handled through the official project channel. I would also prefer that there are no designated "owners" and "experts", for the reasons Fabian mentioned. Ideally there is no need to have "suggested reviewers" either,

Sharing state between subtasks

I'm looking to implement a state sharing mechanism between subtasks (of one or multiple tasks). Our use case is to align watermarks between subtasks of one or multiple sources to prevent some data fetchers to race ahead of others and cause massive state buffering in Flink. Each subtask would share

Re: [DISCUSS] [Contributing] (2) - Review Steps

to help its contributors and committers, then take advantage > of > > >> "open source". > > >> > > >> Best, > > >> tison. > > >> > > >> [1] http://www.apache.org/dev/contributors#patches > > >> > >

Re: Sharing state between subtasks

Thanks for the feedback and comments so far. I want to elaborate more on the need for the shared state and awareness of watermark alignment in the source implementation. Sources like Kafka and Kinesis pull from the external system and then emit the records. For Kinesis, we have multiple consumer t

Re: Spreading Tasks across TaskManagers

Hi Till, Thanks for the pointer, glad that this is being worked on. It almost looks like the non deterministic distribution behavior started with 1.5.x (?) and that surprised us. https://issues.apache.org/jira/browse/BEAM-5713 I agree that there is no one strategy that fits every use case. If a

Re: Sharing state between subtasks

Hi, We are planning to work on the Kinesis consumer in the following order: 1. Add per shard watermarking: https://issues.apache.org/jira/browse/FLINK-5697 - this will be code we already use internally; I will open a PR to add it to the Flink Kinesis consumer 2. Exchange of per subtask watermarks

Re: [DISCUSS] FLIP-27: Refactor Source Interface

Thanks for getting the ball rolling on this! Can the number of splits decrease? Yes, splits can be closed and go away. An example would be a shard merge in Kinesis (2 existing shards will be closed and replaced with a new shard). Regarding advance/poll/take: IMO the least restrictive approach wou

Re: [DISCUSS] FLIP-27: Refactor Source Interface

, 2018 at 1:43 PM Thomas Weise wrote: > Thanks for getting the ball rolling on this! > > Can the number of splits decrease? Yes, splits can be closed and go away. > An example would be a shard merge in Kinesis (2 existing shards will be > closed and replaced with a new shard). > &

Kinesis consumer e2e test

Hi, I'm planning to add an end-to-end test for the Kinesis consumer. We have done something similar at Lyft, using Kinesalite, which can be run as Docker container. I see that some tests already make use of Docker, so we can assume it to be present in the target environment(s)? I also found the

  1   2   3   4   >