Re: Watermark alignment during unit tests

Hi Eugen, It is true that for ITcases this can be difficult and this should be improved in Flink’s testing infrastructure, but for this specific PR, what you need to check is if the allowedLateness parameter is propagated correctly throughout the translation process. The window operator with all

Re: Codespeed deployment for Flink

Thanks for this contribution Piotr and Nico. Tools like this are really useful for Flink’s success. Cheers, Kostas > On Sep 21, 2018, at 4:59 PM, Piotr Nowojski wrote: > > Hello community, > > For almost a year in data Artisans Nico and I were maintaining a setup > that continuously evaluate

Re: [DISCUSS] Dropping flink-storm?

+1 to drop it as nobody seems to be willing to maintain it and it also stands in the way for future developments in Flink. Cheers, Kostas > On Sep 29, 2018, at 8:19 AM, Tzu-Li Chen wrote: > > +1 to drop it. > > It seems few people use it. Commits history of an experimental > module sparse oft

Re: FLINK-9752 (s3 recoverable writer) not actually fixed in 1.6.2?

Hi Addison, This is definitely an error on my end, as the feature is going to be available from Flink 1.7 onwards. I forgot to correctly update the Flink version when closing the JIRA issue. I will update the release notes accordingly. Sorry for the miscommunication, Kostas > On Oct 31, 2018,

Re: [DISCUSS] FLIP-27: Refactor Source Interface

Hi all, >From the discussion, I understand that we are leaning towards a design where the user writes a single-threaded SplitReader, which Flink executes on another thread (not the main task thread). This way the task can have multiple readers running concurrently, each one reading a different spl

Re: [DISCUSS] FLIP-27: Refactor Source Interface

Hi Biao, Thanks for the answer! So given the multi-threaded readers, now we have as open questions: 1) How do we let the checkpoints pass through our multi-threaded reader operator? 2) Do we have separate reader and source operators or not? In the strategy that has a separate source, the source

Re: Looking for relevant sources related to connecting Apache Flink and Edgent.

Hi Felipe, This seems related to your previous question about a custom scheduler that knows which task to run on which machine. As Chesnay said, this is a rather involved and laborious task, if you want to do it as a general framework. But if you know what operation to push down, then why not dec

Re: Looking for relevant sources related to connecting Apache Flink and Edgent.

having to implement a custom source. Cheers, Kostas On Thu, Nov 29, 2018 at 11:08 AM Kostas Kloudas wrote: > Hi Felipe, > > This seems related to your previous question about a custom scheduler that > knows which task to run on which machine. > As Chesnay said, this is a rather in

Re: support/docs for compression in StreamingFileSink

Hi Addison, Sorry for the late reply. I agree that the documentation can be significantly improved and that adding compression could be a nice thing to have. There is already a PR open for supporting writing SequenceFiles with the StreamingFileSink. When this gets merged, you will be able to use

Re: [DISCUSS] Dropping flink-storm?

+1 to drop as well. On Thu, Jan 10, 2019 at 10:15 AM Ufuk Celebi wrote: > +1 to drop. > > I totally agree with your reasoning. I like that we tried to keep it, > but I don't think the maintenance overhead would be justified. > > – Ufuk > > On Wed, Jan 9, 2019 at 4:09 PM Till Rohrmann wrote: > >

Re: [DISCUSS] Bot for stale PRs on GitHub

+1 to try the bot. It may, at first, seem less empathetic than a solution that involves a human monitoring the PRs, but, in essence, having a PR stale for months (or even years) is at least as discouraging for a new contributor. Labels could further reduce the problem of noise, but I think that t

[DISCUSS] FLIP-33: Terminate/Suspend Job with Savepoint

confluence/pages/viewpage.action?pageId=103090212 The implementation sketch is far from complete, but it is worth having a discussion on the semantics as soon as possible. The implementation section is going to be updated soon. Looking forward to the discussion, Kostas -- Kostas Kloudas

Re: StreamingFileSink causing AmazonS3Exception

ae6 >> >> This whole thing is documented here: >> https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/best-practices.html >> >> However, I found that just using the documented property didn't appear to >> work and I had to wrap the InputStream in the B

Re: [DISCUSS] A more restrictive JIRA workflow

Really nice idea Timo, Thanks for taking the initiative to open this discussion. Although a side-effect, I consider it a big argument about my +1 the fact that now we create backpressure whenever needed at the JIRA level, rather than at the open PR level. The reason is that not accepting a PR af

Re: [DISCUSS] FLIP-33: Terminate/Suspend Job with Savepoint

d > > ensure that exactly-once sinks complete their two-phase commits > > successfully. If a savepoint was requested, one should be created. > > > > So in my mind there are two commands, cancel and stop, with appropriate > > semantics. Emitting MAX_WATERMARK before t

Re: [DISCUSS] FLIP-33: Terminate/Suspend Job with Savepoint

t; > > Aljoscha > > > On 7. Mar 2019, at 11:08, Kostas Kloudas wrote: > > > > Hi, > > > > Thanks for the comments. > > I agree with the Ufuk's and Elias' proposal. > > > > - "cancel" remains the good old "cancel"

Re: [Discuss] Semantics of event time for state TTL

Hi all, For GDPR: I am not sure about the regulatory requirements of GDPR but I would assume that the time for deletion starts counting from the time an organisation received the data (i.e. the wall-clock ingestion time of the data), and not the "event time" of the data. In other case, an organisa

Re: Rolling policy when using StreamingFileSink for bulk-encoded output

Hi Ying, Thanks for using the StreamingFileSink. The reason why the StreamingFileSink only supports OnCheckpointRollingPolicy with bulk formats has to do with the fact that currently Flink relies on the Hadoop writer for Parquet. Bulk formats keep important details about how they write the actua

Re: Rolling policy when using StreamingFileSink for bulk-encoded output

ilar to that inside prepareBucketForCheckpointing() > < > https://github.com/apache/flink/blob/3702029f45b7034b767e2b7eb01601c7f76ab35e/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/Bucket.java#L275 > >. > Not sure if I miss anything

Re: Rolling policy when using StreamingFileSink for bulk-encoded output

to address this > issue. > > Thanks again. > > - > Bests > Ying > > > On Tue, Jun 25, 2019 at 2:24 AM Kostas Kloudas wrote: > > > Hi Ying, > > > > You are right! If it is either on checkpoint or on size, then this is > > doable even wi

Re: [ANNOUNCE] Flink 0.10.2 Released

Yes thanks a lot Ufuk! > On Feb 12, 2016, at 3:09 PM, Till Rohrmann wrote: > > Thanks for being our release manager Ufuk :-) Great work! > > On Fri, Feb 12, 2016 at 2:15 PM, Robert Metzger wrote: > >> Thank you for doing a release Ufuk! >> >> I just tweeted about it: >> https://twitter.com/A

Re: [DISCUSS] Release Flink 1.0.3

I am working on: https://issues.apache.org/jira/browse/FLINK-2314?filter=-1 which I believe will also affect: https://issues.apache.org/jira/browse/FLINK-3796 Essentially the FileSourceFunction will become obsolete. > On Apr 22, 2016

Re: [jira] [Updated] (FLINK-2155) Add an additional checkstyle validation for illegal imports

--- >> >>Key: FLINK-2155 >>URL: https://issues.apache.org/jira/browse/FLINK-2155 >>Project: Flink >> Issue Type: Improvement >> Components: Build System >> Affects Versions: 1.1.0 >>

Re: [PROPOSAL] Structure the Flink Open Source Development

Hello, You can also add me to the DataStream API. Kostas > On Jun 16, 2016, at 7:02 PM, Robert Metzger wrote: > > Cool, thank you. > > So now we have at least one shepherd for each component. > Since there were no other comments / complaints about this proposal, I > assume its "active" now.

Re: [DISCUSS] Allowed Lateness in Flink

Hi, In the effort to move the discussion to the mailing list, rather than the doc, there was a comment in the doc: “It seems this proposal marries the allowed lateness of events and the discarding of window state. In most use cases this should be sufficient, but there are instances where havin

Re: sampling function

Hi Do, In DataStream you can always implement your own sampling function, hopefully without too much effort. Adding such functionality it to the API could be a good idea. But given that in sampling there is no “one-size-fits-all” solution (as not every use case needs random sampling and not al

Re: Flink 1.1.0 Preview RC0

There is an open PR now for https://issues.apache.org/jira/browse/FLINK-4207 Kostas > On Jul 21, 2016, at 3:38 PM, Stephan Ewen wrote: > > Looking into - https://github.com/apache/flink/pull/2226: Move Metrics API > to separate module > > On

Re: Flink 1.1.0 Preview RC0

I think Aljoscha is having a look. > On Jul 21, 2016, at 3:39 PM, Kostas Kloudas > wrote: > > There is an open PR now for https://issues.apache.org/jira/browse/FLINK-4207 > <https://issues.apache.org/jira/browse/FLINK-4207> > > Kostas > >> On Ju

Re: [DISCUSS] Allowed Lateness in Flink

Chen Qin wrote: >>>>>> >>>>>>> +1 for allowedLateness scenario. >>>>>>> >>>>>>> The rationale behind is there are backfills or data issues hold >>>>> in-window >>>>>>> data ti

Re: [DISCUSS] Allowed Lateness in Flink

Forgot to say that the signature for the onFire() that I think fits should be: void onFire(Window window, TriggerContext ctx) throws Exception; > On Jul 22, 2016, at 12:47 PM, Kostas Kloudas > wrote: > > Hi, > > I started working on the new triggers proposed here and so fa

Re: [DISCUSS] Allowed Lateness in Flink

heers, > Aljoscha > > On Fri, 22 Jul 2016 at 13:10 Kostas Kloudas > wrote: > >> Forgot to say that the signature for the onFire() that I think fits should >> be: >> >> void onFire(Window window, TriggerContext ctx) throws Exception; >> >>

Re: [DISCUSS] Allowed Lateness in Flink

h all the customization options and have one > SimpleTriggerTrigger that can execute a tree of SimpleTriggers. A rough > sketch of this would be this: > https://gist.github.com/aljoscha/66b0fcab89cd2b6190a63899f461067f > > Cheers, > Aljoscha > > > > On Mon, 25 Ju

Re: Question about Apache Flink Use Case

Hi Suma Cherukuri, From what I understand you have many small files and you want to aggregate them into bigger ones containing the logs of the last 24h. As Max said RollingSinks will allow you to have exactly-once semantics when writing your aggregated results to your FS. As far as reading your

Re: Question about Apache Flink Use Case

Hi Suma Cherukuri, I also replied to your question in the dev list, but I repeat the answer here just in case you missed in. From what I understand you have many small files and you want to aggregate them into bigger ones containing the logs of the last 24h. As Max said RollingSinks will allow

Re: [DISCUSS] Allowed Lateness in Flink

And also I think that the shouldFire has to take as an additional argument the time. This will help differentiate between ON_TIME and EARLY, LATE firings. > On Jul 26, 2016, at 11:02 AM, Kostas Kloudas > wrote: > > Hello, > > This is a nice proposal that I think covers

Re: [DISCUSS] Allowed Lateness in Flink

dealing with user specified triggers since our DSL is strictly >> specified by us >> >> The disadvantages are: >> - No user specified triggers inside the DSL >> >> The last part would be mitigated by still allowing users to write triggers >> for the cu

[DISCUSS] FLIP-9: Trigger DSL

Hi all! I've created a FLIP for the trigger DSL. This is the triggers that we want Apache Flink to support out-of-the-box. This proposal builds on various discussions on the mailing list and aims at serving as a base for further ones. https://cwiki.apache.org/confluence/display/FLINK/FLIP-9%3A+T

Re: [DISCUSS] FLIP-9: Trigger DSL

> On Tue, Aug 16, 2016 at 12:34 PM, Kostas Kloudas > wrote: >> Hi all! >> >> I've created a FLIP for the trigger DSL. This is the triggers >> that we want Apache Flink to support out-of-the-box. This proposal >> builds on various discussions on the mai

Re: [DISCUSS] FLIP-9: Trigger DSL

Thanks Till! Kostas > On Aug 16, 2016, at 3:30 PM, Till Rohrmann wrote: > > Cool design doc Klou. It's well described with a lot of details. I like it > a lot :-) +1 for implementing the trigger DSL. > > Cheers, > Till > > On Tue, Aug 16, 2016 at 3:18 PM, Kost

Re: [DISCUSS] FLIP-9: Trigger DSL

)) > > this gives me early firings whenever I see 5 new elements plus the > watermark firing. > > What do you think? > > On Tue, 16 Aug 2016 at 15:31 Kostas Kloudas > wrote: > >> Thanks Till! >> >> Kostas >> >>> On Aug 16, 2016, at 3:30 PM

Re: [DISCUSS] FLIP-9: Trigger DSL

er(earlyFiringTrigger), > EventTimeTrigger.afterEndOfWindow() > .withLateTrigger(lateFiringTrigger)) > > Is it legal? Will the earlyFiringTrigger and lateFiringTrigger both work ? > > > - Jark Wu > >> 在 2016年8月17日，上午12:24，Kostas Kloudas 写道： >> >> Hi Aljoscha, &

Re: [DISCUSS] FLIP-9: Trigger DSL

r WT) and ((!E and LT) or WT) >> >> now we can eliminate the two parts with E and !E because they can never be >> true and are in an "or": >> >> WT and WT >> >> which yield just "WT". >> >> Hope that makes sense to you. >

Re: [DISCUSS] FLIP-9: Trigger DSL

gt; example, if you have Repeat.forever(Some.trigger()) you know that the >>> finished bit will always be false and so you don't keep any state in the >>> TriggerRunner. If every trigger manually does that bookkeeping you remove >>> that possibility while increasing complexity in e

[DISCUSS] Removing delete*Timer from the WindowOperator.Context

Hi all, As the title of this email suggests, I am proposing to remove the methods deleteProcessingTimeTimer(long time) and deleteEventTimeTimer(long time) from the WindowOperator.Context. With this change, registered timers that have nothing to do (e.g. because their state has already been clea

Re: Releasing Flink 1.1.3

t;>>> start testing. >>>> >>>> The current state of affairs is as follows: >>>> >>>> - FLINK-4732 (Maven junction plugin security issue) has a fix in >>>> https://github.com/apache/flink/pull/2586. @Max: Can you merge this >&g

Re: Releasing Flink 1.1.3

e release. >>> It's cosmetic but it avoids a NPE in case the user jar doesn't contain >>> Flink jobs. >>> >>> On Wed, Oct 5, 2016 at 12:59 PM, Kostas Kloudas >>> wrote: >>>> Hi Ufuk, >>>> >>>> Thanks for bei

Re: [VOTE] Release Apache Flink 1.1.3 (RC1)

Hi all, I tested the Kafka source and continuous file sources and everything seems to be working fine. Kostas > On Oct 6, 2016, at 3:37 PM, Fabian Hueske wrote: > > +1 to release (binding) > > - checked hashes and signatures > - checked diffs against 1.1.2: no dependencies added or modified

Re: [DISCUSS] Removing delete*Timer from the WindowOperator.Context

; > I guessing most of these use cases will be obsolete with the new > Trigger DSL because the trigger logic can be expressed more easily. So > +1 for removing the delete methods from the context. > > On Tue, Sep 27, 2016 at 3:43 PM, Kostas Kloudas > wrote: >> Hi all, >

Re: [DISCUSS] Removing delete*Timer from the WindowOperator.Context

I am not familiar with the new > Trigger DSL though. > > Cheers, > > Konstantin > > On 12.10.2016 15:38, Kostas Kloudas wrote: >> Hi all, >> >> This thread has been dormant for some time now. >> >> Given that this change may affect user code, I am

Re: [DISCUSS] Schedule and Scope for Flink 1.2

Hello, As Aljoshca said, the TriggerDSL depends on another PR, before it can be merged. In addition, this will change also the whole structure of how we test window/trigger related functionality, so some refactoring will be required. Kostas > On Oct 28, 2016, at 3:06 PM, Aljoscha Krettek wrote

[DISCUSS] Adding a dispose() method in the RichFunction.

Hello, I would like to propose the addition of a dispose() method, in addition to the already existing close(), in the RichFunction interface. This will align the lifecycle of a RichFunction, with that of an Operator. After this, the code paths followed when finishing successfully and when ca

Re: [DISCUSS] Adding a dispose() method in the RichFunction.

e used in the DataStream and DataSet APIs. > How would that change affect the DataSet API? > > Best, Fabian > > > 2016-11-10 11:37 GMT+01:00 Kostas Kloudas : > >> Hello, >> >> I would like to propose the addition of a dispose() method, in addition to >>

Re: [DISCUSS] deprecated function need more detail

+1 and we should apply the same to all deprecated interfaces/abstract classes. > On Nov 23, 2016, at 11:13 AM, Aljoscha Krettek wrote: > > +1 That sounds excellent. > > On Wed, 23 Nov 2016 at 11:04 Till Rohrmann wrote: > >> +1 for your proposal. >> >> Cheers, >> Till >> >> On Wed, Nov 23, 2

[DISCUSS] Handling event-time in continuous file processing.

Hi all, This is to open a discussion on how to better handle event-time in continuous file processing. For the sake of illustration of the problem we will use the example of processing hourly server logs. In this case, each server writes its logs in hourly files, with names:

Re: [DISCUSS] Schedule and Scope for Flink 1.2

Hi Robert, I agree with Stefan on the estimation about 4. As for the testing, this can already start, as some basic operators have already been ported. Kostas > On Dec 6, 2016, at 5:50 PM, Stefan Richter > wrote: > > e already restored a job with general window operator and kafka source in a

Re: Taking time off

Hi Max! I may be repeating the previous writers but Thanks a lot for the work and all the contributions to the projects both at the technical and at the community level! Cheers, Kostas > On Jan 23, 2017, at 9:28 AM, Till Rohrmann wrote: > > I totally agree with the previous writers. Thanks

Re: [jira] [Commented] (FLINK-5316) Make the GenericWriteAheadSink backwards compatible.

ake the GenericWriteAheadSink backwards compatible. > > > > > > Key: FLINK-5316 > > URL: https://issues.apache.org/jira/browse/FLINK-5316 > > Project: Flink > > Issue Type: Improvement > > Components: Cassandra Connector &g

Re: [ANNOUNCE] Welcome Stefan Richter as a new committer

Congratulations Stefan! Kostas > On Feb 12, 2017, at 7:17 PM, Chen Qin wrote: > > Congratulations! > > Chen > >> On Feb 10, 2017, at 02:00, Ufuk Celebi wrote: >> >> Hey everyone, >> >> I'm very happy to announce that the Flink PMC has accepted Stefan >> Richter to become a committer of the

Re: [ANNOUNCE] Welcome Jark Wu and Kostas Kloudas as committers

ue, Feb 7, 2017 at 12:16 PM, Fabian Hueske wrote: > >> Hi everybody, >> >> I'm very happy to announce that Jark Wu and Kostas Kloudas accepted the >> invitation of the Flink PMC to become committers of the Apache Flink >> project. >> >> Jark and K

Re: [DISCUSS] Side Outputs and Split/Select

+1 > On Mar 2, 2017, at 1:08 PM, Fabian Hueske wrote: > > +1 > > 2017-03-02 12:11 GMT+01:00 Aljoscha Krettek : > >> Ok, so it seems we have to go with the OutputTag variant for windows as >> well, for now. >> >> For Flink 2.0 we can change that. Would everyone be OK with that? >> >> On Thu,

Re: [VOTE] Release Apache Flink 1.2.1 (RC1)

+1 for 1 > On Apr 3, 2017, at 5:52 PM, Till Rohrmann wrote: > > +1 for option 1) > > On Mon, Apr 3, 2017 at 5:48 PM, Fabian Hueske wrote: > >> +1 to option 1) >> >> 2017-04-03 16:57 GMT+02:00 Ted Yu : >> >>> Looks like #1 is better - 1.2.1 would be at least as stable as 1.2.0 >>> >>> Cheer

Re: Re: [ANNOUNCE] New Apache Flink Committer - Igal Shilman

Congratulations Igal and welcome! Kostas On Wed, Sep 16, 2020 at 6:37 AM Guowei Ma wrote: > > Congratulations :) > Best, > Guowei > > > On Wed, Sep 16, 2020 at 11:54 AM Zhijiang > wrote: > > > Congratulations and welcome, Igal! > > > > > > ---

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

Hi all, Thanks for keeping the discussion running while I was on holidays! I am catching up currently and I will post in the voting thread if I have any comments :) Cheers, Kostas On Wed, Sep 16, 2020 at 11:25 AM David Anderson wrote: > > Aljoscha, > > Thanks for the thorough response. I'm stil

Re: Re: [VOTE] FLIP-134: Batch execution for the DataStream API

+1 My only suggestion (although by no means a blocker) would be to remove from the FLIP the `env.setRuntimeMode()` method. I say that because this is syntactic sugar over the `env.configure()` with the `execution.runtime-mode` option set to BATCH or STREAMING. These methods can be nice but they se

Re: Kafka connection issues

Hi Ramya, Unfortunately your images are blocked. Could you upload them somewhere and post the links here? Also I think that the TaskManager logs may be able to help a bit more. Could you please provide them here? Cheers, Kostas On Tue, Sep 22, 2020 at 8:58 AM Ramya Ramamurthy wrote: > Hi, > >

Re: Kafka connection issues

pods. > > Thanks for your help. > > ~Ramya. > > On Tue, Sep 22, 2020 at 9:18 PM Kostas Kloudas wrote: >> >> Hi Ramya, >> >> Unfortunately your images are blocked. Could you upload them somewhere and >> post the links here? >> Also I think that the Ta

Re: [VOTE] FLIP-143: Unified Sink API

+1 (binding) @Steven Wu I think there will be opportunities to fine tune the API during the implementation. Cheers, Kostas On Sun, Sep 27, 2020 at 7:56 PM Steven Wu wrote: > > +1 (non-binding) > > Although I would love to continue the discussion for tweaking the > CommitResult/GlobaCommitter in

Re: [VOTE] FLIP-142: Disentangle StateBackends from Checkpointing

+1 (binding) Kostas On Tue, Sep 29, 2020 at 4:57 PM Yu Li wrote: > > +1 (binding) > > Thanks all for the patience of answering / addressing my questions in the > discussion thread. > > Best Regards, > Yu > > > On Thu, 17 Sep 2020 at 14:39, Dawid Wysakowicz > wrote: > > > +1 (binding) > > > > On

Re: [DISCUSS] Enforce common opinionated coding style using Spotless

Hi all, +1 for enforcing "a" codestyle using "a" tool. As the project grows both in terms of LOCs and contributors, this becomes more and more important as it eliminates some potential points of friction without any additional effort. >From the discussion, I am leaning towards having a single co

[DISCUSS] Remove flink-connector-filesystem module.

Hi all, As the title suggests, this thread is to discuss the removal of the flink-connector-filesystem module which contains (only) the deprecated BucketingSink. The BucketingSin is deprecated since FLINK 1.9 [1] in favor of the relatively recently introduced StreamingFileSink. For the sake of a

Re: [DISCUSS] Remove flink-connector-filesystem module.

rsions of the module compatible with 1.12+? > > On 10/12/2020 4:30 PM, Kostas Kloudas wrote: > > Hi all, > > > > As the title suggests, this thread is to discuss the removal of the > > flink-connector-filesystem module which contains (only) the deprecated > > Buc

Re: [DISCUSS] Remove flink-connector-filesystem module.

@Chesnay Schepler Off the top of my head, I cannot find an easy way to migrate from the BucketingSink to the StreamingFileSink. It may be possible but it will require some effort because the logic would be "read the old state, commit it, and start fresh with the StreamingFileSink." On Tue, Oct 13

Re: [DISCUSS] Remove flink-connector-filesystem module.

els rushed to remove it at this point. > > On Tue, Oct 13, 2020 at 2:33 PM Kostas Kloudas wrote: > > > @Chesnay Schepler Off the top of my head, I cannot find an easy way > > to migrate from the BucketingSink to the StreamingFileSink. It may be > > possible but it will

Re: [SURVEY] Remove Mesos support

+1 for adding a warning about the removal of Mesos support and I would also propose to state explicitly in the warning the version that we are planning to actually remove it (e.g. 1.13 or even 1.14 if we feel it is too aggressive). This will help as a reminder to users and devs about the upcoming

Re: [SURVEY] Remove Mesos support

Thanks Piyush for the message. After this, I revoke my +1. I agree with the previous opinions that we cannot drop code that is actively used by users, especially if it something that deep in the stack as support for cluster management framework. Cheers, Kostas On Fri, Oct 23, 2020 at 4:15 PM Piyu

Re: [DISCUSS] Releasing Apache Flink 1.11.3

+1 for releasing Flink 1.11.3 as it contains a number of important fixes and thanks Gordon and Xintong for volunteering. Cheers, Kostas On Mon, Oct 26, 2020 at 4:37 AM Yu Li wrote: > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for > volunteering as our release managers. > > Be

Re: [DISCUSS] Remove flink-connector-filesystem module.

gt; > > > Seth > > > > https://github.com/apache/flink/blob/2ff3b771cbb091e1f43686dd8e176cea6d435501/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L170-L172 > > > > On Thu, Oct 15, 2020 at 2:57 PM K

Re: [DISCUSS] Remove flink-connector-filesystem module.

strict. On Wed, Oct 28, 2020 at 10:04 AM Chesnay Schepler wrote: > > If the conclusion is that we shouldn't remove it if _anyone_ is using > it, then we cannot remove it because the user ML obviously does not > reach all users. > > On 10/28/2020 9:28 AM, Kostas Kloudas wrote: &

Re: [DISCUSS] Remove flink-connector-filesystem module.

e users because there are better alternatives. > > On 10/28/2020 10:46 AM, Kostas Kloudas wrote: > > I think that the mailing lists is the best we can do and I would say > > that they seem to be working pretty well (e.g. the recent Mesos > > discussion). > > Of course they ar

Re: [DISCUSS] Remove flink-connector-filesystem module.

st of friction for some users. >> >> To be clear, I, personally, don't have a problem with removing it (we >> have removed other connectors in the past that did not have a migration >> plan), I just reject he argumentation. >> >> On 10/28/2020 12:21 PM, Kostas Kl

[VOTE] Remove flink-connector-filesystem module.

Hi all, Following the discussion in [1], I would like to start a vote on removing the flink-connector-filesystem module which includes the BucketingSink. The vote will be open till November 3rd (72h, excluding the weekend) unless there is an objection or not enough votes. Cheers, Kostas [1] ht

Re: [VOTE] Remove flink-connector-filesystem module.

at 5:05 PM Till Rohrmann wrote: > > +1 > > Cheers, > Till > > On Fri, Oct 30, 2020 at 11:54 AM Guowei Ma wrote: > > > +1 > > Best, > > Guowei > > > > > > On Fri, Oct 30, 2020 at 6:02 PM Aljoscha Krettek > > wrote: > >

Re: [DISCUSS] Enforce common opinionated coding style using Spotless

nk that is what current > > > >>> checkstyle > > > >>> rules serving. > > > >>> > > > >>> For automatic applying part, we suggest developing by IDEA and with > > > >>> Checkstyle Plugin on IDEA applying check

Re: [DISCUSS] Backport broadcast operations in BATCH mode to Flink

+1 on my side as it does not break anything and it can act as motivation for some people to upgrade. Cheers, Kostas On Thu, 7 Jan 2021, 12:39 Aljoscha Krettek, wrote: > 1.12.x > Reply-To: > > Hi, > > what do you think about backporting FLINK-20491 [1] to Flink 1.12.x? > > I (we, including Dawid

Re: [DISCUSS] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

+1 for moving to Hugo. Cheers, Kostas On Thu, Jan 14, 2021 at 1:27 PM Wei Zhong wrote: > > +1 for migrating to Hugo. > > Currently we have developed many plugins based on Jekyll because the native > features of Jekyll cannot meet our needs. It seems all of them can be > supported via Hugo shor

Re: Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

Congrats Roman! Kostas On Wed, Feb 10, 2021 at 2:08 PM Arvid Heise wrote: > > Congrats! Well deserved. > > On Wed, Feb 10, 2021 at 1:54 PM Yun Gao > wrote: > > > Congratulations Roman! > > > > Best, > > Yun > > > > > > --Original Mail -- > > Sender:Till Rohrmann

[DISCUSS] Remove dependency shipping through nested jars during job submission.

Hi all, I would like to bring the discussion in https://issues.apache.org/jira/browse/FLINK-17745 to the dev mailing list, just to hear the opinions of the community. In a nutshell, in the early days of Flink, users could submit their jobs as fat-jars that had a specific structure. More concretel

Re: [Reminder] Prefer {% link %} tag in documentation

Thanks for the heads up Seth! Kostas On Tue, Jun 9, 2020 at 7:27 PM Seth Wiesman wrote: > > The tag is new to Jekyll 4.0 which we only recently updated to. > > There are a lot of existing tags that would need to be updated first :) > I opened a ticket to track that work and then yes that would m

Re: [VOTE] Release 1.11.0, release candidate #4

Hi all, As far as the issue that Chesnay mentioned that leads to a "Caused by: org.apache.flink.api.common.InvalidProgramException:" for DataSet examples with print() collect() or count() as sink, this was a semi-intensional side-effect of the application mode. Before, in these cases, the output

Re: NullPointer Exception while trying to access or read ReadOnly ctx in processElement method in KeyedBroadCastProcessFunction in Apache Flink

Hi all, Just as an addition to what Dawid asked, I would also like to ask: 1) which Flink version are you using? because the stack trace line numbers do not match the current master. 2) as a clarification (although maybe not relevant here), there is no guarantee on the order of the elements, so th

Re: [DISCUSS] Planning Flink 1.12

+1 for end of October from me as well. Cheers, Kostas On Wed, Aug 5, 2020 at 9:59 AM Till Rohrmann wrote: > +1 for end of October from my side as well. > > Cheers, > Till > > On Tue, Aug 4, 2020 at 9:46 PM Stephan Ewen wrote: > > > The end of October sounds good from my side, unless it collide

[DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

Hi all, As described in FLIP-131 [1], we are aiming at deprecating the DataSet API in favour of the DataStream API and the Table API. After this work is done, the user will be able to write a program using the DataStream API and this will execute efficiently on both bounded and unbounded data. But

Re: [DISCUSS] Removing deprecated methods from DataStream API

Thanks a lot for starting this Dawid, Big +1 for the proposed clean-up, and I would also add the deprecated methods of the StreamExecutionEnvironment like: enableCheckpointing(long interval, CheckpointingMode mode, boolean force) enableCheckpointing() isForceCheckpointing() readFile(FileInputFor

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

t; run exactly the same job as in production, except with different sources and > sinks. While it might be a reasonable default, I'm not convinced that > switching a processing time streaming job to read from a bounded source > should always cause it to fail. > > David > &g

Re: [DISCUSS] Removing deprecated methods from DataStream API

nt#readFile,readFileStream(...),socketTextStream(...),socketTextStream(...) > (deprecated in 1.2) > > Looking forward to more opinions on the issue. > > Best, > > Dawid > > > On 17/08/2020 12:49, Kostas Kloudas wrote: > > Thanks a lot for starting this

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

h" and "bounded >> > streaming" to be treated differently. If I've understood it correctly, the >> > section on scheduling allows me to choose STREAMING scheduling even if I >> > have bounded sources. I like that approach, because it recognizes that

Re: [ANNOUNCE] New Flink Committer: David Anderson

Congratulations David! Kostas On Wed, Aug 19, 2020 at 2:33 PM Arvid Heise wrote: > > Congrats David! > > On Wed, Aug 19, 2020 at 11:17 AM Fabian Hueske wrote: > > > Congrats David, well deserved! > > > > Cheers, > > Fabian > > > > Am Mi., 19. Aug. 2020 um 11:05 Uhr schrieb Marta Paes Moreira <

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

mers at the end of > a job would be interesting, and would help in (at least some of) the cases I > have in mind. I don't have a better idea. > > David > > On Mon, Aug 17, 2020 at 8:24 PM Kostas Kloudas wrote: >> >> Hi Kurt and David, >> >> Thanks a lot

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

available in the BATCH mode in current > implementation. > So maybe we need more checks in the AUTOMATIC execution mode. > > Best, > Guowei > > > On Thu, Aug 20, 2020 at 10:27 PM Kostas Kloudas wrote: >> >> Hi all, >> >> Thanks for the comments! >>

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

Thanks a lot for the discussion! I will open a voting thread shortly! Kostas On Mon, Aug 24, 2020 at 9:46 AM Kostas Kloudas wrote: > > Hi Guowei, > > Thanks for the insightful comment! > > I agree that this can be a limitation of the current runtime, but I > think that t

<    1   2   3   4   5   >