Re: AKA and quarantine

Hi Vishal, sorry for the late response. Till (in CC) might be able to answer your Akka / coordination related questions. Best, Fabian 2018-01-24 1:22 GMT+01:00 Vishal Santoshi : > Any suggestions ? I know these are very general issue but these are edge > conditions that we want the community t

Re: ElasticsearchSink in Flink 1.4.0 with Elasticsearch 5.2+

Hi Christophe, great! Thanks for your contribution. I'm quite busy right now, but I agree that we should have support for ES 5.3 and Es 6.x for the next minor release 1.5. Best, Fabian 2018-01-26 23:09 GMT+01:00 Christophe Jolif : > Ok, I got it "done". I have a PR for ES5.3 (FLINK-7386) just

Re: Flink CEP exception during RocksDB update

Hi again Varun, I am investigating the problem you mentioned and I found a bug in the SharedBuffer, but I am not sure if it is the only bug that affects you. Could you please try this branch https://github.com/kl0u/flink/tree/cep-inv and let me know

Advice or best practices on adding metadata to stream events

Hi everyone, I am fairly new to the world of stream processing and I was wondering about best practices when needing to add metadata to a stream in Flink (or stream processing in general). Searching for examples/discussions of this topic did not yield the results I was hoping for, so I figured

Re: ElasticsearchSink in Flink 1.4.0 with Elasticsearch 5.2+

Hi Christophe, Thanks a lot for the contribution! I’ll add reviewing the PR to my backlog. I would like / will try to take a look at the PR by the end of this week, after some 1.4.1 blockers which I’m still busy with. Cheers, Gordon On 29 January 2018 at 9:25:27 AM, Fabian Hueske (fhue...@gmail

Re: How to make savepoints more robust in the face of refactorings ?

Hi, In the Scala API, type serializers may be anonymous classes generated by Scala macros, and would therefore contain a reference to the wrapping class (i.e., your `Operators` class). Since Flink currently serializes serializers into the savepoint to be used for deserialization on restore, and

Re: How to make savepoints more robust in the face of refactorings ?

>One thing that I’m not completely certain with yet, is where in your demonstrated code a anonymous-classed serializer is generated for some type. >From what I see, there shouldn’t be any anonymous-class serializers for the code. Is the code you provided a “simplified” version of the actual code in

Re: ElasticsearchSink in Flink 1.4.0 with Elasticsearch 5.2+

Thanks a lot. Is there any timeline for 1.5 by the way? -- Christophe On Mon, Jan 29, 2018 at 11:36 AM, Tzu-Li (Gordon) Tai wrote: > Hi Christophe, > > Thanks a lot for the contribution! I’ll add reviewing the PR to my backlog. > I would like / will try to take a look at the PR by the end of th

Scheduled/timed source/sink

What is the best way to add timer to a source/sink? (Apologies if I’m asking the question that’s already being answered elsewhere), also is it possible to use process function as source or sink? Thanks Ishwara

Trigger Time vs. Latest Acknowledgement

I'm triggering nightly savepoints at 23:59:00 with crontab on the flink cluster. For example last night's savepoint has this information: Trigger Time: 23:59:14 Latest Acknowledgement: 00:00:59 What are the min/max boundaries for the data contained by the savepoint? Can I deduce from this either

Re: GetExecutionPlan fails with IllegalArgumentException in Comparator

@Julian Could you provide an example job to reproduce this? (Can also be shared privately) On 23.01.2018 18:05, Fabian Hueske wrote: Great! Thank you :-) 2018-01-23 17:58 GMT+01:00 Bauss, Julian >: Hi Fabian, sure! See: https://issues.apache.org

Re: AKA and quarantine

Thank you. On Mon, Jan 29, 2018 at 3:17 AM, Fabian Hueske wrote: > Hi Vishal, > > sorry for the late response. > Till (in CC) might be able to answer your Akka / coordination related > questions. > > Best, Fabian > > 2018-01-24 1:22 GMT+01:00 Vishal Santoshi : > >> Any suggestions ? I know thes

Re: ElasticsearchSink in Flink 1.4.0 with Elasticsearch 5.2+

As of right now there is no specific date, see also https://flink.apache.org/news/2017/11/22/release-1.4-and-1.5-timeline.html. On 29.01.2018 13:41, Christophe Jolif wrote: Thanks a lot. Is there any timeline for 1.5 by the way? -- Christophe On Mon, Jan 29, 2018 at 11:36 AM, Tzu-Li (Gordon)

Re: Scheduled/timed source/sink

AFAIK Flink's source/sink interface do not expose something akin to the ProcessFunction's TimerService, so I would suggest to use an ScheduledExecutorService or similar. Technically you can use process functions as sources, the easiest way being to create a dummy source which broadcasts a singl

Re: AKA and quarantine

Hi Vishal, Akka usually quarantines remote ActorSystems in case of a system message delivery failure or if the death watch was triggered. This can, for example, happen if your machine is under heavy load or has a high GC pressure and does not find enough time to respond to the heartbeats. - If yo

Re: AKA and quarantine

>> If you enable taskmanager.exit-on-fatal-akka-error, then it will stop TMs which got quarantined. This will automatically restart TMs in case that you are running Flink on Yarn. Thus, I would recommend enabling this if possible We do not use yarn. This would end up restarting the jobs on the rem

[Window] Per key window

Hello, How should i create a tumbling window referring same key. each key have it own window instance. Session window referring key, but must have a gap. Regards, Shu li Zheng