Aw: Re: Combine two independant streams

Hi Fabian,   thanks for your very goog explanation. However, I don't exactly know how to increase the watermark by myself. Do you have an example for me? Do I have to override the getCurrentWatermark method?   Thanks, Markus   Gesendet: Dienstag, 28. Februar 2017 um 20:36 Uhr Von: "Fabian H

Re: Flink Application Log :Custom Log properties

Thanks Sunil, this is also a question I wanted to ask to the forum. How to separate application log (aggregated to a file ) just similar to yarn log aggregation but capture flink process related logs from all workers and get it in one central location. I would love to hear suggestion and stories

Re: unclear exception when writing to elasticsearch

Hi Martin, You can do that by adding a dependency to the Elasticsearch client of your desired version in your project. You can also check what Elasticsearch client version the project is using by checking `mvn dependency:tree` from the base directory of your project. Cheers, Gordon On March 1

Re: ElasticsearchSink Exception

Good to know it’s working! Thanks for the update :-) On March 1, 2017 at 6:03:44 AM, Govindarajan Srinivasaraghavan (govindragh...@gmail.com) wrote: Hi Gordon/Flavio, Found out the issue was because of elastic search version mismatch. Another person upgraded ES version to 5.x but I was using

Re: Flink requesting external web service with rate limited requests

Hey Fabian, One of my solutions implements the AsyncFunction but I'm still unable to savepoint because Flink reads the backed up records, thousands of historical records, right off the bat and when I issue a savepoint request it has to wait for all those records to be processed which takes a coupl

Event-time tumbling window doesn't fire- Flink 1.2.0, Kafka-0.8_2.10

Hi, I'm using *Flink* *1.2.0* to read from *Kafka*-0.8.1.1_2.10. I have written a *flink* streaming job that creates (event) time based window and then computes some stats. However, the window function is never called. I used the debug watermark code and noticed that no watermark is generated. If

Event-time tumbling window doesn't fire- Flink 1.2.0, Kafka-0.8_2.10

HiI'm using Flink 1.2.0 to read from Kafka-0.8.1.1_2.10I have written a flink streaming job that creates (event) time based window and then computes some stats. However, the window function is never called. I used the debug watermark code and noticed that no watermark is generated. If I read from f

Re: ElasticsearchSink Exception

Hi Gordon/Flavio, Found out the issue was because of elastic search version mismatch. Another person upgraded ES version to 5.x but I was using 2.x. After changing the version it worked. Thanks for the help. On Mon, Feb 27, 2017 at 6:12 AM, Tzu-Li (Gordon) Tai wrote: > Hi! > > Like wha Flavio s

Re: Flink Application Log :Custom Log properties

Hi Sunil, I have not much experience with logging on Yarn, but Robert (in CC) might be able to help you. Cheers, Fabian 2017-02-28 14:54 GMT+01:00 raikarsunil : > Hi , > > Is there any way we can configure application logging by passing the custom > log4j properties file in flink run command? T

Re: Combine two independant streams

In event-time mode, operators compute their internal time from watermarks. Depending on how watermarks are generated, their time only increases if records with later timestamps are processed. If no records arrive, no new watermarks are generated and the event-time does not increase. Since you want

Re: Combine two independant streams

Hi Fabian, yeah, that's basically it. The events window gets closed only when a newer event arrives (after 10 seconds window). Can I tell Flink to close the event window at timeWindow.getEnd() even if no newer event arrives? Thanks, Markus Am 28.02.17 um 17:19 schrieb Fabian Hueske: Hi Mark

Re: unclear exception when writing to elasticsearch

Hej, thanks for the fast reply. I'm currently running things from inside my IDE so it should not be a packaging problem. That said I added the plugin from the link provided but I'm not sure what elastic search library is needed. Where do I override the elastic search version? The only thing I'm

Re: Flink 1.2 Jobmanager OOME - CheckpointCoordinators

@Konstantion: Could you share a relevant part of the heap dump just to get a second look? The timer tasks are responsible to abort the checkpoint if a checkpoint timeout occurs. You can decrease the timeout via the CheckpointConfig (env.getCheckpointConfig().setCheckpointTimeout(long)), the curren

Re: unclear exception when writing to elasticsearch

Hi! This could be a Elasticsearch server / client version conflict, or that the uber jar of your code wasn’t built properly. For the first possible issue, we’re currently using Elasticsearch 2.3.5 to build the Flink Elasticsearch Connector. Could you try overriding this version to 2.4.1 when b

unclear exception when writing to elasticsearch

Hej, I'm trying to write to elastic search from a streaming application and I get a weird error message I that I can't decipher. Hopefully, someone here can help me. I'm trying to run the java example

Re: Combine two independant streams

Hi Markus, I'm not sure I understood the issue with the second approach. Is it that the stream of application events might be empty for some time such that its event time is not increasing? Best, Fabian 2017-02-28 17:02 GMT+01:00 Markus Klein : > Hello Flink Community, > > I have a question reg

Re: Flink requesting external web service with rate limited requests

A SourceFunction may only emit records when it holds the checkpointLock (just as `ContinuousFileMonitoringFunction` does). Flink only emits a checkpoint if it holds the lock. This ensures correct behavior. Best, Fabian 2017-02-28 10:58 GMT+01:00 Yassine MARZOUGUI : > Hi Fabian, > > I have a rel

Re: Sliding Windows Processing problem with Kafka Queue Event Time and BoundedOutOfOrdernessGenerator

Hi Sujit, as Nico said, somewhat hidden in the middle of his mail: "Since a sliding window is only closed and fired once the watermark is received, its first time firing is actually 5s behind as per your maxOutOfOrderness." In your setup, the watermark is always lagging 5 seconds behind the maximu

Combine two independant streams

Hello Flink Community,   I have a question regarding combining two independant streams.  The first stream is a stream of events with metrics information. It occurs every 10 seconds. What I want is to join a second stream with events from an application. The result should be an event with the metr

Flink 1.2 Jobmanager OOME - CheckpointCoordinators

Hi everyone, I am currently running a small Flink job locally, which checkpoints every 100ms. After a few minutes the JM crashes with an OOME. In the Headump I can see, that a TimerTask holds references to all completed CheckpointCoordinators. I assume this task is supposed to clean these checkpo

Flink Application Log :Custom Log properties

Hi , Is there any way we can configure application logging by passing the custom log4j properties file in flink run command? The aim is to write all the application logs in a specific file rather than the default yarn/flink log file. Currently I am using the below command to run my application :

Re: Set custom file as ClassPath in flink yarn command

Thanks lining jing -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Set-custom-file-as-ClassPath-in-flink-yarn-command-tp11929p11959.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Flink requesting external web service with rate limited requests

Hi Fabian, I have a related question regarding throttling at the source: If there is a sleep in the source as in ContinuousFileMonitoringFunction.java

Re: Evicting elements in EventTimeSessionWindow

Hi Fritz, this seems to be a bug caused by some leftover (overeager) check. The EvictingWindowOperator in fact has the code to deal with merging windows. I opened this issue to track the bug: https://issues.apache.org/jira/browse/FLINK-5933. As a workaround, you can use a purging Trigger. You can

Re: Flink requesting external web service with rate limited requests

Hi Giuliano, Flink 1.2 introduced the AsyncFunction which asynchronously sends requests to external systems (k-v-stores, web services, etc.). You can limit the number of concurrent requests, but AFAIK you cannot specify a limit of requests per minute. Maybe you can configure the function such that

Re: Sliding Windows Processing problem with Kafka Queue Event Time and BoundedOutOfOrdernessGenerator

Hi Nico, Thanks. Yes, we are using the window assigner and allowedLateness parameters as below .window(SlidingEventTimeWindows.of(Time.minutes(6), Time.minutes(2))) // 6 min window with 2 min sliding window .allowedLateness(Time.seconds(5)) We need to use the BoundedOutofOrdernessGenerator to

Re: Running streaming job on every node of cluster

Hi Evgeny, regarding slot allocation, at the moment, you really cannot influence which slots of which TMs are used by your program, e.g. whether two slots of a single TM are used or whether one slot of each of two TMs is used when your program is occupying two slots in total. Afaik there are no