Streaming Exception

Hi All, I see the below error after running my streaming job for a while and when the load increases. After a while the task manager becomes completely dead and the job keeps on restarting. Also when I checked if there is an back pressure in the UI, it kept on saying sampling in progress and no r

questions on custom state with flink window

Hi All I have few questions on understanding state with flink 1) Is it advisable to create custom state within a RichWindowFunction function. I am able to create it but 2) If I create state in the window function then does the state remain for ever as I want the state to stay for ever 3) Also on

Flink - Writing Test Case for the Datastream

Hi Team, I am trying to write test cases to check whether the job is getting executed as desired. I am using the Flink test util. I am trying to do a end to end testing where Flink reads from a Kafka Queue, does some processing and then writes the output to another topic of the Kafka Queue. My obj

Re: TTL for State Entries / FLINK-3089

Hey Aljoscha, thank you for your reply. The amount and quality of response on this list are really great to see and a good way to learn. I will try this and see how this works out. Cheers, Johannes On Thu, Mar 9, 2017 at 3:55 PM, Aljoscha Krettek wrote: > Hi Johannes, > I think what you can

Re: window function not working when control stream broadcast

Hi Aljoscha, Here's the code: private static class DataFilterFunImpl extends RichCoFlatMapFunction { private JSONParser parser; private Map> whiteListMap = new HashMap<>(); @Override // tuple5(domain, device_type, type, key, count_or_sum) public void flatM

Flink Standalone Service

Hi all, Is there a init.d or similar service script for Flink on Redhat (or Centos) 7? Mostly I'm just wanting to make sure when my server restarts, Flink starts up again, but being able to check the status would also be awesome. Cheers, Daniel

Re: Issues with Event Time and Kafka

Thanks for the suggestion, we can definitely try that out. My one concern there is that events technically can lag for days or even months in some cases, but we only care about including the events that lag for 30 minutes or so, and would like the further lagging events to be ignored - I just want

Re: Flink streaming - call external API "after" sink

Hi, this is the second time that something like this is being requested or proposed. This was the first time: [1]. +Seth, who might have an opinion on this. I'm starting to think that we might need to generalise this pattern. Right now, the SinkFunction interface is this: public interface Sin

Flink streaming - call external API "after" sink

Hi, I am using flink-1.2 streaming API to process clickstream and compute some results per cookie. The computed results are stored in Cassandra using flink-cassandra connector. After a result is stored in cassandra, I want to notify an external system (using their API or via Kafka) that result is

Re: Appropriate State to use to buffer events in ProcessFunction

Hi Timo, I thought about the ListState but quickly discarded It as it keeps the insersion order and not events order. After a second thought I think I will reconsider it since my events are occaionally out-of-order. Didn't know that Flink CEP operators 'next' and 'within', can handle event time, s

Re: TTL for State Entries / FLINK-3089

Hi Johannes, I think what you can do is not register a timer for every event but for every key, with a certain granularity. When that timer fires you check what you want to clean up for that key and maybe register another timer for the future. This way, the size of your timer state is bounded by yo

Re: window function not working when control stream broadcast

Hi Sam, could you please also send the code for the DataFilterFunImpl and your timestamps/watermark assigner. That could help in figuring out the problem. Best, Aljoscha On Wed, Mar 8, 2017, at 19:56, Sam Huang wrote: > Hi Timo, > > The window function sinks the data into InfluxDB, and

Re: Event-time tumbling window doesn't fire- Flink 1.2.0, Kafka-0.8_2.10

Great you could figure it out! And thanks for letting us know. On Wed, Mar 8, 2017, at 03:03, Sam Huang wrote: > So sorry I forgot to reply. I've solved the problem, turns out I didn't > input > data which generates a watermark greater than my first window end time, > so > no window was triggered.

Re: Remove Accumulators at runtime

I see, this is not possible with accumulators. You could wrap all counts in a single metric and update that one. Check out Flink's metrics: https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html On Wed, Mar 8, 2017 at 5:04 PM, PedroMrChaves wrote: > Hi, > > I'm buil

Re: ProcessFunction example

Hi Philippe, You are right! Thanks for reporting it! We will fix it asap. Kostas > On Mar 9, 2017, at 8:38 AM, Philippe Caparroy > wrote: > > I think there is an error in the code snippet describing the ProcessFunction > time out example : > https://ci.apache.org/projects/flink/flink-docs