Re: Getting key from keyed stream

:) -Jamie On Thu, Jan 12, 2017 at 12:53 PM, Paul Joireman mailto:paul.joire...@physiq.com>> wrote: Hi all, Is there a simple way to read the key from a KeyedStream. Very simply I'm trying to read a message from Kafka, separate the incoming messages by a field in the message an

Getting key from keyed stream

Hi all, Is there a simple way to read the key from a KeyedStream. Very simply I'm trying to read a message from Kafka, separate the incoming messages by a field in the message and write the original message back to Kafka using that field as a new topic. I chose to partition the incoming str

Event time, watermarks and windows

Hi all, The event timestamp and watermarks documentation (v. 1.1) https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/event_timestamps_watermarks.html states that The AssignerWithPeriodicWatermarks assigns timestamps and generates watermarks periodically (possibly

Watermarks and window firing

Hi all, The documentation for event timestamps and watermarks (https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/event_timestamps_watermarks.html) states that the The AssignerWithPeriodicWatermarks assigns timestamps and generates watermarks periodically (possibly depe

Re: Windows and Watermarks Clarification

me. With allowedLateness() the window evaluation can be deferred to allow late elements (elements whose timestamp is before the current watermark) to join the window before it is evaluated. Let me know if you have further questions, Fabian 2016-09-01 20:16 GMT+02:00 Paul Joireman mailto:pa

Windows and Watermarks Clarification

Hi all, Just a point of clarification on how watermarks are generated. I'd like to use a SlidingEventTime window of say 5 minutes with a 30 second slide. The incoming data stream has elements from which I can extract the timestamp but they may come out of order so I chose to implement the f

Re: Setting EventTime window width using stream data

works if all elements of a key that should fall into the same windows have the same size/slide property, otherwise they would spawn different windows. Cheers, Aljoscha On Tue, 30 Aug 2016 at 21:28 Paul Joireman mailto:paul.joire...@physiq.com>> wrote: Hi all, Is it possible to dynamic

Setting EventTime window width using stream data

Hi all, Is it possible to dynamically set the size/width of a SlidingEventTimeWindow based on a data coming from the stream? Our use case is as follows. We create a stream sourced from external system and coming in as a JSON string which is deserialized to a stream of POJO.The deseria

Re: CountTrigger FIRE or FIRE_AND_PURGE

w.mapr.com/blog/essential-guide-streaming-first-processing-apache-flink [2] http://flink.apache.org/news/2015/12/04/Introducing-windows.html 2016-08-30 0:25 GMT+02:00 Paul Joireman mailto:paul.joire...@physiq.com>>: Hi all, I'm attempting to use long SlidingEventTime window (durat

CountTrigger FIRE or FIRE_AND_PURGE

Hi all, I'm attempting to use long SlidingEventTime window (duration 24 hours) but I would like updates more frequently than the 24 hour length. I naeively attempted to use a simple CountTrigger(10) to give me the window every time 10 samples are collected, however, the window processing func

Updating stored window data

Hello, I'd like to specify a long time duration window (say 1 day) but write a custom trigger to force processing on a more frequent interval (say 10 minutes). Similar, at least initially, to the thread recently started by S. Carey (Firing windows mulitple times). However, each time the win

Window function - iterator data

When you are using a window function the docs: https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/windows.html#windowfunction---the-generic-case state that A WindowFunction gets an Iterable containing all the elements of the window being processed If the input data strea

Using RabbitMQ Sinks

Hi all, The documentation describing the use of RabbitMQ as a sink gives the following example: RMQConnectionConfig connectionConfig = new RMQConnectionConfig.Builder() .setHost("localhost").setPort(5000).setUserName(..) .setPassword(..).setVirtualHost("/").build(); stream.addSink(new RMQSink(

flink batch data processing

I'm evaluating for some processing batches of data. As a simple example say I have 2000 points which I would like to pass through an FIR filter using functionality provided by the Python scipy libraryjk. The scipy filter is a simple function which accepts a set of coefficients and the data to