Re: S3 Access in eu-central-1

2017-11-28 Thread Dominik Bruhn
Hadoop 2.7.x already, or only in Hadoop 2.8? Best, Stephan On Mon, Nov 27, 2017 at 9:47 AM, Dominik Bruhn mailto:domi...@dbruhn.de>> wrote: Hey, can anyone give a hint? Does anyone have flink running with an S3 Bucket in Frankfurt/eu-central-1

Re: S3 Access in eu-central-1

2017-11-27 Thread Dominik Bruhn
Hey, can anyone give a hint? Does anyone have flink running with an S3 Bucket in Frankfurt/eu-central-1 and can share his config and setup? Thanks, Dominik > On 22. Nov 2017, at 17:52, domi...@dbruhn.de wrote: > > Hey everyone, > I'm trying since hours to get Flink 1.3.2 (downloaded for hadoop

Set Parallelism and keyBy

2016-12-26 Thread Dominik Bruhn
Hey, I have a flink job which has a default parallelism set to 2. I want to key the stream and then apply some flatMap on the keyed stream. The flatMap operation is quiet costly, so I want to have a much higher parallelism here (lets say 16). Additionally, it is important that the flatMap oper

Re: Who's hiring, December 2016

2016-12-16 Thread Dominik Bruhn
Relayr, a Berlin based IoT company is using Apache Flink for processing its sensor data. We are still in the learning phase, but are commited to using Flink. Check out our public job ads for Berlin and Munich: https://relayr.io/jobs/ If you are interested, even if there is no clearly matching

Release Process

2016-11-03 Thread Dominik Bruhn
Hey everyone, about three month ago, I made a PR [1] to the flink github project containing a small change for the RabbitMQ source. This PR was merged and the code is in the master. But: This code never made it into a release. In JIRA [2], it is meant to be released with 1.2. How is the polic

BoundedOutOfOrdernessTimestampExtractor and timestamps in the future

2016-11-01 Thread Dominik Bruhn
Hey, I'm using a BoundedOutOfOrdernessTimestampExtractor for assigning my timestamps and discarding to old events (which happens sometimes). Now my problem is that some events, by accident have timestamps in the future. If the timestamps are more in the future than my `maxOutOfOrderness`, I'm

TimeWindow Trigger which only fires when the values have changed

2016-10-04 Thread Dominik Bruhn
Hi, I'm heavily relying on TimeWindows for my real time processing. Roughly my job consumes from an AMQP queue, computes some time buckets and saves the time-buckets to Cassandra. I found the EventTimeTriggerWithEarlyAndLateFiring [1] class which already helped me a lot: Even with long time-w