Re: How to start with the first Kafka Message

If it is a one time thing, you could run the kafka consumer script which has the --from-beginning option, and redirect the output as a socketstream and consume in flink. On Thu, Mar 17, 2016 at 7:24 PM, Till Rohrmann wrote: > Hi Dominique, > > have you tried setting the Kafka property props.put(

Re: Flink job on secure Yarn fails after many hours

Hi Niels, Thanks for the feedback. As far as I know, Hadoop deliberately defaults to the one week maximum life time of delegation tokens. Have you tried increasing the maximum token life time or was that not an option? I wonder why do you use a while loop? Would it be possible to use the Yarn fai

Re: Flink and YARN ship folder

Sounds like a bug. I've only tried this with yarn-sessions. Could it be that the behaviour is different between the two variants? In any case, I sounds a lot like a bug in Flink. Robert can probably chime in here with details. On Wed, Mar 16, 2016 at 6:25 PM, Andrea Sella wrote: > Hi, > > After f

Re: RollingSink with APIs requring fs+path

Hi, you are right, it is currently only possible to write to a FSDataOutputStream. It could be generified as you mentioned. One thing that needs to be taken care of, however, is that the write offsets are correctly checkpointed to ensure exactly-once semantics in case of failure. Right now, we d

Re: RollingSink with APIs requring fs+path

Thanks, great to hear that! - aljoscha > On 18 Mar 2016, at 14:52, Lasse Dalegaard wrote: > > Hello, > > Thanks for verifying my thesis. I've created FLINK-3637( > https://issues.apache.org/jira/browse/FLINK-3637 ) and will start working on > this :-) > > Best regards, > Lasse > ___

RE: Javadoc version

Hi Robert, Thanks for the explanation. Though "1.0-SNAPSHOT" is the default version for many projects throughout their lifespan :) So I think it's a bit confusing. That's the main issue for me, not whether the docs are slightly out of sync with the actual 1.0 release code. In any case, it's a

Re: what happens to the aggregate in a fold on a KeyedStream if the events for a key stop coming ?

Hi Fabian So you're saying that with a windowed stream I can still emit a folded aggregate for each event as it comes in? I didn't realize that, I thought that windows was a sort of micro batching. I'll go read the link you posted Thanks -- Bart van Deenen bartvandee...@fastmail.fm On

Re: what happens to the aggregate in a fold on a KeyedStream if the events for a key stop coming ?

Yes, that's possible. You have to implement a custom trigger for that. The Trigger.onElement() method will be called for each incoming event. If you return TriggerResult.FIRE, it will call the WindowFunction. You can register a timer which will call the Trigger.onXTime() method once time is up and

Re: OutofMemoryError: Java heap space & Loss of Taskmanager

Are you facing these issues with the batch or streaming programs? – Ufuk On Wed, Mar 16, 2016 at 4:30 PM, Till Rohrmann wrote: > If the problem is that your JVMs stall too long, then you can also increase > the akka.ask.timeout configuration value in flink-config.yaml. That will > also increase

Re: what happens to the aggregate in a fold on a KeyedStream if the events for a key stop coming ?

Ok, thanks! I'll do it that way, with a custom trigger and a fold per key. Bart -- Bart van Deenen bartvandee...@fastmail.fm On Fri, Mar 18, 2016, at 13:31, Fabian Hueske wrote: > The "weight" of a window depends on the function that you apply. > If you apply a generic WindowFunction Fli

Re: Input on training exercises

Hi Ken, you can open an issue on the Github repository or send a mail to me. Thanks, Fabian 2016-03-17 23:07 GMT+01:00 Ken Krugler : > Hi list, > > What's the right way to provide input on the training exercises > ? > > Thanks, > > -- Ken

Input on training exercises

Hi list, What's the right way to provide input on the training exercises? Thanks, -- Ken -- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr

Re: Flink Checkpoint on yarn

On Thu, Mar 17, 2016 at 11:51 AM, Stefano Baghino wrote: > does the recovery.zookeeper.path.root property need to be set independently > for each job that is run? No, just per cluster.

Re: operators

Hi Radu, the API call slotSharingGroup was introduced with version 1.0. In the version 0.10 there was something similar called startNewResourceGroup, but it was somewhat broken. Therefore, I would recommend you upgrading to version 1.0. You can find the description of the new method here [1]. The

Re: off-heap size feature request

Updating the documentation will partially remove the confusion. Heap memory is JVM managed while off-heap is outside JVM and I would define its size separately. In my case, I see memory usage going up to full utilisation on a node where full RAM available is 128GB in the following conditions: ta

Re: Flink and YARN ship folder

Thank you very much! Yes, please go ahead an open an issue for this. Thanks for reporting this! On Fri, Mar 18, 2016 at 5:08 PM, Stefano Baghino wrote: > Hi Ufuk, > > I've just run a long running session and in that mode the libraries are > correctly shipped without the need to specify -t lib. >

Not enough free slots to run the job

Hi, For the situation where a program specify a maximum parallelism (so it is supposed to use all available task slots) we can have the possibility that one of the task managers is not registered for various reasons. In this case the job will fail for not enough free slots to run the job. For m