Re: Large Numbers of Dynamically Created Jobs

2016-03-22 Thread David Brelloch
Jamie, That looks fantastic! Thanks for the help. David On Tue, Mar 22, 2016 at 6:22 PM, Jamie Grier wrote: > Hi David, > > Here's an example of something similar to what you're talking about: > https://github.com/jgrier/FilteringExample > > Have a look at the TweetImpressionFilteringJob. > >

Re: Large Numbers of Dynamically Created Jobs

2016-03-22 Thread Jamie Grier
Hi David, Here's an example of something similar to what you're talking about: https://github.com/jgrier/FilteringExample Have a look at the TweetImpressionFilteringJob. -Jamie On Tue, Mar 22, 2016 at 2:24 PM, David Brelloch wrote: > Konstantin, > > Not a problem. Thanks for pointing me in t

Re: Large Numbers of Dynamically Created Jobs

2016-03-22 Thread David Brelloch
Konstantin, Not a problem. Thanks for pointing me in the right direction. David On Tue, Mar 22, 2016 at 5:17 PM, Konstantin Knauf < konstantin.kn...@tngtech.com> wrote: > Hi David, > > interesting use case, I think, this can be nicely done with a comap. Let > me know if you run into problems, u

Re: Large Numbers of Dynamically Created Jobs

2016-03-22 Thread Konstantin Knauf
Hi David, interesting use case, I think, this can be nicely done with a comap. Let me know if you run into problems, unfortunately I am not aware of any open source examples. Cheers, Konstnatin On 22.03.2016 21:07, David Brelloch wrote: > Konstantin, > > For now the jobs will largely just invo

Re: Large Numbers of Dynamically Created Jobs

2016-03-22 Thread David Brelloch
Konstantin, For now the jobs will largely just involve incrementing or decrementing based on the json message coming in. We will probably look at adding windowing later but for now that isn't a huge priority. As an example of what we are looking to do lets say the following 3 message were read fr

Re: Large Numbers of Dynamically Created Jobs

2016-03-22 Thread Konstantin Knauf
Hi David, I have no idea how many parallel jobs are possible in Flink, but generally speaking I do not think this approach will scale, because you will always only have one job manager for coordination. But there is definitely someone on the list, who can tell you more about this. Regarding your

Large Numbers of Dynamically Created Jobs

2016-03-22 Thread David Brelloch
Hi all, We are currently evaluating flink for processing kafka messages and are running into some issues. The basic problem we are trying to solve is allowing our end users to dynamically create jobs to alert based off the messages coming from kafka. At launch we figure we need to support at least