Re: Hardware requirements and learning resources

2015-09-03 Thread Kostas Tzoumas
Well hidden. I added now a link at the menu of http://data-artisans.com/. This material is provided for free by data Artisans but they are not part of the official Apache Flink project. On Thu, Sep 3, 2015 at 2:20 PM, Stefan Winterstein < stefan.winterst...@dfki.de> wrote: > > > Answering to mys

Re: Hardware requirements and learning resources

2015-09-03 Thread Stefan Winterstein
> Answering to myself, I have found some nice training material at > http://dataartisans.github.io/flink-training. Excellent resources! Somehow, I managed not to stumble over them by myself - either I was blind, or they are well hidden... :) Best, -Stefan

Re: Hardware requirements and learning resources

2015-09-02 Thread Juan Rodríguez Hortalá
Hi Robert and Jay, Thanks for your answers. The petstore jobs could indeed be used as a roseta code for Flink and Spark. Regarding the memory requirements, those are very good news to me, just 2GB of RAM is certainly a modest amount of memory, you can use even some Single Board Computers for that

Re: Hardware requirements and learning resources

2015-09-02 Thread Juan Rodríguez Hortalá
Hi Kostas, Thanks a lot for your answer. It's nice to know there are more training videos on their way, they will be on my watch list. I guess you'll be using the data Artisans channel for the new videos too. Greetings, Juan 2015-09-02 14:30 GMT+02:00 Kostas Tzoumas : > Hi Juan, > > Flink is

Re: Hardware requirements and learning resources

2015-09-02 Thread Robert Metzger
@Jay: I've looked into your code, but I was not able to reproduce the issue. I'll start a new discussion thread on the user@flink list for the Flink-BigPetStore discussion. I don't want to take over Juan's hardware-requirements discussion ;) On Wed, Sep 2, 2015 at 3:01 PM, Jay Vyas wrote: > Jus

Re: Hardware requirements and learning resources

2015-09-02 Thread Jay Vyas
Just running the main class is sufficient > On Sep 2, 2015, at 8:59 AM, Robert Metzger wrote: > > Hey jay, > > How can I reproduce the error? > >> On Wed, Sep 2, 2015 at 2:56 PM, jay vyas wrote: >> We're also working on a bigpetstore implementation of flink which will help >> onboard spark/m

Re: Hardware requirements and learning resources

2015-09-02 Thread Robert Metzger
Hey jay, How can I reproduce the error? On Wed, Sep 2, 2015 at 2:56 PM, jay vyas wrote: > We're also working on a bigpetstore implementation of flink which will > help onboard spark/mapreduce folks. > > I have prototypical code here that runs a simple job in memory, > contributions welcome, > >

Re: Hardware requirements and learning resources

2015-09-02 Thread jay vyas
We're also working on a bigpetstore implementation of flink which will help onboard spark/mapreduce folks. I have prototypical code here that runs a simple job in memory, contributions welcome, right now there is a serialization error https://github.com/bigpetstore/bigpetstore-flink . On Wed, Se

Re: Hardware requirements and learning resources

2015-09-02 Thread Stephan Ewen
I have actually run Flink nodes with 50 MB of memory and processed multiple gigabytes, but that is truely a toy setup for experimentation. As Robert said, a Mini-Cluster with two local workers (each around 300-400 MB memory) plus a master node (200-300 MB) gives you 1 GB of total needed memory and

Re: Hardware requirements and learning resources

2015-09-02 Thread Robert Metzger
Hi Juan, I think the recommendations in the Spark guide are quite good, and are similar to what I would recommend for Flink as well. Depending on the workloads you are interested to run, you can certainly use Flink with less than 8 GB per machine. I think you can start Flink TaskManagers with 500

Re: Hardware requirements and learning resources

2015-09-02 Thread Kostas Tzoumas
Hi Juan, Flink is quite nimble with hardware requirements; people have run it in old-ish laptops and also the largest instances available in cloud providers. I will let others chime in with more details. I am not aware of something along the lines of a cheatsheet that you mention. If you actually

Re: Hardware requirements and learning resources

2015-09-02 Thread Juan Rodríguez Hortalá
Answering to myself, I have found some nice training material at http://dataartisans.github.io/flink-training. There are even videos at youtube for some of the slides - http://dataartisans.github.io/flink-training/overview/intro.html https://www.youtube.com/watch?v=XgC6c4Wiqvs - http://da

Hardware requirements and learning resources

2015-09-02 Thread Juan Rodríguez Hortalá
Hi list, I'm new to Flink, and I find this project very interesting. I have experience with Apache Spark, and for I've seen so far I find that Flink provides an API at a similar abstraction level but based on single record processing instead of batch processing. I've read in Quora that Flink exten