Re: ETL process design

Hi Danny, What you describe sounds like you may also consider to use Spring XD instead, at least for the file-centric stuff. Regards Ben Von meinem iPad gesendet > Am 28.01.2015 um 10:42 schrieb Danny Yates : > > Hi, > > My apologies for what has ended up as quite a long email with a lot of

Re: Is Spark the right tool for me?

or react appropriately without knowing the whole context. BTW, to answer your very first question: yes, it looks like Spark will help you! cheers, andy On Mon Dec 01 2014 at 4:36:44 PM Stadin, Benjamin mailto:benjamin.sta...@heidelberg-mobil.com>> wrote: Yes, the processing causes the

Re: Is Spark the right tool for me?

one node (or if your driver is more beefy, do a collect/foreach) to create the index. You could also create a GeoJSON of the geometries and create the r-tree on it (not sure about this one). On Mon Dec 01 2014 at 3:38:00 PM Stadin, Benjamin mailto:benjamin.sta...@heidelberg-mobil.com>>

Re: Is Spark the right tool for me?

, and it's simple SQLite. In case you need an r-tree (or related) index, you're headaches will come from congestion within your database transaction... unless you go to a dedicated database like Vertica (just mentioning) kr, andy On Mon Dec 01 2014 at 2:49:44 PM Stadin, Benjamin mai

Re: Is Spark the right tool for me?

icated database like Vertica (just mentioning) kr, andy On Mon Dec 01 2014 at 2:49:44 PM Stadin, Benjamin mailto:benjamin.sta...@heidelberg-mobil.com>> wrote: Hi all, I need some advise whether Spark is the right tool for my zoo. My requirements share commonalities with „big data“, workf

Is Spark the right tool for me?

Hi all, I need some advise whether Spark is the right tool for my zoo. My requirements share commonalities with „big data“, workflow coordination and „reactive“ event driven data processing (as in for example Haskell Arrows), which doesn’t make it any easier to decide on a tool set. NB: I have