Hi How do you figure out 500gig~3900 partitions? I am trying to do the math. If I assume 64mb block size then 1G~16 blocks and 500g~8000 blocks. If we assume split and block sizes are same, shouldn't we end up with 8k partitions? On 4 May 2015 17:49, "Akhil Das" <[email protected]> wrote:
> 500GB of data will have nearly 3900 partitions and if you can have nearly > that many number of cores and around 500GB of memory then things will be > lightening fast. :) > > Thanks > Best Regards > > On Sun, May 3, 2015 at 12:49 PM, sherine ahmed <[email protected] > > wrote: > >> I need to use spark to upload a 500 GB data from hadoop on standalone mode >> cluster what are the minimum hardware requirements if it's known that it >> will be used for advanced analysis (social network analysis)? >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Hardware-requirements-tp22744.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> >
