> 1. Writing scripts for automated setup of a multi-node cluster for Apache Spark with Hadoop File System (HDFS). This is required since I don’t have a fixed set of machines to run my Spark experiments and hence, need an easy, quick and automated way to do the entire Spark setup.
Where will you be bringing up these instances? If it's on AWS, then flintrock <https://github.com/nchammas/flintrock/> will do all this for you. If not, you can write a custom provider that works for you. It would just need to return an instance of FlintrockCluster <https://github.com/nchammas/flintrock/blob/master/flintrock/core.py#L47>. Thanks, Hariharan On Sat, Apr 10, 2021 at 9:00 AM Dhruv Kumar <[email protected]> wrote: > Hello > > I am new to Apache Spark and am looking for some close guidance or > collaboration for my Spark Project which has the following main components: > > 1. Writing scripts for automated setup of a multi-node cluster for Apache > Spark with Hadoop File System (HDFS). This is required since I don’t have a > fixed set of machines to run my Spark experiments and hence, need an easy, > quick and automated way to do the entire Spark setup. > > 2. Writing scripts for simple SQL queries which read input from HDFS, run > the SQL queries on the multi-node spark cluster and store the output in > HDFS. > > 3. Generating detailed profiling results such as latency, shuffled data > size for every task/operator in the SQL query and generating graphs for the > same. > > Please let me know if anyone is interested. Happy to discuss in more > detail. > > Thanks > Dhruv > [email protected] > > -------------------------------------------------- > *Dhruv Kumar* > PhD Candidate > Computer Science and Engineering > University of Minnesota > www.dhruvkumar.me <http://dhruvkumar.me> >
