回复：Re: how to distributed run a bash shell in spark

[luohui20001]

Thanks Akhil, 
           your code is a big help to me,'cause perl script is the exactly 
thing i wanna try to run in spark. I will have a try.


--------------------------------

 

Thanks&Best regards!
San.Luo

----- 原始邮件 -----
发件人：Akhil Das <ak...@sigmoidanalytics.com>
收件人：罗辉 <luohui20...@sina.com>
抄送人：user <user@spark.apache.org>
主题：Re: how to distributed run a bash shell in spark
日期：2015年05月25日 00点53分

You mean you want to execute some shell commands from spark? Here's something i 
tried a while back. https://github.com/akhld/spark-exploitThanksBest Regards

On Sun, May 24, 2015 at 4:53 PM,  <luohui20...@sina.com> wrote:
hello there      I am trying to run a app in which part of it needs to run a 
shell.how to run a shell distributed in spark cluster.thanks.

here's my code:import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.DataFrame;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SQLContext;
import org.apache.spark.sql.types.DataType;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;

public class ShellCompare {
    public static void main(String[] args) {
        // TODO Auto-generated method stub
        SparkConf conf = new 
SparkConf().setAppName("ShellCompare").setMaster("spark://master:7077").set("spark.executor.memory",
 "6g");
        JavaSparkContext sc = new JavaSparkContext(conf);

        for(int i=1;i<=21;i++){
            execShell(i);
        }
//        execShell(1);
        sc.stop();
    }

    private static void execShell(int i) {
        String shpath="/opt/sh/bin/sort.sh";
        Process process =null;
        
        String var="/opt/data/shellcompare/chr" + i +".txt 
/opt/data/shellcompare/samplechr" + i +".txt /opt/data/shellcompare/result.txt 
600";   
//        String var="/opt/data/chr1.txt /opt/data/chr1sample.txt 
/opt/sh/bin/result.txt 600";  
        String command2 = "sh " + shpath + " " + var;
        try {
            process = Runtime.getRuntime().exec(command2);
            process.waitFor();
        } catch (InterruptedException e1) {
            // TODO Auto-generated catch block
            e1.printStackTrace();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }
}

--------------------------------

 

Thanks&amp;Best regards!
San.Luo