You mean you want to execute some shell commands from spark? Here's
something i tried a while back. https://github.com/akhld/spark-exploit
Thanks
Best Regards
On Sun, May 24, 2015 at 4:53 PM, <luohui20...@sina.com> wrote:
> hello there
>
> I am trying to run a app in which part of it needs to run a
> shell.how to run a shell distributed in spark cluster.thanks.
>
>
> here's my code:
>
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
>
> import org.apache.spark.SparkConf;
> import org.apache.spark.api.java.JavaRDD;
> import org.apache.spark.api.java.JavaSparkContext;
> import org.apache.spark.sql.DataFrame;
> import org.apache.spark.sql.Row;
> import org.apache.spark.sql.SQLContext;
> import org.apache.spark.sql.types.DataType;
> import org.apache.spark.sql.types.StructField;
> import org.apache.spark.sql.types.StructType;
>
> public class ShellCompare {
> public static void main(String[] args) {
> // TODO Auto-generated method stub
> SparkConf conf = new
> SparkConf().setAppName("ShellCompare").setMaster("spark://master:7077").set("spark.executor.memory",
> "6g");
> JavaSparkContext sc = new JavaSparkContext(conf);
>
> for(int i=1;i<=21;i++){
> execShell(i);
> }
> // execShell(1);
> sc.stop();
> }
>
> private static void execShell(int i) {
> String shpath="/opt/sh/bin/sort.sh";
> Process process =null;
>
> String var="/opt/data/shellcompare/chr" + i +".txt
> /opt/data/shellcompare/samplechr" + i +".txt
> /opt/data/shellcompare/result.txt 600";
> // String var="/opt/data/chr1.txt /opt/data/chr1sample.txt
> /opt/sh/bin/result.txt 600";
> String command2 = "sh " + shpath + " " + var;
> try {
> process = Runtime.getRuntime().exec(command2);
> process.waitFor();
> } catch (InterruptedException e1) {
> // TODO Auto-generated catch block
> e1.printStackTrace();
> } catch (IOException e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
> }
>
> }
>
>
>
> --------------------------------
>
> Thanks&Best regards!
> San.Luo
>
