we use spark in web server , no application.jar and submit on the cluster
On 11/2/2018 13:56，Arbab Khalil<akha...@an10.io> wrote：
remove master configuration from code and then submit it to any cluster, it should work.On Fri, Nov 2, 2018 at 10:52 AM 崔苗(数据与人工智能产品开发部) <0049003...@znv.com> wrote:
then how about spark sql and spark MLlib , we use them at most timeOn 11/2/2018 11:58，Daniel de Oliveira Mantovani<daniel.oliveira.mantov...@gmail.com> wrote：Please, read about Spark Streaming or Spark Structured Streaming. Your web application can easily communicate through some API and you won’t have the overhead of start a new spark job, which is pretty heavy.--On Thu, Nov 1, 2018 at 23:01 崔苗(数据与人工智能产品开发部) <0049003...@znv.com> wrote:--------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Hi，we want to execute spark code with out submit application.jar,like this code:public static void main(String args[]) throws Exception{SparkSession spark = SparkSession.builder().master("local[*]").appName("spark test").getOrCreate();Dataset<Row> testData = spark.read().csv(".\\src\\main\\java\\Resources\\no_schema_iris.scv");testData.printSchema();testData.show();}the above code can work well with idea , do not need to generate jar file and submit , but if we replace master("local[*]") with master("yarn") , it can't work , so is there a way to use cluster sparkSession like local sparkSession ? we need to dynamically execute spark code in web server according to the different request , such as filter request will call dataset.filter() , so there is no application.jar to submit .
--Daniel de Oliveira MantovaniPerl Evangelist/Data Hacker+1 786 459 1341--Regards,Arbab KhalilSoftware Design Engineer
