Can't access the data in Kafka Spark Streaming globally

I am trying to stream the data from Kafka to Spark. JavaPairInputDStream directKafkaStream = KafkaUtils.createDirectStream(ssc, String.class, String.class, StringDecoder.class, StringDecoder.class, kafkaParams, topics)

How to perform Join operation using JAVARDD

I tried like this, *CrashData_1.csv:* *CRASH_KEYCRASH_NUMBER CRASH_DATECRASH_MONTH* *2016899114 2016899114 01/02/2016 12:00:00 AM +* *CrashData_2.csv:* *CITY_NAMEZIPCODE CITY STATE* *1945 704

How to read a Multi Line json object via Spark

I tried from Spark-Shell and i am getting the following error: Here is the test.json file: { "colorsArray": [{ "red": "#f00", "green": "#0f0", "blue": "#00f", "cyan": "#0ff", "magenta": "#f0f", "yellow": "#ff0", "black": "#000" }]}

Convert hive sql to spark sql

Hi users i need to test the performance of the query in hive and spark. Can any one convert these sql to spark sql. Here is the sql. SELECT split(DTD.TRAN_RMKS,'/')[0] AS TRAB_RMK1, split(DTD.TRAN_RMKS,'/')[1] AS ATM_ID, DTD.ACID, G.FORACID, DTD.REF_NUM, DTD.TRAN_ID, DTD.TRAN_DATE, DTD.VALUE_DATE

How to convert List into json object / json Array

Here is the snippet of my code : Dataset rows_salaries = spark.read().json("/Users/Macbook/Downloads/rows_salaries.json"); rows_salaries.createOrReplaceTempView("salaries"); List df = spark.sql("select * from salaries").collectAsList(); I need to read the json data from 'List df = spark.sq

How to convert List into json object / json Array

Any suggesttions please.

Converting Dataframe to resultSet in Spark Java

Retrieved the data to DataFrame but i can't convert into ResultSet Is there any possible way how to convert...Any suggestions please... Exception in thread "main" java.lang.ClassCastException: org.apache.spark.sql.DataFrame cannot be cast to com.datastax.driver.core.ResultSet -- Best Regards,

Re: How to insert data into 2000 partitions(directories) of ORC/parquet at a time using Spark SQL?

Hi Spark users, i am new to spark. I am trying to connect hive using SparkJavaContext. Unable to connect to the database. By executing the below code i can see only "default" database. Can anyone help me out. What i need is a sample program for Querying Hive results using SparkJavaContext. Need to

Content-based Recommendation Engine

Can anyone share the code for Content-based Recommendation Engine to recommend the user based on E-mail subject. -- Best Regards, Sreeharsha Eedupuganti Data Engineer innData Analytics Private Limited

Content-based Recommendation Engine

Can anyone share the code for Content-based Recommendation Engine to recommend the user based on E-mail subject. -- Best Regards, Sreeharsha Eedupuganti Data Engineer innData Analytics Private Limited

Spark Select Statement

Hello Spark users, can we query the SQL SELECT statement in Spark using Java. if it is possible any suggestions please. I tried like this.How to pass the database name. Here my database name is nimbus and table name is winbox_opens. *Source Code :* *public class Select { public static class Squar

Need a sample code to load XML files into cassandra database using spark streaming

Hello everyone, new to spark streaming, need a sample code to load xml files from AWS S3 server to cassandra database. Any suggesttions please, Thanks in advance. -- Best Regards, Sreeharsha Eedupuganti Data Engineer innData Analytics Private Limited

How to migrate spark code to spark streaming ?

Hello everyone, Loading XML files from S3 to database [i.e Cassandra]. Right now my code is in Spark Core. I want to migrate my code to Spark Streaming because for every 15 minutes we have to load XML files into database. So in this case i need to migrate my code to Spark Streaming. Any suggestions

How to send a file to database using spark streaming

New to Spark Streaming. My question is i want to load the XML files to database [cassandra] using spark streaming.Any suggestions please.Thanks in Advance. -- Best Regards, Sreeharsha Eedupuganti Data Engineer innData Analytics Private Limited

Getting an error while submitting spark jar

The way how i submitting jar hadoop@localhost:/usr/local/hadoop/spark$ ./bin/spark-submit \ > --class mllib.perf.TesRunner \ > --master spark://localhost:7077 \ > --executor-memory 2G \ > --total-executor-cores 100 \ > /usr/local/hadoop/spark/lib/mllib-perf-tests-assembly.jar \ > 1000

Getting an error in insertion to mysql through sparkcontext in java..

i had 9 rows in my Mysql table options.put("dbtable", "(select * from employee"); options.put("lowerBound", "1"); options.put("upperBound", "8"); options.put("numPartitions", "2"); Error : Parameter index out of range (1 > number of parameters, which is 0) -- Best Regards,