Spark join: grouping of records having same value for a particular column in same partition

partition grouping while reading, then is there another efficient way this can be done other than caching? Regards, Aravind

Re: Spark batch job: failed to compile: java.lang.NullPointerException

Spark version is 2.2 and I think I am running into this issue https://issues.apache.org/jira/browse/SPARK-18016as the dataset schema is pretty huge and nested From: ARAVIND SETHURATHNAM Date: Monday, June 18, 2018 at 4:00 PM To: "user@spark.apache.org" Subject: Spark batch job:

unsubscribe

unsubscribe -- Wealth is not money. Wealth is relationships with people.

Re: SPARK SQL: returns null for a column, while HIVE query returns data for the same column

- this column was added in later partitions and not present in earlier ones. - - i assume partition pruning should just load from that particular partition i am specifying when using spark sql ? - (spark version 2.2) On Fri, May 11, 2018 at 2:24 PM, ARAVIND ARUMUGHAM

SPARK SQL: returns null for a column, while HIVE query returns data for the same column

I have a hive table created on top of s3 DATA in parquet format and partitioned by one column named eventdate. 1) When using HIVE QUERY, it returns data for a column named "headertime" which is in the schema of BOTH the table and the file. select headertime from dbName.test_bug where eventdate=20

Structured streaming: Tried to fetch $offset but the returned record offset was ${record.offset}"

someone provide some direction what could be causing this all of a sudden when consuming from those topics? regards Aravind

Re: java.lang.ClassCastException when I execute a Spark SQL command

sorry, no syntax errors in the sql, I missed the 'like' when I pasted the sql in the email.I'm getting the exception for this sql %sql insert overwrite table  table2 partition(node) select  * from  table1  where  field1 like '%google%'and node = 'DCP

java.lang.ClassCastException when I execute a Spark SQL command

lExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Driver stacktrace:set zeppelin.spark.sql.stacktrace = true to see full stacktrace  P.S. "Arvind" Aravind www.linkedin.com/in/psaravind

Re: Snappy and spark 1.1

Hi Guys, As part of debugging this "native library" error in our environment, it would be great if somebody can help me with this question. What kind of temp, scratch, and staging directories does Spark need and use on the slave nodes in the YARN cluster mode? Thanks, Aravind On

Snappy and spark 1.1

nPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thanks, Aravind

Re: Spark Streaming using File Stream in Java

Hi Akil, It didnt work. Here is the code... package com.paypal; import org.apache.spark.SparkConf; import org.apache.spark.storage.StorageLevel; import org.apache.spark.streaming.api.java.JavaPairInputDStream; import org.apache.spark.streaming.api.java.JavaStreamingContext; import org.apache.sp

Spark Streaming using File Stream in Java

updates? I am a new user of Apache Spark Streaming. Kindly help me as I am totally stuck Thanks in advance. Regards Aravind -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-using-File-Stream-in-Java-tp9115.html Sent from the Apache