Re: IS NOT NULL is not working in programmatic SQL in spark

2016-07-10 Thread Radha krishna
I want to apply null comparison to a column in sqlcontext.sql, is there any way to achieve this? On Jul 10, 2016 8:55 PM, "Radha krishna" wrote: > Ok thank you, how to achieve the requirement. > > On Sun, Jul 10, 2016 at 8:44 PM, Sean Owen wrote: > >> It doesn'

Re: IS NOT NULL is not working in programmatic SQL in spark

2016-07-10 Thread Radha krishna
Ok thank you, how to achieve the requirement. On Sun, Jul 10, 2016 at 8:44 PM, Sean Owen wrote: > It doesn't look like you have a NULL field, You have a string-value > field with an empty string. > > On Sun, Jul 10, 2016 at 3:19 PM, Radha krishna wrote: > > Hi All,IS NOT

IS NOT NULL is not working in programmatic SQL in spark

2016-07-10 Thread radha
Hi All,IS NOT NULL is not working in programmatic sql. check below for input output and code. Input 10,IN 11,PK 12,US 13,UK 14,US 15,IN 16, 17,AS 18,AS 19,IR 20,As val cntdat = sc.textFile("/user/poc_hortonworks/radha/gsd/sample.txt"); case class CNT (id:Int , code : String)

IS NOT NULL is not working in programmatic SQL in spark

2016-07-10 Thread Radha krishna
Hi All,IS NOT NULL is not working in programmatic sql. check below for input output and code. Input 10,IN 11,PK 12,US 13,UK 14,US 15,IN 16, 17,AS 18,AS 19,IR 20,As val cntdat = sc.textFile("/user/poc_hortonworks/radha/gsd/sample.txt"); case class CNT (id:Int , code : String)

Re: Spark Left outer Join issue using programmatic sql joins

2016-07-06 Thread Radha krishna
mer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages a

Re: Spark Left outer Join issue using programmatic sql joins

2016-07-06 Thread radha
Hi , Thanks all, its working fine the issue is with some space for the dept id, I have one more doubt for the non matching records its showing null word, even if i write into HDFS also its showing null word how can we avoid writing null for the non matching columns, i want just empty value ("") s

Re: Spark Left outer Join issue using programmatic sql joins

2016-07-06 Thread Radha krishna
1004,abf,30,30,IT] >> [1008,abk,30,30,IT] >> [1008,abk,30,30,IT] >> >> >> >> Note that you need to enforce ordering >> >> HTH >> >> >> >> >> >> >> >> >> >> >> >> Dr Mich Talebzadeh >

Spark Left outer Join issue using programmatic sql joins

2016-07-06 Thread radha
Hi All, Please check below for the code and input and output, i think the output is not correct, i am missing any thing? pls guide Code public class Test { private static JavaSparkContext jsc = null; private static SQLContext sqlContext = null; private static Configurat

Spark Left outer Join issue using programmatic sql joins

2016-07-06 Thread Radha krishna
ev| |10| 1010| abq|10| dev| |20| 1002| abs| null|null| |20| 1006| abh| null|null| |20| 1009| abl| null|null| |30| 1004| abf| null|null| |30| 1008| abk| null|null| +--+--++--++ Regards Radha

How to write the DataFrame results back to HDFS with other then \n as record separator

2016-06-28 Thread Radha krishna
o hdfs with the same line separator (RS[\u001e]) Thanks & Regards Radha krishna

How to write the DataFrame results back to HDFS with other then \n as record separator

2016-06-24 Thread Radha krishna
) using java Can any one suggest.. Note: i need to use other than \n bez my data contains \n as part of the column value. Thanks & Regards Radha krishna