RE: How to export data from hive into hdfs in spark program?

[LinQili]

Hi Leo:Thanks for your reply.I am talking about using hive from spark to export 
data from hive to hdfs.maybe like:      val exportData = s"insert overwrite 
directory '/user/linqili/tmp/src' select * from $DB.$tableName"      
hiveContext.sql(exportData)but it was unsupported in spark now:Exception in 
thread "Thread-3" java.lang.reflect.InvocationTargetException    at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)   
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)    at 
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:183)Caused
 by: java.lang.RuntimeException: Unsupported language features in query: insert 
overwrite directory '/user/linqili/tmp/src' select * from 
test_spark.srcTOK_QUERY  TOK_FROM    TOK_TABREF      TOK_TABNAME        
test_spark        src   TOK_INSERT    TOK_DESTINATION      TOK_DIR        
'/user/linqili/tmp/src'    TOK_SELECT      TOK_SELEXPR        TOK_ALLCOLREF
        at scala.sys.package$.error(package.scala:27)    at 
org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:256)    at 
org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:106)    at 
org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:110)    at 
com.nd.huayuedu.HiveExportTest$.main(HiveExportTest.scala:35)    at 
com.nd.huayuedu.HiveExportTest.main(HiveExportTest.scala)    ... 5 moreDate: 
Tue, 23 Dec 2014 16:47:11 +0800
From: leo.chen.cip...@outlook.com
To: lin_q...@outlook.com
Subject: Re: How to export data from hive into hdfs in spark program?


  
    
  
  
    Hi,

    

    If you are talking about using spark's thriftserver, this query
    should work:

    export table $DB.$tableName to '/user/linqili/tmp/src';

    However you need to take care of that folder (by deleting it I
    presume) first.

    

    Cheers,

    Leo

    On 2014/12/23 16:09, LinQili wrote:

    
    
      
      Hi all:
        I wonder if is there a way to export data from table of
          hive into hdfs using spark?
        like this:  INSERT OVERWRITE DIRECTORY
          '/user/linqili/tmp/src' select * from $DB.$tableName