Hi all,I wonder if there is a way to create some child streaming while using
spark streaming?For example, I create a netcat main stream, read data from a
socket, then create 3 different child streams on the main stream,in stream1, we
do fun1 on the input data then print result to screen;in strea
Hi, allI wonder how to delete hdfs file/directory using spark API?
Hi all,I was doing select using spark sql like:
insert into table startup_log_uid_20150227select * from
bak_startup_log_uid_20150227where login_time < 1425027600
Usually, it got a exception:
org.apache.hadoop.hive.ql.metadata.Hive.checkPaths(Hive.java:2157)org.apache.hadoop.hive.ql.metadata.Hive.c
conf.getBoolVar(HiveConf.ConfVars.HIVE_HADOOP_SUPPORTS_SUBDIRECTORIES) &&
item.isDir()) {
throw new HiveException("checkPaths: " + src.getPath()
+ " has nested directory" + itemSource);
}
|
On 3/3/15 14:36, LinQili wrot
Hi all:I was using spark sql on hive for doing some statistics. While using
spark sql, I found that, sometimes, spark-sql got the incorrect results while
select. And if I executed the same select statement again, it returned the
different results which was correct. That is saying, sometime, with
Hi all.
I was launching a spark sql job on my own machine, not on the spark cluster
machines, and failed. The excpetion info is:
15/04/28 16:28:04 INFO yarn.ApplicationMaster: Final app status: FAILED,
exitCode: 15, (reason: User class threw exception: java.lang.RuntimeException:
Unable to insta
Hi all:I deployed a spark client in my own machine. I put SPARK in path:`
/home/somebody/spark`, and the cluster's worker spark home path is
`/home/spark/spark` .While I launched the jar, it shows that: `
AppClient$ClientActor: Executor updated: app-20141124170955-11088/12 is now
FAILED (java.i
Hi, all:
According to https://github.com/apache/spark/pull/2732, When a spark job fails
or exits nonzero in yarn-cluster mode, the spark-submit will get the
corresponding return code of the spark job. But I tried in spark-1.1.1 yarn
cluster, spark-submit return zero anyway.
Here is my spark code
I tried in spark client mode, spark-submit can get the correct return code from
spark job. But in yarn-cluster mode, It failed.
From: lin_q...@outlook.com
To: u...@spark.incubator.apache.org
Subject: Issue on [SPARK-3877][YARN]: Return code of the spark-submit in
yarn-cluster mode
Date: Fri, 5 D
I tried anather test code: def main(args: Array[String]) {if (args.length
!= 1) { Util.printLog("ERROR", "Args error - arg1: BASE_DIR")
exit(101) }val currentFile = args(0).toStringval DB = "test_spark"
val tableName = "src"
val sparkConf = new SparkConf().setApp
Hi all:I was running HiveFromSpark on yarn-cluster. While I got the hive
select's result schemaRDD and tried to run `collect()` on it, the application
got stuck and don't know what's wrong with it. Here is my code:
val sqlStat = s"SELECT * FROM $TABLE_NAME" val result =
hiveContext.hql(sqlStat)
I checked my code again, and located the issue that, if we do the `load data
inpath` before select statement, the application will get stuck, if don't, it
won't get stuck.Log info: 14/12/09 17:29:33 ERROR actor.ActorSystemImpl:
Uncaught fatal error from thread [sparkDriver-akka.actor.default-dis
I checked my code again, and located the issue that, if we do the `load data
inpath` before select statement, the application will get stuck, if don't, it
won't get stuck.Get stuck code: val sqlLoadData = s"LOAD DATA INPATH
'$currentFile' OVERWRITE INTO TABLE $tableName "
hiveContext.
Hi all,In yarn-cluster mode, we know that the driver machine is running on
applicationMaster, applicationMaster is one of the cluster machines.I am
wondering if there is a way to specify the driver on a specific machine of the
cluster in yarn cluster mode?Thanks.
Hi all,On yarn-cluster mode, can we let the driver running on a specific
machine that we choose in cluster ? Or, even the machine not in the cluster?
Hi all:I wonder if is there a way to export data from table of hive into hdfs
using spark?like this: INSERT OVERWRITE DIRECTORY '/user/linqili/tmp/src'
select * from $DB.$tableName
Hi Leo:Thanks for your reply.I am talking about using hive from spark to export
data from hive to hdfs.maybe like: val exportData = s"insert overwrite
directory '/user/linqili/tmp/src' select * from $DB.$tableName"
hiveContext.sql(exportData)but it was
Hi all:I am using spark sql to read and write hive tables. But There is a issue
that how to select the first row in each group by group?In hive, we could write
hql like this:SELECT imeiFROM (SELECT imei,
row_number() over (PARTITION BY imei ORDER BY login_time ASC)
18 matches
Mail list logo
