It does not matter whether you start your spark with local or other mode. If
you have hdfs-site.xml somewhere and spark configuration pointing to that
config, you will read/write to HDFS.
Thanks.
Zhan Zhang
From: Madhu
Sent: Saturday, September 19, 201
Juan, I wouldn't go as far as suggesting we switch from programming using
RDDs to using SparkIterable. For example, all methods involving context,
jobs or partitions should only be part of the RDD API and not part of
SparkIterable. That said, the Spark community would benefit from a
consistent set
Great!
Jon / Shane: Thanks for handling this.
On Saturday, September 19, 2015, shane knapp wrote:
> we're up and building! time for breakfast... :)
>
> https://amplab.cs.berkeley.edu/jenkins/
>
> On Sat, Sep 19, 2015 at 7:35 AM, shane knapp > wrote:
> > it was definitely one of our servers..
Hi Sim,
I understand that what you propose is defining a trait SparkIterable (and
also PairSparkIterable for RDDs of pairs) that encapsulates the methods in
RDDs, and then program using that trait instead of RDD. That is similar to
programming using scala.collection.GenSeq to abstract from using a
Looks like you didn't specify sparkr profile when building.
Cheers
On Sat, Sep 19, 2015 at 12:30 PM, Devl Devel
wrote:
> Hi All,
>
> I've built spark 1.5.0 with hadoop 2.6 with a fresh download :
>
> build/mvn -Phadoop-2.6 -Dhadoop.version=2.6.0 -DskipTests clean package
>
> I try to run Spark
Hi All,
I've built spark 1.5.0 with hadoop 2.6 with a fresh download :
build/mvn -Phadoop-2.6 -Dhadoop.version=2.6.0 -DskipTests clean package
I try to run SparkR it launches the normal R without the spark addons:
./bin/sparkR --master local[*]
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/shar
Thanks guys.
I do have HADOOP_INSTALL set, but Spark 1.4.1 did not seem to mind.
Seems like there's a difference in behavior between 1.5.0 and 1.4.1 for some
reason.
To the best of my knowledge, I just downloaded each tgz and untarred them in
/opt
I adjusted my PATH to point to one or the other,
It sounds a lot like you have some local Hadoop config pointing to a
cluster, and you're picking that up when you run the shell. Look for
HADOOP_* env variables and clear them, and use --master local[*]
On Sat, Sep 19, 2015 at 5:14 PM, Madhu wrote:
> I downloaded spark-1.5.0-bin-hadoop2.6.tgz rec
Maybe you have a hdfs-site.xml lying around somewhere?
On Sat, Sep 19, 2015 at 9:14 AM, Madhu wrote:
> I downloaded spark-1.5.0-bin-hadoop2.6.tgz recently and installed on
> CentOS.
> All my local Spark code works fine locally.
>
> For some odd reason, spark-shell doesn't work in local mode.
>
we're up and building! time for breakfast... :)
https://amplab.cs.berkeley.edu/jenkins/
On Fri, Sep 18, 2015 at 9:30 PM, jon kuroda wrote:
> Starting tonight at about 6:45PM, the AMP Jenkins instance, which is
> hosted at the main UC Berkeley Campus Datacenter, went dark. This is
> almost cert
we're up and building! time for breakfast... :)
https://amplab.cs.berkeley.edu/jenkins/
On Sat, Sep 19, 2015 at 7:35 AM, shane knapp wrote:
> it was definitely one of our servers... we have no ETA on when
> jenkins will be back online. we will need to inspect the rack closely
> before we plu
I downloaded spark-1.5.0-bin-hadoop2.6.tgz recently and installed on CentOS.
All my local Spark code works fine locally.
For some odd reason, spark-shell doesn't work in local mode.
It looks like it want's to connect to HDFS, even if I use --master local or
specify local mode in the conf.
Even sc.
Hi all,
I'm using Spark (1.4.1) + Hive (0.13.1), I found that a large number of
network IO appeared when query a parquet table *with only one part file* use
SparkSQL.
The SQL is: SELECT concat(year(fkbb5855f0), "-", month(fkbb5855f0), "-",
day(fkbb5855f0), " 00:00:00"),COUNT(fk919b1d80) FROM tes
> On 19 Sep 2015, at 08:28, shane knapp wrote:
>
> TL; DR: jenkins is currently down and will probably not be brought
> back up until monday morning.
>
> a machine caught fire in the colo this evening, and this tripped the
> halon, and now IST is overheating... it looks like it may have been
TL; DR: jenkins is currently down and will probably not be brought
back up until monday morning.
a machine caught fire in the colo this evening, and this tripped the
halon, and now IST is overheating... it looks like it may have been
one of our servers that popped and caused the event, and thank
15 matches
Mail list logo
