Hi,
Does Spark MLLIB support Contextual Bandit? How can we use Spark MLLIB to
implement Contextual Bandit?
Thanks.
Best regards,
Ey-Chih
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
-
To unsubscribe e
Hi,
For a query against the LocalRelation, is there anybody know what does the
corresponding SQL looks like? Thanks.
Best regards,
Ey-Chih Chow
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/corresponding-sql-for-query-against-LocalRelation-tp26093
. What else I should do to fix the problem?
Ey-Chih Chow
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/IncompatibleClassChangeError-tp21934.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
have better performance for large input data? Thanks.
Ey-Chih Chow
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/how-to-improve-performance-of-spark-job-with-large-input-to-executor-tp21856.html
Sent from the Apache Spark User List mailing list archive
-20150209210707-0007/0 is:
spark-etl-0.0.1-SNAPSHOT.jar stderr stdout
Is there any way we can disallow the system to copy the jar file?
Ey-Chih Chow
To: 2dot7kelvin@gmail.comCC: gen.tan...@gmail.com; user@spark.apache.org
Subject: RE: no space left at worker node
Date: Mon, 9 Feb 2015 12:07:17
back with the following. What's
wrong with this?
Ey-Chih Chow
===
Date: Sun, 8 Feb 2015 22:27:17 -0800Sending launch command to
spark://ec2-54-213-73-150.us-west-2.compute.amazonaws.com:7077Driver
successfully submitted as driver-20150209185453-0010... waiting before polling
m
Thanks. But, in spark-submit, I specified the jar file in the form of
local:/spark-etl-0.0.1-SNAPSHOT.jar. It comes back with the following. What's
wrong with this?
Ey-Chih Chow
===
Date: Sun, 8 Feb 2015 22:27:17 -0800Sending launch command to
spark://ec2-54-213-73-150.us-w
Is there any way we can disable Spark copying the jar file to the corresponding
directory. I have a fat jar and is already copied to worker nodes using the
command copydir. Why Spark needs to save the jar to ./spark/work/appid each
time a job get started?
Ey-Chih Chow
Date: Sun, 8 Feb
, the input and output paths of the job are all in s3. I did not
use paths of hdfs as input or output.
Best regards,
Ey-Chih Chow
From: eyc...@hotmail.com
To: gen.tan...@gmail.com
CC: user@spark.apache.org
Subject: RE: no space left at worker node
Date: Sun, 8 Feb 2015 14:57:15 -0800
Hi Gen
By this way, the input and output paths of the job are all in s3. I did not
use paths of hdfs as input or output.
Best regards,
Ey-Chih Chow
From: eyc...@hotmail.com
To: gen.tan...@gmail.com
CC: user@spark.apache.org
Subject: RE: no space left at worker node
Date: Sun, 8 Feb 2015 14:57:15 -0800
Hi Gen,
Thanks. I save my logs in a file under /var/log. This is the only place to
save data. Will the problem go away if I use a better machine?
Best regards,
Ey-Chih Chow
Date: Sun, 8 Feb 2015 23:32:27 +0100
Subject: Re: no space left at worker node
From: gen.tan...@gmail.com
To: eyc
before hadoop starts.
-Mike
From: gen tang
To: ey-chih chow
Cc: "user@spark.apache.org"
Sent: Sunday, February 8, 2015 6:09 AM
Subject: Re: no space left at worker node
Hi,I fact, I met this problem before. it is a bug of AWS. Which type of machine
do you use?If I gues
Thanks Gen. How can I check if /dev/sdc is well mounted or not? In general,
the problem shows up when I submit the second or third job. The first job I
submit most likely will succeed.
Ey-Chih Chow
Date: Sun, 8 Feb 2015 18:18:03 +0100
Subject: Re: no space left at worker node
From: gen.tan
defaults0 0proc/proc
procdefaults0 0/dev/sdb/mntauto
defaults,noatime,nodiratime,comment=cloudconfig 0 0/dev/sdc/mnt2
autodefaults,noatime,nodiratime,comment=cloudconfig 0 0
There is no entry of /dev/xvdb.
Ey-Chih Chow
Date
/dev/xvdb 30963708 1729652 27661192 6% /mnt
Does anybody know how to fix this? Thanks.
Ey-Chih Chow
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/no-space-left-at-worker-node-tp21545.html
Sent from the Apache Spark User List mailing list
Hi,
I would like to submit spark jobs one by one, in that the next job will not
be submitted until the previous one succeeds. Spark_submit can only submit
jobs asynchronously. Is there any way I can submit jobs sequentially?
Thanks.
Ey-Chih Chow
--
View this message in context:
http
28/00,s3://pixlogstxt/ETL/output/2015/01/28/01"
But, instead, it seemed to me this was expanded to
"s3://pixlogstxt/ETL/output/2015/01/28/00","s3://pixlogstxt/ETL/output/2015/01/28/01"
Anybody knows what spark is doing this? Thanks.
Ey-Chih Chow
--
View this mess
I use the default value, which I think is 512MB. If I change to 1024MB, Spark
submit will fail due to not enough memory for rdd.
Ey-Chih Chow
From: moham...@glassbeam.com
To: eyc...@hotmail.com; user@spark.apache.org
Subject: RE: unknown issue in submitting a spark job
Date: Fri, 30 Jan 2015 00
The worker node has 15G memory, 1x32 GB SSD, and 2 core. The data file is from
S3. If I don't set mapred.max.split.size, it is fine with only one partition.
Otherwise, it will generate OOME.
Ey-Chih Chow
> From: moham...@glassbeam.com> To: eyc...@hotmail.com; user@spark.apache.or
Hi,
I submitted a job using spark-submit and got the following exception.
Anybody knows how to fix this? Thanks.
Ey-Chih Chow
15/01/29 08:53:10 INFO storage.BlockManagerMasterActor: Registering block
manager ip-10-10-8-191.us-west-2
file
> On Jan 26, 2015, at 8:57 PM, ey-chih chow wrote:
>
> Hi,
>
> I used the spark-ec2 script of spark 1.2 to launch a cluster. I have
> modified the script according to
>
> https://github.com/grzegorz-dubicki/spark/commit/5dd8458d2ab9753aae939b3bb33be9
ready'
state.
Any additional thing I should do to make it succeed? Thanks.
Ey-Chih Chow
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-ec2-launch-script-hang-tp21381.html
Sent from the Apache
wing that you are running an
> old Hadoop 1.x HDFS? there seem to be a number of possible
> incompatibilities here.
>
> On Fri, Jan 23, 2015 at 11:38 PM, ey-chih chow wrote:
> > Sorry I still did not quiet get your resolution. In my jar, there are
> > following three related cl
wing that you are running an
> old Hadoop 1.x HDFS? there seem to be a number of possible
> incompatibilities here.
>
> On Fri, Jan 23, 2015 at 11:38 PM, ey-chih chow wrote:
> > Sorry I still did not quiet get your resolution. In my jar, there are
> > following three related cl
?
Date: Fri, 23 Jan 2015 17:01:48 +
Subject: RE: spark 1.1.0 save data to hdfs failed
From: so...@cloudera.com
To: eyc...@hotmail.com
Are you receiving my replies? I have suggested a resolution. Look at the
dependency tree next.
On Jan 23, 2015 2:43 PM, "ey-chih chow" wrote:
I l
1.1.0 save data to hdfs failed
From: so...@cloudera.com
To: eyc...@hotmail.com
Are you receiving my replies? I have suggested a resolution. Look at the
dependency tree next.
On Jan 23, 2015 2:43 PM, "ey-chih chow" wrote:
I looked into the source code of SparkHadoopMapReduceUtil.scala.
;
> These are all definitely symptoms of mixing incompatible versions of
> libraries.
>
> I'm not suggesting you haven't excluded Spark / Hadoop, but, this is
> not the only way Hadoop deps get into your app. See my suggestion
> about investigating the dependency tree.
spark.apache.org
>
> These are all definitely symptoms of mixing incompatible versions of
> libraries.
>
> I'm not suggesting you haven't excluded Spark / Hadoop, but, this is
> not the only way Hadoop deps get into your app. See my suggestion
> about investigating the d
version has to match. You should also use mvn
> dependency:tree to see all the dependencies coming in. There may be
> many sources of a Hadoop dep.
>
> On Fri, Jan 23, 2015 at 1:05 AM, ey-chih chow wrote:
> > Thanks. But after I replace the maven de
netty
the warning message is still shown up in the namenode log. Is there any other
thing I need to do?
Thanks.
Ey-Chih Chow
> From: so...@cloudera.com
> Date: Thu, 22 Jan 2015 22:34:22 +
>
io.netty
netty
the warning message is still shown up in the namenode log. Is there any other
thing I need to do?
Thanks.
Ey-Chih Chow
> From:
...@gmail.com
CC: user@spark.apache.org
Subject: RE: spark 1.1.0 save data to hdfs failed
Date: Wed, 21 Jan 2015 23:12:56 -0800
The hdfs release should be hadoop 1.0.4.
Ey-Chih Chow
Date: Wed, 21 Jan 2015 16:56:25 -0800
Subject: Re: spark 1.1.0 save data to hdfs failed
From: yuzhih...@gmail.com
To
The hdfs release should be hadoop 1.0.4.
Ey-Chih Chow
Date: Wed, 21 Jan 2015 16:56:25 -0800
Subject: Re: spark 1.1.0 save data to hdfs failed
From: yuzhih...@gmail.com
To: eyc...@hotmail.com
CC: user@spark.apache.org
What hdfs release are you using ?
Can you check namenode log around time of
f[AvroKey[GenericRecord]],
classOf[NullWritable],
classOf[AvroKeyOutputFormat[GenericRecord]],
job.getConfiguration)
But it failed with the following error messages. Is there any people who
can help?
1.1.0 - spark-submit
failedFrom: yuzhih...@gmail.com
To: eyc...@hotmail.com
CC: user@spark.apache.org
Please check which netty jar(s) are on the classpath.
NioWorkerPool(Executor workerExecutor, int workerCount) was added in netty 3.5.4
Cheers
On Tue, Jan 20, 2015 at 4:15 PM, ey-chih chow wrote
how to fix the problem? Thanks.
Ey-Chih Chow
==
Launch Command: "/usr/lib/jvm/java-1.7.0/bin/java" "-cp"
"/root/spark/work/driver-20150120200843-/spark-etl-0.0.1-SNAPSHOT.jar/root/ephemeral-hdfs/conf:/root/spark/conf:/root
the /home/hadoop directory.
However, afterwords, ssh was no longer working.
Thanks.
Best regards,
Ey-Chih Chow
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/issue-with-spark-ec2-of-spark-release-1-1-0-tp21195.html
Sent from the Apache Spark User List ma
I should rephrase my question as follows:
How to use the corresponding Hadoop Configuration of a HadoopRDD in defining
a function as an input parameter to the MapPartitions function?
Thanks.
Ey-Chih Chow
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com
rtitions will generate the
following message?
Cause: java.io.NotSerializableException: org.apache.hadoop.mapreduce.Job
If I take out 'val config = job.getConfiguration()' in the mapPartitions,
the code works fine, even through
job.getConfiguration() shows up also in newAPIHadoopFile().
Ey-Ch
It's working now. Probably I didn't specify the excluded list correctly. I
kept revising it and now it's working. Thanks.
Ey-Chih Chow
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Debugging-a-Spark-application-using-Eclipse-throws-Se
h5.2.0
provided
javax.servlet
*
I appreciate if somebody can help me to identify the issue.
Best regards,
Ey-Chih Chow
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Debugging-a-Spark-application-using-Eclipse-throws-SecurityException-tp20843.html
Sent from t
41 matches
Mail list logo