Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/148#issuecomment-37752740
Currently, Spark doesn't ship a log4j.properties. It uses the
log4j.properties that comes from Hadoop. This log4j.properties is meant for
Hadoop services, not
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/148
SPARK-1252. On YARN, use container-log4j.properties for executors
container-log4j.properties is a file that YARN provides so that containers
can have log4j.properties distinct from that of the
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/138#issuecomment-37613741
Thanks Patrick
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/138#issuecomment-37609081
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/86#issuecomment-37601581
Yeah, workin on it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/120#issuecomment-37564441
Updated patch with feedback
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user sryza closed the pull request at:
https://github.com/apache/spark/pull/90
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/86#discussion_r10554729
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/120#discussion_r10554025
--- Diff:
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
---
@@ -133,11 +148,11 @@ class ClientArguments(val args: Array[String
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/120#issuecomment-37491980
Updated the patch to fix things on yarn-alpha. I verified it on a
pseudo-distributed cluster with 0.23.7.
---
If your project is set up for it, you can reply to this
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/120#issuecomment-37477894
The goal isn't to replace Spark's names with YARN's names, but rather to be
consistent with the terminology used in the rest of Spark. "Master"
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/130#discussion_r10530856
--- Diff: docs/running-on-yarn.md ---
@@ -99,16 +99,16 @@ With this mode, your application is actually run on the
remote machine where the
## Launch
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/119#discussion_r10528875
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -130,6 +130,16 @@ class SparkContext(
val isLocal = (master == "
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/129#issuecomment-37434030
+1. Sorry again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/127#issuecomment-37426702
Oh, you beat me to it. +1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/91#discussion_r10524194
--- Diff: core/pom.xml ---
@@ -17,274 +17,260 @@
-->
http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSc
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/120#issuecomment-37383673
Updated patch addresses Patrick's comments
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pr
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/120#discussion_r10508210
--- Diff: docs/running-on-yarn.md ---
@@ -60,11 +60,11 @@ The command to launch the Spark application on the
cluster is as follows:
--jar
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/119#discussion_r10507644
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -130,6 +130,16 @@ class SparkContext(
val isLocal = (master == "
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/86#issuecomment-37381283
Updated patch takes review comments form @mridulm and @pwendell into
account.
spark.max.cores is now correctly handled. Jars passed in with --more-jars
are not
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/119#discussion_r10506355
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -130,6 +130,16 @@ class SparkContext(
val isLocal = (master == "
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/119#discussion_r10506306
--- Diff: docs/configuration.md ---
@@ -393,6 +394,16 @@ Apart from these, the following properties are also
available, and may be useful
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/119#discussion_r10506244
--- Diff: docs/configuration.md ---
@@ -375,7 +376,7 @@ Apart from these, the following properties are also
available, and may be useful
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/119#discussion_r10506248
--- Diff: docs/configuration.md ---
@@ -430,7 +441,7 @@ Apart from these, the following properties are also
available, and may be useful
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/119#discussion_r10506241
--- Diff: docs/configuration.md ---
@@ -111,6 +111,7 @@ Apart from these, the following properties are also
available, and may be useful
it if you
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/119#discussion_r10506221
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -726,6 +736,8 @@ class SparkContext(
* Adds a JAR dependency for all tasks to be
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37374693
Unfortunately, the alpha/stable distinction doesn't fully capture the
differences here because the APIs are different between the 0.23 Hadoop line
and the 2.0 line, bo
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10505297
--- Diff:
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala ---
@@ -379,9 +381,48 @@ object ClientBase {
// Based on code
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37371001
Updated patch takes out the SBT stuff, adds the comment requested by
@pwendell, and uses reflection to work around the incompatibilities pointed out
by @gzm55
---
If your
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10478785
--- Diff:
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala ---
@@ -379,7 +381,17 @@ object ClientBase {
// Based on code
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/120
SPARK-1183. Don't use "worker" to mean executor
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sryza/spark sandy-spark-1183
Al
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10460757
--- Diff: project/SparkBuild.scala ---
@@ -236,7 +236,8 @@ object SparkBuild extends Build {
"com.novocode" % "junit-int
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10459998
--- Diff: project/SparkBuild.scala ---
@@ -236,7 +236,8 @@ object SparkBuild extends Build {
"com.novocode" % "junit-int
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10458903
--- Diff:
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala ---
@@ -379,7 +381,17 @@ object ClientBase {
// Based on code
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10458859
--- Diff: project/SparkBuild.scala ---
@@ -236,7 +236,8 @@ object SparkBuild extends Build {
"com.novocode" % "junit-int
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/118
SPARK-1211. In ApplicationMaster, set spark.master system property to "y...
...arn-cluster"
You can merge this pull request into a Git repository by running:
$ git pull https://github
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37154606
Updated version works with sbt as well
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/102
SPARK-1064
This reopens PR 649 from incubator-spark against the new repo
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sryza/spark sandy-spark
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/91#issuecomment-37089032
Upmerged
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/86#issuecomment-37088962
Newest patch includes tests and doc. @pwendell, do you have a link to the
addJar patch? If it's definitely going to happen, I'll take out the
classloader
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/100#issuecomment-37085469
That only works when getting Hadoop dependencies by packaging everything in
an uber jar. On yarn, for example, the Hadoop jars and dependencies are pulled
in by pointing
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/100#issuecomment-37085083
Newer Hadoop versions pull in ASM 3.1 (through Jersey)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/100#issuecomment-37082518
This patch looks good to me. ASM still needs to be excluded from chill,
right?
---
If your project is set up for it, you can reply to this email and have your
reply
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/86#discussion_r10399046
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/86#discussion_r10397696
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/86#discussion_r10397547
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/86#discussion_r10397541
--- Diff: bin/spark-submit ---
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/86#issuecomment-36966736
I uploaded a new patch that doesn't start a new JVM and parses
--driver-memory in bash. It wasn't as bad as I expected (thanks to some help
from @umbran
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/95#issuecomment-36965043
If there's consensus on a different identifier, I'd be happy to post an
addendum patch.
---
If your project is set up for it, you can reply to this email and
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/95#issuecomment-36956664
Updated patch incorporates review feedback
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/30#issuecomment-36953787
Updated to 1.0.0 and removed incubating
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/30#issuecomment-36953590
You need to run make inside the python directory first. Did you do that?
(This obviously needs to be documented).
---
If your project is set up for it, you can reply to
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/86#issuecomment-36937078
I uploaded a new patch that takes most of the review feedback into account.
Includes the following changes:
* changes Opt to OptionAssigner and uses default parameters
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/86#discussion_r10360322
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkAppArguments.scala ---
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/86#discussion_r10359569
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApp.scala ---
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/86#discussion_r10359483
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApp.scala ---
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/95
Change yarn-standalone to yarn-cluster and fix up running on YARN docs
This patch changes "yarn-standalone" to "yarn-cluster" (but still supports
the former). It also cleans up
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/86#issuecomment-36828689
Thanks for taking a look, Matei. If we use system properties instead of
env variables, the remaining reason we'd want to start a second JVM is to be
able to have a --d
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/90#issuecomment-36827609
Oy, good point. Ok, I'll try that out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/91
SPARK-1193. Fix indentation in pom.xmls
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sryza/spark sandy-spark-1193
Alternatively you can review
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/90
SPARK-782. Shade ASM
This solves the issue of Spark's version of ASM conflicting with the ASM
versions of its transitive dependencies (e.g. Hadoop). The ASM packages, with
their package
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/86#discussion_r10331365
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkAppArguments.scala ---
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/86#discussion_r10331320
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApp.scala ---
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/86#discussion_r10331089
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApp.scala ---
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/86
SPARK-1126. spark-app preliminary
This is a starting version of the spark-app script for running compiled
binaries against Spark. It still needs tests and some polish. The only
testing I've do
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/29#issuecomment-36372574
Uploaded a rebased patch
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/28#issuecomment-36368855
Thanks Tom!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/29#issuecomment-36311392
Oops, actually pushed it this time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/29#issuecomment-36293609
No problem. I attached a new patch that does this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/29#issuecomment-36275981
I'm not familiar with what the secrets are used for as opposed to the
tokens. Do you know the reason they're not needed in the MR1 code? Either wa
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/30
SPARK-1004. PySpark on YARN
This reopens https://github.com/apache/incubator-spark/pull/640 against the
new repo
You can merge this pull request into a Git repository by running:
$ git pull
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/29
SPARK-1051. On YARN, executors don't doAs submitting user
This reopens https://github.com/apache/incubator-spark/pull/538 against the
new repo
You can merge this pull request into a Git reposito
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/28
SPARK-1032. If Yarn app fails before registering, app master stays aroun...
...d long after
This reopens https://github.com/apache/incubator-spark/pull/648 against the
new repo.
You can merge
73 matches
Mail list logo
