[
https://issues.apache.org/jira/browse/SPARK-17814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-17814:
------------------------------
Priority: Minor (was: Major)
OK, yeah with quoting fixed, I think there's still an issue here. I think the
issue is that with cluster mode, these args become an argument again to a new
command line, and although it's escaped again, it seems something goes funny
with curly braces.
In cluster mode,
{code}
spark2-submit --verbose --master yarn --deploy-mode cluster --class
org.apache.spark.examples.SparkPi
/opt/cloudera/parcels/SPARK2/lib/spark2/examples/jars/spark-examples_2.11-2.0.0.cloudera.beta1.jar
"{\"mode\":\"wf\", \"arrays\":{\"array\":[1]}}"
...
childArgs [{"mode":"wf", "arrays":{"array":[1]}}]
...
--arg
{"mode":"wf", "arrays":{"array":[1]}}
{code}
I end up with the error ...
{code}
java.lang.NumberFormatException: For input string: "{"mode":"wf",
"arrays":{"array":[1]"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at
scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272)
at scala.collection.immutable.StringOps.toInt(StringOps.scala:29)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:32)
...
{code}
Obviously a NFE is expected here, but, the input string it fails on is not
expected. It seems specific to braces, but not sure why that is.
> spark submit arguments are truncated in yarn-cluster mode
> ---------------------------------------------------------
>
> Key: SPARK-17814
> URL: https://issues.apache.org/jira/browse/SPARK-17814
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit, YARN
> Affects Versions: 1.6.1
> Reporter: shreyas subramanya
> Priority: Minor
>
> {noformat}
> One of our customers is trying to pass in json through spark-submit as
> follows:
> spark-submit --verbose --class SimpleClass --master yarn-cluster ./simple.jar
> "{\"mode\":\"wf\", \"arrays\":{\"array\":[1]}}"
> The application reports the passed arguments as: {"mode":"wf",
> "arrays":{"array":[1]
> If the same application is submitted in yarn-client mode, as follows:
> spark-submit --verbose --class SimpleClass --master yarn-client ./simple.jar
> "{\"mode\":\"wf\", \"arrays\":{\"array\":[1]}}"
> The application reports the passed args as: {"mode":"wf",
> "arrays":{"array":[1]}}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
