[jira] [Created] (HIVE-9040) Spark Memory can be formatted string

[Brock Noland (JIRA)]

Brock Noland created HIVE-9040:
----------------------------------

             Summary: Spark Memory can be formatted string
                 Key: HIVE-9040
                 URL: https://issues.apache.org/jira/browse/HIVE-9040
             Project: Hive
          Issue Type: Sub-task
            Reporter: Brock Noland


Here: 
https://github.com/apache/hive/blob/spark/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java#L72

we call {{getInt}} on {{spark.executor.memory}} but this is a formatted string, 
example here: http://spark.apache.org/docs/1.0.1/configuration.html as such, I 
get:

{noformat}
2014-12-08 03:04:48,114 WARN  [HiveServer2-Handler-Pool: Thread-34]: 
spark.SetSparkReducerParallelism (SetSparkReducerParallelism.java:process(141)) 
- Failed to create spark client.
java.lang.NumberFormatException: For input string: "23000m"
        at 
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Integer.parseInt(Integer.java:492)
        at java.lang.Integer.parseInt(Integer.java:527)
        at 
scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)
        at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
        at 
org.apache.spark.SparkConf$$anonfun$getInt$2.apply(SparkConf.scala:184)
        at 
org.apache.spark.SparkConf$$anonfun$getInt$2.apply(SparkConf.scala:184)
        at scala.Option.map(Option.scala:145)
        at org.apache.spark.SparkConf.getInt(SparkConf.scala:184)
        at 
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.getMemoryAndCores(SparkSessionImpl.java:72)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)