Hi,
you can get the same with:
import org.apache.spark.sql.functions._
import sqlContext.implicits._
import org.apache.spark.sql.types.{IntegerType, StringType, StructField,
StructType}
val schema = StructType(Array(StructField("name", StringType),
StructField("age", IntegerType) ))
val lst = List(Row("Shmuel", 13), Row("Blitz", 23))
val rdd = sc.parallelize(lst)
val df = sqlContext.createDataFrame(rdd,schema)
df.withColumn("newName", concat($"name" , lit("abc")) ).show()
On Mon, Mar 26, 2018 at 6:36 AM, 1427357...@qq.com <1427357...@qq.com>
wrote:
> Hi all,
>
> I have a table like below:
>
> +---+---------+-----------+
> | id| name|sharding_id|
> +---+---------+-----------+
> | 1|leader us| 10000|
> | 3| mycat| 10000|
> +---+---------+-----------+
>
> My schema is :
> root
> |-- id: integer (nullable = false)
> |-- name: string (nullable = true)
> |-- sharding_id: integer (nullable = false)
>
> I want add a new column named newName. The new column is based on "name"
> and append "abc" after it. My code looks like:
>
> stud_scoreDF.withColumn("newName", stud_scoreDF.col("name") + "abc" ).show()
>
> When I run the code, I got the reslult:
> +---+---------+-----------+-------+
> | id| name|sharding_id|newName|
> +---+---------+-----------+-------+
> | 1|leader us| 10000| null|
> | 3| mycat| 10000| null|
> +---+---------+-----------+-------+
>
>
> I checked the code, the key code is in arithmetic.scala. line 165.
> It looks like:
>
> override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode =
> dataType match {
> case dt: DecimalType =>
> defineCodeGen(ctx, ev, (eval1, eval2) => s"$eval1.$$plus($eval2)")
> case ByteType | ShortType =>
> defineCodeGen(ctx, ev,
> (eval1, eval2) => s"(${ctx.javaType(dataType)})($eval1 $symbol $eval2)")
> case CalendarIntervalType =>
> defineCodeGen(ctx, ev, (eval1, eval2) => s"$eval1.add($eval2)")
> case _ =>
> defineCodeGen(ctx, ev, (eval1, eval2) => s"$eval1 $symbol $eval2")
> }
>
>
> My issue is:
> Can we add case StringType in this class to support string append please?
>
>
>
> ------------------------------
> 1427357...@qq.com
>
--
Shmuel Blitz
Big Data Developer
Email: shmuel.bl...@similarweb.com
www.similarweb.com
<https://www.facebook.com/SimilarWeb/>
<https://www.linkedin.com/company/429838/> <https://twitter.com/similarweb>
