This is working! Thank you so much :) Stefan Panayotov, PhD Home: 610-355-0919 Cell: 610-517-5586 email: [email protected] [email protected] [email protected]
From: [email protected] Date: Tue, 21 Jul 2015 12:08:04 -0700 Subject: Re: Add column to DF To: [email protected] CC: [email protected] Try instead: import org.apache.spark.sql.functions._ val determineDayPartID = udf((evntStDate: String, evntStHour: String) => { val stFormat = new java.text.SimpleDateFormat("yyMMdd") var stDateStr:String = evntStDate.substring(2,8) val stDate:Date = stFormat.parse(stDateStr) val stHour = evntStHour.substring(1,3).toDouble + 0.1 var bucket = Math.ceil(stHour/3.0).toInt val cal:Calendar = Calendar.getInstance cal.setTime(stDate) var dayOfWeek = cal.get(Calendar.DAY_OF_WEEK) if (dayOfWeek == 1) dayOfWeek = 8 if (dayOfWeek > 6) bucket = bucket + 8 bucket }) input.withColumn("DayPartID", determineDayPartID (col("StartDate"), col("EventStartHour")))
