61810...@qq.com>;
"dev"; "user";
: Re: Time is ugly in Spark Streaming
Java's SimpleDateFormat is not thread safe. You can consider using
DateTimeFormatter if you are using Java 8 or Joda-time
On Sat, Jun 27, 2015 at 3:32 AM, Tathagata Das wrote:
Coul
Java's SimpleDateFormat is not thread safe. You can consider using
DateTimeFormatter if you are using Java 8 or Joda-time
On Sat, Jun 27, 2015 at 3:32 AM, Tathagata Das wrote:
> Could you print the "time" on the driver (that is, in foreachRDD but
> before RDD.foreachPartition) and see if it is
Could you print the "time" on the driver (that is, in foreachRDD but before
RDD.foreachPartition) and see if it is behaving weird?
TD
On Fri, Jun 26, 2015 at 3:57 PM, Emrehan Tüzün
wrote:
>
>
>
>
> On Fri, Jun 26, 2015 at 12:30 PM, Sea <261810...@qq.com> wrote:
>
>> Hi, all
>>
>> I find a probl
On Fri, Jun 26, 2015 at 12:30 PM, Sea <261810...@qq.com> wrote:
> Hi, all
> I find a problem in spark streaming, when I use the time in function
> foreachRDD... I find the time is very interesting.
> val messages = KafkaUtils.createDirectStream[String, String, StringDecoder,
> StringDecoder](s
Yes, I make it.
-- --
??: "Gerard Maas";;
: 2015??6??26??(??) 5:40
??: "Sea"<261810...@qq.com>;
: "user"; "dev";
: Re: Time is ugly in Spark Streaming
Are you shari
Are you sharing the SimpleDateFormat instance? This looks a lot more like
the non-thread-safe behaviour of SimpleDateFormat (that has claimed many
unsuspecting victims over the years), than any 'ugly' Spark Streaming. Try
writing the timestamps in millis to Kafka and compare.
-kr, Gerard.
On Fri,
Hi, all
I find a problem in spark streaming, when I use the time in function
foreachRDD... I find the time is very interesting.
val messages = KafkaUtils.createDirectStream[String, String, StringDecoder,
StringDecoder](ssc, kafkaParams, topicsSet)
dataStream.map(x => createGroup(x._2,
dimensio
