Issues with Spark Streaming and Manual Clock used for Unit Tests

2015-06-02 Thread mobsniuk
ating checkpoint data for time 4000 ms 15/06/02 10:39:38 INFO DStreamGraph: Updated checkpoint data for time 4000 ms 15/06/02 10:39:38 INFO CheckpointWriter: Saving checkpoint for time 2000 ms to file 'file:/Users/mobsniuk/perforce/IB/proj/analytics-dnst/platform/checkpoint/checkpoint-2000' 15/06/0

Converting SchemaRDD/Dataframe to RDD[vector]

2015-02-26 Thread mobsniuk
I've been searching around and see others have asked similar questions. Given a schemaRDD I extract a restless that contains numbers, both Int and Doubles. How do I construct a RDD[Vector]? In 1.2 I wrote the results to a textile and then read them back in splitting them with some code I found in