Sorry for the late response. With the amount of data you are planning join, any system would take time. However, between Hive's MapRduce joins, and Spark's basic shuffle, and Spark SQL's join, the latter wins hands down. Furthermore, with the APIs of Spark and Spark Streaming, you will have to do strictly less work to set the infrastructure that you want to build.
Yes, Spark Streaming currently does not support providing own timer, because the logic to handle delays etc, is pretty complex and specific to each application. Usually that logic can be implemented on top of the windowing solutoin that Spark Streaming already provides. TD On Thu, Feb 5, 2015 at 7:37 AM, Zilvinas Saltys <zilvinas.sal...@gmail.com> wrote: > The challenge I have is this. There's two streams of data where an event > might look like this in stream1: (time, hashkey, foo1) and in stream2: > (time, hashkey, foo2) > The result after joining should be (time, hashkey, foo1, foo2) .. The join > happens on hashkey and the time difference can be ~30 mins between events. > The amount of data is enormous .. hundreds of billions of events per > month. I need not only join the existing history data but continue to do so > with incoming data (comes in batches not really streamed) > > For now I was thinking to implement this in MapReduce and sliding windows > .. I'm wondering if spark can actually help me with this sort of challenge? > How would a join of two huge streams of historic data would actually > happen internally within spark and would it be more efficient than let's > say hive map reduce stream join of two big tables? > > I also saw spark streaming has windowing support but it seems you cannot > provide your own timer? As in I cannot make the time be derived from events > itself rather than having an actual clock running. > > Thanks, >
