Re: Use case question

2014-11-24 Thread Gordon Benjamin
Great thanks On Monday, November 24, 2014, Akhil Das wrote: > I'm not quiet sure if i understood you correctly, but here's the thing, if > you use sparkstreaming, it is more likely to refresh your dashboard for > each batch. So for every batch your dashboard will be updated with the new > data.

Re: Use case question

2014-11-24 Thread Akhil Das
I'm not quiet sure if i understood you correctly, but here's the thing, if you use sparkstreaming, it is more likely to refresh your dashboard for each batch. So for every batch your dashboard will be updated with the new data. And yes, the end use won't feel anything while you do the coalesce/repa

Re: Use case question

2014-11-24 Thread Gordon Benjamin
Thanks. Yes d3 ones. Just to clarify--we could take our current system, which is incrementally adding partitions and overlay an Apache streaming layer to ingest these partitions? Then nightly, we could coalesce these partitions for example? I presume that while we are carrying out a coalesce, the e

Re: Use case question

2014-11-24 Thread Akhil Das
Streaming would be easy to implement, all you have to do is to create the stream, do some transformation (depends on your usecase) and finally write it to your dashboards backend. What kind of dashboards are you building? For d3.js based ones, you can have websocket and write the stream output to t