Jags,
Thanks for the details. This makes things much clearer. I saw in the Spark
roadmap that version 2.1 will add the SQL capabilities mentioned here. It looks
like, gradually, the Spark community is coming to the same conclusions that the
SnappyData folks have come to a while back in terms of
The plan is to fully integrate with the new structured streaming API and
implementation in an upcoming release. But, we will continue offering
several extensions. Few noted below ...
- the store (streaming sink) will offer a lot more capabilities like
transactions, replicated tables, partitioned r
Jags,
I should have been more specific. I am referring to what I read at
http://snappydatainc.github.io/snappydata/streamingWithSQL/, especially the
Streaming Tables part. It roughly coincides with the Streaming DataFrames
outlined here
https://docs.google.com/document/d/1NHKdRSNCbCmJbinLmZuqN
Ben,
Note that Snappydata's primary objective is to be a distributed
in-memory DB for mixed workloads (i.e. streaming with transactions and
analytic queries). On the other hand, Spark, till date, is primarily
designed as a processing engine over myriad storage engines (SnappyData
being one). So,
I recently got a sales email from SnappyData, and after reading the
documentation about what they offer, it sounds very similar to what Structured
Streaming will offer w/o the underlying in-memory, spill-to-disk, CRUD
compliant data storage in SnappyData. I was wondering if Structured Streaming
