[
https://issues.apache.org/jira/browse/IGNITE-9357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947450#comment-16947450
]
Alexey Kukushkin commented on IGNITE-9357:
------------------------------------------
[~zaleslaw], that was only a proof-of-concept (PoC). The solution is still far
from being production-ready. The items that must be implemented before we can
merge it are:
# The PoC supports only SQL-enabled caches with an incremental field: Spark
maintains "start" and "end" offsets depending on the streaming mode ("append"
or "aggregate") and passes the offsets to Ignite. Right now, the patch only
works with incremental and timestamp fields. I think we need to somehow support
any cache before we merge it.
# Poor performance: Spark works using “micro-batches.” Currently the patch
uses a naive approach of just firing the SQL ("SELECT ... FROM ... WHERE offset
>= start AND offset < end") at Ignite. This doesn’t scale. It would be better
to use a continuous query.
# Work that would have to be done to get to a point where it could be merged
into Ignite: cleaning up the code, documenting it, etc.
Unfortunately I do not have time to complete it in near future. Please feel
free to take the task and complete it.
> Spark Structured Streaming with Ignite as data source and sink
> --------------------------------------------------------------
>
> Key: IGNITE-9357
> URL: https://issues.apache.org/jira/browse/IGNITE-9357
> Project: Ignite
> Issue Type: New Feature
> Components: spark
> Affects Versions: 2.9
> Reporter: Alexey Kukushkin
> Assignee: Alexey Kukushkin
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We are working on a PoC where we want to use Ignite as a data storage and
> Spark as a computation engine. We found that Ignite is supported neither as a
> source nor as a Sink when using Spark Structured Streaming, which is a must
> for us.
> We are enhancing Ignite to support Spark streaming with Ignite. We will send
> docs and code for review for the Ignite Community to consider if the
> community wants to accept this feature.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
