[
https://issues.apache.org/jira/browse/HBASE-30414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil resolved HBASE-30414.
------------------------------------------
Release Note:
Follows Spark 4 Structured Streaming model to offer a declarative, catalog
based streaming service. Spark's engine discovers HBaseTableProvider via
ServiceLoader, finds STREAMING_WRITE capability, calls newWriteBuilder() to
obtain a HBaseStreamingWrite instance which in turn re-uses HBaseDataWriter for
the actual HBase writes. The catalog JSON replaces the client provided
transformation function.
In the spark3 module, streaming implemented the now deprecated DStream API, it
was a separate concern (HBaseDStreamFunctions in the spark package) layered on
top of HBaseContext and required client application to provide all the mapping
logic between streamed data and the schema. In spark4, streaming is just
another capability of the same Table interface that handles batch reads and
writes, provided with the datasource package as part of SparkSQL, leading to a
unified API for batch and streaming.
Resolution: Fixed
> Add streaming sink to spark4 streaming
> ---------------------------------------
>
> Key: HBASE-30414
> URL: https://issues.apache.org/jira/browse/HBASE-30414
> Project: HBase
> Issue Type: Task
> Reporter: Wellington Chevreuil
> Assignee: Wellington Chevreuil
> Priority: Major
> Labels: pull-request-available
>
> The spark3 module has streaming via the
> deprecated DStream/StreamingContext API (HBaseDStreamFunctions: hbaseBulkPut,
> hbaseBulkDelete,
> hbaseBulkGet, hbaseForeachPartition, hbaseMapPartitions).
>
> This adds the streaming sink (STREAMING_WRITE) to spark4 module.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
