Hello,
While the Arrow IPC format is primarily meant for RecordBatch transport,
it also has dedicated messages for transporting N-dimensional Tensor and
SparseTensor data.
The Tensor message was added in 2017
(https://github.com/apache/arrow/pull/435), while the SparseTensor
message was added in 2019 (https://github.com/apache/arrow/pull/3651).
Both have not had any significant additions since 2020. Furthermore:
* They are still marked experimental.
* They are not exercised by the the cross-implementation integration
test suite.
* Convenience IPC reading APIs are generally architected around
RecordBatch iteration, requiring more manual plumbing to read Tensor and
SparseTensor messages from an IPC stream.
* They are generally not reachable from Flight RPC APIs, or other
high-level APIs such the C++ Dataset API.
* A code search on GitHub reveals no apparent third-party usage of these
message types.
Nowadays, Arrow has canonical extension types for fixed-shape and
variable-shaped tensors which allow passing (dense) N-dimensional
tensors as regular RecordBatch columns. We could also add canonical
extension types for the different kinds of sparse tensors, if there is
demand for it.
Despite not seeing much usage, these message types carry their
maintenance baggage. Recently, we've had a number of security reports
around the lack of validation of untrusted Tensor and SparseTensor
messages in Arrow C++, requiring core developer intervention to analyze
and fix those issues.
I would like to suggest that we start deprecating these messages types
in the IPC format, and that we recommend deprecating the corresponding
APIs in all major Arrow implementations. What do you think?
Regards
Antoine.
