Jungtaek Lim created SPARK-55732:
------------------------------------
Summary: StateStore.putList to receive Iterator (non-materialized
list of data)
Key: SPARK-55732
URL: https://issues.apache.org/jira/browse/SPARK-55732
Project: Spark
Issue Type: Task
Components: Structured Streaming
Affects Versions: 4.2.0
Reporter: Jungtaek Lim
Currently, StateStore.putList() takes Array[UnsafeRow] as the value list, which
enforces the values in the array to be materialized. This is particularly
problematic when the elements of the array are generated from the same
UnsafeProjection, which require the caller of putList() to copy all UnsafeRows
to store to the array before calling the method.
We should have an overloaded version of putList() which takes Iterator or Seq
to avoid enforcing copying UnsafeRows for the above case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
