[
https://issues.apache.org/jira/browse/SPARK-48199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar resolved SPARK-48199.
----------------------------------------
Resolution: Fixed
> [Arbitrary State Support] Allow users to set EventTime timestamp properly in
> TransformWithState for watermark evaluation
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-48199
> URL: https://issues.apache.org/jira/browse/SPARK-48199
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 4.0.0
> Reporter: Bhuwan Sahni
> Priority: Major
>
> TransformWithState operator allows users to provide a eventTimeColumn. This
> column is used for watermark calculation in any operation done after
> transformWithState.
> Spark currently validates that the rows emitted from transformWithState
> operator adhere to the watermark boundary (specifically,
> watermarkForLateEvents). However, the user currently does not have a
> mechanism to obtain the watermarkForLateEvents.
> We need to either 1) introduce a method to users which gives a watermark
> value before advancing (late events) or 2) provide a mechanism for users to
> set the event time timestamp properly without watermark value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
