[
https://issues.apache.org/jira/browse/SPARK-57365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-57365:
-----------------------------------
Labels: pull-request-available (was: )
> Fix flaky KafkaRelationSuite resolved-offset test by using distinct message
> timestamps
> --------------------------------------------------------------------------------------
>
> Key: SPARK-57365
> URL: https://issues.apache.org/jira/browse/SPARK-57365
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.1.0
> Reporter: Iván Morales
> Priority: Minor
> Labels: pull-request-available
>
> The test "resolved start offset greater than end offset (without latest)" in
> KafkaRelationSuite is flaky. It produces two messages per partition without
> explicit timestamps and expects offsetsForTimes to resolve to offset 1.
> When both messages land in the same millisecond they share a CreateTime
> timestamp, offsetsForTimes resolves to offset 0, and the hardcoded "resolved
> end offset 1" assertion fails. he existing eventually(60.seconds) does not
> help: produced timestamps are fixed at produce time, so retrying always
> resolves the same offset.
> Introduced by SPARK-52096 (commit 2b5cd7dc59, 2025-05-15).
> Affects master, 4.1.x, 4.2.x.
> Fix: produce the two messages with explicit increasing timestamps (base,
> base+1)
> via RecordBuilder so the second message deterministically resolves to offset
> 1.
> Test-only change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
