Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2

[russell . spitzer]

Exactly once should be output sink dependent, what sink was being used? Sent from my iPhone On Sep 14, 2023, at 4:52 PM, Jerry Peng <jerry.boyang.p...@gmail.com> wrote:  Craig, Thanks! Please let us know the result! Best, Jerry On Thu, Sep 14, 2023 at 12:22 PM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: Hi Craig, Can you please clarify what this bug is and provide sample code causing this issue? HTH  Mich Talebzadeh, Distinguished Technologist, Solutions Architect & Engineer London United Kingdom    view my Linkedin profile  https://en.everybodywiki.com/Mich_Talebzadeh   Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.   On Thu, 14 Sept 2023 at 17:48, Craig Alfieri <craig.alfi...@antithesis.com> wrote: Hello Spark Community-   As part of a research effort, our team here at Antithesis tests for correctness/fault tolerance of major OSS projects. Our team recently was testing Spark’s Structured Streaming, and we came across a data duplication bug we’d like to work with the teams on to resolve.   Our intention is to utilize this as a future case study for our platform, but prior to doing so we like to have a resolution in place so that an announcement isn’t alarming to the user base.   Attached is a high level .pdf that reviews the High Availability set-up put under test. This was also tested across the three latest versions, and the same behavior was observed.   We can reproduce this error readily, since our environment is fully deterministic, we are just not Spark experts and would like to work with someone in the community to resolve this.   Please let us know at your earliest convenience.   Best   Craig Alfieri c: 917.841.1652 craig.alfi...@antithesis.com New York, NY. Antithesis.com   We can't talk about most of the bugs that we've found for our customers, but some customers like to speak about their work with us: https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis     ----------------------------- This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity for whom they are addressed. If you received this message in error, please notify the sender and remove it from your system. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org