Re: When Spark job shows FetchFailedException it creates few duplicate data and we see few data also missing , please explain why

thanks Mich, in a nutshell if fetchFailedException occurs due to data node reboot then it can create duplicate / missing data . so this is more of hardware(env issue ) rather than spark issue . On Sat, Mar 2, 2024 at 7:45 AM Mich Talebzadeh wrote: > Hi, > > It seems to me that there are i

Re: [ANNOUNCE] Apache Spark 3.5.1 released

Shall we revisit this functionality? The API doc is built with individual versions, and for each individual version we depend on other released versions. This does not seem to be right to me. Also, the functionality is only in PySpark API doc which does not seem to be consistent as well. I don't t