Re: [DISCUSS] Versionless Spark Programming Guide Proposal

2024-06-04 Thread Nimrod Ofek
Hi Neil, While you wrote you don't mean the api docs (of course), the programming guides are also different between versions since features are being added, configs are being added/ removed/ changed, defaults are being changed etc. I know of "backport hell" - which is why I wrote that once a ver

Re: [DISCUSS] Versionless Spark Programming Guide Proposal

2024-06-04 Thread Neil Ramaswamy
Hi Nimrod, Quick clarification—my proposal will not touch API-specific documentation for the specific reasons you mentioned (signatures, behavior, etc.). It just aims to make the *programming guides *versionless. Programming guides should teach fundamentals of Spark, and the fundamentals of Spark

Re: [DISCUSS] Versionless Spark Programming Guide Proposal

2024-06-04 Thread Nimrod Ofek
Hi, While I think that the documentation needs a lot of improvement and important details are missing - and detaching the documentation from the main project can help iterating faster on documentation specific tasks, I don't think we can nor should move to versionless documentation. Documentation

Re: [DISCUSS] Versionless Spark Programming Guide Proposal

2024-06-04 Thread Praveen Gattu
+1. This helps for greater velocity in improving docs. However, we might still need a way to provide version specific information isn't it, i.e. what features are available in which version etc. On Mon, Jun 3, 2024 at 3:08 PM Neil Ramaswamy wrote: > Hi all, > > I've written up a proposal to migr