So IMO maintaining outside in a separate repo is going to be harder. That was why I asked.
________________________________ From: Maciej Szymkiewicz <mszymkiew...@gmail.com> Sent: Tuesday, August 4, 2020 12:59 PM To: Sean Owen Cc: Felix Cheung; Hyukjin Kwon; Driesprong, Fokko; Holden Karau; Spark Dev List Subject: Re: [PySpark] Revisiting PySpark type annotations On 8/4/20 9:35 PM, Sean Owen wrote > Yes, but the general argument you make here is: if you tie this > project to the main project, it will _have_ to be maintained by > everyone. That's good, but also exactly I think the downside we want > to avoid at this stage (I thought?) I understand for some > undertakings, it's just not feasible to start outside the main > project, but is there no proof of concept even possible before taking > this step -- which more or less implies it's going to be owned and > merged and have to be maintained in the main project. I think we have a bit different understanding here ‒ I believe we have reached a conclusion that maintaining annotations within the project is OK, we only differ when it comes to specific form it should take. As of POC ‒ we have stubs, which have been maintained over three years now and cover versions between 2.3 (though these are fairly limited) to, with some lag, current master. There is some evidence there are used in the wild (https://github.com/zero323/pyspark-stubs/network/dependents?package_id=UGFja2FnZS02MzU1MTc4Mg%3D%3D), there are a few contributors (https://github.com/zero323/pyspark-stubs/graphs/contributors) and at least some use cases (https://stackoverflow.com/q/40163106/). So, subjectively speaking, it seems we're already beyond POC. -- Best regards, Maciej Szymkiewicz Web: https://zero323.net Keybase: https://keybase.io/zero323 Gigs: https://www.codementor.io/@zero323 PGP: A30CEF0C31A501EC
