Looking at it a second time, I think it is only mypy that's complaining:
MacBook-Pro-van-Fokko:spark fokkodriesprong$ git diff
*diff --git a/python/pyspark/accumulators.pyi
b/python/pyspark/accumulators.pyi*
*index f60de25704..6eafe46a46 100644*
*--- a/python/pyspark/accumulators.pyi*
*+++ b/python/pyspark/accumulators.pyi*
@@ -30,7 +30,7 @@ U = TypeVar("U", bound=SupportsIAdd)
import socketserver as SocketServer
-_accumulatorRegistry: Dict = {}
+# _accumulatorRegistry: Dict = {}
class Accumulator(Generic[T]):
aid: int
MacBook-Pro-van-Fokko:spark fokkodriesprong$ ./dev/lint-python
starting python compilation test...
python compilation succeeded.
starting pycodestyle test...
pycodestyle checks passed.
starting flake8 test...
flake8 checks passed.
starting mypy test...
mypy checks failed:
python/pyspark/worker.py:34: error: Module 'pyspark.accumulators' has no
attribute '_accumulatorRegistry'
Found 1 error in 1 file (checked 185 source files)
1
Sorry for the noise, just my excitement to see this happen. Any action
points that we can define and that I can help on? I'm fine with taking the
route that Hyukjin suggests :)
Cheers, Fokko
Op do 27 aug. 2020 om 18:45 schreef Maciej <mszymkiew...@gmail.com>:
> Well, technically speaking annotation and actual are not the same thing.
> Many parts of Spark API might require heavy overloads to either capture
> relationships between arguments (for example in case of ML) or to capture
> at least rudimentary relationships between inputs and outputs (i.e. udfs).
>
> Just saying...
>
>
>
> On 8/27/20 6:09 PM, Driesprong, Fokko wrote:
>
> Also, it is very cumbersome to add everything to the pyi file. In
> practice, this means copying the method definition from the py file and
> paste it into the pyi file. This hurts my developers' heart, as it
> violates the DRY principle.
>
>
>
> I see many big projects using regular annotations:
> - Pandas:
> https://github.com/pandas-dev/pandas/blob/master/pandas/io/parquet.py#L51
>
> That's probably not a good example, unless something changed significantly
> lately. The last time I participated in the discussion Pandas didn't type
> check and had no clear timeline for advertising annotations.
>
>
> --
> Best regards,
> Maciej Szymkiewicz
>
> Web: https://zero323.net
> Keybase: https://keybase.io/zero323
> Gigs: https://www.codementor.io/@zero323
> PGP: A30CEF0C31A501EC
>
>
