Oh, this is probably because of how annotations are handled. In general stubs take preference over inline annotations and are considered the only source of type hints, unless packaged is marked as partially typed (https://www.python.org/dev/peps/pep-0561/#id21). In such case however is all-or-nothing on the module basis.
Nonetheless ecosystem is still somewhat, so different tools might choose
different approach.
On 8/27/20 10:24 PM, Driesprong, Fokko wrote:
> Looking at it a second time, I think it is only mypy that's complaining:
>
> MacBook-Pro-van-Fokko:spark fokkodriesprong$ git diff
>
> *diff --git a/python/pyspark/accumulators.pyi
> b/python/pyspark/accumulators.pyi*
>
> *index f60de25704..6eafe46a46 100644*
>
> *--- a/python/pyspark/accumulators.pyi*
>
> *+++ b/python/pyspark/accumulators.pyi*
>
> @@ -30,7 +30,7 @@U = TypeVar("U", bound=SupportsIAdd)
>
>
>
> import socketserver as SocketServer
>
>
>
> -_accumulatorRegistry: Dict = {}
>
> +# _accumulatorRegistry: Dict = {}
>
>
>
> class Accumulator(Generic[T]):
>
> aid: int
>
>
> MacBook-Pro-van-Fokko:spark fokkodriesprong$ ./dev/lint-python
>
> starting python compilation test...
>
> python compilation succeeded.
>
>
> starting pycodestyle test...
>
> pycodestyle checks passed.
>
>
> starting flake8 test...
>
> flake8 checks passed.
>
>
> starting mypy test...
>
> mypy checks failed:
>
> python/pyspark/worker.py:34: error: Module 'pyspark.accumulators' has
> no attribute '_accumulatorRegistry'
>
> Found 1 error in 1 file (checked 185 source files)
>
> 1
>
>
> Sorry for the noise, just my excitement to see this happen. Any action
> points that we can define and that I can help on? I'm fine with taking
> the route that Hyukjin suggests :)
>
> Cheers, Fokko
>
> Op do 27 aug. 2020 om 18:45 schreef Maciej <mszymkiew...@gmail.com
> <mailto:mszymkiew...@gmail.com>>:
>
> Well, technically speaking annotation and actual are not the same
> thing. Many parts of Spark API might require heavy overloads to
> either capture relationships between arguments (for example in
> case of ML) or to capture at least rudimentary relationships
> between inputs and outputs (i.e. udfs).
>
> Just saying...
>
>
>
> On 8/27/20 6:09 PM, Driesprong, Fokko wrote:
>> Also, it is very cumbersome to add everything to the pyi file. In
>> practice, this means copying the method definition from the py
>> file and paste it into the pyi file. This hurts my developers'
>> heart, as it violates the DRY principle.
>
>
>>
>> I see many big projects using regular annotations:
>> -
>> Pandas:
>> https://github.com/pandas-dev/pandas/blob/master/pandas/io/parquet.py#L51
>
> That's probably not a good example, unless something changed
> significantly lately. The last time I participated in the
> discussion Pandas didn't type check and had no clear timeline for
> advertising annotations.
>
>
> --
> Best regards,
> Maciej Szymkiewicz
>
> Web: https://zero323.net
> Keybase: https://keybase.io/zero323
> Gigs: https://www.codementor.io/@zero323
> PGP: A30CEF0C31A501EC
>
--
Best regards,
Maciej Szymkiewicz
Web: https://zero323.net
Keybase: https://keybase.io/zero323
Gigs: https://www.codementor.io/@zero323
PGP: A30CEF0C31A501EC
signature.asc
Description: OpenPGP digital signature
