On Tue, 23 Oct 2018 at 21:32, Sean Owen <sro...@gmail.com> wrote: > >> The comments say that it is not possible to overload 'and' and 'or', >> which would have been more natural. >> > Yes, unfortunately, Python does not allow you to override and, or, or not. They are not implemented as “dunder” method (e.g. __add__()) and they implement special short-circuiting logic that’s not possible to reproduce with a function call. I think we made the most practical choice in overriding the bitwise operators.
In any case, I’ll file a JIRA ticket about this, and maybe also submit a PR to close it, adding documentation about PySpark column boolean operators to the programming guide. Nick
