Re: [DISCUSS] Pythonic approach of setting Spark SQL configurations

On your point ...I believe there are better ways to improve the pythonic surface of PySpark. .. Can you please elaborate? HTH Mich Talebzadeh, Architect | Data Science | Financial Crime | GDPR & Compliance Specialist PhD Imperial College Lon

Re: [DISCUSS] Pythonic approach of setting Spark SQL configurations

Generally function and column composition can be quite verbose. Maybe this is something to invest some brain power. I see to often folks fall back to expr() or selectEpxr(). The other one I stumbled across was the idea of dynamic selectors like Polara has them. https://docs.pola.rs/api/python/sta

Re: [DISCUSS] Pythonic approach of setting Spark SQL configurations

On top proper deep integration with pytest would be awesome. On Sun, Dec 29, 2024 at 16:40 Martin Grund wrote: > Generally function and column composition can be quite verbose. Maybe this > is something to invest some brain power. I see to often folks fall back to > expr() or selectEpxr(). > > T