Re: [DISCUSS] Pythonic approach of setting Spark SQL configurations

On top proper deep integration with pytest would be awesome. On Sun, Dec 29, 2024 at 16:40 Martin Grund wrote: > Generally function and column composition can be quite verbose. Maybe this > is something to invest some brain power. I see to often folks fall back to > expr() or selectEpxr(). > > T

Re: [DISCUSS] Pythonic approach of setting Spark SQL configurations

Generally function and column composition can be quite verbose. Maybe this is something to invest some brain power. I see to often folks fall back to expr() or selectEpxr(). The other one I stumbled across was the idea of dynamic selectors like Polara has them. https://docs.pola.rs/api/python/sta

Re: [DISCUSS] Pythonic approach of setting Spark SQL configurations

On your point ...I believe there are better ways to improve the pythonic surface of PySpark. .. Can you please elaborate? HTH Mich Talebzadeh, Architect | Data Science | Financial Crime | GDPR & Compliance Specialist PhD Imperial College Lon

Re: [DISCUSS] Pythonic approach of setting Spark SQL configurations

I'm not a fan of this approach. Spark configuration keys are defined as string values in Spark and used as Strings everywhere. I don't necessarily see the benefit of changing conf["keyName"] vs conf.get("keyName") or even spark.conf.keyName Trying to wrap this into magic getattr calls is not ide

Re: [DISCUSS] Pythonic approach of setting Spark SQL configurations

On the surface it looks like a good idea. In essence,writing code that is not just functional but also reflects the spirit and style of the Python language . It is about writing code that is readable, and maintainable. The core objective (if I am correct) of this

Re: [DISCUSS] Pythonic approach of setting Spark SQL configurations

I think having automatic gettr/settr on spark.conf object seems reasonable to me. On Thu, Dec 26, 2024 at 9:32 PM Reynold Xin wrote: > I actually think this might be confusing (just in general adding too many > different ways to do the same thing is also un-Pythonic). > > On Thu, Dec 26, 2024 at

Re: [DISCUSS] Pythonic approach of setting Spark SQL configurations

I actually think this might be confusing (just in general adding too many different ways to do the same thing is also un-Pythonic). On Thu, Dec 26, 2024 at 4:58 PM Hyukjin Kwon wrote: > Hi all, > > I hope you guys are enjoying the holiday season. I just wanted to have > some quick feedback about

[DISCUSS] Pythonic approach of setting Spark SQL configurations

Hi all, I hope you guys are enjoying the holiday season. I just wanted to have some quick feedback about this PR https://github.com/apache/spark/pull/49297 This PR allows you do set/unset SQL configurations in Pythonic way, e.g., >>> spark.conf["spark.sql.optimizer.runtime.rowLevelOperationGrou