Re: [DISCUSS] Pythonic approach of setting Spark SQL configurations

2024-12-26 Thread Reynold Xin
I actually think this might be confusing (just in general adding too many different ways to do the same thing is also un-Pythonic). On Thu, Dec 26, 2024 at 4:58 PM Hyukjin Kwon wrote: > Hi all, > > I hope you guys are enjoying the holiday season. I just wanted to have > some quick feedback about

[DISCUSS] Pythonic approach of setting Spark SQL configurations

2024-12-26 Thread Hyukjin Kwon
Hi all, I hope you guys are enjoying the holiday season. I just wanted to have some quick feedback about this PR https://github.com/apache/spark/pull/49297 This PR allows you do set/unset SQL configurations in Pythonic way, e.g., >>> spark.conf["spark.sql.optimizer.runtime.rowLevelOperationGrou

Re: [DISCUSS] Pythonic approach of setting Spark SQL configurations

2024-12-26 Thread Holden Karau
I think having automatic gettr/settr on spark.conf object seems reasonable to me. On Thu, Dec 26, 2024 at 9:32 PM Reynold Xin wrote: > I actually think this might be confusing (just in general adding too many > different ways to do the same thing is also un-Pythonic). > > On Thu, Dec 26, 2024 at