This is an automated email from the ASF dual-hosted git repository.
ueshin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e953cf26f727 [SPARK-55156][PS][FOLLOW-UP] Fix fallback functions
e953cf26f727 is described below
commit e953cf26f72794ad876e35d1bd1bf5d44f56508c
Author: Takuya Ueshin <[email protected]>
AuthorDate: Wed Mar 4 17:51:44 2026 -0800
[SPARK-55156][PS][FOLLOW-UP] Fix fallback functions
### What changes were proposed in this pull request?
This is a follow-up of apache/spark#54285.
Fixes fallback functions.
### Why are the changes needed?
apache/spark#54285 changed the behavior of `groupby-apply` with pandas 3,
which caused a failure of fallback functions, like `DataFrame.asfreq`,
`DataFrame.asof`, etc.
```py
>>> psdf.asfreq(freq="30s")
Traceback (most recent call last):
...
KeyError: "['__tmp_aggregate_col_for_frame_asfreq__'] not found in axis"
```
### Does this PR introduce _any_ user-facing change?
Yes, it will behave more like pandas 3.
### How was this patch tested?
The existing tests should pass.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #54630 from ueshin/issues/SPARK-55156/fallback.
Authored-by: Takuya Ueshin <[email protected]>
Signed-off-by: Takuya Ueshin <[email protected]>
---
python/pyspark/pandas/frame.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/pandas/frame.py b/python/pyspark/pandas/frame.py
index ed6e63138e86..8377f72cd376 100644
--- a/python/pyspark/pandas/frame.py
+++ b/python/pyspark/pandas/frame.py
@@ -14086,7 +14086,8 @@ defaultdict(<class 'list'>, {'col..., 'col...})]
# TODO(SPARK-46859): specify the return type if possible
def compute_function(pdf: pd.DataFrame): # type:
ignore[no-untyped-def]
- pdf = pdf.drop(columns=[tmp_agg_column_name])
+ if LooseVersion(pd.__version__) < "3.0.0":
+ pdf = pdf.drop(columns=[tmp_agg_column_name])
pdf = pdf.set_index(tmp_idx_column_name, drop=True)
pdf = pdf.sort_index()
pdf = getattr(pdf, method)(*args, **kwargs)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
