milenkovicm commented on code in PR #215:
URL: https://github.com/apache/datafusion-site/pull/215#discussion_r4422362581
##########
content/blog/2026-10-29-ballista-python-udfs.md:
##########
@@ -0,0 +1,298 @@
+---
+layout: post
+title: Distributed Python UDFs in Apache DataFusion Ballista
+date: 2026-10-29
+author: pmc
+categories: [subprojects]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+The Python client for [Apache DataFusion Ballista] can now run Python
user-defined functions (UDFs) across a
+cluster. Scalar, aggregate, and window UDFs written in Python are serialized
into the query plan, shipped to
+the scheduler, and executed on Ballista executors, from both the DataFrame API
and SQL.
+
+This comes alongside a larger change in how the Ballista Python client works.
Instead of wrapping
+[DataFusion Python], Ballista now plugs into a standard
`datafusion.SessionContext` through DataFusion's FFI
+layer. If you know the DataFusion Python API, you already know how to use
Ballista.
+
+Neither change would have been possible without a long series of improvements
to DataFusion Python and the
+`datafusion-ffi` crate. This post explains what changed, how it works, and who
made it happen.
+
+[Apache DataFusion Ballista]: https://datafusion.apache.org/ballista/
+[DataFusion Python]: https://datafusion.apache.org/python/
+
+## What it looks like
+
+Python UDFs are most useful when logic is awkward to write with built-in SQL
functions but natural to express
+in Python, such as business rules kept in a Python data structure. To show
what this looks like on a cluster,
+we will rewrite a query from the [TPC-H] benchmark, the same benchmark
Ballista uses to measure its own
+performance, so you can try it on data you generate yourself.
+
+[TPC-H Query 19] computes the revenue from parts that qualify for a discount.
The rules differ by brand, and
+the reference SQL spells them out as three nearly identical blocks joined with
`OR`. With a Python UDF, the
+rules can live in an ordinary Python dictionary, and the query is reduced to a
join, the shared filters, and a
+single call to the UDF.
+
+To follow along, generate the two tables the query needs with [tpchgen-cli]:
+
+```shell
+pip install tpchgen-cli
+tpchgen-cli parquet --scale-factor 1 --tables lineitem,part --output-dir data
+```
+
+Then start a local cluster. The `ballista` Python package can run both the
scheduler and the executor. Start
+each in its own terminal, from the same directory as `data/`, so the executor
can find the files:
+
+```shell
+# terminal 1: scheduler, listening on localhost:50050
+python -c "from ballista import BallistaScheduler; s = BallistaScheduler();
s.start(); s.wait_for_termination()"
+
+# terminal 2: executor
+python -c "from ballista import BallistaExecutor; e = BallistaExecutor();
e.start(); e.wait_for_termination()"
+```
+
+Alternatively, `ballista.setup_test_cluster()` starts both inside the current
Python process and returns the
+host and port to connect to. Until the next release, the `ballista` package
needs to be built from source, as
+described in [Trying it out](#trying-it-out).
+
+The UDF is written with [pyarrow.compute], so each batch stays in Arrow rather
than being converted to Python
+objects:
+
+```python
+import functools
+import os
+
+import pyarrow as pa
+import pyarrow.compute as pc
+from datafusion import col, lit, udf
+from datafusion import functions as f
+from ballista import BallistaSessionContext
+
+ctx = BallistaSessionContext("df://localhost:50050")
Review Comment:
It would be good to make a comment here that the this line is actually how
df python is converted to ballista python, and not others changes are needed
(fingers crossed 🤞🏻)
We can add commented line which shows df py context commented
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
