I think it's interesting but I'm hesistant to add only PySpark functions, there's no reason we can't build a proper cross-language implementation like the rest of our Spark functionality. I've got a bit of a larger WIP doc for looking at how we could add a common set (and how we might do it differently than "regular" UDFs), would love to collaborate on that together if your open to it :) https://docs.google.com/document/d/16WwnJjON3mT0ls1aF20nlHMS72-wU1LQfR9BTH5x_vM/edit?usp=sharing
On 2026-11-12 00:37 UTC Alejandro Morgante wrote: > (Resending now that I’m subscribed, as my earlier message doesn’t appear in > the public archive. Apologies if you receive this twice) > > Hi Spark devs, > > I’d like to propose *ai_query* for *PySpark*. It takes a prompt column and > returns the model’s response as a column. *Databricks SQL* and *Google > BigQuery *offer similar capabilities through *ai_query > <https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_query> > *and > *AI*.*GENERATE* > <https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate>, > respectively. Here’s what this could look like in a *PySpark DataFrame.* > > from pyspark.sql.ai import ai_query, register_modelfrom > pyspark.sql.functions import col, concat, lit > register_model( > "summarizer", > "databricks-gpt-oss-20b", > provider="databricks", > profile="my-workspace", > auth_type="databricks-cli", > ) > df.withColumn( > "summary", > ai_query( > endpoint="summarizer", > request=concat(lit("Summarize in one sentence: "), > col("description")), > ), > ) > > *register_model* maps *summarizer* to a model endpoint. You can switch > between *Databricks Model Serving* and *Amazon Bedrock* by changing the > registration, while keeping the same DataFrame code. > *The prototype passed end-to-end tests with both providers using Spark and > Arrow.* > What do you think of this *API* for *PySpark*? *I’d love to hear how you’d > use it and what you’d change.* > > Draft PR #60311 <https://github.com/apache/spark/pull/60311> > JIRA SPARK-61404 <https://issues.apache.org/jira/browse/SPARK-61404> > > Thanks, > Morgan > --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
