I think it's interesting but I'm hesistant to add only PySpark functions, 
there's no reason we can't build a proper cross-language implementation like 
the rest of our Spark functionality. I've got a bit of a larger WIP doc for 
looking at how we could add a common set (and how we might do it differently 
than "regular" UDFs), would love to collaborate on that together if your open 
to it :) 
https://docs.google.com/document/d/16WwnJjON3mT0ls1aF20nlHMS72-wU1LQfR9BTH5x_vM/edit?usp=sharing


On 2026-11-12 00:37 UTC Alejandro Morgante wrote:
> (Resending now that I’m subscribed, as my earlier message doesn’t appear in
> the public archive. Apologies if you receive this twice)
> 
> Hi Spark devs,
> 
> I’d like to propose *ai_query* for *PySpark*. It takes a prompt column and
> returns the model’s response as a column. *Databricks SQL* and *Google
> BigQuery *offer similar capabilities through *ai_query
> <https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_query>
> *and
> *AI*.*GENERATE*
> <https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate>,
> respectively. Here’s what this could look like in a *PySpark DataFrame.*
> 
> from pyspark.sql.ai import ai_query, register_modelfrom
> pyspark.sql.functions import col, concat, lit
> register_model(
>     "summarizer",
>     "databricks-gpt-oss-20b",
>     provider="databricks",
>     profile="my-workspace",
>     auth_type="databricks-cli",
> )
> df.withColumn(
>     "summary",
>     ai_query(
>         endpoint="summarizer",
>         request=concat(lit("Summarize in one sentence: "), 
> col("description")),
>     ),
> )
> 
> *register_model* maps *summarizer* to a model endpoint. You can switch
> between *Databricks Model Serving* and *Amazon Bedrock* by changing the
> registration, while keeping the same DataFrame code.
> *The prototype passed end-to-end tests with both providers using Spark and
> Arrow.*
> What do you think of this *API* for *PySpark*? *I’d love to hear how you’d
> use it and what you’d change.*
> 
> Draft PR #60311 <https://github.com/apache/spark/pull/60311>
> JIRA SPARK-61404 <https://issues.apache.org/jira/browse/SPARK-61404>
> 
> Thanks,
> Morgan
> 

---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]

Reply via email to