(Resending now that I’m subscribed, as my earlier message doesn’t appear in the public archive. Apologies if you receive this twice)
Hi Spark devs, I’d like to propose *ai_query* for *PySpark*. It takes a prompt column and returns the model’s response as a column. *Databricks SQL* and *Google BigQuery *offer similar capabilities through *ai_query <https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_query> *and *AI*.*GENERATE* <https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate>, respectively. Here’s what this could look like in a *PySpark DataFrame.* from pyspark.sql.ai import ai_query, register_modelfrom pyspark.sql.functions import col, concat, lit register_model( "summarizer", "databricks-gpt-oss-20b", provider="databricks", profile="my-workspace", auth_type="databricks-cli", ) df.withColumn( "summary", ai_query( endpoint="summarizer", request=concat(lit("Summarize in one sentence: "), col("description")), ), ) *register_model* maps *summarizer* to a model endpoint. You can switch between *Databricks Model Serving* and *Amazon Bedrock* by changing the registration, while keeping the same DataFrame code. *The prototype passed end-to-end tests with both providers using Spark and Arrow.* What do you think of this *API* for *PySpark*? *I’d love to hear how you’d use it and what you’d change.* Draft PR #60311 <https://github.com/apache/spark/pull/60311> JIRA SPARK-61404 <https://issues.apache.org/jira/browse/SPARK-61404> Thanks, Morgan
