Quick test on on 3.2 confirms everything should be working as expected
scala> spark.createDataset(Seq(("foo", "bar")))
res0: org.apache.spark.sql.Dataset[(String, String)] = [_1: string, _2: string]
scala> spark.createDataset(Seq(("foo", "bar"))).createTempView("temp")
scala> spark.sql("SELECT replace(_1, 'fo', 'bo') from temp").show
+-------------------+
|replace(_1, fo, bo)|
+-------------------+
| boo|
+-------------------+
> On Oct 2, 2022, at 12:21 PM, Russell Spitzer <[email protected]>
> wrote:
>
> https://spark.apache.org/docs/3.3.0/api/sql/index.html#replace
> <https://spark.apache.org/docs/3.3.0/api/sql/index.html#replace>
>
> This was added in Spark 2.3.0 as far as I can tell.
>
> https://github.com/apache/spark/pull/18047
> <https://github.com/apache/spark/pull/18047>
>
>> On Oct 2, 2022, at 11:19 AM, Khalid Mammadov <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Hi,
>>
>> As you know there's no string "replace" function inside
>> pyspark.sql.functions for PySpark nor in org.apache.sql.functions for
>> Scala/Java and was wondering why is that so? And I know there's
>> regexp_replace instead and na.replace or SQL with expr.
>>
>> I think it's one of the fundamental functions in users/developers toolset
>> and available almost in every language. It takes time for new Spark devs to
>> realise it's not there and to use alternative ones. So, I think it would be
>> nice to have one.
>> I had already got a prototype for Scala (which is just a sugar over
>> regexp_replace) and works like a charm:)
>>
>> Would like to know your opinion to contribute or not needed...
>>
>> Thanks
>> Khalid
>>
>
