Re: ***pyspark.sql.functions.monotonically_increasing_id()***

2023-04-28 Thread Winston Lai
__ From: Karthick Nk Sent: Friday, April 28, 2023 7:59:29 PM To: user@spark.apache.org Subject: ***pyspark.sql.functions.monotonically_increasing_id()*** Hi @all, I am using monotonically_increasing_id(), in the pyspark function, for removing one field from json field in one colu

***pyspark.sql.functions.monotonically_increasing_id()***

2023-04-28 Thread Karthick Nk
Hi @all, I am using monotonically_increasing_id(), in the pyspark function, for removing one field from json field in one column from the delta table, please refer the below code df = spark.sql(f"SELECT * from {database}.{table}") df1 = spark.read.json(df.rdd.map(lambda x: x.data), multiLine = Tr