java.lang.AssertionError: assertion failed: Found duplicate rewrite attributes

Hello Experts During a join operation, I see this error below (spark 3.0.2) Any suggestions on how to debug? Error: java.lang.AssertionError: assertion failed: Found duplicate rewrite attribute Source code: val dfFilteredFinal=dfFiltered .join(dfScenarioSite, Seq("tid","site"), "left_oute

Issue with spark 3.1.2 accessing Cloud bucket throwing errors

Hi, I have a PySpark job that uses a Kubernetes cluster (GKE) on Google cloud. The Spark image was built on PySpark 3.1.2, Scala 3.7, Java 8 as below with some Python packages *3.1.2*-scala_2.12-8-jre-slim-buster-addedpackages When run it throws this error 21/08/28 07:43:18 WARN NativeCodeLoad