Spark History Server in GCP

Hi There, I have been able to start the Spark History Server in GKE Kubernetes Cluster. And I have created a Service Account in my google project with permissions as storage admin,storage object admin and owner. Now when I have tried to submit the job using Spark Submit, using below options to w

Re: loop of spark jobs leads to increase in memory on worker nodes and eventually faillure

Clear-probably not a good idea. But a previous comment said “you are doing everything in the end in one go”. So this made me wonder: in case your only action is a write in the end after lots of complex transformations, then what is the alternative for writing in the end which means doing everyth