Spark Executor pod not getting created on kubernetes cluster

Hi Team I am trying to create a spark cluster on kubernetes with rbac enabled using spark submit job. I am using spark-2.4.1 version. Spark submit is able to launch the driver pod by contacting Kubernetes API server but executor Pod is not getting launched. I can see the below warning message in t

Re: Announcing .NET for Apache Spark 0.5.0

Congratulations on the release :) On Mon, Sep 30, 2019 at 9:38 AM Terry Kim wrote: > We are thrilled to announce that .NET for Apache Spark 0.5.0 has been just > released ! > > > > Some of the highlights of this release include: > >- Delta

Announcing .NET for Apache Spark 0.5.0

We are thrilled to announce that .NET for Apache Spark 0.5.0 has been just released ! Some of the highlights of this release include: - Delta Lake 's *DeltaTable *APIs - UDF improvements - Support f

How to handle this use-case in spark-sql-streaming

Hi, I have scenario like below https://stackoverflow.com/questions/58134379/how-to-handle-backup-scenario-in-spark-structured-streaming-using-joins How to handle this use-case ( back-up scenario) in spark-structured-streaming? Any clues would be highly appreciated. Thanks, Shyam

Re: Read text file row by row and apply conditions

You can use csv reader with delimiter as '|' to split data and create a dataframe on top of the file data. Second step, filter the dataframe on column value like indicator type=A,D etc and put it in tables. Saving to tables you can use dataframewriter(not sure what is you destination db type here).