Hello, I am new to Spark, so I have a basic question which I couldn't find an answer online.
If I want to run SQL queries on a Spark dataframe, do I have to create a temporary table first? I know I could use the Spark SQL API, but is there a way of simply reading the data and run SQL queries without creating a temporary table? If that is impossible, does that mean that running SQL queries is less efficient than using the API since it does not require the use of temporary tables? Any help would be appreciated. Thanks,
