Re: Insertable records in Datasource v2.

Hi Rahul, I think it's not and will not be supported. You should report an issue in JIRA at https://issues.apache.org/jira/projects/SPARK, but don't expect a solution with temporary views though. I did manage to reproduce the AnalysisException: unresolved operator and even though INSERT INTO view

Re: Use case advice

Thanks, Muru, very helpful suggestion! Delta Lake is amazing, completely changed a few of my projects! One question regarding that. When I use the following statement, all works fine and I can use delta properly, in the spark context that jupyter initiates automatically. export PYSPARK_DRIVER_PYT

Re: Use case advice

sorry missed out a bit. Added, highlighted with yellow. On Thu, 14 Jan 2021 at 13:54, András Kolbert wrote: > Thanks, Muru, very helpful suggestion! Delta Lake is amazing, completely > changed a few of my projects! > > One question regarding that. > When I use the following statement, all works

Re: Spark 3.0.1 not connecting with Hive 2.1.1

I am having the a similar problem to the original post. I am trying to run in Spark 3 and connect to CDH Hive 2.1.1. I have run with the same spark.sql.hive.metastore options. The main difference in my environment is that I am trying to run in Spark on K8S using spark-operator, making it a little m

Adding third party specific jars to Spark

The primer for this was the process of developing code for accessing BigQuery data from PyCharm on premises so that advanced analytics and graphics can be done on local. Writes are an issue as BiqQuery buffers data in a temporary storage on GS bucket before pushing it into BigQuery database One o

Re: Insertable records in Datasource v2.

Thank you Jacek, for trying it out and clarifying. Appreciate it. Best, Rahul -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Use case advice

You need to make sure the delta-core_2.11-0.6.1. jar file in your $SPARK_HOME/jars folder. On Thu, Jan 14, 2021 at 4:59 AM András Kolbert wrote: > sorry missed out a bit. Added, highlighted with yellow. > > On Thu, 14 Jan 2021 at 13:54, András Kolbert > wrote: > >> Thanks, Muru, very helpful su

Re: Use case advice

Unsubscribe

Spark 3.0.1 giving warning while running with Java 11

Hi All, Getting warning while running spark3.0.1 with Java11 . WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform ( file:/opt/spark/jars/spark-unsafe_2.12-3.0.1.jar) to constructor java.nio.DirectByteBuffer(long,int

Re: Spark 3.0.1 giving warning while running with Java 11

You can ignore that. Spark 3.x works with Java 11 but it will generate some warnings that are safe to disregard. On Thu, Jan 14, 2021 at 11:26 PM Sachit Murarka wrote: > Hi All, > > Getting warning while running spark3.0.1 with Java11 . > > > WARNING: An illegal reflective access operation has o