Thanks Kartik,
Yes indeed this is a BigQuery issue with Spark. Those two setting (below)
did not work in spark-submit or adding to
$SUBASE_HOME/conf/spark-defaults.conf
--conf
spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \
--conf
spark.executor.extraJavaOptions="-
Hi Mich!
It looks like the issue comes from the BigQuery Connector and not Spark
itself. For reference, see
https://github.com/GoogleCloudDataproc/spark-bigquery-connector/issues/256
and
https://github.com/GoogleCloudDataproc/spark-bigquery-connector/issues/350.
These issues also mention a few poss
Thanks for the hint.
The issue is that it can write data to the BigQuery table as I can see it
in the BigQuery dashboard.
SELECT * FROM test.randomData WHERE DATE(op_time) = "2021-08-07" LIMIT 1
[
{
"ID": "43",
"CLUSTERED": "0.42",
"SCATTERED": "42.0",
"RANDOMISED": "4.0",
Without seeing the code and the whole stack trace, just a wild guess if
you set the config param for enabling arrow
(spark.sql.execution.arrow.pyspark.enabled)? If not in your code, you
would have to set it in the spark-default.conf. Please note that the
parameter spark.sql.execution.arrow.e
Hi,
I encounter the error:
"java.lang.UnsupportedOperationException: sun.misc.Unsafe or
java.nio.DirectByteBuffer.(long, int) not available"
When reading from Google BigQuery (GBQ) table using Kubernetes cluster
built on debian buster
The current debian bustere from the docker image is:
r
Apparently you were not using the right formatting string. For
sub-second formatting, use capital S instead of lower case s. See
Spark's doc at
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html. Hope
this helps...
-- ND
On 8/4/21 4:42 PM, Tzahi File wrote:
Hi All,
I'm us
