ilto:mykid...@gmail.com>>;
user@spark.apache.org<mailto:user@spark.apache.org>
mailto:user@spark.apache.org>>
Subject: Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL
Hi,
I completely agree with Saurabh, the use of BQ with SPARK does not make sense
at all,
lar and presto didn't support auto
> scaling.
>
> TIA
> Saurabh
> --------------
> *From:* Mich Talebzadeh
> *Sent:* 22 February 2022 16:49
> *To:* Kidong Lee ; Saurabh Gulati <
> saurabh.gul...@fedex.com>
> *Cc:* user@spark.apache.org
> *Sub
Mich Talebzadeh ; Kidong Lee
; user@spark.apache.org
Subject: Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL
Hi,
I completely agree with Saurabh, the use of BQ with SPARK does not make sense
at all, if you are trying to cut down your costs. I think that costs do matter
to
container.
>
> We use Spark on GKE setup to run thrift-server which spawns workers
> depending on the load. For buckets we use gcs.
>
>
> TIA
> Saurabh
> --------------
> *From:* Mich Talebzadeh
> *Sent:* 22 February 2022 16:05
> *To:* Saurabh Gulati
> *Cc:* u
in a third container.
>
> We use Spark on GKE setup to run thrift-server which spawns workers
> depending on the load. For buckets we use gcs.
>
>
> TIA
> Saurabh
> --------------
> *From:* Mich Talebzadeh
> *Sent:* 22 February 2022 16:05
> *To:* Saurabh Gulati
>
e.org
Subject: Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL
Ok interesting.
I am surprised why you are not using BigQuery and using Hive. My assumption is
that your Spark is version 3.1.1 with standard GKE on auto-scaler. What
benefits are you getting from Using Hive here? A
--
> *From:* Mich Talebzadeh
> *Sent:* 22 February 2022 16:05
> *To:* Saurabh Gulati
> *Cc:* user@spark.apache.org
> *Subject:* [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark
> SQL
>
> *Caution! This email originated outside of FedEx. Please do not open
> a
: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL
Thanks Sean for your response.
@Mich Talebzadeh<mailto:mich.talebza...@gmail.com> We run all workloads on GKE
as docker containers. So to answer your questions, Hive is running in a
container as K8S service and spark thrift-ser
eed to make WHERE clause compulsory in Spark SQL
Caution! This email originated outside of FedEx. Please do not open attachments
or click links from an unknown or suspicious origin.
Is your hive on prem with external tables in cloud storage?
Where is your spark running from and what cloud bucke
