ilto:mykid...@gmail.com>>;
user@spark.apache.org<mailto:user@spark.apache.org>
mailto:user@spark.apache.org>>
Subject: Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL
Hi,
I completely agree with Saurabh, the use of BQ with SPARK does not make sense
at all,
lar and presto didn't support auto
> scaling.
>
> TIA
> Saurabh
> --------------
> *From:* Mich Talebzadeh
> *Sent:* 22 February 2022 16:49
> *To:* Kidong Lee ; Saurabh Gulati <
> saurabh.gul...@fedex.com>
> *Cc:* user@spark.apache.org
> *Sub
Mich Talebzadeh ; Kidong Lee
; user@spark.apache.org
Subject: Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL
Hi,
I completely agree with Saurabh, the use of BQ with SPARK does not make sense
at all, if you are trying to cut down your costs. I think that costs do matter
to
container.
>
> We use Spark on GKE setup to run thrift-server which spawns workers
> depending on the load. For buckets we use gcs.
>
>
> TIA
> Saurabh
> --------------
> *From:* Mich Talebzadeh
> *Sent:* 22 February 2022 16:05
> *To:* Saurabh Gulati
> *Cc:* u
in a third container.
>
> We use Spark on GKE setup to run thrift-server which spawns workers
> depending on the load. For buckets we use gcs.
>
>
> TIA
> Saurabh
> --------------
> *From:* Mich Talebzadeh
> *Sent:* 22 February 2022 16:05
> *To:* Saurabh Gulati
>
e.org
Subject: Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL
Ok interesting.
I am surprised why you are not using BigQuery and using Hive. My assumption is
that your Spark is version 3.1.1 with standard GKE on auto-scaler. What
benefits are you getting from Using Hive here? A
--
> *From:* Mich Talebzadeh
> *Sent:* 22 February 2022 16:05
> *To:* Saurabh Gulati
> *Cc:* user@spark.apache.org
> *Subject:* [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark
> SQL
>
> *Caution! This email originated outside of FedEx. Please do not open
> a
: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL
Thanks Sean for your response.
@Mich Talebzadeh<mailto:mich.talebza...@gmail.com> We run all workloads on GKE
as docker containers. So to answer your questions, Hive is running in a
container as K8S service and spark thrift-ser
eed to make WHERE clause compulsory in Spark SQL
Caution! This email originated outside of FedEx. Please do not open attachments
or click links from an unknown or suspicious origin.
Is your hive on prem with external tables in cloud storage?
Where is your spark running from and what cloud bucke
Is your hive on prem with external tables in cloud storage?
Where is your spark running from and what cloud buckets are you using?
HTH
On Tue, 22 Feb 2022 at 12:36, Saurabh Gulati
wrote:
> Hello,
> We are trying to setup Spark as the execution engine for exposing our data
> stored in lake. We
Spark does not use Hive for execution, so Hive params will not have an
effect. I don't think you can enforce that in Spark. Typically you enforce
things like that at a layer above your SQL engine, or can do so, because
there is probably other access you need to lock down.
On Tue, Feb 22, 2022 at 6
11 matches
Mail list logo
