Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL

ilto:mykid...@gmail.com>>; user@spark.apache.org<mailto:user@spark.apache.org> mailto:user@spark.apache.org>> Subject: Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL Hi, I completely agree with Saurabh, the use of BQ with SPARK does not make sense at all,

Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL

lar and presto didn't support auto > scaling. > > TIA > Saurabh > -------------- > *From:* Mich Talebzadeh > *Sent:* 22 February 2022 16:49 > *To:* Kidong Lee ; Saurabh Gulati < > saurabh.gul...@fedex.com> > *Cc:* user@spark.apache.org > *Sub

Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL

Mich Talebzadeh ; Kidong Lee ; user@spark.apache.org Subject: Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL Hi, I completely agree with Saurabh, the use of BQ with SPARK does not make sense at all, if you are trying to cut down your costs. I think that costs do matter to

Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL

com> > *Cc:* user@spark.apache.org > *Subject:* Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in > Spark SQL > > Ok interesting. > > I am surprised why you are not using BigQuery and using Hive. My > assumption is that your Spark is version 3.1.1 with standard

Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL

l...@fedex.com> > *Cc:* user@spark.apache.org > *Subject:* Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in > Spark SQL > > Ok interesting. > > I am surprised why you are not using BigQuery and using Hive. My > assumption is that your Spark is version 3.1.1 with s

Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL

e.org Subject: Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL Ok interesting. I am surprised why you are not using BigQuery and using Hive. My assumption is that your Spark is version 3.1.1 with standard GKE on auto-scaler. What benefits are you getting from Using Hive here? A

Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL

Ok interesting. I am surprised why you are not using BigQuery and using Hive. My assumption is that your Spark is version 3.1.1 with standard GKE on auto-scaler. What benefits are you getting from Using Hive here? As you have your hive tables on gs buckets, you can easily download your hive tables

Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL

: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL Thanks Sean for your response. @Mich Talebzadeh<mailto:mich.talebza...@gmail.com> We run all workloads on GKE as docker containers. So to answer your questions, Hive is running in a container as K8S service and spark thrift-ser

Re: [EXTERNAL] Re: Need to make WHERE clause compulsory in Spark SQL

Thanks Sean for your response. @Mich Talebzadeh We run all workloads on GKE as docker containers. So to answer your questions, Hive is running in a container as K8S service and spark thrift-server in another container as a service and Superset in a third contai