Hi Steve,
Can you shed some light why do they need $JAVA_HOME at all if everything is
already in place?
Regards,
- Sergey
On Mon, Jul 18, 2022 at 4:31 AM Stephen Coy
wrote:
> Hi Szymon,
>
> There seems to be a common misconception that setting JAVA_HOME will set
> the version of Java that is u
The suggestion is to check:
1. Used format for write
2. Used parallelism
On Thu, Apr 14, 2022 at 7:13 PM Anil Dasari wrote:
> Hello,
>
>
>
> We are upgrading spark from 2.4.7 to 3.0.1. we use spark sql (hive) to
> checkpoint data frames (intermediate data). DF write is very slow in 3.0.1
> comp
In order for the Spark to see Hive metastore you need to build Spark
Session accordingly:
val spark = SparkSession.builder()
.master("local[2]")
.appName("myApp")
.config("hive.metastore.uris","thrift://localhost:9083")
.enableHiveSupport()
.getOrCreate()
On Mon, Nov 12, 2018 at 11:49 A
Try to make encoding right.
E.g,, if you read from `csv` or other sources, specify encoding, which is
most probably `cp1251`:
df = sqlContext.read.csv(filePath, encoding="cp1251")
On Linux cli encoding can be found with `chardet` utility
On Wed, Jan 18, 2017 at 3:53 PM, AlexModestov
wrote:
>
I have a asked a similar question here
http://stackoverflow.com/questions/40701518/spark-2-0-redefining-sparksession-params-through-getorcreate-and-not-seeing-cha
Please see the answer, basically stating that it's impossible to change
Session config as soon as it was initiated
On Mon, Dec 19, 20
