RE: error while connecting to azure blob storage

Hi Team, I have written a small code to connect to azure blob storage but go error. I have attached the error log. Please help Calling command -- ./spark-submit stg.py --jars /home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-azure-3.2.0.jar,/home/citus/spark/spark-2.3.3-bin-hadoop2.7/j

Classloading issues when using connectors with Uber jars with improper Shading in single Spark job

Hi Guys When using connectors with Uber jars, we are hitting classloading issues in Spark 2.3.0. Upon investigation we found out that the classloading issues were caused by improper shading of certain classes in these uber jars. The aim of this email is to start a discussion on whether such iss

Re: Questions for platform to choose

Hi Eliza, As I mentioned to you in the Kafka mailing list when you asked this there, there are pros and cons to all of the technologies you've mentioned, and you really need to sit down and try each solution to see what suits your needs best. Kind regards, Liam Clarke On Wed, Aug 21, 2019 at 9:

Spark scanning every Hive Database

Hi, I am using Spark with Java and connecting to Hive. The steps in my code are: a) Create SparkSession : SparkSession.builder().config(conf).enableHiveSupport().getOrCreate(); b) Check existense of a Hive database : spark.catalog().databaseExists(config.getHiveDbName()), if false I create the da

Re: [External Sender] Execute Spark model without Spark

Hi you can checkout mLeap - https://github.com/combust/mleap But I must warn you - their support is minimal at best. Femi Sent from my iPhone On Aug 22, 2019, at 1:13 PM, Yeikel wrote: Hi , I have a GBTClassificationModel < https://urldefense.proofpoint.com/v2/url?u=https-3A__spark.apache.

Execute Spark model without Spark

Hi , I have a GBTClassificationModel that I generated using Spark. How can I export this model and use without a Spark cluster? I would like to serve it outside of Spark -- Sent

RE: Release Spark 2.3.4

The following PRs regarding SPARK-28699 have been merged into branch-2.3. https://github.com/apache/spark/pull/25491 https://github.com/apache/spark/pull/25498 -> https://github.com/apache/spark/pull/25508　(backport to 2.3) I will cut `2.3.4-rc1` tag during weekend and starts 2.3.1 RC1 on next Mo

Re: Creating custom Spark-Native catalyst/codegen functions

https://github.com/DataSystemsLab/GeoSpark should be public. Am Do., 22. Aug. 2019 um 09:50 Uhr schrieb Arwin Tio : > Hey, > > It seems like the GeoSpark repo is not publicly accessible? > > But from the filepath it seems like the Spark codebase itself was forked > or modified. > > The examples t

How to support hive built in functions in saprk

HI i want to use the hive built-in functions in saprk. i tried current_username() function in spark local mode. i am getting the below error. Error: org.apache.spark.sql.AnalysisException: Undefined function: 'current_user'. This function is neither a registered temporary function nor a permanent

Re: Creating custom Spark-Native catalyst/codegen functions

Hey, It seems like the GeoSpark repo is not publicly accessible? But from the filepath it seems like the Spark codebase itself was forked or modified. The examples that I've seen seem to suggest that you need to register custom Spark-Native functions inside Spark's private namespace like you s