Re: Announcing Delta Lake 0.2.0

Hi Ayan, Delta is obviously well thought through, its been available in Databricks since last year and a half now I think and besides that it is from some of the best minds at work :) But what may not be well tested in Delta is its availability as a storage class for HIVE. How about your testing

Re: Announcing Delta Lake 0.2.0

Hi Liwen, its done https://github.com/delta-io/delta/issues/73 Please let me know in case the description looks fine. I can also contribute to the test cases in case required. Regards, Gourav On Thu, Jun 20, 2019 at 12:52 AM Liwen Sun wrote: > Hi Gourav, > > Thanks for the suggestion. Please

Re: What is the compatibility between releases?

Hi Community , I am still looking for an answer for this question I am running a cluster using Spark 2.3.1 , but I wondering if it is safe to include Spark 2.4.1 and use new features such as higher order functions. Thank you. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com

Re: Override jars in spark submit

Hi Naresh, You could use "--conf spark.driver.extraClassPath=". Note that the jar will not be shipped to the executors, if its a class that is needed on the executors as well you should provide "--conf spark.executor.extraClassPath=". Note that if you do provide executor extraclasspath the jar fil

connecting spark with mysql

Hi everyone, I tried to manipulate MySQL tables from spark, I do not want to move these tables from MySQL to spark, as these tables can easily get very big. It is ideal that the data stays in the database where it was stored. For me, spark is only used to speed up the read and write process (as

Re: Spark SQL

Just to make it more clear, Spark sql uses hive metastore and run queries using its own engine and not uses hive execution engine. Please correct me if it’s not true. On Mon, Jun 10, 2019 at 2:29 PM Russell Spitzer wrote: > Spark can use the HiveMetastore as a catalog, but it doesn't use the

Override jars in spark submit

Hello All, How can we override jars in spark submit? We have hive-exec-spark jar which is available as part of default spark cluster jars. We wanted to override above mentioned jar in spark submit with latest version jar. How do we do that ? Thank you, Naresh -- Thanks, Naresh www.linkedin.com/

Re: Announcing Delta Lake 0.2.0

Hi We are using Delta features. The only problem we faced till now is Hive can not read DELTA outputs by itself (even if the Hive metastore is shared). However, if we create hive external table pointing to the folder (and with Vacuum), it can read the data. Other than that, the feature looks good

Re: Announcing Delta Lake 0.2.0

Hi Gourav, Thanks for the suggestion. Please open a Github issue at https://github.com/delta-io/delta/issues to describe your use case and requirements for "external tables" so we can better track this feature and also get feedback from the community. Regards, Liwen On Wed, Jun 19, 2019 at 12:11

Re: Announcing Delta Lake 0.2.0

Hi, does Delta support external tables? I think that most users will be needing this. Regards, Gourav On Wed, Jun 19, 2019 at 8:04 PM Liwen Sun wrote: > We are delighted to announce the availability of Delta Lake 0.2.0! > > To try out Delta Lake 0.2.0, please follow the Delta Lake Quickstart:

Re: Announcing Delta Lake 0.2.0

Hi, this is fantastic :) Regards, Gourav Sengupta On Wed, Jun 19, 2019 at 8:04 PM Liwen Sun wrote: > We are delighted to announce the availability of Delta Lake 0.2.0! > > To try out Delta Lake 0.2.0, please follow the Delta Lake Quickstart: > https://docs.delta.io/0.2.0/quick-start.html > > T

Announcing Delta Lake 0.2.0

We are delighted to announce the availability of Delta Lake 0.2.0! To try out Delta Lake 0.2.0, please follow the Delta Lake Quickstart: https://docs.delta.io/0.2.0/quick-start.html To view the release notes: https://github.com/delta-io/delta/releases/tag/v0.2.0 This release introduces two main

pyspark cached dataframe shows deserialized at StorageLevel

Hi, Spark document describes "Since the data is always serialized on the Python side, all the constants use the serialized formats.". http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.StorageLevel But when I cached dataframe and looked StorageLevel, it shows that cached d

RE: tcps oracle connection from spark

Connecting to Oracle from Spark using the TPCS protocol works OK for me. Maybe try to turn debug on with -Djavax.net.debug=all? See also: https://blogs.oracle.com/dev2dev/ssl-connection-to-oracle-db-using-jdbc%2c-tlsv12%2c-jks-or-oracle-wallets Regards, L. From: Richard Xin Sent: Wednesday, June

[webinar] TFX Chicago Taxi example on Mini Kubeflow (MiniKF)

Hi all, I would like to invite you to our webinar "Kubeflow Pipelines on-prem". It will take place on Friday, June 21 at 9am Pacific Time. You can register here: https://zoom.us/webinar/register/WN_j_HJbkISTluMckyyr706eg We are going to demonstrate the end-to-end TFX Chicago Taxi example ru

Unsubscribe

-- Tushar Marne 9011062432

Re: Ask for ARM CI for spark

Thanks for your reply. As I said before, I met some problem of build or test for spark on aarch64 server, so it will be better to have the ARM CI to make sure the spark is compatible for AArch64 platforms. I’m from OpenLab team(https://openlabtesting.org/ ,a community to do open source project te