Re: Spark SQL query

Sanchit, It seems I have to do some sort of analysis from the plan to get the query. Appreciate all your help on this. Thanks, Arpan Bhandari -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscrib

Re: Spark SQL query

Hi Mich, Repeated the steps as suggested, but still there is no such folder created in the home directory. Do we need to enable some property so that it creates one. Thanks, Arpan Bhandari -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

Re: Spark SQL query

Hi Arpan. I believe all applications including spark and scala create a hidden history file You can go to home directory cd # see list of all hidden files ls -a | egrep '^\.' If you are using scala do you see .scala_history file? .scala_history HTH LinkedIn * https://www.linkedin.com/pr

Re: Spark SQL query

Hi Mich, I do see the .scala_history directory, but it contains all the queries which got executed uptill now, but if i have to map a specific query to an application Id in yarn that would not correlate, hence this method alone won't suffice Thanks, Arpan Bhandari -- Sent from: http://apache

Re: Spark SQL query

Yes i can see the jobs on 8088 and also on the spark history url. spark history server is showing up the plan details on the sql tab but not giving the query. Thanks, Arpan Bhandari -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

Spark 3 datasource v2: Can't extract user provided schema Dataframewriter save operation

Dear friends,I'm implementing datasource v2 for a custom NoSql database. I'm facing following issuea) It seems while doing save operation, there is no way to access user specified schema on dataframe. There is an existing unresolved ongoing conversation here

Re: Spark SQL query

create a directory in hdfs hdfs dfs -mkdir /spark_event_logs modify file $SPARK_HOME/conf/spark-defaults.conf and add these two lines spark.eventLog.enabled=true # do not use quotes below spark.eventLog.dir=hdfs://rhes75:9000/spark_event_logs Then run a job and check it hdfs dfs -ls /spark_eve

S3a Committer

Hi, I have some issues at the moment with S3 API of Openstack Swift (S3a). This one is eventually consistent and it causes lots of issues with my distributed jobs in Spark. Is the S3A committer able to fix that ? Or an "S3guard like" implementation is the only way ? David

Exception on Avro Schema Object Serialization

We tried to standardize the SQL data source management using the Avro schema, but encountered some serialization exceptions when trying to use the data.  The interesting part is that we didn't have any problems in reading the Avro schema JSON file and converting the Avro schema into a SQL Struc

Re: Exception on Avro Schema Object Serialization

Your function is somehow capturing the actual Avro schema object, which won't seiralize. Try rewriting it to ensure that it isn't used in the function. On Tue, Feb 2, 2021 at 2:32 PM Artemis User wrote: > We tried to standardize the SQL data source management using the Avro > schema, but encount

Re: S3a Committer

S3 is strongly consistent now https://aws.amazon.com/s3/consistency/ Regards, Henoc On Tue, Feb 2, 2021, 10:27 PM David Morin wrote: > Hi, > > I have some issues at the moment with S3 API of Openstack Swift (S3a). > This one is eventually consistent and it causes lots of issues with my > distri

Re: S3a Committer

Yes, that's true but this is not (yet) the case of the Openstack Swift S3 API Le mar. 2 févr. 2021 à 21:41, Henoc a écrit : > S3 is strongly consistent now > https://aws.amazon.com/s3/consistency/ > > Regards, > Henoc > > On Tue, Feb 2, 2021, 10:27 PM David Morin > wrote: > >> Hi, >> >> I have

Re: Exception on Avro Schema Object Serialization

Thanks Sean.  But the real issue is on the surface, my UDF should have no knowledge of the Avro schema at all!  Here are the high-level steps of what happened: 1. read the JSON to create a Avro Schema object  -- OK 2. Convert the Avro schema to a StructType using the Spark's SchemaConverter

Re: Spark SQL query

Mich, The directory is already there and event logs are getting generated, I have checked them it contains the query plan but not the actual query. Thanks, Arpan Bhandari -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --

RE: S3a Committer

This December AWS announced https://aws.amazon.com/s3/consistency/, are you sure this is your problem? I think all these s3guard like wrappers are irrelevant right now. Please correct me if I am wrong. From: David Morin Sent: Tuesday, 2 February 2021 22:26 To: user@spark.apache.org Subject: S3a

Re: S3a Committer

Hi, I have almost the same problem with Ceph RGW, and currently do research about Apache Iceberg and Databricks Delta(opensource version). I think these libraries can address the problem. 2021년 2월 3일 (수) 오후 3:44, Boris Litvak 님이 작성: > This December AWS announced https://aws.amazon.com/s3/consis