Re: [Structured Streaming] NullPointerException in long running query

The stack trace is omitted by JVM when an exception is thrown too many times. This usually happens when you have multiple Spark tasks on the same executor JVM throwing the same exception. See https://stackoverflow.com/a/3010106 Best Regards, Ryan On Tue, Apr 28, 2020 at 10:45 PM lec ssmi wrote:

Re: How can I add extra mounted disk to HDFS

Use Hadoop NFSv3 gateway to mount FS. @*JB*Δ Le mar. 28 avr. 2020 à 23:18, Chetan Khatri a écrit : > Hi Spark Users, > > My spark job gave me an error No Space left on the device >

Unsubscribe

-- Thanks Deepak www.bigdatabig.com www.keosha.net

Re: OFFICIAL USA REPORT TODAY India Most Dangerous : USA Religious Freedom Report out TODAY

I am unsubscribing until these hatemongers like Zahid Amin are removed or blocked . FYI Zahid Amin , Indian govt rejected the false report already . On Wed, 29 Apr 2020 at 11:58 AM, Gaurav Agarwal wrote: > Spark moderator supress this user please. Unnecessary Spam or apache spark > account is

Re: OFFICIAL USA REPORT TODAY India Most Dangerous : USA Religious Freedom Report out TODAY

Spark moderator supress this user please. Unnecessary Spam or apache spark account is hacked ? On Wed, Apr 29, 2020, 11:56 AM Zahid Amin wrote: > How can it be rumours ? > Of course you want to suppress me. > Suppress USA official Report out TODAY . > > > Sent: Wednesday, April 29, 2020 at 8:

OFFICIAL USA REPORT TODAY India Most Dangerous : USA Religious Freedom Report out TODAY

How can it be rumours ? Of course you want to suppress me. Suppress USA official Report out TODAY . > Sent: Wednesday, April 29, 2020 at 8:17 AM > From: "Deepak Sharma" > To: "Zahid Amin" > Cc: user@spark.apache.org > Subject: Re: India Most Dangerous : USA Religious Freedom Report > > Can so

Re: India Most Dangerous : USA Religious Freedom Report out Today

> Sent: Wednesday, April 29, 2020 at 8:17 AM > From: "Deepak Sharma" > To: "Zahid Amin" > Cc: user@spark.apache.org > Subject: Re: India Most Dangerous : USA Religious Freedom Report > > Can someone block this email ? > He is spreading rumours and spamming. > > On Wed, 29 Apr 2020 at 11:46 AM, Za

Re: India Most Dangerous : USA Religious Freedom Report

Can someone block this email ? He is spreading rumours and spamming. On Wed, 29 Apr 2020 at 11:46 AM, Zahid Amin wrote: > USA report states that India is now the most dangerous country for Ethnic > Minorities. > > Remember Martin Luther King. > > > https://www.mail.com/int/news/us/9880960-religi

India Most Dangerous : USA Religious Freedom Report

USA report states that India is now the most dangerous country for Ethnic Minorities. Remember Martin Luther King. https://www.mail.com/int/news/us/9880960-religious-freedom-watchdog-pitches-adding-india-to.html#.1258-stage-set1-3 It began with Kasmir and still in locked down Since August 2019.

Re: [Structured Streaming] NullPointerException in long running query

It should be a problem of my data quality. It's curious why the driver-side exception stack has no specific exception information. Edgardo Szrajber 于2020年4月28日周二 下午3:32写道： > The exception occured while aborting the stage. It might be interesting to > try to understand the reason for the abortion

Re: Read Hive ACID Managed table in Spark

Hi Amogh, Thanks for your reply. I'll send you seperate email for the communication. On Thu, Apr 9, 2020 at 1:11 PM amogh margoor wrote: > Sorry for the late reply. > I can help you with getting started with > https://github.com/qubole/spark-acid to read Hive ACID tables. Feel free > to drop m

Re: Unablee to get to_timestamp with Timezone Information

Thanks Enrico, Magnus On Thu, Apr 2, 2020 at 11:49 AM Enrico Minack wrote: > Once parsed into a Timestamp the timestamp is store internally as UTC and > printed as your local timezone (e.g. as defined by > spark.sql.session.timeZone). Spark is good at hiding timezone information > from you. > >

How can I add extra mounted disk to HDFS

Hi Spark Users, My spark job gave me an error No Space left on the device

[no subject]

Unsubscribe Get Outlook for Android

Re: Converting a date to milliseconds with time zone in Scala with fixed date str

import java.time._ import java.util.Date import java.text.SimpleDateFormat import java.util.Locale import java.util.SimpleTimeZone import org.joda.time import org.joda.time.DateTime object CalendarDemo extends App { println("Calendar Demo") val pattern = "E dd M HH:mm:ss.SSSZ

Re: Spark 2.3 and Kafka client library version

You should be able to override the Kafka client version. The Kafka APIs used by Structured Streaming exist in new Kafka versions. There is a known correctness issue in Kafka 0.10.1.*. Other versions should be fine. Best Regards, Ryan On Tue, Apr

Re: Converting a date to milliseconds with time zone in Scala

Hi, Thank you all, I am just thinking of passing that date 06/04/2020 12:03:43 and getting the correct format from the module. In effect This date format -MM-dd'T'HH:mm:ss.SZ as pattern in other words rather than new Date() pass "06/04/2020 12:03:43" as string REgards, Dr Mi

Spark 2.3 and Kafka client library version

I have a keberized HDFS cluster. When I use structured streaming with Kafka (with SASL_SSL/PLAINTEXT), I believe I’m blocked by Kafka-5294. It seems like fix version in 0.11.0.0 Kafka client library. I have a Spark 2.3 cluster, and it’s using 0.10.0.1 kafka client library. Do you know if I can

Re: Converting a date to milliseconds with time zone in Scala

import java.time._ import java.util.Date import java.text.SimpleDateFormat import java.util.Locale import java.util.SimpleTimeZone object CalendarDemo extends App { println("Calendar Demo") val pattern = "E dd M HH:mm:ss.SSSZ"; val simpleDateFormat = (new SimpleDateForma

Re: Converting a date to milliseconds with time zone in Scala

import java.time._ import java.util.Date import java.text.SimpleDateFormat import java.util.Locale import java.util.SimpleTimeZone object CalendarDemo extends App { println("Calendar Demo") val pattern =3D "E dd M HH:mm:ss.SSSZ"; val simpleDateFormat =3D (new SimpleDateF

Re: Converting a date to milliseconds with time zone in Scala

Hiplease check combining unix_timestamp and from_unixtime, Something like:  from_unixtime(unix_timestamp( "06-04-2020 12:03:43"),"-MM-dd'T'HH:mm:ss Z") please note that I just wrote without any validation. In any case, you might want to check the documentation of both functions to check all

Re: Converting a date to milliseconds with time zone in Scala

Unfortunately that did not work. any other suggestions? thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress

Re: Converting a date to milliseconds with time zone in Scala

Thanks Neeraj, I'll check it out. ! Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com *Disclaimer:* Use it a

Re: Converting a date to milliseconds with time zone in Scala

Hi Mich, You can try Spark DateTime function here and see if that helps. https://medium.com/expedia-group-tech/deep-dive-into-apache-spark-datetime-functions-b66de737950a Regards, Neeraj On Tue, Apr 28, 2020 at 5:15 PM Mich Talebzadeh wrote: > Hi, > > I have a date in format like 06/04/202

Converting a date to milliseconds with time zone in Scala

Hi, I have a date in format like 06/04/2020 12:03:43 and we want it to be displayed as follows: -MM-dd'T'HH:mm:ss.SZ format So the input date is GMT date time just we do not receive the information with it The output should have timezone information Appreciate any ideas. Thanks

Unsubscribe

unsubscribe From: Yasir Elgohary [mailto:ygoh...@gmail.com] Sent: Tuesday, April 28, 2020 8:18 AM To: user@spark.apache.org Subject: Unsubscribe... unsubscribe This communication and any attached files may contain information that is confidential or privileged. If this communication has been re

Unsubscribe...

unsubscribe

Structured Streaming using Kafka Avro Record in 2.3.0

Hi How can we deserialise avro record read from kafka in spark 2.3.0 in optimised manner. I could see that native support for avro was added in 2.4.x. Currently i am using following library which is very slow. com.twitter bijection-avro_2.11 Kind Regards Harsh Takkar

Spark stable release for Hadoop 3

Hello, We are considering whether to use Hadoop or Kubernetes as the cluster manager for Spark. We would prefer to have Hadoop 3 because of its native support for scheduling GPUs. Although there is a Spark 3.0.0 pre-view2 version available that is pre-built for Hadoop 3, I would like to know when

Re: [Structured Streaming] NullPointerException in long running query

The exception occured while aborting the stage. It might be interesting to try to understand the reason for the abortion.Maybe timeout? How long the query run?Bentzi Sent from Yahoo Mail on Android On Tue, Apr 28, 2020 at 9:25, Jungtaek Lim wrote: The root cause of exception is occurred

Re: SparkLauncher reliability and scalability

How about Apache Livy? the purpose is similar as SparkLauncher, but through Restful API to launch Spark Jobs From: mhd wrk Sent: Monday, April 27, 2020 11:38 PM To: user@spark.apache.org Subject: SparkLauncher reliability and scalability [Warning]: This email o