Re: Parallel write to different partitions

Found this issue reported earlier but was bulk closed: https://issues.apache.org/jira/browse/SPARK-27030 Regards, Shrikant On Fri, 22 Sep 2023 at 12:03 AM, Shrikant Prasad wrote: > Hi all, > > We have multiple spark jobs running in parallel trying to write into same > hive table

Parallel write to different partitions

Hi all, We have multiple spark jobs running in parallel trying to write into same hive table but each job writing into different partition. This was working fine with Spark 2.3 and Hadoop 2.7. But after upgrading to Spark 3.2 and Hadoop 3.2.2, these parallel jobs are failing with FileNotFound exc

Re: Spark migration from 2.3 to 3.0.1

move the sesion inside main(), not a > member. > Or what other explanation do you have? I don't understand. > > On Mon, Jan 2, 2023 at 10:10 AM Shrikant Prasad > wrote: > >> If that was the case and deserialized session would not work, the >> application would not hav

Re: Spark migration from 2.3 to 3.0.1

: > It silently allowed the object to serialize, though the > serialized/deserialized session would not work. Now it explicitly fails. > > On Mon, Jan 2, 2023 at 9:43 AM Shrikant Prasad > wrote: > >> Thats right. But the serialization would be happening in Spark 2.3 also,

Re: Spark migration from 2.3 to 3.0.1

ause you are trying to use TestMain methods > in your program. > This was never correct, but now it's an explicit error in Spark 3. The > session should not be a member variable. > > On Mon, Jan 2, 2023 at 9:24 AM Shrikant Prasad > wrote: > >> Please see these logs. The

Re: Spark migration from 2.3 to 3.0.1

executor; that's not the issue. See your stack > trace, where it clearly happens in the driver. > > On Mon, Jan 2, 2023 at 8:58 AM Shrikant Prasad > wrote: > >> Even if I set the master as yarn, it will not have access to rest of the >> spark confs. It will need spark.yar

Re: Spark migration from 2.3 to 3.0.1

wrote: > So call .setMaster("yarn"), per the error > > On Mon, Jan 2, 2023 at 8:20 AM Shrikant Prasad > wrote: > >> We are running it in cluster deploy mode with yarn. >> >> Regards, >> Shrikant >> >> On Mon, 2 Jan 2023 at 6:15 PM, Steli

Re: Spark migration from 2.3 to 3.0.1

ding to where you want to run this > > On Mon, 2 Jan 2023 at 14:38, Shrikant Prasad > wrote: > >> Hi, >> >> I am trying to migrate one spark application from Spark 2.3 to 3.0.1. >> >> The issue can be reproduced using below sample code: >> >>

Spark migration from 2.3 to 3.0.1

at TestMain$.(TestMain.scala) >From the exception it appears that it tries to create spark session on executor also in Spark 3 whereas its not created again on executor in Spark 2.3. Can anyone help in identfying why there is this change in behavior? Thanks and Regards, Shrikant -- Regards, Shrikant Prasad

Re: sequence file write

I have tried with that also. It gives same exception: ClassNotFoundException: sequencefile.DefaultSource Regards, Shrikant On Mon, 14 Nov 2022 at 6:35 PM, Jie Han wrote: > It seems that the name is “sequencefile”. > > > 2022年11月14日 20:59，Shrikant Prasad 写道： > > > &g

sequence file write

Spark 3.2. Is there any change in sequence file support in 3.2 or any code change is required to make it work? Thanks and regards, Shrikant -- Regards, Shrikant Prasad

Re: Dynamic allocation on K8

dynamic allocation is available, however I am not sure how > it works. Spark official docs > <https://spark.apache.org/docs/latest/running-on-kubernetes.html#future-work> > say that shuffle service is not yet available. > > Thanks > > Nikhil > -- Regards, Shrikant Prasad