Re: How more than one spark job can write to same partition in the parquet file

Hey Chetan, I have not got your question. Are you trying to write to a partition from two actions ?? or you are looking for writing from two jobs. Except for maintaining the state for the dataset completeness in that case, I dont see any issues. We are writing data to a Partition using two differ

Re: How more than one spark job can write to same partition in the parquet file

We partitioned data logically for 2 different jobs...in our use case based on geography... On Thu, 12 Dec 2019 at 3:39 pm, Chetan Khatri wrote: > Thanks, If you can share alternative change in design. I would love to > hear from you. > > On Wed, Dec 11, 2019 at 9:34 PM ayan guha wrote: > >> No

Re: How more than one spark job can write to same partition in the parquet file

Thanks, If you can share alternative change in design. I would love to hear from you. On Wed, Dec 11, 2019 at 9:34 PM ayan guha wrote: > No we faced problem with that setup. > > On Thu, 12 Dec 2019 at 11:14 am, Chetan Khatri < > chetan.opensou...@gmail.com> wrote: > >> Hi Spark Users, >> would t

Re: How more than one spark job can write to same partition in the parquet file

No we faced problem with that setup. On Thu, 12 Dec 2019 at 11:14 am, Chetan Khatri wrote: > Hi Spark Users, > would that be possible to write to same partition to the parquet file > through concurrent two spark jobs with different spark session. > > thanks > -- Best Regards, Ayan Guha