Would it be an option to just write the results of each job into separate 
tables and then run a UNION on all of them at the end into a final target 
table? Just thinking of an alternative!

Thanks,
Subhash

Sent from my iPhone

> On Apr 20, 2017, at 3:48 AM, Rick Moritz <rah...@gmail.com> wrote:
> 
> Hi List,
> 
> I'm wondering if the following behaviour should be considered a bug, or 
> whether it "works as designed":
> 
> I'm starting multiple concurrent (FIFO-scheduled) jobs in a single 
> SparkContext, some of which write into the same tables.
> When these tables already exist, it appears as though both jobs [at least 
> believe that they] successfully appended to the table (i.e., both jobs 
> terminate succesfully, but I haven't checked whether the data from both jobs 
> was actually written, or if one job overwrote the other's data, despite 
> Mode.APPEND). If the table does not exist, both jobs will attempt to create 
> the table, but whichever job's turn is second (or  later) will then fail with 
> a AlreadyExistsException (org.apache.spark.sql.AnalysisException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: AlreadyExistsException).
> 
> I think the issue here is, that both jobs don't register the table with the 
> metastore, until they actually start writing to it, but determine early on 
> that they will need to create it. The slower job then oobviously fails 
> creating the table, and instead of falling back to appending the data to the 
> existing table crashes out.
> 
> I would consider this a bit of a bug, but I'd like to make sure that it isn't 
> merely a case of me doing something stupid elsewhere, or indeed simply an 
> inherent architectural limitation of working with the metastore, before going 
> to Jira with this.
> 
> Also, I'm aware that running the jobs strictly sequentially would work around 
> the issue, but that would require reordering jobs before sending them off to 
> Spark, or kill efficiency.
> 
> Thanks for any feedback,
> 
> Rick

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Reply via email to