[
https://issues.apache.org/jira/browse/HIVE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15673317#comment-15673317
]
Steve Loughran commented on HIVE-15199:
---------------------------------------
# as sahil notes, blobstore copy calls must be in object store to get internal
bandwidth *and avoid download costs*.
# that can currently only be done in rename(); if we ever added a copy()
command to the FS API, your life would be better
# the fact that rename returns "false" without details makes things worse. FWIW
I'm modifying spark's inner rename to throw exceptions —but as as that isn't
the public FS API, it's of no use here. Though I could add an option to s3a to
always throw those exceptions (a subclass of IOE) if the caller needed it.
Would that help? it'd only be broadly useful if HDFS &c also did that
Now, what is needed to be done here?
# handle the situation where a list of an object store path can be out of sync
with the actual contents, something that surfaces in S3 and swift. the
exists/getFileStatus() call is more robust there.
# handle the situation where rename(src, dest), src is a file and dest is a
file, will copy src onto dest.
Short term: check the destination. I know some people will worry about the cost
of the existence check, but remember this is going to trigger a copy operation,
which is O(data) at about 6-8 MB/s: if you are committing large files, things
will be slow.
Looking at the rename semantics as tested in {{AbstractContractRenameTest}},
the test {{testRenameFileOverExistingFile}} has the comment "handles
filesystems that will overwrite the destination as well as those that do not
(i.e. HDFS).". It's not just s3a which lets you rename over a test, so
apparently does local file://. Interestingly: azure wasb:// doesn't, nor does
s3n.
I think we can consider the fact that s3a lets you overwrite an existing
destination to be a bug, based on its inconsistency with HDFS, Azure, s3n, etc.
Indeed, the difference between s3a and s3n makes it harder to say "s3a is a
drop-in replacement for s3n". Created HADOOP-13823 for you.
> INSERT INTO data on S3 is replacing the old rows with the new ones
> ------------------------------------------------------------------
>
> Key: HIVE-15199
> URL: https://issues.apache.org/jira/browse/HIVE-15199
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Reporter: Sergio Peña
> Assignee: Sergio Peña
> Priority: Critical
> Attachments: HIVE-15199.1.patch, HIVE-15199.2.patch,
> HIVE-15199.3.patch
>
>
> Any INSERT INTO statement run on S3 tables and when the scratch directory is
> saved on S3 is deleting old rows of the table.
> {noformat}
> hive> set hive.blobstore.use.blobstore.as.scratchdir=true;
> hive> create table t1 (id int, name string) location 's3a://spena-bucket/t1';
> hive> insert into table t1 values (1,'name1');
> hive> select * from t1;
> 1 name1
> hive> insert into table t1 values (2,'name2');
> hive> select * from t1;
> 2 name2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
