[
https://issues.apache.org/jira/browse/HUDI-1268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liwei updated HUDI-1268:
------------------------
Description:
一、issue：in UpgradeDowngrade.run()
fs.rename(updatedPropsFilePath, propsFilePath);
the fs.rename have different action,such as:
// a) for hdfs : if propsFilePath already exist,fs.rename will not replace
propsFilePath, but just return false
// b) for localfs: if propsFilePath already exist,fs.rename will replace
propsFilePath, and return ture
// c) for aliyun ossfs: if propsFilePath already exist,will throw
FileAlreadyExistsException
// so we should delete the old propsFilePath. also upgrade and downgrade is
Idempotent
二、for aliyun oss filesystem
when using HoodieWriteClient API to write data to hudi with following config:
```
Properties properties = new Properties();
properties.setProperty(HoodieTableConfig.HOODIE_TABLE_NAME_PROP_NAME,
tableName);
properties.setProperty(HoodieTableConfig.HOODIE_TABLE_TYPE_PROP_NAME,
tableType.name());
properties.setProperty(HoodieTableConfig.HOODIE_PAYLOAD_CLASS_PROP_NAME,
OverwriteWithLatestAvroPayload.class.getName());
properties.setProperty(HoodieTableConfig.HOODIE_ARCHIVELOG_FOLDER_PROP_NAME,
"archived");
return HoodieTableMetaClient.initTableAndGetMetaClient(hadoopConf, basePath,
properties);
```
the exception will be thrown with FileAlreadyExistsException in aliyun OSS,
after debugging, it is the following code throws the exception.
```
// Rename the .updated file to hoodie.properties. This is atomic in hdfs, but
not in cloud stores.
// But as long as this does not leave a partial hoodie.properties file, we are
okay.
fs.rename(updatedPropsFilePath, propsFilePath);
```
however, we would ignore the FileAlreadyExistsException since hoodie.properties
already exists.
was:
when using HoodieWriteClient API to write data to hudi with following config:
```
Properties properties = new Properties();
properties.setProperty(HoodieTableConfig.HOODIE_TABLE_NAME_PROP_NAME,
tableName);
properties.setProperty(HoodieTableConfig.HOODIE_TABLE_TYPE_PROP_NAME,
tableType.name());
properties.setProperty(HoodieTableConfig.HOODIE_PAYLOAD_CLASS_PROP_NAME,
OverwriteWithLatestAvroPayload.class.getName());
properties.setProperty(HoodieTableConfig.HOODIE_ARCHIVELOG_FOLDER_PROP_NAME,
"archived");
return HoodieTableMetaClient.initTableAndGetMetaClient(hadoopConf, basePath,
properties);
```
the exception will be thrown with FileAlreadyExistsException in aliyun OSS,
after debugging, it is the following code throws the exception.
```
// Rename the .updated file to hoodie.properties. This is atomic in hdfs, but
not in cloud stores.
// But as long as this does not leave a partial hoodie.properties file, we are
okay.
fs.rename(updatedPropsFilePath, propsFilePath);
```
however, we would ignore the FileAlreadyExistsException since hoodie.properties
already exists.
> Fix UpgradeDowngrade fs Rename issue for hdfs and aliyun oss
> ------------------------------------------------------------
>
> Key: HUDI-1268
> URL: https://issues.apache.org/jira/browse/HUDI-1268
> Project: Apache Hudi
> Issue Type: Bug
> Components: Writer Core
> Reporter: leesf
> Assignee: liwei
> Priority: Major
> Fix For: 0.6.1
>
>
> 一、issue：in UpgradeDowngrade.run()
> fs.rename(updatedPropsFilePath, propsFilePath);
> the fs.rename have different action,such as:
> // a) for hdfs : if propsFilePath already exist,fs.rename will not replace
> propsFilePath, but just return false
> // b) for localfs: if propsFilePath already exist,fs.rename will replace
> propsFilePath, and return ture
> // c) for aliyun ossfs: if propsFilePath already exist,will throw
> FileAlreadyExistsException
> // so we should delete the old propsFilePath. also upgrade and downgrade is
> Idempotent
>
> 二、for aliyun oss filesystem
> when using HoodieWriteClient API to write data to hudi with following config:
> ```
> Properties properties = new Properties();
> properties.setProperty(HoodieTableConfig.HOODIE_TABLE_NAME_PROP_NAME,
> tableName);
> properties.setProperty(HoodieTableConfig.HOODIE_TABLE_TYPE_PROP_NAME,
> tableType.name());
> properties.setProperty(HoodieTableConfig.HOODIE_PAYLOAD_CLASS_PROP_NAME,
> OverwriteWithLatestAvroPayload.class.getName());
> properties.setProperty(HoodieTableConfig.HOODIE_ARCHIVELOG_FOLDER_PROP_NAME,
> "archived");
> return HoodieTableMetaClient.initTableAndGetMetaClient(hadoopConf, basePath,
> properties);
> ```
> the exception will be thrown with FileAlreadyExistsException in aliyun OSS,
> after debugging, it is the following code throws the exception.
>
> ```
> // Rename the .updated file to hoodie.properties. This is atomic in hdfs, but
> not in cloud stores.
> // But as long as this does not leave a partial hoodie.properties file, we
> are okay.
> fs.rename(updatedPropsFilePath, propsFilePath);
> ```
> however, we would ignore the FileAlreadyExistsException since
> hoodie.properties already exists.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
