Re: [Spark Catalog API] Support for metadata Backup/Restore

2021-05-11 Thread Tianchen Zhang
le enough and persistent enough, >>> why dont just use external metastore? It is fairly straightforward process. >>> Also regardless you are in cloud or not, database bkp is a routine and >>> established pattern in most organizations. >>> You can also enhance HA and D

Re: [Spark Catalog API] Support for metadata Backup/Restore

2021-05-07 Thread Tianchen Zhang
x27;t see a strong reason to > expose the APIs to Spark. Do you plan to add new SQL commands in Spark to > backup/restore a catalog? > > On Tue, May 4, 2021 at 2:39 AM Tianchen Zhang > wrote: > >> Hi all, >> >> Currently the user-facing Catalog API doesn't s

[Spark Catalog API] Support for metadata Backup/Restore

2021-05-03 Thread Tianchen Zhang
Hi all, Currently the user-facing Catalog API doesn't support backup/restore metadata. Our customers are asking for such functionalities. Here is a usage example: 1. Read all metadata of one Spark cluster 2. Save them into a Parquet file on DFS 3. Read the Parquet file and restore all metadata in

[no subject]

2021-05-03 Thread Tianchen Zhang
Hi all, Currently the user-facing Catalog API doesn't support backup/restore metadata. Our customers are asking for such functionalities. Here is a usage example: 1. Read all metadata of one Spark cluster 2. Save them into a Parquet file on DFS 3. Read the Parquet file and restore all metadata in