In my opinion, after improving the commit mechanism of FileSystemCatalog, hadoopcatalog can be initially put into production environment.Currently, most customers use hadoopcatalog for external tables.We can make sure that there are no problems for this group of users. Secondly, as for iceberg's view support.I think it's a low priority.It's tempting, but it's not the most pressing need. I think the most pressing need is to further improve hadoopcatalogTables, because we are actually implementing a file system based way of managing catalogues.It should not be tied to hadoop. Hadoop should only be an implementation of it.I think it would be better to change the implementation of this catalog to an abstract FileSystemCatalog, similar to the current implementation of FileIO.Once we've done that, I think all the rest will be much easier. Otherwise, I think we'll come back to this sooner or later.
在 2024-07-04 22:04:17，"Jean-Baptiste Onofré" <j...@nanthrax.net> 写道： >Hi, > >Thanks for the heads up and working on this ! > >My understanding of the HadoopCatalog is that we would need more than >an improved commit mechanism to be production ready (I'm thinking on >scalability, or view support). What's your thoughts? >By the way, I'm happy to take a look at adding view support if it helps. > >Regards >JB > >On Thu, Jul 4, 2024 at 8:27 AM lisoda <lis...@yeah.net> wrote: >> >> Hi Team. >> I've refactored the logic of the commit method in HadoopTableOptions.With >> this refactoring, I believe that hadoopCatalog is ready to be used in a >> production environment. Now HadoopTableOptions can implement atomic commits >> while being compatible with the differences in behaviour between block and >> object stores.Concurrency control is also supported.if anyone can assist me >> in reiewing this PR, that would be great. >> Also, any FileSystemCatalog's user can comment on this PR. Any advice would >> be invaluable to me. >> Thank you all. >> >> PR:https://github.com/apache/iceberg/pull/10623 >> SLACK:https://apache-iceberg.slack.com/archives/C03LG1D563F/p1719993403208859
