Based on my understanding, we are not inventing anything new here. Basically, we are building a federated database system especially after we supporting multiple catalog. There are many mature commercial products in the market. For example, https://www.ibm.com/support/knowledgecenter/en/SSEPGG_10.5.0/com.ibm.swg.im.iis.db.fed.overview.doc/topics/cfpint25.html is the link for explaining the federated catalog in DB2.
Could we answer the difference and tradeoff in the new proposal, compared with these existing system? If different, what is the motivation we do not want to follow the traditional solution? If the same, do we have some user-facing APIs? Cheers, Xiao Wenchen Fan <cloud0...@gmail.com> 于2018年11月28日周三 下午6:41写道： > Hi Ryan, > > Thanks for hosting the discussion! I think the table catalog is super > useful, but since this is the first time we allow users to extend catalog, > it's better to write down some details from end-user APIs to internal > management. > 1. How would end-users register/unregister catalog with SQL API and > Scala/Java API? > 2. How would end-users manage catalogs? like LIST CATALOGS, USE CATALOG > xyz? > 3. How to separate the abilities of catalog? Can we create a bunch of > mixin triats for catalog API like SupportsTable, SupportsFunction, > SupportsView, etc.? > 4. How should Spark resolve identifies with catalog name? How to resolve > ambiguity? What if the catalog doesn't support database? Can users write > `catalogName.tblName` directly? > 5. Where does Spark store the catalog list? In an in-memory map? > 6. How to support atomic CTAS? > 7. The data/schema of table may change over time, when should Spark > determine the table content? During analysis or planning? > 8. ... > > Since the catalog API is not only developer facing, but also user-facing, > I think it's better to have a doc explaining what the developers concern > and what the end users concern. The doc is also good for future reference, > and can be used in release notes. > > Thanks, > Wenchen > > On Wed, Nov 28, 2018 at 12:54 PM JackyLee <qcsd2...@163.com> wrote: > >> +1 >> >> Please add me to the Google Hangout invite. >> >> >> >> -- >> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
