We can always consider moving interfaces from core to the API when they are relatively stable. For SPI classes, I'm not sure there's a good reason to do it because the API and SPI are purposely different. API is for using Iceberg from a processing engine or data management service. SPI is for customizing how the core table implementations work. Maybe we could move SPI to its own module, eventually.
On Tue, Mar 19, 2019 at 8:36 PM Sandeep Nayak <osgig...@gmail.com> wrote: > Thanks for pointing me in the right direction Ryan. > > One question comes to mind, the `TableOperations` is in the core project > but the interface is identified as an SPI. If one were to plug in other > metastores should we consider restricting any such extensions to be only > against classes in api or maybe a separate spi sub-project v/s core > sub-project? That pattern would allow for restricting dependencies in one > isolated area and allowing other areas (like core) to evolve a bit more > freely without worrying about breaking any dependents. > > Thoughts? > > -Sandeep > > On Tue, Mar 19, 2019 at 2:24 PM Ryan Blue <rb...@netflix.com.invalid> > wrote: > >> Sandeep, we use the `TableOperations` API to plug in other metastores, as >> well as other customizations like `FileIO` implementations. Thanks to Matt >> and Yifei for their work extending this area! >> >> On Mon, Mar 18, 2019 at 8:49 PM Sandeep Nayak <osgig...@gmail.com> wrote: >> >>> To Xabriel's point, it would be good to have a Store abstraction so that >>> one could plug-in an implementation be it HMS or something else. >>> >>> >>> On Mon, Mar 18, 2019 at 3:20 PM Xabriel Collazo Mojica >>> <xcoll...@adobe.com.invalid> wrote: >>> >>>> +1 for having a tool/API to migrate tables from HMS into Iceberg. >>>> >>>> >>>> >>>> We do not use HMS in my current project, but since HMS is the de facto >>>> catalog in most companies doing Hadoop, I think such a tool would be vital >>>> for incentivizing Iceberg adoption and/or PoCs. >>>> >>>> >>>> >>>> *Xabriel J Collazo Mojica* | Senior Software Engineer | Adobe | >>>> xcoll...@adobe.com >>>> >>>> >>>> >>>> *From: *<aokolnyc...@apple.com> on behalf of Anton Okolnychyi >>>> <aokolnyc...@apple.com.INVALID> >>>> *Reply-To: *"dev@iceberg.apache.org" <dev@iceberg.apache.org> >>>> *Date: *Monday, March 18, 2019 at 2:22 PM >>>> *To: *"dev@iceberg.apache.org" <dev@iceberg.apache.org>, Ryan Blue < >>>> rb...@netflix.com> >>>> *Subject: *Re: Extend SparkTableUtil to Handle Tables Not Tracked in >>>> Hive Metastore >>>> >>>> >>>> >>>> I definitely support this idea. Having a clean and reliable API to >>>> migrate existing Spark tables to Iceberg will be helpful. >>>> >>>> I propose to collect all requirements for the new API in this thread. >>>> Then I can come up with a doc that we will discuss within the community. >>>> >>>> >>>> >>>> From the feature perspective, I think it would be important to support >>>> tables that persist partition information in HMS as well as tables that >>>> derive partition information from the folder structure. Also, migrating >>>> just a partition of a table would be useful. >>>> >>>> >>>> >>>> >>>> >>>> On 18 Mar 2019, at 18:28, Ryan Blue <rb...@netflix.com.INVALID> wrote: >>>> >>>> >>>> >>>> I think that would be fine, but I want to throw out a quick warning: >>>> SparkTableUtil was initially written as a few handy helpers, so it wasn't >>>> well designed as an API. It's really useful, so I can understand wanting to >>>> extend it. But should we come up with a real API for these conversion tasks >>>> instead of updating the hacks? >>>> >>>> >>>> >>>> On Mon, Mar 18, 2019 at 11:11 AM Anton Okolnychyi < >>>> aokolnyc...@apple.com.invalid> wrote: >>>> >>>> Hi, >>>> >>>> SparkTableUtil can be helpful for migrating existing Spark tables into >>>> Iceberg. Right now, SparkTableUtil assumes that the partition information >>>> is always tracked in Hive metastore. >>>> >>>> What about extending SparkTableUtil to handle Spark tables that don’t >>>> rely on Hive metastore? I have a local prototype that makes use of Spark >>>> InMemoryFileIndex to infer the partitioning info. >>>> >>>> Thanks, >>>> Anton >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Ryan Blue >>>> >>>> Software Engineer >>>> >>>> Netflix >>>> >>>> >>>> >>> >> >> -- >> Ryan Blue >> Software Engineer >> Netflix >> > -- Ryan Blue Software Engineer Netflix
