Sounds good! -Jack
On Wed, Dec 6, 2023 at 10:16 AM Tim Brown <t...@onehouse.ai> wrote: > Hi Ryan, > > Apologies for the noise. > > Jack and Walaa, let's move any conversations to the Discussions > <https://github.com/onetable-io/onetable/discussions> board on github for > the project. Also feel free to reach out to me directly if you prefer. > > I'm personally looking forward to learning more about the Rest Catalog and > making sure we integrate well with it from OneTable. > > -Tim Brown > > On Wed, Dec 6, 2023 at 11:55 AM Ryan Blue <b...@tabular.io> wrote: > >> I'm not sure that this is the right place for a discussion about the >> merits of their approach. >> >> This list is for Iceberg development. I encourage anyone interested to >> follow up on the appropriate incubator list rather than here. >> >> I also think it's debatable whether advertising other projects is helpful >> or wanted here, but I'd rather not add to the noise either way. >> >> Ryan >> >> On Tue, Dec 5, 2023 at 8:36 PM Jack Ye <yezhao...@gmail.com> wrote: >> >>> I recently did an analysis of the OneTable project, overall it made me a >>> bit confused. >>> >>> From an end user's perspective, no one really wants to use all these 3 >>> formats, and most companies do not have the engineering resources to >>> maintain a stack of all these 3 formats. Eventually people pick one and >>> just stick with it. >>> >>> If the goal is to provide a converter, then individual communities have >>> developed different tools, such as Delta's Uniform, Iceberg's snapshot and >>> migrate procedures, Hudi's bootstrap methods. The advantage of those tools >>> is that the specific community knows the best way to convert a foreign data >>> source to its native format, and can declare compatibility and fail >>> whenever necessary. It is not bounded to the expressiveness of an internal >>> data model like OneTable, OneField, OneSchema, etc. >>> >>> If the goal is format unification, at least for me being in the Iceberg >>> community with a bit bias, a more straightforward way to achieve the goal >>> is to extend the feature of "Iceberg external tables", where we can map >>> Hive, Delta, Hudi and other table formats directly to Iceberg format behind >>> a REST catalog, and make that readable. This is kind of related to a >>> recent email thread I sent regarding the EXTERNAL/MANAGED syntax >>> <https://lists.apache.org/thread/ohqfvhf4wofzkhrvff1lxl58blh432o6>. And >>> linking back to this thread, that essentially makes Iceberg the unified >>> format, and we are actually pretty close to achieving that. With this >>> approach, you get not just conversion, you can (1) not do physical metadata >>> conversion but directly convert table metadata at runtime to Iceberg data >>> model, (2) query all the tables using a single unified Iceberg connector in >>> all supported engines, and (3) it is a very standardized external table >>> concept that all database system folks immediately understand. >>> >>> This makes me feel that we are trying to make OneTable a new table >>> format without saying it is a new table format. Although the Apache >>> Incubation proposal clearly says "OneTable is NOT a new table format", it >>> is hard for me to envision a long-term roadmap that does not eventually >>> make it a table format, with connectors and data maintenance features built >>> directly against this internal model, which is kind of feels like what the >>> commercial entity OneHouse is trying to do right now, but maybe I am wrong. >>> >>> What do you think? >>> >>> Best, >>> Jack Ye >>> >>> On Tue, Dec 5, 2023 at 3:30 PM Jesús Camacho Rodríguez < >>> jcama...@apache.org> wrote: >>> >>>> Currently, there is no established group discussions. The project was >>>> recently open-sourced, and communication is currently done through GitHub. >>>> (If the project is accepted into the ASF incubator, mailing lists will be >>>> created). If you're interested in regular meetings, feel free to suggest it >>>> to the community on GitHub. >>>> >>>> Thanks, >>>> Jesús >>>> >>>> >>>> On 2023/12/05 06:30:38 Gaurav Agarwal wrote: >>>> > HI >>>> > Thanks for this mail , I would like to know is there any group >>>> discussion >>>> > also happened or any call to discuss the issues. >>>> > >>>> > thanks >>>> > >>>> > >>>> > On Tue, Dec 5, 2023 at 9:29 AM Walaa Eldin Moustafa < >>>> wa.moust...@gmail.com> >>>> > wrote: >>>> > >>>> > > Thanks Jesus for sharing OneTable. Looks like it touches upon some >>>> of the >>>> > > topics we discussed in the Rise of Table Formats panel at VLDB >>>> > > <https://ceur-ws.org/Vol-3462/CDMS18.pdf> back in September. I was >>>> > > browsing through the source code, and I ran into the OneField >>>> > > < >>>> https://github.com/onetable-io/onetable/blob/main/api/src/main/java/io/onetable/model/schema/OneField.java> >>>> class >>>> > > and noticed it has support for default values, which is good, but >>>> in the >>>> > > Iceberg spec, there are two default values >>>> > > <https://iceberg.apache.org/spec/#default-values> (more details in >>>> the >>>> > > spec and respective PR). I was pointing this out as an example of >>>> small >>>> > > nuances that can differ from one format to another and was >>>> wondering how >>>> > > OneTable is planning to bridge them? >>>> > > >>>> > > Thanks, >>>> > > Walaa. >>>> > > >>>> > > >>>> > >>>> >>> >> >> -- >> Ryan Blue >> Tabular >> >
