Thanks Ryan! Yes there is an active discussion on the PR on the spec aspect.
On Fri, Feb 11, 2022 at 8:47 AM Ryan Blue <b...@tabular.io> wrote: > Sounds great. Thanks for the update! That PR is on my list to take a look > at, but I still recommend starting with the spec changes. For example, how > should default values be stored in Iceberg metadata for each type? > Currently, the spec changes just mention defaults without going into detail > about how they are tracked and what rules there are about them. > > On Wed, Feb 9, 2022 at 6:32 PM Walaa Eldin Moustafa <wa.moust...@gmail.com> > wrote: > >> Thanks Ryan and Owen! Glad we have converged on this. Next steps for us: >> >> * Continuing the discussion on the default value PR (already ongoing [1]). >> * Filing the union type conversion PR (ETA end of next week). >> * Moving listing-based Hive table scan using Iceberg to a separate repo >> (likely open source). For this I expect introducing some extension points >> to Iceberg such as making some classes SPI. I hope that the community is >> okay with that. >> >> By the way, Owen and I synced on the Hive casing behavior, and it is a >> bit more involved: Hive lowers the schema case for all fields (including >> nested fields) in the Avro case, but only lowers top-level field case and >> preserves inner field case for other formats (we experimented with ORC and >> Text). Hope this clarifies the confusion. >> >> [1] https://github.com/apache/iceberg/pull/2496 >> >> Thanks, >> Walaa. >> >> >> >> On Wed, Feb 2, 2022 at 2:40 PM Ryan Blue <b...@tabular.io> wrote: >> >>> Walaa, thanks for this list. I think most of these are definitely >>> useful. I think the best one to focus on first is the default values, since >>> those will make Iceberg tables behave more like standard SQL tables, which >>> is the goal. >>> >>> I'm really curious to learn more about #1, but I don't think that I have >>> enough detail to know whether it is something that fits in the Iceberg >>> project. At Netflix, we had an alternative implementation of Hive and Spark >>> tables (Spark tables are slightly different) that we similarly used. But we >>> didn't write to both at the same time. >>> >>> For the others, I'm interested in hearing what other people in the >>> community find valuable. I don't think I would use #2 or #3, for example. >>> That's because we already support a flag for case insensitive column >>> resolution that is well supported throughout Iceberg. If you wanted to use >>> alternative names, then I'd probably recommend just turning that on... >>> although that may not be an option depending on how you're working with a >>> table. It would work in Spark, though. This may be a better feature for >>> your system that is built on Iceberg. >>> >>> Reading unions as structs has come up a couple times so that seems like >>> people will want it. I think someone attempted to add this support in the >>> past, but ran into issues because the spec is clear that these are NOT >>> Iceberg files. There is no guarantee that other implementations will read >>> them and Iceberg cannot write them in this form. I'm fairly confident that >>> not allowing unions to be written is a good choice, but I would support >>> being able to read them. >>> >>> Ryan >>> >>> On Mon, Jan 31, 2022 at 4:32 PM Owen O'Malley <owen.omal...@gmail.com> >>> wrote: >>> >>>> >>>> >>>> On Thu, Jan 27, 2022 at 10:26 PM Walaa Eldin Moustafa < >>>> wa.moust...@gmail.com> wrote: >>>> >>>>> *2. Iceberg schema lower casing:* Before Iceberg, when users read >>>>> Hive tables from Spark, the returned schema is lowercase since Hive stores >>>>> all metadata in lowercase mode. If users move to Iceberg, such readers >>>>> could break once Iceberg returns proper case schema. This feature is to >>>>> add >>>>> lowercasing for backward compatibility with existing scripts. This feature >>>>> is added as an option and is not enabled by default. >>>>> >>>> >>>> This isn't quite correct. Hive lowercases top-level columns. It does >>>> not lowercase field names inside structs. >>>> >>>> >>>>> *3. Hive table proper casing:* conversely, we leverage the Avro >>>>> schema to supplement the lower case Hive schema when reading Hive tables. >>>>> This is useful if someone wants to still get proper cased schemas while >>>>> still in the Hive mode (to be forward-compatible with Iceberg). The same >>>>> flag used in (2) is used here. >>>>> >>>> >>>> Are there users of Avro schemas in Hive outside of LinkedIn? I've never >>>> seen it used. I don't think you should tie #2 and #3 together. >>>> >>>> Supporting default values and union types are useful extensions. >>>> >>>> .. Owen >>>> >>> >>> >>> -- >>> Ryan Blue >>> Tabular >>> >> > > -- > Ryan Blue > Tabular >
