Benny, thank you so much for performing the experiment. Glad
that using UUIDs as keys in the state map makes more sense now.
For the issue with the view spec being restrictive, I agree
and I have raised the concern on the view spec PR last year
[1]. I think there is some area of improvement here. At the
least, if it is restrictive, it should be explicitly stated.
I will start a thread on how to approach the view spec. We
may need to get more insight on the view spec before
finalizing the MV spec, because view spec will determine if
we should proceed with one lineage (with the implicitly
assumed common catalog name), or with multiple lineages (one
per engine or catalog name).
[1]
https://github.com/apache/iceberg/pull/7992#issuecomment-1763172619
Thanks,
Walaa.
On Mon, Sep 9, 2024 at 3:28 AM Benny Chow <btc...@gmail.com>
wrote:
Hi Walaa
I did some testing with two different engines (Spark and
Dremio) against the same Nessie catalog and created the
attached materialized view metadata.json. I see your
point now about the SQL identifiers being tightly
coupled to the engines. In the metadata JSON, spark
refers to the catalog as "SparkNessie", whereas Dremio
refers to the catalog as "LocalNessie". So, this means
that the fully qualified view and table identifiers are
engine specific and Dremio can't lookup a Spark
identifier and vice versa.
*So, I think it does make sense now for the
refresh-state to key off the UUIDs and not use engine
specific identifiers. *This also means that the
materization consumer will have to fully expand the
query tree and basically diff the UUID + latest snapshot
ids against the refresh state. Would it ever make sense
for the Iceberg Catalog to expose a bulk lookup API by UUID?
As a side note, it seems that for a materialized view to
work with multiple engines, the default-catalog and
default-namespace can't be used unless both engines use
the same catalog name which seems pretty restrictive to me.
Thanks for the great discussions
Benny
On Sat, Sep 7, 2024 at 2:49 AM Walaa Eldin Moustafa
<wa.moust...@gmail.com> wrote:
Jan, we definitely can store SQL identifiers of
multiple representations in Approach 1.
The takeaway is that SQL identifiers are highly
coupled with engines, just like views. It makes
sense to track both together for consistency.
Thanks,
Walaa.
On Sat, Sep 7, 2024 at 8:15 AM Jan Kaul
<jank...@mailbox.org.invalid>
<mailto:jank...@mailbox.org.invalid> wrote:
Walaa, thanks you for bringing up this use case.
I think we need to keep in mind that we require
identifiers to interface with the catalog. We
cannot use UUIDs.
Which means you also wouldn't be able to use
Approach 1 for your use case because you can't
store the catalog names of multiple
representations in the lineage. You would need
to fallback to parsing the SQL for a particular
representation and rebuilding the full query
tree to obtain the identifiers.
You could do the same for Approach 2. So I don't
see why Approach 1 would yield any benefits.
Thanks,
Jan
On 07.09.24 00:01, Steven Wu wrote:
Benny, `default-catalog` is optional, while
`default-namespace` is required.
I will retract my comment on the `summary`. it
indicates the engine that made the revision to
the current view version. it doesn't really
matter for multi-engine/representation support.
On Fri, Sep 6, 2024 at 2:49 PM Benny Chow
<btc...@gmail.com> wrote:
Steven - Ideally, the lineage is engine
agnostic so I'd hope it wouldn't have to be
under a specific representation.
Walaa - That's a serious concern... If the
same catalog is aliased differently by two
different engines, then the basic view spec
seems broken to me since
"default-namespace" includes the catalog
alias and is outside of the SQL
representation. Does that mean for a view
to be interoperable, we require different
engines to use the same catalog name?
Thanks
On Fri, Sep 6, 2024 at 1:29 PM Steven Wu
<stevenz...@gmail.com> wrote:
Walaa, thanks for bringing up the
interesting case of multiple
representations (for different
engines), which definitely requires
more discussion from the community.
When I am looking at the view spec, I
am seeing some conflict. "summary"
field seems meant for only one engine,
while "representations" support
multiple engines.
"summary" : {
<https://iceberg.apache.org/view-spec/#__codelineno-5-16>"engine-name"
: "Spark",
<https://iceberg.apache.org/view-spec/#__codelineno-5-17>"engineVersion"
: "3.3.2"
<https://iceberg.apache.org/view-spec/#__codelineno-5-18>},
<https://iceberg.apache.org/view-spec/#__codelineno-5-19>"representations"
: [ {
<https://iceberg.apache.org/view-spec/#__codelineno-5-20>"type"
: "sql",
<https://iceberg.apache.org/view-spec/#__codelineno-5-21>"sql"
: "SELECT\n COUNT(1), CAST(event_ts AS
DATE)\nFROM events\nGROUP BY 2",
<https://iceberg.apache.org/view-spec/#__codelineno-5-22>"dialect"
: "spark"
<https://iceberg.apache.org/view-spec/#__codelineno-5-23>}
]
With multiple representations/engines,
I guess one engine will be responsible
for the storage table refresh and other
engines are read only. If we want to
store the lineage info in the view, it
probably needs to be part of the
"representation" struct so that each
engine/representation stores its own
lineage info..
Who is to validate/ensure that the SQL
representation is actually semantically
identical (minus syntax differences
across engines)? I guess this
responsibility is left to the user who
owns and manages the view.
