Benny, thank you so much for performing the experiment.
Glad that using UUIDs as keys in the state map makes
more sense now.
For the issue with the view spec being restrictive, I
agree and I have raised the concern on the view spec PR
last year [1]. I think there is some area of
improvement here. At the least, if it is restrictive,
it should be explicitly stated. I will start a thread
on how to approach the view spec. We may need to get
more insight on the view spec before finalizing the MV
spec, because view spec will determine if we should
proceed with one lineage (with the implicitly assumed
common catalog name), or with multiple lineages (one
per engine or catalog name).
[1]
https://github.com/apache/iceberg/pull/7992#issuecomment-1763172619
Thanks,
Walaa.
On Mon, Sep 9, 2024 at 3:28 AM Benny Chow
<btc...@gmail.com> wrote:
Hi Walaa
I did some testing with two different engines
(Spark and Dremio) against the same Nessie catalog
and created the attached materialized view
metadata.json. I see your point now about the SQL
identifiers being tightly coupled to the engines.
In the metadata JSON, spark refers to the catalog
as "SparkNessie", whereas Dremio refers to the
catalog as "LocalNessie". So, this means that the
fully qualified view and table identifiers are
engine specific and Dremio can't lookup a Spark
identifier and vice versa.
*So, I think it does make sense now for the
refresh-state to key off the UUIDs and not use
engine specific identifiers. *This also means that
the materization consumer will have to fully expand
the query tree and basically diff the UUID + latest
snapshot ids against the refresh state. Would it
ever make sense for the Iceberg Catalog to expose a
bulk lookup API by UUID?
As a side note, it seems that for a materialized
view to work with multiple engines, the
default-catalog and default-namespace can't be used
unless both engines use the same catalog name which
seems pretty restrictive to me.
Thanks for the great discussions
Benny
On Sat, Sep 7, 2024 at 2:49 AM Walaa Eldin Moustafa
<wa.moust...@gmail.com> wrote:
Jan, we definitely can store SQL identifiers of
multiple representations in Approach 1.
The takeaway is that SQL identifiers are highly
coupled with engines, just like views. It makes
sense to track both together for consistency.
Thanks,
Walaa.
On Sat, Sep 7, 2024 at 8:15 AM Jan Kaul
<jank...@mailbox.org.invalid>
<mailto:jank...@mailbox.org.invalid> wrote:
Walaa, thanks you for bringing up this use
case. I think we need to keep in mind that
we require identifiers to interface with
the catalog. We cannot use UUIDs.
Which means you also wouldn't be able to
use Approach 1 for your use case because
you can't store the catalog names of
multiple representations in the lineage.
You would need to fallback to parsing the
SQL for a particular representation and
rebuilding the full query tree to obtain
the identifiers.
You could do the same for Approach 2. So I
don't see why Approach 1 would yield any
benefits.
Thanks,
Jan
On 07.09.24 00:01, Steven Wu wrote:
Benny, `default-catalog` is optional,
while `default-namespace` is required.
I will retract my comment on the
`summary`. it indicates the engine that
made the revision to the current view
version. it doesn't really matter for
multi-engine/representation support.
On Fri, Sep 6, 2024 at 2:49 PM Benny Chow
<btc...@gmail.com> wrote:
Steven - Ideally, the lineage is
engine agnostic so I'd hope it
wouldn't have to be under a specific
representation.
Walaa - That's a serious concern...
If the same catalog is aliased
differently by two different engines,
then the basic view spec seems broken
to me since "default-namespace"
includes the catalog alias and is
outside of the SQL representation.
Does that mean for a view to be
interoperable, we require different
engines to use the same catalog name?
Thanks
On Fri, Sep 6, 2024 at 1:29 PM Steven
Wu <stevenz...@gmail.com> wrote:
Walaa, thanks for bringing up the
interesting case of multiple
representations (for different
engines), which definitely
requires more discussion from the
community.
When I am looking at the view
spec, I am seeing some conflict.
"summary" field seems meant for
only one engine, while
"representations" support multiple
engines.
"summary" : {
<https://iceberg.apache.org/view-spec/#__codelineno-5-16>"engine-name"
: "Spark",
<https://iceberg.apache.org/view-spec/#__codelineno-5-17>"engineVersion"
: "3.3.2"
<https://iceberg.apache.org/view-spec/#__codelineno-5-18>},
<https://iceberg.apache.org/view-spec/#__codelineno-5-19>"representations"
: [ {
<https://iceberg.apache.org/view-spec/#__codelineno-5-20>"type"
: "sql",
<https://iceberg.apache.org/view-spec/#__codelineno-5-21>"sql"
: "SELECT\n COUNT(1),
CAST(event_ts AS DATE)\nFROM
events\nGROUP BY 2",
<https://iceberg.apache.org/view-spec/#__codelineno-5-22>"dialect"
: "spark"
<https://iceberg.apache.org/view-spec/#__codelineno-5-23>}
]
With multiple
representations/engines, I guess
one engine will be responsible for
the storage table refresh and
other engines are read only. If we
want to store the lineage info in
the view, it probably needs to be
part of the "representation"
struct so that each
engine/representation stores its
own lineage info..
Who is to validate/ensure that the
SQL representation is actually
semantically identical (minus
syntax differences across
engines)? I guess this
responsibility is left to the user
who owns and manages the view.
