The option of using catalog identifiers in the state map still
requires keeping lineage information in the view because REFRESH MV
needs the latest fully expanded children (which could have changed
from the set of children currently in the state map),
without reparsing the view tree. Therefore, catalog identifiers in the
state map, does not eliminate the need for tracking children in the
form of catalog identifiers in the lineage side (but in this case
lineage will be a set instead of just a map).
Hence, my concerns with using catalog identifiers (as opposed to
UUIDs) are:
* The fundamental issue where the table spec depends on/refers to the
view spec (because such catalog identifiers are not defined in the
table spec and the only place they have a meaning is in the view spec
lineage information).
* (less fundamental) The denormalization introduced by this
arrangement, where each identifier is 3-parts and all of them repeat
in both lineage info and state map.
I am not very concerned with recursive expansion (through multiple
calls), as it is always the case with views.
On a positive note, looks like we agree to move past sequence numbers :)
Thanks,
Walaa.
On Thu, Aug 15, 2024 at 4:07 PM Micah Kornfield
<emkornfi...@gmail.com> wrote:
I think given the constraint that catalog lookup has to be by
identifier and not UUID, I'd prefer using identifier in the
refresh state. If we use identifiers, we can directly
parallelize the catalog calls to fetch the latest state. If we
use UUID, the engine has to go back to the MV and possibly
additional views to reconstruct the lineage map. It's just a
lot slower and more work for the engine when there is a MV
that references a lot of views (and those views reference
additional views).
I'm +1 on using catalog identifiers as the key. As you point out
this is inline with #2 (try to minimize serial catalog lookups) in
addition to supporting requirement #3.
On Thu, Aug 15, 2024 at 3:27 PM Benny Chow <btc...@gmail.com> wrote:
I think given the constraint that catalog lookup has to be by
identifier and not UUID, I'd prefer using identifier in the
refresh state. If we use identifiers, we can directly
parallelize the catalog calls to fetch the latest state. If
we use UUID, the engine has to go back to the MV and possibly
additional views to reconstruct the lineage map. It's just a
lot slower and more work for the engine when there is a MV
that references a lot of views (and those views reference
additional views).
Thanks
Benny
On Thu, Aug 15, 2024 at 2:14 PM Walaa Eldin Moustafa
<wa.moust...@gmail.com> wrote:
Thanks Jan, Micah, and Karuppayya for chiming in.
I do not think 3 and 4 are at odds with each other (for
example maintaining both lineage map and state map through
UUID can achieve both). Also, I do not think we can drop
the lineage map since in many catalogs, the only lookup
method is by the catalog identifier, and not the UUID.
I think if we go with UUIDs in the state, we should have a
lineage map (from identifiers to UUIDs) to go with it.
Thanks,
Walaa.
On Thu, Aug 15, 2024 at 1:45 PM karuppayya
<karuppayya1...@gmail.com> wrote:
+1 to storing the refresh state as a map of UUIDs to
snapshot IDs, and deferring the inclusion of lineage
to a future iteration.(like Micha mentioned)
This would greatly simplify the current design.
Also in terms of identifiers to use(UUID or catalog
identifier) for the refresh state
We will not be able to fetch the table/View using the
UUID alone, for example from Hive based catalog.
We do not have the direct mapping between UUID and
table/view.
Which leaves us only with the catalog identifiers?
Thanks & Regards
Karuppayya
On Thu, Aug 15, 2024 at 9:16 AM Micah Kornfield
<emkornfi...@gmail.com> wrote:
I think it might be worth restating perceived
requirements and making sure there is alignment on
them.
If I am reading correctly, I think the following
are perceived requirements:
1. An engine must be able to unambiguously detect
that an underlying queried entity has changed or
not via metadata to decide if materialized table
data can be used.
2. The number of sequential catalog reads an
engine needs to make to make use of a materialized
table state at read time is minimized.
3. Engines that don't understand a SQL dialect can
still use MV information if it is not stale.
4. Table refs (catalog identifiers) should not
appear in the materialized table metadata (i.e.
state).
5. The view part of the MV definition should not
need a new revision for any changes to objects it
queries as long as their schemas stay compatible
(only state information on the materialized table
need to change).
In my mind, requirement 1, is the only true
requirement. I think this necessitates having
UUID + snapshot ID as part of the state
information (not necessarily part of the
Lineage). I think it also necessitates having a
denormalized view of all entities that are inputs
into the MV in the state information (a view
object might not change but its underlying tables
or views could change and that must be detected).
Requirements 2 and 5 are somewhat at odds with
each other. If information is denormalized (fully
expanded) in Lineage, it means if table
information is somehow dropped from an
intermediate view, one would need to update the
view (or make excess calls to the catalog). In my
mind, this argues for normalization of the lineage
stored on the view (with the cost of potentially 1
additional serial catalog lookup once the state
information is retrieved).
I think #3 is at odds with #4. I think #3 is more
worthwhile, then keeping #4 (and as Jan noted #4
adds complexity).
I think the last remaining question is if lineage
serves any purpose. I think it is useful for the
following reasons:
a) When there are no intermediate views queried,
it allows for fully parallelized lookup calls to
the catalog without having to parse the SQL
statement first
b) Allows tools that don't need to lookup state
information or parse SQL but still navigate
MV/view trees.
Both of these seem relatively minor, so lineage
could perhaps be left out in the first iteration.
As it applies to Jan's questions:
1. Should we move the identifiers out of the
refresh-state into a new lineage record that
is stored as part of the view metadata?
No, I don't think so, I think #5 is a reasonable
requirement and I think this violates it.
2. If yes, should the lineage in the view be
fully expanded?
No, I think only the state should be fully
expanded (for reasons mentioned above, it
potentially requires more updates to the view then
necessary).
3. What should be used as an identifier in the
lineage to reference entries in the refresh-state?
Catalog identifiers make sense to me. If we agree
requirement #3 is not a requirement then it seems
like this could also be UUIDs.
Thanks,
Micah
On Thu, Aug 15, 2024 at 7:57 AM Benny Chow
<btc...@gmail.com> wrote:
If we go with either UUID or Table
Identifier + VersionID/SnapshotId in the
refresh state, then this list is fully
expanded already. So, to validate the
freshness of a materialization, the engine
doesn't even need to look at the view
lineage. IMO, the view lineage is nice to
have but not a necessary requirement for MVs.
The view lineage makes sharing of views
between engines without common SQL dialects
possible.
Benny
On Thu, Aug 15, 2024 at 12:22 AM Jan Kaul
<jank...@mailbox.org.invalid> wrote:
Hi all,
I would like to reemphasize the purpose of
the refresh-state for materialized views.
The purpose is to determine if the
precomputed data is fresh, stale or
invalid. For that the current snapshot-id
of every table in the query tree has to be
fetched from the catalog by using its full
identifier and ref. Additionally the
refresh state stores the snapshot-id of
the last refresh.
To summarize: *To determine the freshness
of the precomputed data we require the
full identifier + ref and snapshot-id of
the last refresh for every table in the
fully expanded query tree*
This is a requirement from how the catalog
works and independent from how we design
the lineage/refresh state. Additionally we
previously agreed that we should be able
to obtain the full list of identifiers
without needing to parse the SQL definition.
Now we are having a discussion in how to
store and obtain the fully expanded list
of table identifiers and snapshot-ids. To
move the discussion forward I think it
would be valuable to answer the following
3 questions:
1. Should we move the identifiers out of
the refresh-state into a new lineage
record that is stored as part of the view
metadata?
2. If yes, should the lineage in the view
be fully expanded?
3. What should be used as an identifier in
the lineage to reference entries in the
refresh-state?
1. Question:
We already agreed that this would be a
good idea because we wouldn't introduce
the identifier concept to the table
metadata. However, looking at the
complexity that comes with the
alternatives, I would like to keep this
question open.
2. Question:
I'm against using a not fully expanded
lineage in the view struct. To recall we
require every identifier in the fully
expanded query tree to determine the
freshness. Not storing all identifiers in
the lineage would mean to recursively call
the catalog and expand the query tree at
read time. This can lead to a large
overhead for determining the refresh state
compared to expanding the query tree once
at creation time and then storing the
fully expanded lineage.
3. Question:
This depends on Question 2.
For a not fully expanded lineage, the only
options would be uuids or catalog identifiers.
For a fully expanded lineage the question
isn't all that relevant. The current
design specifies that the lineage is a map
from an identifier to an id and the
refresh-state is a map from such id to a
snapshot-id. For this to work we don't
have to specify which kind of identifier
has to be used. One query engine could use
uuids, the other engine sequence-ids. The
important assumption we are making is that
every id that is used in the refresh-state
has to be defined in the lineage.
So the question about using uuids is
rather, can the query engine trust that
the id defined in the lineage is the uuid
of the table.
Regarding the complexity that comes from
introducing the lineage in the view I
would like to revisit question 1.
Introducing the lineage in the view
metadata opens up the question of when
should the lineage be fully expanded. We
see that we have 3 options:
1. Not fully expanded lineage -> Expansion
at read time
2. Fully expanded lineage -> Expansion at
creation time
3. No lineage (use identifiers in
refresh-state) -> Expansion at refresh time
As reading is expected to be the most
frequent operation I see option 1 as not
favorable. As the query engine has to
fully expand the query tree for a refresh
anyway, I see option 3 as the most
natural. For a refresh operation the query
engine must understand the SQL dialects of
all views in the query tree and therefore
is guaranteed to successfully expand the
lineage. This might not be the case at
creation time, which makes option 2 less
favorable.
As can be seen, I'm in favor of just
storing the refresh-state as a map from
identifier to snapshot-id and not using
the lineage. I know that this introduces
the concept of a catalog identifiers to
the table metadata spec, but in my opinion
it is by far the simplest option.
I'm interested in your opinions.
Best wishes,
Jan
On 14.08.24 22:24, Walaa Eldin Moustafa wrote:
Thanks Benny. For refs, I am +1 to
represent them as UUID + optional ref,
although we can iterate ohe exact JSON
structure (e.g., another option is
splitting for (UUID) state from (UUID +
ref) state into two separate higher-level
fields).
Generally agree on REFRESH VIEW strategy
could be up to the engine, but it seems
like an area where Iceberg could have an
opinion/spec on. I will start a separate
thread for that.
Thanks,
Walaa.
