Very excited for MV to be in Iceberg :)
Keeping in the same doc. would be helpful, to have the trail.
But also agreed, if there are too many directions/threads, then keep
closing the old one, if there are no more questions.
And put down the assumptions for the initial version to move forward.
On Tue, Feb 20, 2024 at 12:17 PM Walaa Eldin Moustafa
<wa.moust...@gmail.com> wrote:
I would vote to keep a log in the doc with open questions, and
keep the doc updated with open questions as they arise/get resolved.
On Tue, Feb 20, 2024 at 11:37 AM Jack Ye <yezhao...@gmail.com> wrote:
Thanks for the response from everyone!
Before proceeding further, I see a few people referring back
to the current design from Jan. I specifically raised this
thread based on the information in the doc and a few latest
discussions we had there. Because there are many threads in
the doc, and each thread points further to other discussion
threads in the same doc or other doc, it is now quite hard to
follow and continue discussing all different topics there.
I hope we can make incremental consensus of the questions in
the doc through devlist, because it provides more visibility,
and also a single thread instead of multiple threads going on
at the same time. If we think this format is not effective, I
propose that we create a new mv channel in Iceberg Slack
workspace, and people interested can join and discuss all
these points directly. What do we think?
Best,
Jack Ye
On Mon, Feb 19, 2024 at 6:03 PM Szehon Ho
<szehon.apa...@gmail.com> wrote:
Hi,
Great to see more discussion on the MV spec. Actually,
Jan's document "Iceberg Materialized View Spec"
<https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A>
has
been organized , with a "Design Questions" section to
track these debates, and it would be nice to centralize
the debates there, as Micah mentions.
For Dan's question, I think this debate was tracked in
"DesignQuestion 3: Should the storage table be registered
in the catalog?". I think the general idea there was to
not expose it directly via Catalog as it is then exposed
to user modification. If the engine wants to access
anything about the storage table (including audit and
storage), it is of course there via the storage table
pointer. I think Walaa's point is also good, we could
expose it as we expose metadata tables, but I am still not
sure if there is still some use-cases of engine access not
covered?
It is true that for Jack's initial question (Do we really
want to go with the MV = view + storage table design
approach for Iceberg MV?), unfortunately we did not
capture it as a "Design Question" in Jan's doc, as it was
an implicit assumption of 'yes', because it is the choice
of Hive, Trino, and other engines , as others have pointed
out.
Jack's point about potential evolution of MV (like to add
partitioning) is an interesting one, but definitely hard
to grasp. I think it makes sense to add this as a
separate Design Question in the doc, and add the options.
This will allow us to flesh out this alternative
option(s). Maybe Micah's point about modifying existing
proposal to 'embed' the required table metadata fields in
the existing view metadata, is one middle ground option.
Or we add a totally new MV object spec for MV, separate
than existing View spec?
Also , as Jack pointed out, it may make sense to have the
REST / Catalog API proposal in the doc to educate the
above decision.
Thanks
Szehon
On Mon, Feb 19, 2024 at 4:08 PM Walaa Eldin Moustafa
<wa.moust...@gmail.com> wrote:
I think it would help if we answer the question of
whether an MV is a view + storage table (and degree of
exposing this underlying implementation) in the
context of the user interfacing with those concepts:
For the end user, interfacing with the engine APIs
(e.g., through SQL), materialized view APIs should be
almost the same as regular view APIs (except for
operations specific to materialized views like REFRESH
command etc). Typically, the end user interacts with
the (materialized) view object as a view, and the
engine performs the abstraction over the storage table.
For the engines interfacing with Iceberg, it sounds
the correct abstraction at this layer is indeed view +
storage table, and engines could have access to both
objects to optimize queries.
So in a sense, the engine will ultimately hide most of
the storage detail from the end user (except for
advanced users who want to explicitly access the
storage table with a modifier like
"db.view.storageTable" -- and they can only read it),
while Iceberg will expose the storage details to the
engine catalog to use it in scans if needed. So the
storage table is hidden or exposed based on the
context/the actual users. From Iceberg point of view
(which interacts with the engines), the storage table
is exposed. Note that this does not necessarily mean
that the storage table is registered in the catalog
with its own independent name (e.g., where we can drop
the view but keep the storage table and access it from
the catalog). Addressing the storage table using a
virtual namespace like "db.view.storageTable" sounds
like a good middle ground. Anyways, end users should
not need to directly access the storage table in most
cases.
Thanks,
Walaa.
On Mon, Feb 19, 2024 at 3:38 PM Micah Kornfield
<emkornfi...@gmail.com> wrote:
Hi Jack,
In my mind, the first key point we all need to
agree upon to move this design forward is*: Do
we really want to go with the MV = view +
storage table design approach for Iceberg MV?*
I think we want this to the extent that we do not
want to redefine the same concept with different
representations/naming to the greatest degree
possible. This is why borrowing the concepts from
the view (e.g. multiple ways of expressing the
same view logic in different dialects) and aspects
of the materialized data (e.g. partitioning,
ordering) feels most natural. IIUC your proposal,
I think you are saying maybe two modifications to
the existing proposals in the document:
1. No separate storage table link, instead embed
most of the metadata of the materialized table
into the MV document (the exception seems to be
snapshot history)
2. For snapshot history, have one unified history
specific to the MV.
This seems fairly reasonable to me and I think I
can solve some challenges with the existing
proposal in an elegant way. If this is correct (or
maybe if it isn't quite correct) perhaps you can
make suggestions to the document so all of the
trade-offs can be discussed in one place?
I think the one thing the current draft of the
materialized view ignores is how to store
algebraic summaries (e.g. separate sum and count
for averages, or other sketches), so that new data
can be incrementally incorporated. But
representing these structures feels like it
potentially has value beyond just MVs (e.g. it can
be a natural way to express summary statistics in
table metadata), so I think it deserves at least a
try in incorporating the concepts in the table
specification, so the definitions can be shared.
I was imagining this could come as part of the
next revision of MV specification.
The MV internal structure could evolve in a
way that works more efficiently with the
reduced scope of functionalities, without
relying on table to offer the same
capabilities. I can at least say that is true
based on my internal knowledge of how Redshift
MVs work.
I'm not sure I fully understand this point, but it
seems the main question here is what would break
if it started to evolve in this direction. Is it
purely additive or do we suspect some elements
would need to be removed? My gut feeling here is
the main concerns here are getting the
cardinatities correct (i.e. 1 MV should probably
have 0, 1 or more materialized storage tables
associated with it, to support more advanced
algebraic structures listed above, and perhaps a
second without them, and additional metadata to
distinguish between these two different modes).
If after the evaluation, we are confident that
the MV = view + storage table approach is the
right way to go, then we can debate the other
issues, and I think the next issue to reach
consensus should be "Should the storage table
be registered in the catalog?".
I actually think there are actually more
fundamental questions posed:
1. Should be considering how items should be
modelled in the REST API concurrently with the
Iceberg spec, as that potentially impacts design
decision (I think the answer is yes, and we should
update the doc with sketches on new endpoints and
operations on the endpoints to ensure things align).
2. Going forward should new aspects of Iceberg
artifacts rely on the fact that a catalog is
present and we can rely on a naming convention for
looking up other artifacts in a catalog as
pointers (I lean yes on this, but I'm a little bit
more ambivalent).
Thanks,
Micah
On Mon, Feb 19, 2024 at 12:52 PM Jack Ye
<yezhao...@gmail.com> wrote:
I suggest we need a step-by-step process to
make incremental consensus, otherwise we are
constantly talking about many different
debates at the same time.
In my mind, the first key point we all need to
agree upon to move this design forward is*: Do
we really want to go with the MV = view +
storage table design approach for Iceberg MV?*
I think we (at least me) started with this
assumption, mostly because this is how Trino
implements MV, and how Hive tables store MV
information today. But does it mean we should
design it that way in Iceberg?
Now I look back at how we did the view spec
design, we could also say that we just add a
representation field in the table spec to
store view, and an Iceberg view is just a
table with no data but with representations
defined. But we did not do that. So it feels
now quite inconsistent to say we want to just
add a few fields in the table and view spec to
call it an Iceberg MV.
If we look into most of the other database
systems (e.g. Redshift, BigQuery, Snowflake),
they never expose such implementation details
like storage table. Apart from being
close-sourced systems, I think it is also for
good technical reasons. There are many more
things that a table needs to support, but does
not really apply to MV. The MV internal
structure could evolve in a way that works
more efficiently with the reduced scope of
functionalities, without relying on table to
offer the same capabilities. I can at least
say that is true based on my internal
knowledge of how Redshift MVs work.
I think we should fully evaluate both
directions, and commit to one first before
debating more things.
If we have a new and independent Iceberg MV
spec, then an Iceberg MV is under-the-hood a
single object containing all MV information.
It has its own name, snapshots, view
representation, etc. I don't believe we will
be blocked by Trino due to its MV SPIs
currently requiring the existence of a storage
table, as it will just be a different
implementation from the existing one in
Trino-Iceberg. In this direction, I don't
think we need to have any further debate about
pointers, metadata locations, storage table,
etc. because everything will be new.
If after the evaluation, we are confident that
the MV = view + storage table approach is the
right way to go, then we can debate the other
issues, and I think the next issue to reach
consensus should be "Should the storage table
be registered in the catalog?".
What do we think?
-Jack
On Mon, Feb 19, 2024 at 11:32 AM Daniel Weeks
<dwe...@apache.org> wrote:
Jack,
I think we should consider either allowing
the storage table to be fully
exposed/addressable via the catalog or
allow access via namespacing like with
metadata tables. E.g.
<catalog>.<database>.<table>.<storage>,
which would allow for full access to the
underlying table.
For other engines to interact with the
storage table (e.g. to execute the query
to materialize the table), it may be
necessary that the table is fully
addressable. Whether the storage table is
returned as part of list operations may be
something we leave up to the catalog
implementation.
I don't think the table should reference a
physical location (only a logical
reference) since things will be
changing behind the view definition and
I'm not confident we want to have to
update the view representation everytime
the storage table is updated.
I think there's still some exploration as
to whether we need to model this as
separate from view endpoints, but there
may be enough overlap that it's not
necessary to have yet another set of
endpoints for materialized views (maybe
filter params if you need to distinguish?).
-Dan
On Sun, Feb 18, 2024 at 6:57 PM Renjie Liu
<liurenjie2...@gmail.com> wrote:
Hi, Jack:
Thanks for raising this.
In most database systems, MV, view
and table are considered
independent objects, at least at
API level. It is very rare for a
system to support operations like
"materializing a logical view" or
"upgrading a logical view to MV",
because view and MV are very
different in almost every aspect
of user experience. Extending the
existing view or table spec to
accommodate MV might give us a MV
implementation similar to the
current Trino or Hive views, save
us some effort and a few APIs in
REST, but it binds us to a very
specific design of MV, which we
might regret in the future.
When I reviewed the doc, I thought we
were discussing the spec of
materialized view, just like the spec
of table metadata, but didn't not the
user facing api. I would definitely
agree that we should consider MV as
another kind of database object in
user facing api, even though it's
internally modelled as a view +
storage table pointer.
If we want to make the REST
experience good for MV, I think we
should at least consider directly
describing the full metadata of
the storage table in Iceberg view,
instead of pointing to a JSON file.
Do you mean we need to add components
like `LoadMaterializedViewResponse`,
if so, I would +1 for this.
*Q2: what REST APIs do we expect
to use for interactions with MVs?*
As I have mentioned above, I think we
should consider MV as another database
object, so I think we should add a set
of apis specifically designed for MV,
such as `loadMV`, `freshMV`.
On Sat, Feb 17, 2024 at 11:14 AM Jack
Ye <yezhao...@gmail.com> wrote:
Hi everyone,
As we are discussing the spec
change for materialized view,
there has been a missing aspect
that is technically also related,
and might affect the MV spec
design: *how do we want to add MV
support to the REST spec?*
*
*
I would like to discuss this in a
new thread to collect people's
thoughts. This topic expands to
the following 2 sub-questions:
*Q1: how would the MV spec change
affect the REST spec?*
In the current proposal, it looks
like we are using a design where a
MV is modeled as an Iceberg view
linking to an Iceberg storage
table. At the same time, we do not
want to expose this storage table
in the catalog, thus the Iceberg
view has a pointer to only a
metadata JSON file of the Iceberg
storage table. Each MV refresh
updates the pointer to a new
metadata JSON file.
I feel this does not play very
well with the direction that REST
is going. The REST catalog is
trying to remove the dependency to
the metadata JSON file. For
example, in LoadTableResponse the
only required field is the
metadata, and metadata-location is
actually optional.
If we want to make the REST
experience good for MV, I think we
should at least consider directly
describing the full metadata of
the storage table in Iceberg view,
instead of pointing to a JSON file.
*Q2: what REST APIs do we expect
to use for interactions with MVs?*
So far we have been thinking about
amending the view spec to
accommodate MV. This entails
likely having MVs also being
handled through the view APIs in
REST spec.
We need to agree with that first
in the community, because this has
various implications, and I am not
really sure at this point if it is
the best way to go.
If MV interactions are through the
view APIs, the view APIs need to
be updated to accommodate MV
constructs that are not really
related to logical views. In fact,
most actions performed on MVs are
more similar to actions performed
on table rather than view, which
involve configuring data layout,
read and write constructs. For
example, users might run something
like:
CREATE MATERIALIZED VIEW mv
PARTITION BY col1
CLUSTER BY col2
AS ( // some sql )
then the CreateView API needs to
accept partition spec and sort
order that are completely not
relevant for logical views.
When reading a MV, we might even
want to have a
PlanMaterializedView API similar
to the PlanTable API we are adding.
*My personal take*
It feels like we need to
reconsider the question of what is
the best way to model MV in
Iceberg. Should it be (1) a view
linked to a storage table, or (2)
a table with a view SQL associated
with it, or (3) it's a completely
independent thing. This topic was
discussed in the past in this doc
<https://docs.google.com/document/d/1QAuy-meSZ6Oy37iPym8sV_n7R2yKZOHunVR-ZWhhZ6Q/edit?pli=1>,
but at that time we did not have
much perspective about aspects
like REST spec, and the view
integration was also not fully
completed yet. With the new
knowledge, currently I am actually
leaning a bit more towards (3).
In most database systems, MV, view
and table are considered
independent objects, at least at
API level. It is very rare for a
system to support operations like
"materializing a logical view" or
"upgrading a logical view to MV",
because view and MV are very
different in almost every aspect
of user experience. Extending the
existing view or table spec to
accommodate MV might give us a MV
implementation similar to the
current Trino or Hive views, save
us some effort and a few APIs in
REST, but it binds us to a very
specific design of MV, which we
might regret in the future.
If we make a new MV spec, it can
be made up of fields that already
exist in the table and view specs,
but it is a whole new spec. In
this way, the spec can evolve
independently to accommodate MV
specific features, and we can also
create MV-related REST endpoints
that will evolve independently
from table and view REST APIs.
But on the other side it is
definitely associated with more
work to maintain a new spec, and
potentially big refactoring in the
codebase to make sure operations
today that work on table or view
can now support MV as a different
object. And it definitely has
other problems that I have
overlooked. I would greatly
appreciate any thoughts about this!
Best,
Jack Ye
