Hi Marc,
Could you clarify where you store the view definitions in this case, and
how the syntax looks like?
Thanks,
Walaa.
On Tue, Nov 15, 2022 at 2:34 PM Ryan Blue wrote:
> Hi Marc,
>
> This is expected. Although the ViewCatalog SPIP was approved by the Spark
> community, the implementation
or the Hive table resolution) to route to V2 and Iceberg
readers when applicable; hence we could switch between the Hive and Iceberg
versions of the table transparently.
Thanks,
Walaa.
On Tue, Nov 15, 2022 at 2:38 PM Walaa Eldin Moustafa
wrote:
> Hi Marc,
>
> Could you clarify wher
ing catalog & namespace). Our iceberg tables are backed by a HMS
> so it would presumably be stored there?
>
> Thanks again for your responses!
>
> On Tue, Nov 15, 2022 at 5:38 PM Walaa Eldin Moustafa <
> wa.moust...@gmail.com> wrote:
>
>> Hi Marc,
>>
>>
+1 (non-binding)
On Wed, Aug 28, 2024 at 9:36 AM Steven Wu wrote:
> +1 (binding)
>
> On Wed, Aug 28, 2024 at 9:29 AM Micah Kornfield
> wrote:
>
>> I propose to merge https://github.com/apache/iceberg/pull/10780 as a
>> starting place for describing community norms around merging/discussing PRs.
Hi Jan,
I think we need to close the discussion on the UUID vs table identifier
options and possibly cast a vote before having a productive discussion on
the PR. I did not get a chance yet to post the document on the UUID vs
table identifier discussion. I will do that by next week.
Thanks,
Walaa.
> participating in the discussion have a similar stance.
>
> I'm not sure if we really need another document for the discussion.
>
> Best wishes,
>
> Jan
> On 29.08.24 21:10, Walaa Eldin Moustafa wrote:
>
> Hi Jan,
>
> I think we need to close the discussion
sed my comment on UUID.
>
> On Mon, Sep 2, 2024 at 4:35 PM Walaa Eldin Moustafa
> wrote:
>
>> Hi Jan,
>>
>> I think we need further discussion for a few reasons:
>>
>> * Semantic issues are significant, especially in a spec. They could
>> create gaps that
ons like RTAS or Drop/Create on tables
> referenced by a view, #1 might not work. My understanding is that most
> database systems wouldn't allow that, but it could also be hard to enforce
> without a more complicated catalog implementation.
>
> -Dan
>
> On Tue, Sep 3, 2024 at
t make sense then to store the lineage as part of the
>> representation:
>>
>> {
>>
>> "type": "sql",
>>
>> "sql": "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP
>> BY 2",
>>
>>
d in a federated catalog. the references to source tables should
> include the source catalog names, which are standardized by the federated
> catalog.
>
> Thanks,
> Steven
>
>
> On Mon, Oct 7, 2024 at 11:16 PM Walaa Eldin Moustafa <
> wa.moust...@gmail.com> wrote:
&g
uld be
> consistent across engines. catalog federation can achieve the
> normalization/standardization of the catalog names
>
>
>
> On Tue, Oct 8, 2024 at 6:17 PM Walaa Eldin Moustafa
> wrote:
>
>> Just opened Comment access to the doc. Link here again for convenience
&g
Hi Dan,
I think there are a few questions that we should solve to decide the path
forward:
** Does the current spec contain implicit assumptions?*
I think the answer is yes. I think this is also what Ryan indicated here
[1].
** Do these implicit assumptions make it difficult to adopt the spec or
es set depending on the
> USE context. If Dremio intended for that view to be readable by Spark, it
> would have to adhere to all those restrictions I listed before.
>
> Thanks
> Benny
>
> On Fri, Oct 11, 2024 at 10:00 AM Walaa Eldin Moustafa <
> wa.moust...@gmail.
onfusing to people, I would be open to
> discussing an "unsupported configurations" callout, but I don't think this
> blocks work and am somewhat skeptical that it's necessary.
>
> -Dan
>
>
>
> On Thu, Oct 10, 2024 at 2:47 PM Walaa Eldin Moustafa <
&g
sing to people, I would be open to
>> discussing an "unsupported configurations" callout, but I don't think this
>> blocks work and am somewhat skeptical that it's necessary.
>>
>> -Dan
>>
>>
>>
>> On Thu, Oct 10, 2024
Just opened Comment access to the doc. Link here again for convenience [1].
[1]
https://docs.google.com/document/d/1e5orD_sBv0VlNNLZRgUtalVUllGuztnAGTtqo8J0UG8/edit
Thanks,
Walaa.
On Tue, Oct 8, 2024 at 10:42 AM Walaa Eldin Moustafa
wrote:
> Thanks Steven! I think this fits in the framew
Benny, "Iceberg View Spec Improvements" includes documenting what is
supported and what is not. You listed a few restrictions. Many of them are
not documented on the current spec. Documenting them is what this thread is
about. We are trying to reach a consensus on the necessary constraints (so
we a
Hi Everyone,
Thanks for all the discussion so far. I have created a PR to document the
requirements https://github.com/apache/iceberg/pull/11365. Please feel free
to review it or discuss further in the thread.
Thanks,
Walaa.
On Fri, Oct 11, 2024 at 5:19 PM Walaa Eldin Moustafa
wrote:
>
Thanks Ryan and everyone who left feedback on the doc. Let me clarify a few
things.
"Improving the spec" also includes making the implicit assumptions
explicitly stated in the spec.
Explicitly stating the assumptions is discussed under the "Portable table
identifiers" section in the doc. I am onb
Hi Everyone,
As part of our discussions on the Materialized View (MV) spec, the topic of
"SQL table identifiers" has been a constant source of complexity. After
several iterations, the community has agreed not to use SQL table
identifiers in the table-side representation of MVs. However, that stil
a
> dataframe-based world. Walaa, do you have any outlook on Coral
> Python/Rust/Go support?
>
> Kind regards,
> Fokko
>
>
> Op vr 25 okt 2024 om 22:16 schreef Walaa Eldin Moustafa <
> wa.moust...@gmail.com>:
>
>> I think this may need some more discussion.
t;> on how to resolve this. If needed, we can schedule another online meeting
>> to discuss in person.
>>
>> Best regards, Jan
>>
>> 1.
>> https://lists.apache.org/list?dev@iceberg.apache.org:lte=1M:view%20spec%20improvements
>> On 14.10.24 23:31, Walaa
Hi Russel,
I am not sure if materialized views change the table spec. Table spec
remains largely the same when it comes to handling MVs. Further, I think
there may be unresolved questions on the view part before proceeding with
the MV spec (namely on catalog names).
There is also a PR on default
be based on *engine agnostic*
>>> identifiers without the catalog name. The MV and storage table would have
>>> to be in the same catalog.
>>>
>>> Thanks
>>> Benny
>>>
>>>
>>>
>>> On Fri, Sep 13, 2024 at 2:08 AM Jan Kaul
I think this may need some more discussion.
To me, a "serialized IR" is another form of a "dialect". In this case, this
dialect will be mostly specific to Iceberg, and compute engines will still
support reading views in their native SQL. There are some data points on
this from the Trino community
gt;> work with the respective IR community to help close those gaps or if needed
>>> consider other paths.
>>> >>
>>> >> Thanks,
>>> >> Amogh Jahagirdar
>>> >>
>>> >> On Mon, Oct 28, 2024 at 11:43 AM Piotr Fin
Thanks Manu for starting this discussion. That is definitely a valid
feature. I have always found maintaining snapshots by day makes it harder
to provide different types of guarantees/contracts especially when tables
change rates are diverse or irregular. Maintaining by snapshot count makes
a lot o
Hi Ryan,
For Table Format V3, we could point out that the default value support for
Avro has been merged and support for other formats is ongoing.
Thanks,
Walaa.
On Wed, Dec 11, 2024 at 12:51 PM rdb...@gmail.com wrote:
> Hi everyone,
>
> It’s time to report to the board again. Great to see al
101 - 128 of 128 matches
Mail list logo
