Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Thanks Martijn. Regards Venkata krishnan On Fri, Sep 22, 2023 at 1:51 AM Martijn Visser wrote: > Sure thing, I've made the necessary changes. Thnx for clarifying > > On Thu, Sep 21, 2023 at 8:24 PM Venkatakrishnan Sowrirajan > wrote: > > > > Got it, Martijn. > > > > Unfortunately, I don't hav

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Sure thing, I've made the necessary changes. Thnx for clarifying On Thu, Sep 21, 2023 at 8:24 PM Venkatakrishnan Sowrirajan wrote: > > Got it, Martijn. > > Unfortunately, I don't have edit access to the already created JIRA - > FLINK-20767 . If y

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Got it, Martijn. Unfortunately, I don't have edit access to the already created JIRA - FLINK-20767 . If you can remove the task from the EPIC FLINK-16987 FLIP-95: Add new table source and sink interfaces

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Hi Venkatakrishnan, The reason why I thought it's abandoned because the Jira ticket is part of the umbrella ticket for FLIP-95. Let's move the Jira ticket to its own dedicated task instead of nested to a FLIP-95 ticket. Thanks, Martijn On Wed, Sep 20, 2023 at 4:34 PM Becket Qin wrote: > > Hi M

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Hi Martijn, This FLIP has passed voting[1]. It is a modification on top of the FLIP-95 interface. Thanks, Jiangjie (Becket) Qin [1] https://lists.apache.org/thread/hysv9y1f48gtpr5vx3x40wtjb6cp9ky6 On Wed, Sep 20, 2023 at 9:29 PM Martijn Visser wrote: > For clarity purposes, this FLIP is bein

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

No, this is not abandoned. This is accepted with enough binding votes. I didn't get why you think that this FLIP is abandoned. Could you please clarify? On Wed, Sep 20, 2023, 6:11 AM Martijn Visser wrote: > For clarity purposes, this FLIP is being abandoned because it was part > of FLIP-95? > >

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

For clarity purposes, this FLIP is being abandoned because it was part of FLIP-95? On Thu, Sep 7, 2023 at 3:01 AM Venkatakrishnan Sowrirajan wrote: > > Hi everyone, > > Posted a PR (https://github.com/apache/flink/pull/23313) to add nested > fields filter pushdown. Please review. Thanks. > > Rega

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Hi everyone, Posted a PR (https://github.com/apache/flink/pull/23313) to add nested fields filter pushdown. Please review. Thanks. Regards Venkata krishnan On Tue, Sep 5, 2023 at 10:04 PM Venkatakrishnan Sowrirajan wrote: > Based on an offline discussion with Becket Qin, I added *fieldIndices

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Based on an offline discussion with Becket Qin, I added *fieldIndices *back which is the field index of the nested field at every level to the *NestedFieldReferenceExpression *in FLIP-356 *. *2 rea

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Hi Venkata, > Also I made minor changes to the *NestedFieldReferenceExpression, *instead > of *fieldIndexArray* we can just do away with *fieldNames *array that > includes fieldName at every level for the nested field. I don't think keeping only the field names array would work. At the end of t

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

is a top level >> > > record >> > > > > or >> > > > > > > > nested >> > > > > > > > > > >> > record. >> > > > > > > > > > >> > > >> > > > > > > >

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

t; > > > > > > every > > > > > > > > > > >> > > > methods in 2-steps, but is it good to change API > > > back > > > > > and > > > > > > > > forth > > > > > > > > > > for > > &g

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

r the reply, Jark. > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > I think it will be helpful to understand the > final > > > > state we > > > > > > > > want > > &

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

gt;> > > > just > > > > > > > > >> > > > > a one step change. > > > > > > > > >> > > > >- Regarding the supportsNestedFilterPushDown() > > > method, if > > > > > >

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

gt;> > > > > 2. Extend the existing FiledReferenceExpression > class > > to > > > > > > support > > > > > > > >> > nested > > > > > > > >> > > > > fields; SupportsFi

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

t; > > > a. introduce a new method > > > > > > >> > tryApplyFilters(List) > > > > > > >> > > > to > > > > > > >> > > > > SupportsFilterPushDown, which ma

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

> > > >> > > > > tryApplyProjections method > > > > returning > > > > > a > > > > > >> > > Result. > > > > > >> > > > > The Result al

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

nt > > > > >> > > > > and intuitive. Given the size of Flink, keeping APIs in the > > > same

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

gt; > > > > > I have some minor comments about the FLIP. > > > >> > > > > > > > > >> > > > > > 1. I think we don't need to > > > >> > > > > > add SupportsFilterPushDown#suppo

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

gt; > > the interface will be filled with lots of flags (e.g., > > >> > > supportsBetween, > > >> > > > > > supportsIN). > > >> > > > > > > > >> > > > > > 2. NestedFieldReferenceExpre

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

> and > >> > this > >> > > > may > >> > > > > > lead to unexpected errors. > >> > > > > > Having a separate NestedFieldReferenceExpression sounds safer > to > >&g

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

t; > >> > > > > > >> > > > > > >> > > > > > On Tue, 22 Aug 2023 at 05:55, Venkatakrishnan Sowrirajan < >> > > > > vsowr...@asu.edu >> > > > > > > >> > > > >

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

; in > > > > > > > the end state? > > > > > > > > > > > > > > The current proposal works if we do not want to migrate > > > > > > > > SupportsFilterPushdown to also use > > NestedFieldReferenceExpression > > > > in > >

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

t; > removed > > > > > > in > > > > > > > the next major version bump. > > > > > > > > > > > > I assume getFieldIndex would return fieldIndexArray[0], right? > > > > > > > > > > > > Thanks &g

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

sal works if we do not want to migrate > > > > > > SupportsFilterPushdown to also use NestedFieldReferenceExpression > > in > > > > the > > > > > > long term. > > > > > > > > > > > Did you mean *FieldReferenceExpressi

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

; 2. Extend the FieldReferenceExpression to support nested fields. > > > > > - Change the index field type from int to int[]. > > > > > > > > - Add a new method int[] getFieldIndexArray(). > > > > > - Deprecate the int getFieldIndex() method, the code will be

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

gt; > - add a new method > applyProjection(List, > > > > DataType), with default implementation invoking > > applyProjection(int[][], > > > > DataType) > > > > - deprecate the current applyProjection(int[][], DataType) method > > > > > >

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

d > > > > > > Phase 2 (in the next major version bump) > > > 1. remove the deprecated methods. > > > > > > Phase 3 (optional) > > > 1. deprecate and remove the supportsNestedFilters() / > > > supportsNestedProjection() methods from the SupportsFilterPush

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

t; > > > Jiangjie (Becket) Qin > > > > On Sat, Aug 19, 2023 at 12:09 AM Venkatakrishnan Sowrirajan < > > vsowr...@asu.edu> wrote: > > > > > Gentle ping for reviews/feedback. > > > > > &

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

s, > > Jiangjie (Becket) Qin > > On Sat, Aug 19, 2023 at 12:09 AM Venkatakrishnan Sowrirajan < > vsowr...@asu.edu> wrote: > > > Gentle ping for reviews/feedback. > > > > On Tue, Aug 15, 2023, 5:37 PM Venkatakrishnan Sowrirajan < > vsowr...@as

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

ping for reviews/feedback. > > On Tue, Aug 15, 2023, 5:37 PM Venkatakrishnan Sowrirajan > > wrote: > > > Hi All, > > > > I am opening this thread to discuss FLIP-356: Support Nested Fields > > Filter Pushdown. The FLIP can be found at > > > https://cwik

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Gentle ping for reviews/feedback. On Tue, Aug 15, 2023, 5:37 PM Venkatakrishnan Sowrirajan wrote: > Hi All, > > I am opening this thread to discuss FLIP-356: Support Nested Fields > Filter Pushdown. The FLIP can be found at > https://cwiki.apache.org/confluence/display/FLINK/FLIP

[DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Hi All, I am opening this thread to discuss FLIP-356: Support Nested Fields Filter Pushdown. The FLIP can be found at https://cwiki.apache.org/confluence/display/FLINK/FLIP-356%3A+Support+Nested+Fields+Filter+Pushdown This FLIP adds support for pushing down nested fields filters to the