I am using this thread since the question seems related. As of now I can not say a way to project a nested record (FYI scalar works). https://jira.apache.org/jira/browse/CALCITE-4065 @Igor any idea if this can be done without major work on the SqlRelToRelConverter ? Also I am thinking about turning off the flatten stage but not sure this is going to happen (seems like a pandora box kind of flag where you do not know what to expect)
On Thu, Oct 24, 2019 at 3:53 AM Igor Guzenko <[email protected]> wrote: > Hello Naveen, > > 1. If I understand correctly, then yes you can extract nested fields from > struct type. The syntax depends on StructKind value for your data type, > for example for FULLY_QUALIFIED struct you should first > make alias for your table and then request nested field like, > table_alias.struct_column.nested_field. In rel tree such expressions are > presented as RexCall with SqlItemOperator operator. > 2. Yes, this ability was implemented in CALCITE-3138 [1]. It builds call to > ROW type constructor function on top of flattened tree for necessary > columns. > 3. Yes, examples of such functions are ROW(...), ANY_VALUE(...) etc. > > In current implementation of flattener invocation of ROW constructor > function is done despite of null handling same issue exists for some > aggregate function flattening, like COUNT(struct_column). > Proper null handling is real pain for flattener, original idea was to > handle special null indicator for each flattened struct, but in practice I > recognized that it's really hard to deal with flattened fields indices when > related methods are called from very different points, so for now the > problem remains unsolved. > If you can't avoid dealing with null values in your struct columns you > could try to avoid invocation to SqlToRelConverter.flattenTypes(...) and > check whether final plan acceptable for you. As far as I know > there is no reading material for given topic, you can investigate source > code by debugging RelStructuredTypeFlattener and reading some related plans > in SqlToRelConverterTest.java and SqlToRelConverterTest.xml. > > [1] https://issues.apache.org/jira/browse/CALCITE-3138 > > Thanks, > Igor > > On Thu, Oct 24, 2019 at 12:57 PM Naveen Kumar > <[email protected]> wrote: > > > Hi, > > > > I work at Flipkart, we are using Calcite in our streaming platform. In > most > > of our use cases, input data is nested. I understand Calcite flattens > > structs in scan and references fields positionally. > > > > I had a few questions on handling nested data - > > > > 1. Can RelNode DAG work with nested data (instead of flattened fields) > > by referencing fields through their nested structure eg, > > data.order.orderId > > 2. In the current flattened behavior, can output of a query be a > struct. > > Eg if *orderId, orderData.timestamp, orderData.category* are output of > > select query, can I declaratively organise output to below json > > structure - > > 1. > > > > > > > > > > > > *{ "orderId": "order1", "orderData": { "timestamp": 1571904384814, > > "category": "shoes" } }* > > 3. Can output of a UDF be struct type > > > > Please point me to any reading material or example that would help with > > these questions. > > > > Regards, > > Naveen > > > > -- > > > > > > > > > > > *-----------------------------------------------------------------------------------------* > > > > > > *This email and any files transmitted with it are confidential and > > intended solely for the use of the individual or entity to whom they are > > addressed. If you have received this email in error, please notify the > > system manager. This message contains confidential information and is > > intended only for the individual named. If you are not the named > > addressee, > > you should not disseminate, distribute or copy this email. Please notify > > the sender immediately by email if you have received this email by > mistake > > and delete this email from your system. If you are not the intended > > recipient, you are notified that disclosing, copying, distributing or > > taking any action in reliance on the contents of this information is > > strictly prohibited.***** > > > > **** > > > > *Any views or opinions presented in this > > email are solely those of the author and do not necessarily represent > > those > > of the organization. Any information on shares, debentures or similar > > instruments, recommended product pricing, valuations and the like are for > > information purposes only. It is not meant to be an instruction or > > recommendation, as the case may be, to buy or to sell securities, > > products, > > services nor an offer to buy or sell securities, products or services > > unless specifically stated to be so on behalf of the Flipkart group. > > Employees of the Flipkart group of companies are expressly required not > to > > make defamatory statements and not to infringe or authorise any > > infringement of copyright or any other legal right by email > > communications. > > Any such communication is contrary to organizational policy and outside > > the > > scope of the employment of the individual concerned. The organization > will > > not accept any liability in respect of such communication, and the > > employee > > responsible will be personally liable for any damages or other liability > > arising.***** > > > > **** > > > > *Our organization accepts no liability for the > > content of this email, or for the consequences of any actions taken on > the > > basis of the information *provided,* unless that information is > > subsequently confirmed in writing. If you are not the intended recipient, > > you are notified that disclosing, copying, distributing or taking any > > action in reliance on the contents of this information is strictly > > prohibited.* > > > > > > > > > _-----------------------------------------------------------------------------------------_ > > > > >
