Some question about lattices

Hi, community I was investigating lattice recently and encountered some problems: 1. Lattice requires the estimated number of rows of cuboids (tiles) for optimization. I've found the method: `TileSuggester.StatisticsProviderImpl # getRowCount`, `LatticeStatisticProvider # cardinality`, but I found

When will the exchange node(Distribution) be added to the execution plan

Hi I wonder when will the exchange node be added to the execution plan. For example, In Spark, if a join is SMJ(SortMergeJoin), it will add an exchange and a sort node to the execution plan: [image: 3631580619602_.pic.jpg] In Calcite, Let me use CsvTest#testReadme for example and I can find a sor

Re: Re: [DISCUSS] On-demand traitset request

Jinfeng Ni's proposal +1, enhance method satisfies maybe more reasonable. Danny Chan's thoughts +1 Regards! Aron Tao XING JIN 于2019年11月8日周五 下午8:09写道： > Hi Haisheng, > > Thanks a lot for sharing this great proposal ~ > For short I understand your idea as below: > 1. Derive the distribution

Re: When will the exchange node(Distribution) be added to the execution plan

xample in [1]. [1] https://github.com/apache/calcite/blob/a6f544eb48a87f4f71f76ed422584398c0c9baa3/core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java#L6377 -- Kind Regards Roman Kondakov On 02.02.2020 08:01, JiaTao Tao wrote: > Hi > I wonder when will the exchange node be add

Re: When will the exchange node(Distribution) be added to the execution plan

ant part of the original plan is as follows 7:EnumerableTableScan(table=[[USERS]]) Empty subset 1: rel#49:Subset#1.ENUMERABLE.[0].hash[0], the relevant part of the original plan is as follows 8:EnumerableTableScan(table=[[JOBS]]) My table has no collation. Regards! Aron Tao JiaTao Tao 于2020年2月3

Question about HASH_DISTRIBUTED

Hi When I see the example of HASH_DISTRIBUTED, e.g. RelDistributions.hash(ImmutableList.of(0)), or in RelDistributionImpl#satisfies I can't find the info about "hash partition num" of "hash func", if we don't know these message, how can we perform a Bucket Join (Collocated Join)? Or do I miss some

Re: When will the exchange node(Distribution) be added to the execution plan

https://www.slideshare.net/JordanHalterman/introduction-to-apache-calcite > > > -- > Kind Regards > Roman Kondakov > > > On 03.02.2020 16:38, JiaTao Tao wrote: > > The detail message is as follows, and I can see LogicalSort and > > LogicalExchange has been generated though

Is CBO really useful(you can not estimate cost very well )?

Under big data, does CBO have such a big effect? Node like filter/join/aggregate, their cost is estimated. There's one case, I call it runtime optimizing, it means optimizing while c alculating, You adjust your execution plan in real-time based on the execution statistics of the previous step(like

Re: Is CBO really useful(you can not estimate cost very well )?

st-based optimizer” is one that is nondeterministic. > By which I mean, it can investigate multiple paths, thereby avoiding local > minima. > > Therefore I am a fan of simple, crappy stats such as estimated row count. > > Julian > > [1] http://www.vldb.org/pvldb/vol9/p204-leis.pdf &l

How to trace a column back to its original column

SQL like this: ``` SELECT T1.NID, T1.NAME, T2.COMPANY FROM (SELECT (U.ID +100) AS NID, U.NAME AS NAME FROM USERS U) T1 JOIN (SELECT (J.ID +100) AS NID, J.COMPANY FROM JOBS J) T2 ON T1.NID = T2.NID ``` What I wanted is that T1.NID comes from USER.ID, T1.NA

Re: How to trace a column back to its original column

What I really need is to collect every part of the SQL, Which columns are used as filters, which are used as projection(Columns on the source table), But the existence of nested subqueries complicates the issue. Regards! Aron Tao JiaTao Tao 于2020年2月19日周三 下午2:17写道： > SQL like t

Re: How to trace a column back to its original column

Hi Thanks a lot. Can you give more details, say which method/field? Regards! Aron Tao Seliverstov Igor 于2020年2月19日周三 下午2:31写道： > You can use their origins (says where a column came from). > > It's accessable from SqlValidatorImpl or jdbc result set > > ср, 19 февр. 2020

Re: How to trace a column back to its original column

ache/calcite/rel/metadata/RelMdExpressionLineage.java > > On Tue, Feb 18, 2020 at 10:31 PM Seliverstov Igor > wrote: > > > > You can use their origins (says where a column came from). > > > > It's accessable from SqlValidatorImpl or jdbc result set > > &g

Re: How to trace a column back to its original column

calcite.sql.validate.SqlValidatorImpl#getFieldOrigins > java.sql.ResultSetMetaData#getTableName > > Regards, > Igor > > > 19 февр. 2020 г., в 10:29, JiaTao Tao написал(а): > > > > Hi > > Thanks a lot. > > Can you give more details, say which meth

Re: How to trace a column back to its original column

t; > I can’t come up with another way. > > Regards, > Igor > > > 19 февр. 2020 г., в 15:02, JiaTao Tao написал(а): > > > > Thanks a lot > > > > But I found if the select is from the subquery, `getFieldOrigins` > returns null, because you can not find the

Re: [DISCUSS] Towards Calcite 1.22.0

+1 Regards! Aron Tao Danny Chan 于2020年2月20日周四 下午2:15写道： > Hi all, > > Approximately 5 months has passed from the previous release > (Calcite 1.21.0) and I was thinking that it is the time to have the next > release by the end of February. To do this I think we should try to have an > RC aroun

Re: [VOTE] Release apache-calcite-1.22.0 (release candidate 2)

+1 (non-binding) Regards! Aron Tao Danny Chan 于2020年2月26日周三 下午1:23写道： > Hi all, > > I have created a build for Apache Calcite 1.22.0, release candidate 2. > > Thanks to everyone who has contributed to this release. > You can read the release notes here: > https://github.com/apache/calcite/bl

javacc problem: support "!" as "NOT"

Hi I want to identify “!” as “NOT” in SQL, so I do So I made the following changes in parser.jj: | < NOT: "NOT"|"!" > It can work, but some test fails(testNoUnintendedNewReservedKeywords), then I found in SqlParserImplConstants, “NOT” becomes "" as in the pic. [image: image.png] It should be “\

Re: javacc problem: support "!" as "NOT"

Thank Danny Can you give an example? And I don't understand why "NOT" becomes "". Regards! Aron Tao Danny Chan 于2020年2月27日周四 上午11:39写道： > You can add a token and alias it as you like. The token is always a > reserved keyword. > > Best, > Danny Chan > 在 2020年2月27日 +0800 AM11:23，dev@calcite.ap

SQL has order by, there will be one more column(the sort column) in result plan

SQL; ``` SELECT u.ID, u.NAME FROM USERS u ORDER BY u.AGE ``` PLAN: ``` LogicalSort(sort0=[$2], dir0=[ASC]) LogicalProject(ID=[$0], NAME=[$1], AGE=[$2]) EnumerableTableScan(table=[[RECR, USERS]]) ``` Plan's row type: RecordType(INTEGER ID, VARCHAR NAME, INTEGER AGE) But the SQL only project

Re: SQL has order by, there will be one more column(the sort column) in result plan

t; > > > On Mar 4, 2020, at 3:43 AM, JiaTao Tao wrote: > > > > SQL; > > ``` > > SELECT u.ID, u.NAME > > FROM USERS u > > ORDER BY u.AGE > > ``` > > > > PLAN: > > ``` > > LogicalSort(sort0=[$2], dir0=[ASC]) > > Logical

Re: SQL has order by, there will be one more column(the sort column) in result plan

Seems `root.project()` works, thanks a lot! Regards! Aron Tao JiaTao Tao 于2020年3月5日周四 上午11:05写道： > Thanks, Julian > > I can see the fields in RelRoot, and I'm a little confused, I want to > use RelVisitor to visit the RelNode, and here comes the RelRoot, any advice >

[DISCUSS] get RexExecutor from RexSimplify in method reduceExpressionsInternal

In method reduceExpressionsInternal, we get RexExecutor from cluster, it can be null: [image: image.png] But in the outside(reduceExpressions), `final RexExecutor executor = Util.first(cluster.getPlanner().getExecutor(), RexUtil.EXECUTOR)`, it can't be null. And reduceExpressions is the only cal

Re: [DISCUSS] get RexExecutor from RexSimplify in method reduceExpressionsInternal

must relay on the engine to plugin their RexExecutor to make a > > constant reduction(some engine use code generation, some use Java > > reflection). > > > > So, in total, the executor in RexSimplify has a fallback is because it’s > > expression to reduce is simple enough.

Re: [DISCUSS] get RexExecutor from RexSimplify in method reduceExpressionsInternal

r may be different, but it still makes people confusing. IMHO, if "return RexUtil.EXECUTOR" >= "return null", we can do the modify. If you think so, I can open a JIRA and do this minor change. Hope to hear your voice. Regards! Aron Tao JiaTao Tao 于2020年3月17日周二 下午4:02写

set SqlToRelConverter.ConfigBuilder#expand default to false

Hi Now the default "expand" in SqlToRelConverter.ConfigBuilder is true but in calcite's main process, actually, it is false( `withExpand(THREAD_EXPAND.get())`) [image: image.png] [image: image.png] That leads we need to explicitly set `withExpand` to false when we use SqlToRelConverter. So I

Re: set SqlToRelConverter.ConfigBuilder#expand default to false

with a release note. Julian > On Mar 23, 2020, at 1:16 AM, JiaTao Tao wrote: > > Hi > > Now the default "expand" in SqlToRelConverter.ConfigBuilder is true but in calcite's main process, actually, it is false( `withExpand(THREAD_EXPAND.get())`) > > >

Doubts about TPCH queries with volcano planner

Hi all Currently, I'm studying the volcano planner and found that many TPCH queries failed to optimize. See in org.apache.calcite.adapter.tpch.TpchTest: And here comes the question: Is volcano planner production ready? If so, why there are so many TPCH queries fails? Do we have plans to support t

what's the difference between RelOptRules#ABSTRACT_RELATIONAL_RULES and BASE_RULES

The rules in ABSTRACT_RELATIONAL_RULES seem pretty "base", why we put them ABSTRACT_RELATIONAL_RULES, not in "BASE"? Regards! Aron Tao

Re: Re: Doubts about TPCH queries with volcano planner

alcite is a framework for paring and optimizing the query. The volcano > planner is always available if you leverage it correctly. > > Besides, am I the only person who cannot see the picture? > > > Best, > Chunwei > > > On Thu, Mar 26, 2020 at 1:30 PM JiaTao Tao wro

RelMetadataQuery.getRowCount stackoverflow

Hi Has anyone encountered this problem before? Just a simple query(no more than 20 lines, two joins, no union). And I see this ticket: https://issues.apache.org/jira/browse/CALCITE-2057, but there's no follow up, also I see flink may occur this problem( https://developer.aliyun.com/ask/129548) ja

Re: RelMetadataQuery.getRowCount stackoverflow

> I can usually see this happening when I add a > System.out.println(RelOptUtil.dumpPlan()) to the line before the > call.transformTo(newRelationNode) > > On Mon, Apr 20, 2020 at 3:13 AM JiaTao Tao wrote: > > > Hi > > Has anyone encountered this problem before? Just

Re: [VOTE] Release apache-calcite-1.23.0 (release candidate 1)

Vote: +1 (non-binding) CALCITE-3819 really helps infinite planning in our use case. And I'm looking forward to CALCITE-3997. --- Environment: Mac-OS JDK version: 1.8.0_212 Build with tests: OK Regards! Aron Tao Vladimir Sitnikov 于2020年5月2

Assign CALCITE JIRA permissions

Hi, Could anyone assign me the CALCITE JIRA's permission, so that I can assign JIRA to myself. Regards! Aron Tao

Re: Assign CALCITE JIRA permissions

Thank you very much Regards! Aron Tao Stamatis Zampetakis 于2020年6月17日周三 上午5:05写道： > Hi Aron, > > I just added you as a contributor in JIRA. You can now assign issues to > yourself. > > Best, > Stamatis > > On Tue, Jun 16, 2020 at 3:24 PM JiaTao Tao wrote: > &

Calcite PR CI offten failed due to elasticsearch:test

Seems an unstable case. Regards! Aron Tao

Re: Calcite PR CI offten failed due to elasticsearch:test

an issue there and disable the case ? > > Best, > Danny Chan > 在 2020年6月26日 +0800 PM6:00，JiaTao Tao ，写道： > > Seems an unstable case. > > > > > > Regards! > > > > Aron Tao >

Re: Calcite PR CI offten failed due to elasticsearch:test

Danny Chan > 在 2020年7月2日 +0800 AM11:15，JiaTao Tao ，写道： > > Hi Danny > > > > > > There is 4 failed ut, and the root cause is java.net > .SocketTimeoutException: > > 30,000 milliseconds timeout on connection http-outgoing-2 : > > > > elasticsearc

Doubts about VolcanoPlannerPhase

Seems only use OPTIMIZE phase, can we remove this mechanism? Regards! Aron Tao

Re: Doubts about VolcanoPlannerPhase

bout this several months ago [1]. > > [1] https://github.com/apache/calcite/pull/1840#discussion_r387967624 > > -- > Roman Kondakov > > On 07.07.2020 09:52, JiaTao Tao wrote: > > Seems only use OPTIMIZE phase, can we remove this mechanism? > > > > > > Regards! > > > > Aron Tao > > >

Re: Calcite PR CI offten failed due to elasticsearch:test

if it is only a CI problem > then it would be nice if we could disable the tests only there. > > Best, > Stamatis > > On Thu, Jul 2, 2020 at 5:58 AM JiaTao Tao wrote: > > > Ok, thanks for your suggestion. I'll open a JIRA and track the status. > > > > >

Re: Calcite PR CI offten failed due to elasticsearch:test

if it is only a CI problem > then it would be nice if we could disable the tests only there. > > Best, > Stamatis > > On Thu, Jul 2, 2020 at 5:58 AM JiaTao Tao wrote: > > > Ok, thanks for your suggestion. I'll open a JIRA and track the status. > > > > >

Re: Calcite PR CI offten failed due to elasticsearch:test

;[=($0, 1), OR(AND(=($2, 3), =($1, 2)), =($1, 4))]" Regards! Aron Tao JiaTao Tao 于2020年7月9日周四 下午2:46写道： > Hi Stamatis Zampetakis > > I agree with you, on my local test, it is never failing, seems only a PR > CI problem. > Do you have any idea about disabling the tests on

Re: [DISCUSSION] Rename master branch to main

Agree with Danny Chan, -1 for this change. Regards! Aron Tao Michael Mior 于2020年8月5日周三 下午6:41写道： > My apologies for misinterpreting your previous statement then. However, I > don't see evidence to support your claims that this will bring no new > contributors nor make anyone feel more welcom

Re: RelToSqlConverter NullPointerException

Hi tonytao In your code, you connect pg, can you reproduce this in RelToSqlConverterTest? Regards! Aron Tao Stamatis Zampetakis 于2020年8月12日周三 上午5:38写道： > Thanks for reporting this. Please file a JIRA case with the necessary > information. > > Also it would be helpful if you could refactor you

How about add a method for "RelNode" that can get the same string like RelOptUtil.toString

As a calcite dev, I call this method dozens of times, especially in debugging, it's not so convenient. "RelNode" has an API for its own digest(not the tree), how about adding an API can get it tree digest just like RelOptUtil.toString did. Regards! Aron Tao

TableScan#explainTerms may be not enough for digest

Hi fellows Here's the problem: We have our own RelOptTable implementation, and we replace the RelOptTableImpl during a rule in HepPlanner. Something unexpected occurred: We only replaced the `table` in TableScan, and the digest is not changed after the rule, so in `HepPlanner#addRelToGraph`, direct

Re: TableScan#explainTerms may be not enough for digest

first: override def getQualifiedName: util.List[String] = { val names = super.getQualifiedName extraDigests.foreach(builder.add) builder.build() } But this way, there's dirty info when we call getQualifiedName. Regards! Aron Tao JiaTao Tao 于2020年10月9日周五 下午3:27写道： &g

Re: TableScan#explainTerms may be not enough for digest

Hi Danny So maybe it is better to add an interface for rel opt table to provide extra digest and consider this in TableScan#explainTerms, but it changes the base output of TableScan, seems a quite big change. Regards! Aron Tao Danny Chan 于2020年10月9日周五 下午5:29写道： > Thanks for driving this disc

Re: TableScan#explainTerms may be not enough for digest

Hi Danny, Does this info mainly for planner use? If this we can add an interface mainly for the planner. The problem bothering me is that the transformation is not effective in HepPlanner due to this. Regards! Aron Tao JiaTao Tao 于2020年10月9日周五 下午5:46写道： > > Hi Danny > So maybe it

Re: TableScan#explainTerms may be not enough for digest

> Aron, what do you mean by "the transformation is not effective"? > > Haisheng > > On 2020/10/09 09:49:27, JiaTao Tao wrote: > > Hi Danny, > > Does this info mainly for planner use? If this we can add an interface > > mainly for the planner. > > &

why DECIMAL is in SqlTypeName#EXACT_TYPES but DOUBLE is in SqlTypeName#APPROX_TYPES

org.apache.calcite.sql.type.SqlTypeName#APPROX_TYPES org.apache.calcite.sql.type.SqlTypeName#NUMERIC_TYPES Regards! Aron Tao

Re: why DECIMAL is in SqlTypeName#EXACT_TYPES but DOUBLE is in SqlTypeName#APPROX_TYPES

In calcite, you can not mod doubles. In Java, you can, in C you can't use "%" to mod doubles, but you can use fmod. Regards! Aron Tao JiaTao Tao 于2020年10月15日周四 下午10:00写道： > org.apache.calcite.sql.type.SqlTypeName#APPROX_TYPES > org.apache.calcite.sql.type.SqlT

Re: why DECIMAL is in SqlTypeName#EXACT_TYPES but DOUBLE is in SqlTypeName#APPROX_TYPES

tice the way we represent a number can lead to an exact (scale + > precision) or an approximate (base + mantissa) representation, which suits > very different needs. > > Best regards, > Alessandro > > On Thu, 15 Oct 2020 at 16:16, JiaTao Tao wrote: > > > In calcite, y

Re: why DECIMAL is in SqlTypeName#EXACT_TYPES but DOUBLE is in SqlTypeName#APPROX_TYPES

n that you are trying to apply the modulus (mod) > > operator over an approximate number, which is not supported as we have > seen > > above. > > > > Best regards, > > Alessandro > > > > On Fri, 16 Oct 2020 at 05:02, JiaTao Tao wrote: > >

Generate a empty Aggregate after RelFieldTrimmer#trim

Hi fellows I found this in my plan, after RelFieldTrimmer#trim, `relBuilder.aggregate(groupKey, newAggCallList);`, the groupKey and newAggCallList are all empty, I think it doesn't meet expectations. LogicalAggregate(group=[{}]) LogicalAggregate(group=[{0, 1, 2}]) Regards! Aron Tao

RelBuilder#project should try keep alias in optimizer

Hi fellows I found in some place, we just call `project(Iterable nodes) `, but this will lose alias, why don't we try to call `project(Iterable nodes, Iterable fieldNames) ` to keep alias? Regards! Aron Tao

Re: RelBuilder#project should try keep alias in optimizer

rowse/CALCITE-4037 > [2] https://issues.apache.org/jira/browse/CALCITE-3662 > [3] https://issues.apache.org/jira/browse/CALCITE-1584 > > Best, > Danny Chan > 在 2020年10月21日 +0800 PM2:49，JiaTao Tao ，写道： > > Hi fellows > > I found in some place, we just call `project(Itera

Re: RelBuilder#project should try keep alias in optimizer

in 1584. I think that answers your question. > > Julian > > > On Oct 21, 2020, at 23:35, JiaTao Tao wrote: > > > > Hi Danny > > Thanks for your reply, can you explain "Calcite only optimize based on > the > > field input refs, if we always keep the a

Re: RelBuilder#project should try keep alias in optimizer

Some rules we use `RelBuilder project(Iterable nodes) `, sometimes use `RelBuilder project(Iterable nodes, Iterable fieldNames)`, and this really makes me confused. Regards! Aron Tao JiaTao Tao 于2020年10月22日周四 下午4:13写道： > Thanks, Julian > One more question is that I can se

Re: RelBuilder#project should try keep alias in optimizer

道： > Did you read the javadoc for the two ‘project’ methods? > > Julian > > > > On Oct 22, 2020, at 1:15 AM, JiaTao Tao wrote: > > > > Some rules we use `RelBuilder project(Iterable nodes) > `, > > sometimes use `RelBuilder project(Iterable nodes, &g

Re: Generate a empty Aggregate after RelFieldTrimmer#trim

s.apache.org/jira/browse/CALCITE-4317>. Let’s continue > discussion there. > > Julian > > > On Oct 20, 2020, at 8:29 PM, JiaTao Tao wrote: > > > > Hi fellows > > I found this in my plan, after RelFieldTrimmer#trim, > > `relBuilder.aggregate(groupKey,

Re: [DISCUSS] Apache Calcite Online Meetup January 2021

I think it's a nice idea, hope to hear Julian's topic about lattice/mv, we have a big demand on this and want to learn more. Regards! Aron Tao Julian Hyde 于2020年10月16日周五 下午12:50写道： > Nice idea, Stamatis. I'm happy to give a talk if people want to hear > it - possible topics include geospatia

Why ProjectJoinTransposeRule not in Calcite default rule set.

This rule is only used in RelOptMaterializations#substitute, is not in Calcite's main optimizer process(RelOptUtil#registerDefaultRules), wonder why, seems a common rule. Regards! Aron Tao

Potential memory leak in VolcanoPlanner#materializations due to circular reference

Hi fellows We use Calcite's materialization view in our product env, and it OOM times, I dig the memory dump and found there all of RelOptMaterialization instance. We will call Planner#clear before every query, and inside the dump, there are lots of planner instance, soI suspect it is a circular r

Re: Why ProjectJoinTransposeRule not in Calcite default rule set.

it tests need to be updated? > > > > > > -Rui > > > > On Mon, Oct 26, 2020 at 2:14 AM Fan Liya wrote: > > > > > I also think the ProjectJoinTransposeRule should be included in the > main > > > optimizer process. > > > > &

Re: Why ProjectJoinTransposeRule not in Calcite default rule set.

This rule is used by Flink for a while(since 2019/3), so we are confident that we can introduce it into the main optimize process. Regards! Aron Tao JiaTao Tao 于2020年10月27日周二 下午4:10写道： > Hi > I'll enable this rule and run UTs, Why I noticed this rule is because > Flink uses i

Re: [DISCUSS] State of the project 2020

Hi Stamatis, Thanks for your great work. 2) What areas do we need to do better? 1. Lattice/Materialized view 2. Streaming(we can more contact with Flink project) 3) Which other candidates should we consider for PMC chair? I think Haisheng and Danny are all excellent choices. Regards! Aron Tao

Re: [Question] How to leverage Calcite adaptor for federated SQL query without using Calcite parser

There seems to be no easy way to do this, you can take look at Hive, it uses its own parser to parse SQL and use Caclite to do the optimization. By the way, Calcite's parser is quite OK both in compatibility and performance(javacc is faster than Antlr), in Bytedance, we use Calcite to do the federa

Re: [Question] How to leverage Calcite adaptor for federated SQL query without using Calcite parser

In fact, parse twice's impact is little, in Apache Kylin, every time we do the transformation to SQL, we re-parse it. What really takes time is validation (use metadata like getting it from HMS) and optimization. Regards! Aron Tao Juan Pan 于2020年11月22日周日 下午2:32写道： > Hi community, > > > > > Th

Re: A question regarding querying Google Cloud BigTable or Spanner through Apache Calcite

I think you are talking about query federation, yes, it's a good case for Caclite. Regards! Aron Tao Jason Chen 于2020年11月4日周三 上午8:06写道： > Hey, > > I am Jason Chen from Shopify Data Science and Engineering team. I have a > few questions regarding the Apache Calcite, and I am not sure if the A

[DISCUSS] Does anybody think this is debuging unfriendly: "call.transformTo(relBuilder.build())"

Hi I've been developed Calcite full time for a quite long time, and I ofter debug in the rule to see the transformations, but code like this is not debuging friendly in my opinion: "call.transformTo(relBuilder.build())" I want to see the relBuilder.build()'s result, I have to debug into the "tran

Re: [DISCUSS] Does anybody think this is debuging unfriendly: "call.transformTo(relBuilder.build())"

t;transformTo", you have to go back to see these. Regards! Aron Tao JiaTao Tao 于2020年11月22日周日 下午5:23写道： > Hi > I've been developed Calcite full time for a quite long time, and I ofter > debug in the rule to see the transformations, but code like this is not >

Re: A question regarding querying Google Cloud BigTable or Spanner through Apache Calcite

; > Juan Pan (Trista) > > Senior DBA & PMC of Apache ShardingSphere > E-mail: panj...@apache.org > > > > > On 11/22/2020 16:39，JiaTao Tao wrote： > I think you are talking about query federation, yes, it's a good case for > Caclite. > > > Regard

Re: [Question] How to leverage Calcite adaptor for federated SQL query without using Calcite parser

he.org/document/current/en/features/sharding/principle/parse > > > Juan Pan (Trista) > > Senior DBA & PMC of Apache ShardingSphere > E-mail: panj...@apache.org > > > > > On 11/22/2020 16:17，JiaTao Tao wrote： > In fact, parse twice's impact is little, in Apac

Re: [DISCUSS] Does anybody think this is debuging unfriendly: "call.transformTo(relBuilder.build())"

/11/23 02:42:21, Danny Chan wrote: > > I kind of agree, but it's more like a programming specification, we can > > tell people how to write codes but they may not follow those rules. > > > > JiaTao Tao 于2020年11月22日周日 下午5:27写道： > > > > > Why I don'

Re: [Question] How to leverage Calcite adaptor for federated SQL query without using Calcite parser

Conn)`, which > will bring Calcite parser to parser SQL instead of my own. : ( > Is there any approach to make Calcite call the custom adapter and > third-party parser? > > > Best wishes, > Trista > > > > > Juan Pan (Trista) > > Senior DBA & PMC of A

Re: [DISCUSS] Does anybody think this is debuging unfriendly: "call.transformTo(relBuilder.build())"

es before they > > return them, e.g. > > > > final RelNode r = relBuilder.build(); > > return r; > > > > I am not fond of that pattern either. > > > > Julian > > > > > > > > > On Nov 23, 2020, at 6:32 AM,

Re: Decouple core from linq4j and Avatica

+1 for this idea, I have been developing Calcite for a long time(counting during project Kylin), we all treat calcite as an optimizer, but we need to consider overhead. I aggre with Stamatis: "since those dependencies were not causing any real trouble." What really troubling me is that when we d

Re: [DISCUSS] Does anybody think this is debuging unfriendly: "call.transformTo(relBuilder.build())"

e. > 2) there might be other codes that are not debug-friendly and we can not > change them all. > > > Best, > Chunwei > > > On Tue, Nov 24, 2020 at 10:59 AM JiaTao Tao wrote: > > > Hi James > > My point is not all intermediate variables, please don&#

Re: [DISCUSS] Does anybody think this is debuging unfriendly: "call.transformTo(relBuilder.build())"

that are not debug-friendly and we can not > change them all. > > > Best, > Chunwei > > > On Tue, Nov 24, 2020 at 10:59 AM JiaTao Tao wrote: > > > Hi James > > My point is not all intermediate variables, please don't expand the > scope, > > you

TpchTest#testQuery07 timeout when running continuous-integration/appveyor/pr

Hi fellows My PR failed due to, seems an unstable ut, have anyone met this? FAILURE 300.8sec, org.apache.calcite.adapter.tpch.TpchTest > testQuery07() java.util.concurrent.TimeoutException: testQuery07() timed out after 5 minutes https://github.com/apache/calcite/pull/2284 https://ci.appveyo

Re: TpchTest#testQuery07 timeout when running continuous-integration/appveyor/pr

Hi Vladimir Thanks for your reply: 1. I run TpchTest#testQuery07 in my MPB, the avg time is 43s, and after re-run the PR ut, the ut is passed, so this ut timeout may be an occasional situation. 2. For the pull/, it due increase the time of TpchTest#testQuery07, about 23%, sorry for that, we te

Re: TpchTest#testQuery07 timeout when running continuous-integration/appveyor/pr

reslove all those cases. Regards! Aron Tao Vladimir Sitnikov 于2020年12月3日周四 下午5:48写道： JiaTao Tao, thanks for the details! JiaTao Tao> found. Without pull/, some cases will fail due to "There are not enough JiaTao Tao> rules to produce a node with desired properties: There is 1 em

Re: TpchTest#testQuery07 timeout when running continuous-integration/appveyor/pr

Hi the flame graph is IDEA's feature, you can directly run with the flame graph. As I said does increase the time, but the time is almost from getRelList, so if we can opt getRelList, the overhead is gone. Regards! Aron Tao Vladimir Sitnikov 于2020年12月3日周四 下午9:02写道： > >From the flame grap

Re: TpchTest#testQuery07 timeout when running continuous-integration/appveyor/pr

and paste your findings. > I would like to look into it if I find some time. > > Best, > Chunwei > > > On Thu, Dec 3, 2020 at 1:15 PM JiaTao Tao wrote: > > > Hi Vladimir > > Thanks for your reply: > > 1. I run TpchTest#testQuery07 in my MPB, the avg time

Re: TpchTest#testQuery07 timeout when running continuous-integration/appveyor/pr

Hi Julian Thanks, Julian, your advice is really helpful! Do you mind I pull a request to merge your code so that we can improve the performance very quickly? About " stored the list of RelNodes ", do you concern about the duplicate "relnode" store in "relsubset"? IMO, what we store is a reference,

Re: TpchTest#testQuery07 timeout when running continuous-integration/appveyor/pr

Hi Vladimir I got your opinion, may we do a quick fix about the performance, then I'll take a look again about this. Regards! Aron Tao Vladimir Sitnikov 于2020年12月4日周五 上午5:13写道： The improvement to RelTraitSet.satisfies would be nice, however, I still believe getRelList must not be used in 222

Re: TpchTest#testQuery07 timeout when running continuous-integration/appveyor/pr

I will not merge a fix for getRelList performance until after the algorithmic problem has been resolved. On Thu, Dec 3, 2020 at 6:54 PM JiaTao Tao wrote: > > Hi Julian > > Thanks, Julian, your advice is really helpful! Do you mind I pull a request > to merge your code so that we

Re: TpchTest#testQuery07 timeout when running continuous-integration/appveyor/pr

gt; problem. So, I will not merge a fix for getRelList performance until > after the algorithmic problem has been resolved. > > On Thu, Dec 3, 2020 at 6:54 PM JiaTao Tao wrote: > > > > Hi Julian > > > > Thanks, Julian, your advice is really helpful! Do you mind I pull a

Re: [HELP] Local build fails

Currently, I've not encountered such a situation, my java version is 212. But I'm always encountered with geode: timeout, wonder anybody has met this problem. Regards! Aron Tao Chunwei Lei 于2020年12月4日周五 下午6:15写道： > My Java version is: > > java version "1.8.0_151" > Java(TM) SE Runtime Environ

grouping() function occurs value large than 1

Hi After AggregateExpandDistinctAggregatesRule, I got a plan like this: The $10 in the project node is $g=[GROUPING($0, $1, $2, $3, $4, $5, $6, $7, $8)]) and we can see it is compared with value 1/2/3, but I check the def of grouping(), both pg and oracle, the value of grouping is 0 or 1. pg:http

Re: grouping() function occurs value large than 1

it > returns an integer bitmask with N bits. > > PostgreSQL has the same behavior: see example in > https://www.postgresql.org/docs/9.5/functions-aggregate.html. > > Julian > > On Tue, Dec 8, 2020 at 12:30 AM JiaTao Tao wrote: > > > > Hi > > After AggregateEx

Re: grouping() function occurs value large than 1

; Of course you can use whichever you like in your queries. > > Julian > > > On Dec 8, 2020, at 19:00, JiaTao Tao wrote: > > > > Hi Julian > > I see, thanks, maybe use grouping id is better? Cuz seems not every > engine > > has this grouping behavior, in

Why we repalce the literals in RexWindowBound with RexInputRef and put these literals to Window#constants

Hi Current we got like this in Window#Group if we wrote SQL like `rows between 0 preceding and 2 following` $3 PRECEDING $4 FOLLOWING $3 and $4 is input ref, and we had to find $3 and $4 in Window#constant, and all digest about Window#Group doesn't replace the ref back to literal, so it lost the i

Re: Using Calcite at LinkedIn

Hi Walaa Very happy to see this, our team basically do the same thing, a unified SQL layer: 1. Spark: RelNode -> Spark DataFrame plan 2. Presto: RelNode -> In string SQL 3. Clickhouse: RelNode -> Serialized RelNode 4. Flink -> TBD(with datastream API or table API) I do point 1 both in my previous

Re: Using Calcite at LinkedIn

/tree/master/coral-spark-plan > [2] > > https://github.com/linkedin/coral/blob/master/coral-spark-plan/src/test/java/com/linkedin/coral/sparkplan/SparkPlanToIRRelConverterTest.java > [3] > > https://jaceklaskowski.gitbooks.io/mastering-spark-sql/content/spark-sql-SparkStrategy-D

Re: [ANNOUNCE] New Calcite PMC chair: Haisheng Yuan

Congratulations, Haisheng! Thank you for your service, Stamatis. Regards! Aron Tao Danny Chan 于2020年12月18日周五 上午10:05写道： > Congratulations, Haisheng! > > Thank you for your service, Stamatis. > > Julian Hyde 于2020年12月18日周五 上午9:58写道： > > > Congratulations, Haisheng! > > > > Thank you for your

Re: Proposal to extend Calcite into a incremental query optimizer

Seems interesting, the pic can not be seen in the mail, may you open a JIRA for this, people who are interested in this can subscribe to the JIRA? Regards! Aron Tao Botong Huang 于2020年12月24日周四 上午3:18写道： > Hi all, > > This is a proposal to extend the Calcite optimizer into a general > increme

Re: How to rewrite a query using calcite. Any Example?

Hi Try this: SqlParser.create(expr).parseExpression(); Regards! Aron Tao kant kodali 于2021年1月5日周二 上午9:34写道： > Hi All, > > is there a way to parse only the where clause using calcite (any sample > code)? meaning, for example, assume the input is something like > > view1.col1 = value1 or view

  1   2   >