[jira] [Commented] (HIVE-9574) Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark Branch]

[Chao (JIRA)]

    [ 
https://issues.apache.org/jira/browse/HIVE-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312987#comment-14312987
 ] 

Chao commented on HIVE-9574:
----------------------------

Regarding cbo_gby_empty.q, I found this error in hive.log:

{noformat}
...
NestedThrowablesStackTrace:
Method/Identifier expected at character 37 in "tableName == t1 && dbName == t2 
&& ()"
org.datanucleus.store.query.QueryCompilerSyntaxException: Method/Identifier 
expected at character 37 in "tableName == t1 && dbName == t2 && ()"
        at 
org.datanucleus.query.compiler.JDOQLParser.processPrimary(JDOQLParser.java:810)
        at 
org.datanucleus.query.compiler.JDOQLParser.processUnaryExpression(JDOQLParser.java:656)
        at 
org.datanucleus.query.compiler.JDOQLParser.processMultiplicativeExpression(JDOQLParser.java:582)
        at 
org.datanucleus.query.compiler.JDOQLParser.processAdditiveExpression(JDOQLParser.java:553)
        at 
org.datanucleus.query.compiler.JDOQLParser.processRelationalExpression(JDOQLParser.java:467)
        at 
org.datanucleus.query.compiler.JDOQLParser.processAndExpression(JDOQLParser.java:450)
        at 
org.datanucleus.query.compiler.JDOQLParser.processExclusiveOrExpression(JDOQLParser.java:436)
        at 
org.datanucleus.query.compiler.JDOQLParser.processInclusiveOrExpression(JDOQLParser.java:422)
        at 
org.datanucleus.query.compiler.JDOQLParser.processConditionalAndExpression(JDOQLParser.java:408)
        at 
org.datanucleus.query.compiler.JDOQLParser.processConditionalOrExpression(JDOQLParser.java:389)
        at 
org.datanucleus.query.compiler.JDOQLParser.processExpression(JDOQLParser.java:378)
        at 
org.datanucleus.query.compiler.JDOQLParser.processPrimary(JDOQLParser.java:785)
        at 
org.datanucleus.query.compiler.JDOQLParser.processUnaryExpression(JDOQLParser.java:656)
        at 
org.datanucleus.query.compiler.JDOQLParser.processMultiplicativeExpression(JDOQLParser.java:582)
        at 
org.datanucleus.query.compiler.JDOQLParser.processAdditiveExpression(JDOQLParser.java:553)
        at 
org.datanucleus.query.compiler.JDOQLParser.processRelationalExpression(JDOQLParser.java:467)
        at 
org.datanucleus.query.compiler.JDOQLParser.processAndExpression(JDOQLParser.java:450)
        at 
org.datanucleus.query.compiler.JDOQLParser.processExclusiveOrExpression(JDOQLParser.java:436)
        at 
org.datanucleus.query.compiler.JDOQLParser.processInclusiveOrExpression(JDOQLParser.java:422)
        at 
org.datanucleus.query.compiler.JDOQLParser.processConditionalAndExpression(JDOQLParser.java:412)
        at 
org.datanucleus.query.compiler.JDOQLParser.processConditionalOrExpression(JDOQLParser.java:389)
        at 
org.datanucleus.query.compiler.JDOQLParser.processExpression(JDOQLParser.java:378)
        at org.datanucleus.query.compiler.JDOQLParser.parse(JDOQLParser.java:99)
        at 
org.datanucleus.query.compiler.JavaQueryCompiler.compileFilter(JavaQueryCompiler.java:467)
        at 
org.datanucleus.query.compiler.JDOQLCompiler.compile(JDOQLCompiler.java:113)
        at 
org.datanucleus.store.query.AbstractJDOQLQuery.compileInternal(AbstractJDOQLQuery.java:367)
        at 
org.datanucleus.store.rdbms.query.JDOQLQuery.compileInternal(JDOQLQuery.java:240)
        at org.datanucleus.store.query.Query.executeQuery(Query.java:1744)
        at org.datanucleus.store.query.Query.executeWithArray(Query.java:1672)
        at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:312)
        at 
org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:5969)
        at 
org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:159)
        at 
org.apache.hadoop.hive.metastore.ObjectStore$6.getJdoResult(ObjectStore.java:6025)
        at 
org.apache.hadoop.hive.metastore.ObjectStore$6.getJdoResult(ObjectStore.java:6017)
{noformat}

I'm not sure what this mean.

> Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark 
> Branch]
> --------------------------------------------------------------------------------
>
>                 Key: HIVE-9574
>                 URL: https://issues.apache.org/jira/browse/HIVE-9574
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Rui Li
>            Assignee: Jimmy Xiang
>             Fix For: spark-branch
>
>         Attachments: HIVE-9574.1-spark.patch, HIVE-9574.2-spark.patch, 
> HIVE-9574.3-spark.patch, HIVE-9574.4-spark.patch, HIVE-9574.5-spark.patch
>
>
> {{RowContainer.first}} may call {{InputFormat.getSplits}}, which is 
> expensive. If we switch {{container}} and {{backupContainer}} frequently in 
> {{HiveKVResultCache}}, it will downgrade performance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)