I'm working on CALCITE-2376 which unifies ES2 and ES5 adapters. I'll make sure it is passing both javadoc / site tasks in maven.
Would you consider merging this partial javadoc fix (PR 741 <https://github.com/apache/calcite/pull/741>) as precursor for CALCITE-2376 ? Former PR has also fixes javadoc issues for mongo adapter. On a separate note, do you think it makes sense to add javadoc generation to travis build ? On Mon, Jun 25, 2018 at 1:52 PM Julian Hyde <jh...@apache.org> wrote: > Your changes seem to be generally beneficial and are much appreciated. But > they do not fix the main problem, which is the missing package on JDK 9, > 10, 11. (I believe it has always worked on JDK 8.) For example, here is > output from the run against your branch on JDK 10 Ubuntu: > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on > project calcite: Error generating maven-javadoc-plugin:3.0.1:test-aggregate > report: > [ERROR] Exit code: 1 - > /home/jhyde/regress/calcite/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29: > error: package org.elasticsearch.node.internal does not exist > [ERROR] import org.elasticsearch.node.internal.InternalSettingsPreparer; > [ERROR] ^ > [ERROR] > > Let’s work on fixing the main problem first, then we’ll consider your PR. > > We had issues before (there is a JIRA case, don’t recall the number) where > ES2 and ES5 could not share the same class path. I think this is another > manifestation of that — maybe javadoc tries to generate for all modules > simultaneously, and therefore when ES5’s dependencies have been loaded into > javadoc, ES2’s dependencies cannot be seen. > > The solution may be to disable javadoc for the ES2 adapter. Or perhaps > just for that one java file. > > Julian > > > > On Jun 25, 2018, at 8:10 AM, Volodymyr Vysotskyi <volody...@apache.org> > wrote: > > > > Thanks for the changes, I have checked and it passed on my env. > > > > I think it should be also checked with other JDK versions besides 8. > > > > Kind regards, > > Volodymyr Vysotskyi > > > > > > пн, 25 черв. 2018 о 07:05 Andrei Sereda <and...@sereda.cc> пише: > > > >> Can you please check the branch in my fork ? > >> > >> fork: https://github.com/asereda-gs/calcite > >> branch: javadoc-fixes > >> > >> I've tried to remove most of the warnings and javadoc, javadoc-test and > >> site all work for me. > >> > >> ~/I/c/elasticsearch5 ❯❯❯ mvn --version > >> javadoc-fixes ✭ > >> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; > >> 2018-06-17T14:33:14-04:00) > >> Maven home: /usr/local/Cellar/maven/3.5.4/libexec > >> Java version: 1.8.0_151, vendor: Oracle Corporation, runtime: > >> /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre > >> Default locale: en_US, platform encoding: UTF-8 > >> OS name: "mac os x", version: "10.13.2", arch: "x86_64", family: "mac" > >> > >> > >> > >> On Sun, Jun 24, 2018 at 12:40 PM Volodymyr Vysotskyi < > volody...@apache.org > >>> > >> wrote: > >> > >>> I have also checked "mvn javadoc:javadoc javadoc:test-javadoc" on my > >>> machine (Ubuntu, JDK 8, Maven 3.3.9) and it fails on elasticsearch2 > >> module > >>> (didn't check this problem for the next modules). > >>> > >>> "mvn clean -DskipTests site" fails for MongoDB module. > >>> > >>> Kind regards, > >>> Volodymyr Vysotskyi > >>> > >>> > >>> сб, 23 черв. 2018 о 03:12 Julian Hyde <jh...@apache.org> пише: > >>> > >>>> Same failure for me on JDK 8/macOS. The most likely other variable is > >>>> maven: I have 3.5.2 on macOS, 3.5.3 on ubuntu. > >>>> > >>>> I’m mystified why no one else is seeing javadoc failures on macOS. > >>>> > >>>> Julian > >>>> > >>>> > >>>>> On Jun 22, 2018, at 4:35 PM, Julian Hyde <jh...@apache.org> wrote: > >>>>> > >>>>> On JDK 8/ubuntu, “mvn clean -DskipTests site” gives the following > >>>> failure: > >>>>> > >>>>> [ERROR] Failed to execute goal > >>>> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on > >>>> project calcite-mongodb: Error generating > >>>> maven-javadoc-plugin:3.0.1:test-javadoc report: > >>>>> [ERROR] Exit code: 1 - > >>>> > >>> > >> > /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:22: > >>>> error: cannot find symbol > >>>>> [ERROR] import org.apache.calcite.test.CalciteAssert; > >>>>> [ERROR] ^ > >>>>> [ERROR] symbol: class CalciteAssert > >>>>> [ERROR] location: package org.apache.calcite.test > >>>>> [ERROR] > >>>> > >>> > >> > /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:137: > >>>> error: package CalciteAssert does not exist > >>>>> [ERROR] private CalciteAssert.AssertThat assertModel(String model) > >> { > >>>>> [ERROR] ^ > >>>>> [ERROR] > >>>> > >>> > >> > /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:145: > >>>> error: package CalciteAssert does not exist > >>>>> [ERROR] private CalciteAssert.AssertThat assertModel(URL url) { > >>>>> [ERROR] ^ > >>>>> [ERROR] > >>>> > >>> > >> > /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/test/MongoAssertions.java:43: > >>>> error: reference not found > >>>>> [ERROR] /** Similar to {@link CalciteAssert#checkResultUnordered}, > >>> but > >>>> filters strings > >>>>> [ERROR] ^ > >>>>> > >>>>> > >>>>> > >>>>>> On Jun 22, 2018, at 3:46 PM, Andrei Sereda <and...@sereda.cc> > >> wrote: > >>>>>> > >>>>>> Hmm. Both master and c12cb4b0de work for me. Perhaps the issue is > >> JDK > >>> 8 > >>>> vs > >>>>>> 9 ? > >>>>>> > >>>>>> # JDK8 (oracle) / macOS > >>>>>> $ mvn clean -DskipTests site > >>>>>> > >>>>>> On Fri, Jun 22, 2018 at 5:56 PM Julian Hyde <jh...@apache.org> > >> wrote: > >>>>>> > >>>>>>> I just tried it again. The following fails for me (jdk9, ubuntu): > >>>>>>> > >>>>>>> $ git checkout c12cb4b0de > >>>>>>> $ mvn clean -DskipTests site > >>>>>>> > >>>>>>> Constructing Javadoc information... > >>>>>>> 1 error > >>>>>>> [INFO] > >>>>>>> > >>>> > >> ------------------------------------------------------------------------ > >>>>>>> [INFO] Reactor Summary: > >>>>>>> [INFO] > >>>>>>> [INFO] Calcite 1.17.0-SNAPSHOT ............................ FAILURE > >>>> [02:29 > >>>>>>> min] > >>>>>>> [INFO] Calcite Linq4j ..................................... SKIPPED > >>>>>>> [INFO] Calcite Core ....................................... SKIPPED > >>>>>>> [INFO] Calcite Cassandra .................................. SKIPPED > >>>>>>> [INFO] Calcite Druid ...................................... SKIPPED > >>>>>>> [INFO] Calcite Elasticsearch .............................. SKIPPED > >>>>>>> [INFO] Calcite Elasticsearch5 ............................. SKIPPED > >>>>>>> [INFO] Calcite Examples ................................... SKIPPED > >>>>>>> [INFO] Calcite Example CSV ................................ SKIPPED > >>>>>>> [INFO] Calcite Example Function ........................... SKIPPED > >>>>>>> [INFO] Calcite File ....................................... SKIPPED > >>>>>>> [INFO] Calcite Geode ...................................... SKIPPED > >>>>>>> [INFO] Calcite MongoDB .................................... SKIPPED > >>>>>>> [INFO] Calcite Pig ........................................ SKIPPED > >>>>>>> [INFO] Calcite Piglet ..................................... SKIPPED > >>>>>>> [INFO] Calcite Plus ....................................... SKIPPED > >>>>>>> [INFO] Calcite Server ..................................... SKIPPED > >>>>>>> [INFO] Calcite Spark ...................................... SKIPPED > >>>>>>> [INFO] Calcite Splunk ..................................... SKIPPED > >>>>>>> [INFO] Calcite Ubenchmark 1.17.0-SNAPSHOT ................. SKIPPED > >>>>>>> [INFO] > >>>>>>> > >>>> > >> ------------------------------------------------------------------------ > >>>>>>> [INFO] BUILD FAILURE > >>>>>>> [INFO] > >>>>>>> > >>>> > >> ------------------------------------------------------------------------ > >>>>>>> [INFO] Total time: 02:29 min > >>>>>>> [INFO] Finished at: 2018-06-22T14:53:13-07:00 > >>>>>>> [INFO] > >>>>>>> > >>>> > >> ------------------------------------------------------------------------ > >>>>>>> [ERROR] Failed to execute goal > >>>>>>> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) > >> on > >>>>>>> project calcite: Error generating > >>>> maven-javadoc-plugin:3.0.1:test-aggregate > >>>>>>> report: > >>>>>>> [ERROR] Exit code: 1 - > >>>>>>> > >>>> > >>> > >> > /home/jhyde/open1/calcite.4/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29: > >>>>>>> error: package org.elasticsearch.node.internal does not exist > >>>>>>> [ERROR] import > >>>> org.elasticsearch.node.internal.InternalSettingsPreparer; > >>>>>>> [ERROR] ^ > >>>>>>> [ERROR] > >>>>>>> [ERROR] Command line was: /usr/lib/jvm/jdk9/bin/javadoc @options > >>>> @packages > >>>>>>> @argfile > >>>>>>> [ERROR] > >>>>>>> [ERROR] Refer to the generated Javadoc files in > >>>>>>> '/home/jhyde/open1/calcite.4/target/site/testapidocs' dir. > >>>>>>> [ERROR] -> [Help 1] > >>>>>>> [ERROR] > >>>>>>> [ERROR] To see the full stack trace of the errors, re-run Maven > >> with > >>>> the > >>>>>>> -e switch. > >>>>>>> [ERROR] Re-run Maven using the -X switch to enable full debug > >>> logging. > >>>>>>> [ERROR] > >>>>>>> [ERROR] For more information about the errors and possible > >> solutions, > >>>>>>> please read the following articles: > >>>>>>> [ERROR] [Help 1] > >>>>>>> > >>>> > >> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException > >>>>>>> > >>>>>>> > >>>>>>>> On Jun 22, 2018, at 2:41 PM, Michael Mior <mm...@apache.org> > >> wrote: > >>>>>>>> > >>>>>>>> Odd, mvn clean site still works fine for me. > >>>>>>>> -- > >>>>>>>> Michael Mior > >>>>>>>> mm...@apache.org > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Le ven. 22 juin 2018 à 17:12, Julian Hyde <jh...@apache.org> a > >>> écrit > >>>> : > >>>>>>>> > >>>>>>>>> Looks like this change broke “mvn site” (perhaps also “mvn > >>>>>>>>> javadoc:test-javadoc”). > >>>>>>>>> > >>>>>>>>> [ERROR] Failed to execute goal > >>>>>>>>> org.apache.maven.plugins:maven-site-plugin:3.7:site > >> (default-site) > >>> on > >>>>>>>>> project calcite: Error generating > >>>>>>> maven-javadoc-plugin:3.0.1:test-aggregate > >>>>>>>>> report: > >>>>>>>>> [ERROR] Exit code: 1 - > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > /home/jhyde/regress/calcite/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29: > >>>>>>>>> error: package org.elasticsearch.node.internal does not exist > >>>>>>>>> [ERROR] import > >>>> org.elasticsearch.node.internal.InternalSettingsPreparer; > >>>>>>>>> [ERROR] ^ > >>>>>>>>> [ERROR] > >>>>>>>>> > >>>>>>>>>> On Jun 21, 2018, at 3:39 AM, mm...@apache.org wrote: > >>>>>>>>>> > >>>>>>>>>> [CALCITE-2347] running ElasticSearch in embedded mode for unit > >>>> tests of > >>>>>>>>> ES adapter (Andrei Sereda) > >>>>>>>>>> > >>>>>>>>>> After discussion on dev-list Integration tests (for ES) have > >> been > >>>>>>>>> removed. They're now > >>>>>>>>>> superseded by unit tests (which execute queries against a real > >>>> elastic > >>>>>>>>> instance) > >>>>>>>>>> > >>>>>>>>>> Added local file (zips-mini.json) which contains a small subset > >> of > >>>>>>>>> original zips.json > >>>>>>>>>> (allows to bootstrap tests faster) > >>>>>>>>>> > >>>>>>>>>> Created separate ES JUnit rule which can be re-used across > >>> different > >>>>>>>>> tests. > >>>>>>>>>> > >>>>>>>>>> Both v2 and v5 of ES adapters are supported. > >>>>>>>>>> > >>>>>>>>>> Close apache/calcite#716 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Project: http://git-wip-us.apache.org/repos/asf/calcite/repo > >>>>>>>>>> Commit: > >>>> http://git-wip-us.apache.org/repos/asf/calcite/commit/c12cb4b0 > >>>>>>>>>> Tree: > >>> http://git-wip-us.apache.org/repos/asf/calcite/tree/c12cb4b0 > >>>>>>>>>> Diff: > >>> http://git-wip-us.apache.org/repos/asf/calcite/diff/c12cb4b0 > >>>>>>>>>> > >>>>>>>>>> Branch: refs/heads/master > >>>>>>>>>> Commit: c12cb4b0de1baa3f7cbb9952ee350fdd1701662d > >>>>>>>>>> Parents: 37944bb > >>>>>>>>>> Author: Andrei Sereda <and...@nospam.com> > >>>>>>>>>> Authored: Thu May 31 18:19:10 2018 -0400 > >>>>>>>>>> Committer: Michael Mior <mm...@uwaterloo.ca> > >>>>>>>>>> Committed: Thu Jun 21 06:38:50 2018 -0400 > >>>>>>>>>> > >>>>>>>>>> > >>>> ---------------------------------------------------------------------- > >>>>>>>>>> .../AbstractElasticsearchTable.java | 12 + > >>>>>>>>>> .../elasticsearch/ElasticsearchProject.java | 61 ++- > >>>>>>>>>> elasticsearch2/pom.xml | 6 + > >>>>>>>>>> .../Elasticsearch2Enumerator.java | 12 +- > >>>>>>>>>> .../elasticsearch2/Elasticsearch2Schema.java | 16 +- > >>>>>>>>>> .../elasticsearch2/Elasticsearch2Table.java | 9 +- > >>>>>>>>>> .../ElasticSearch2AdapterTest.java | 395 > >>>>>>> ++++++++++++++++++ > >>>>>>>>>> .../elasticsearch2/EmbeddedElasticNode.java | 147 +++++++ > >>>>>>>>>> .../elasticsearch2/EmbeddedElasticRule.java | 97 +++++ > >>>>>>>>>> .../org/apache/calcite/test/ElasticChecker.java | 49 +++ > >>>>>>>>>> .../calcite/test/Elasticsearch2AdapterIT.java | 270 > >>> ------------- > >>>>>>>>>> .../resources/elasticsearch-zips-model.json | 50 --- > >>>>>>>>>> .../src/test/resources/zips-mini.json | 149 +++++++ > >>>>>>>>>> elasticsearch5/pom.xml | 31 ++ > >>>>>>>>>> .../elasticsearch5/Elasticsearch5Schema.java | 17 +- > >>>>>>>>>> .../elasticsearch5/Elasticsearch5Table.java | 11 +- > >>>>>>>>>> .../ElasticSearch5AdapterTest.java | 399 > >>>>>>> +++++++++++++++++++ > >>>>>>>>>> .../elasticsearch5/EmbeddedElasticNode.java | 153 +++++++ > >>>>>>>>>> .../elasticsearch5/EmbeddedElasticRule.java | 98 +++++ > >>>>>>>>>> .../org/apache/calcite/test/ElasticChecker.java | 49 +++ > >>>>>>>>>> .../calcite/test/Elasticsearch5AdapterIT.java | 270 > >>> ------------- > >>>>>>>>>> .../resources/elasticsearch-zips-model.json | 50 --- > >>>>>>>>>> elasticsearch5/src/test/resources/log4j2.xml | 16 + > >>>>>>>>>> .../src/test/resources/zips-mini.json | 149 +++++++ > >>>>>>>>>> pom.xml | 20 +- > >>>>>>>>>> 25 files changed, 1866 insertions(+), 670 deletions(-) > >>>>>>>>>> > >>>> ---------------------------------------------------------------------- > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java > >>>>>>>>>> > >>>> ---------------------------------------------------------------------- > >>>>>>>>>> diff --git > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java > >>>>>>>>>> index 0980469..8cc5933 100644 > >>>>>>>>>> --- > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java > >>>>>>>>>> +++ > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java > >>>>>>>>>> @@ -75,6 +75,18 @@ public abstract class > >>> AbstractElasticsearchTable > >>>>>>>>> extends AbstractQueryableTable > >>>>>>>>>> relOptTable, this, null); > >>>>>>>>>> } > >>>>>>>>>> > >>>>>>>>>> + /** > >>>>>>>>>> + * In ES 5.x scripted fields start with {@code > >>>> params._source.foo} > >>>>>>>>> while in ES2.x > >>>>>>>>>> + * {@code _source.foo}. Helper method to build correct query > >>>> based > >>>>>>> on > >>>>>>>>> runtime version of elastic. > >>>>>>>>>> + * > >>>>>>>>>> + * @see <a href=" > >>>>>>>>> https://github.com/elastic/elasticsearch/issues/20068";>_source > >>>>>>>>> variable</a> > >>>>>>>>>> + * @see <a href=" > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-fields.html > >>>>>>> ">Scripted > >>>>>>>>> Fields</a> > >>>>>>>>>> + */ > >>>>>>>>>> + protected String scriptedFieldPrefix() { > >>>>>>>>>> + // this is default pattern starting 5.x > >>>>>>>>>> + return "params._source"; > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> /** Executes a "find" operation on the underlying type. > >>>>>>>>>> * > >>>>>>>>>> * <p>For example, > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java > >>>>>>>>>> > >>>> ---------------------------------------------------------------------- > >>>>>>>>>> diff --git > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java > >>>>>>>>>> index b42abd7..961c8b0 100644 > >>>>>>>>>> --- > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java > >>>>>>>>>> +++ > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java > >>>>>>>>>> @@ -27,11 +27,15 @@ import > >>>>>>>>> org.apache.calcite.rel.metadata.RelMetadataQuery; > >>>>>>>>>> import org.apache.calcite.rel.type.RelDataType; > >>>>>>>>>> import org.apache.calcite.rex.RexNode; > >>>>>>>>>> import org.apache.calcite.util.Pair; > >>>>>>>>>> -import org.apache.calcite.util.Util; > >>>>>>>>>> + > >>>>>>>>>> +import com.google.common.base.Function; > >>>>>>>>>> +import com.google.common.collect.Lists; > >>>>>>>>>> > >>>>>>>>>> import java.util.ArrayList; > >>>>>>>>>> import java.util.List; > >>>>>>>>>> > >>>>>>>>>> +import javax.annotation.Nullable; > >>>>>>>>>> + > >>>>>>>>>> /** > >>>>>>>>>> * Implementation of {@link org.apache.calcite.rel.core.Project} > >>>>>>>>>> * relational expression in Elasticsearch. > >>>>>>>>>> @@ -57,41 +61,60 @@ public class ElasticsearchProject extends > >>>> Project > >>>>>>>>> implements ElasticsearchRel { > >>>>>>>>>> implementor.visitChild(0, getInput()); > >>>>>>>>>> > >>>>>>>>>> final List<String> inFields = > >>>>>>>>>> - > >>>>>>>>> > >>> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType()); > >>>>>>>>>> + > >>>>>>>>> > >>> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType()); > >>>>>>>>>> final ElasticsearchRules.RexToElasticsearchTranslator > >> translator > >>> = > >>>>>>>>>> - new ElasticsearchRules.RexToElasticsearchTranslator( > >>>>>>>>>> - (JavaTypeFactory) getCluster().getTypeFactory(), > >>>>>>> inFields); > >>>>>>>>>> + new > >> ElasticsearchRules.RexToElasticsearchTranslator( > >>>>>>>>>> + (JavaTypeFactory) > >>>> getCluster().getTypeFactory(), > >>>>>>>>> inFields); > >>>>>>>>>> > >>>>>>>>>> - final List<String> findItems = new ArrayList<>(); > >>>>>>>>>> - final List<String> scriptFieldItems = new ArrayList<>(); > >>>>>>>>>> + final List<String> fields = new ArrayList<>(); > >>>>>>>>>> + final List<String> scriptFields = new ArrayList<>(); > >>>>>>>>>> for (Pair<RexNode, String> pair: getNamedProjects()) { > >>>>>>>>>> final String name = pair.right; > >>>>>>>>>> final String expr = pair.left.accept(translator); > >>>>>>>>>> > >>>>>>>>>> if (expr.equals("\"" + name + "\"")) { > >>>>>>>>>> - findItems.add(ElasticsearchRules.quote(name)); > >>>>>>>>>> + fields.add(name); > >>>>>>>>>> } else if (expr.matches("\"literal\":.+")) { > >>>>>>>>>> - scriptFieldItems.add(ElasticsearchRules.quote(name) > >>>>>>>>>> - + ":{\"script\": " > >>>>>>>>>> - + expr.split(":")[1] + "}"); > >>>>>>>>>> + scriptFields.add(ElasticsearchRules.quote(name) > >>>>>>>>>> + + ":{\"script\": " > >>>>>>>>>> + + expr.split(":")[1] + "}"); > >>>>>>>>>> } else { > >>>>>>>>>> - scriptFieldItems.add(ElasticsearchRules.quote(name) > >>>>>>>>>> - + ":{\"script\":\"params._source." > >>>>>>>>>> - + expr.replaceAll("\"", "") + "\"}"); > >>>>>>>>>> + scriptFields.add(ElasticsearchRules.quote(name) > >>>>>>>>>> + + ":{\"script\":" > >>>>>>>>>> + // _source (ES2) vs params._source (ES5) > >>>>>>>>>> + + "\"" + > >>>>>>>>> implementor.elasticsearchTable.scriptedFieldPrefix() + "." > >>>>>>>>>> + + expr.replaceAll("\"", "") + "\"}"); > >>>>>>>>>> + } > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + StringBuilder query = new StringBuilder(); > >>>>>>>>>> + if (scriptFields.isEmpty()) { > >>>>>>>>>> + List<String> newList = Lists.transform(fields, new > >>>>>>>>> Function<String, String>() { > >>>>>>>>>> + @Nullable > >>>>>>>>>> + @Override public String apply(@Nullable String input) { > >>>>>>>>>> + return ElasticsearchRules.quote(input); > >>>>>>>>>> + } > >>>>>>>>>> + }); > >>>>>>>>>> + > >>>>>>>>>> + final String findString = String.join(", ", newList); > >>>>>>>>>> + query.append("\"_source\" : > >>>> [").append(findString).append("]"); > >>>>>>>>>> + } else { > >>>>>>>>>> + // if scripted fields are present, ES ignores _source > >>>> attribute > >>>>>>>>>> + for (String field: fields) { > >>>>>>>>>> + scriptFields.add(ElasticsearchRules.quote(field) + > >>>>>>>>> ":{\"script\": " > >>>>>>>>>> + // _source (ES2) vs params._source (ES5) > >>>>>>>>>> + + "\"" + > >>>>>>>>> implementor.elasticsearchTable.scriptedFieldPrefix() + "." > >>>>>>>>>> + + field + "\"}"); > >>>>>>>>>> } > >>>>>>>>>> + query.append("\"script_fields\": {" + String.join(", ", > >>>>>>>>> scriptFields) + "}"); > >>>>>>>>>> } > >>>>>>>>>> - final String findString = Util.toString(findItems, "", ", > >> ", > >>>> ""); > >>>>>>>>>> - final String scriptFieldString = "\"script_fields\": {" > >>>>>>>>>> - + Util.toString(scriptFieldItems, "", ", ", "") + "}"; > >>>>>>>>>> - final String fieldString = "\"_source\" : [" + findString + > >>> "]" > >>>>>>>>>> - + ", " + scriptFieldString; > >>>>>>>>>> > >>>>>>>>>> for (String opfield : implementor.list) { > >>>>>>>>>> if (opfield.startsWith("\"_source\"")) { > >>>>>>>>>> implementor.list.remove(opfield); > >>>>>>>>>> } > >>>>>>>>>> } > >>>>>>>>>> - implementor.add(fieldString); > >>>>>>>>>> + implementor.add(query.toString()); > >>>>>>>>>> } > >>>>>>>>>> } > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/pom.xml > >>>>>>>>>> > >>>> ---------------------------------------------------------------------- > >>>>>>>>>> diff --git a/elasticsearch2/pom.xml b/elasticsearch2/pom.xml > >>>>>>>>>> index f24622c..6fbee03 100644 > >>>>>>>>>> --- a/elasticsearch2/pom.xml > >>>>>>>>>> +++ b/elasticsearch2/pom.xml > >>>>>>>>>> @@ -73,6 +73,12 @@ limitations under the License. > >>>>>>>>>> <version>${elasticsearch-java-driver.version}</version> > >>>>>>>>>> </dependency> > >>>>>>>>>> <dependency> > >>>>>>>>>> + <!-- Lang groovy dependency is needed for testing with > >>>> embedded > >>>>>>>>> ES (scripted fields like loc[0]) --> > >>>>>>>>>> + <groupId>org.elasticsearch.module</groupId> > >>>>>>>>>> + <artifactId>lang-groovy</artifactId> > >>>>>>>>>> + <scope>test</scope> > >>>>>>>>>> + </dependency> > >>>>>>>>>> + <dependency> > >>>>>>>>>> <groupId>com.carrotsearch</groupId> > >>>>>>>>>> <artifactId>hppc</artifactId> > >>>>>>>>>> <version>${hppc.version}</version> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java > >>>>>>>>>> > >>>> ---------------------------------------------------------------------- > >>>>>>>>>> diff --git > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java > >>>>>>>>>> index 84370ab..c3d2ac0 100644 > >>>>>>>>>> --- > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java > >>>>>>>>>> +++ > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java > >>>>>>>>>> @@ -26,6 +26,7 @@ import org.elasticsearch.search.SearchHit; > >>>>>>>>>> import java.util.Date; > >>>>>>>>>> import java.util.Iterator; > >>>>>>>>>> import java.util.List; > >>>>>>>>>> +import java.util.Locale; > >>>>>>>>>> import java.util.Map; > >>>>>>>>>> > >>>>>>>>>> /** > >>>>>>>>>> @@ -101,15 +102,18 @@ public class Elasticsearch2Enumerator > >>>> implements > >>>>>>>>> Enumerator<Object> { > >>>>>>>>>> private static Function1<SearchHit, Object[]> listGetter( > >>>>>>>>>> final List<Map.Entry<String, Class>> fields) { > >>>>>>>>>> return new Function1<SearchHit, Object[]>() { > >>>>>>>>>> - public Object[] apply(SearchHit searchHitFields) { > >>>>>>>>>> + public Object[] apply(SearchHit hit) { > >>>>>>>>>> Object[] objects = new Object[fields.size()]; > >>>>>>>>>> for (int i = 0; i < fields.size(); i++) { > >>>>>>>>>> final Map.Entry<String, Class> field = fields.get(i); > >>>>>>>>>> final String name = field.getKey(); > >>>>>>>>>> - if (searchHitFields.fields().isEmpty()) { > >>>>>>>>>> - objects[i] = > >>>>>>> convert(searchHitFields.getSource().get(name), > >>>>>>>>> field.getValue()); > >>>>>>>>>> + if (hit.fields().isEmpty()) { > >>>>>>>>>> + objects[i] = convert(hit.getSource().get(name), > >>>>>>>>> field.getValue()); > >>>>>>>>>> + } else if (hit.fields().containsKey(name)) { > >>>>>>>>>> + objects[i] = convert(hit.field(name).getValue(), > >>>>>>>>> field.getValue()); > >>>>>>>>>> } else { > >>>>>>>>>> - objects[i] = > >>>>>>>>> convert(searchHitFields.field(name).getValue(), > >> field.getValue()); > >>>>>>>>>> + throw new IllegalStateException( > >>>>>>>>>> + String.format(Locale.getDefault(), "No > >> result > >>>> for > >>>>>>>>> %s", field)); > >>>>>>>>>> } > >>>>>>>>>> } > >>>>>>>>>> return objects; > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java > >>>>>>>>>> > >>>> ---------------------------------------------------------------------- > >>>>>>>>>> diff --git > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java > >>>>>>>>>> index 668402b..46e3fc5 100644 > >>>>>>>>>> --- > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java > >>>>>>>>>> +++ > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java > >>>>>>>>>> @@ -22,6 +22,9 @@ import > >>>> org.apache.calcite.schema.impl.AbstractSchema; > >>>>>>>>>> > >>>>>>>>>> import com.carrotsearch.hppc.cursors.ObjectObjectCursor; > >>>>>>>>>> > >>>>>>>>>> +import com.google.common.annotations.VisibleForTesting; > >>>>>>>>>> +import com.google.common.base.Preconditions; > >>>>>>>>>> + > >>>>>>>>>> import com.google.common.collect.ImmutableList; > >>>>>>>>>> import com.google.common.collect.ImmutableMap; > >>>>>>>>>> > >>>>>>>>>> @@ -86,6 +89,16 @@ public class Elasticsearch2Schema extends > >>>>>>>>> AbstractSchema > >>>>>>>>>> } > >>>>>>>>>> } > >>>>>>>>>> > >>>>>>>>>> + /** > >>>>>>>>>> + * Allows schema to be instantiated from existing elastic > >>> search > >>>>>>>>> client. > >>>>>>>>>> + * This constructor is used in tests. > >>>>>>>>>> + */ > >>>>>>>>>> + @VisibleForTesting > >>>>>>>>>> + Elasticsearch2Schema(Client client, String index) { > >>>>>>>>>> + this.client = Preconditions.checkNotNull(client, "client"); > >>>>>>>>>> + this.index = Preconditions.checkNotNull(index, "index"); > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> @Override protected Map<String, Table> getTableMap() { > >>>>>>>>>> final ImmutableMap.Builder<String, Table> builder = > >>>>>>>>> ImmutableMap.builder(); > >>>>>>>>>> > >>>>>>>>>> @@ -120,7 +133,8 @@ public class Elasticsearch2Schema extends > >>>>>>>>> AbstractSchema > >>>>>>>>>> > >>>>>>>>>> final List<DiscoveryNode> nodes = > >>>>>>>>> ImmutableList.copyOf(transportClient.connectedNodes()); > >>>>>>>>>> if (nodes.isEmpty()) { > >>>>>>>>>> - throw new RuntimeException("Cannot connect to any > >>>> elasticsearch > >>>>>>>>> nodes"); > >>>>>>>>>> + throw new IllegalStateException("Cannot connect to any > >>>>>>>>> elasticsearch node: " > >>>>>>>>>> + + transportNodes); > >>>>>>>>>> } > >>>>>>>>>> > >>>>>>>>>> client = transportClient; > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java > >>>>>>>>>> > >>>> ---------------------------------------------------------------------- > >>>>>>>>>> diff --git > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java > >>>>>>>>>> index 636aa5f..2928835 100644 > >>>>>>>>>> --- > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java > >>>>>>>>>> +++ > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java > >>>>>>>>>> @@ -45,8 +45,15 @@ public class Elasticsearch2Table extends > >>>>>>>>> AbstractElasticsearchTable { > >>>>>>>>>> this.client = client; > >>>>>>>>>> } > >>>>>>>>>> > >>>>>>>>>> + /** > >>>>>>>>>> + * ES version 2.x. To access document attributes ES2 uses > >>> {@code > >>>>>>>>> _source.foo} syntax. > >>>>>>>>>> + */ > >>>>>>>>>> + @Override protected String scriptedFieldPrefix() { > >>>>>>>>>> + return "_source"; > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> @Override protected Enumerable<Object> find(String index, > >>>> List<String> > >>>>>>>>> ops, > >>>>>>>>>> - List<Map.Entry<String, Class>> fields) { > >>>>>>>>>> + > >>>> List<Map.Entry<String, > >>>>>>>>> Class>> fields) { > >>>>>>>>>> final String dbName = index; > >>>>>>>>>> > >>>>>>>>>> final String queryString = "{" + Util.toString(ops, "", ", ", > >>> "") + > >>>>>>>>> "}"; > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java > >>>>>>>>>> > >>>> ---------------------------------------------------------------------- > >>>>>>>>>> diff --git > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java > >>>>>>>>>> new file mode 100644 > >>>>>>>>>> index 0000000..287e094 > >>>>>>>>>> --- /dev/null > >>>>>>>>>> +++ > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java > >>>>>>>>>> @@ -0,0 +1,395 @@ > >>>>>>>>>> +/* > >>>>>>>>>> + * Licensed to the Apache Software Foundation (ASF) under one > >> or > >>>> more > >>>>>>>>>> + * contributor license agreements. See the NOTICE file > >>> distributed > >>>>>>> with > >>>>>>>>>> + * this work for additional information regarding copyright > >>>> ownership. > >>>>>>>>>> + * The ASF licenses this file to you under the Apache License, > >>>> Version > >>>>>>>>> 2.0 > >>>>>>>>>> + * (the "License"); you may not use this file except in > >>> compliance > >>>>>>> with > >>>>>>>>>> + * the License. You may obtain a copy of the License at > >>>>>>>>>> + * > >>>>>>>>>> + * http://www.apache.org/licenses/LICENSE-2.0 > >>>>>>>>>> + * > >>>>>>>>>> + * Unless required by applicable law or agreed to in writing, > >>>> software > >>>>>>>>>> + * distributed under the License is distributed on an "AS IS" > >>>> BASIS, > >>>>>>>>>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express > >>> or > >>>>>>>>> implied. > >>>>>>>>>> + * See the License for the specific language governing > >>> permissions > >>>> and > >>>>>>>>>> + * limitations under the License. > >>>>>>>>>> + */ > >>>>>>>>>> +package org.apache.calcite.adapter.elasticsearch2; > >>>>>>>>>> + > >>>>>>>>>> +import org.apache.calcite.jdbc.CalciteConnection; > >>>>>>>>>> +import org.apache.calcite.schema.SchemaPlus; > >>>>>>>>>> +import org.apache.calcite.schema.impl.ViewTable; > >>>>>>>>>> +import org.apache.calcite.schema.impl.ViewTableMacro; > >>>>>>>>>> +import org.apache.calcite.test.CalciteAssert; > >>>>>>>>>> +import org.apache.calcite.test.ElasticChecker; > >>>>>>>>>> + > >>>>>>>>>> +import com.google.common.io.LineProcessor; > >>>>>>>>>> +import com.google.common.io.Resources; > >>>>>>>>>> + > >>>>>>>>>> +import org.elasticsearch.action.bulk.BulkItemResponse; > >>>>>>>>>> +import org.elasticsearch.action.bulk.BulkRequestBuilder; > >>>>>>>>>> +import org.elasticsearch.action.bulk.BulkResponse; > >>>>>>>>>> +import org.elasticsearch.common.xcontent.XContentBuilder; > >>>>>>>>>> +import org.elasticsearch.common.xcontent.XContentFactory; > >>>>>>>>>> + > >>>>>>>>>> +import org.junit.BeforeClass; > >>>>>>>>>> +import org.junit.ClassRule; > >>>>>>>>>> +import org.junit.Test; > >>>>>>>>>> + > >>>>>>>>>> +import java.io.IOException; > >>>>>>>>>> +import java.nio.charset.StandardCharsets; > >>>>>>>>>> +import java.sql.Connection; > >>>>>>>>>> +import java.sql.DriverManager; > >>>>>>>>>> +import java.sql.SQLException; > >>>>>>>>>> +import java.util.Arrays; > >>>>>>>>>> +import java.util.Collections; > >>>>>>>>>> +import java.util.Locale; > >>>>>>>>>> + > >>>>>>>>>> +/** > >>>>>>>>>> + * Set of tests for ES adapter. Uses real instance via {@link > >>>>>>>>> EmbeddedElasticRule}. Document > >>>>>>>>>> + * source is local {@code zips-mini.json} file (located in the > >>>>>>>>> classpath). > >>>>>>>>>> + */ > >>>>>>>>>> +public class ElasticSearch2AdapterTest { > >>>>>>>>>> + > >>>>>>>>>> + @ClassRule //init once for all tests > >>>>>>>>>> + public static final EmbeddedElasticRule NODE = > >>>>>>>>> EmbeddedElasticRule.create(); > >>>>>>>>>> + > >>>>>>>>>> + private static final String ZIPS = "zips"; > >>>>>>>>>> + > >>>>>>>>>> + /** > >>>>>>>>>> + * Used to create {@code zips} index and insert some data > >>>>>>>>>> + */ > >>>>>>>>>> + @BeforeClass > >>>>>>>>>> + public static void setupInstance() throws Exception { > >>>>>>>>>> + // define mapping so fields are searchable (term query) > >>>>>>>>>> + XContentBuilder mapping = > >>>>>>>>> XContentFactory.jsonBuilder().startObject() > >>>>>>>>>> + .startObject("properties") > >>>>>>>>>> + .startObject("city").field("type", "string") > >>>>>>>>>> + .field("index", "not_analyzed").endObject() > >>>>>>>>>> + .startObject("state").field("type", "string") > >>>>>>>>>> + .field("index", "not_analyzed").endObject() > >>>>>>>>>> + .startObject("pop").field("type", > >> "long").endObject() > >>>>>>>>>> + .endObject() > >>>>>>>>>> + .endObject(); > >>>>>>>>>> + > >>>>>>>>>> + // create index > >>>>>>>>>> + NODE.client().admin().indices() > >>>>>>>>>> + .prepareCreate(ZIPS) > >>>>>>>>>> + .addMapping(ZIPS, mapping) > >>>>>>>>>> + .get(); > >>>>>>>>>> + > >>>>>>>>>> + BulkRequestBuilder bulk = > >>>>>>>>> NODE.client().prepareBulk().setRefresh(true); > >>>>>>>>>> + > >>>>>>>>>> + // load records from file > >>>>>>>>>> + > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > Resources.readLines(ElasticSearch2AdapterTest.class.getResource("/zips-mini.json"), > >>>>>>>>>> + StandardCharsets.UTF_8, new LineProcessor<Void>() { > >>>>>>>>>> + @Override public boolean processLine(String line) > >>>> throws > >>>>>>>>> IOException { > >>>>>>>>>> + line = line.replaceAll("_id", "id"); // _id is > >> a > >>>>>>>>> reserved attribute in ES > >>>>>>>>>> + bulk.add(NODE.client().prepareIndex(ZIPS, > >>>>>>>>> ZIPS).setSource(line)); > >>>>>>>>>> + return true; > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + @Override public Void getResult() { > >>>>>>>>>> + return null; > >>>>>>>>>> + } > >>>>>>>>>> + }); > >>>>>>>>>> + > >>>>>>>>>> + if (bulk.numberOfActions() == 0) { > >>>>>>>>>> + throw new IllegalStateException("No records to be > >>> indexed"); > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + BulkResponse response = bulk.execute().get(); > >>>>>>>>>> + > >>>>>>>>>> + if (response.hasFailures()) { > >>>>>>>>>> + throw new IllegalStateException( > >>>>>>>>>> + String.format(Locale.getDefault(), "Failed to > >>>> populate > >>>>>>>>> %s:\n%s", NODE.httpAddress(), > >>>>>>>>>> + > >>>>>>>>> > >>> Arrays.stream(response.getItems()).filter(BulkItemResponse::isFailed) > >>>>>>>>>> + > >>>>>>>>> > >>>>>>> > >>>> > >>> > >> > .map(BulkItemResponse::getFailureMessage).findFirst().orElse("<unknown>"))); > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + private CalciteAssert.ConnectionFactory > >> newConnectionFactory() > >>> { > >>>>>>>>>> + return new CalciteAssert.ConnectionFactory() { > >>>>>>>>>> + @Override public Connection createConnection() throws > >>>>>>>>> SQLException { > >>>>>>>>>> + final Connection connection = > >>>>>>>>> DriverManager.getConnection("jdbc:calcite:"); > >>>>>>>>>> + final SchemaPlus root = > >>>>>>>>> connection.unwrap(CalciteConnection.class).getRootSchema(); > >>>>>>>>>> + > >>>>>>>>>> + root.add("elastic", new > >>> Elasticsearch2Schema(NODE.client(), > >>>>>>>>> ZIPS)); > >>>>>>>>>> + > >>>>>>>>>> + // add calcite view programmatically > >>>>>>>>>> + final String viewSql = "select cast(_MAP['city'] AS > >>>>>>>>> varchar(20)) AS \"city\", " > >>>>>>>>>> + + " cast(_MAP['loc'][0] AS float) AS > >>>> \"longitude\",\n" > >>>>>>>>>> + + " cast(_MAP['loc'][1] AS float) AS > >>>> \"latitude\",\n" > >>>>>>>>>> + + " cast(_MAP['pop'] AS integer) AS \"pop\", " > >>>>>>>>>> + + " cast(_MAP['state'] AS varchar(2)) AS > >>> \"state\", > >>>> " > >>>>>>>>>> + + " cast(_MAP['id'] AS varchar(5)) AS \"id\" " > >>>>>>>>>> + + "from \"elastic\".\"zips\""; > >>>>>>>>>> + > >>>>>>>>>> + ViewTableMacro macro = ViewTable.viewMacro(root, > >> viewSql, > >>>>>>>>>> + Collections.singletonList("elastic"), > >>>>>>>>> Arrays.asList("elastic", "view"), false); > >>>>>>>>>> + root.add("ZIPS", macro); > >>>>>>>>>> + > >>>>>>>>>> + return connection; > >>>>>>>>>> + } > >>>>>>>>>> + }; > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + private CalciteAssert.AssertThat calciteAssert() { > >>>>>>>>>> + return CalciteAssert.that() > >>>>>>>>>> + .with(newConnectionFactory()); > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + /** > >>>>>>>>>> + * Tests using calcite view > >>>>>>>>>> + */ > >>>>>>>>>> + @Test > >>>>>>>>>> + public void view() throws Exception { > >>>>>>>>>> + calciteAssert() > >>>>>>>>>> + .query("select * from zips where \"city\" = > >>> 'BROOKLYN'") > >>>>>>>>>> + .returns("city=BROOKLYN; longitude=-73.956985; > >>>>>>>>> latitude=40.646694; " > >>>>>>>>>> + + "pop=111396; state=NY; id=11226\n") > >>>>>>>>>> + .returnsCount(1); > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + @Test > >>>>>>>>>> + public void emptyResult() { > >>>>>>>>>> + CalciteAssert.that() > >>>>>>>>>> + .with(newConnectionFactory()) > >>>>>>>>>> + .query("select * from zips limit 0") > >>>>>>>>>> + .returnsCount(0); > >>>>>>>>>> + > >>>>>>>>>> + CalciteAssert.that() > >>>>>>>>>> + .with(newConnectionFactory()) > >>>>>>>>>> + .query("select * from \"elastic\".\"zips\" where > >>>>>>>>> _MAP['Foo'] = '_MISSING_'") > >>>>>>>>>> + .returnsCount(0); > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + @Test > >>>>>>>>>> + public void basic() throws Exception { > >>>>>>>>>> + CalciteAssert.that() > >>>>>>>>>> + .with(newConnectionFactory()) > >>>>>>>>>> + .query("select * from \"elastic\".\"zips\" where > >>>>>>>>> _MAP['city'] = 'BROOKLYN'") > >>>>>>>>>> + .returnsCount(1); > >>>>>>>>>> + > >>>>>>>>>> + CalciteAssert.that() > >>>>>>>>>> + .with(newConnectionFactory()) > >>>>>>>>>> + .query("select * from \"elastic\".\"zips\" where" > >>>>>>>>>> + + " _MAP['city'] in ('BROOKLYN', > >>>> 'WASHINGTON')") > >>>>>>>>>> + .returnsCount(2); > >>>>>>>>>> + > >>>>>>>>>> + // lower-case > >>>>>>>>>> + CalciteAssert.that() > >>>>>>>>>> + .with(newConnectionFactory()) > >>>>>>>>>> + .query("select * from \"elastic\".\"zips\" where " > >>>>>>>>>> + + "_MAP['city'] in ('brooklyn', 'Brooklyn', > >>>>>>>>> 'BROOK') ") > >>>>>>>>>> + .returnsCount(0); > >>>>>>>>>> + > >>>>>>>>>> + // missing field > >>>>>>>>>> + CalciteAssert.that() > >>>>>>>>>> + .with(newConnectionFactory()) > >>>>>>>>>> + .query("select * from \"elastic\".\"zips\" where > >>>>>>>>> _MAP['CITY'] = 'BROOKLYN'") > >>>>>>>>>> + .returnsCount(0); > >>>>>>>>>> + > >>>>>>>>>> + // limit works > >>>>>>>>>> + CalciteAssert.that() > >>>>>>>>>> + .with(newConnectionFactory()) > >>>>>>>>>> + .query("select * from \"elastic\".\"zips\" limit > >> 42") > >>>>>>>>>> + .returnsCount(42); > >>>>>>>>>> + > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + @Test public void testSort() { > >>>>>>>>>> + final String explain = > >>>> "PLAN=ElasticsearchToEnumerableConverter\n" > >>>>>>>>>> + + " ElasticsearchSort(sort0=[$4], dir0=[ASC])\n" > >>>>>>>>>> + + " ElasticsearchProject(city=[CAST(ITEM($0, > >>>>>>>>> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE > >>>>>>>>> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0, > >>> 'loc'), > >>>>>>>>> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT], > >>>>>>>>> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0, > >>>>>>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE > >>>>>>>>> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0, > >> 'id')):VARCHAR(5) > >>>>>>>>> CHARACTER SET \"ISO-8859-1\" COLLATE > >>>> \"ISO-8859-1$en_US$primary\"])\n" > >>>>>>>>>> + + " ElasticsearchTableScan(table=[[elastic, > >>>> zips]])"; > >>>>>>>>>> + > >>>>>>>>>> + calciteAssert() > >>>>>>>>>> + .query("select * from zips order by \"state\"") > >>>>>>>>>> + .returnsCount(10) > >>>>>>>>>> + .explainContains(explain); > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + @Test public void testSortLimit() { > >>>>>>>>>> + final String sql = "select \"state\", \"id\" from zips\n" > >>>>>>>>>> + + "order by \"state\", \"id\" offset 2 rows fetch > >>> next > >>>> 3 > >>>>>>>>> rows only"; > >>>>>>>>>> + calciteAssert() > >>>>>>>>>> + .query(sql) > >>>>>>>>>> + .returnsUnordered("state=AK; id=99801", > >>>>>>>>>> + "state=AL; id=35215", > >>>>>>>>>> + "state=AL; id=35401") > >>>>>>>>>> + .queryContains( > >>>>>>>>>> + ElasticChecker.elasticsearchChecker( > >>>>>>>>>> + "\"_source\" : [\"state\", > >> \"id\"]", > >>>>>>>>>> + "\"sort\": [ {\"state\": \"asc\"}, > >>>>>>> {\"id\": > >>>>>>>>> \"asc\"}]", > >>>>>>>>>> + "\"from\": 2", > >>>>>>>>>> + "\"size\": 3")); > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + > >>>>>>>>>> + > >>>>>>>>>> + @Test public void testOffsetLimit() { > >>>>>>>>>> + final String sql = "select \"state\", \"id\" from zips\n" > >>>>>>>>>> + + "offset 2 fetch next 3 rows only"; > >>>>>>>>>> + calciteAssert() > >>>>>>>>>> + .query(sql) > >>>>>>>>>> + .runs() > >>>>>>>>>> + .queryContains( > >>>>>>>>>> + ElasticChecker.elasticsearchChecker( > >>>>>>>>>> + "\"from\": 2", > >>>>>>>>>> + "\"size\": 3", > >>>>>>>>>> + "\"_source\" : [\"state\", > >>> \"id\"]")); > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + @Test public void testLimit() { > >>>>>>>>>> + final String sql = "select \"state\", \"id\" from zips\n" > >>>>>>>>>> + + "fetch next 3 rows only"; > >>>>>>>>>> + > >>>>>>>>>> + calciteAssert() > >>>>>>>>>> + .query(sql) > >>>>>>>>>> + .runs() > >>>>>>>>>> + .queryContains( > >>>>>>>>>> + ElasticChecker.elasticsearchChecker( > >>>>>>>>>> + "\"size\": 3", > >>>>>>>>>> + "\"_source\" : [\"state\", > >>> \"id\"]")); > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + @Test public void testFilterSort() { > >>>>>>>>>> + final String sql = "select * from zips\n" > >>>>>>>>>> + + "where \"state\" = 'CA' and \"id\" >= '70000'\n" > >>>>>>>>>> + + "order by \"state\", \"id\""; > >>>>>>>>>> + final String explain = > >>>> "PLAN=ElasticsearchToEnumerableConverter\n" > >>>>>>>>>> + + " ElasticsearchSort(sort0=[$4], sort1=[$5], > >>>> dir0=[ASC], > >>>>>>>>> dir1=[ASC])\n" > >>>>>>>>>> + + " ElasticsearchProject(city=[CAST(ITEM($0, > >>>>>>>>> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE > >>>>>>>>> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0, > >>> 'loc'), > >>>>>>>>> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT], > >>>>>>>>> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0, > >>>>>>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE > >>>>>>>>> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0, > >> 'id')):VARCHAR(5) > >>>>>>>>> CHARACTER SET \"ISO-8859-1\" COLLATE > >>>> \"ISO-8859-1$en_US$primary\"])\n" > >>>>>>>>>> + + " > >>>>>>> ElasticsearchFilter(condition=[AND(=(CAST(ITEM($0, > >>>>>>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE > >>>>>>>>> \"ISO-8859-1$en_US$primary\", 'CA'), >=(CAST(ITEM($0, > >>>> 'id')):VARCHAR(5) > >>>>>>>>> CHARACTER SET \"ISO-8859-1\" COLLATE > >> \"ISO-8859-1$en_US$primary\", > >>>>>>>>> '70000'))])\n" > >>>>>>>>>> + + " ElasticsearchTableScan(table=[[elastic, > >>>>>>> zips]])"; > >>>>>>>>>> + calciteAssert() > >>>>>>>>>> + .query(sql) > >>>>>>>>>> + .returnsOrdered("city=LOS ANGELES; > >>>> longitude=-118.258189; > >>>>>>>>> latitude=34.007856; " > >>>>>>>>>> + + "pop=96074; state=CA; id=90011", > >>>>>>>>>> + "city=BELL GARDENS; longitude=-118.17205; > >>>>>>>>> latitude=33.969177; " > >>>>>>>>>> + + "pop=99568; state=CA; id=90201", > >>>>>>>>>> + "city=NORWALK; longitude=-118.081767; > >>>>>>>>> latitude=33.90564; " > >>>>>>>>>> + + "pop=94188; state=CA; id=90650") > >>>>>>>>>> + .queryContains( > >>>>>>>>>> + > >>> ElasticChecker.elasticsearchChecker("\"query\" > >>>> : " > >>>>>>>>>> + + > >>>>>>>>> "{\"constant_score\":{\"filter\":{\"bool\":" > >>>>>>>>>> + + > >>>>>>>>> "{\"must\":[{\"term\":{\"state\":\"CA\"}}," > >>>>>>>>>> + + > >>>>>>>>> "{\"range\":{\"id\":{\"gte\":\"70000\"}}}]}}}}", > >>>>>>>>>> + "\"script_fields\": > >>>>>>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, " > >>>>>>>>>> + + > >>>>>>>>> "\"latitude\":{\"script\":\"_source.loc[1]\"}, " > >>>>>>>>>> + + "\"city\":{\"script\": > >>>>>>>>> \"_source.city\"}, " > >>>>>>>>>> + + "\"pop\":{\"script\": > >>>>>>>>> \"_source.pop\"}, " > >>>>>>>>>> + + "\"state\":{\"script\": > >>>>>>>>> \"_source.state\"}, " > >>>>>>>>>> + + "\"id\":{\"script\": \"_ > >>>>>>> source.id > >>>>>>>>> \"}}", > >>>>>>>>>> + "\"sort\": [ {\"state\": \"asc\"}, > >>>>>>> {\"id\": > >>>>>>>>> \"asc\"}]")) > >>>>>>>>>> + .explainContains(explain); > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + @Test public void testFilterSortDesc() { > >>>>>>>>>> + final String sql = "select * from zips\n" > >>>>>>>>>> + + "where \"pop\" BETWEEN 95000 AND 100000\n" > >>>>>>>>>> + + "order by \"state\" desc, \"pop\""; > >>>>>>>>>> + calciteAssert() > >>>>>>>>>> + .query(sql) > >>>>>>>>>> + .limit(4) > >>>>>>>>>> + .returnsOrdered( > >>>>
