I have updated existing PR <https://github.com/apache/calcite/pull/716> and migrated all integration tests to run against local instance. In fact, integration tests have become unit tests.
Both ES2 and ES5 adapters have been fixed and switched to query embedded ES node. Like mongo adapter, it is using a small version of zips resources (zips-mini.json). Please take a look and let me know what you think. On Wed, Jun 13, 2018 at 12:32 AM Christian Beikov < [email protected]> wrote: > Interesting, I tested both environments back then and it seemed to work. > Good thing that we are about to replace the IT tests with using local ES > nodes. This will finally allow us to run these tests regularly. > > Regarding ES2 and ES5, I'd suggest you have separate > ElasticsearchProjectRule for each version. In Elasticsearch2Table and > Elasticsearch5Table you add the rules to the ElasticsearchTableScan > constructor. > > > Mit freundlichen Grüßen, > ------------------------------------------------------------------------ > *Christian Beikov* > Am 13.06.2018 um 06:15 schrieb Andrei Sereda: > > The issue seems to have started since CALCITE-1967 (ES5 support). > > ES2 expects keyword _source in scripted fields while ES5 param._source. > > See ElasticsearchProject > > < > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java#L79 > > > > . > > > > Also ES2 doesn't like when both script_fields and _source are present in > > the query. So if there are some explicit projections all have to be > present > > in script_fields. > > > > Any elegant way to differentiate at runtime ES2 vs ES5 inside > > ElasticsearchProject ? > > > > > > > > On Mon, Jun 11, 2018 at 5:46 PM Christian Beikov < > [email protected]> > > wrote: > > > >> I'm fine with a single PR as long as you split commits up. > >> > >> Andrei Sereda <[email protected]> schrieb am Mo., 11. Juni 2018, 23:24: > >> > >>> Hi Christian, > >>> > >>> I'm working in parallel on CALCITE-2331 evaluation of predicate "(A or > B) > >>> and C" for ES. Can I fix existing bugs as part of that (or separate) > >> commit > >>> ? > >>> > >>> I was planning to do it just not in the same PR. > >>> > >>> On Mon, Jun 11, 2018 at 4:31 PM, Christian Beikov < > >>> [email protected]> wrote: > >>> > >>>> For me that's fine, though it would be great if you could fix those > >> given > >>>> your time permits it. > >>>> > >>>> Andrei Sereda <[email protected]> schrieb am Mo., 11. Juni 2018, > 22:12: > >>>> > >>>>> There are some tests which have been failing for while. Is it OK if I > >>>>> temporary mark them @Ignore(d) ? > >>>>> > >>>>> On Mon, Jun 11, 2018 at 3:48 PM, Christian Beikov < > >>>>> [email protected]> wrote: > >>>>> > >>>>>> If you migrate the existing tests to use the local node in that PR, > >>> I'd > >>>>>> consider merging that PR given that Travis CI reports no test > >>> failures. > >>>>>> > >>>>>> Mit freundlichen Grüßen, > >>>>>> ------------------------------------------------------------ > >>>> ------------ > >>>>>> *Christian Beikov* > >>>>>> Am 11.06.2018 um 14:33 schrieb Andrei Sereda: > >>>>>> > >>>>>>> Do the tests succeed with the local node? > >>>>>>> IT tests haven't been migrated to local node yet. I wanted to do > >> it > >>> as > >>>>>>> separate commit. > >>>>>>> There are new tests which run against embedded instance > >>> successfully. > >>>>> Pls > >>>>>>> see PR 716 for details. > >>>>>>> > >>>>>>> I don't see a reason for running ES tests again on a "real" ES > >>>> instance. > >>>>>>> What does that even mean? > >>>>>>> The only reason, I see, in connecting to external instance is > >>> testing > >>>>>>> compatibility with different versions of the engine. > >>>>>>> > >>>>>>> On Mon, Jun 11, 2018 at 7:16 AM, Michael Mior <[email protected]> > >>>> wrote: > >>>>>>> Good point. Since ES is written in Java and we're running the same > >>>> code > >>>>>>>> testing on a local node, it's probably not necessary to test > >>> against > >>>> a > >>>>>>>> separate ES cluster. > >>>>>>>> -- > >>>>>>>> Michael Mior > >>>>>>>> [email protected] > >>>>>>>> > >>>>>>>> > >>>>>>>> Le lun. 11 juin 2018 à 01:44, Christian Beikov < > >>>>>>>> [email protected] > >>>>>>>> a écrit : > >>>>>>>> > >>>>>>>> I don't see a reason for running ES tests again on a "real" ES > >>>>> instance. > >>>>>>>>> What does that even mean? > >>>>>>>>> > >>>>>>>>> With a local node, we would be testing functionality against > >>> almost > >>>>> the > >>>>>>>>> real thing. The only difference is that a real setup has a > >>> cluster, > >>>>> but > >>>>>>>>> that doesn't change the functionality. So I'm all in on using > >>> just a > >>>>>>>>> local node which is also something I wanted to do for a long > >> time, > >>>> but > >>>>>>>>> you got to it before I did. > >>>>>>>>> > >>>>>>>>> Do the tests succeed with the local node? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Mit freundlichen Grüßen, > >>>>>>>>> ------------------------------------------------------------ > >>>>>>>>> ------------ > >>>>>>>>> *Christian Beikov* > >>>>>>>>> Am 07.06.2018 um 20:47 schrieb Andrei Sereda: > >>>>>>>>> > >>>>>>>>>> Hi Volodymyr, > >>>>>>>>>> > >>>>>>>>>> We're trying to bridge the gap between unit and integration > >> tests > >>>>>>>>> (which > >>>>>>>>> are run infrequently) using fakes > >>>>>>>>>> <https://martinfowler.com/bliki/InMemoryTestDatabase.html> in > >>>>> Calcite. > >>>>>>>>>> Currently the plan is to use fongo (see CALCITE-2345 for mongo > >>>>> adapter) > >>>>>>>>> and > >>>>>>>>> > >>>>>>>>>> local elastic node (see CALCITE-2347 for elastic). > >>>>>>>>>> > >>>>>>>>>> During unit tests code will run against fake instances. For IT > >>>>> profile > >>>>>>>>> they > >>>>>>>>> > >>>>>>>>>> will run against real instances (as before). > >>>>>>>>>> > >>>>>>>>>> Regards, > >>>>>>>>>> Andrei. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Thu, Jun 7, 2018 at 2:32 PM, Julian Hyde <[email protected]> > >>>>> wrote: > >>>>>>>>>> Unfortunately, I don’t know. We don’t run the integration tests > >>> as > >>>>>>>>>> often > >>>>>>>>> as we should. > >>>>>>>>>>> On Jun 7, 2018, at 10:20 AM, Vova Vysotskyi <[email protected] > >>>>>>>>>>> wrote: > >>>>>>>>> Hi all, > >>>>>>>>>>>> I tried to run Elasticsearch integration tests for the > >> current > >>>>>>>>>>> master, > >>>>>>>>> and > >>>>>>>>>>>> many tests failed. > >>>>>>>>>>>> > >>>>>>>>>>>> Is it a known problem, or just an issue of my env? > >>>>>>>>>>>> > >>>>>>>>>>>> I noticed that one failure appears after the change made in > >>>>>>>>>>>> > >>>>>>>>>>> CALCITE-2316. > >>>>>>>>>> Most of the other failures are observed for 1.16 version also. > >>>>>>>>>>>> Kind regards, > >>>>>>>>>>>> Volodymyr Vysotskyi > >>>>>>>>>>>> > >
