Yes. There is an API to list all indexes / types in elastic. They can be automatically imported into a schema.
What needs to be agreed upon is how to expose those elements in calcite schema (naming / behaviour). 1) Many (most?) of setups are single type per index. Natural way to name would be "elastic.$index" (elastic being schema name). Multiple indexes would be under same schema "elastic.index1" "elastic.index2" etc. 2) What if index has several types should they exported as calcite tables: "elastic.$index_type1" "elastic.$index_type2" ? Or (current behaviour) as "elastic.type1" and "elastic.type2". Or as subschema "elastic.$index.type1" ? Now what if one has combination of (1) and (2) ? Setup (2) is already deprecated (and will be unsupported in next version) On Thu, Jun 28, 2018 at 7:31 PM Christian Beikov <christian.bei...@gmail.com> wrote: > Is there an API to discover indexes? If there is, I'd suggest we allow a > config option that to make the adapter discover the possible indexes. > We'd still have to adapt the code a bit, but internally, the schema > could just keep a cache of type name to index name map and be able to > support both scenarios. > > > Mit freundlichen Grüßen, > ------------------------------------------------------------------------ > *Christian Beikov* > Am 29.06.2018 um 00:12 schrieb Andrei Sereda: > >> 1) What's the time horizon for the current adapter no longer working > with these > > changes to ES ? > > Current adapter will be working for a while with existing setup. The > > problem is nomenclature and ease of use. > > > > Their new SQL concepts mapping > > < > https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html > > > > drops > > the notion of ES type (which before was equivalent of RDBMS table) and > uses > > ES index as new table equivalent (before ES index was equal to database). > > Most users use elastic this way (one type , one index) index == table. > > > > Currently calcite requires schema per index. In RDBMS parlance database > per > > table (I'd like to change that). > > > >> 2) Any guess how complicated it would be to maintain code paths for both > >> behaviours? I know this is probably really challenging to estimate, but > I > >> really have no idea of the scope of these changes. Would it mean two > >> different ES adapters? > > One can have just a separate calcite schema implementations (same > adapter / > > module) : > > 1) LegacySchema (old). Schema can have only one index (but multiple > > types). Type == table in this case. > > 2) NewSchema (new). Single schema can have multiple indexes (type is > > dropped). Index == table in this case > > > >> 3) Do we really need compatibility with the current version of the > > adapter? > >> IMO this depends on what versions of ES we would lose support for and > how > >> complex it would be for users of the current ES adapter to make updates > > for > >> any Calcite API changes. > > The issue is not in adapter but how calcite schema exposes tables. > Should > > it expose index as individual table (new), or ES type (old) ? > > > > Andrei. > > > > On Thu, Jun 28, 2018 at 5:23 PM Michael Mior <mm...@apache.org> wrote: > > > >> Unfortunately I know very little about ES so I'm not in a great > position to > >> asses the impact of these changes. I will say that that legacy > >> compatibility is great, but maintaining two sets of logic is always a > >> challenge. A few follow up questions: > >> > >> 1) What's the time horizon for the current adapter no longer working > with > >> these changes to ES? > >> > >> 2) Any guess how complicated it would be to maintain code paths for both > >> behaviours? I know this is probably really challenging to estimate, but > I > >> really have no idea of the scope of these changes. Would it mean two > >> different ES adapters? > >> > >> 3) Do we really need compatibility with the current version of the > adapter? > >> IMO this depends on what versions of ES we would lose support for and > how > >> complex it would be for users of the current ES adapter to make updates > for > >> any Calcite API changes. > >> > >> Thanks for your continued work on the ES adapter Andrei! > >> > >> -- > >> Michael Mior > >> mm...@apache.org > >> > >> > >> > >> Le jeu. 28 juin 2018 à 12:57, Andrei Sereda <and...@sereda.cc> a écrit > : > >> > >>> Hello, > >>> > >>> Elastic announced > >>> < > >>> > >> > https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html > >>> that they will be deprecating mapping types in ES6 and indexes will be > >>> single-typed only. > >>> > >>> Historical analogy <https://www.elastic.co/blog/index-vs-type> between > >>> RDBMS and elastic was that index is equivalent to a database and type > >>> corresponds to table in that database. In a couple of releases (ES6-8) > >> this > >>> shall not longer be true. > >>> > >>> Recent SQL addition > >>> <https://www.elastic.co/blog/elasticsearch-6-3-0-released> to elastic > >>> confirms > >>> this trend > >>> < > >>> > >> > https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html > >>>> . > >>> Index is equivalent to a table and there are no more ES types. > >>> > >>> I would like to propose to include this logic in Calcite ES adapter. > IE, > >>> expose each ES single-typed index as a separate table inside calcite > >>> schema. This is in contrast to current integration where schema can > only > >>> have a single index. Current approach forces you to create multiple > >> schemas > >>> to query single-typed indexes (on the same ES cluster). > >>> > >>> Legacy compatibility can always be controlled with configuration > >>> parameters. > >>> > >>> Do you agree with such changes ? If yes, would you consider a PR ? > >>> > >>> Regards, > >>> Andrei. > >>> > >
