I agree there should be a configuration option. How about the following
approach.
Expose both variables ${index} and ${type} in configuration (JSON) and user
will use them to generate table name in calcite schema.
Example
"table_name": "${type}" // current
"table_name": "${index}" // new (default?)
"table_name": "${index}_${type}" // most generic. supports multiple types
per index
On Fri, Jun 29, 2018 at 9:26 AM Michael Mior <mm...@apache.org> wrote:
> I think it sounds like you and Andrei are in a good position to tackle this
> one so I'm happy to have you both work on whatever solution you think is
> best.
>
> --
> Michael Mior
> mm...@apache.org
>
>
>
> Le ven. 29 juin 2018 à 04:19, Christian Beikov <christian.bei...@gmail.com
> >
> a écrit :
>
> > IMO the best solution would be to make it configurable by introducing a
> > "table_mapping" config with values
> >
> > * type - every type in the known indices is mapped as table
> > * index - every known index is mapped as table
> >
> > We'd probably also need a "type_field" configuration for defining which
> > field to use for the type determination as one of the possible future
> > ways to do things is to introduce a custom field:
> >
> >
> https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html#_custom_type_field_2
> >
> > We already detect the ES version, so we can set a smart default for this
> > setting. Let's make the index config param optional.
> >
> > * When no index is given, we discover indexes, the default for
> > "table_mapping" then is "index"
> > * When index is given, the we only discover types according to the
> > "type_field" configuration and the default for "table_mapping" is
> > "type"
> >
> > This would also allow to discover indexes but still use "type" as
> > "table_mapping".
> >
> > What do you think?
> >
> > Mit freundlichen Grüßen,
> > ------------------------------------------------------------------------
> > *Christian Beikov*
> > Am 29.06.2018 um 02:41 schrieb Andrei Sereda:
> > > Yes. There is an API to list all indexes / types in elastic. They can
> be
> > > automatically imported into a schema.
> > >
> > > What needs to be agreed upon is how to expose those elements in calcite
> > > schema (naming / behaviour).
> > >
> > > 1) Many (most?) of setups are single type per index. Natural way to
> name
> > > would be "elastic.$index" (elastic being schema name). Multiple
> indexes
> > > would be under same schema "elastic.index1" "elastic.index2" etc.
> > >
> > > 2) What if index has several types should they exported as calcite
> > tables:
> > > "elastic.$index_type1" "elastic.$index_type2" ? Or (current behaviour)
> > as
> > > "elastic.type1" and "elastic.type2". Or as subschema
> > > "elastic.$index.type1" ?
> > >
> > > Now what if one has combination of (1) and (2) ?
> > > Setup (2) is already deprecated (and will be unsupported in next
> version)
> > >
> > >
> > > On Thu, Jun 28, 2018 at 7:31 PM Christian Beikov <
> > christian.bei...@gmail.com>
> > > wrote:
> > >
> > >> Is there an API to discover indexes? If there is, I'd suggest we
> allow a
> > >> config option that to make the adapter discover the possible indexes.
> > >> We'd still have to adapt the code a bit, but internally, the schema
> > >> could just keep a cache of type name to index name map and be able to
> > >> support both scenarios.
> > >>
> > >>
> > >> Mit freundlichen Grüßen,
> > >>
> ------------------------------------------------------------------------
> > >> *Christian Beikov*
> > >> Am 29.06.2018 um 00:12 schrieb Andrei Sereda:
> > >>>> 1) What's the time horizon for the current adapter no longer working
> > >> with these
> > >>> changes to ES ?
> > >>> Current adapter will be working for a while with existing setup. The
> > >>> problem is nomenclature and ease of use.
> > >>>
> > >>> Their new SQL concepts mapping
> > >>> <
> > >>
> >
> https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html
> > >>> drops
> > >>> the notion of ES type (which before was equivalent of RDBMS table)
> and
> > >> uses
> > >>> ES index as new table equivalent (before ES index was equal to
> > database).
> > >>> Most users use elastic this way (one type , one index) index ==
> table.
> > >>>
> > >>> Currently calcite requires schema per index. In RDBMS parlance
> database
> > >> per
> > >>> table (I'd like to change that).
> > >>>
> > >>>> 2) Any guess how complicated it would be to maintain code paths for
> > both
> > >>>> behaviours? I know this is probably really challenging to estimate,
> > but
> > >> I
> > >>>> really have no idea of the scope of these changes. Would it mean two
> > >>>> different ES adapters?
> > >>> One can have just a separate calcite schema implementations (same
> > >> adapter /
> > >>> module) :
> > >>> 1) LegacySchema (old). Schema can have only one index (but multiple
> > >>> types). Type == table in this case.
> > >>> 2) NewSchema (new). Single schema can have multiple indexes (type is
> > >>> dropped). Index == table in this case
> > >>>
> > >>>> 3) Do we really need compatibility with the current version of the
> > >>> adapter?
> > >>>> IMO this depends on what versions of ES we would lose support for
> and
> > >> how
> > >>>> complex it would be for users of the current ES adapter to make
> > updates
> > >>> for
> > >>>> any Calcite API changes.
> > >>> The issue is not in adapter but how calcite schema exposes tables.
> > >> Should
> > >>> it expose index as individual table (new), or ES type (old) ?
> > >>>
> > >>> Andrei.
> > >>>
> > >>> On Thu, Jun 28, 2018 at 5:23 PM Michael Mior <mm...@apache.org>
> wrote:
> > >>>
> > >>>> Unfortunately I know very little about ES so I'm not in a great
> > >> position to
> > >>>> asses the impact of these changes. I will say that that legacy
> > >>>> compatibility is great, but maintaining two sets of logic is always
> a
> > >>>> challenge. A few follow up questions:
> > >>>>
> > >>>> 1) What's the time horizon for the current adapter no longer working
> > >> with
> > >>>> these changes to ES?
> > >>>>
> > >>>> 2) Any guess how complicated it would be to maintain code paths for
> > both
> > >>>> behaviours? I know this is probably really challenging to estimate,
> > but
> > >> I
> > >>>> really have no idea of the scope of these changes. Would it mean two
> > >>>> different ES adapters?
> > >>>>
> > >>>> 3) Do we really need compatibility with the current version of the
> > >> adapter?
> > >>>> IMO this depends on what versions of ES we would lose support for
> and
> > >> how
> > >>>> complex it would be for users of the current ES adapter to make
> > updates
> > >> for
> > >>>> any Calcite API changes.
> > >>>>
> > >>>> Thanks for your continued work on the ES adapter Andrei!
> > >>>>
> > >>>> --
> > >>>> Michael Mior
> > >>>> mm...@apache.org
> > >>>>
> > >>>>
> > >>>>
> > >>>> Le jeu. 28 juin 2018 à 12:57, Andrei Sereda <and...@sereda.cc> a
> > écrit
> > >> :
> > >>>>> Hello,
> > >>>>>
> > >>>>> Elastic announced
> > >>>>> <
> > >>>>>
> > >>
> >
> https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html
> > >>>>> that they will be deprecating mapping types in ES6 and indexes will
> > be
> > >>>>> single-typed only.
> > >>>>>
> > >>>>> Historical analogy <https://www.elastic.co/blog/index-vs-type>
> > between
> > >>>>> RDBMS and elastic was that index is equivalent to a database and
> type
> > >>>>> corresponds to table in that database. In a couple of releases
> > (ES6-8)
> > >>>> this
> > >>>>> shall not longer be true.
> > >>>>>
> > >>>>> Recent SQL addition
> > >>>>> <https://www.elastic.co/blog/elasticsearch-6-3-0-released> to
> > elastic
> > >>>>> confirms
> > >>>>> this trend
> > >>>>> <
> > >>>>>
> > >>
> >
> https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html
> > >>>>>> .
> > >>>>> Index is equivalent to a table and there are no more ES types.
> > >>>>>
> > >>>>> I would like to propose to include this logic in Calcite ES
> adapter.
> > >> IE,
> > >>>>> expose each ES single-typed index as a separate table inside
> calcite
> > >>>>> schema. This is in contrast to current integration where schema
> can
> > >> only
> > >>>>> have a single index. Current approach forces you to create multiple
> > >>>> schemas
> > >>>>> to query single-typed indexes (on the same ES cluster).
> > >>>>>
> > >>>>> Legacy compatibility can always be controlled with configuration
> > >>>>> parameters.
> > >>>>>
> > >>>>> Do you agree with such changes ? If yes, would you consider a PR ?
> > >>>>>
> > >>>>> Regards,
> > >>>>> Andrei.
> > >>>>>
> > >>
> >
> >
>
