Maybe there could be a separator char as one of the adapter’s parameters.
People should choose a value, say ‘$’ or ‘#’, that is legal in an unquoted SQL
identifier but does not occur in any of their index or type names.
If not specified, the adapter would end up in a simple mode, say looking for
indexes first, then looking for types, and people would need to make sure
indexes and types have distinct names. After the transition to single-type
indexes, people could stop using the parameter.
Julian
> On Jun 29, 2018, at 4:43 PM, Andrei Sereda <and...@sereda.cc> wrote:
>
> That's a valid point. Then user would define a different pattern like
> "i$index_t$type" for his cluster.
>
> I think we should first answer wherever such scenarios should be supported
> by calcite (given that they're already deprecated by the vendor). If yes,
> what should be collision strategy ? User defined pattern like above or
> failure or auto generated name ?
>
> On Fri, Jun 29, 2018, 19:14 Julian Hyde <jh...@apache.org> wrote:
>
>>> In elastic (index/type) pair is guaranteed to be unique therefore
>>> "${index}_${type}" will be also unique (as string). This is only
>> necessary
>>> when we have several types per index. Valid question is wherever user
>>> should be allowed such flexibility.
>>
>> Uniqueness is not my concern.
>>
>> Suppose there is an index called "x_y" with a type called "z", and
>> another index called "x" with a type called "y_z". If I write "x_y_z"
>> it's not clear how it should be broken into index/type.
>>
>>
>> On Fri, Jun 29, 2018 at 3:15 PM, Andrei Sereda <and...@sereda.cc> wrote:
>>>> Can you show how those examples affect SQL against the ES adapter and/or
>>> how they affect JSON models?
>>>
>>> The discussion is how to properly bridge (index/type) concept from ES
>> into
>>> relational world. Proposal to use placeholders ($index / $type) affects
>>> only how table is named in calcite. They're not used as SQL literals. IE
>> it
>>> affects only configuration phase of the schema.
>>> Pretty much we're doing string/replace to derive table name from
>>> ($index/$type).
>>>
>>>> You seem to be using '_' as a separator character. Are we sure that
>>>> people will never use it in index or type name? Separator characters
>>>> often cause problems.
>>> In elastic (index/type) pair is guaranteed to be unique therefore
>>> "${index}_${type}" will be also unique (as string). This is only
>> necessary
>>> when we have several types per index. Valid question is wherever user
>>> should be allowed such flexibility.
>>>
>>>
>>>
>>> On Fri, Jun 29, 2018 at 2:19 PM Julian Hyde <jh...@apache.org> wrote:
>>>
>>>> Andrei,
>>>>
>>>> I'm not an ES user so I don't fully understand this issue, but my two
>>>> cents anyway...
>>>>
>>>> Can you show how those examples affect SQL against the ES adapter
>>>> and/or how they affect JSON models?
>>>>
>>>> You seem to be using '_' as a separator character. Are we sure that
>>>> people will never use it in index or type name? Separator characters
>>>> often cause problems.
>>>>
>>>> Julian
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Jun 29, 2018 at 10:58 AM, Andrei Sereda <and...@sereda.cc>
>> wrote:
>>>>> I agree there should be a configuration option. How about the
>> following
>>>>> approach.
>>>>>
>>>>> Expose both variables ${index} and ${type} in configuration (JSON) and
>>>> user
>>>>> will use them to generate table name in calcite schema.
>>>>>
>>>>> Example
>>>>> "table_name": "${type}" // current
>>>>> "table_name": "${index}" // new (default?)
>>>>> "table_name": "${index}_${type}" // most generic. supports multiple
>> types
>>>>> per index
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Jun 29, 2018 at 9:26 AM Michael Mior <mm...@apache.org>
>> wrote:
>>>>>
>>>>>> I think it sounds like you and Andrei are in a good position to
>> tackle
>>>> this
>>>>>> one so I'm happy to have you both work on whatever solution you
>> think is
>>>>>> best.
>>>>>>
>>>>>> --
>>>>>> Michael Mior
>>>>>> mm...@apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> Le ven. 29 juin 2018 à 04:19, Christian Beikov <
>>>> christian.bei...@gmail.com
>>>>>>>
>>>>>> a écrit :
>>>>>>
>>>>>>> IMO the best solution would be to make it configurable by
>> introducing
>>>> a
>>>>>>> "table_mapping" config with values
>>>>>>>
>>>>>>> * type - every type in the known indices is mapped as table
>>>>>>> * index - every known index is mapped as table
>>>>>>>
>>>>>>> We'd probably also need a "type_field" configuration for defining
>>>> which
>>>>>>> field to use for the type determination as one of the possible
>> future
>>>>>>> ways to do things is to introduce a custom field:
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html#_custom_type_field_2
>>>>>>>
>>>>>>> We already detect the ES version, so we can set a smart default for
>>>> this
>>>>>>> setting. Let's make the index config param optional.
>>>>>>>
>>>>>>> * When no index is given, we discover indexes, the default for
>>>>>>> "table_mapping" then is "index"
>>>>>>> * When index is given, the we only discover types according to
>> the
>>>>>>> "type_field" configuration and the default for "table_mapping"
>> is
>>>>>>> "type"
>>>>>>>
>>>>>>> This would also allow to discover indexes but still use "type" as
>>>>>>> "table_mapping".
>>>>>>>
>>>>>>> What do you think?
>>>>>>>
>>>>>>> Mit freundlichen Grüßen,
>>>>>>>
>>>> ------------------------------------------------------------------------
>>>>>>> *Christian Beikov*
>>>>>>> Am 29.06.2018 um 02:41 schrieb Andrei Sereda:
>>>>>>>> Yes. There is an API to list all indexes / types in elastic. They
>>>> can
>>>>>> be
>>>>>>>> automatically imported into a schema.
>>>>>>>>
>>>>>>>> What needs to be agreed upon is how to expose those elements in
>>>> calcite
>>>>>>>> schema (naming / behaviour).
>>>>>>>>
>>>>>>>> 1) Many (most?) of setups are single type per index. Natural way
>> to
>>>>>> name
>>>>>>>> would be "elastic.$index" (elastic being schema name). Multiple
>>>>>> indexes
>>>>>>>> would be under same schema "elastic.index1" "elastic.index2" etc.
>>>>>>>>
>>>>>>>> 2) What if index has several types should they exported as
>> calcite
>>>>>>> tables:
>>>>>>>> "elastic.$index_type1" "elastic.$index_type2" ? Or (current
>>>> behaviour)
>>>>>>> as
>>>>>>>> "elastic.type1" and "elastic.type2". Or as subschema
>>>>>>>> "elastic.$index.type1" ?
>>>>>>>>
>>>>>>>> Now what if one has combination of (1) and (2) ?
>>>>>>>> Setup (2) is already deprecated (and will be unsupported in next
>>>>>> version)
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jun 28, 2018 at 7:31 PM Christian Beikov <
>>>>>>> christian.bei...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Is there an API to discover indexes? If there is, I'd suggest we
>>>>>> allow a
>>>>>>>>> config option that to make the adapter discover the possible
>>>> indexes.
>>>>>>>>> We'd still have to adapt the code a bit, but internally, the
>> schema
>>>>>>>>> could just keep a cache of type name to index name map and be
>> able
>>>> to
>>>>>>>>> support both scenarios.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Mit freundlichen Grüßen,
>>>>>>>>>
>>>>>>
>> ------------------------------------------------------------------------
>>>>>>>>> *Christian Beikov*
>>>>>>>>> Am 29.06.2018 um 00:12 schrieb Andrei Sereda:
>>>>>>>>>>> 1) What's the time horizon for the current adapter no longer
>>>> working
>>>>>>>>> with these
>>>>>>>>>> changes to ES ?
>>>>>>>>>> Current adapter will be working for a while with existing
>> setup.
>>>> The
>>>>>>>>>> problem is nomenclature and ease of use.
>>>>>>>>>>
>>>>>>>>>> Their new SQL concepts mapping
>>>>>>>>>> <
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html
>>>>>>>>>> drops
>>>>>>>>>> the notion of ES type (which before was equivalent of RDBMS
>> table)
>>>>>> and
>>>>>>>>> uses
>>>>>>>>>> ES index as new table equivalent (before ES index was equal to
>>>>>>> database).
>>>>>>>>>> Most users use elastic this way (one type , one index) index ==
>>>>>> table.
>>>>>>>>>>
>>>>>>>>>> Currently calcite requires schema per index. In RDBMS parlance
>>>>>> database
>>>>>>>>> per
>>>>>>>>>> table (I'd like to change that).
>>>>>>>>>>
>>>>>>>>>>> 2) Any guess how complicated it would be to maintain code
>> paths
>>>> for
>>>>>>> both
>>>>>>>>>>> behaviours? I know this is probably really challenging to
>>>> estimate,
>>>>>>> but
>>>>>>>>> I
>>>>>>>>>>> really have no idea of the scope of these changes. Would it
>> mean
>>>> two
>>>>>>>>>>> different ES adapters?
>>>>>>>>>> One can have just a separate calcite schema implementations
>> (same
>>>>>>>>> adapter /
>>>>>>>>>> module) :
>>>>>>>>>> 1) LegacySchema (old). Schema can have only one index (but
>>>> multiple
>>>>>>>>>> types). Type == table in this case.
>>>>>>>>>> 2) NewSchema (new). Single schema can have multiple indexes
>>>> (type is
>>>>>>>>>> dropped). Index == table in this case
>>>>>>>>>>
>>>>>>>>>>> 3) Do we really need compatibility with the current version of
>>>> the
>>>>>>>>>> adapter?
>>>>>>>>>>> IMO this depends on what versions of ES we would lose support
>> for
>>>>>> and
>>>>>>>>> how
>>>>>>>>>>> complex it would be for users of the current ES adapter to
>> make
>>>>>>> updates
>>>>>>>>>> for
>>>>>>>>>>> any Calcite API changes.
>>>>>>>>>> The issue is not in adapter but how calcite schema exposes
>> tables.
>>>>>>>>> Should
>>>>>>>>>> it expose index as individual table (new), or ES type (old) ?
>>>>>>>>>>
>>>>>>>>>> Andrei.
>>>>>>>>>>
>>>>>>>>>> On Thu, Jun 28, 2018 at 5:23 PM Michael Mior <mm...@apache.org
>>>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Unfortunately I know very little about ES so I'm not in a
>> great
>>>>>>>>> position to
>>>>>>>>>>> asses the impact of these changes. I will say that that legacy
>>>>>>>>>>> compatibility is great, but maintaining two sets of logic is
>>>> always
>>>>>> a
>>>>>>>>>>> challenge. A few follow up questions:
>>>>>>>>>>>
>>>>>>>>>>> 1) What's the time horizon for the current adapter no longer
>>>> working
>>>>>>>>> with
>>>>>>>>>>> these changes to ES?
>>>>>>>>>>>
>>>>>>>>>>> 2) Any guess how complicated it would be to maintain code
>> paths
>>>> for
>>>>>>> both
>>>>>>>>>>> behaviours? I know this is probably really challenging to
>>>> estimate,
>>>>>>> but
>>>>>>>>> I
>>>>>>>>>>> really have no idea of the scope of these changes. Would it
>> mean
>>>> two
>>>>>>>>>>> different ES adapters?
>>>>>>>>>>>
>>>>>>>>>>> 3) Do we really need compatibility with the current version of
>>>> the
>>>>>>>>> adapter?
>>>>>>>>>>> IMO this depends on what versions of ES we would lose support
>> for
>>>>>> and
>>>>>>>>> how
>>>>>>>>>>> complex it would be for users of the current ES adapter to
>> make
>>>>>>> updates
>>>>>>>>> for
>>>>>>>>>>> any Calcite API changes.
>>>>>>>>>>>
>>>>>>>>>>> Thanks for your continued work on the ES adapter Andrei!
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Michael Mior
>>>>>>>>>>> mm...@apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Le jeu. 28 juin 2018 à 12:57, Andrei Sereda <and...@sereda.cc>
>> a
>>>>>>> écrit
>>>>>>>>> :
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> Elastic announced
>>>>>>>>>>>> <
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html
>>>>>>>>>>>> that they will be deprecating mapping types in ES6 and
>> indexes
>>>> will
>>>>>>> be
>>>>>>>>>>>> single-typed only.
>>>>>>>>>>>>
>>>>>>>>>>>> Historical analogy <
>> https://www.elastic.co/blog/index-vs-type>
>>>>>>> between
>>>>>>>>>>>> RDBMS and elastic was that index is equivalent to a database
>> and
>>>>>> type
>>>>>>>>>>>> corresponds to table in that database. In a couple of
>> releases
>>>>>>> (ES6-8)
>>>>>>>>>>> this
>>>>>>>>>>>> shall not longer be true.
>>>>>>>>>>>>
>>>>>>>>>>>> Recent SQL addition
>>>>>>>>>>>> <https://www.elastic.co/blog/elasticsearch-6-3-0-released>
>> to
>>>>>>> elastic
>>>>>>>>>>>> confirms
>>>>>>>>>>>> this trend
>>>>>>>>>>>> <
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html
>>>>>>>>>>>>> .
>>>>>>>>>>>> Index is equivalent to a table and there are no more ES
>> types.
>>>>>>>>>>>>
>>>>>>>>>>>> I would like to propose to include this logic in Calcite ES
>>>>>> adapter.
>>>>>>>>> IE,
>>>>>>>>>>>> expose each ES single-typed index as a separate table inside
>>>>>> calcite
>>>>>>>>>>>> schema. This is in contrast to current integration where
>> schema
>>>>>> can
>>>>>>>>> only
>>>>>>>>>>>> have a single index. Current approach forces you to create
>>>> multiple
>>>>>>>>>>> schemas
>>>>>>>>>>>> to query single-typed indexes (on the same ES cluster).
>>>>>>>>>>>>
>>>>>>>>>>>> Legacy compatibility can always be controlled with
>> configuration
>>>>>>>>>>>> parameters.
>>>>>>>>>>>>
>>>>>>>>>>>> Do you agree with such changes ? If yes, would you consider a
>>>> PR ?
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Andrei.
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>
