Thanks for the proposal Timo! I've done a pass and added some comments (mostly asking for clarification, details). Overall, this is going into a very good direction. I think the tables which are stored in different systems and using a format definition to define other formats require some more discussions. However, these are also not the features that we would start with.
>From a compatibility point of view, an important question to answer would be whether we can drop the support for field mapping, i.e., do we have users who take advantage of mapping format fields to fields with a different name in the schema. Besides that, all existing functionality is preserved although the syntax changes a bit. Best, Fabian Am Mo., 1. Okt. 2018 um 10:53 Uhr schrieb Timo Walther <twal...@apache.org>: > Hi everyone, > > as some of you might have noticed, in the last two releases we aimed to > unify SQL connectors and make them more modular. The first connectors > and formats have been implemented and are usable via the SQL Client and > Java/Scala/SQL APIs. > > However, after writing more connectors/example programs and talking to > users, there are still a couple of improvements that should be applied > to unified SQL connector API. > > I wrote a design document [1] that discusses limitations that I have > observed and consideres feedback that I have collected over the last > months. I don't know whether we will implement all of these > improvements, but it would be great to get feedback for a satisfactory > API and for future priorization. > > The general goal should be to connect to external systems as convenient > and type-safe as possible. Any feedback is highly appreciated. > > Thanks, > > Timo > > [1] > > https://docs.google.com/document/d/1Yaxp1UJUFW-peGLt8EIidwKIZEWrrA-pznWLuvaH39Y/edit?usp=sharing > >
