Thanks Timo for merging a couple of the PRs. Are you also able to
review the others that I mentioned? Xuefu I would like to incorporate
your feedback too.
Check out this short demonstration of using a catalog in SQL Client:
https://asciinema.org/a/C8xuAjmZSxCuApgFgZQyeIHuo
Thanks again!
On Thu, Jan 3, 2019 at 9:37 AM Eron Wright <eronwri...@gmail.com
<mailto:eronwri...@gmail.com>> wrote:
Would a couple folks raise their hand to make a review pass thru
the 6 PRs listed above? It is a lovely stack of PRs that is 'all
green' at the moment. I would be happy to open follow-on PRs to
rapidly align with other efforts.
Note that the code is agnostic to the details of the
ExternalCatalog interface; the code would not be obsolete if/when
the catalog interface is enhanced as per the design doc.
On Wed, Jan 2, 2019 at 1:35 PM Eron Wright <eronwri...@gmail.com
<mailto:eronwri...@gmail.com>> wrote:
I propose that the community review and merge the PRs that I
posted, and then evolve the design thru 1.8 and beyond. I
think having a basic infrastructure in place now will
accelerate the effort, do you agree?
Thanks again!
On Wed, Jan 2, 2019 at 11:20 AM Zhang, Xuefu
<xuef...@alibaba-inc.com <mailto:xuef...@alibaba-inc.com>> wrote:
Hi Eron,
Happy New Year!
Thank you very much for your contribution, especially
during the holidays. Wile I'm encouraged by your work, I'd
also like to share my thoughts on how to move forward.
First, please note that the design discussion is still
finalizing, and we expect some moderate changes,
especially around TableFactories. Another pending change
is our decision to shy away from scala, which our work
will be impacted by.
Secondly, while your work seemed about plugging in
catalogs definitions to the execution environment, which
is less impacted by TableFactory change, I did notice some
duplication of your work and ours. This is no big deal,
but going forward, we should probable have a better
communication on the work assignment so as to avoid any
possible duplication of work. On the other hand, I think
some of your work is interesting and valuable for
inclusion once we finalize the overall design.
Thus, please continue your research and experiment and let
us know when you start working on anything so we can
better coordinate.
Thanks again for your interest and contributions.
Thanks,
Xuefu
------------------------------------------------------------------
From:Eron Wright <eronwri...@gmail.com
<mailto:eronwri...@gmail.com>>
Sent At:2019 Jan. 1 (Tue.) 18:39
To:dev <dev@flink.apache.org
<mailto:dev@flink.apache.org>>; Xuefu
<xuef...@alibaba-inc.com <mailto:xuef...@alibaba-inc.com>>
Cc:Xiaowei Jiang <xiaow...@gmail.com
<mailto:xiaow...@gmail.com>>; twalthr
<twal...@apache.org <mailto:twal...@apache.org>>;
piotr <pi...@data-artisans.com
<mailto:pi...@data-artisans.com>>; Fabian Hueske
<fhue...@gmail.com <mailto:fhue...@gmail.com>>;
suez1224 <suez1...@gmail.com
<mailto:suez1...@gmail.com>>; Bowen Li
<bowenl...@gmail.com <mailto:bowenl...@gmail.com>>
Subject:Re: [DISCUSS] Integrate Flink SQL well with
Hive ecosystem
Hi folks, there's clearly some incremental steps to be
taken to introduce catalog support to SQL Client,
complementary to what is proposed in the Flink-Hive
Metastore design doc. I was quietly working on this
over the holidays. I posted some new sub-tasks, PRs,
and sample code to FLINK-10744.
What inspired me to get involved is that the catalog
interface seems like a great way to encapsulate a
'library' of Flink tables and functions. For example,
the NYC Taxi dataset (TaxiRides, TaxiFares, various
UDFs) may be nicely encapsulated as a catalog
(TaxiData). Such a library should be fully consumable
in SQL Client.
I implemented the above. Some highlights:
1. A fully-worked example of using the Taxi dataset in
SQL Client via an environment file.
- an ASCII video showing the SQL Client in action:
https://asciinema.org/a/C8xuAjmZSxCuApgFgZQyeIHuo
- the corresponding environment file (will be even
more concise once 'FLINK-10696 Catalog UDFs' is merged):
_https://github.com/EronWright/flink-training-exercises/blob/3be008d64be975ced0f1a7e3901a8c5353f72a7e/src/main/dist/conf/sql-client-defaults.yaml_
- the typed API for standalone table applications:
_https://github.com/EronWright/flink-training-exercises/blob/3be008d64be975ced0f1a7e3901a8c5353f72a7e/src/main/java/com/dataartisans/flinktraining/examples/table_java/examples/ViaCatalog.java#L50_
2. Implementation of the core catalog descriptor and
factory. I realize that some renames may later occur
as per the design doc, and would be happy to do that
as a follow-up.
https://github.com/apache/flink/pull/7390
3. Implementation of a connect-style API on
TableEnvironment to use catalog descriptor.
https://github.com/apache/flink/pull/7392
4. Integration into SQL-Client's environment file:
https://github.com/apache/flink/pull/7393
I realize that the overall Hive integration is still
evolving, but I believe that these PRs are a good
stepping stone. Here's the list (in bottom-up order):
- https://github.com/apache/flink/pull/7386
- https://github.com/apache/flink/pull/7388
- https://github.com/apache/flink/pull/7389
- https://github.com/apache/flink/pull/7390
- https://github.com/apache/flink/pull/7392
- https://github.com/apache/flink/pull/7393
Thanks and enjoy 2019!
Eron W
On Sun, Nov 18, 2018 at 3:04 PM Zhang, Xuefu
<xuef...@alibaba-inc.com
<mailto:xuef...@alibaba-inc.com>> wrote:
Hi Xiaowei,
Thanks for bringing up the question. In the current
design, the properties for meta objects are meant to
cover anything that's specific to a particular catalog
and agnostic to Flink. Anything that is common (such
as schema for tables, query text for views, and udf
classname) are abstracted as members of the respective
classes. However, this is still in discussion, and
Timo and I will go over this and provide an update.
Please note that UDF is a little more involved than
what the current design doc shows. I'm still refining
this part.
Thanks,
Xuefu
------------------------------------------------------------------
Sender:Xiaowei Jiang <xiaow...@gmail.com
<mailto:xiaow...@gmail.com>>
Sent at:2018 Nov 18 (Sun) 15:17
Recipient:dev <dev@flink.apache.org
<mailto:dev@flink.apache.org>>
Cc:Xuefu <xuef...@alibaba-inc.com
<mailto:xuef...@alibaba-inc.com>>; twalthr
<twal...@apache.org <mailto:twal...@apache.org>>;
piotr <pi...@data-artisans.com
<mailto:pi...@data-artisans.com>>; Fabian Hueske
<fhue...@gmail.com <mailto:fhue...@gmail.com>>;
suez1224 <suez1...@gmail.com <mailto:suez1...@gmail.com>>
Subject:Re: [DISCUSS] Integrate Flink SQL well with
Hive ecosystem
Thanks Xuefu for the detailed design doc! One question
on the properties associated with the catalog objects.
Are we going to leave them completely free form or we
are going to set some standard for that? I think that
the answer may depend on if we want to explore catalog
specific optimization opportunities. In any case, I
think that it might be helpful for standardize as much
as possible into strongly typed classes and use leave
these properties for catalog specific things. But I
think that we can do it in steps.
Xiaowei
On Fri, Nov 16, 2018 at 4:00 AM Bowen Li
<bowenl...@gmail.com <mailto:bowenl...@gmail.com>> wrote:
Thanks for keeping on improving the overall design,
Xuefu! It looks quite
good to me now.
Would be nice that cc-ed Flink committers can help to
review and confirm!
One minor suggestion: Since the last section of
design doc already touches
some new sql statements, shall we add another section
in our doc and
formalize the new sql statements in SQL Client and
TableEnvironment that
are gonna come along naturally with our design? Here
are some that the
design doc mentioned and some that I came up with:
To be added:
- USE <catalog> - set default catalog
- USE <catalog.schema> - set default schema
- SHOW CATALOGS - show all registered catalogs
- SHOW SCHEMAS [FROM catalog] - list schemas in
the current default
catalog or the specified catalog
- DESCRIBE VIEW view - show the view's definition
in CatalogView
- SHOW VIEWS [FROM schema/catalog.schema] - show
views from current or a
specified schema.
(DDLs that can be addressed by either our design
or Shuyi's DDL design)
- CREATE/DROP/ALTER SCHEMA schema
- CREATE/DROP/ALTER CATALOG catalog
To be modified:
- SHOW TABLES [FROM schema/catalog.schema] - show
tables from current or
a specified schema. Add 'from schema' to existing
'SHOW TABLES' statement
- SHOW FUNCTIONS [FROM schema/catalog.schema] -
show functions from
current or a specified schema. Add 'from schema'
to existing 'SHOW TABLES'
statement'
Thanks, Bowen
On Wed, Nov 14, 2018 at 10:39 PM Zhang, Xuefu
<xuef...@alibaba-inc.com <mailto:xuef...@alibaba-inc.com>>
wrote:
> Thanks, Bowen, for catching the error. I have
granted comment permission
> with the link.
>
> I also updated the doc with the latest class
definitions. Everyone is
> encouraged to review and comment.
>
> Thanks,
> Xuefu
>
>
------------------------------------------------------------------
> Sender:Bowen Li <bowenl...@gmail.com
<mailto:bowenl...@gmail.com>>
> Sent at:2018 Nov 14 (Wed) 06:44
> Recipient:Xuefu <xuef...@alibaba-inc.com
<mailto:xuef...@alibaba-inc.com>>
> Cc:piotr <pi...@data-artisans.com
<mailto:pi...@data-artisans.com>>; dev
<dev@flink.apache.org <mailto:dev@flink.apache.org>>;
Shuyi
> Chen <suez1...@gmail.com <mailto:suez1...@gmail.com>>
> Subject:Re: [DISCUSS] Integrate Flink SQL well with
Hive ecosystem
>
> Hi Xuefu,
>
> Currently the new design doc
>
<https://docs.google.com/document/d/1Y9it78yaUvbv4g572ZK_lZnZaAGjqwM_EhjdOv4yJtw/edit>
> is on “view only" mode, and people cannot leave
comments. Can you please
> change it to "can comment" or "can edit" mode?
>
> Thanks, Bowen
>
>
> On Mon, Nov 12, 2018 at 9:51 PM Zhang, Xuefu
<xuef...@alibaba-inc.com <mailto:xuef...@alibaba-inc.com>>
> wrote:
> Hi Piotr
>
> I have extracted the API portion of the design and
the google doc is here
>
<https://docs.google.com/document/d/1Y9it78yaUvbv4g572ZK_lZnZaAGjqwM_EhjdOv4yJtw/edit?usp=sharing>.
> Please review and provide your feedback.
>
> Thanks,
> Xuefu
>
>
------------------------------------------------------------------
> Sender:Xuefu <xuef...@alibaba-inc.com
<mailto:xuef...@alibaba-inc.com>>
> Sent at:2018 Nov 12 (Mon) 12:43
> Recipient:Piotr Nowojski <pi...@data-artisans.com
<mailto:pi...@data-artisans.com>>; dev <
> dev@flink.apache.org <mailto:dev@flink.apache.org>>
> Cc:Bowen Li <bowenl...@gmail.com
<mailto:bowenl...@gmail.com>>; Shuyi Chen
<suez1...@gmail.com <mailto:suez1...@gmail.com>>
> Subject:Re: [DISCUSS] Integrate Flink SQL well with
Hive ecosystem
>
> Hi Piotr,
>
> That sounds good to me. Let's close all the open
questions ((there are a
> couple of them)) in the Google doc and I should be
able to quickly split
> it into the three proposals as you suggested.
>
> Thanks,
> Xuefu
>
>
------------------------------------------------------------------
> Sender:Piotr Nowojski <pi...@data-artisans.com
<mailto:pi...@data-artisans.com>>
> Sent at:2018 Nov 9 (Fri) 22:46
> Recipient:dev <dev@flink.apache.org
<mailto:dev@flink.apache.org>>; Xuefu
<xuef...@alibaba-inc.com <mailto:xuef...@alibaba-inc.com>>
> Cc:Bowen Li <bowenl...@gmail.com
<mailto:bowenl...@gmail.com>>; Shuyi Chen
<suez1...@gmail.com <mailto:suez1...@gmail.com>>
> Subject:Re: [DISCUSS] Integrate Flink SQL well with
Hive ecosystem
>
> Hi,
>
>
> Yes, it seems like the best solution. Maybe someone
else can also suggests if we can split it further?
Maybe changes in the interface in one doc, reading
from hive meta store another and final storing our
meta informations in hive meta store?
>
> Piotrek
>
> > On 9 Nov 2018, at 01:44, Zhang, Xuefu
<xuef...@alibaba-inc.com
<mailto:xuef...@alibaba-inc.com>> wrote:
> >
> > Hi Piotr,
> >
> > That seems to be good idea!
> >
>
> > Since the google doc for the design is currently
under extensive review, I will leave it as it is for
now. However, I'll convert it to two different FLIPs
when the time comes.
> >
> > How does it sound to you?
> >
> > Thanks,
> > Xuefu
> >
> >
> >
------------------------------------------------------------------
> > Sender:Piotr Nowojski <pi...@data-artisans.com
<mailto:pi...@data-artisans.com>>
> > Sent at:2018 Nov 9 (Fri) 02:31
> > Recipient:dev <dev@flink.apache.org
<mailto:dev@flink.apache.org>>
> > Cc:Bowen Li <bowenl...@gmail.com
<mailto:bowenl...@gmail.com>>; Xuefu
<xuef...@alibaba-inc.com <mailto:xuef...@alibaba-inc.com>
> >; Shuyi Chen <suez1...@gmail.com
<mailto:suez1...@gmail.com>>
> > Subject:Re: [DISCUSS] Integrate Flink SQL well
with Hive ecosystem
> >
> > Hi,
> >
>
> > Maybe we should split this topic (and the design
doc) into couple of smaller ones, hopefully
independent. The questions that you have asked Fabian
have for example very little to do with reading
metadata from Hive Meta Store?
> >
> > Piotrek
> >
> >> On 7 Nov 2018, at 14:27, Fabian Hueske
<fhue...@gmail.com <mailto:fhue...@gmail.com>> wrote:
> >>
> >> Hi Xuefu and all,
> >>
> >> Thanks for sharing this design document!
>
> >> I'm very much in favor of restructuring /
reworking the catalog handling in
> >> Flink SQL as outlined in the document.
>
> >> Most changes described in the design document
seem to be rather general and
> >> not specifically related to the Hive integration.
> >>
>
> >> IMO, there are some aspects, especially those at
the boundary of Hive and
> >> Flink, that need a bit more discussion. For example
> >>
> >> * What does it take to make Flink schema
compatible with Hive schema?
> >> * How will Flink tables (descriptors) be stored
in HMS?
> >> * How do both Hive catalogs differ? Could they
be integrated into to a
> >> single one? When to use which one?
>
> >> * What meta information is provided by HMS? What
of this can be leveraged
> >> by Flink?
> >>
> >> Thank you,
> >> Fabian
> >>
> >> Am Fr., 2. Nov. 2018 um 00:31 Uhr schrieb Bowen
Li <bowenl...@gmail.com <mailto:bowenl...@gmail.com>
> >:
> >>
> >>> After taking a look at how other discussion
threads work, I think it's
> >>> actually fine just keep our discussion here.
It's up to you, Xuefu.
> >>>
> >>> The google doc LGTM. I left some minor comments.
> >>>
> >>> On Thu, Nov 1, 2018 at 10:17 AM Bowen Li
<bowenl...@gmail.com <mailto:bowenl...@gmail.com>> wrote:
> >>>
> >>>> Hi all,
> >>>>
> >>>> As Xuefu has published the design doc on
google, I agree with Shuyi's
>
> >>>> suggestion that we probably should start a new
email thread like "[DISCUSS]
>
> >>>> ... Hive integration design ..." on only dev
mailing list for community
> >>>> devs to review. The current thread sends to
both dev and user list.
> >>>>
>
> >>>> This email thread is more like validating the
general idea and direction
>
> >>>> with the community, and it's been pretty long
and crowded so far. Since
>
> >>>> everyone is pro for the idea, we can move
forward with another thread to
> >>>> discuss and finalize the design.
> >>>>
> >>>> Thanks,
> >>>> Bowen
> >>>>
> >>>> On Wed, Oct 31, 2018 at 12:16 PM Zhang, Xuefu <
> xuef...@alibaba-inc.com
<mailto:xuef...@alibaba-inc.com>>
> >>>> wrote:
> >>>>
> >>>>> Hi Shuiyi,
> >>>>>
>
> >>>>> Good idea. Actually the PDF was converted
from a google doc. Here is its
> >>>>> link:
> >>>>>
> >>>>>
>
https://docs.google.com/document/d/1SkppRD_rE3uOKSN-LuZCqn4f7dz0zW5aa6T_hBZq5_o/edit?usp=sharing
> >>>>> Once we reach an agreement, I can convert it
to a FLIP.
> >>>>>
> >>>>> Thanks,
> >>>>> Xuefu
> >>>>>
> >>>>>
> >>>>>
> >>>>>
------------------------------------------------------------------
> >>>>> Sender:Shuyi Chen <suez1...@gmail.com
<mailto:suez1...@gmail.com>>
> >>>>> Sent at:2018 Nov 1 (Thu) 02:47
> >>>>> Recipient:Xuefu <xuef...@alibaba-inc.com
<mailto:xuef...@alibaba-inc.com>>
> >>>>> Cc:vino yang <yanghua1...@gmail.com
<mailto:yanghua1...@gmail.com>>; Fabian Hueske <
> fhue...@gmail.com <mailto:fhue...@gmail.com>>;
> >>>>> dev <dev@flink.apache.org
<mailto:dev@flink.apache.org>>; user
<u...@flink.apache.org <mailto:u...@flink.apache.org>>
> >>>>> Subject:Re: [DISCUSS] Integrate Flink SQL
well with Hive ecosystem
> >>>>>
> >>>>> Hi Xuefu,
> >>>>>
>
> >>>>> Thanks a lot for driving this big effort. I
would suggest convert your
>
> >>>>> proposal and design doc into a google doc,
and share it on the dev mailing
>
> >>>>> list for the community to review and comment
with title like "[DISCUSS] ...
>
> >>>>> Hive integration design ..." . Once
approved, we can document it as a FLIP
>
> >>>>> (Flink Improvement Proposals), and use JIRAs
to track the implementations.
> >>>>> What do you think?
> >>>>>
> >>>>> Shuyi
> >>>>>
> >>>>> On Tue, Oct 30, 2018 at 11:32 AM Zhang, Xuefu <
> xuef...@alibaba-inc.com
<mailto:xuef...@alibaba-inc.com>>
> >>>>> wrote:
> >>>>> Hi all,
> >>>>>
> >>>>> I have also shared a design doc on Hive
metastore integration that is
>
> >>>>> attached here and also to FLINK-10556[1].
Please kindly review and share
> >>>>> your feedback.
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>> Xuefu
> >>>>>
> >>>>> [1]
https://issues.apache.org/jira/browse/FLINK-10556
> >>>>>
------------------------------------------------------------------
> >>>>> Sender:Xuefu <xuef...@alibaba-inc.com
<mailto:xuef...@alibaba-inc.com>>
> >>>>> Sent at:2018 Oct 25 (Thu) 01:08
> >>>>> Recipient:Xuefu <xuef...@alibaba-inc.com
<mailto:xuef...@alibaba-inc.com>>; Shuyi Chen <
> >>>>> suez1...@gmail.com <mailto:suez1...@gmail.com>>
> >>>>> Cc:yanghua1127 <yanghua1...@gmail.com
<mailto:yanghua1...@gmail.com>>; Fabian Hueske <
> fhue...@gmail.com <mailto:fhue...@gmail.com>>;
> >>>>> dev <dev@flink.apache.org
<mailto:dev@flink.apache.org>>; user
<u...@flink.apache.org <mailto:u...@flink.apache.org>>
> >>>>> Subject:Re: [DISCUSS] Integrate Flink SQL
well with Hive ecosystem
> >>>>>
> >>>>> Hi all,
> >>>>>
> >>>>> To wrap up the discussion, I have attached a
PDF describing the
>
> >>>>> proposal, which is also attached to
FLINK-10556 [1]. Please feel free to
> >>>>> watch that JIRA to track the progress.
> >>>>>
> >>>>> Please also let me know if you have
additional comments or questions.
> >>>>>
> >>>>> Thanks,
> >>>>> Xuefu
> >>>>>
> >>>>> [1]
https://issues.apache.org/jira/browse/FLINK-10556
> >>>>>
> >>>>>
> >>>>>
------------------------------------------------------------------
> >>>>> Sender:Xuefu <xuef...@alibaba-inc.com
<mailto:xuef...@alibaba-inc.com>>
> >>>>> Sent at:2018 Oct 16 (Tue) 03:40
> >>>>> Recipient:Shuyi Chen <suez1...@gmail.com
<mailto:suez1...@gmail.com>>
> >>>>> Cc:yanghua1127 <yanghua1...@gmail.com
<mailto:yanghua1...@gmail.com>>; Fabian Hueske <
> fhue...@gmail.com <mailto:fhue...@gmail.com>>;
> >>>>> dev <dev@flink.apache.org
<mailto:dev@flink.apache.org>>; user
<u...@flink.apache.org <mailto:u...@flink.apache.org>>
> >>>>> Subject:Re: [DISCUSS] Integrate Flink SQL
well with Hive ecosystem
> >>>>>
> >>>>> Hi Shuyi,
> >>>>>
>
> >>>>> Thank you for your input. Yes, I agreed with
a phased approach and like
>
> >>>>> to move forward fast. :) We did some work
internally on DDL utilizing babel
> >>>>> parser in Calcite. While babel makes
Calcite's grammar extensible, at
> >>>>> first impression it still seems too
cumbersome for a project when too
>
> >>>>> much extensions are made. It's even
challenging to find where the extension
>
> >>>>> is needed! It would be certainly better if
Calcite can magically support
>
> >>>>> Hive QL by just turning on a flag, such as
that for MYSQL_5. I can also
>
> >>>>> see that this could mean a lot of work on
Calcite. Nevertheless, I will
>
> >>>>> bring up the discussion over there and to see
what their community thinks.
> >>>>>
> >>>>> Would mind to share more info about the
proposal on DDL that you
> >>>>> mentioned? We can certainly collaborate on this.
> >>>>>
> >>>>> Thanks,
> >>>>> Xuefu
> >>>>>
> >>>>>
------------------------------------------------------------------
> >>>>> Sender:Shuyi Chen <suez1...@gmail.com
<mailto:suez1...@gmail.com>>
> >>>>> Sent at:2018 Oct 14 (Sun) 08:30
> >>>>> Recipient:Xuefu <xuef...@alibaba-inc.com
<mailto:xuef...@alibaba-inc.com>>
> >>>>> Cc:yanghua1127 <yanghua1...@gmail.com
<mailto:yanghua1...@gmail.com>>; Fabian Hueske <
> fhue...@gmail.com <mailto:fhue...@gmail.com>>;
> >>>>> dev <dev@flink.apache.org
<mailto:dev@flink.apache.org>>; user
<u...@flink.apache.org <mailto:u...@flink.apache.org>>
> >>>>> Subject:Re: [DISCUSS] Integrate Flink SQL
well with Hive ecosystem
> >>>>>
> >>>>> Welcome to the community and thanks for the
great proposal, Xuefu! I
>
> >>>>> think the proposal can be divided into 2
stages: making Flink to support
>
> >>>>> Hive features, and make Hive to work with
Flink. I agreed with Timo that on
>
> >>>>> starting with a smaller scope, so we can make
progress faster. As for [6],
>
> >>>>> a proposal for DDL is already in progress,
and will come after the unified
>
> >>>>> SQL connector API is done. For supporting
Hive syntax, we might need to
> >>>>> work with the Calcite community, and a recent
effort called babel (
> >>>>>
https://issues.apache.org/jira/browse/CALCITE-2280) in
Calcite might
> >>>>> help here.
> >>>>>
> >>>>> Thanks
> >>>>> Shuyi
> >>>>>
> >>>>> On Wed, Oct 10, 2018 at 8:02 PM Zhang, Xuefu <
> xuef...@alibaba-inc.com
<mailto:xuef...@alibaba-inc.com>>
> >>>>> wrote:
> >>>>> Hi Fabian/Vno,
> >>>>>
>
> >>>>> Thank you very much for your encouragement
inquiry. Sorry that I didn't
>
> >>>>> see Fabian's email until I read Vino's
response just now. (Somehow Fabian's
> >>>>> went to the spam folder.)
> >>>>>
>
> >>>>> My proposal contains long-term and
short-terms goals. Nevertheless, the
> >>>>> effort will focus on the following areas,
including Fabian's list:
> >>>>>
> >>>>> 1. Hive metastore connectivity - This covers
both read/write access,
>
> >>>>> which means Flink can make full use of Hive's
metastore as its catalog (at
> >>>>> least for the batch but can extend for
streaming as well).
>
> >>>>> 2. Metadata compatibility - Objects
(databases, tables, partitions, etc)
>
> >>>>> created by Hive can be understood by Flink
and the reverse direction is
> >>>>> true also.
> >>>>> 3. Data compatibility - Similar to #2, data
produced by Hive can be
> >>>>> consumed by Flink and vise versa.
>
> >>>>> 4. Support Hive UDFs - For all Hive's native
udfs, Flink either provides
> >>>>> its own implementation or make Hive's
implementation work in Flink.
> >>>>> Further, for user created UDFs in Hive, Flink
SQL should provide a
>
> >>>>> mechanism allowing user to import them into
Flink without any code change
> >>>>> required.
> >>>>> 5. Data types - Flink SQL should support all
data types that are
> >>>>> available in Hive.
> >>>>> 6. SQL Language - Flink SQL should support
SQL standard (such as
>
> >>>>> SQL2003) with extension to support Hive's
syntax and language features,
> >>>>> around DDL, DML, and SELECT queries.
>
> >>>>> 7. SQL CLI - this is currently developing in
Flink but more effort is
> >>>>> needed.
>
> >>>>> 8. Server - provide a server that's
compatible with Hive's HiverServer2
>
> >>>>> in thrift APIs, such that HiveServer2 users
can reuse their existing client
> >>>>> (such as beeline) but connect to Flink's
thrift server instead.
>
> >>>>> 9. JDBC/ODBC drivers - Flink may provide its
own JDBC/ODBC drivers for
> >>>>> other application to use to connect to its
thrift server
> >>>>> 10. Support other user's customizations in
Hive, such as Hive Serdes,
> >>>>> storage handlers, etc.
>
> >>>>> 11. Better task failure tolerance and task
scheduling at Flink runtime.
> >>>>>
> >>>>> As you can see, achieving all those requires
significant effort and
>
> >>>>> across all layers in Flink. However, a
short-term goal could include only
>
> >>>>> core areas (such as 1, 2, 4, 5, 6, 7) or
start at a smaller scope (such as
> >>>>> #3, #6).
> >>>>>
>
> >>>>> Please share your further thoughts. If we
generally agree that this is
>
> >>>>> the right direction, I could come up with a
formal proposal quickly and
> >>>>> then we can follow up with broader discussions.
> >>>>>
> >>>>> Thanks,
> >>>>> Xuefu
> >>>>>
> >>>>>
> >>>>>
> >>>>>
------------------------------------------------------------------
> >>>>> Sender:vino yang <yanghua1...@gmail.com
<mailto:yanghua1...@gmail.com>>
> >>>>> Sent at:2018 Oct 11 (Thu) 09:45
> >>>>> Recipient:Fabian Hueske <fhue...@gmail.com
<mailto:fhue...@gmail.com>>
> >>>>> Cc:dev <dev@flink.apache.org
<mailto:dev@flink.apache.org>>; Xuefu
<xuef...@alibaba-inc.com <mailto:xuef...@alibaba-inc.com>
> >; user <
> >>>>> u...@flink.apache.org
<mailto:u...@flink.apache.org>>
> >>>>> Subject:Re: [DISCUSS] Integrate Flink SQL
well with Hive ecosystem
> >>>>>
> >>>>> Hi Xuefu,
> >>>>>
>
> >>>>> Appreciate this proposal, and like Fabian, it
would look better if you
> >>>>> can give more details of the plan.
> >>>>>
> >>>>> Thanks, vino.
> >>>>>
> >>>>> Fabian Hueske <fhue...@gmail.com
<mailto:fhue...@gmail.com>> 于2018年10月10日周三
下午5:27写道：
> >>>>> Hi Xuefu,
> >>>>>
>
> >>>>> Welcome to the Flink community and thanks for
starting this discussion!
> >>>>> Better Hive integration would be really great!
> >>>>> Can you go into details of what you are
proposing? I can think of a
> >>>>> couple ways to improve Flink in that regard:
> >>>>>
> >>>>> * Support for Hive UDFs
> >>>>> * Support for Hive metadata catalog
> >>>>> * Support for HiveQL syntax
> >>>>> * ???
> >>>>>
> >>>>> Best, Fabian
> >>>>>
> >>>>> Am Di., 9. Okt. 2018 um 19:22 Uhr schrieb
Zhang, Xuefu <
> >>>>> xuef...@alibaba-inc.com
<mailto:xuef...@alibaba-inc.com>>:
> >>>>> Hi all,
> >>>>>
> >>>>> Along with the community's effort, inside
Alibaba we have explored
>
> >>>>> Flink's potential as an execution engine not
just for stream processing but
> >>>>> also for batch processing. We are encouraged
by our findings and have
>
> >>>>> initiated our effort to make Flink's SQL
capabilities full-fledged. When
>
> >>>>> comparing what's available in Flink to the
offerings from competitive data
>
> >>>>> processing engines, we identified a major gap
in Flink: a well integration
>
> >>>>> with Hive ecosystem. This is crucial to the
success of Flink SQL and batch
>
> >>>>> due to the well-established data ecosystem
around Hive. Therefore, we have
>
> >>>>> done some initial work along this direction
but there are still a lot of
> >>>>> effort needed.
> >>>>>
> >>>>> We have two strategies in mind. The first one
is to make Flink SQL
>
> >>>>> full-fledged and well-integrated with Hive
ecosystem. This is a similar
>
> >>>>> approach to what Spark SQL adopted. The
second strategy is to make Hive
>
> >>>>> itself work with Flink, similar to the
proposal in [1]. Each approach bears
>
> >>>>> its pros and cons, but they don’t need to be
mutually exclusive with each
> >>>>> targeting at different users and use cases.
We believe that both will
> >>>>> promote a much greater adoption of Flink
beyond stream processing.
> >>>>>
> >>>>> We have been focused on the first approach
and would like to showcase
>
> >>>>> Flink's batch and SQL capabilities with Flink
SQL. However, we have also
> >>>>> planned to start strategy #2 as the follow-up
effort.
> >>>>>
>
> >>>>> I'm completely new to Flink(, with a short
bio [2] below), though many
>
> >>>>> of my colleagues here at Alibaba are
long-time contributors. Nevertheless,
>
> >>>>> I'd like to share our thoughts and invite
your early feedback. At the same
>
> >>>>> time, I am working on a detailed proposal on
Flink SQL's integration with
> >>>>> Hive ecosystem, which will be also shared
when ready.
> >>>>>
> >>>>> While the ideas are simple, each approach
will demand significant
>
> >>>>> effort, more than what we can afford. Thus,
the input and contributions
> >>>>> from the communities are greatly welcome and
appreciated.
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>>
> >>>>> Xuefu
> >>>>>
> >>>>> References:
> >>>>>
> >>>>> [1]
https://issues.apache.org/jira/browse/HIVE-10712
>
> >>>>> [2] Xuefu Zhang is a long-time open source
veteran, worked or working on
> >>>>> many projects under Apache Foundation, of
which he is also an honored
>
> >>>>> member. About 10 years ago he worked in the
Hadoop team at Yahoo where the
>
> >>>>> projects just got started. Later he worked at
Cloudera, initiating and
>
> >>>>> leading the development of Hive on Spark
project in the communities and
>
> >>>>> across many organizations. Prior to joining
Alibaba, he worked at Uber
>
> >>>>> where he promoted Hive on Spark to all Uber's
SQL on Hadoop workload and
> >>>>> significantly improved Uber's cluster efficiency.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
>
> >>>>> "So you have to trust that the dots will
somehow connect in your future."
> >>>>>
> >>>>>
> >>>>> --
>
> >>>>> "So you have to trust that the dots will
somehow connect in your future."
> >>>>>
>
>
