--
From:Timo Walther
Sent At:2019 Jan. 7 (Mon.) 05:59
To:dev
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
Hi everyone,
Xuefu and I had multiple iterations over the catalog design document
[1]. I believe that it is in a good shape now to be converted into FLIP.
Maybe we
LIP-30%3A+Unified+Catalog+APIs
>
>
> --
> From:Timo Walther
> Sent At:2019 Jan. 7 (Mon.) 05:59
> To:dev
> Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
>
> Hi everyone,
>
> Xuefu and I had multiple iterations over the catalog
--
From:Timo Walther
Sent At:2019 Jan. 7 (Mon.) 05:59
To:dev
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
Hi everyone,
Xuefu and I had multiple iterations over the catalog design document
[1]. I believe that it is in a good shape now to be converted into
8 Nov 18 (Sun) 15:17
Recipient:dev mailto:dev@flink.apache.org>>
Cc:Xuefu mailto:xuef...@alibaba-inc.com>>; twalthr
mailto:twal...@apache.org>>;
piotr mailto:pi...@data-artisans.com>>; Fabian Huesk
mailto:twal...@apache.org>>;
piotr mailto:pi...@data-artisans.com>>; Fabian Hueske
mailto:fhue...@gmail.com>>;
suez1224 mailto:suez1...@gmail.com>>; Bowen Li
mailto:bowenl...@gmail.com>>
Su
le for
>>> inclusion once we finalize the overall design.
>>>
>>> Thus, please continue your research and experiment and let us know when
>>> you start working on anything so we can better coordinate.
>>>
>>> Thanks again for your interest and contributions.
>>>
>>&g
n for your interest and contributions.
>>
>> Thanks,
>> Xuefu
>>
>>
>>
>> ----------------------
>> From:Eron Wright
>> Sent At:2019 Jan. 1 (Tue.) 18:39
>> To:dev ; Xuefu
>> Cc:Xiaowei Jiang ; twa
nate.
>
> Thanks again for your interest and contributions.
>
> Thanks,
> Xuefu
>
>
>
> --
> From:Eron Wright
> Sent At:2019 Jan. 1 (Tue.) 18:39
> To:dev ; Xuefu
> Cc:Xiaowei Jiang ; twalthr ;
> piotr ; Fabian Hueske ;
> suez1224 ; Bowen L
nks,
Xuefu
--
From:Eron Wright
Sent At:2019 Jan. 1 (Tue.) 18:39
To:dev ; Xuefu
Cc:Xiaowei Jiang ; twalthr ; piotr
; Fabian Hueske ; suez1224
; Bowen Li
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
Hi folks, there's clearly some incremental steps to be taken to int
the current
> design doc shows. I'm still refining this part.
>
> Thanks,
> Xuefu
>
>
> --
> Sender:Xiaowei Jiang
> Sent at:2018 Nov 18 (Sun) 15:17
> Recipient:dev
> Cc:Xuefu ; twalthr ; piotr <
> pi...@data-artisans.com>;
nks,
Xuefu
--
Sender:Xiaowei Jiang
Sent at:2018 Nov 18 (Sun) 15:17
Recipient:dev
Cc:Xuefu ; twalthr ; piotr
; Fabian Hueske ; suez1224
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
Thanks Xuefu for the detailed design doc! One q
error. I have granted comment permission
> > with the link.
> >
> > I also updated the doc with the latest class definitions. Everyone is
> > encouraged to review and comment.
> >
> > Thanks,
> > Xuefu
> >
> > -
---
> Sender:Bowen Li
> Sent at:2018 Nov 14 (Wed) 06:44
> Recipient:Xuefu
> Cc:piotr ; dev ; Shuyi
> Chen
> Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
>
> Hi Xuefu,
>
> Currently the new design do
Sent at:2018 Nov 14 (Wed) 06:44
Recipient:Xuefu
Cc:piotr ; dev ; Shuyi Chen
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
Hi Xuefu,
Currently the new design doc is on “view only" mode, and people cannot leave
comments. Can you please change it to "can comment&quo
eview and provide your feedback.
>
> Thanks,
> Xuefu
>
> --
> Sender:Xuefu
> Sent at:2018 Nov 12 (Mon) 12:43
> Recipient:Piotr Nowojski ; dev <
> dev@flink.apache.org>
> Cc:Bowen Li ; Shuyi Chen
> Subject:Re: [DISCUSS] I
Li ; Shuyi Chen
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
Hi Piotr,
That sounds good to me. Let's close all the open questions ((there are a couple
of them)) in the Google doc and I should be able to quickly split it into the
three proposals as you suggested.
T
Xuefu
--
Sender:Piotr Nowojski
Sent at:2018 Nov 9 (Fri) 22:46
Recipient:dev ; Xuefu
Cc:Bowen Li ; Shuyi Chen
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
Hi,
Yes, it seems like the best solution. Maybe someone else can also suggests if
we can split it further? Maybe changes i
it sound to you?
>
> Thanks,
> Xuefu
>
>
> --
> Sender:Piotr Nowojski
> Sent at:2018 Nov 9 (Fri) 02:31
> Recipient:dev
> Cc:Bowen Li ; Xuefu ; Shuyi
> Chen
> Subject:Re: [DISCUSS] Integrate F
Xuefu
--
Sender:Piotr Nowojski
Sent at:2018 Nov 9 (Fri) 02:31
Recipient:dev
Cc:Bowen Li ; Xuefu ; Shuyi Chen
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
Hi,
Maybe we should split this topic (and the design doc) into couple of smaller
ommunity to review and comment with title like "[DISCUSS] ...
>>>> Hive integration design ..." . Once approved, we can document it as a FLIP
>>>> (Flink Improvement Proposals), and use JIRAs to track the implementations.
>>>> What do you think?
>>>>
>>>> Shuyi
>>>>
gt; link:
>>>
>>> https://docs.google.com/document/d/1SkppRD_rE3uOKSN-LuZCqn4f7dz0zW5aa6T_hBZq5_o/edit?usp=sharing
>>> Once we reach an agreement, I can convert it to a FLIP.
>>>
>>> Thanks,
>>> Xuefu
>>>
>>>
>>>
>>> -
4f7dz0zW5aa6T_hBZq5_o/edit?usp=sharing
>> Once we reach an agreement, I can convert it to a FLIP.
>>
>> Thanks,
>> Xuefu
>>
>>
>>
>> ----------
>> Sender:Shuyi Chen
>> Sent at:2018 Nov 1 (Thu) 02:47
>&
convert it to a FLIP.
>
> Thanks,
> Xuefu
>
>
>
> --
> Sender:Shuyi Chen
> Sent at:2018 Nov 1 (Thu) 02:47
> Recipient:Xuefu
> Cc:vino yang ; Fabian Hueske ;
> dev ; user
> Subject:Re: [DISCUSS]
--
Sender:Shuyi Chen
Sent at:2018 Nov 1 (Thu) 02:47
Recipient:Xuefu
Cc:vino yang ; Fabian Hueske ; dev
; user
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
Hi Xuefu,
Thanks a lot for driving this big effort. I would suggest convert your
01:08
> Recipient:Xuefu ; Shuyi Chen
> Cc:yanghua1127 ; Fabian Hueske ;
> dev ; user
> Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
>
> Hi all,
>
> To wrap up the discussion, I have attached a PDF describing the proposal,
> which is also attach
--
Sender:Xuefu
Sent at:2018 Oct 25 (Thu) 01:08
Recipient:Xuefu ; Shuyi Chen
Cc:yanghua1127 ; Fabian Hueske ; dev
; user
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
Hi all,
To wrap up the discussion, I have attached a PDF describing the proposal, which
is also
://issues.apache.org/jira/browse/FLINK-10556
--
Sender:Xuefu
Sent at:2018 Oct 16 (Tue) 03:40
Recipient:Shuyi Chen
Cc:yanghua1127 ; Fabian Hueske ; dev
; user
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
Hi Shuyi,
Thank
Chen
Sent at:2018 Oct 14 (Sun) 08:30
Recipient:Xuefu
Cc:yanghua1127 ; Fabian Hueske ; dev
; user
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
Welcome to the community and thanks for the great proposal, Xuefu! I think the
proposal can be divided into 2 stages: makin
:Bowen
Sent at:2018 Oct 13 (Sat) 21:55
Recipient:Xuefu ; Fabian Hueske
Cc:dev ; user
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
Thank you Xuefu, for bringing up this awesome, detailed proposal! It will
resolve lots of existing pain for users like me.
In general, I tot
t;
> Thanks,
> Xuefu
>
>
>
> --------------
> Sender:vino yang
> Sent at:2018 Oct 11 (Thu) 09:45
> Recipient:Fabian Hueske
> Cc:dev ; Xuefu ; user <
> u...@flink.apache.org>
> Subject:Re: [DISC
use to connect to its thrift server
>> 10. Support other user's customizations in Hive, such as Hive Serdes,
>> storage handlers, etc.
>> 11. Better task failure tolerance and task scheduling at Flink runtime.
>>
>> As you can see, achieving all those requires signific
> Sender:Jörn Franke
> Sent at:2018 Oct 11 (Thu) 13:54
> Recipient:Xuefu
> Cc:vino yang ; Fabian Hueske ; dev
> ; user
> Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
>
> Would it maybe make sense to provide Flink as an engine o
06
> Recipient:Xuefu
> Cc:Rong Rong ; Timo Walther ;
> dev ; jornfranke ; vino yang <
> yanghua1...@gmail.com>; Fabian Hueske ; user <
> u...@flink.apache.org>
> Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
>
> One other thought on the same
o Walther ; dev
; jornfranke ; vino yang
; Fabian Hueske ; user
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
One other thought on the same lines was to use hive tables to store kafka
information to process streaming tables. Something like
"create table streaming_tabl
at Flink
>> provides. It's usually the servicer owner who provides drivers to their
>> services. We weren't talking about JDBC/ODBC driver to external DB systems.
>>
>> Let me know if you have further questions.
>>
>> Thanks,
>> Xuefu
>>
>> -----
> Sender:Rong Rong
> Sent at:2018 Oct 12 (Fri) 01:52
> Recipient:Timo Walther
> Cc:dev ; jornfranke ; Xuefu <
> xuef...@alibaba-inc.com>; vino yang ; Fabian
> Hueske ; user
> Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
>
>
Sender:Rong Rong
Sent at:2018 Oct 12 (Fri) 01:52
Recipient:Timo Walther
Cc:dev ; jornfranke ; Xuefu
; vino yang ; Fabian Hueske
; user
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
Hi Xuefu,
Thanks for putting together the overview. I would like to add some mo
;> all layers in Flink. However, a short-term goal could include only core
>> areas (such as 1, 2, 4, 5, 6, 7) or start at a smaller scope (such as #3,
>> #6).
>>
>> Please share your further thoughts. If we generally agree that this is the
>> right direction, I could come up with a formal proposal quickly and then we
>> can follow up wit
nificant effort and
> across all layers in Flink. However, a short-term goal could include only
> core areas (such as 1, 2, 4, 5, 6, 7) or start at a smaller scope (such as
> #3, #6).
> >>
> >> Please share your further thoughts. If we generally agree that this is
> the rig
feedback.
Thanks,
Xuefu
--
Sender:Jörn Franke
Sent at:2018 Oct 11 (Thu) 13:54
Recipient:Xuefu
Cc:vino yang ; Fabian Hueske ; dev
; user
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
Would it maybe make sense to provide Flink as an engine on Hive
(„flink
irection, I could come up with a formal proposal quickly and then we can
follow up with broader discussions.
Thanks,
Xuefu
--
Sender:vino yang
Sent at:2018 Oct 11 (Thu) 09:45
Recipient:Fabian Hueske
Cc:dev ; Xuefu ; use
n we
> can follow up with broader discussions.
>
> Thanks,
> Xuefu
>
>
>
> ------
> Sender:vino yang
> Sent at:2018 Oct 11 (Thu) 09:45
> Recipient:Fabian Hueske
> Cc:dev ; Xuefu ; user
>
> Subject:R
d then we can
follow up with broader discussions.
Thanks,
Xuefu
--
Sender:vino yang
Sent at:2018 Oct 11 (Thu) 09:45
Recipient:Fabian Hueske
Cc:dev ; Xuefu ; user
Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
Hi Xuefu,
Hi Xuefu,
Appreciate this proposal, and like Fabian, it would look better if you can
give more details of the plan.
Thanks, vino.
Fabian Hueske 于2018年10月10日周三 下午5:27写道：
> Hi Xuefu,
>
> Welcome to the Flink community and thanks for starting this discussion!
> Better Hive integration would be re
Hi Xuefu,
Welcome to the Flink community and thanks for starting this discussion!
Better Hive integration would be really great!
Can you go into details of what you are proposing? I can think of a couple
ways to improve Flink in that regard:
* Support for Hive UDFs
* Support for Hive metadata cat
Hi all,
Along with the community's effort, inside Alibaba we have explored Flink's
potential as an execution engine not just for stream processing but also for
batch processing. We are encouraged by our findings and have initiated our
effort to make Flink's SQL capabilities full-fledged. When c
46 matches
Mail list logo
