Hi Rong, Thanks for pointing out the missing FLIPs in the FLIP main page. I added all the missing FLIP (incl. FLIP-14, FLIP-22, FLIP-29, FLIP-30, FLIP-31) to the page.
I also include @xuef...@alibaba-inc.com <xuef...@alibaba-inc.com> and @Bowen Li <bowenl...@gmail.com> into the thread who are familiar with the latest catalog design. Thanks, Jark On Thu, 18 Apr 2019 at 02:39, Rong Rong <walter...@gmail.com> wrote: > Thanks Artsem for looking into this problem and Thanks Dawid for bringing > up the discussion on FLIP-30. > > We've observe similar scenarios when we also would like to reuse the schema > registry of both Kafka stream as well as the raw ingested kafka messages in > datalake. > FYI another more catalog-oriented document can be found here [1]. I do have > one question to follow up with Dawid's point (2): are we suggesting that > different kafka topics (e.g. test-topic-prod, test-topic-non-prod, etc) > considered as a "view" of a logical table with schema (e.g. test-topic) ? > > Also, seems like a few of the FLIPs, like the FLIP-30 page is not linked in > the main FLIP confluence wiki page [2] for some reason. > I tried to fix that be seems like I don't have permission. Maybe someone > can also take a look? > > Thanks, > Rong > > > [1] > > https://docs.google.com/document/d/1Y9it78yaUvbv4g572ZK_lZnZaAGjqwM_EhjdOv4yJtw/edit#heading=h.xp424vn7ioei > [2] > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > On Wed, Apr 17, 2019 at 2:30 AM Artsem Semianenka <artfulonl...@gmail.com> > wrote: > > > Thank you, Dawid! > > This is very helpful information. I will keep a close eye on the updates > of > > FLIP-30 and contribute whenever it possible. > > I guess I may create a Jira ticket for my proposal in which I describe > the > > idea and attach intermediate pull request based on current API(just for > > initial discuss). But the final pull request definitely will be based on > > FLIP-30 API. > > > > Best regards, > > Artsem > > > > On Wed, 17 Apr 2019 at 09:36, Dawid Wysakowicz <dwysakow...@apache.org> > > wrote: > > > > > Hi Artsem, > > > > > > I think it totally makes sense to have a catalog for the Schema > > > Registry. It is also good to hear you want to contribute that. There is > > > few important things to consider though: > > > > > > 1. The Catalog interface is currently under rework. You make take a > look > > > at the corresponding FLIP-30[1], and also have a look at the first PR > > > that introduces the basic interfaces[2]. I think it would be worth to > > > already consider those changes. I cc Xuefu who is participating in the > > > efforts of Catalog integration. > > > > > > 2. There is still ongoing discussion about what properties should we > > > store for streaming tables and how. I think this might affect (but > maybe > > > doesn't have to) the design of the Catalog.[3] I cc Timo who might give > > > more insights if those should be blocking for the work around this > > Catalog. > > > > > > Best, > > > > > > Dawid > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs > > > > > > [2] https://github.com/apache/flink/pull/8007 > > > > > > [3] > > > > > > > > > https://docs.google.com/document/d/1Yaxp1UJUFW-peGLt8EIidwKIZEWrrA-pznWLuvaH39Y/edit#heading=h.egn858cgizao > > > > > > On 16/04/2019 17:35, Artsem Semianenka wrote: > > > > Hi guys! > > > > > > > > I'm working on External Catalog for Confluent Kafka. The main idea to > > > > register the external catalog which provides the list of Kafka topics > > and > > > > execute SQL queries like : > > > > Select * form kafka.topic_name > > > > > > > > I'm going to receive the table schema from Confluent schema registry. > > The > > > > main disadvantage is: we should have the topic name with the same > name > > > > (prefix and postfix are accepted ) as this schema subject in Schema > > > > Registry. > > > > For example : > > > > topic: test-topic-prod > > > > schema subject: test-topic > > > > > > > > I would like to contribute this solution into the main Flink branch > and > > > > would like to discuss the pros and cons of this approach. > > > > > > > > Best regards, > > > > Artsem > > > > > > > > > > > > > > -- > > > > С уважением, > > Артем Семененко > > >
