Hi All ,
Thank you for all your valuable suggestions and questions regarding the
proposals.
In case there are more queries or questions from the community , I will
keep this discussion Thread open for a couple of more days and proceed with
next steps.
Bests
Samrat
On Wed, Dec 14, 2022 at 9:41 P
Thank you Danny for more insights on the flink-connector-aws-base[1].
It looks like localstack supports glue [2], we already use localstack for
> integration tests so we can follow suite here.
As GlueCatalog will be a part of flink-connector-aws-base. As per
suggestion, we will reuse code and re
Hello Samrat,
Sorry for the late response.
+1 for a native Glue Data Catalog integration. We have
internally developed a Glue Data Catalog catalog implementation that shims
hive. We have been meaning to contribute, but this solution can replace our
internal one.
+1 for putting this in the flink-
Hi Konstantin Knauf,
Can you explain how users are expected to authenticate with AWS Glue? I
> don't see any catalog options regardng authx. So I assume the credentials
> are taken from the environment?
We are planning to put GlueCatalog in flink-connector-aws[1].
flink-connector-aws already pro
Hi Jark,
Apologies for late reply.
Thank you for your valuable input.
Besides, I have a question about Glue Namespace. Could you share the
> documentation of the Glue
> Namespaces? (Sorry, I didn't find it.) According to the "Flink Glue
> Metaspace Mapping" section,
> if there is a database "mydb
Hi Samrat,
+1 to the effort and +1 to adding it to flink-connector-aws.
Can you explain how users are expected to authenticate with AWS Glue? I
don't see any catalog options regardng authx. So I assume the credentials
are taken from the environment?
Best,
Konstantin
Am Fr., 9. Dez. 2022 um 1
Hi Samrat,
Thanks a lot for driving the new catalog, and sorry for jumping into the
discussion late.
As Flink SQL is becoming the first-class citizen of the Flink API, we are
planning to push Catalog
to become the first-class citizen of the connector instead of Source &
Sink. For Flink SQL users,
Hi Samrat,
Sorry for the late reply. Yeah I am referring to creating a similar
external repo such as flink-catalog-glue. flink-connector-aws is already
named with `connector` so it seems a bit weird to put a catalog there.
Thanks!
Dong
On Wed, Dec 7, 2022 at 1:04 PM Samrat Deb wrote:
> Hi Dong
Hi Dong Lin,
Since this is the first proposal for adding a vendor-specific catalog
> library in Flink, I think maybe we should also externalize those catalog
> libraries similar to how we are externalizing connector libraries. It is
> likely that we might want to add catalogs for other vectors in
Hi Dong Lin,
Aws Glue Data catalog is vendor specific and in future we will get such
type of implementation from different providers. We should
definitely externalize these catalog libraries similar to flink connectors.
I am thinking of creating
flink-catalog similar to flink-connector under the r
Hi Yuxia,
Thank you for reviewing the flip and putting forward your observations and
comments.
1: I noticed there's a YAML part in the section of "Using the Catalog",
> what do you mean by that? Do you mean how to use glue catalog in sql
> client? If so, just for your information, it's not suppor
Hi Samrat,
Thanks for the FLIP!
Since this is the first proposal for adding a vendor-specific catalog
library in Flink, I think maybe we should also externalize those catalog
libraries similar to how we are externalizing connector libraries. It is
likely that we might want to add catalogs for oth
pdd780wl4f26p447fohvm9osky2r9fhh
[2] https://issues.apache.org/jira/browse/FLINK-22540
Best regards,
Yuxia
- 原始邮件 -
发件人: "Samrat Deb"
收件人: "dev"
抄送: "prabhujose gates"
发送时间: 星期六, 2022年 12 月 03日 下午 12:29:16
主题: [DISCUSS] FLIP-277: Native GlueCatalog Support in Fl
Hi everyone,
I would like to open a discussion[1] on providing GlueCatalog support
in Flink.
Currently, Flink offers 3 major types of catalog[2]. Out of which only
HiveCatalog is a persistent catalog backed by Hive Metastore. We would like
to introduce GlueCatalog in Flink offering another option
14 matches
Mail list logo
