Thanks all for the feedback. +1 on the single repo and version for AWS
connectors. The reduced maintenance cost and complexity is a clear winner
here.
I will open a vote thread for this matter.
Thanks all!
On Tue, 25 Oct 2022, 03:04 Jark Wu, wrote:
> TBH, I suspect the way of “a single reposit
TBH, I suspect the way of “a single repository per connector”, considering
there are hundreds of connectors out there (Airbyte[1], Kafka[2]).
I don’t think the community is feasible to maintain hundreds of repositories.
It makes sense to combine some connectors to reduce the maintenance burden.
Hi Danny,
I'm also leaning slightly towards the single AWS connector repo direction.
Bumps in the underlying AWS SDK would bump all of the connectors in any
case. And if a change occurs that is isolated to a single connector, then
those that do not use that connector can just skip the release.
C
I like the single repo with single version idea.
Pros:
- Better discoverability for connectors for AWS services means a better
experience for Flink users
- Natural placement of AWS-related utils (Credentials, SDK Retry strategy)
Caveats:
- As you mentioned, it is not desirable if we have to evol
Thanks Chesnay for the suggestion, I will investigate this option.
Related to the single repo idea, I have considered it in the past. Are you
proposing we also use a single version between all connectors? If we have a
single version then it makes sense to combine them in a single repo, if
they are
I agree with Jark. It would be easier for the further development and
maintenance, if all aws related connectors and the base module are in the
same repo. It might make sense to upgrade the flink-connector-dynamodb to
flink-connector-aws and move the other modules including the
flink-connector-aws-
How about creating a new repository flink-connector-aws and merging dynamodb,
kinesis firehouse into it?
This can reduce the maintenance for complex dependencies and make the release
easy.
I think the maintainers of aws-releated connectors are the same people.
Best,
Jark
> 2022年10月21日 17:41，C
I would not go with 2); I think it'd just be messy .
Here's another option:
Create another repository (aws-connector-base) (following the
externalization model), add it as a sub-module to the downstream
repositories, and make it part of the release process of said connector.
I.e., we never c
Hello all,
Currently we have 2 AWS Flink connectors in the main Flink codebase
(Kinesis Data Streams and Kinesis Data Firehose) and one new externalized
connector in progress (DynamoDB). Currently all three of these use common
AWS utilities from the flink-connector-aws-base module. Common code
inc
