yxu-valleytider opened a new pull request #6968: Consuming data from DynamoDB streams to flink URL: https://github.com/apache/flink/pull/6968 ## What is the purpose of the change This PR introduces a new Flink source to consume directly from [dynamodb streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html). This new source is built on top of the existing Kinesis connector. It interacts with the dynamodb streams via a [dynamodb-streams-kinesis-adapter](https://github.com/awslabs/dynamodb-streams-kinesis-adapter) client. ## Brief change log New data stream can be constructed to directly pull data from DynamoDB streams. DataStream<String> dynamodbStreams = env.addSource(new FlinkDynamodbStreamsConsumer(streamName, ..., ...)) Other changes include: - *DynamodbStreamsProxy class which uses a `dynamodbstreams-kinesis` adapter client to interact with Dynamodb streams.* - *Port the `describeStream` API from the flink-1.5 branch into DynamodbStreamsProxy.* - *Minimalist DynamodbStreamsSchema class which helps deserialize dynamodb streams records* ## Verifying this change This change is already covered by most of the existing Flink kinesis connector tests. Manual tests are provided to verify that the dynamodbstreams connector can: - *consume from a stream given its ARN* - *consume from a given table with new stream created* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes) Added optional dependency on AWS [dynamodb-streams-kinesis-adapter](https://github.com/awslabs/dynamodb-streams-kinesis-adapter). - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (JavaDocs)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
