Hello Blake,
Sorry for the delay, I have posted a few comments to your PR.
Thanks for the contribution!
Danny
On Fri, Apr 15, 2022 at 6:03 PM Blake Wilson
wrote:
> Great to know! Thanks for the reference.
>
> On Fri, Apr 15, 2022 at 6:15 AM Danny Cranmer
> wrote:
>
> > Yes, the Flink Kinesis
Great to know! Thanks for the reference.
On Fri, Apr 15, 2022 at 6:15 AM Danny Cranmer
wrote:
> Yes, the Flink Kinesis Consumer detects aggregated records and seamlessly
> de-aggregates them for you [1].
>
> Thanks,
>
> [1]
>
> https://github.com/apache/flink/blob/master/flink-connectors/flink-c
Yes, the Flink Kinesis Consumer detects aggregated records and seamlessly
de-aggregates them for you [1].
Thanks,
[1]
https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/publisher/RecordBatch.ja
Thanks for offering to review, Danny.
Thanks also for pointing out that KCL can de-aggregate records aggregated
by KPL. Several applications I've worked on batch multiple records without
using the KPL unfortunately.
Is de-aggregation supported by the Kinesis Connector Source? I found
mention of a
Just to clarify, the native KCL/KPL aggregation [1] handles the partition
key rebalancing for you out of the box.
[1] https://docs.aws.amazon.com/streams/latest/dev/kinesis
-kpl-concepts.html#kinesis-kpl-concepts-aggretation
On Thu, Apr 14, 2022 at 8:48 AM Danny Cranmer
wrote:
> Hey Blake,
>
>
Hey Blake,
I am happy to take a look, but I will not have capacity until next week.
The current way to achieve multiple records per PUT is to use the native
KCL/KPL aggregation [1], which is supported by the Flink connector. A
downside of aggregation is that the sender has to manage the partition
Hello, I recently submitted a pull request to support the Collector API for
the Kinesis Streams Connector.
The ability to use this API would save a great deal of shuttling bytes
around in multiple Flink programs I've worked on. This is because to
construct a stream of the desired type without Coll
