Re: [DESIGN] Timers, Watermark Holds, Loops, Batch and Drain

hi Kenn, I would add option 3 Option 3: Use current way of using tag, add drain in additional proto object - The tag & 0b0111 indicates whether it is using FIRST, ONE_INDEX or TWO_INDICES encoding and future options if needed. - tag & 0b1 ("WITH_EXTENSION" ) indicates there i

Re: [DESIGN] Timers, Watermark Holds, Loops, Batch and Drain

I love using one WITH_EXTENSIONS bit and then adding a proto. Especially for full WindowedValue where there is already ~10-12 bytes overhead, it doesn't make sense to create our own extensible encoding protocol. - I think we can actually do this without a breaking change, too. Which part were you

Re: [DESIGN] Timers, Watermark Holds, Loops, Batch and Drain

Breaking change is when a portable runner (e.g. UW) running on a newer version, starts to add an extension to WindowedValue which the old SDK hasn't expected. Old SDK only expects FIRST, ONE_INDEX or TWO_INDICES, without an extension bit. Radek On Fri, Apr 25, 2025 at 4:33 PM Kenneth Knowles wro

Re: [DESIGN] Timers, Watermark Holds, Loops, Batch and Drain

Ah, yes. We have this field where an SDK can declare its capabilities: https://github.com/apache/beam/blob/908d43e65ca8281f4a8d6188dac30a5964cb213c/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto#L1559 Kenn On Fri, Apr 25, 2025 at 10:44 AM Radek Stankiewicz

Re: [DISCUSS] Proposal for Debezium IO Upgrade to 3.1.1 (and Java 17 Build Enablement)

Hi Tobi, Thanks for your interest and willing to take on the task. First of all, could you please create a GitHub Issue to track the effort? I would like to share some comments if find useful - We already have infrastructure for the goal of "PR 1". We use BeamPluginModulePlugin [1] to initialize

[DISCUSS] Proposal for Debezium IO Upgrade to 3.1.1 (and Java 17 Build Enablement)

Hi Beam Devs, I'm interested in helping to upgrade the Debezium IO connector ( sdks:java:io:debezium) to Debezium version 3.1.1.Final. This version of Debezium requires its dependencies (like debezium-core) to be compiled against Java 17

Re: [DISCUSS] Proposal for Debezium IO Upgrade to 3.1.1 (and Java 17 Build Enablement)

Thank you both! I have checked and couldn't find a matching issue, so I created this one: https://github.com/apache/beam/issues/34747 On Sat, Apr 26, 2025 at 12:32 AM XQ Hu via dev wrote: > Can we create the GitHub issue to track this work if we haven't? > > On Fri, Apr 25, 2025, 4:39 PM Yi Hu v

Re: [DISCUSS] Proposal for Debezium IO Upgrade to 3.1.1 (and Java 17 Build Enablement)

Can we create the GitHub issue to track this work if we haven't? On Fri, Apr 25, 2025, 4:39 PM Yi Hu via dev wrote: > Hi Tobi, > > Thanks for your interest and willing to take on the task. First of all, > could you please create a GitHub Issue to track the effort? > > I would like to share some