Beam High Priority Issue Report (51)

2024-01-10 Thread beamactions
This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/29926 [Bug]: FileIO: lack

[YAML] ReadFromKafka with yaml

2024-01-10 Thread Yarden BenMoshe
Hi, I am trying to consume a kafka topic using ReadFromKafka transform. If i got it right, since ReadFromKafka is originally written in java, an expansion service is needed and default env is set to DOCKER, and in current implementation I can see that expansion service field is not adjustable (im

Re: [YAML] ReadFromKafka with yaml

2024-01-10 Thread Ferran Fernández Garrido
Hi Yarden, If you are using Dataflow as a runner, you can already use ReadFromKafka (introduced originally in version 2.52). Dataflow will handle the expansion service automatically, so you don't have to do anything. If you want to run it locally for development purposes, you'll have to build the

Beam 2.54.0 Release

2024-01-10 Thread Robert Burke
Hey everyone, Happy New Year! The next release (2.54.0) branch cut is scheduled for Jan 24, 2024, 2 weeks from today, according to the release calendar [1]. I'd like to perform this release; I will cut the branch on that date, and cherrypick remaining release-blocking fixes afterwards, if any. Plea

ByteBuddy DoFnInvokers Write Up

2024-01-10 Thread John Casey via dev
The team at Google recently held an internal hackathon, and my hack involved modifying how our ByteBuddy DoFnInvokers work. My hack didn't end up going anywhere, but I learned a lot about how our code generation works. It turns out we have no documentation or design docs about our code generation,

Re: ByteBuddy DoFnInvokers Write Up

2024-01-10 Thread Robert Burke
That's neat! Thanks for writing that up! On Wed, Jan 10, 2024, 11:12 AM John Casey via dev wrote: > The team at Google recently held an internal hackathon, and my hack > involved modifying how our ByteBuddy DoFnInvokers work. My hack didn't end > up going anywhere, but I learned a lot about how

Re: Beam 2.54.0 Release

2024-01-10 Thread Robert Burke
Not sure why newlines were eaten. Hopefully reflowed inline below. On 2024/01/10 17:53:56 Robert Burke wrote: > Hey everyone, Happy New Year! > > The next release (2.54.0) branch cut is scheduled for Jan 24, 2024, 2 weeks > from today, according to the release calendar [1]. I'd like to perform th

RE: Re: [YAML] ReadFromKafka with yaml

2024-01-10 Thread Yarden BenMoshe
Thanks for the detailed answer. I forgot to mention that I am using FlinkRunner as my Setup. Will this work with this runner as well? On 2024/01/10 13:34:28 Ferran Fernández Garrido wrote: > Hi Yarden, > > If you are using Dataflow as a runner, you can already use > ReadFromKafka (introduced or