Beam High Priority Issue Report (33)

2024-12-13 Thread beamactions
This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/32997 [Bug]: Non Retained

[PROPOSAL] Implement Beam SDK harness initialization capability for Python

2024-12-13 Thread Valentyn Tymofieiev via dev
Hi everyone, Currently we don't have a straightforward and documented way to do simple initialization steps on every Beam Python SDK worker before data processing starts. It is a rough edge that I've encountered on several occasions myself and in conversations with Beam users I put together some

Re: [PROPOSAL] Implement Beam SDK harness initialization capability for Python

2024-12-13 Thread Robert Bradshaw via dev
We already have https://github.com/apache/beam/blob/release-2.40.0/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L141 that allows arbitrary code to be imported and executed on worker startup. (Perhaps we could generalize to let it also reference a function to be called rather than just

Re: [Design] Anomaly Detection with Beam

2024-12-13 Thread Danny McCormick via dev
Thanks - this is exciting! I left a couple comments, but I am a big +1 to this effort! On Fri, Dec 13, 2024 at 3:07 PM XQ Hu via dev wrote: > Thanks for sharing. Great doc! > > On Fri, Dec 13, 2024 at 1:27 PM Shunping Huang > wrote: > >> Hi all, >> >> Recently, I have been working on a design f

Re: [PROPOSAL] Implement Beam SDK harness initialization capability for Python

2024-12-13 Thread Danny McCormick via dev
Thanks - I actually was thinking about this today and was annoyed that we don't have this ability. I'm +1 to the proposed approach. I dropped a comment, but also upleveling in case there is broader interest; it would be nice to have a similar capability for expansion service containers as well. O

[Design] Anomaly Detection with Beam

2024-12-13 Thread Shunping Huang
Hi all, Recently, I have been working on a design for a new Anomaly Detection PTransform in Beam. https://docs.google.com/document/d/1tE8lz9U_vjlNn2H7t-GRrs3vfhQ5UuCgWiHXCRHRPns/edit?usp=sharing If you are interested, please feel free to review the proposal and leave comments. Any feedback is gr

Re: [Design] Anomaly Detection with Beam

2024-12-13 Thread XQ Hu via dev
Thanks for sharing. Great doc! On Fri, Dec 13, 2024 at 1:27 PM Shunping Huang wrote: > Hi all, > > Recently, I have been working on a design for a new Anomaly Detection > PTransform in Beam. > > > https://docs.google.com/document/d/1tE8lz9U_vjlNn2H7t-GRrs3vfhQ5UuCgWiHXCRHRPns/edit?usp=sharing >

[Design] Anomaly Detection with Beam

2024-12-13 Thread Shunping Huang
Hi all, Recently, I have been working on a design for a new Anomaly Detection PTransform in Beam. https://docs.google.com/document/d/1tE8lz9U_vjlNn2H7t-GRrs3vfhQ5UuCgWiHXCRHRPns/edit?usp=sharing If you are interested, please feel free to review the proposal and leave comments. Any feedback is gr

Re: [Design] Anomaly Detection with Beam

2024-12-13 Thread Robert Bradshaw via dev
Thanks, this'll be a great addition to Beam. Left a couple of comments on the doc. On Fri, Dec 13, 2024 at 12:40 PM Danny McCormick via dev wrote: > > Thanks - this is exciting! I left a couple comments, but I am a big +1 to > this effort! > > On Fri, Dec 13, 2024 at 3:07 PM XQ Hu via dev wrote