junaiddshaukat commented on issue #18479:
URL: https://github.com/apache/beam/issues/18479#issuecomment-3864268067
Thank you @je-ik for the detailed response and the clear direction!
I'd love to sketch a design document and iterate on it with you.
Your points give a solid foundation to start from:
1. **Fn API** as the target — agreed, this ensures all SDKs work from
day one with room for Java-specific optimizations later.
2. **Processor API** over DSL — makes sense given the flexibility
needed for Beam's model. I'll analyze the alignment more deeply
in the design doc.
3. **Minimum transforms**: Read, stateless ParDo, GBK, CBK, Window —
with stateful ParDo and splittable DoFn as stretch goals.
I'll start drafting the design document this week, covering:
- High-level architecture (pipeline translation flow)
- Mapping of Beam primitives to Kafka Streams Processor API constructs
- Watermark propagation strategy
- Bundle management and Beam's guarantees
- Timer support considerations
I'll share the initial draft here for your feedback before we take it
to the dev@ list. Looking forward to collaborating!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
