nikitagrover19 commented on issue #36201: URL: https://github.com/apache/beam/issues/36201#issuecomment-3851083426
Thanks for the context — reading through #31608 helped clarify why this is more complex than just ` ordering_key ` through. I understand now that simply wiring the key into ` _PubSubWriteDoFn ` would only affect DirectRunner and could give a misleading impression of Dataflow support, given the batching logic and sink overrides used by DataflowRunner. I’m happy to adjust the scope of this work based on your guidance. A few options I see are: 1. **Document the limitation clearly** – add docstring or user-facing warnings that ` ordering_key ` is only honored with DirectRunner 2. **Add runtime guards** – warn or fail fast when ` ordering_key ` is used with DataflowRunner, similar to the Java-side handling in #31608 3. **Keep as-is** – merge DirectRunner-only support and track DataflowRunner support separately I’m also curious whether a Python equivalent of ` --experiments=enable_custom_pubsub_sink ` would be feasible as a shorter-term option, or if that’s blocked by deeper Dataflow backend work. I’m relatively new to Beam internals, so guidance on the intended direction would be really helpful. Happy to make whatever changes make sense. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
