Hi Utharsh,
the cause of the error is the use_deprecated_read flag you pass to the
expansion service. Dataflow v2 expands KafkaIO using SDF [1] and does
not support (as the error says) the depreacted Read transform. Just
remove the flag and it should work.
Best,
Jan
[1] https://beam.apache.org/blog/splittable-do-fn/
On 2/26/25 08:31, Utkarsh Parekh wrote:
Hello,
I’m encountering the following error while working with ReadFromKafka
with the Dataflow runner V2 in Apache Beam:
2025-02-25 23:14:19.435 PST
Error translating pipeline. Unknown non-composite transform urn
beam:transform:read:v1 for
external_1Reading-messages-from-Kafka-KafkaIO-Read-KafkaIO-Read-ReadFromKafkaViaUnbounded-Read-KafkaUnboundedS
Below is the code snippet triggering the error:
fromapache_beam.io.kafka importReadFromKafka, default_io_expansion_service
pcoll =self.pipeline |"Reading messages from Kafka">>(ReadFromKafka(
self.consumer_config,
self.kafka_config.get('topics'),
commit_offset_in_finalize=True,
with_metadata=True,
expansion_service=default_io_expansion_service(append_args=['--experiments=use_deprecated_read']))
)
*Background:*
I’m looking to switch from the default connector (ReadFromKafkaViaSDF)
to the legacy connector (ReadFromKafkaViaUnboundedSource), as the
default connector is causing the Dataflow job to generate many Kafka
connections.
Any guidance or suggestions to resolve this issue would be greatly
appreciated.
Thank you
--
Utkarsh
Sr. Data Engineer
Albertsons Companies
Mobile: (860)-578-5208
Email: utkarsh.par...@albertsons.com
------------------------------------------------------------------------
Warning: All e-mail sent to this address will be received by the
corporate e-mail system, and is subject to archival and review by
someone other than the recipient. This e-mail may contain proprietary
information and is intended only for the use of the intended
recipient(s). If the reader of this message is not the intended
recipient(s), you are notified that you have received this message in
error and that any review, dissemination, distribution or copying of
this message is strictly prohibited. If you have received this message
in error, please notify the sender immediately.
------------------------------------------------------------------------
