[Question] Apache Beam library upgrade causing IllegalStateExceptions with setRowSchema and setCoder

2022-04-15 Thread Jimmy Headdon
Hello I'm attempting to upgrade the Apache Beam libraries from v2.19.0 to v2.37.0 (Java 8 & Maven), but have run into an issue with a breaking change that I'd appreciate some support with. Sorry this is quite a long one, I wanted to capture as much context as I could, but please shout if there's

[Question] Infer schema from a Pcollection of Python dicts

2022-04-15 Thread Nivaldo Tokuda
Hi, I have a pipeline with a Pcollection of dicts in Python, and I'd like to apply a schema to it for use with SQLTransforms. The schema is defined as follows: class RowSchema(typing.NamedTuple): colA: str colB: typing.Optional[str] beam.coders.registry.register_coder(RowSchema, be

RE: [Question] Infer schema from a Pcollection of Python dicts

2022-04-15 Thread Nivaldo Tokuda
I apologize for the formatting. Never used a mailing list before. I restructured the question here with the intended layout: https://gist.github.com/nivaldo-tokuda-g/4f9b0b3aafb28d7c5740d7da5a4cdc8c Este e-mail e seus anexos s?o pa

RE: [Question] Infer schema from a Pcollection of Python dicts

2022-04-15 Thread Nivaldo Tokuda
Hi Brian, I was fully confident I had double checked this before, but the type mismatch in the schema was exactly the issue. I changed one of the fields to int and the schema is now inferred correctly. Thanks a lot for the help! Es

Re: [Question] Apache Beam library upgrade causing IllegalStateExceptions with setRowSchema and setCoder

2022-04-15 Thread Jimmy Headdon
Thanks for the swift response Brian, Andrew. I've tried your suggestion Brian, and sadly I get the same error as the lengthy call stack from the end of my original post (IllegalStateException) - it appears the PCollection might have been finalised my the DoFn, and therefore I cannot setRowSchema a