Re: Implementing a custom I/O Connector

Okay, beam.Impulse() does solve it now Thanks so much for your help. On Thu, Jul 14, 2022 at 10:59 PM Chamikara Jayalath wrote: > You cannot directly apply 'beam.ParDo' on the pipeline object. > Instead you feed the source description element to the ParDo, for example, > p | beam.Create([source

Re: Implementing a custom I/O Connector

You cannot directly apply 'beam.ParDo' on the pipeline object. Instead you feed the source description element to the ParDo, for example, p | beam.Create([source_description]) | beam.ParDo(CountFn(10)) If the 'source_description' element is trivial (or gets ignored in the source), you can replace

Re: Implementing a custom I/O Connector

It has this error After some looking around, I think the problem was because I treated the Splittable DoFn as a regular DoFn. And they weren't any PCollection in the pipeline. --- > AttributeErrorT

Re: Implementing a custom I/O Connector

Do you have the full stacktrace ? Also, what does the Read() transform in the example entail ? Thanks, Cham On Thu, Jul 14, 2022 at 7:39 AM Damian Akpan wrote: > Hi Everyone, > > I've been working on implementing a Google Sheets IO source for my > pipeline. I've tried this example >

Implementing a custom I/O Connector

Hi Everyone, I've been working on implementing a Google Sheets IO source for my pipeline. I've tried this example along with this blog . I have an example here on colab