Hey
No it was unrelated. Actually this was confusing: the pipeline wasn’t stuck
- it was working, it just didn’t show on Flink’s UI. I guess because it was
a bounded PCollection but not sure.
Nir
On Thu, 25 Feb 2021 at 22:06 Pablo Estrada <pabl...@google.com> wrote:
> hi Nir!
> was this fixed by the PR you submitted?
>
> On Wed, Feb 24, 2021 at 8:55 AM Nir Gazit <nir....@gmail.com> wrote:
>
>> Hey,
>> When trying to read a file from S3 with a combine action, the pipeline
>> seems to be stuck. When replacing it with a GCP source it works fine.
>> Furthermore - if I comment out the Count.PerElement part it also works.
>>
>> Anyone has an idea why that is?
>>
>> lines = p | beam.io.ReadFromText('s3://...')
>> transformed = (
>> lines
>> | 'SplitLine' >>
>> (beam.ParDo(WordExtractingDoFn()).with_output_types(unicode))
>> | 'Count' >> beam.combiners.Count.PerElement()
>> | 'Format' >> beam.MapTuple(lambda w, c: f'{w}: {c}')
>> )
>>
>> transformed | 'Write' >> beam.io.WriteToText('s3://...')
>>
>> Thanks!
>> Nir
>>
>
