+u...@beam.apache.org <u...@beam.apache.org> Reshuffle on Google Cloud Dataflow for a bounded pipeline waits till all the data has been read before the next transforms can run. After the reshuffle, the data should have been processed in parallel across the workers. Did you see this?
Are you able to change the input of your pipeline to use an uncompressed file or many compressed files? On Fri, May 10, 2019 at 1:03 PM Allie Chen <yifangc...@google.com> wrote: > Hi, > > > I am trying to load a gzip file to BigQuey using Dataflow. Since the > compressed file is not splittable, one worker is allocated to read the > file. The same worker will do all the other transforms since Dataflow fused > all transforms together. There are a large amount of data in the file, and > I expect to see more workers spinning up after reading transforms. I tried > to use Reshuffle Transform > <https://github.com/apache/beam/blob/release-2.3.0/sdks/python/apache_beam/transforms/util.py#L516> > to prevent the fusion, but it is not scalable since it won’t proceed until > all data arrived at this point. > > Is there any other ways to allow more workers working on all the other > transforms after reading? > > Thanks, > > Allie > >
