Is there a long-term plan for preventing fusion in Dataflow pipelines? Maybe a simple flag --disableFusion ?
I have read a few of the discussions in the Beam mailing lists, and I haven't found any sentiment that something should be changed about Dataflow, only that Dataflow users should work around this with sometimes-costly GBKs. My example is a streaming pipeline: PubSub => memcached read => http request => memcached write => GCS write The three middle steps are implemented as few lines of readable code. That's how pipeline steps/transformations should be written, agreed? But, since they are fused, they are all slow. I have tried some hacks successfully. For example, a DoFn handles an element by (1) start an async network request then (2) output any network responses responses waiting in a response queue. One DoFn instance can handle thousands of network requests concurrently. Also, this feels like I'm using Beam incorrectly. Jacob
