Hi beam-users, I'm not sure if this is the right place for a Dataflow question - I think this is where the issue appears.
I've been trying to run some large backfills since last night and they run well, then get to almost 100 % on the main fused part of the pipeline ending with a GroupByKey, and the job gets stuck. Most of my jobs haven't finished - some of the small-ish-er ones have *eventually* finished after several hours of being stuck at 97-99 %. We ran this job recently with no issues, and with much more data in one go. To try to isolate the issue, I ran the current data with the older image where the job succeeded, and I also ran (a subset of) the older data with the current image. Both of these jobs also got stuck, so that leads me to believe it's a Dataflow issue rather than an issue on our end. All our jobs have --dataflow_service_options = ['enable_prime'] set, which I know is necessary for larger jobs where GroupByKey is used. Is there anything going on with Dataflow right now that might cause this issue? Is this even the right place to ask? Thanks! -Lina
