Does anyone actually use Streaming Autoscaling with cloud Dataflow? I have seen scale-ups based on CPU but never on backlog. Now I do not see scale up events at all. If this works can you please point me to a working example.
On 2019/01/09 20:09:46, Ken Barr <ken.b...@solace.com> wrote: > Hello > > I have been directed to this mail list by posting to Stack Overflow. If this > is inappropriate use of mail list please let me > know.https://stackoverflow.com/questions/54100870/autoscale-stuck-at-1-never-see-getsplitbacklogbytes-execute > > I am creating a custom IO connector in Java that runs is streaming mode with > 2.8.0 SDK. It reads from a messaging queue for further pipeline processing, > similar to JMS IO. All works well except the auto-scaling does not seem to > work. The odd part for me is that I have implemented a getSplitBacklogBytes() > method, but as far as the logs show, it is never executed. > > I am testing with a modified wordcount example to test, executed with: > > --runner=DataflowRunner --autoscalingAlgorithm=THROUGHPUT_BASED > --maxNumWorkers=4 > > > > The Pipeline options and logs seem to show that auto-scaling is being set up > correctly. > > PipelineOptions: Current Settings: > > autoscalingAlgorithm: THROUGHPUT_BASED > > maxNumWorkers: 4 > > numWorkers: 0 > > Logs show: > > 20:50:54:977 .. [INFO] .. - 2019-01-08T20:50:40.937Z: Autoscaling is enabled > for job 2019-01-08_12_50_40-3227727519433807139. The number of workers will > be between 1 and 4. > > 20:50:54:991 .. [INFO] .. - 2019-01-08T20:50:42.943Z: Autoscaling: Enabled > for job 2019-01-08_12_50_40-3227727519433807139 between 1 and 4 worker > processes. > > 20:50:55:052 .. [INFO] .. - 2019-01-08T20:50:49.186Z: Generating 4 persistent > disks which will allow autoscaling from 1 to 4 workers. > > 20:50:55:053 .. [INFO] .. - 2019-01-08T20:50:49.493Z: Starting 1 workers... > > > > Is there anything else that needs to be done to enable this feature, or > anything I might have done to inadvertently disable this feature? > > Source code lives here: > https://github.com/SolaceLabs/solace-beam-integration/tree/development > >
