Is there any kind of multiprocess parallelism built into the Python sdk
worker? In other words, is there a way for my runner to start a worker and
have it has multiple cores instead of having one worker per core? I thought
I saw this capability somewhere but now that I look I can only see the
pipeline option for it but nothing in sdk_worker or sdk_worker_main.
I'm starting to think about how to scale into the 10,000s of running
workers and I'm concerned that having 10,000 threads on the driver to
maintain the grpc connections will be a bit much to manage.

Cheers,
Joey

Reply via email to