"Operations on the executor will generally calculate and store a seed once"
Can you elaborate more this? Does Spark try to seed RNGs to ensure overall quality of random number generating? To give an extremely example, if all workers use the same seed, then RNGs repeat the same numbers on each worker, which is obviously a poor choice. Best, ---- Ben Du Personal Blog<http://www.legendu.net/> | GitHub<https://github.com/dclong/> | Bitbucket<https://bitbucket.org/dclong/> | Docker Hub<https://hub.docker.com/r/dclong/> ________________________________ From: Sean Owen <[email protected]> Sent: Monday, October 4, 2021 1:00 PM To: Benjamin Du <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [RNG]: How does Spark handle RNGs? The 2nd approach. Spark doesn't work in the 1st way in any context - the driver and executor processes do not cooperate during execution. Operations on the executor will generally calculate and store a seed once, and use that in RNGs, to make its computation reproducible. On Mon, Oct 4, 2021 at 2:20 PM Benjamin Du <[email protected]<mailto:[email protected]>> wrote: Hi everyone, I'd like to ask how does Spark (or more generally, distributed computing engines) handle RNGs? High-level speaking, there are two ways, 1. Use a single RNG on the driver and random numbers generating on each work makes request to the single RNG on the driver. 2. Use a separate RNG on each worker. If the 2nd approach above is used, may I ask how does Spark seed RNGs on different works to ensure the overall quality of random number generating? Best, ---- Ben Du Personal Blog<http://www.legendu.net/> | GitHub<https://github.com/dclong/> | Bitbucket<https://bitbucket.org/dclong/> | Docker Hub<https://hub.docker.com/r/dclong/>
