Yes, the probability of this happening is astronomical, but in production
with 128 core servers with 7000 max_connections, with petabyte scale data,
this did repro 2 times in the last month. We had to move to a local
approach to manager our ratelimiting counters.
This is not reproducible very easily. I feel that we should at least shield
ourselves with the following change, so that we at least increase the delay
by 1000us every time. We will follow a linear back off, but better than no
backoff.
status->cur_delay += max(1000, (int) (status->cur_delay *
pg_prng_double(&pg_global_prng_state) +
0.5));
On Wed, Apr 10, 2024 at 9:43 AM Robert Haas <robertmh...@gmail.com> wrote:
> On Wed, Apr 10, 2024 at 12:40 PM Parag Paul <parag.p...@gmail.com> wrote:
> > The reason why this could be a problem is a flaw in the RNG with the
> enlarged Hamming belt.
> > I attached an image here, with the RNG outputs from 2 backends. I ran
> our code for weeks, and collected ther
> > values generated by the RNG over many backends. The one in Green (say
> backend id 600), stopped flapping values and
> > only produced low (near 0 ) values for half an hour, whereas the
> Blue(say backend 700), kept generating good values and had
> > a range between [0-1)
> > During this period, the backed 600 suffered and ended up with spinlock
> stuck condition.
>
> This is a very vague description of a test procedure. If you provide a
> reproducible series of steps that causes a stuck spinlock, I imagine
> everyone will be on board with doing something about it. But this
> graph is not going to convince anyone of anything.
>
> --
> Robert Haas
> EDB: http://www.enterprisedb.com
>
