Re: Fastest way to clone schema ~1000x

An update on this for anyone else who has similar issues/considerations: - As suggested, we are sticking to 1 database-per-test, and not a schema-per-test. - We moved our testing setup to maintain at most 1 connection per backend worker. If a request for a different test comes in, we close and reco

Re: Fastest way to clone schema ~1000x

On 2/26/24 01:06, Emiel Mols wrote: On Mon, Feb 26, 2024 at 3:50 PM Daniel Gustafsson > wrote: There is a measurable overhead in connections, regardless of if they are used or not.  If you are looking to squeeze out performance then doing more over alr

Re: Fastest way to clone schema ~1000x

On Mon, Feb 26, 2024 at 3:50 PM Daniel Gustafsson wrote: > There is a measurable overhead in connections, regardless of if they are > used > or not. If you are looking to squeeze out performance then doing more over > already established connections, and reducing max_connections, is a good > pla

Re: Fastest way to clone schema ~1000x

> On 26 Feb 2024, at 09:46, Emiel Mols wrote: > The 2048 we actually need for our test setup. There is a measurable overhead in connections, regardless of if they are used or not. If you are looking to squeeze out performance then doing more over already established connections, and reducing ma

Re: Fastest way to clone schema ~1000x

We've experimented with shared buffers to no effect. The 2048 we actually need for our test setup. The way this works is that we have a single preforked backend for all tests where each backend worker maintains persistent connections *per test* (in database-per-test), so with say 50 backend proces

Re: Fastest way to clone schema ~1000x

po 26. 2. 2024 v 8:08 odesílatel Emiel Mols napsal: > Thanks, as indicated we're using that right now. The 30% spinlock overhead > unfortunately persists. > try to increase shared_buffer 128MB can be too low max_connection = 2048 - it unrealistic high > - Fsync was already disabled, too. Com

Re: Fastest way to clone schema ~1000x

Thanks, as indicated we're using that right now. The 30% spinlock overhead unfortunately persists. - Fsync was already disabled, too. Complete postgresql.conf used in testing: listen_addresses = '' max_connections = 2048 unix_socket_directories = '..' shared_buffers = 128MB log_line_prefix = '' sy

Re: Fastest way to clone schema ~1000x

Hi po 26. 2. 2024 v 7:28 odesílatel Emiel Mols napsal: > Hello, > > To improve our unit and end-to-end testing performance, we are looking to > optimize initialization of around 500-1000 database *schemas* from a > schema.sql file. > > Background: in postgres, you cannot change databases on > ex

Fastest way to clone schema ~1000x

Hello, To improve our unit and end-to-end testing performance, we are looking to optimize initialization of around 500-1000 database *schemas* from a schema.sql file. Background: in postgres, you cannot change databases on existing connections, and using schemas would allow us to minimize the amo