Re: Hardware for writing/updating 12,000,000 rows per hour

> On Jul 27, 2019, at 11:55 AM, Arya F wrote: > > On Sat, Jul 27, 2019 at 11:49 AM farjad.farid > wrote: > With this kind of design requirements it is worth considering hardware > "failure & recovery". Even SSDs can and do fail. > > It is not just a mat

Re: Why does backend send buffer size hardcoded at 8KB?

Hi, On 2019-07-27 19:10:22 -0400, Tom Lane wrote: > Andres Freund writes: > > Additionally we perhaps ought to just not use the send buffer when > > internal_putbytes() is called with more data than can fit in the > > buffer. We should fill it with as much data as fits in it (so the > > pending d

Re: Why does backend send buffer size hardcoded at 8KB?

Andres Freund writes: > On 2019-07-27 18:34:50 -0400, Tom Lane wrote: >> Yeah. The existing commentary about that is basically justifying 8K >> as being large enough to avoid performance issues; if somebody can >> show that that's not true, I wouldn't have any hesitation about >> kicking it up.

Re: Why does backend send buffer size hardcoded at 8KB?

Hi, On 2019-07-27 18:34:50 -0400, Tom Lane wrote: > Andres Freund writes: > > It might be better to just use larger send sizes however. I think most > > kernels are going to be better than us knowing how to chop up the send > > size. > Yeah. The existing commentary about that is basically justi

Re: Why does backend send buffer size hardcoded at 8KB?

Andres Freund writes: > It might be better to just use larger send sizes however. I think most > kernels are going to be better than us knowing how to chop up the send > size. Yeah. The existing commentary about that is basically justifying 8K as being large enough to avoid performance issues; i

RE: Hardware for writing/updating 12,000,000 rows per hour

HI Arya, It is not clear what is the budget and why there is so much data? Is this a real time system, e.g. 24/7 operation. Even if each row takes up just 50 bytes, that is a lot of data in/out of your CPUs/memory/hard disk, any one of which could fail. Personally I would recommend analyzing t

Re: Why does backend send buffer size hardcoded at 8KB?

Hi, On 2019-07-27 11:09:06 -0400, Tom Lane wrote: > Artemiy Ryabinkov writes: > > Does it make sense to make this parameter configurable? > > Not without some proof that it makes a performance difference on > common setups (which you've not provided). I think us unnecessarily fragmenting into so

Re: Why does backend send buffer size hardcoded at 8KB?

On 2019-07-27 14:43:54 +0300, Artemiy Ryabinkov wrote: > Why backend send buffer use exactly 8KB? > (https://github.com/postgres/postgres/blob/249d64999615802752940e017ee5166e726bc7cd/src/backend/libpq/pqcomm.c#L134) > > > I had this question when I try to measure the speed of reading data. The

Re: Default ordering option

On 2019-07-26 09:52:52 +0200, Cyril Champier wrote: > No, the code I pasted was an existing production bug: the last_name > should have been unique, so the selected patient would always be the > same. This should have been detected in tests, but since the order was > "almost always the same", our t

Re: Hardware for writing/updating 12,000,000 rows per hour

On Sat, Jul 27, 2019 at 11:49 AM farjad.farid < farjad.fa...@checknetworks.com> wrote: > With this kind of design requirements it is worth considering hardware > "failure & recovery". Even SSDs can and do fail. > > It is not just a matter of just speed. RAID disks of some kind, depending > on the

RE: Hardware for writing/updating 12,000,000 rows per hour

With this kind of design requirements it is worth considering hardware "failure & recovery". Even SSDs can and do fail. It is not just a matter of just speed. RAID disks of some kind, depending on the budget is worth the effort. -Original Message- From: Alvaro Herrera Sent: 2019

Re: Why does backend send buffer size hardcoded at 8KB?

Artemiy Ryabinkov writes: > Does it make sense to make this parameter configurable? Not without some proof that it makes a performance difference on common setups (which you've not provided). Even with some proof, I'm not sure I'd bother with exposing a user-tunable knob, as opposed to just maki

Why does backend send buffer size hardcoded at 8KB?

Why backend send buffer use exactly 8KB? (https://github.com/postgres/postgres/blob/249d64999615802752940e017ee5166e726bc7cd/src/backend/libpq/pqcomm.c#L134) I had this question when I try to measure the speed of reading data. The bottleneck was a read syscall. With strace I found that in mos