Re: [PATCH] heap_insert() and heap_update() optimization

2018-10-16 Thread Andres Freund
Hi, On 2018-10-17 09:48:19 +0300, Andrey Klychkov wrote: > > Interesting. That's with an optimized build, or an assertion build? > > Hello, > That was an optimized build. > > However I've just done some extra time tests and didn't notice so significant > difference as early. > Even more - avg o

Re[2]: [PATCH] heap_insert() and heap_update() optimization

2018-10-16 Thread Andrey Klychkov
> Interesting. That's with an optimized build, or an assertion build? Hello, That was an optimized build. However I've just done some extra time tests and didn't notice so significant difference as early. Even more - avg origin 1272, avg patched 1303. Maybe there was the autovacuum / analyze /

Fwd: Query Optimizer Postgresql

2018-10-16 Thread Sumit Chaturvedi
Hello Everyone, I had some questions about the query optimization engine and will be grateful if someone can answer them -- Forwarded message - From: Sumit Chaturvedi Date: Sun, Oct 14, 2018 at 8:50 PM Subject: Query Optimizer Postgresql To: Cc: Adwait Godbole , Nilay Pande ,

Re: pageinspect: add tuple_data_record()

2018-10-16 Thread Andres Freund
Hi, On 2018-10-16 22:05:28 -0400, James Coleman wrote: > > I don't think it's entirely safe to do so for invisible rows. The toast > > references could be reused, constraints be violated, etc. So while I > > could have used this several times before, it's also very likely a good > > way to cause

Re: pageinspect: add tuple_data_record()

2018-10-16 Thread Peter Geoghegan
On Tue, Oct 16, 2018 at 6:39 PM James Coleman wrote: > Summary: > The new function tuple_data_record() parallels the existing > tuple_data_split() function, but instead of returning a bytea array of raw > attribute heap values, it returns a row type of the relation being examined. I've been doi

Re: pageinspect: add tuple_data_record()

2018-10-16 Thread James Coleman
> I don't think it's entirely safe to do so for invisible rows. The toast > references could be reused, constraints be violated, etc. So while I > could have used this several times before, it's also very likely a good > way to cause trouble. It'd probably be ok to just fetch the binary > value

Re: pageinspect: add tuple_data_record()

2018-10-16 Thread Andres Freund
Hi, On 2018-10-16 21:39:02 -0400, James Coleman wrote: > Background: > In my usage of pageinspect for debugging or other purposes I often find it > frustrating that I don't have a way to easily view a heap page's tuple data > as actual records rather than binary data. After encountering this again

pageinspect: add tuple_data_record()

2018-10-16 Thread James Coleman
Background: In my usage of pageinspect for debugging or other purposes I often find it frustrating that I don't have a way to easily view a heap page's tuple data as actual records rather than binary data. After encountering this again last week while doing some data recovery after an accidental de

Re: TupleTableSlot abstraction

2018-10-16 Thread Andres Freund
Hi, On 2018-10-15 12:12:03 +0530, Amit Khandekar wrote: > On Sat, 13 Oct 2018 at 04:02, Andres Freund wrote: > > I'd just have ExecInterpExpr() continue calling ExecEvalSysVar. There's > > no reason for it to be inline. > Can you explain more why you think there should be a ExecEvalSysVar() > de

Re: Large writable variables

2018-10-16 Thread Tom Lane
BTW, I looked around for .o files with large BSS numbers, and came across $ size src/interfaces/ecpg/ecpglib/prepare.o textdata bss dec hex filename 4023 4 1048576 1052603 100fbb src/interfaces/ecpg/ecpglib/prepare.o That megabyte is from a statically allocated statem

Re: pgsql: Add TAP tests for pg_verify_checksums

2018-10-16 Thread Andres Freund
On October 16, 2018 3:47:55 PM PDT, Tom Lane wrote: >Michael Paquier writes: >> On Mon, Oct 15, 2018 at 11:20:28AM -0400, Andrew Dunstan wrote: >>> OK. In that case should we note that the utility is broken on >Windows? > >> Not sure if that's worth adding in the documentation, something in >t

Re: pgsql: Add TAP tests for pg_verify_checksums

2018-10-16 Thread Tom Lane
Michael Paquier writes: > On Mon, Oct 15, 2018 at 11:20:28AM -0400, Andrew Dunstan wrote: >> OK. In that case should we note that the utility is broken on Windows? > Not sure if that's worth adding in the documentation, something in the > press release would be more adapted? I can't get too exci

Re: Large writable variables

2018-10-16 Thread Tom Lane
I wrote: > 1. tzdefrules_s is filled in the postmaster, and if it's not filled there > it'd be filled by every child process, so there's zero point in converting > it to malloc-on-the-fly style. This is because tzparse() always wants > it filled. That's actually a tad annoying, because it looks t

Re: pgsql: Add TAP tests for pg_verify_checksums

2018-10-16 Thread Michael Paquier
On Mon, Oct 15, 2018 at 11:20:28AM -0400, Andrew Dunstan wrote: > On 10/15/2018 11:05 AM, Tom Lane wrote: >> Andrew Dunstan writes: >>> We should fix this in PG11 rather than ship a broken utility. If >>> everyone is happy, I can apply this. >> >> At this point I'd be happier if you waited till af

Re: Large writable variables

2018-10-16 Thread Andres Freund
On 2018-10-16 16:36:12 -0400, Tom Lane wrote: > Andres Freund writes: > > Attached is a patch that shrinks fmgr_builtins by 25%. That seems > > worthwhile, it's pretty frequently accessed, making it more dense is > > helpful. Unless somebody protests soon, I'm going to apply that... > > Hah. I'

Re: Large writable variables

2018-10-16 Thread Andres Freund
Hi, On 2018-10-17 09:38:18 +1300, Gavin Flower wrote: > On 17/10/2018 09:36, Tom Lane wrote: > > Andres Freund writes: > > > Attached is a patch that shrinks fmgr_builtins by 25%. That seems > > > worthwhile, it's pretty frequently accessed, making it more dense is > > > helpful. Unless somebody

Re: Large writable variables

2018-10-16 Thread Gavin Flower
On 17/10/2018 09:36, Tom Lane wrote: Andres Freund writes: Attached is a patch that shrinks fmgr_builtins by 25%. That seems worthwhile, it's pretty frequently accessed, making it more dense is helpful. Unless somebody protests soon, I'm going to apply that... Hah. I'm pretty sure that struc

Re: Large writable variables

2018-10-16 Thread Tom Lane
Andres Freund writes: > Attached is a patch that shrinks fmgr_builtins by 25%. That seems > worthwhile, it's pretty frequently accessed, making it more dense is > helpful. Unless somebody protests soon, I'm going to apply that... Hah. I'm pretty sure that struct *was* set up with an eye to padd

Re: Large writable variables

2018-10-16 Thread Andres Freund
On 2018-10-16 10:16:33 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2018-10-16 01:59:00 -0400, Tom Lane wrote: > >> Also, I noticed that the biggest part of those structs are arrays of > >> FormatNode, which has been designed with complete lack of thought about > >> size or padding issues

Re: [PATCH] heap_insert() and heap_update() optimization

2018-10-16 Thread Andres Freund
Hi, On 2018-10-16 11:28:17 +0300, Andrey Klychkov wrote: > I suggest the small attached patch that gives a bit of heap_insert() and > heap_update() optimization > by reducing calls of BufferGetPage(buffer) into them. > I measured call time of these: > heap_insert(): avg origin 13394 ns, avg patch

Re: pg_dump multi VALUES INSERT

2018-10-16 Thread Tom Lane
Surafel Temesgen writes: > According to the documentation –inserts option is mainly useful for making > dumps that can be loaded into non-PostgreSQL databases and to reduce the > amount of rows that might lost during error in reloading but multi values > insert command are equally portable and com

Re: [RFC] Removing "magic" oids

2018-10-16 Thread Robert Haas
On Sun, Oct 14, 2018 at 6:43 PM Andres Freund wrote: > I'm not sure we want that however - yes, the short time pain will be > lower, but do we really want to inflict the confusion about invisible > oids on our users for the next 20 years? I think there's a fair argument > to be made that we should

Re: PG vs macOS Mojave

2018-10-16 Thread Tom Lane
I wrote: > If we go this route, then the configure-time sysroot path will get > embedded into $perl_includedir and $TCL_INCLUDE_SPEC, but noplace > else. I don't know how far that would improve your concern about > extensions, but it should help a little. > I'm still a bit dubious about the safety

Re: Large writable variables

2018-10-16 Thread Andres Freund
Hi, On 2018-10-16 15:13:43 -0400, Robert Haas wrote: > On Mon, Oct 15, 2018 at 4:08 PM Andres Freund wrote: > > So we have 500kb of not-initialized memory mapped into every > > process. That's, uh, not nothing. > > Thinking about this a bit more, why is this bad? I mean, if the > memory is neve

Re: Large writable variables

2018-10-16 Thread Andres Freund
Hi, On 2018-10-15 13:07:54 -0700, Andres Freund wrote: > Hi, > > while briefly thinking about the per-process overhead of postgres, I > looked at the section sizes of a modern postgres. In particular which > memory areas are *not* shared between processes. > > If you look at the section sizes that

Re: Index Skip Scan

2018-10-16 Thread Dmitry Dolgov
> On Fri, 12 Oct 2018 at 19:44, Robert Haas wrote: > On Thu, Sep 13, 2018 at 11:40 AM Jesper Pedersen > wrote: > > > I noticed that current implementation doesn't > > > perform well when we have lots of small groups of equal values. Here is > > > the execution time of index skip scan vs unique ov

Re: Large writable variables

2018-10-16 Thread Robert Haas
On Mon, Oct 15, 2018 at 4:08 PM Andres Freund wrote: > So we have 500kb of not-initialized memory mapped into every > process. That's, uh, not nothing. Thinking about this a bit more, why is this bad? I mean, if the memory is never touched, the OS does not really need to allocate or zero any pag

Re: Large writable variables

2018-10-16 Thread Tom Lane
Andres Freund writes: > On 2018-10-16 12:41:44 -0400, Robert Haas wrote: >> In terms of a function that returns both const and non-const >> variables, it seems a bit sketchy that the caller would know what the >> function is doing in particular cases and make decisions based on it, >> but maybe th

Re: Large writable variables

2018-10-16 Thread Andres Freund
Hi, On 2018-10-16 12:41:44 -0400, Robert Haas wrote: > On Tue, Oct 16, 2018 at 12:26 PM Andres Freund wrote: > > /* > > * Macro that allows to cast constness away from a variable, but doesn't > > * allow changing the underlying type. Enforcement of the latter > > * currently only works for gc

Re: Large writable variables

2018-10-16 Thread Robert Haas
On Tue, Oct 16, 2018 at 12:26 PM Andres Freund wrote: > /* > * Macro that allows to cast constness away from a variable, but doesn't > * allow changing the underlying type. Enforcement of the latter > * currently only works for gcc like compilers. > * > * Please note IT IS NOT SAFE to cast c

Re: Large writable variables

2018-10-16 Thread Andres Freund
On 2018-10-16 12:19:55 -0400, Robert Haas wrote: > On Tue, Oct 16, 2018 at 12:06 PM Andres Freund wrote: > > But yea, it definitely should have a big red warning > > label. > > That's all I'm asking. And the warning label shouldn't just say "use > with caution!" but should rather explain how to

Re: Large writable variables

2018-10-16 Thread Robert Haas
On Tue, Oct 16, 2018 at 12:06 PM Andres Freund wrote: > But yea, it definitely should have a big red warning > label. That's all I'm asking. And the warning label shouldn't just say "use with caution!" but should rather explain how to know whether you're doing the right thing. -- Robert Haas E

pg_dump multi VALUES INSERT

2018-10-16 Thread Surafel Temesgen
Hello, According to the documentation –inserts option is mainly useful for making dumps that can be loaded into non-PostgreSQL databases and to reduce the amount of rows that might lost during error in reloading but multi values insert command are equally portable and compact and also faster to re

Re: Large writable variables

2018-10-16 Thread Andres Freund
On 2018-10-16 11:22:31 -0400, Robert Haas wrote: > On Tue, Oct 16, 2018 at 2:30 AM Andres Freund wrote: > > This just reminded me that a couple times I wanted a cast that casts > > away const, but otherwise makes sure the type stays the same. I don't > > think there's a way to do that in C, but we

Re: Large writable variables

2018-10-16 Thread Robert Haas
On Tue, Oct 16, 2018 at 2:30 AM Andres Freund wrote: > This just reminded me that a couple times I wanted a cast that casts > away const, but otherwise makes sure the type stays the same. I don't > think there's a way to do that in C, but we can write one that verifies > the cast doesn't do someth

Re: PG vs macOS Mojave

2018-10-16 Thread Tom Lane
I wrote: > Jakob Egger writes: >> Am 25. Sep. 2018 um 06:49 schrieb Tom Lane : >>> It's a bit scary to be adding -isysroot globally. >> This breaks building extensions with a different SDK than the server itself. > Hm, under what circumstances would that be safe or a good idea? Oh, well *this*

Re: Large writable variables

2018-10-16 Thread Tom Lane
Andres Freund writes: > On 2018-10-16 01:59:00 -0400, Tom Lane wrote: >> Also, I noticed that the biggest part of those structs are arrays of >> FormatNode, which has been designed with complete lack of thought about >> size or padding issues. We can very easily cut it in half on 64-bit >> machin

Re: Creating Certificates

2018-10-16 Thread Bruce Momjian
On Tue, Oct 16, 2018 at 02:02:48PM +0200, Magnus Hagander wrote: > > > On Tue, Oct 16, 2018 at 4:49 AM Bruce Momjian wrote: > > On Tue, Oct 16, 2018 at 11:45:53AM +0900, Tatsuo Ishii wrote: > > > I'm not opposed to simplifying the instructions, however. > > > > Ok, attached is a

Re: PG vs macOS Mojave

2018-10-16 Thread Tom Lane
Jakob Egger writes: > Am 25. Sep. 2018 um 06:49 schrieb Tom Lane : >> It's a bit scary to be adding -isysroot globally. > This breaks building extensions with a different SDK than the server itself. Hm, under what circumstances would that be safe or a good idea? The concerns that strike my mind

Re: WIP: Avoid creation of the free space map for small tables

2018-10-16 Thread Amit Kapila
On Tue, Oct 16, 2018 at 4:27 PM John Naylor wrote: > > [proposal for a cache of blocks to try] > > That's interesting. I'll have to do some reading elsewhere in the > codebase, and then I'll follow up. > Thanks, I have changed the status of this patch as "Waiting on Author". Feel free to change

Re: Creating Certificates

2018-10-16 Thread Magnus Hagander
On Tue, Oct 16, 2018 at 4:49 AM Bruce Momjian wrote: > On Tue, Oct 16, 2018 at 11:45:53AM +0900, Tatsuo Ishii wrote: > > > I'm not opposed to simplifying the instructions, however. > > > > Ok, attached is a proposal to simplify the instructions. > > I am against this simplification for the reason

Re: PG vs macOS Mojave

2018-10-16 Thread Jakob Egger
Am 25. Sep. 2018 um 06:49 schrieb Tom Lane : > It's a bit scary to be adding -isysroot globally. > (...) > I've tested this on all the macOS versions I have at hand, and it > doesn't seem to break anything. > (...) > Thoughts? > > regards, tom lane This breaks building ext

Re: TupleTableSlot abstraction

2018-10-16 Thread Amit Khandekar
On Tue, 9 Oct 2018 at 20:46, Amit Khandekar wrote: > There is still one more regression test failure in polygon.sql which I > am yet to analyze. Below is a narrowed down testcase which reproduces the failure in polygon.sql : create table tab (n int, dist int, id integer); insert into tab values

Re: WIP: Avoid creation of the free space map for small tables

2018-10-16 Thread John Naylor
On 10/15/18, Amit Kapila wrote: > I think you can avoid calling RelationGetNumberOfBlocks, if you call > smgrexists before This is done in the attached v5, 0001. > and for the purpose of vacuum, we can get that as an > input parameter. I think one can argue for not changing the interface > func

[PATCH] heap_insert() and heap_update() optimization

2018-10-16 Thread Andrey Klychkov
Hello, hackers I suggest the small attached patch that gives a bit of heap_insert() and heap_update() optimization by reducing calls of BufferGetPage(buffer) into them. I measured call time of these: heap_insert(): avg origin 13394 ns, avg patched 12685 ns; perf increases +5.59% heap_update(): av

Re: Function to promote standby servers

2018-10-16 Thread Laurenz Albe
Ashwin Agrawal wrote: > Just curious to know, is promotion via function only allowed for > hot-standby or works for any standby? Only for hot standby - how do you want to execute a function on a standby server that doesn't allow connections? Yours, Laurenz Albe

Re: pg11rc1 DROP INDEX: NOTICE: drop_trigger

2018-10-16 Thread Sergei Kornilov
Hi Maybe you have some event trigger? Please check pg_event_trigger system view: https://www.postgresql.org/docs/11/static/catalog-pg-event-trigger.html regards, Sergei