Re: Documentation for bootstrap data conversion

2018-04-19 Thread John Naylor
On 4/18/18, Tom Lane wrote: > John Naylor writes: >> and dug through a bit to find cases where 'catalog' is clearly a >> better term. Most of these are in the pg_*.h/.dat file boilerplate >> comments, which would be easy enough to change with a script. If we&#

lingering references to V0 calling convention

2018-04-23 Thread John Naylor
ion of the configure options for --disable-floatN-byval, since this thread proposed something a bit more invasive: https://www.postgresql.org/message-id/flat/10862.1519228208%40sss.pgh.pa.us#10862.1519228...@sss.pgh.pa.us -John Naylor diff --git a/doc/src/sgml/plhandler.sgml b/doc/src/sgml/plhandler.

Re: unused_oids script is broken with bsd sed

2018-04-25 Thread John Naylor
ce they're run at the same time anyway (see catalog/Makefile), and we've already read all the data. -John Naylor

Re: unused_oids script is broken with bsd sed

2018-04-25 Thread John Naylor
t to expose toast and index oids directly rather than in strings formatted specifically for the bki file, as in the attached. Thoughts? -John Naylor diff --git a/src/backend/catalog/Catalog.pm b/src/backend/catalog/Catalog.pm index 6305a2b..601d680 100644 --- a/src/backend/catalog/Catalog.pm +++ b

Re: unused_oids script is broken with bsd sed

2018-04-25 Thread John Naylor
#x27;re in the catalog directory. I can hack on that part tomorrow if no one else has. -John Naylor

Re: unused_oids script is broken with bsd sed

2018-04-26 Thread John Naylor
On 4/26/18, Tom Lane wrote: > John Naylor writes: >> For those following along, these scripts still assume we're in the >> catalog directory. I can hack on that part tomorrow if no one else >> has. > > I didn't touch this point. Patch 0002 is my attempt a

generating catcache control data

2019-10-10 Thread John Naylor
get their attrs from schemapg.h. 3. Simplify cat/syscache.c Is this something worth doing? [1] https://www.postgresql.org/message-id/1295.1507918074%40sss.pgh.pa.us -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: generating catcache control data

2019-10-18 Thread John Naylor
hich amounts to nearly 4% improvement in the first tps number, which isn't earth-shattering, but it's something. Opinions? It wouldn't be a lot of additional work to put together a WIP patch. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: pgsql: Avoid creation of the free space map for small heap relations, t

2019-02-27 Thread John Naylor
; What if get some valid block from the first call to > GetPageWithFreeSpace via local map which has required space? I think > in that case we will need the call at hio.c:682. Am I missing > something? Are you referring to the call at line 393? Then the map will be cleared on line 507 before

Re: pgsql: Avoid creation of the free space map for small heap relations, t

2019-02-27 Thread John Naylor
t;, to "This prevents us from using the map". Perhaps also a comma after "first". -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: WIP: Avoid creation of the free space map for small tables

2019-03-06 Thread John Naylor
to be independent of block-size, was in [2]. [1] https://www.postgresql.org/message-id/CACPNZCu4cOdm3uGnNEGXivy7Gz8UWyQjynDpdkPGabQ18_zK6g%40mail.gmail.com [2] https://www.postgresql.org/message-id/CACPNZCsWa%3Ddd0K%2BFiODwM%3DLEsepAHVJCoSx2Avew%3DxBEX3Ywjw%40mail.gmail.com -- John Naylor

Re: WIP: Avoid creation of the free space map for small tables

2019-03-10 Thread John Naylor
pensated by the fact > that we are now not transferring *fsm files for small relations. How > about constructing a test where all relations are greater than 4 pages > and then try to upgrade them. We can check for a cluster with a > different number of relations say 10K

Re: Why don't we have a small reserved OID range for patch revisions?

2019-03-10 Thread John Naylor
-target-oid 2000 *.dat To prevent a maintenance headache, I didn't copy any of the formatting logic over. You'll also have to run reformat_dat_files.pl afterwards to restore that. It seems to work, but I haven't tested thoroughly. -- John Naylor

Re: Why don't we have a small reserved OID range for patch revisions?

2019-03-12 Thread John Naylor
On Tue, Mar 12, 2019 at 5:36 AM Tom Lane wrote: > This seems committable from my end --- any further comments? I gave it a read and it looks good to me, but I haven't tried to run it. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Re

Re: WIP: Avoid creation of the free space map for small tables

2019-03-13 Thread John Naylor
patch 5.59s 5.64s The variation within the builds is up to +/- 0.2s, so there is no difference, as expected. 1 page: masterpatch 5.62s 4.25s Clearly, linking is much slower than stat. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: WIP: Avoid creation of the free space map for small tables

2019-03-13 Thread John Naylor
t the > patch? If it is only with patch, then we definitely need to > investigate. Only with the patch. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: WIP: Avoid creation of the free space map for small tables

2019-03-13 Thread John Naylor
> [segfault problems] This now seems spurious. I ran make distclean, git pull, reapplied the patch (leaving out the gettimeofday() calls), and now my upgrade perf test works with default compiler settings. Not sure what happened, but hopefully we can move forward. -- John Nay

Re: WIP: Avoid creation of the free space map for small tables

2019-03-14 Thread John Naylor
eader to be consistent with the > nearby code, run pgindent on the code and changed the commit message. Looks good to me, thanks. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

outdated reference to tuple header OIDs

2019-03-14 Thread John Naylor
It seems this is a leftover from commit 578b229718e8. Patch attached. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services remove-ref-to-sys-oid-col.patch Description: Binary data

Re: WIP: Avoid creation of the free space map for small tables

2019-03-15 Thread John Naylor
eries, right? Great, thanks! I'll keep an eye on the buildfarm as well. I just spotted two comments in freespace.c that were true during earlier patch revisions, but are no longer true, so I've attached a fix for those. There are no other patches in the series. -- John Naylor

Re: jsonpath

2019-03-19 Thread John Naylor
s in the committed patch. This is not default, it's chosen for maximum performance at the expense of binary/memory size. That's fine, but with a little effort you can also make the scanner non-backtracking for additional performance, as in the attached. -- John Naylor

Re: jsonpath

2019-03-21 Thread John Naylor
a performance penalty, in which case a comment to describe the additional complexity would be helpful. {unicode} { parseUnicode(yytext, yyleng); } {hex_char} { parseHexChars(yytext, yyleng); } {unicodefail} { yyerror(NULL, "Unicode sequence is invalid"); } {hex_fail} { y

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-30 Thread John Naylor
I believe I found a typo in mcv.c, fix attached. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services mcv-comment-fix.patch Description: Binary data

Re: benchmarking Flex practices

2019-07-24 Thread John Naylor
On Sun, Jul 21, 2019 at 3:14 AM Tom Lane wrote: > > John Naylor writes: > > The pre-existing ecpg var "state_before" was a bit confusing when > > combined with the new var "state_before_quote_stop", and the former is > > also u

Re: [proposal] de-TOAST'ing using a iterator

2019-07-28 Thread John Naylor
ld off on that for now. 5. Would it be a good idea to add tests (not always practical), or more Assert()'s? You probably already know this, but as a reminder it's good to develop with asserts enabled, but never build with them for performance testing. I think that's enough for now. If y

Re: benchmarking Flex practices

2019-08-01 Thread John Naylor
On Mon, Jul 29, 2019 at 10:40 PM Tom Lane wrote: > > John Naylor writes: > > > The lexer returns UCONST from xus and UIDENT from xui. The grammar has > > rules that are effectively: > > > SCONST { do nothing} > > | UCONST { esc char is backslash } > >

Re: [proposal] de-TOAST'ing using a iterator

2019-08-02 Thread John Naylor
On Tue, Jul 30, 2019 at 8:20 PM Binguo Bao wrote: > > John Naylor 于2019年7月29日周一 上午11:49写道: >> >> 1). For every needle comparison, text_position_next_internal() >> calculates how much of the value is needed and passes that to >> detoast_iterate(), which then calcul

Re: Bison state table

2019-08-13 Thread John Naylor
https://www.postgresql.org/message-id/74dd0f55-f3cf-447e-acf2-88c01e42a...@lrde.epita.fr -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services gram-combine.patch Description: Binary data

Re: [proposal] de-TOAST'ing using a iterator

2019-08-13 Thread John Naylor
On Sat, Aug 3, 2019 at 11:11 PM Binguo Bao wrote: > > John Naylor 于2019年8月2日周五 下午3:12写道: >> >> I like the use of a macro here. However, I think we can find a better >> location for the definition. See the header comment of fmgr.h: >> "Definitions for the Pos

Re: [proposal] de-TOAST'ing using a iterator

2019-08-18 Thread John Naylor
om the + * source buffer in the next iteration, see pglz_decompress_iterate(). + */ +#define INVALID_CTRLC 8 I think the macro might be better placed in pg_lzcompress.h, and for consistency used in pglz_decompress(). Then the comment can be shorter and more general. With my additional commen

Re: [proposal] de-TOAST'ing using a iterator

2019-08-21 Thread John Naylor
On Thu, Aug 22, 2019 at 12:10 AM Binguo Bao wrote: > [v9 patch] Thanks, looks good. I'm setting it to ready for committer. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: MCV lists for highly skewed distributions

2018-01-07 Thread John Naylor
ta, ran analyze a few times and eyeballed the results, of which I give a typical example. I used a low stat target to make it easier to see the general picture. Suggestions welcome. -John Naylor test_analyze_highly_skewed_v1.sql Description: application/sql

Re: WIP: a way forward on bootstrap data

2018-01-12 Thread John Naylor
disclose concerns of my own: 1. MSVC dependency tracking is certainly broken until such time as I can shave that yak and test. 2. Keeping the oid symbols with the data entries required some Makefile trickery to make them visible to .c files outside the backend (patch 0007). It builds fine, but the dependency tracking might have bugs. -- [1] https://www.postgresql.org/message-id/15697.1479161432%40sss.pgh.pa.us Thanks, John Naylor

Re: WIP: a way forward on bootstrap data

2018-01-13 Thread John Naylor
hema will simply not be written to pg_proc.dat, so the old columns are now gone. The result is found in patch 04. -- Note: You could theoretically also load the source data into tables, do the updates with SQL, and dump back out again. I made some progress with this method, but it's not

Re: WIP: a way forward on bootstrap data

2018-01-13 Thread John Naylor
my $rettypeoid = $regtypeoids{ $bki_values{prorettype}}; + $bki_values{prorettype} = $rettypeoid + if defined($rettypeoid); This is all in patch 0011. -John Naylor

Re: WIP: a way forward on bootstrap data

2018-01-14 Thread John Naylor
ut of indexing.h and toasting.h for no good > reason whatever. We'll have quite enough code thrash and pending-patch > breakage from this patch set; we don't need to take on rearrangements that > aren't buying anything. I don't have a convincing rebuttal, so I'll withdraw it. -- [1] https://www.postgresql.org/message-id/25254.1248533810%40sss.pgh.pa.us -John Naylor

Re: WIP: a way forward on bootstrap data

2018-01-15 Thread John Naylor
ids.h into a static file that just #includes catalog/pg_proc_d.h. Thoughts? And I'm curious, what is "_d" intended to convey? (While I'm thinking outloud, I'm beginning to think that these headers lie outside the scope of genbki.pl, and belong in a separate script.) -John Naylor

Re: MCV lists for highly skewed distributions

2018-01-19 Thread John Naylor
sometimes, whether for highly uniform or highly non-uniform distributions. [1] https://www.postgresql.org/message-id/flat/32261.1496611829%40sss.pgh.pa.us#32261.1496611...@sss.pgh.pa.us -John Naylor

Re: MCV lists for highly skewed distributions

2018-01-19 Thread John Naylor
class size in the sample, as denoted by gamma in the Haas and Stokes paper on page 7. -John Naylor

Re: MCV lists for highly skewed distributions

2018-01-20 Thread John Naylor
. > [1] > https://www.postgresql.org/message-id/flat/32261.1496611829%40sss.pgh.pa.us#32261.1496611...@sss.pgh.pa.us -John Naylor

stricter MCV tests for uniform distributions (was Re: MCV lists for highly skewed distributions)

2018-01-22 Thread John Naylor
(Starting a new thread so as not to distract review) On 1/21/18, Dean Rasheed wrote: > On 21 January 2018 at 07:26, John Naylor wrote: >> I spent a few hours hacking on this, and it turns out calculating the >> right number of MCVs taking into account both uniform and highly

Re: MCV lists for highly skewed distributions

2018-02-04 Thread John Naylor
on real data, is still needed of course, but I think this is definitely in the right direction. -John Naylor test_mcvstats_v1.sql Description: application/sql

Re: Copyright information in source files

2019-11-23 Thread John Naylor
h is one less thing to update and one less cause of noise in the change log for rarely-changed files. Is there actually a good reason to update the year? -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: benchmarking Flex practices

2019-11-26 Thread John Naylor
could be made to do something smarter with more work, but > it's not clear to me that it's worth the trouble.) Hmm, I thought we only allowed Unicode escapes in the first place if the server encoding was UTF-8. Or did you mean something else? > If this seems like a reasonable approach

Re: benchmarking Flex practices

2019-12-03 Thread John Naylor
cleaner this way, and possibly faster. I also used this to reunite the xcc and xcsql states. This whole part could be split out into a separate refactoring patch to be applied first, if desired. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services v10-handle-uescapes-in-parser.patch Description: Binary data

Re: Add support for automatically updating Unicode derived files

2019-12-19 Thread John Naylor
unicode_combining_table.pl, do not edit */\n\n"; I would print out the full boilerplate like for other generated headers. Lastly, src/common/unicode/README is outdated (and possibly no longer useful at all?). -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

reduce size of fmgr_builtins array

2019-12-23 Thread John Naylor
ce in the binary over using string pointers. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services v1-fmgr-func-names.patch Description: Binary data

use CLZ instruction in AllocSetFreeIndex()

2019-12-26 Thread John Naylor
also. This is done in the attached patch. [1] https://www.postgresql.org/message-id/407d949e0907201811i13c73e18x58295566d27aadcc%40mail.gmail.com -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 0001-Use-the

Re: use CLZ instruction in AllocSetFreeIndex()

2019-12-27 Thread John Naylor
constant-time, it doesn't with CLZ. That makes sense -- I'll look into doing that. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: use CLZ instruction in AllocSetFreeIndex()

2019-12-27 Thread John Naylor
On Fri, Dec 27, 2019 at 11:05 AM Tom Lane wrote: > > John Naylor writes: > > On Fri, Dec 27, 2019 at 9:54 AM Tom Lane wrote: > >> ... but couldn't the > >> right shift be elided in favor of changing the constant we > >> subtract clz's result fr

Re: use CLZ instruction in AllocSetFreeIndex()

2019-12-27 Thread John Naylor
On Fri, Dec 27, 2019 at 9:16 PM David Fetter wrote: > On Fri, Dec 27, 2019 at 07:02:02PM -0500, John Naylor wrote: > > The lookup table case is less clear. Removing the shift results in > > assembly that looks more like the C code and is slower for me. The > > standard lookup

Re: use CLZ instruction in AllocSetFreeIndex()

2019-12-28 Thread John Naylor
v2 had an Assert that was only correct while experimenting with eliding right shift. Fixed in v3. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services v3-0001-Use-the-CLZ-instruction-in-AllocSetFreeIndex.p

remove separate postgres.(sh)description files

2019-12-30 Thread John Naylor
d then let the generic code path output it into postgres.bki. The attached patch does that and simplifies the catalog makefile and initdb.c. I'll add a commitfest entry for this. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Trai

Re: reduce size of fmgr_builtins array

2020-01-01 Thread John Naylor
istic workload, but reducing the array size by 1/3 couldn't hurt. I'll go ahead and add this to the commitfest. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services drive_builtin_by_name.c Description: Binary data

Re: benchmarking Flex practices

2020-01-02 Thread John Naylor
unicode1.33s1.34s uescape 1.44s1.58s -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Add support for automatically updating Unicode derived files

2020-01-03 Thread John Naylor
On Thu, Dec 26, 2019 at 12:39 PM Peter Eisentraut wrote: > > On 2019-12-19 23:48, John Naylor wrote: > > I would print out the full boilerplate like for other generated headers. > > Hmm, you are probably comparing with > src/common/unicode/generate-unicode_norm_tab

Re: [proposal] de-TOAST'ing using a iterator

2020-01-11 Thread John Naylor
so that text_position_next_internal() didn't have to call the iterator function every time the needle advances, which caused a noticeable performance penalty. The toast code has moved around quite a bit since then, and I'm not sure of the best way forward. Also, c60e520f6e0 changed the st

Re: benchmarking Flex practices

2020-01-13 Thread John Naylor
us whitespace changes -revised scan.l comment about the performance benefits of no backtracking -split the ECPG C-comment scanning cleanup into a separate patch, as I did for v6. I include it here since it's related (merging scanner states), but not relevant to making the core scanner smaller. -w

Re: benchmarking Flex practices

2020-01-13 Thread John Naylor
On Tue, Jan 14, 2020 at 4:12 AM Tom Lane wrote: > > John Naylor writes: > > [ v11 patch ] > > I pushed this with some small cosmetic adjustments. Thanks for your help hacking on the token filter. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Develop

Re: Use compiler intrinsics for bit ops in hash

2020-01-17 Thread John Naylor
rectly tied to anything in this function, or even in the function it calls, and could get out of date easily. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: reduce size of fmgr_builtins array

2020-01-17 Thread John Naylor
would be worth the effort involved. I've marked it rejected. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: remove separate postgres.(sh)description files

2020-01-19 Thread John Naylor
1 if exist > >> src\backend\catalog\postgres.shdescription del /q > >> src\backend\catalog\postgres.shdescription > > Pushed, thanks! Thanks for taking care of those loose ends -- that was a bit sloppy of me. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-04-16 Thread John Naylor
elations and > reopens it from scratch. Shouldn't we at the very least check > smgr_fsm_nblocks beforehand, so this is only done once? Hmm, I can look into that. On Wed, Apr 17, 2019 at 3:16 AM Andres Freund wrote: > > Hi, > > On 2019-04-16 14:31:25 -0400, Tom Lane wrote: > > and in any case it seems pretty inefficient for workloads that insert > > into multiple tables. > > As is, it's inefficient for insertions into a *single* relation. The > RelationGetTargetBlock() makes it not crazily expensive, but it's still > plenty expensive. Performance testing didn't reveal any performance regression. If you have a realistic benchmark in mind that stresses this logic more heavily, I'd be happy to be convinced otherwise. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: jsonpath

2019-04-17 Thread John Naylor
, while I have the thought in my head, for v13 we should consider replacing the keyword binary search with the perfect hash technique added in c64d0cd5ce2 -- it might give a small performance boost to the scanner. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7

Re: jsonpath

2019-04-17 Thread John Naylor
On Thu, Apr 18, 2019 at 1:43 AM Tom Lane wrote: > > John Naylor writes: > > Attached is a patch to fix some minor issues: > > -misspelling of an error message > > Yeah, I'd noticed that one too :-(. I think the whole jsonpath patch > needs a sweep to bring its e

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-04-18 Thread John Naylor
hreshold to reduce wasted space) 3. reduce calls to smgr_exists() Thoughts? -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-04-19 Thread John Naylor
On Fri, Apr 19, 2019 at 10:38 AM Amit Kapila wrote: > > On Thu, Apr 18, 2019 at 2:10 PM John Naylor > wrote: > > Agreed. I suspect the most realistic way to address most of the > > objections in a short amount of time would be to: > > > > 1. rip out the local

Re: jsonpath

2019-04-21 Thread John Naylor
On Sun, Apr 21, 2019 at 2:01 AM Alexander Korotkov wrote: > > On Thu, Apr 18, 2019 at 4:09 AM John Naylor > wrote: > > I was wondering about that. I measured the current size of > > yy_transition to be 36492 on my machine. With the flag -Cfe, which > > gives the smal

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-04-24 Thread John Naylor
e to store the actual free space in the same format as the FSM. That might be v13 material, but in any case, I'll hold off on A for now. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services optimize-smgrexists.patch Description: Binary data

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-04-25 Thread John Naylor
On Wed, Apr 24, 2019 at 1:58 PM Amit Kapila wrote: > Sorry for not noticing earlier, but this patch causes a regression test failure for me (attached) -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-04-25 Thread John Naylor
n, say, block 2 won't actually create the FSM on the standby. When block 5 fills up enough, then the xlog call will cause the FSM to be created. Not sure if this is best, but it saves another syscall, and this function is only called when freespace is less than 20%, IIRC. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services untie-local-map-from-heap_v1.patch Description: Binary data

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-04-25 Thread John Naylor
On Fri, Apr 26, 2019 at 11:52 AM Amit Kapila wrote: > > On Thu, Apr 25, 2019 at 12:39 PM John Naylor > wrote: > > > > On Wed, Apr 24, 2019 at 1:58 PM Amit Kapila wrote: > > > > > > > Sorry for not noticing earlier, but this patch causes a re

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-04-29 Thread John Naylor
on to notify the existence of > FSM, this can be done both by vacuum and backend. I still don't claim to be anything but naive in this area, but does the attached get us any closer? -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remot

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-04-30 Thread John Naylor
, as discussed above if we issue invalidation > call (in RecordPageWithFreeSpace) when the page becomes empty, then we > shouldn't encounter this. John, can we try this out and see if the > failure goes away? I added a clear/inval call in RecordPageWithFreeSpace and the failure goe

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-04-30 Thread John Naylor
On Tue, Apr 30, 2019 at 12:48 PM Amit Kapila wrote: > > On Fri, Apr 26, 2019 at 10:46 AM John Naylor > wrote: > > > > On Wed, Apr 17, 2019 at 2:04 AM Andres Freund wrote: > > > > > > Hi, > > > > > > I'm somewhat unhappy in how much

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-04-30 Thread John Naylor
above emails, I am also bit > scared that it should not lead to many invalidation messages for small > relations, so may be we should send the invalidation message only when > the entire page is empty. One way would be to send the inval if the new free space is greater than some pe

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-05-01 Thread John Naylor
tly as mature as the one in HEAD. It's not ready. Not wanting to attempt Promise Driven Development, I'd rather revert, and only try again if there's enough time and interest. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-05-02 Thread John Naylor
ion call. Isn't it better to directly call > CacheInvalidateRelcache here to notify other backends that their local > maps are invalid now? Yes, you're quite correct. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-05-02 Thread John Naylor
On Thu, May 2, 2019 at 4:57 PM Amit Kapila wrote: > > On Thu, May 2, 2019 at 12:39 PM John Naylor > wrote: > > > Okay, I have updated the patch to incorporate your changes and call > relcache invalidation at required places. I have updated comments at a > few places as

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-05-04 Thread John Naylor
On Sat, May 4, 2019 at 5:25 PM Amit Kapila wrote: > Attached is a revert patch. John, can you please once double-check to > ensure I have not missed anything? Looks complete to me. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remo

copy-past-o comment in lock.h

2019-05-07 Thread John Naylor
Attached is an attempt to match surrounding code. More broadly, though, it seems the "ID info" comments belong with the SET_LOCKTAG_* macros rather than with the LockTagType enum members. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support,

Re: copy-past-o comment in lock.h

2019-05-07 Thread John Naylor
On Tue, May 7, 2019 at 4:00 PM Michael Paquier wrote: > > On Tue, May 07, 2019 at 03:41:50PM +0800, John Naylor wrote: > > Attached is an attempt to match surrounding code. More broadly, > > though, it seems the "ID info" comments belong with the SET_LOCKTAG_* >

Re: copy-past-o comment in lock.h

2019-05-08 Thread John Naylor
On Wed, May 8, 2019 at 3:10 PM Michael Paquier wrote: > > On Tue, May 07, 2019 at 04:12:31PM +0800, John Naylor wrote: > > That's probably better. > > Would you like to send an updated patch? Perhaps you have a better > idea? > -- > Michael In the attached, I

WIP: Avoid creation of the free space map for small tables

2018-10-05 Thread John Naylor
l.com [2] https://www.postgresql.org/message-id/11360.1345502641%40sss.pgh.pa.us -- I'll add this to the November commitfest. -John Naylor fsmtest.sql Description: application/sql From 77c85f633f915bd247c554b691a134fac1f32316 Mon Sep 17 00:00:00 2001 From: John Naylor Date: Sat, 6 Oct 2018 00:

Re: WIP: Avoid creation of the free space map for small tables

2018-10-07 Thread John Naylor
On 10/6/18, Thomas Munro wrote: > On Sat, Oct 6, 2018 at 7:47 AM John Naylor wrote: >> A while back, Robert Haas noticed that the space taken up by very >> small tables is dominated by the FSM [1]. Tom suggested that we could >> prevent creation of the FSM until the heap

Re: WIP: Avoid creation of the free space map for small tables

2018-10-08 Thread John Naylor
On 10/7/18, Tom Lane wrote: > John Naylor writes: >> On 10/6/18, Thomas Munro wrote: >>> On Sat, Oct 6, 2018 at 7:47 AM John Naylor wrote: >>>> A while back, Robert Haas noticed that the space taken up by very >>>> small tables is dominated b

Re: WIP: Avoid creation of the free space map for small tables

2018-10-13 Thread John Naylor
On 10/13/18, Amit Kapila wrote: > On Sat, Oct 6, 2018 at 12:17 AM John Naylor wrote: >> -For normal mode, I taught fsm_set_and_search() to switch to a >> non-extending buffer call, but the biggest missing piece is WAL >> replay. >> > > fsm_set_and_search() &

Re: inconsistency and inefficiency in setup_conversion()

2018-10-13 Thread John Naylor
Attached is v6, a simple rebase. -John Naylor From e37cb80ab8e7baaa5231fc3b8dbc9d96ec253018 Mon Sep 17 00:00:00 2001 From: John Naylor Date: Sat, 13 Oct 2018 19:28:08 +0700 Subject: [PATCH v6 1/2] Add pg_language lookup. This didn't seem worth doing before, but an upcoming commit will a

Re: WIP: Avoid creation of the free space map for small tables

2018-10-14 Thread John Naylor
a relfilenode and not a relation, I had to reimplement that separately, but the logic is basically the same. It works under streaming replication. I've also attached a couple SQL scripts which, when the aforementioned DEBUG1 calls are enabled, show what the heap insert code i

Re: WIP: Avoid creation of the free space map for small tables

2018-10-15 Thread John Naylor
we have already, much less all of them: /* * Check if some other backend has extended a block for us while * we were waiting on the lock. */ targetBlock = GetPageWithFreeSpace(relation, len + saveFreeSpace); By the time we get to this call, we likely wouldn't trigger the logic to try every block, but I don't think we can guarantee that. We could add a boolean parameter that means "consider trying every block", but I don't think the FSM code should have so much state passed to it. Thanks for reviewing, -John Naylor

Re: WIP: Avoid creation of the free space map for small tables

2018-10-16 Thread John Naylor
CUUM callers. > [proposal for a cache of blocks to try] That's interesting. I'll have to do some reading elsewhere in the codebase, and then I'll follow up. Thanks, -John Naylor From 0dafd220e437ed1cf9ed77e41cf3090365a63798 Mon Sep 17 00:00:00 2001 From: John Naylor Date: Tue, 16

ERROR's turning FATAL in BRIN regression tests

2018-10-19 Thread John Naylor
ver closed the connection unexpectedly + This probably means the server terminated abnormally + before or while processing the request. + connection to server was lost I'm thoroughly stumped -- anyone have an idea where to look next? Thanks, -John Naylor

Re: WIP: Avoid creation of the free space map for small tables

2018-10-21 Thread John Naylor
27;ll redo the logic to extend the relation. If we're about to successfully return a buffer, whether from an existing block, or by extension, we clear the local map. Once this is in shape, I'll do some performance testing. -John Naylor From 529fa1f57946d70736b2304c2883213e45f7c077 Mon Se

Re: WIP: Avoid creation of the free space map for small tables

2018-10-23 Thread John Naylor
nce not having 3 blocks of FSM in shared buffers exactly makes up for the 3 other blocks of heap that are checked when free space runs out. I can run additional tests if there's interest. -John Naylor fsm-copy-test.sql Description: application/sql

Re: WIP: Avoid creation of the free space map for small tables

2018-10-31 Thread John Naylor
they have no data at all. This isn't anything to get excited about, but it would be nice to investigate, at least so it can be documented. A cursory dig hasn't found the cause, but I'll keep doing that as time permits. -John Naylor From 428b56e12e3f5d2bd8fc81c9fb5fe7169e9da580 Mon Sep

Re: PostgreSQL Limits and lack of documentation about them.

2018-10-31 Thread John Naylor
of toasted columns per tuple). + Maximum Identifier Length + 63 characters + Can this be increased with recompiling, if not conveniently? + Maximum Indexed Columns + 32 + Can be increased by recompiling PostgreSQL How about the max number of included columns in a covering index? > I'm not so sure about detailing limits of GUCs since the limits of > those are mentioned in pg_settings. Maybe we could just have a link to that section in the docs. -- -John Naylor

Re: PostgreSQL Limits and lack of documentation about them.

2018-10-31 Thread John Naylor
x27;s header + toast table oid + chunk_id + logical size + compressed size. This is one area where visual diagrams would be nice. -John Naylor

Re: [HACKERS] generated columns

2018-11-01 Thread John Naylor
lumn: 'c' + 's'/'v' This way, we'd still end up with 1 fewer column (2 new ones minus atthasdef, attidentity, and atthasmissing). A bit crazier, what if "d = dropped" was another allowed value in attmaterialized -- we could then get rid of attisdropped as well. That has obvious disadvantages, but the broader idea is that this design may have use cases we haven't thought of yet. Thoughts? -John Naylor

Re: WIP: Avoid creation of the free space map for small tables

2018-11-02 Thread John Naylor
l I had to do was compile with a threshold much larger than required, and then test inserting into X number of pages, to simulate a threshold of X. I increased X until I saw a regression. That's where the 32 came from, sorry if that was misleading, in my head it was obvious. I'd be happy test other scenarios. I'm not sure how to test redo -- seems more difficult to get meaningful results than the normal case. -John Naylor

Re: WIP: Avoid creation of the free space map for small tables

2018-11-04 Thread John Naylor
d in a fairly different way. Were you referring to performance or just functionality? Because the threshold works on the standby, but I don't know about the performance there. -John Naylor fsm-copy-test-v2.sql Description: application/sql

<    1   2   3   4   5   6   7   8   9   10   >