Re: Amcheck: do rightlink verification with lock coupling

2020-01-13 Thread Tomas Vondra
On Mon, Jan 13, 2020 at 03:49:40PM -0800, Peter Geoghegan wrote: On Sat, Jan 11, 2020 at 4:25 AM Tomas Vondra wrote: Understood. Is that a reason to not commit of this patch now, though? It could use some polishing. Are you interested in committing it? Not really - as a CFM I was trying

Re: planner support functions: handle GROUP BY estimates ?

2020-01-14 Thread Tomas Vondra
7;t store stuff in pg_statistic without having a relation (i.e. table / index / ...) but maybe we could invent a new relation type for this purpose. Of course, it'd require some catalog work to represent this ... Ultimately I think we'd want both things, it's not one or the other. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: planner support functions: handle GROUP BY estimates ?

2020-01-14 Thread Tomas Vondra
On Tue, Jan 14, 2020 at 04:21:57PM -0500, Tom Lane wrote: Tomas Vondra writes: On Tue, Jan 14, 2020 at 03:12:21PM -0500, Tom Lane wrote: cc'ing Tomas in case he has any thoughts about it. Well, I certainly do thoughts about this - it's pretty much exactly what I proposed yesterd

Re: planner support functions: handle GROUP BY estimates ?

2020-01-14 Thread Tomas Vondra
On Tue, Jan 14, 2020 at 04:52:44PM -0500, Tom Lane wrote: Tomas Vondra writes: On Tue, Jan 14, 2020 at 04:21:57PM -0500, Tom Lane wrote: Yeah, it seems likely to me that the infrastructure for this would be somewhat different --- the user-facing syntax could be basically the same, but

Re: planner support functions: handle GROUP BY estimates ?

2020-01-14 Thread Tomas Vondra
On Tue, Jan 14, 2020 at 05:37:53PM -0500, Tom Lane wrote: Tomas Vondra writes: On Tue, Jan 14, 2020 at 04:52:44PM -0500, Tom Lane wrote: The main issue for sticking the results into pg_statistic is that the primary key there is (starelid, staattnum), and we haven't got a suitable attnu

Re: Complete data erasure

2020-01-15 Thread Tomas Vondra
what I'd prefer is encryption + rotation of the keys. Which should work properly even on COW filesystems, the performance impact is kinda low and amortized etc. Of course, we're discussing built-in encryption for quite a bit of time. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-16 Thread Tomas Vondra
27;t see why not to just do what Tom proposed, i.e. allocate this as part of the memory context in SlabContextCreate(), when memory context checking is enabled. It seems much more convenient / simpler than the globals (no repalloc, ...). regards -- Tomas Vondra http://www.2ndQ

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-16 Thread Tomas Vondra
cause we want to allocate the space as part of the context header, but nothin too bad. We might optimize it a bit by using a regular bitmap (instead of just an array of bools), but I haven't done that. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Develop

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-16 Thread Tomas Vondra
On Thu, Jan 16, 2020 at 08:48:49AM -0800, Andres Freund wrote: Hi, On 2020-01-16 17:25:00 +0100, Tomas Vondra wrote: On Thu, Jan 16, 2020 at 10:27:01AM -0500, Tom Lane wrote: > Andres Freund writes: > > ... I thought you were asking whether > > any additional memory could

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-16 Thread Tomas Vondra
On Thu, Jan 16, 2020 at 11:43:34AM -0500, Tom Lane wrote: Tomas Vondra writes: The attached fix should do the trick - it pre-allocates the space when creating the context. There is a bit of complexity because we want to allocate the space as part of the context header, but nothin too bad. We

Re: psql - add SHOW_ALL_RESULTS option

2020-01-16 Thread Tomas Vondra
Hi, This is one of the patches already marked as RFC (since September by Alvaro). Anyone interested in actually pushing it, so that it does not fall through to yet another commitfest? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-16 Thread Tomas Vondra
On Thu, Jan 16, 2020 at 12:33:03PM -0500, Tom Lane wrote: Tomas Vondra writes: On Thu, Jan 16, 2020 at 08:48:49AM -0800, Andres Freund wrote: I don't get why it's advantageous to allocate this once for each slab, rather than having it as a global once for all slabs. But anyway, sti

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-16 Thread Tomas Vondra
On Thu, Jan 16, 2020 at 03:15:41PM -0500, Tom Lane wrote: Tomas Vondra writes: I think the one possible argument against this approach might be that it adds a field to the struct, so if you have an extension using a Slab context, it'll break if you don't rebuild it. But that only mat

Re: psql - add SHOW_ALL_RESULTS option

2020-01-16 Thread Tomas Vondra
On Thu, Jan 16, 2020 at 01:08:16PM -0500, Tom Lane wrote: Tomas Vondra writes: This is one of the patches already marked as RFC (since September by Alvaro). Anyone interested in actually pushing it, so that it does not fall through to yet another commitfest? TBH, I think we'd be bette

Re: FETCH FIRST clause PERCENT option

2020-01-16 Thread Tomas Vondra
quot; - at this point we need to decide which of the approaches is the right one. regards [1] https://www.postgresql.org/message-id/flat/87o8wvz253@news-spur.riddles.org.uk -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Recognizing superuser in pg_hba.conf

2020-01-16 Thread Tomas Vondra
Hi, I see this patch is marked as RFC since 12/30, but there seems to be quite a lot of discussion about the syntax, keywords and how exactly to identify the superuser. So I'll switch it back to needs review, which I think is a better representation of the current state. regards --

Re: remove separate postgres.(sh)description files

2020-01-16 Thread Tomas Vondra
/q src\backend\catalog\postgres.shdescription Barring objections, I'll remove those too, and commit this. +1 from me. Let's remove these small RFC patches out of the way. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Supp

Re: Run-time pruning for ModifyTable

2020-01-16 Thread Tomas Vondra
is patch is marked as "waiting on author" since 11/27, and there have been no updates or responses since then. Do you plan to submit a new patch version in this CF? We're already half-way through, so there's not much time ... regards -- Tomas Vondra http://ww

Re: row filtering for logical replication

2020-01-16 Thread Tomas Vondra
. Euler, this thread is waiting for input from you regarding the latest comments from Amit. Euler, this patch is still in "waiting on author" since 11/25. Do you plan to review changes made by Amit in the patches he submitted, or what are your plans with this patch? regards -- To

Re: [PATCH] ltree, lquery, and ltxtquery binary protocol support

2020-01-16 Thread Tomas Vondra
mplex, all I know is cputube can't apply it. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Improve search for missing parent downlinks in amcheck

2020-01-16 Thread Tomas Vondra
provide a fixed patch? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-17 Thread Tomas Vondra
On Thu, Jan 16, 2020 at 06:04:32PM +0100, Tomas Vondra wrote: On Thu, Jan 16, 2020 at 11:43:34AM -0500, Tom Lane wrote: Tomas Vondra writes: The attached fix should do the trick - it pre-allocates the space when creating the context. There is a bit of complexity because we want to allocate

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-17 Thread Tomas Vondra
On Thu, Jan 16, 2020 at 01:41:39PM -0500, Tom Lane wrote: Andres Freund writes: On 2020-01-16 18:01:53 +0100, Tomas Vondra wrote: and it's only really used in debug builds anyway. So I'm not all that woried about this wasting a couple extra kB of memory. IDK, making memory

Re: jsonb_set() strictness considered harmful to data

2020-01-17 Thread Tomas Vondra
warnings check-world passed without any problems I'll mark this patch as ready for commiter Thank you for your work Thanks for the review. I propose to commit this shortly. Now that this was committed, I've updated the patch status accordingly. Thanks! -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

SLRU statistics

2020-01-19 Thread Tomas Vondra
the SLRU stats, similarly to how we can reset bgwriter stats. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c index f8e7670f8d

Re: SLRU statistics

2020-01-20 Thread Tomas Vondra
On Mon, Jan 20, 2020 at 01:04:33AM +, tsunakawa.ta...@fujitsu.com wrote: From: Tomas Vondra One of the stats I occasionally wanted to know are stats for the SLRU stats (we have couple of those - clog, subtrans, ...). So here is a WIP version of a patch adding that. How can users take

Re: SLRU statistics

2020-01-20 Thread Tomas Vondra
On Mon, Jan 20, 2020 at 03:01:36PM -0300, Alvaro Herrera wrote: On 2020-Jan-20, Tomas Vondra wrote: On Mon, Jan 20, 2020 at 01:04:33AM +, tsunakawa.ta...@fujitsu.com wrote: > From: Tomas Vondra > > One of the stats I occasionally wanted to know are stats for the SLRU > >

Re: SLRU statistics

2020-01-21 Thread Tomas Vondra
On Tue, Jan 21, 2020 at 05:09:33PM +0900, Masahiko Sawada wrote: On Tue, 21 Jan 2020 at 01:38, Tomas Vondra wrote: On Mon, Jan 20, 2020 at 01:04:33AM +, tsunakawa.ta...@fujitsu.com wrote: >From: Tomas Vondra >> One of the stats I occasionally wanted to know are stats for

Re: SLRU statistics

2020-01-21 Thread Tomas Vondra
On Tue, Jan 21, 2020 at 06:24:29AM +, tsunakawa.ta...@fujitsu.com wrote: From: Tomas Vondra You're right the users can't really take advantage of this - my primary motivation was providing a feedback for devs, benchmarking etc. That might have been done with DEBUG messages or

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-01-21 Thread Tomas Vondra
is patch before 2020-03, with the hope it can still make it into 13. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: allow online change primary_conninfo

2020-01-21 Thread Tomas Vondra
s of this patch? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

We're getting close to the end of 2020-01 CF

2020-01-21 Thread Tomas Vondra
ot;needs review" state too, although it's less clear who to ping for those. Maybe if one of your patches is waiting for a review, try pinging people who already did a review in the past. sincerely, your commitfest dictator -- Tomas Vondra http://www.2ndQuadrant.com Pos

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-01-21 Thread Tomas Vondra
On Tue, Jan 21, 2020 at 09:37:01AM -0500, James Coleman wrote: On Tue, Jan 21, 2020 at 9:25 AM Tomas Vondra wrote: Hi, This patch has been marked as WoA since end of November, and there has been no discussion/reviews since then :-( Based on off-list discussion with James I don't think t

Re: We're getting close to the end of 2020-01 CF

2020-01-21 Thread Tomas Vondra
On Tue, Jan 21, 2020 at 12:25:14PM -0300, Alvaro Herrera wrote: On 2020-Jan-21, Tomas Vondra wrote: About half of the WoA patches are inactive for a long time, i.e. have been marked like that before 2020-01 (and sometimes long before that) and there have been no substantive updates. The chance

Re: Protect syscache from bloating with negative cache entries

2020-01-21 Thread Tomas Vondra
t the appveyor build (it seems to be about jsonb_set_lax), but on travis it fails like this: catcache.c:820:1: error: no previous prototype for ‘CatalogCacheFlushCatalog2’ [-Werror=missing-prototypes] so I'll leave it in WoA for now. regards -- Tomas Vondra http://www.2ndQu

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-01-21 Thread Tomas Vondra
review" - if that's not the right status, let me know. Also, the patch was moved forward mostly by Amit and Dilip, so I've added them as authors in the CF app (well, what matters is the commit message, of course, but let's keep this up to date too). regards -- Tomas Vondra

Re: proposal: schema variables

2020-01-21 Thread Tomas Vondra
DO_TABLE: case DO_PROCLANG: 14) namespace.c defines VariableIsVisible twice: extern bool VariableIsVisible(Oid relid); ... extern bool VariableIsVisible(Oid varid); 15) I'd say lookup_variable and identify_variable should use camelcase just like the other functions in the same file. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: We're getting close to the end of 2020-01 CF

2020-01-22 Thread Tomas Vondra
On Wed, Jan 22, 2020 at 02:09:39PM +0900, Michael Paquier wrote: On Tue, Jan 21, 2020 at 05:20:17PM +0100, Tomas Vondra wrote: Yeah, you're right returning them with feedback seems more appropriate, given the long inactivity. Plus, the CF app apparently does not allow moving WoA patches t

Re: fix for BUG #3720: wrong results at using ltree

2020-01-24 Thread Tomas Vondra
Unfortunately, the current code is somewhat undercommented :-( Anyway, I don't quite understand why we need these caps. It kinda seems like a band-aid for potential overflow. Why should it be OK for the values to even get past the maximum, with sane input data? And isn't there a

Re: Expose lock group leader pid in pg_stat_activity

2020-01-28 Thread Tomas Vondra
ng the leader_id will change, that, of course. But I think it's still mostly OK, even with the possible inconsistency. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Expose lock group leader pid in pg_stat_activity

2020-01-28 Thread Tomas Vondra
On Tue, Jan 28, 2020 at 02:26:34PM +0100, Julien Rouhaud wrote: On Tue, Jan 28, 2020 at 2:09 PM Tomas Vondra wrote: I agree a separate "leader_id" column is easier to work with, as it does not require unnesting and so on. As for the consistency, I agree we probably can't ma

Re: Is custom MemoryContext prohibited?

2020-01-28 Thread Tomas Vondra
ntexts are part of the same hierarchy as "normal" contexts? That would be a bit confusing, I think. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Is custom MemoryContext prohibited?

2020-01-28 Thread Tomas Vondra
On Tue, Jan 28, 2020 at 11:32:49PM +0900, Kohei KaiGai wrote: 2020年1月28日(火) 23:09 Tomas Vondra : On Tue, Jan 28, 2020 at 10:55:11AM +0900, Kohei KaiGai wrote: >Hello, > >I noticed MemoryContextIsValid() called by various kinds of memory context >routines checks its node-ta

Re: Complete data erasure

2020-01-28 Thread Tomas Vondra
sort of queue. And then a background worker would "erase" these relfilenodes outside the COMMIT. And yes, we need to do this in a way that works with replicas, i.e. we need to WAL-log it somehow. And it should to be done in a way that works when the replica is on a different type of filesystem. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: FETCH FIRST clause WITH TIES option

2019-04-03 Thread Tomas Vondra
pSize, count_est + (...)); } ... } Firstly, this seriously needs some comment explaining why we do this. But more importantly - shouldn't it really be count_rows = Max(avgGroupSize, count_rows + (...)); instead of using count_est again (which might easily be -1 anyway)? reg

Re: FETCH FIRST clause WITH TIES option

2019-04-03 Thread Tomas Vondra
On Wed, Apr 03, 2019 at 03:08:05PM -0400, Tom Lane wrote: Tomas Vondra writes: I've tried to fix the merge conflict (essentially by moving some of the code to adjust_limit_rows_costs(), but I'm wondering if the code added to create_limit_path is actually correct ... Firstly, this

Re: explain plans with information about (modified) gucs

2019-04-03 Thread Tomas Vondra
Hi, I've committed this, with some minor documentation tweaks. I've also fixed a minor bug in the last patch, where the group with settings was not properly labeled in some formats (e.g. json). Thanks to all the reviewers! regards -- Tomas Vondra http://www.2ndQu

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Tomas Vondra
o CopyFrom, considering that's unsupported and so can't be tested etc. It's not a huge amount of code, of course. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: FETCH FIRST clause PERCENT option

2019-04-05 Thread Tomas Vondra
On Fri, Apr 05, 2019 at 03:14:56PM +0300, Surafel Temesgen wrote: On Tue, Mar 26, 2019 at 5:46 PM Tomas Vondra wrote: On Tue, Mar 26, 2019 at 03:06:52PM +0300, Surafel Temesgen wrote: >On Thu, Feb 28, 2019 at 11:16 PM Tomas Vondra < tomas.von...@2ndquadrant.com> >wrote: > >

Re: FETCH FIRST clause WITH TIES option

2019-04-06 Thread Tomas Vondra
s not change anything because limit does not project, but it seems confusing. cheers -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml index

Re: shared-memory based stats collector

2019-04-09 Thread Tomas Vondra
agree with Andres the patch did not quite get to committable state in the last CF. Conidering how sensitive part it touches, I suggest we try to get it committed early in the PG13 cycle. I'm willing to spend some time on doing test/benchmarks and reviewing the code, if needed. re

Re: POC: GROUP BY optimization

2019-04-09 Thread Tomas Vondra
ing those patches, if there is progress. That still holds, if there are new patch versions I'll look at them. cheers -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-04-09 Thread Tomas Vondra
On Tue, Apr 09, 2019 at 11:29:18AM +0900, Michael Paquier wrote: On Sat, Mar 30, 2019 at 09:13:01PM +0100, Tomas Vondra wrote: Hmmm, what's the right status in the CF app when a part of a patch was committed and the rest should be moved to the next CF? Committed, Moved to next CF, or some

Re: jsonpath

2019-04-09 Thread Tomas Vondra
try to disassemble jsonpath_scan.o and see whether content of yyparse2 influences assembly of other functions. Have you tried other compiler version / different optimization level? Or running it under valgrind. Not sure how difficult that is on Windows. regards -- Tomas Vondra

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-04-09 Thread Tomas Vondra
On Tue, Apr 09, 2019 at 12:14:47PM -0400, Alvaro Herrera wrote: On 2019-Mar-27, Tomas Vondra wrote: Attached are the remaining parts of this patch series - the multivariate histograms, and also a new patch tweaking regression tests for the old statistic types (ndistinct, dependencies) to adopt

Re: shared-memory based stats collector

2019-04-10 Thread Tomas Vondra
On Wed, Apr 10, 2019 at 09:39:29AM +0900, Kyotaro HORIGUCHI wrote: At Tue, 9 Apr 2019 17:03:33 +0200, Tomas Vondra wrote in <20190409150333.5iashyjxm5jmraml@development> Unfortunately, now that we're past code freeze it's clear this is a PG12 matter now :-( I personally con

Re: PostgreSQL pollutes the file system

2019-04-13 Thread Tomas Vondra
correct name for initdb? Maybe > pg_createcluster? The Debian packagers already use pg_createcluster for their script which wraps initdb, and while pg_initdb is a bit misleading (it creates a cluster rather than a database) it is not that bad. Andreas regards -- Tomas Vondra htt

Re: Berserk Autovacuum (let's save next Mandrill)

2019-04-13 Thread Tomas Vondra
ure. I find it rather difficult to manage more and more different types of cleanup workers. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Commit message / hash in commitfest page.

2019-04-13 Thread Tomas Vondra
nformation. And I'd probably try searching in git log first anyway. regard -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Should the docs have a warning about pg_stat_reset()?

2019-04-13 Thread Tomas Vondra
y available monitoring tools will do that. If only we had a way to regularly snapshot the data from within the database, and then compute the deltas on that. If only we could insert data from one table into another one a then do some analysics on it, with like small windows moving over the data or

Re: Zedstore - compressed in-core columnar storage

2019-04-14 Thread Tomas Vondra
ikely to happen in the same release, considering how much work it will require. And I have no idea if zedstore or VOPS are / will be the only candidates - it's way too early at this point. FWIW I personally plan to focus primarily on the features that aim to be included in core, and that appli

Re: Zedstore - compressed in-core columnar storage

2019-04-14 Thread Tomas Vondra
d to have AM piece in place, which fits the postgres ecosystem and supports all the features heap does. Not sure it's quite orthogonal. Sure, you can apply it to rowstores too, but I'd say column stores are naturally better suited for it. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Zedstore - compressed in-core columnar storage

2019-04-14 Thread Tomas Vondra
execution directly on the compressed data (like RLE, dictionary and similar approaches). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Zedstore - compressed in-core columnar storage

2019-04-14 Thread Tomas Vondra
ginal length + 1 bit, but it's usually not hard to find a place for one extra bit. Don't we already have that flag, though? I see ZSCompressedBtreeItem has t_flags, and there's ZSBT_COMPRESSED, but maybe it's more complicated. -- Tomas Vondra http://www

Re: Zedstore - compressed in-core columnar storage

2019-04-14 Thread Tomas Vondra
o almost nothing. Implementation == Insert: Inserting a new row, splits the row into datums. Then for first column decide which block to insert the same to, and pick a TID for it, and write undo record for the same. Rest of the columns are inserted using that same

Re: Zedstore - compressed in-core columnar storage

2019-04-14 Thread Tomas Vondra
On Sun, Apr 14, 2019 at 09:45:10AM -0700, Andres Freund wrote: Hi, On 2019-04-14 18:36:18 +0200, Tomas Vondra wrote: I think those comparisons are cute and we did a fair amount of them when considering a drop-in replacement for pglz, but ultimately it might be a bit pointless because: (a) it

Re: Should the docs have a warning about pg_stat_reset()?

2019-04-14 Thread Tomas Vondra
On Sun, Apr 14, 2019 at 09:11:52AM -0500, Joe Conway wrote: On 4/13/19 3:42 PM, Tomas Vondra wrote: If only we had a way to regularly snapshot the data from within the database, and then compute the deltas on that. If only we could insert data from one table into another one a then do some

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Tomas Vondra
On Sun, Apr 14, 2019 at 06:39:47PM +0200, Tomas Vondra wrote: On Thu, Apr 11, 2019 at 06:20:47PM +0300, Heikki Linnakangas wrote: On 11/04/2019 17:54, Tom Lane wrote: Ashwin Agrawal writes: Thank you for trying it out. Yes, noticed for certain patterns pg_lzcompress() actually requires much

Re: Multivariate MCV lists -- pg_mcv_list_items() seems to be broken

2019-04-15 Thread Tomas Vondra
ke a look. There should probably be regression test coverage of that function. Agreed. I plan to rework the existing tests to use the same approach as the MCV, so I'll add a test for this function too. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Develop

Re: Multivariate MCV lists -- pg_mcv_list_items() seems to be broken

2019-04-15 Thread Tomas Vondra
does memcpy(ptr, mcvlist->types, sizeof(Oid) * ndims); while it should be doing memcpy(mcvlist->types, ptr, sizeof(Oid) * ndims); Will fix. cheers -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Tomas Vondra
On Mon, Apr 15, 2019 at 09:29:37AM -0700, Ashwin Agrawal wrote: On Sun, Apr 14, 2019 at 9:40 AM Tomas Vondra wrote: On Thu, Apr 11, 2019 at 06:20:47PM +0300, Heikki Linnakangas wrote: >On 11/04/2019 17:54, Tom Lane wrote: >>Ashwin Agrawal writes: >>>Thank

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Tomas Vondra
On Mon, Apr 15, 2019 at 10:50:21AM -0700, Ashwin Agrawal wrote: On Mon, Apr 15, 2019 at 10:33 AM Tomas Vondra wrote: ... I see. Perhaps it'd be better to call the flag ZSBT_CONTAINER, when it means "this is a container". And then have another flag to track

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Tomas Vondra
On Mon, Apr 15, 2019 at 11:57:49AM -0700, Ashwin Agrawal wrote: On Mon, Apr 15, 2019 at 11:18 AM Tomas Vondra wrote: Maybe. I'm not going to pretend I fully understand the internals. Does that mean the container contains ZSUncompressedBtreeItem as elements? Or just the

Re: Strange coding in mvdistinct.c

2019-04-15 Thread Tomas Vondra
r calculation later in the file that gets it right.) Thanks. I noticed some of the bugs while investigating the recent MCV serialization, and I plan to fix them soon. This week, hopefully. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Sup

Re: Zedstore - compressed in-core columnar storage

2019-04-16 Thread Tomas Vondra
maybe all we'll have to do is define the ranges differently - instead of "number of pages" we may define them as "number of rows" and it might be working. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Strange coding in mvdistinct.c

2019-04-21 Thread Tomas Vondra
to remove the macros, although it's unlikely anyone else uses them. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: finding changed blocks using WAL scanning

2019-04-22 Thread Tomas Vondra
ied in the next few megabytes of WAL. So just doing it once per checkpoint (or even anything above a single WAL segment) and removing all the detailed LSN location makes it useless for this use case. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24

Re: finding changed blocks using WAL scanning

2019-04-22 Thread Tomas Vondra
ing the WAL files in pg_wal and having them streamed and archived gives that flexibility. I agree - this would be quite useful for the prefetching use case I've already mentioned in my previous message. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Dev

Re: finding changed blocks using WAL scanning

2019-04-22 Thread Tomas Vondra
step. Now, maybe the incremental backup use case is so much more important the right thing to do is ignore this other use case, and I'm OK with that - as long as it's a conscious choice. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: finding changed blocks using WAL scanning

2019-04-22 Thread Tomas Vondra
On Mon, Apr 22, 2019 at 07:44:45PM -0400, Bruce Momjian wrote: On Tue, Apr 23, 2019 at 01:21:27AM +0200, Tomas Vondra wrote: On Sat, Apr 20, 2019 at 04:21:52PM -0400, Robert Haas wrote: > On Sat, Apr 20, 2019 at 12:42 AM Stephen Frost wrote: > > > Oh. Well, I already explained

Re: finding changed blocks using WAL scanning

2019-04-23 Thread Tomas Vondra
On Tue, Apr 23, 2019 at 10:22:46AM -0400, Stephen Frost wrote: Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: On Sat, Apr 20, 2019 at 04:21:52PM -0400, Robert Haas wrote: >On Sat, Apr 20, 2019 at 12:42 AM Stephen Frost wrote: >>> Oh. Well, I already explained

Re: finding changed blocks using WAL scanning

2019-04-23 Thread Tomas Vondra
On Tue, Apr 23, 2019 at 11:43:05AM -0400, Stephen Frost wrote: Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: On Tue, Apr 23, 2019 at 10:22:46AM -0400, Stephen Frost wrote: >* Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: >>On Sat, Apr 20, 2019 at 04:21:52PM -040

Re: finding changed blocks using WAL scanning

2019-04-23 Thread Tomas Vondra
On Tue, Apr 23, 2019 at 09:34:54AM -0700, Andres Freund wrote: Hi, On 2019-04-23 18:07:40 +0200, Tomas Vondra wrote: Well, the thing is that for prefetching to be possible you actually have to be a bit behind. Otherwise you can't really look forward which blocks will be needed, right?

Re: finding changed blocks using WAL scanning

2019-04-23 Thread Tomas Vondra
On Tue, Apr 23, 2019 at 10:09:39AM -0700, Andres Freund wrote: Hi, On 2019-04-23 19:01:29 +0200, Tomas Vondra wrote: On Tue, Apr 23, 2019 at 09:34:54AM -0700, Andres Freund wrote: > Hi, > > On 2019-04-23 18:07:40 +0200, Tomas Vondra wrote: > > Well, the thing is that for pr

Re: finding changed blocks using WAL scanning

2019-04-24 Thread Tomas Vondra
e/sort them after we recycle the WAL segment, or something like that. That way the incremental backups don't need to deal with excessive number of modblock files. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Why is it OK for dbsize.c to look at relation files directly?

2019-04-25 Thread Tomas Vondra
is one of the rare examples of a TODO item that is actually doable and not "I have no idea how to do this so I'll stick it into the TODO list". And it seems like a fairly well isolated stuff, so it might be a nice work for someone new. regards -- Tomas Vondra h

Re: TRACE_SORT defined by default

2019-04-25 Thread Tomas Vondra
se it'd be useful to have some actual numbers showing what overhead this actually has, and whether disabling it would make any difference. I can't run anything right away, but I could get us some numbers in a couple of days, assuming there is some agreement on which cases we need to test.

Re: Segfault when restoring -Fd dump on current HEAD

2019-04-26 Thread Tomas Vondra
clean all that now, so +1 from me to apply the patch. +1 from me too -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Data streaming between different databases

2019-04-28 Thread Tomas Vondra
x27;on the basis of equality and differences' and how it all should work. Maybe try explaining it in more detail, with some examples or something. Or if there are other products doing something like that, share a link. regards -- Tomas Vondra http://www.2ndQuadrant.com Postgre

walsender vs. XLogBackgroundFlush during shutdown

2019-04-30 Thread Tomas Vondra
callers anyway, so something like XLogBackgroundFlush(bool) would be simple enough. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: walsender vs. XLogBackgroundFlush during shutdown

2019-05-01 Thread Tomas Vondra
On Wed, May 01, 2019 at 02:13:10PM +0200, Alexander Kukushkin wrote: Hi Tomas, On Wed, 1 May 2019 at 02:28, Tomas Vondra wrote: I see there's an ongoing discussion about race conditions in walreceiver blocking shutdown, so let me start a new thread about a race condition in walsender d

Re: walsender vs. XLogBackgroundFlush during shutdown

2019-05-01 Thread Tomas Vondra
On Wed, May 01, 2019 at 08:53:15AM -0700, Andres Freund wrote: Hi, On 2019-05-01 02:28:45 +0200, Tomas Vondra wrote: The reason for that is pretty simple - the walsenders are doing logical decoding, and during shutdown they're waiting for WalSndCaughtUp=true. But per XLogSendLogical()

Re: walsender vs. XLogBackgroundFlush during shutdown

2019-05-02 Thread Tomas Vondra
On Wed, May 01, 2019 at 07:12:52PM +0200, Alexander Kukushkin wrote: Hi, On Wed, 1 May 2019 at 17:02, Tomas Vondra wrote: OK, so that seems like a separate issue, somewhat unrelated to the issue I reported. And I'm not sure it's a walsender issue - it seems it might be a psycopg

Re: error messages in extended statistics

2019-05-03 Thread Tomas Vondra
igger an error condition, then yes ereport, but if we're reporting situations that couldn't happen without a server bug then elog seems fine. Yeah, I agree. Most of (peshaps all) those errors are internal errors, and thus should be elog. I'll take care of clening this up a bit. r

Re: POC: GROUP BY optimization

2019-05-03 Thread Tomas Vondra
On Fri, May 03, 2019 at 10:28:21PM +0200, Dmitry Dolgov wrote: On Tue, Apr 9, 2019 at 5:21 PM Tomas Vondra wrote: So I personally would suggest to treat those patches as independent until the very last moment, develop the costing improvements needed by each of them, and then decide which of

accounting for memory used for BufFile during hash joins

2019-05-03 Thread Tomas Vondra
https://www.postgresql.org/message-id/flat/bc138e9f-c89e-9147-5395-61d51a757b3b%40gusw.net -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/src/backend/executor/nodeHash.c b/src/backend/executor/nodeHash.c

Re: Logging the feature of SQL-level read/write commits

2019-05-04 Thread Tomas Vondra
Or perhaps something even more complex. And no, I don't think the database can deduce this - those operations often happen in the application code, outside the database. This is why applications (e.g. banking systems) implement this functionality as pretty much new transaction, doing application-specific things. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: error messages in extended statistics

2019-05-04 Thread Tomas Vondra
On Fri, May 03, 2019 at 09:42:17PM +0200, Tomas Vondra wrote: On Fri, May 03, 2019 at 12:21:36PM -0400, Tom Lane wrote: Alvaro Herrera writes: Error reporting in extended statistics is inconsistent -- many messages that are ereport() in mvdistinct.c are elog() in the other modules. ... I

Re: accounting for memory used for BufFile during hash joins

2019-05-06 Thread Tomas Vondra
On Tue, May 07, 2019 at 01:48:40PM +1200, Thomas Munro wrote: On Tue, May 7, 2019 at 9:58 AM Melanie Plageman wrote: On Fri, May 3, 2019 at 5:34 PM Tomas Vondra wrote: The second patch tries to enforce work_mem more strictly. That would be impossible if we were to keep all the BufFile

Re: accounting for memory used for BufFile during hash joins

2019-05-07 Thread Tomas Vondra
On Mon, May 06, 2019 at 11:18:28PM -0400, Tom Lane wrote: Tomas Vondra writes: Do we actually check how many duplicates are there during planning? Certainly that's part of the planner's cost estimates ... but it's only as good as the planner's statistical knowledge.

<    7   8   9   10   11   12   13   14   15   16   >