Re: Built-in connection pooling

2018-01-19 Thread Tomas Vondra
On 01/19/2018 05:17 PM, Konstantin Knizhnik wrote: > > > On 19.01.2018 18:53, Tomas Vondra wrote: >> >> ... >> >> The questions I'm asking myself are mostly these: >> >> 1) When assigning a backend, we first try to get one from a pool, which &

Re: Built-in connection pooling

2018-01-19 Thread Tomas Vondra
it is possible to achieve > millions of TPS. > It is illustrated by the results I have sent in the previous mail: by > spawning 10 instances of pgbouncer I was able to receive 7 times bigger > speed. > AFAICS making pgbouncer multi-threaded would not be hugely complicated. A simp

Re: Built-in connection pooling

2018-01-19 Thread Tomas Vondra
cond (zmq), pgbouncer shouldn't be radically different. > The trouble is pgbouncer is not handling individual packets. It needs to do additional processing to assemble the messages, understand the state of the connection (e.g. to do transaction pooling) etc. Or handle SSL. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Built-in connection pooling

2018-01-19 Thread Tomas Vondra
solution. > Sure, I wasn't really suggesting it's a clear win. I was responding to your argument that pgbouncer in some cases reaches 100% CPU utilization - that can be mitigated to a large extent by adding threads. Of course, the cost for extra level of indirection is not zero. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Built-in connection pooling

2018-01-19 Thread Tomas Vondra
;t have to > proxy all data, it just delegates handling of a command to a backend, > and forgets about that socket. > > Sounds like it could work. > How could it do all that without actually processing all the data? For example, how could it determine the statement/transaction boundaries? -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Built-in connection pooling

2018-01-19 Thread Tomas Vondra
e possible? Perhaps not for prepared statements with simple protocol, but I'm pretty sure it's doable for extended protocol (which seems like a reasonable limitation). That being said, I think it's a mistake to turn this thread into a pgbouncer vs. the world battle. I could name things that are possible only with standalone connection pool - e.g. pausing connections and restarting the database without interrupting the clients. But that does not mean built-in connection pool is not useful. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Built-in connection pooling

2018-01-19 Thread Tomas Vondra
On 01/19/2018 07:35 PM, Claudio Freire wrote: > > > On Fri, Jan 19, 2018 at 2:22 PM, Tomas Vondra > mailto:tomas.von...@2ndquadrant.com>> wrote: > > > > On 01/19/2018 06:13 PM, Claudio Freire wrote: > > > > > > On F

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-19 Thread Tomas Vondra
not interrupted half-way through, leaving some of the catalogs in inconsistent state. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2018-01-19 Thread Tomas Vondra
On 01/19/2018 03:34 PM, Tomas Vondra wrote: > Attached is v5, fixing a silly bug in part 0006, causing segfault when > creating a subscription. > Meh, there was a bug in the sgml docs ( vs. ), causing another failure. Hopefully v6 will pass the CI build, it does pass a build with

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-19 Thread Tomas Vondra
On 01/19/2018 08:33 PM, Robert Haas wrote: > On Fri, Jan 19, 2018 at 2:16 PM, Tomas Vondra > wrote: >> I think an important piece of this puzzle is that we only really care >> about catalog changes made in a transaction that aborts after doing some >> additional changes,

Re: Built-in connection pooling

2018-01-22 Thread Tomas Vondra
On 01/22/2018 05:05 PM, Konstantin Knizhnik wrote: > > > On 19.01.2018 20:28, Tomas Vondra wrote: >>> >>> With pgbouncer you will never be able to use prepared statements which >>> slows down simple queries almost twice (unless my patch with >>> au

Re: [HACKERS] log_destination=file

2018-01-22 Thread Tomas Vondra
r elsewhere, then dup whichever output source is currently > selected onto stderr, then dup the other one if the config is changed > later). > I think the hook system is a really powerful tool, but it seems a bit awkward to force people to use it to improve performance like this ... That seems like something the core should to out of the box. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: WIP: BRIN multi-range indexes

2018-01-23 Thread Tomas Vondra
I'll reorder the patch series along those lines in the next few days. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: copy.c allocation constant

2018-01-24 Thread Tomas Vondra
c/free calls, which may be somewhat expensive). At the glibc level ... I'm not so sure. AFAIK glibc uses an allocator with similar ideas (freelists, ...) so hopefully it's fine too. And then there are the systems without glibc, or with other libc implementations. No idea about those. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [PROPOSAL] Shared Ispell dictionaries

2018-01-24 Thread Tomas Vondra
oc0 - can you >> elaborate a bit why that was necessary? Also, when benchmarking the >> impact of this make sure to measure not only the time but also memory >> consumption. > > It seems to me that there is no need compact_palloc0() anymore. Tests > show that czech dictio

Re: Setting BLCKSZ 4kB

2018-01-26 Thread Tomas Vondra
ources? FWIW even if it's not save in general, it would be useful to understand what are the requirements to make it work. I mean, conditions that need to be met on various levels (sector size of the storage device, page size of of the file system, filesystem alignment, ...). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Setting BLCKSZ 4kB

2018-01-26 Thread Tomas Vondra
On 01/27/2018 12:06 AM, Andres Freund wrote: > Hi, > > On 2018-01-26 23:53:33 +0100, Tomas Vondra wrote: >> But more importantly, I don't see why the size of the internal page >> would matter here at all? SSDs have non-volatile write cache (DRAM with >> batte

Re: Setting BLCKSZ 4kB

2018-01-27 Thread Tomas Vondra
On 01/27/2018 05:01 AM, Bruce Momjian wrote: > On Fri, Jan 26, 2018 at 11:53:33PM +0100, Tomas Vondra wrote: >> >> ... >> >> FWIW even if it's not save in general, it would be useful to >> understand what are the requirements to make it work. I mean, >&g

Re: Write lifetime hints for NVMe

2018-01-27 Thread Tomas Vondra
ns establishing) > ``` > > ``` > with patch > number of transactions actually processed: 521805 > latency average = 2.300 ms > tps = 869.665330 (including connections establishing) > tps = 869.668026 (excluding connections establishing) > ``` > Aren't those numbers f

Re: Write lifetime hints for NVMe

2018-01-27 Thread Tomas Vondra
On 01/27/2018 08:06 PM, Dmitry Dolgov wrote: >> On 27 January 2018 at 16:03, Tomas Vondra >> wrote: >> >> Aren't those numbers far lower that you'd expect from NVMe storage? I do >> have a NVMe drive (Intel 750) in my machine, and I can do thousands o

Re: STATISTICS retained in CREATE TABLE ... LIKE (INCLUDING ALL)?

2018-01-27 Thread Tomas Vondra
; The attached adds this. > Thanks for working on a patch. This should have been in the statistics patch, no doubt about that. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: STATISTICS retained in CREATE TABLE ... LIKE (INCLUDING ALL)?

2018-01-27 Thread Tomas Vondra
W, I'd call it a new feature. > I'm not sure what exactly the feature would be? I mean "keep statistics even if you only ask for indexes" does not seem particularly helpful to me. So I see it more like a bug - I certainly think it should have been handled differently in

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-29 Thread Tomas Vondra
). The cases where this would matter are large ETL jobs, upgrade scripts and so on - these tend to be large and mix DDL (temporary tables, ALTER TABLE, ...). That's unfortunate, as it's one of the cases the streaming was supposed to help. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-29 Thread Tomas Vondra
On 01/29/2018 02:49 PM, Simon Riggs wrote: > On 29 January 2018 at 13:34, Tomas Vondra > wrote: > >> The important detail is that we only really care >> about aborts in transactions that modified catalogs in some way (e.g. by >> doing DDL). But we can safely decode

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-29 Thread Tomas Vondra
On 01/29/2018 03:17 PM, Simon Riggs wrote: > On 29 January 2018 at 14:13, Tomas Vondra > wrote: > >> 4) inspect the new row (which we still have in reorderbuffer) >> >> 5) Kabooom! The row has column "c" which we don't see in the catalog. > >

Re: JIT compiling with LLVM v9.0

2018-01-29 Thread Tomas Vondra
didate multilib: .;@m64 Candidate multilib: 32;@m32 Selected multilib: .;@m64 regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services Writing fmgroids.h Writing fmgrprotos.h Writing fmgrtab.c Writing postgres.bki W

Re: JIT compiling with LLVM v9.0

2018-01-29 Thread Tomas Vondra
On 01/29/2018 10:57 PM, Andres Freund wrote: > Hi, > > On 2018-01-29 22:51:38 +0100, Tomas Vondra wrote: >> Hi, I wanted to look at this, but my attempts to build the jit branch >> fail with some compile-time warnings (uninitialized variables) and >> errors (unknown

Re: JIT compiling with LLVM v9.0

2018-01-29 Thread Tomas Vondra
On 01/29/2018 11:17 PM, Andres Freund wrote: > On 2018-01-29 23:01:14 +0100, Tomas Vondra wrote: >> On 01/29/2018 10:57 PM, Andres Freund wrote: >>> Hi, >>> >>> On 2018-01-29 22:51:38 +0100, Tomas Vondra wrote: >>>> Hi, I wanted to look at this, b

Re: JIT compiling with LLVM v9.0

2018-01-29 Thread Tomas Vondra
On 01/29/2018 11:49 PM, Tomas Vondra wrote: > > ... > > and that indeed changes the failure to this: > > Writing postgres.bki > Writing schemapg.h > Writing postgres.description > Writing postgres.shdescription > llvmjit_error.cpp: In function

Re: JIT compiling with LLVM v9.0

2018-01-30 Thread Tomas Vondra
On 01/30/2018 12:24 AM, Andres Freund wrote: > Hi, > > On 2018-01-30 00:16:46 +0100, Tomas Vondra wrote: >> FWIW I've installed llvm 5.0.1 from distribution package, and now >> everything builds fine (I don't even need the configure tweak). >> >>

Re: STATISTICS retained in CREATE TABLE ... LIKE (INCLUDING ALL)?

2018-02-01 Thread Tomas Vondra
> > That seems quite rational. > > To prevent this getting lost I've added it to the March commitfest [1]. > > In the commitfest application I've classed it (for now) as a bug fix. > If that changes then we can alter it in the commitfest app. > > [1] https:

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2018-02-01 Thread Tomas Vondra
On 01/31/2018 07:53 AM, Masahiko Sawada wrote: > On Sat, Jan 20, 2018 at 7:08 AM, Tomas Vondra > wrote: >> On 01/19/2018 03:34 PM, Tomas Vondra wrote: >>> Attached is v5, fixing a silly bug in part 0006, causing segfault when >>> creating a subscription. >>>

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2018-02-01 Thread Tomas Vondra
y are ready to be continued. > Will do. Thanks for the feedback. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Re: Improve OR conditions on joined columns (common star schema problem)

2018-02-02 Thread Tomas Vondra
eries) help? I'm willing to spend some cycles on that, if considered helpful. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Re: Improve OR conditions on joined columns (common star schema problem)

2018-02-02 Thread Tomas Vondra
On 02/02/2018 03:26 PM, Tom Lane wrote: > Tomas Vondra writes: >> ISTM this patch got somewhat stuck as we're not quite sure the >> transformation is correct in all cases. Is my impression correct? > > Yeah, that's the core issue. > >> If yes, how to w

Re: WIP: BRIN multi-range indexes

2018-02-05 Thread Tomas Vondra
and be replaced by timestamp and * interval. */ But then why adding BRIN opclasses at all? And if adding them, why not to test them? We all know how long deprecation takes, particularly for data types. For me the question is whether to bother with adding the multi-minmax opclasses, of

Re: WIP: BRIN multi-range indexes

2018-02-05 Thread Tomas Vondra
On 02/06/2018 12:40 AM, Tom Lane wrote: > Tomas Vondra writes: >> Yeah, that's what I've been wondering about too. There's also this >> comment in nabstime.h: > >> /* >> * Although time_t generally is a long int on 64 bit systems, these two >

Re: [HACKERS] MERGE SQL Statement for PG11

2018-02-06 Thread Tomas Vondra
nt status is (or my understanding of it). That is (a) what are the missing pieces, (b) why are they missing, (c) how we plan to address them in the future and (d) opinions on these issues expressed by others on this thread. kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2018-02-06 Thread Tomas Vondra
he buildfarm go green. > Did you do a test with "-O0"? In my experience that makes valgrind tests much more reliable and repeatable. Some time ago we've seen cases that were failing for me but not for others, and I suspect it was due to me using "-O0". (This is mo

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2018-02-06 Thread Tomas Vondra
On 02/06/2018 10:14 PM, Peter Geoghegan wrote: > On Tue, Feb 6, 2018 at 1:04 PM, Tomas Vondra > wrote: >> Did you do a test with "-O0"? In my experience that makes valgrind tests >> much more reliable and repeatable. Some time ago we've seen cases that >>

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2018-02-06 Thread Tomas Vondra
On 02/06/2018 10:39 PM, Peter Geoghegan wrote: > On Tue, Feb 6, 2018 at 1:30 PM, Tomas Vondra > wrote: >> I have little idea what -Og exactly means. It seems to be focused on >> debugging experience, and so still does some of the optimizations. > > As I under

Re: [HACKERS] FOSDEM PGDay_2018_Developer_Meeting notes

2018-02-09 Thread Tomas Vondra
opment part to me. Shitty code is my natural talent. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-02-09 Thread Tomas Vondra
nforce the rule, because some writes may not produce WAL (think of unlogged tables). I also suspect it may be incorrect "in the opposite direction" because a query may not do any changes and yet it may produce WAL (e.g. due to wal_hint_bits=true). So we may need to think

Using multiple extended statistics for estimates

2019-10-28 Thread Tomas Vondra
I've both in the loop, but I think that'd be wrong - the MCV list is expected to contain more information about individual values (compared to functional deps, which are column-level). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Supp

Re: Creating foreign key on partitioned table is too slow

2019-10-30 Thread Tomas Vondra
esting it's how it should be done ultimately. The other bit (speed of planning with 8k partitions) is probably a more general issue, and I suppose we'll improve that over time. I don't think there's a simple change magically improving that. regards -- Tomas Vondra

Re: Parallel leader process info in EXPLAIN

2019-10-30 Thread Tomas Vondra
e whole block (which only really happens for oversized chunks) the accounting info does not change. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: MarkBufferDirtyHint() and LSN update

2019-10-30 Thread Tomas Vondra
by adding a sleep or a breakpoint, I guess. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Removing alignment padding for byval types

2019-10-31 Thread Tomas Vondra
hether there's any byval columns with alignment requirements at slot creation / ExecSetSlotDescriptor() time (or even set a different callback getsomeattrs callback, but that's a bit more complicated). Thoughts? Seems reasonable. I certainly agree this padding is pretty annoying, so

Re: Proposal: Global Index

2019-10-31 Thread Tomas Vondra
archies ... At present I don't actually use inheritance; instead I put triggers on the child tables that do an insert on the parent table, which has the effect of enforcing the uniqueness I want. Does it? Are you sure it actually works in READ COMMITTED? What exactly does the trigger do? re

Re: Adding percentile metrics to pg_stat_statements module

2019-10-31 Thread Tomas Vondra
able. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Removing alignment padding for byval types

2019-10-31 Thread Tomas Vondra
On Thu, Oct 31, 2019 at 12:24:33PM -0700, Andres Freund wrote: Hi, On 2019-10-31 20:15:12 +0100, Tomas Vondra wrote: On Thu, Oct 31, 2019 at 11:48:21AM -0700, Andres Freund wrote: > We currently align byval types such as int4/8, float4/8, timestamp *, > date etc, even though we mostly

Re: Postgres cache

2019-10-31 Thread Tomas Vondra
to look at the code and comments in the related files, particularly in src/backend/utils/cache/syscache.c src/backend/utils/cache/relcache.c Not sure if there's a better source of information :-( regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Devel

Re: merging HashJoin and Hash nodes

2019-10-31 Thread Tomas Vondra
ualizing execution plans, so maybe backwards compatibility of the output is a concern too (I know we don't promise anything, though). The fact that EXPLAIN doesn't label relations seems to be a separate concern that applies equally to nestloop joins, and could perhaps be addressed with some more optional verbosity, not a fake node? Yeah, that seems separate. -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Adding percentile metrics to pg_stat_statements module

2019-11-01 Thread Tomas Vondra
introduce additional errors into the estimates, because it discards the weights/frequencies. If it turns out that memory consumption on real workloads is small enough, it could eventually be turned on by default. Maybe, but it's not just about memory consumption. CPU matters too. re

Re: pglz performance

2019-11-01 Thread Tomas Vondra
code. I'd try running the benchmarks to verify the numbers, and maybe do some additional tests, but it's not clear to me which patches should I use. I think the last patches with 'hacked' and 'hacked8' in the name are a couple of months old, and the recent posts att

Re: On disable_cost

2019-11-01 Thread Tomas Vondra
actually compare the cost) is probably the way forward. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: 64 bit transaction id

2019-11-01 Thread Tomas Vondra
roblems (maybe later) Well, not fatal, but naive approaches can increase per-tuple overhead. And we already have plenty of that, hence there were proposals to use page epochs and so on. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support,

Re: 64 bit transaction id

2019-11-01 Thread Tomas Vondra
a separate page, we don't have any link between those tuples. And adding it may easily mean more overhead than the 8B we'd save by only storing a single XID. IMO the most promising solution to this is the "page epoch" approach discussed some time ago (1-2 years

Re: 64 bit transaction id

2019-11-02 Thread Tomas Vondra
, that has to be stored somewhere ... -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: 64 bit transaction id

2019-11-02 Thread Tomas Vondra
rsion" which may be on another page. That's pretty bad, bot for I/O and cache hit ratio. I don't think that's a reasonable trade-off (at least compared to simply making the XIDs 64bit). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: 64 bit transaction id

2019-11-03 Thread Tomas Vondra
tely cheap, we certainly are not going to do lookups of other pages or looking for older versions of the row, and so on. Being able to do visibility decisions based on the tuple alone (or possibly page-level + tuple information) has a lot of value, and I don't think we want to make this more

Re: Log statement sample - take two

2019-11-03 Thread Tomas Vondra
maximum to disable it.[4] Tomas Vondra proposed to use two minimum thresholds: 1) log_min_duration_sample - enables sampling of commands, using the existing GUC log_statement_sample_rate 2) log_min_duration_statement - logs all commands exceeding this This patch implement this idea. PS: I notice I

Re: 64 bit transaction id

2019-11-04 Thread Tomas Vondra
tuple overhead without causing significant issues elsewhere), we'd be crazy not to consider it. The bar is pretty high, though, because this touches one of the core pieces of the database. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Suppo

Re: 64 bit transaction id

2019-11-04 Thread Tomas Vondra
On Mon, Nov 04, 2019 at 10:04:09AM -0800, Andres Freund wrote: Hi, (I've not read the rest of this thread yet) On 2019-11-04 16:07:23 +0100, Tomas Vondra wrote: On Mon, Nov 04, 2019 at 04:39:44PM +0300, Павел Ерёмин wrote: > And yet, if I try to implement a similar mechanism, if su

Re: 64 bit transaction id

2019-11-04 Thread Tomas Vondra
On Mon, Nov 04, 2019 at 10:44:53AM -0800, Andres Freund wrote: Hi, On 2019-11-04 19:39:18 +0100, Tomas Vondra wrote: On Mon, Nov 04, 2019 at 10:04:09AM -0800, Andres Freund wrote: > And "without causing significant issues elsewhere" unfortunately > includes continuing to all

Re: [Patch] Optimize dropping of relation buffers using dlist

2019-11-05 Thread Tomas Vondra
//travis-ci.org/postgresql-cfbot/postgresql/builds/607563900 regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Log statement sample - take two

2019-11-06 Thread Tomas Vondra
Pushed, with some minor tweaks and rewording to the documentation. The first bit, documenting the log_transaction_sample_rate as PG_SUSET, got backpatched to 12, where it was introduced. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support

Re: idea: log_statement_sample_rate - bottom limit for sampling

2019-11-06 Thread Tomas Vondra
On Sun, Aug 04, 2019 at 11:41:54PM +0200, Tomas Vondra wrote: On Sun, Aug 04, 2019 at 10:48:48PM +0200, Tomas Vondra wrote: On Sun, Aug 04, 2019 at 04:25:12PM -0400, Tom Lane wrote: Tomas Vondra writes: On Sun, Aug 04, 2019 at 03:16:12PM -0400, Tom Lane wrote: Isn't the issue her

Re: idea: log_statement_sample_rate - bottom limit for sampling

2019-11-06 Thread Tomas Vondra
On Wed, Nov 06, 2019 at 08:00:57PM +0100, Adrien Nayrat wrote: On 11/6/19 7:21 PM, Tomas Vondra wrote: I've pushed the reworked version of log_statement_sample_rate patch [1]. If I understand correctly, that makes this patch unnecessary, and we should mark it as rejected. Or do we still ne

Re: Using multiple extended statistics for estimates

2019-11-06 Thread Tomas Vondra
On Mon, Oct 28, 2019 at 04:20:48PM +0100, Tomas Vondra wrote: Hi, PostgreSQL 10 introduced extended statistics, allowing us to consider correlation between columns to improve estimates, and PostgreSQL 12 added support for MCV statistics. But we still had the limitation that we only allowed

Re: Using multiple extended statistics for estimates

2019-11-06 Thread Tomas Vondra
On Wed, Nov 06, 2019 at 08:54:40PM +0100, Tomas Vondra wrote: On Mon, Oct 28, 2019 at 04:20:48PM +0100, Tomas Vondra wrote: Hi, PostgreSQL 10 introduced extended statistics, allowing us to consider correlation between columns to improve estimates, and PostgreSQL 12 added support for MCV

Re: Using multiple extended statistics for estimates

2019-11-07 Thread Tomas Vondra
On Thu, Nov 07, 2019 at 01:38:20PM +0900, Kyotaro Horiguchi wrote: Hello. At Wed, 6 Nov 2019 20:58:49 +0100, Tomas Vondra wrote in >Here is a slightly polished v2 of the patch, the main difference being >that computing clause_attnums was moved to a separate function. > This time

Re: Reorderbuffer crash during recovery

2019-11-07 Thread Tomas Vondra
ons. Now I have tried without using sub-transactions and could still reproduce this issue. You are right Issue 1 will appear in both the cases with and without subtransactions. Okay, thanks for the confirmation. I'm a bit confused - does this happen only with the logical_work_mem patches,

Re: Monitoring disk space from within the server

2019-11-08 Thread Tomas Vondra
data, which is nice, but it still needs to be fed to some motinoring and alerting system. And every monitoring system has a good plugin to collect this type of data, so why not to use that? Surely, we can't rely on this for any internal logic - so why not to provide this as an extension?

Re: Monitoring disk space from within the server

2019-11-08 Thread Tomas Vondra
On Fri, Nov 08, 2019 at 04:06:21PM +0100, Christoph Berg wrote: Re: Tomas Vondra 2019-11-08 <20191108145025.d7pfcip6plufxiah@development> While I agree monitoring disk space is important, I think pretty much every deployment already does that using some other monitoring tool (whic

Re: ssl passphrase callback

2019-11-09 Thread Tomas Vondra
g, just store it in a separate file and it's about the same as storing it in the .so library. Is there something that can be done with a .so library but can't be done with a shell command (which may just call a binary, with all the config included, making it equal to the .

Re: Missing constant propagation in planner on hash quals causes join slowdown

2019-11-09 Thread Tomas Vondra
ould need to be more complicated (requiring more joins, or something like that). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Using multiple extended statistics for estimates

2019-11-10 Thread Tomas Vondra
On Sat, Nov 09, 2019 at 12:33:05PM -0800, Mark Dilger wrote: On 11/6/19 11:58 AM, Tomas Vondra wrote: On Wed, Nov 06, 2019 at 08:54:40PM +0100, Tomas Vondra wrote: On Mon, Oct 28, 2019 at 04:20:48PM +0100, Tomas Vondra wrote: Hi, PostgreSQL 10 introduced extended statistics, allowing us to

Re: Using multiple extended statistics for estimates

2019-11-10 Thread Tomas Vondra
On Sat, Nov 09, 2019 at 02:32:27PM -0800, Mark Dilger wrote: On 11/9/19 12:33 PM, Mark Dilger wrote: On 11/6/19 11:58 AM, Tomas Vondra wrote: On Wed, Nov 06, 2019 at 08:54:40PM +0100, Tomas Vondra wrote: On Mon, Oct 28, 2019 at 04:20:48PM +0100, Tomas Vondra wrote: Hi, PostgreSQL 10

Re: Index Skip Scan

2019-11-10 Thread Tomas Vondra
12.120 rows=1331 loops=1) Output: a, c, b Skip scan: true Heap Fetches: 1331 Planning Time: 0.053 ms Execution Time: 12.632 ms (6 rows) This is a more generic issue, not specific to this patch, of course. I think we saw it with the incremental sort patch, IIRC. I wonder how difficult would

Re: [Patch] Optimize dropping of relation buffers using dlist

2019-11-12 Thread Tomas Vondra
On Tue, Nov 12, 2019 at 10:49:49AM +, k.jami...@fujitsu.com wrote: On Thurs, November 7, 2019 1:27 AM (GMT+9), Robert Haas wrote: On Tue, Nov 5, 2019 at 10:34 AM Tomas Vondra wrote: > 2) This adds another hashtable maintenance to BufferAlloc etc. but > you've only

Re: Using multiple extended statistics for estimates

2019-11-13 Thread Tomas Vondra
ng. So what this does instead is simply merging all the dependencies from all the relevant stats, and treating them as a single collection. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From 0e09

Re: ssl passphrase callback

2019-11-13 Thread Tomas Vondra
trol that if the command string starts with a # or something. Very good idea If it's about securely passing sensitive information (i.e. passphrase) as was suggested, then I think that only applies to fairly small number of GUCs. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: ssl passphrase callback

2019-11-13 Thread Tomas Vondra
s. Also, there were vague references to issues with passing parameters to archive_command. A link to details, past discussion, or brief explanation would be nice. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Missing dependency tracking for TableFunc nodes

2019-11-13 Thread Tomas Vondra
, the patch is over 2.2MB, gzip'd. I'll only bother distilling it if you don't already know about these cache lookup failures. Not sure. But I do wonder if we see the same issue. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Optimize partial TOAST decompression

2019-11-14 Thread Tomas Vondra
was supposed to fix, but it just masked them instead. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index 47a03f

Re: Using multiple extended statistics for estimates

2019-11-14 Thread Tomas Vondra
On Wed, Nov 13, 2019 at 10:04:36AM -0800, Mark Dilger wrote: On 11/13/19 7:28 AM, Tomas Vondra wrote: Hi, here's an updated patch, with some minor tweaks based on the review and added tests (I ended up reworking those a bit, to make them more like the existing ones). Thanks, Tomas, fo

Re: ssl passphrase callback

2019-11-14 Thread Tomas Vondra
On Thu, Nov 14, 2019 at 11:34:24AM -0500, Andrew Dunstan wrote: On 11/14/19 11:07 AM, Bruce Momjian wrote: On Thu, Nov 14, 2019 at 11:42:05AM +0100, Magnus Hagander wrote: On Wed, Nov 13, 2019 at 9:23 PM Tomas Vondra I think it would be beneficial to explain why shared object is more

Re: Using multiple extended statistics for estimates

2019-11-14 Thread Tomas Vondra
On Thu, Nov 14, 2019 at 10:23:44AM -0800, Mark Dilger wrote: On 11/14/19 7:55 AM, Tomas Vondra wrote: On Wed, Nov 13, 2019 at 10:04:36AM -0800, Mark Dilger wrote: On 11/13/19 7:28 AM, Tomas Vondra wrote: Hi, here's an updated patch, with some minor tweaks based on the review and

Re: Missing dependency tracking for TableFunc nodes

2019-11-14 Thread Tomas Vondra
AE lock on the relation in RemoveStatisticsById, per the attached patch. It's possible we'll need to do something more complicated once join stats are added, but for now this should be enough (and backpatchable). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL

Re: Using multiple extended statistics for estimates

2019-11-14 Thread Tomas Vondra
On Thu, Nov 14, 2019 at 03:16:04PM -0500, Tom Lane wrote: Tomas Vondra writes: For the case with equal Const values that should be mostly obvious, i.e. "a=1 AND a=1 AND a=1" has the same selectivity as "a=1". The case with different Const values is harder, unfortun

Re: Using multiple extended statistics for estimates

2019-11-14 Thread Tomas Vondra
On Thu, Nov 14, 2019 at 01:17:02PM -0800, Mark Dilger wrote: On 11/14/19 12:04 PM, Tomas Vondra wrote: Aha, I think I understand now - thanks for the explanation. You're right the comment is trying to explain why just taking the last clause for a given attnum is fine. I'll try t

Re: Missing dependency tracking for TableFunc nodes

2019-11-14 Thread Tomas Vondra
On Thu, Nov 14, 2019 at 04:36:54PM -0500, Tom Lane wrote: Tomas Vondra writes: On Wed, Nov 13, 2019 at 08:37:59PM -0500, Tom Lane wrote: I concur with Tomas' suspicion that this must be a race condition during DROP STATISTICS. If we're going to allow people to do that separ

Re: Missing dependency tracking for TableFunc nodes

2019-11-14 Thread Tomas Vondra
On Thu, Nov 14, 2019 at 07:27:29PM -0300, Alvaro Herrera wrote: On 2019-Nov-14, Tomas Vondra wrote: Isn't that solved by RemoveObjects() doing this? /* Get an ObjectAddress for the object. */ address = get_object_address(stmt->removeType,

Re: Missing dependency tracking for TableFunc nodes

2019-11-14 Thread Tomas Vondra
On Thu, Nov 14, 2019 at 05:35:06PM -0500, Tom Lane wrote: Tomas Vondra writes: On Thu, Nov 14, 2019 at 04:36:54PM -0500, Tom Lane wrote: Hm. No, it's not enough, unless you add more logic to deal with the possibility that the stats object is gone by the time you have the table

Re: why doesn't optimizer can pull up where a > ( ... )

2019-11-20 Thread Tomas Vondra
ation, because no one considered it interesting enough to submit a patch. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: why doesn't optimizer can pull up where a > ( ... )

2019-11-20 Thread Tomas Vondra
n in the original query will include rows where that isn't true.) Yeah. I think the examples are a bit messed up, and surely there are other restrictions on applicability of this optimization. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: why doesn't optimizer can pull up where a > ( ... )

2019-11-20 Thread Tomas Vondra
On Wed, Nov 20, 2019 at 12:36:50PM -0500, Tom Lane wrote: Tomas Vondra writes: On Wed, Nov 20, 2019 at 11:12:56AM -0500, Tom Lane wrote: I'm content to say that the application should have written the query with a GROUP BY to begin with. I'm not sure I agree with that. The probl

Re: why doesn't optimizer can pull up where a > ( ... )

2019-11-20 Thread Tomas Vondra
On Wed, Nov 20, 2019 at 12:34:25PM -0800, Xun Cheng wrote: On Wed, Nov 20, 2019 at 11:18 AM Tomas Vondra wrote: On Wed, Nov 20, 2019 at 12:36:50PM -0500, Tom Lane wrote: >Tomas Vondra writes: >> On Wed, Nov 20, 2019 at 11:12:56AM -0500, Tom Lane wrote: >>> I'm

Re: why doesn't optimizer can pull up where a > ( ... )

2019-11-21 Thread Tomas Vondra
, I will search more about this. Having said that, the best form of criticism is a patch. If somebody actually wrote the code to do something like this, we could look at how much time it wasted in which unsuccessful cases and then have an informed discussion about whether it was worth a

<    5   6   7   8   9   10   11   12   13   14   >