Re: Auto explain after query timeout

2022-09-20 Thread James Coleman
On Tue, Sep 20, 2022 at 2:12 PM Gurjeet wrote: > > On Tue Sep 20, 2022 at 10:34 AM PDT, James Coleman wrote: > > Hopefully I'm not missing something obvious, but as far as I know > > there's no way to configure auto explain to work fire > > statement_timeout f

Re: Auto explain after query timeout

2022-09-20 Thread James Coleman
On Tue, Sep 20, 2022 at 3:06 PM Robert Haas wrote: > > On Tue, Sep 20, 2022 at 2:35 PM James Coleman wrote: > > Either I'm missing something (and/or this was fixed in a later PG > > version), but I don't think this is how it works. We have this >

Virtual tx id

2022-09-20 Thread James Sewell
Hello Hackers! Is it possible to get the current virtual txid from C somehow? I've looked through the code, but can't seem to find anything other than getting a NULL when there is no (real) xid assigned. Maybe I'm missing something? Cheers, James

Re: Support pg_attribute_aligned and noreturn in MSVC

2022-09-21 Thread James Coleman
On Tue, Sep 20, 2022 at 9:18 PM Michael Paquier wrote: > > On Tue, Sep 20, 2022 at 08:01:20AM -0400, James Coleman wrote: > > I don't have access to a Windows machine for testing, but re-reading > > the documentation it looks like the issue is that our noreturn macr

Re: Consider parallel for lateral subqueries with limit

2022-09-22 Thread James Coleman
On Mon, Sep 19, 2022 at 4:29 PM Robert Haas wrote: > > On Mon, Sep 19, 2022 at 3:58 PM James Coleman wrote: > > But in the case where there's correlation via LATERAL we already don't > > guarantee unique executions for a given set of params into the lateral > &

Re: An inefficient query caused by unnecessary PlaceHolderVar

2023-05-30 Thread James Coleman
d rel is under the same lowest nulling outer + * join. I think this is clearer: "references something outside the subquery being pulled up and is not under the same lowest outer join." One other thing: it would be helpful to have the test query output be stable between HEAD and this patch; perhaps add: order by 1, 2, 3, 4, 5, 6, 7 to ensure stability? Thanks, James Coleman

Re: An inefficient query caused by unnecessary PlaceHolderVar

2023-06-01 Thread James Coleman
On Wed, May 31, 2023 at 10:30 PM Richard Guo wrote: > > > On Wed, May 31, 2023 at 1:27 AM James Coleman wrote: >> >> This looks good to me. > > > Thanks for the review! Sure thing! >> >> A few small tweaks suggested to comment wording: >> &g

Re: RFC: Logging plan of the running query

2023-06-02 Thread James Coleman
ning, and I think it would be worth adding that. Are you interested in re-opening this patch? I'd be happy to provide further review and help to try to push this along. I've rebased the patch and attached as v26. Thanks, James Coleman v26-0001-Add-function-to-log-the-plan-of-the-query.patch Description: Binary data

Re: RFC: Logging plan of the running query

2023-06-05 Thread James Coleman
On Mon, Jun 5, 2023 at 4:30 AM torikoshia wrote: > > On 2023-06-03 02:51, James Coleman wrote: > > Hello, > > > > Thanks for working on this patch! Sure thing! I'm *very interested* in seeing this available, and I think it paves the way for some additional features l

Add last_commit_lsn to pg_stat_database

2023-06-09 Thread James Coleman
) fits more naturally into the stats system. I'd originally thought of exposing it in pg_stat_wal, but that's per-cluster rather than per-database (indeed, this is a flaw I hadn't considered in the original patch), so I think pg_stat_database is the correct location. I've

Re: Let's make PostgreSQL multi-threaded

2023-06-10 Thread James Addison
I don't have an objection, but I do wonder: can one (or perhaps a few) queries/workloads be provided where threading would be significantly beneficial? (some material there could help get people on-board with the idea and potentially guide many of the smaller questions that arise along the way) O

Re: Parallelize correlated subqueries that execute within each worker

2023-06-11 Thread James Coleman
On Tue, Jun 6, 2023 at 4:36 AM Richard Guo wrote: > > > On Mon, Jan 23, 2023 at 10:00 PM James Coleman wrote: >> >> Which this patch we do in fact now see (as expected) rels with >> non-empty lateral_relids showing up in generate_[useful_]gather_paths. >> And t

Re: RFC: Logging plan of the running query

2023-06-12 Thread James Coleman
On Sun, Jun 11, 2023 at 11:07 PM torikoshia wrote: > > On 2023-06-06 03:26, James Coleman wrote: > > On Mon, Jun 5, 2023 at 4:30 AM torikoshia > > wrote: > >> > >> On 2023-06-03 02:51, James Coleman wrote: > >> > Hello, > >> > >

Re: RFC: Logging plan of the running query

2023-06-13 Thread James Coleman
On Tue, Jun 13, 2023 at 11:22 AM torikoshia wrote: > > On 2023-06-13 00:52, James Coleman wrote: > >> > >> > I've attached v27. The important change here in 0001 is that it > >> > guarantees the interrupt handler is re-entrant, since that was a bug >

Re: RFC: Logging plan of the running query

2023-06-14 Thread James Coleman
On Tue, Jun 13, 2023 at 11:53 AM James Coleman wrote: > > ... > I'm going to re-run tests with my patch version + resetting the flag > on SIGINT (and any other error condition) to be certain that the issue > you uncovered (where backends get stuck after a SIGINT not respondin

Re: Let's make PostgreSQL multi-threaded

2023-06-14 Thread James Addison
On Mon, 12 Jun 2023 at 20:24, Andres Freund wrote: > > Hi, > > On 2023-06-12 16:23:14 +0400, Pavel Borisov wrote: > > Is the following true or not? > > > > 1. If we switch processes to threads but leave the amount of session > > local variables unchanged, there would be hardly any performance gain

Re: Let's make PostgreSQL multi-threaded

2023-06-14 Thread James Addison
On Wed, 14 Jun 2023 at 20:48, Robert Haas wrote: > > On Wed, Jun 14, 2023 at 3:16 PM James Addison wrote: > > I think that they're practical performance-related questions about the > > benefits of performing a technical migration that could involve > > significant de

Re: Let's make PostgreSQL multi-threaded

2023-06-14 Thread James Addison
On Tue, 13 Jun 2023 at 07:55, Konstantin Knizhnik wrote: > > > > On 12.06.2023 3:23 PM, Pavel Borisov wrote: > > Is the following true or not? > > > > 1. If we switch processes to threads but leave the amount of session > > local variables unchanged, there would be hardly any performance gain. > >

Re: Let's make PostgreSQL multi-threaded

2023-06-15 Thread James Addison
On Thu, 15 Jun 2023 at 08:12, Konstantin Knizhnik wrote: > > > > On 15.06.2023 1:23 AM, James Addison wrote: > > On Tue, 13 Jun 2023 at 07:55, Konstantin Knizhnik wrote: > > > On 12.06.2023 3:23 PM, Pavel Borisov wrote: > > Is the following true or not? > &g

Re: RFC: Logging plan of the running query

2023-06-15 Thread James Coleman
On Thu, Jun 15, 2023 at 9:00 AM torikoshia wrote: > > On 2023-06-15 01:48, James Coleman wrote: > > On Tue, Jun 13, 2023 at 11:53 AM James Coleman > > wrote: > >> > >> ... > >> I'm going to re-run tests with my patch version + resetting the flag

deb’s pg_upgradecluster(1) vs streaming replication

2023-06-17 Thread James Cloos
uld I expect things to work easily? -JimC -- James Cloos OpenPGP: 0x997A9F17ED7DAEA6

path->param_info only set for lateral?

2023-06-18 Thread James Coleman
ateral reference) we're not going to get any ParamPathInfo added to the path or the rel. Is there a reason why we don't track the required relids providing the PARAM_EXEC params in this case? Thanks, James Coleman 1: https://www.postgresql.org/message-id/CAMbWs4_evjcMzN8Gw78bHfhfo2FKJThqhEjRJRmoMZx%3DNXcJ7w%40mail.gmail.com

Re: path->param_info only set for lateral?

2023-06-20 Thread James Coleman
On Sun, Jun 18, 2023 at 10:57 PM Tom Lane wrote: > > James Coleman writes: > > Over in "Parallelize correlated subqueries that execute within each > > worker" [1} Richard Guo found a bug in the current version of my patch > > in that thread. While debugging

Re: Use of additional index columns in rows filtering

2023-06-21 Thread James Coleman
Rows Removed by Index Recheck: 197780 > Filter: (b = 4) > Buffers: shared hit=544 > Planning Time: 0.105 ms > Execution Time: 13.690 ms > (10 rows) > > ... I did also confirm that this properly identifies cases Jeff had mentioned to me like "Index Filter: (((a * 2) > 50) AND ((b % 10) = 4))". I noticed also you still had questions/TODOs about handling index scans for join clauses. Regards, James Coleman 1: https://www.postgresql.org/message-id/20230609000600.syqy447e6metnvyj%40awork3.anarazel.de

Opportunistically pruning page before update

2023-06-21 Thread James Coleman
t had time to devote properly to that, so I'm wondering if there's anyone who might be interested in collaborating on that part. Other TODOs: - Audit other callers of RelationSetTargetBlock() to ensure they don't hold pointers into the page. Regards, James Coleman v1-000

Re: Use of additional index columns in rows filtering

2023-06-21 Thread James Coleman
On Wed, Jun 21, 2023 at 11:28 AM Tomas Vondra wrote: > > > > On 6/21/23 14:45, James Coleman wrote: > > Hello, > > > > I've cc'd Jeff Davis on this due to a conversation we had at PGCon > > about applying filters on index tuples during index scans.

Re: Memory leak in incremental sort re-scan

2023-06-21 Thread James Coleman
sorted_keys). That avoids unnecessary recreation of the sort states, but it also fixes the problem Tom noted as well: the call to preparePresortedCols() is already guarded by a test on fullsort_state being NULL, so with this change we also won't unnecessarily redo that work. Regards, James Coleman v2-0001-Fix-memory-leak-in-incremental-sort-rescan.patch Description: Binary data

Stampede of the JIT compilers

2023-06-23 Thread James Coleman
ered implemented a GUC/feature like "max_concurrent_jit_compilations" to cap the number of backends that may be compiling a query at any given point so that we avoid an optimization from running amok and consuming all of a servers resources? Regards, James Coleman

Re: Stampede of the JIT compilers

2023-06-24 Thread James Coleman
On Sat, Jun 24, 2023 at 7:40 AM Tomas Vondra wrote: > > > > On 6/24/23 02:33, David Rowley wrote: > > On Sat, 24 Jun 2023 at 02:28, James Coleman wrote: > >> There are a couple of issues here. I'm sure it's been discussed > >> before, and it'

Re: Stampede of the JIT compilers

2023-06-24 Thread James Coleman
On Sat, Jun 24, 2023 at 1:54 PM Tom Lane wrote: > > James Coleman writes: > > In that context capping the number of backends compiling, particularly > > where plans (and JIT?) might be cached, could well save us (depending > > on workload). > > TBH I do not find t

Re: Stampede of the JIT compilers

2023-06-24 Thread James Coleman
On Sat, Jun 24, 2023 at 8:14 PM David Rowley wrote: > > On Sun, 25 Jun 2023 at 05:54, Tom Lane wrote: > > > > James Coleman writes: > > > On Sat, Jun 24, 2023 at 7:40 AM Tomas Vondra > > > wrote: > > >> On 6/24/23 02:33, David Rowley wrote: >

Re: Stampede of the JIT compilers

2023-06-25 Thread James Coleman
does, but I don't > have a Google Cloud SQL or RDS instance running right to verify their > settings. I do seem to remember that they did as well though, at least a > while back. > > > Michael I believe it's off by default in Aurora Postgres also. Regards, James Coleman

Analyze on table creation?

2023-06-26 Thread James Coleman
so I assume people have considered it before. If so, I'd like to understand why the conclusion was not to do it, or, alternatively if it's a lack of tuits. Regards, James Coleman

Re: Analyze on table creation?

2023-06-26 Thread James Coleman
On Mon, Jun 26, 2023 at 1:45 PM Pavel Stehule wrote: > > > > po 26. 6. 2023 v 19:43 odesílatel Pavel Stehule > napsal: >> >> Hi >> >> po 26. 6. 2023 v 19:41 odesílatel James Coleman napsal: >>> >>> Hello, >>> >>> Have w

Re: Analyze on table creation?

2023-06-26 Thread James Coleman
cc'ing Tom because I'm curious if he's willing to provide some greater context on the commit in question. On Mon, Jun 26, 2023 at 2:16 PM Pavel Stehule wrote: > > > > po 26. 6. 2023 v 19:48 odesílatel James Coleman napsal: >> >> On Mon, Jun 26, 2

Re: Analyze on table creation?

2023-06-26 Thread James Coleman
On Mon, Jun 26, 2023 at 4:00 PM Andres Freund wrote: > > Hi, > > On 2023-06-26 13:40:49 -0400, James Coleman wrote: > > Have we ever discussed running an analyze immediately after creating a > > table? > > That doesn't make a whole lot of sense to me - we coul

Re: Analyze on table creation?

2023-06-27 Thread James Coleman
On Mon, Jun 26, 2023 at 4:16 PM James Coleman wrote: > > On Mon, Jun 26, 2023 at 4:00 PM Andres Freund wrote: > > > > Hi, > > > > On 2023-06-26 13:40:49 -0400, James Coleman wrote: > > > Have we ever discussed running an analyze immediately after creati

pgindent (probably my missing something obvious)

2023-07-03 Thread James Coleman
aving some very long function calls. I've downloaded the latest typedefs list, but I haven't added any types anyway. What obvious thing am I missing? Thanks, James Coleman

Re: Parallelize correlated subqueries that execute within each worker

2023-07-03 Thread James Coleman
On Sun, Jun 11, 2023 at 10:23 PM James Coleman wrote: > > ... > > And while trying the v9 patch I came across a crash with the query > > below. > > > > set min_parallel_table_scan_size to 0; > > set parallel_setup_cost to 0; > > set parallel_tuple_cost to

Re: pgindent (probably my missing something obvious)

2023-07-03 Thread James Coleman
On Mon, Jul 3, 2023 at 9:20 PM Tom Lane wrote: > > James Coleman writes: > > This is the first time I've run pgindent on my current machine, and it > > doesn't seem to be making any modifications to source files. For > > example this command: > > >

[BUG] Security bugs affected version detected.

2024-08-29 Thread James Watt
Our tool have detected that postgre in the version of REL9_6_18~ REL9_6_24 may also affected by the vulnerability CVE-2022-2625. The vulnerability database does not include these versions and you may not fix it in the REL9_6 branch. Is there a need to backport the patch of CVE-2022-2625?

Seq scan instead of index scan querying single row from primary key on large table

2024-07-18 Thread James Coleman
7;s true I think the early return cost multiplication of the LIMIT is being applied very naively on the seq scan node. Or perhaps the issue is that the startup cost for a single tuple on a seq scan like this shouldn't really have a startup cost of 0 -- that cost is presumably for tuples being returned _without_ having applied the filter. That seems slightly odd to me, because the cost of getting the first row out of that node -- in my naive view thinking about it for all of 5 seconds -- should be calculated based on applying the filter (and thus the likelihood that that filter matches right away). If we did that then this path would never win. But that 0.00 startup cost for the seq scan with a filter shows up in PG14 and PG11 also, not just PG16, so that's not something that's changed. To recap: the estimation of rows is correct, the estimated high (total) cost of the seq scan is correct, but the seq scan is chosen over the index scan anyway for a plan that returns a single "random" row based on the primary key. Am I right to be surprised here? James Coleman

Re: Seq scan instead of index scan querying single row from primary key on large table

2024-07-18 Thread James Coleman
On Thu, Jul 18, 2024 at 2:38 PM Tom Lane wrote: > > James Coleman writes: > > The plan generated by the planner changed suddenly one morning this > > week, and in a very surprising way: the innermost scan (of "objects") > > started choosing a seq scan, des

Re: [HACKERS] Multiple synchronous_standby_names rules

2020-06-08 Thread James Sewell
On Thu, 12 Jan 2017 at 12:06, Michael Paquier wrote: > On Thu, Jan 12, 2017 at 9:53 AM, James Sewell > wrote: > > What is needed to support this is the ability to configure Px with > something like: > > > > 1 (P1, P2, P3), 1 (D1, D2, D3) > > > > Would th

Threading in BGWorkers (!)

2020-06-22 Thread James Sewell
w things up in anyway (if you aren't following the standards / code comments). James -- The contents of this email are confidential and may be subject to legal or professional privilege and copyright. No representation is made that this email is free of viruses or other defects. If you hav

Re: Threading in BGWorkers (!)

2020-06-22 Thread James Sewell
On Tue, 23 Jun 2020 at 13:38, Tom Lane wrote: > James Sewell writes: > > I was talking about PostgreSQL and threading on IRC the other day - > which I > > know is a frowned upon topic - and just wanted to frame the same > questions > > here and hopefully get a discus

Re: Threading in BGWorkers (!)

2020-06-23 Thread James Sewell
> Using multithreading in bgworker is possible if you do not use any > Postgres runtime inside thread procedures or do it in exclusive critical > section. > It is not so convenient but possible. The most difficult thing from my > point of view is error reporting. > Happy to be proved wrong, but I

Re: Threading in BGWorkers (!)

2020-06-23 Thread James Sewell
On Tue, 23 Jun 2020 at 17:15, James Sewell wrote: > Using multithreading in bgworker is possible if you do not use any >> Postgres runtime inside thread procedures or do it in exclusive critical >> section. >> It is not so convenient but possible. The most difficult thin

Re: Threading in BGWorkers (!)

2020-06-23 Thread James Sewell
On Tue, 23 Jun 2020 at 17:26, Konstantin Knizhnik wrote: > On 23.06.2020 10:15, James Sewell wrote: > > Using multithreading in bgworker is possible if you do not use any >> Postgres runtime inside thread procedures or do it in exclusive critical >> section. >>

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-06-24 Thread James Coleman
On Fri, Jun 19, 2020 at 12:04 AM Justin Pryzby wrote: > > On Tue, Apr 07, 2020 at 08:40:30AM -0400, James Coleman wrote: > > On Tue, Apr 7, 2020 at 12:25 AM Justin Pryzby wrote: > > > On Mon, Apr 06, 2020 at 09:57:22PM +0200, Tomas Vondra wrote: > > > > I'

Re: Open Item: Should non-text EXPLAIN always show properties?

2020-06-25 Thread James Coleman
lly > do things. We try to keep the text format as humanly readable as > possible but don't really expect humans to be commonly reading the > other supported formats, so we care less about including additional > details there. > > There's also an open item regarding th

Re: Open Item: Should non-text EXPLAIN always show properties?

2020-06-25 Thread James Coleman
On Thu, Jun 25, 2020 at 12:33 PM Tom Lane wrote: > > Robert Haas writes: > > On Thu, Jun 25, 2020 at 8:42 AM James Coleman wrote: > >> Yesterday I'd replied [1] to Justin's proposal for this WRT > >> incremental sort and expressed my opinion tha

Re: Threading in BGWorkers (!)

2020-07-01 Thread James Sewell
#x27;s at least a way forward / a way to progress this discussion. Cheers, James [1] https://pubs.opengroup.org/onlinepubs/9699919799/functions/sigprocmask.html -- The contents of this email are confidential and may be subject to legal or professional privilege and copyright. No representation is

Re: suggest to rename enable_incrementalsort

2020-07-02 Thread James Coleman
more readable (at one point it was enable_incsort...which is short...but does not have an obvious meaning). I've attached a patch to make the change, though if people are interested in Tom's suggestion of enable_sort_incremental I could switch to that. James v1-0001-Rename-enable_incr

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-07-02 Thread James Coleman
true. The change has already been made and pushed for hash disk spilling, so I think we ought to use Justin's patch here. James [1] https://www.postgresql.org/message-id/2276865.1593102811%40sss.pgh.pa.us

Re: Use of "long" in incremental sort code

2020-07-02 Thread James Coleman
ke this new code follow that. I've attached a patch to make this change but with one tweak: I decided to use unint64 for both memory and disk (rather than Size in some cases) since we aggregated across multiple runs and have shared code that deals with both values. James v1-0001-Use-unint64-instead-of-long-for-space-used-variab.patch Description: Binary data

Re: Use of "long" in incremental sort code

2020-07-02 Thread James Coleman
rly limited degree (it uses -1 as a magic > value). Do you think it's reasonable to use int64 across the board for memory and disk space numbers then? If so, I can update the patch. James

Re: Use of "long" in incremental sort code

2020-07-02 Thread James Coleman
On Thu, Jul 2, 2020 at 3:39 PM Tom Lane wrote: > > Peter Geoghegan writes: > > On Thu, Jul 2, 2020 at 10:53 AM James Coleman wrote: > >> Do you think it's reasonable to use int64 across the board for memory > >> and disk space numbers then? If so, I can updat

Re: Patch for nodeIncrementalSort comment correction.

2020-07-19 Thread James Coleman
; Regards, > Vignesh > EnterpriseDB: http://www.enterprisedb.com > Thanks for fixing this. Looks correct to me. James

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 7:31 PM Tomas Vondra wrote: > > On Mon, Apr 06, 2020 at 07:09:11PM -0400, James Coleman wrote: > >On Mon, Apr 6, 2020 at 6:13 PM Tomas Vondra > > wrote: > >> > >> On Mon, Apr 06, 2020 at 05:47:48PM -0400, James Coleman wrote: > &

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
debug printouts, is that sortMethod is > frequently zero when we reach the EXPLAIN output for a worker. In many of > the tests this happens even though there is no visible failure, because > we've got a filter function hiding the output :-( > > So I concur with James' conclu

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 10:09 PM Tom Lane wrote: > > James Coleman writes: > > On Mon, Apr 6, 2020 at 9:46 PM Tom Lane wrote: > >> I think the correct fix is to change the enum declaration. > > > Hmm. I don't actually really like that, because it mean

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-07 Thread James Coleman
in show_hash_info() and show_hashagg_info(), and in your own text output, > that's called "Peak": > ExplainPropertyInteger("Peak Memory Usage", "kB", memPeakKb, es); > ExplainPropertyInteger("Peak Memory Usage", "kB", > spacePeakKb, es); Yes, that's a miss and should be fixed. James

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-07 Thread James Coleman
e pointers, but there's no reason to >* repay the setup cost, and because guard setting up pivot comparator >* state similarly, doing so might actually cause a leak. > > I can't figure out what should be. James, do you recall what this > should be? Yep, it&#x

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-07 Thread James Coleman
On Tue, Apr 7, 2020 at 7:58 PM Tomas Vondra wrote: > > On Tue, Apr 07, 2020 at 07:50:26PM -0400, James Coleman wrote: > >On Tue, Apr 7, 2020 at 7:02 PM Tomas Vondra > > wrote: > >> > >> On Mon, Apr 06, 2020 at 11:25:21PM -0500, Justin Pryzby wrote: > >&

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-08 Thread James Coleman
t; should do the trick. Looking at the tests that failed, I think we should consider just adding: set enable_sort = off; because several of those tests have very specific amounts of data to ensure we test the transition points around the different modes in the incremental sort node. James

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-08 Thread James Coleman
On Wed, Apr 8, 2020 at 11:02 AM Tomas Vondra wrote: > > On Wed, Apr 08, 2020 at 04:08:39PM +0200, Tomas Vondra wrote: > >On Wed, Apr 08, 2020 at 09:54:42AM -0400, James Coleman wrote: > >>On Wed, Apr 8, 2020 at 9:43 AM Tomas Vondra > >> wrote: > >>> >

Re: [PATCH] Incremental sort

2020-04-08 Thread James Coleman
On Wed, Apr 8, 2020 at 11:29 AM David Steele wrote: > > On 4/8/20 11:13 AM, James Coleman wrote: > >> > >> James, can you verify it that's still true? > > I marked this entry as committed in the 2020-03 CF but it's not clear to > me if that's enti

Re: Multiple FPI_FOR_HINT for the same block during killing btree index items

2020-04-09 Thread James Coleman
ltiple times. > > I like the idea of checking !ItemIdIsDead(iid) as a further condition > of killing the item -- there is clearly no point in doing work to kill > an item that is already dead. I don't like the idea of using an > exclusive buffer lock (even if it's just with wal_log_hints = on), > though. I don't have a strong opinion on the lock. James

Re: Multiple FPI_FOR_HINT for the same block during killing btree index items

2020-04-09 Thread James Coleman
try to run some numbers tomorrow to confirm, but I believe that the created_at value is almost (if not completely) unique. So, no, it's not a low cardinality case like that. I believe the write pattern to this table likely looks like: - INSERT - UPDATE - DELETE for every row. But tomorrow I can do some more digging if needed. James

Re: Multiple FPI_FOR_HINT for the same block during killing btree index items

2020-04-10 Thread James Coleman
On Thu, Apr 9, 2020 at 10:08 PM Peter Geoghegan wrote: > > On Thu, Apr 9, 2020 at 6:47 PM James Coleman wrote: > > I believe the write pattern to this table likely looks like: > > - INSERT > > - UPDATE > > - DELETE > > for every row. But tomorrow I can do

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-10 Thread James Coleman
One thing I just noticed and had a question about: in preparePresortedCols (which sets up a function call context), do we need to call pg_proc_aclcheck? James

execExprInterp() questions / How to improve scalar array op expr eval?

2020-04-11 Thread James Coleman
#x27;d be interested in that too. Side question: when we do: arr = DatumGetArrayTypeP(*op->resvalue); in ExecEvalScalarArrayOp() is that going to be expensive each time through a seq scan? Or is it (likely) going to resolve to an already in-memory array and effectively be the cost of retrieving that pointer? James

Re: execExprInterp() questions / How to improve scalar array op expr eval?

2020-04-11 Thread James Coleman
On Sat, Apr 11, 2020 at 2:01 PM Andres Freund wrote: > > Hi, > > > Tom, CCing you because of expanded datum question at bottom. > > > On 2020-04-11 08:58:46 -0400, James Coleman wrote: > > - Does the execExpr/execExprInterp framework allow a scalar array op > &

Re: execExprInterp() questions / How to improve scalar array op expr eval?

2020-04-11 Thread James Coleman
On Sat, Apr 11, 2020 at 3:33 PM Tom Lane wrote: > > Andres Freund writes: > > On 2020-04-11 08:58:46 -0400, James Coleman wrote: > >> - Does the execExpr/execExprInterp framework allow a scalar array op > >> to get an already expanded array (unless I'm mis

Re: execExprInterp() questions / How to improve scalar array op expr eval?

2020-04-11 Thread James Coleman
On Sat, Apr 11, 2020 at 3:57 PM James Coleman wrote: > .. > > It seems like this might be somewhat related to the currently-moribund > > patch to allow caching of the values of stable subexpressions from > > one execution to the next. If we had that infrastructure you could

Re: execExprInterp() questions / How to improve scalar array op expr eval?

2020-04-12 Thread James Coleman
On Sat, Apr 11, 2020 at 5:32 PM Andres Freund wrote: > > Hi, > > On 2020-04-11 15:53:11 -0400, James Coleman wrote: > > On Sat, Apr 11, 2020 at 2:01 PM Andres Freund wrote: > > > > - If not, is there a way in that framework to know if the array expr > > >

Re: execExprInterp() questions / How to improve scalar array op expr eval?

2020-04-13 Thread James Coleman
on only on constants, re-planning as `IN (VALUES ...)`) is something reasonable enough relative to the amount of effort to be worth working on. James [1]: https://www.postgresql.org/message-id/19001.1178823208%40sss.pgh.pa.us

Re: sqlsmith crash incremental sort

2020-04-14 Thread James Coleman
the prepunion.c code alongside cost_incremental_sort, it seems that we don't have access to the same level of information as the prepunion code (i.e., we're only looking at the result of the union, not the components of it), and trying descend down into it seems even more gross, so, see below... > Another option is to use something as simple as checking for Vars with > varno==0 in cost_incremental_sort() and ignoring them somehow. We could > simply use some arbitrary estimate - by assuming the rows are unique or > something like that. Yes, I agree it's pretty ugly and I'd much rather > find a way to generate something sensible, but I'm not even sure we can > generate good estimate when doing UNION of data from different relations > and so on. The attached (ugly) patch does this. ...therefore I think this is worth proceeding with. James

Re: [DOC] Document concurrent index builds waiting on each other

2020-04-15 Thread James Coleman
IC from blocking other CICs, but it wouldn't solve the problem of CIC blocking vacuum on unrelated tables, right? Perhaps that's orthogonal though. James

Re: Incremental sorts and EXEC_FLAG_REWIND

2020-04-15 Thread James Coleman
on s.a = t.a where t.a in (1, 2); > > Alexander, Tomas, any thoughts? I'll try to respond more fully later today when I can dig up the specific change. In the meantime, your question is primarily about making sure the code/comments/etc. are consistent and not a behavioral problem or failure you've seen in testing? James

Re: sqlsmith crash incremental sort

2020-04-15 Thread James Coleman
On Wed, Apr 15, 2020 at 10:47 AM Tomas Vondra wrote: > > On Tue, Apr 14, 2020 at 01:16:33PM -0400, James Coleman wrote: > >On Sun, Apr 12, 2020 at 8:09 PM Tomas Vondra > > wrote: > >> > >> On Sun, Apr 12, 2020 at 12:44:45AM +0200, Tomas Vondra wrote: > >&

Re: Incremental sorts and EXEC_FLAG_REWIND

2020-04-15 Thread James Coleman
On Wed, Apr 15, 2020 at 11:02 AM James Coleman wrote: > > On Tue, Apr 14, 2020 at 2:53 AM Michael Paquier wrote: > > > > Hi, > > > > When initializing an incremental sort node, we have the following as > > of ExecInitIncrementalSort(): > > /* >

Re: [DOC] Document concurrent index builds waiting on each other

2020-04-15 Thread James Coleman
On Wed, Apr 15, 2020 at 6:31 PM Andres Freund wrote: > > Hi, > > On 2020-04-15 09:31:58 -0400, James Coleman wrote: > > On Wed, Mar 25, 2020 at 3:58 PM Andres Freund wrote: > > > On 2020-03-25 16:30:10 -0300, Alvaro Herrera wrote: > > > > I posted this

Re: sqlsmith crash incremental sort

2020-04-16 Thread James Coleman
oo >>-> Seq Scan on foo foo_1 >> (6 rows) >> > > Attached is what I'm thinking about this optimization. Does it make any > sense? Shouldn't this go one either a new thread or on the thread for the patch Tomas was referencing (by Teodor I believe)? Or are you saying you believe this patch guarantees we never see this problem in incremental sort costing? James

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-16 Thread James Coleman
On Fri, Apr 10, 2020 at 10:12 AM James Coleman wrote: > > One thing I just noticed and had a question about: in > preparePresortedCols (which sets up a function call context), do we > need to call pg_proc_aclcheck? Background: this came up because I noticed that pg_proc_aclcheck is c

Re: sqlsmith crash incremental sort

2020-04-16 Thread James Coleman
On Thu, Apr 16, 2020 at 8:54 PM Tomas Vondra wrote: > > On Wed, Apr 15, 2020 at 11:26:12AM -0400, James Coleman wrote: > >On Wed, Apr 15, 2020 at 10:47 AM Tomas Vondra > > wrote: > >> > >> ... > >> > >> Yeah. And I'm not even sure havin

Re: [DOC] Document concurrent index builds waiting on each other

2020-04-16 Thread James Coleman
On Thu, Apr 16, 2020 at 6:12 PM Andres Freund wrote: > > Hi, > > On 2020-04-15 21:44:48 -0400, James Coleman wrote: > > On Wed, Apr 15, 2020 at 6:31 PM Andres Freund wrote: > > > If it's about the xmin horizon for vacuum: I think we could probably > > >

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-16 Thread James Coleman
On Thu, Apr 16, 2020 at 1:10 PM Tom Lane wrote: > > James Coleman writes: > > On Fri, Apr 10, 2020 at 10:12 AM James Coleman wrote: > >> One thing I just noticed and had a question about: in > >> preparePresortedCols (which sets up a function call con

Re: execExprInterp() questions / How to improve scalar array op expr eval?

2020-04-17 Thread James Coleman
On Mon, Apr 13, 2020 at 10:40 AM James Coleman wrote: > > I've read through all of the previous discussions related to stable > subexpression caching, and I'm planning to send a summary email with > all of those links in one place. > > But I also happened to stumble

Summary: State of Caching Stable Subexpressions

2020-04-17 Thread James Coleman
resolved the issue, but seemed like a pretty significant (and perceptually unnecessary) gotcha. I'm hoping collating this all in one place is helpful; at the very least it will be helpful to me as a reference should I find the time to push this forward some more. James [1]: https://ww

Re: sqlsmith crash incremental sort

2020-04-18 Thread James Coleman
ORTED is preferred then? And choose some value like "1/2 of the normal DEFAULT_NUM_DISTINCT groups" or some such? James

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-19 Thread James Coleman
On Sat, Apr 18, 2020 at 10:36 PM Justin Pryzby wrote: > > On Tue, Apr 07, 2020 at 10:53:05AM -0500, Justin Pryzby wrote: > > On Tue, Apr 07, 2020 at 08:40:30AM -0400, James Coleman wrote: > > > > And, should it use two spaces before "Sort Method", "Memory&qu

Re: Incremental sorts and EXEC_FLAG_REWIND

2020-04-19 Thread James Coleman
On Wed, Apr 15, 2020 at 2:04 PM James Coleman wrote: > > On Wed, Apr 15, 2020 at 11:02 AM James Coleman wrote: > > > > On Tue, Apr 14, 2020 at 2:53 AM Michael Paquier wrote: > > > > > > Hi, > > > > > > When initializing an in

Binary search in ScalarArrayOpExpr for OR'd constant arrays

2020-04-20 Thread James Coleman
000) n(i) where i in (<1000 random integers in the series>) shows ~30ms for the patch versus ~640ms on master. James [1]: https://www.postgresql.org/message-id/flat/CAAaqYe-UQBba7sScrucDOyHb7cDoNbWf_rcLrOWeD4ikP3_qTQ%40mail.gmail.com From 08742543d7865d5f25c24c26bf1014924035c9eb Mon Sep 17 00

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2020-04-23 Thread James Coleman
On Thu, Apr 23, 2020 at 8:47 AM Tomas Vondra wrote: > > On Mon, Apr 20, 2020 at 09:27:34PM -0400, James Coleman wrote: > >Over in "execExprInterp() questions / How to improve scalar array op > >expr eval?" [1] I'd mused about how we might be able to optimi

Re: [PATCH] Fix division by zero (explain.c)

2020-04-23 Thread James Coleman
on my part, and the patch looks correct to me. Tomas: agreed? Thanks, James

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2020-04-24 Thread James Coleman
On Thu, Apr 23, 2020 at 10:55 AM Tomas Vondra wrote: > > On Thu, Apr 23, 2020 at 09:02:26AM -0400, James Coleman wrote: > >On Thu, Apr 23, 2020 at 8:47 AM Tomas Vondra > > wrote: > >> > >> On Mon, Apr 20, 2020 at 09:27:34PM -0400, James Coleman wrote: >

Re: Incremental sorts and EXEC_FLAG_REWIND

2020-04-24 Thread James Coleman
On Sun, Apr 19, 2020 at 12:14 PM James Coleman wrote: > > On Wed, Apr 15, 2020 at 2:04 PM James Coleman wrote: > > > > On Wed, Apr 15, 2020 at 11:02 AM James Coleman wrote: > > > > > > On Tue, Apr 14, 2020 at 2:53 AM Michael Paquier > > > wr

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2020-04-24 Thread James Coleman
On Fri, Apr 24, 2020 at 5:55 PM Tomas Vondra wrote: > > On Fri, Apr 24, 2020 at 09:38:54AM -0400, James Coleman wrote: > >On Thu, Apr 23, 2020 at 10:55 AM Tomas Vondra > > wrote: > >> > >> On Thu, Apr 23, 2020 at 09:02:26AM -0400, James Coleman wrote: > &g

<    1   2   3   4   5   6   7   >