Re: Coccinelle for PostgreSQL development [1/N]: coccicheck.py

2025-03-11 Thread Mats Kindahl
On Sun, Mar 2, 2025 at 2:26 PM Mats Kindahl wrote: > On Sat, Jan 18, 2025 at 8:44 PM Mats Kindahl wrote: > >> On Tue, Jan 14, 2025 at 4:19 PM Aleksander Alekseev < >> aleksan...@timescale.com> wrote: >> >>> IMO the best solution would be re-submitting a

Re: Coccinelle for PostgreSQL development [1/N]: coccicheck.py

2025-03-02 Thread Mats Kindahl
On Sat, Jan 18, 2025 at 8:44 PM Mats Kindahl wrote: > On Tue, Jan 14, 2025 at 4:19 PM Aleksander Alekseev < > aleksan...@timescale.com> wrote: > >> IMO the best solution would be re-submitting all the patches to this >> thread. Also please make sure the patchset is r

Re: Coccinelle for PostgreSQL development [1/N]: coccicheck.py

2025-01-19 Thread Mats Kindahl
On Sun, Jan 19, 2025 at 2:10 AM Michael Paquier wrote: > On Sat, Jan 18, 2025 at 08:44:00PM +0100, Mats Kindahl wrote: > > For PostgreSQL 16, Peter extended the palloc()/pg_malloc() interface in > > commit 2016055a92f to provide more type-safety, but these functions are > n

Re: Coccinelle for PostgreSQL development [4/N]: correcting palloc() use

2025-01-08 Thread Mats Kindahl
On Wed, Jan 8, 2025 at 10:02 PM Peter Eisentraut wrote: > On 07.01.25 20:49, Mats Kindahl wrote: > > This is the first example semantic patch and shows how to capture and > > fix a common problem. > > > > If you use an palloc() to allocate memory for an object (or an a

Re: Coccinelle for PostgreSQL development [1/N]: coccicheck.py

2025-01-08 Thread Mats Kindahl
On Wed, Jan 8, 2025 at 11:42 AM Andrew Dunstan wrote: > > On 2025-01-07 Tu 2:44 PM, Mats Kindahl wrote: > > I got some time over during the holidays, so I spent some of it > > doing something I've been thinking about for a while. > > > > For those of you that a

Re: Coccinelle for PostgreSQL development [2/N]: autoconf support [RESEND]

2025-01-07 Thread Mats Kindahl
On Tue, Jan 7, 2025 at 8:56 PM Daniel Gustafsson wrote: > On 7 Jan 2025, at 20:54, Mats Kindahl wrote: > > On Tue, Jan 7, 2025 at 8:51 PM Daniel Gustafsson wrote: > >> > On 7 Jan 2025, at 20:47, Mats Kindahl wrote: >> >> > This second patch adds support fo

Re: Coccinelle for PostgreSQL development [2/N]: autoconf support [RESEND]

2025-01-07 Thread Mats Kindahl
On Tue, Jan 7, 2025 at 8:51 PM Daniel Gustafsson wrote: > > On 7 Jan 2025, at 20:47, Mats Kindahl wrote: > > > This second patch adds support for coccicheck > > Forgive me if I'm daft, but I feel there is a slight lack of context here. > What is coccicheck, what

Coccinelle for PostgreSQL development [4/N]: correcting palloc() use

2025-01-07 Thread Mats Kindahl
an "LLVMBasicBlockRef*", which strictly speaking is not correct (it should be "sizeof(LLVMBasicBlockRef)"). However, since they are both pointers, there is no risk of incorrect allocation size. -- Best wishes, Mats Kindahl, Timescale From eedf99b78a0e63b4de6db14617091f18adbd0d83

Coccinelle for PostgreSQL development [3/N]: meson support

2025-01-07 Thread Mats Kindahl
extra flags: SPFLAGS=--debug ninja coccicheck-context [1]: https://www.postgresql.org/docs/current/install-meson.html -- Best wishes, Mats Kindahl, Timescale From bb908e22e7a255a66b738f9180f4c6aa97e85624 Mon Sep 17 00:00:00 2001 From: Mats Kindahl Date: Wed, 1 Jan 2025 14:15:51 +0100 Subje

Coccinelle for PostgreSQL development [2/N]: autoconf support [RESEND]

2025-01-07 Thread Mats Kindahl
select a different directory to apply the semantic patches to, and COCCI to use a single specific semantic patch rather than all available. I have not added support for this right now, but if you think this is valuable, it should be straightforward to add. -- Best wishes, Mats Kindahl, Timescale

Coccinelle for PostgreSQL development [2/N]: autoconf support

2025-01-07 Thread Mats Kindahl
patches to, and COCCI to use a single specific semantic patch rather than all available. I have not added support for this right now, but if you think this is valuable, it should be straightforward to add. -- Best wishes, Mats Kindahl, Timescale

Coccinelle for PostgreSQL development [1/N]: coccicheck.py

2025-01-07 Thread Mats Kindahl
options to the execution of that semantic patch, respectively. -- Best wishes, Mats Kindahl, Timescale From 55f5caba3d6cb88e3729985571286c16171f36b3 Mon Sep 17 00:00:00 2001 From: Mats Kindahl Date: Sun, 29 Dec 2024 19:35:58 +0100 Subject: Add initial coccicheck script The coccicheck.py script can be u

Re: Potential ABI breakage in upcoming minor releases

2024-11-15 Thread Mats Kindahl
break if you pass it back and forth between extension and server code (where sizeof(ResultRelInfo) is different). For this case, a List might be better since it will just contain pointers and the extension will in that case just not read the added fields. -- Best wishes, Mats Kindahl, Timescale

Re: Potential ABI breakage in upcoming minor releases

2024-11-15 Thread Mats Kindahl
aledb avoid a rebuild. The target audience > would > be someone who can get a new PostgreSQL build but can't get a new > timescaledb > build. That feels like a small audience. What's missing in that analysis? > > Going forward, for struct field additions, I plan to make my own patches > more > ABI-breakage-averse than the postgr.es/c/e54a42a standard. > > > -- Best wishes, Mats Kindahl, Timescale

Re: Potential ABI breakage in upcoming minor releases

2024-11-15 Thread Mats Kindahl
rc/compression/compression.c-1562- Relation single_index_relation; > > I find this rather suspect. > Yes, I agree, trying to figure out other ways to deal with this piece of code. It is not possible to make a copy of the ResultRelInfo node here since copyObject() does not support T_ResultRelInfo. Trying to use CatalogOpenIndexes() causes another warning. > -- > Álvaro Herrera PostgreSQL Developer — > https://www.EnterpriseDB.com/ > "La espina, desde que nace, ya pincha" (Proverbio africano) > > > -- Best wishes, Mats Kindahl, Timescale

Re: Potential ABI breakage in upcoming minor releases

2024-11-15 Thread Mats Kindahl
ay. > I think it was mentioned elsewhere, but this wouldn't be a problem if makeNode was not a macro. -- Best wishes, Mats Kindahl, Timescale

Re: Use streaming read API in ANALYZE

2024-09-19 Thread Mats Kindahl
On Wed, Sep 18, 2024 at 5:13 AM Thomas Munro wrote: > On Sun, Sep 15, 2024 at 12:14 AM Mats Kindahl wrote: > > I used the combination of your patch and making the computation of > vacattrstats for a relation available through the API and managed to > implement something that I

Re: Use streaming read API in ANALYZE

2024-09-14 Thread Mats Kindahl
On Fri, Sep 13, 2024 at 10:10 AM Mats Kindahl wrote: > On Tue, Sep 10, 2024 at 6:04 AM Thomas Munro > wrote: > >> On Tue, Sep 10, 2024 at 10:27 AM Thomas Munro >> wrote: >> > Mats, what do you think about >> > this? (I haven't tried to preserve th

Re: Use streaming read API in ANALYZE

2024-09-13 Thread Mats Kindahl
ybe > the redesign work will result in something completely > different/better... just a thought... } > I'll take a look at the thread. I really think the ReadStream abstraction is a good step in the right direction. -- Best wishes, Mats Kindahl, Timescale

Re: Use streaming read API in ANALYZE

2024-09-13 Thread Mats Kindahl
re to talk about.) > Yes, they are, so this is kind-of-a-hack-to-get-it-roughly-correct. The ideal scenario would be to be able to run the same analysis of that is done in do_analyze_rel on the "hidden" relation to get an accurate targetrows. This is what I am trying now with the attached patch. > > . o O { We need that wiki page listing TAMs with links to the open > source ones... } -- Best wishes, Mats Kindahl, Timescale 0001-Add-function-to-compute-vacuum-attribute-statistics.patch.v1 Description: Binary data

Re: Use streaming read API in ANALYZE

2024-09-10 Thread Mats Kindahl
On Thu, Sep 5, 2024 at 11:12 AM Thomas Munro wrote: > On Thu, Sep 5, 2024 at 6:45 PM Mats Kindahl wrote: > > Forgive me for asking, but I am not entirely sure why the ReadStream > struct is opaque. The usual reasons are: > > > > You want to provide an ABI to allow ext

Re: Use streaming read API in ANALYZE

2024-09-04 Thread Mats Kindahl
paque, so this could be the reason. Is it either of these reasons, or is there another reason? Making the ReadStream API non-opaque (that is, moving the definition to the header file) would at least solve our problem (unless I am mistaken). However, I am ignorant about long-term plans which might affect this, so there might be a good reason to revert it for reasons I am not aware of. -- Best wishes, Mats Kindahl, Timescale

Re: Use streaming read API in ANALYZE

2024-08-29 Thread Mats Kindahl
On Sat, Aug 24, 2024 at 5:31 AM Thomas Munro wrote: > On Thu, Aug 22, 2024 at 7:31 PM Mats Kindahl wrote: > > The alternate version proposed by Nazir allows you to deide which > interface to use. > > Reverting the patch entirely would also solve the problem. > After diggi

Re: Use streaming read API in ANALYZE

2024-08-22 Thread Mats Kindahl
ns at different stages (For example, variable ring_size, needed to set up the buffer access strategy is computed in ExecVacuum(); variable targrows, used to set up the buffer sampler, is computed inside acquire_sample_rows(), which in turn requires to decide what attributes to analyze, which is compu

Re: Table AM Interface Enhancements

2024-05-23 Thread Mats Kindahl
aïve assumption of what slot implementation should be used, but that is a separate discussion.) Best wishes, Mats Kindahl

Re: Hooking into ExplainOneQuery() complicated by missing standard_ExplainOneQuery

2024-03-11 Thread Mats Kindahl
> to have this standard explain wrapper at the beginning of next week. > > If others have any comments, feel free. > > Well, done as of a04ddd077e61. > Thanks Michael! -- Best wishes, Mats Kindahl, Timescale

Re: Hooking into ExplainOneQuery() complicated by missing standard_ExplainOneQuery

2024-03-05 Thread Mats Kindahl
dd something to the node explain and the > summary, if only because they can significantly influence the planner > and executor's behaviour. > > [1] > > https://www.postgresql.org/message-id/flat/6cd5caa7-06e1-4460-bf35-00a59da3f677%40garret.ru This is an excellent example of where such a hook would be useful. -- Best wishes, Mats Kindahl, Timescale

Re: Hooking into ExplainOneQuery() complicated by missing standard_ExplainOneQuery

2024-03-04 Thread Mats Kindahl
own and also allow this information to propagate upwards. We happen to have buffer information right now, but allowing something similar to be added dynamically by extending ExplainNode and passing down a callback to standard_ExplainOneQuery. Best wishes, Mats Kindahl

Hooking into ExplainOneQuery() complicated by missing standard_ExplainOneQuery

2024-03-04 Thread Mats Kindahl
summarize information from custom scan nodes, but I had no immediate use for that so did not add it here. I would still be interested in hearing if you think this is something that would be useful to the community. Best wishes, Mats Kindahl, Timescale From eaab4d7c088ff3ee9b0e6ec3de96677bafd184c0 Mon

Re: glibc qsort() vulnerability

2024-02-17 Thread Mats Kindahl
On Fri, Feb 16, 2024 at 9:09 PM Nathan Bossart wrote: > On Fri, Feb 16, 2024 at 01:45:52PM +0100, Mats Kindahl wrote: > > Looks good to me. > > Committed. > Thanks Nathan! > > -- > Nathan Bossart > Amazon Web Services: https://aws.amazon.com >

Re: glibc qsort() vulnerability

2024-02-16 Thread Mats Kindahl
On Fri, Feb 16, 2024 at 12:32 AM Nathan Bossart wrote: > Here is what I have staged for commit. > Looks good to me. Checked that all of the comparisons are in the expected order, except inside compDESC, cmp_lsn, and resource_priority_cmp, where the order is reversed. Best wishes, Mats K

Re: glibc qsort() vulnerability

2024-02-13 Thread Mats Kindahl
< b ? true : a > b ? false : false a < b ? true : a > b ? false : false a < b ? true : false a < b When it comes to (a < b) - (a > b) there are no such rules added since it is not a very common case. Clang fares better for this case and at least shows the two alternatives as equal. Maybe we should change to use the original version equivalent to the inline function above since that works better with surrounding code? Best wishes, Mats Kindahl > > Greetings, > > Andres >

Re: glibc qsort() vulnerability

2024-02-12 Thread Mats Kindahl
On Mon, Feb 12, 2024 at 4:57 PM Nathan Bossart wrote: > On Sun, Feb 11, 2024 at 03:44:42PM +0100, Mats Kindahl wrote: > > On Sat, Feb 10, 2024 at 9:53 PM Nathan Bossart > > > wrote: > >> and I think we should expand on some of the commentary in int.h. > >> Fo

Re: glibc qsort() vulnerability

2024-02-11 Thread Mats Kindahl
On Sat, Feb 10, 2024 at 9:53 PM Nathan Bossart wrote: > On Sat, Feb 10, 2024 at 08:59:06AM +0100, Mats Kindahl wrote: > > Split the code into two patches: one that just adds the functions > > (including the new pg_cmp_size()) to common/int.h and one that starts > using >

Re: glibc qsort() vulnerability

2024-02-09 Thread Mats Kindahl
On Fri, Feb 9, 2024 at 9:08 PM Nathan Bossart wrote: > On Fri, Feb 09, 2024 at 08:43:21PM +0100, Mats Kindahl wrote: > > QQ: right now it looks like this: > > > > static inline int > > pg_cmp_u16(uint16 a, uint16 b) > > { > > > > return (

Re: glibc qsort() vulnerability

2024-02-09 Thread Mats Kindahl
On Fri, Feb 9, 2024 at 8:26 PM Mats Kindahl wrote: > On Fri, Feb 9, 2024 at 5:24 PM Nathan Bossart > wrote: > >> On Fri, Feb 09, 2024 at 08:52:26AM +0100, Mats Kindahl wrote: >> > Here is a new version introducing pg_cmp_s32 and friends and use them >> > instea

Re: glibc qsort() vulnerability

2024-02-09 Thread Mats Kindahl
On Fri, Feb 9, 2024 at 5:27 PM Tom Lane wrote: > Nathan Bossart writes: > > On Fri, Feb 09, 2024 at 08:52:26AM +0100, Mats Kindahl wrote: > >> The types "int" and "size_t" are treated as s32 and u32 respectively > since > >> that seems to be t

Re: glibc qsort() vulnerability

2024-02-09 Thread Mats Kindahl
On Fri, Feb 9, 2024 at 5:27 PM Tom Lane wrote: > Nathan Bossart writes: > > On Fri, Feb 09, 2024 at 08:52:26AM +0100, Mats Kindahl wrote: > >> The types "int" and "size_t" are treated as s32 and u32 respectively > since > >> that seems to be t

Re: glibc qsort() vulnerability

2024-02-09 Thread Mats Kindahl
On Fri, Feb 9, 2024 at 5:24 PM Nathan Bossart wrote: > On Fri, Feb 09, 2024 at 08:52:26AM +0100, Mats Kindahl wrote: > > Here is a new version introducing pg_cmp_s32 and friends and use them > > instead of the INT_CMP macro introduced before. It also moves the > > definition

Re: glibc qsort() vulnerability

2024-02-08 Thread Mats Kindahl
since that seems to be the case for most of the code, even if strictly not correct (size_t can be an unsigned long int for some architecture). I also noted that in many situations size_t values are treated as "int" so there is an overflow risk here, even if small. For example, the re

Re: glibc qsort() vulnerability

2024-02-08 Thread Mats Kindahl
> > > I think it's worth following int.h's pattern of including > [s]igned/[u]nsigned > > in the name, an efficient implementation for signed might not be the > same as > > for unsigned. And if we use static inlines, we need to do so for correct > > semantics anyway. > > Seems reasonable to me. > Agree. Best wishes, Mats Kindahl > > -- > Nathan Bossart > Amazon Web Services: https://aws.amazon.com >

Re: glibc qsort() vulnerability

2024-02-08 Thread Mats Kindahl
On Thu, Feb 8, 2024 at 7:44 PM Tom Lane wrote: > Nathan Bossart writes: > > On Thu, Feb 08, 2024 at 02:16:11PM +0100, Mats Kindahl wrote: > >> +/* > >> + * Compare two integers and return -1, 0, or 1 without risking > overflow. > >> + * > >> + * T

Re: glibc qsort() vulnerability

2024-02-08 Thread Mats Kindahl
integers and other common types. Also noted it is quite common to have this pattern in various places to do lexicographic sort of multiple values and continue the comparison if they are equal. Not sure if that is something we should look at. Best wishes, Mats Kindahl > > -- > Nathan Bos

Re: glibc qsort() vulnerability

2024-02-08 Thread Mats Kindahl
On Thu, Feb 8, 2024 at 12:01 PM Mats Kindahl wrote: > On Wed, Feb 7, 2024 at 9:56 PM Nathan Bossart > wrote: > >> On Wed, Feb 07, 2024 at 08:46:56PM +0200, Heikki Linnakangas wrote: >> > Doesn't hurt to fix the comparison functions, and +1 on using the same >

Re: glibc qsort() vulnerability

2024-02-08 Thread Mats Kindahl
On Thu, Feb 8, 2024 at 4:08 AM Nathan Bossart wrote: > Mats, I apologize for steamrolling a bit here. I'll take a step back into > a reviewer role. > No worries. :) > > -- > Nathan Bossart > Amazon Web Services: https://aws.amazon.com >

Re: glibc qsort() vulnerability

2024-02-08 Thread Mats Kindahl
On Wed, Feb 7, 2024 at 9:56 PM Nathan Bossart wrote: > On Wed, Feb 07, 2024 at 08:46:56PM +0200, Heikki Linnakangas wrote: > > Doesn't hurt to fix the comparison functions, and +1 on using the same > > pattern everywhere. > > I attached a new version of the patch with some small adjustments. I >

Re: glibc qsort() vulnerability

2024-02-07 Thread Mats Kindahl
can cause an overflow. This method is used for some int16 as well but since standard conversions in C will perform the arithmetics in "int" precision, this cannot overflow, so added a comment there. It might still be a good idea to follow the same pattern for the int16 routines, but sinc

Re: glibc qsort() vulnerability

2024-02-07 Thread Mats Kindahl
On Tue, Feb 6, 2024 at 4:11 PM Tom Lane wrote: > Mats Kindahl writes: > > There is a bug in glibc's qsort() algorithm that runs the risk of > creating > > an out-of-bounds error if the comparison function is not transitive, for > > example, if subtraction is

glibc qsort() vulnerability

2024-02-06 Thread Mats Kindahl
e few cases and add a comment to one case that was not clear that it could not overflow. Best wishes, Mats Kindahl, Timescale From 83e2f14ab259f568034b07c2f99e34c6dff2c7b5 Mon Sep 17 00:00:00 2001 From: Mats Kindahl Date: Tue, 6 Feb 2024 14:53:53 +0100 Subject: Ensure comparison functions are trans

Re: Add test module for Table Access Method

2024-01-16 Thread Mats Kindahl
I wanted to implement is a table access method that you can use to test the interface, something like a "mock TAM" where you can programmatically decide on the responses to unit-test the API. I was thinking that you could implement a framework that allows you to implement the T

Re: Add test module for Table Access Method

2024-01-16 Thread Mats Kindahl
could provide a proxy TAM for a heap TAM > which does nothing but logging used TAM methods, its arguments and > return values. This would be a good example and also potentially can > be used as a debugging tool. > We wrote a table access method for experimenting with and

Re: Table AM and DROP TABLE [ Was: Table AM and DDLs]

2022-11-16 Thread Mats Kindahl
verage the full power of this concept. On Tue, Aug 2, 2022 at 1:44 AM Andres Freund wrote: > Hi, > > On 2021-04-05 21:57:12 +0200, Mats Kindahl wrote: > >2. In the storage layer, the function RelationDropStorage is called, > >which will record the table to be droppe

Re: Table AM and DROP TABLE [ Was: Table AM and DDLs]

2022-07-29 Thread Mats Kindahl
t many of these problems can be worked around by placing restrictions on how DDLs can be used in transactions, but that would create unnecessary restrictions for the end-user. It might also be possible to find implementation workarounds by placing code at strategic points in the implementation, but this is error prone and the risk of making an error is higher. Best wishes, Mats Kindahl > > - Heikki >

Re: RFC: Table access methods and scans

2021-06-03 Thread Mats Kindahl
Hi Jeff, On Fri, Jun 4, 2021 at 2:52 AM Jeff Davis wrote: > Hi, > > On Wed, 2021-03-31 at 22:10 +0200, Mats Kindahl wrote: > > As an example of how this is useful, I noticed the work by Heikki and > > Ashwin [1], where they return a `TableScanDesc` that contains >

Table AM and DROP TABLE [ Was: Table AM and DDLs]

2021-04-05 Thread Mats Kindahl
On Mon, Mar 22, 2021 at 12:16 AM Andres Freund wrote: > Hi, > > On 2021-03-03 22:15:18 +0100, Mats Kindahl wrote: > > On Tue, Feb 23, 2021 at 2:11 AM Andres Freund > wrote: > > Thanks for the answer and sorry about the late reply. > > Mine is even later ;) >

RFC: Table access methods and scans

2021-03-31 Thread Mats Kindahl
ngly enough, `ScanKey` is generated for `IndexScan` and I think that the same approach could be used for sequential scans: pick out the quals that can be used for filtering and offer them to the table access method through the `scan_begin` callback. Thoughts around this? Best wishes, Mats Kindahl [1

Re: Table AM and DDLs

2021-03-03 Thread Mats Kindahl
On Tue, Feb 23, 2021 at 2:11 AM Andres Freund wrote: > Hi, > Hi Andres, Thanks for the answer and sorry about the late reply. > On 2021-02-22 08:33:21 +0100, Mats Kindahl wrote: > > I started to experiment with the table access method interface to see if > it > > can

Table AM and DDLs

2021-02-21 Thread Mats Kindahl
might want to optimize the internal storage when the schema changes. I have not been able to find any discussions around this, but it seems like a natural thing to do with a table. Have I misunderstood how this works? Best wishes, Mats Kindahl Timescale From