Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-05 Thread Amit Kapila
On Thu, Jun 5, 2025 at 7:24 PM Feike Steenbergen wrote: > > On Thu, 5 Jun 2025 at 12:49, Peter Eisentraut wrote: > > I propose to address this by not allowing the use of user-defined > > functions in generation expressions for now. The attached patch > > implements this. This assumes that all b

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-05 Thread Robert Haas
On Thu, Jun 5, 2025 at 11:19 AM jian he wrote: > I think it will work. > because we already require the generated column expression to be > immutable functions. > > The above functions you mentioned are all not immutable. Hmm. I guess I have no evidence that we have built-in immutable functions t

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-05 Thread Christoph Berg
Re: Tom Lane > > Extending the idea, perhaps the check could be moved to run-time and > > recursively check that only immutable functions are called, including > > user-defined immutable functions? > > I don't think I'd trust that. UDFs can claim to be immutable but > be lying about it. That's w

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-05 Thread Tom Lane
Christoph Berg writes: > So the question is, are all built-in *immutable* functions safe? Perhaps. > Extending the idea, perhaps the check could be moved to run-time and > recursively check that only immutable functions are called, including > user-defined immutable functions? I don't think I'd

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-05 Thread Christoph Berg
Re: Robert Haas > I don't think this is sufficient to fix the problem. We have built-in > functions that are unsafe. These include LO functions like loread(), > lowrite(), lo_unlink(); functions that change session state like > set_config() and setseed(); functions that allow arbitrary query > exec

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-05 Thread jian he
On Thu, Jun 5, 2025 at 10:39 PM Robert Haas wrote: > > On Thu, Jun 5, 2025 at 6:49 AM Peter Eisentraut wrote: > > I propose to address this by not allowing the use of user-defined > > functions in generation expressions for now. The attached patch > > implements this. This assumes that all buil

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-05 Thread Robert Haas
On Thu, Jun 5, 2025 at 6:49 AM Peter Eisentraut wrote: > I propose to address this by not allowing the use of user-defined > functions in generation expressions for now. The attached patch > implements this. This assumes that all built-in functions are > trustworthy, for this purpose, which seem

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-05 Thread Feike Steenbergen
On Thu, 5 Jun 2025 at 12:49, Peter Eisentraut wrote: > I propose to address this by not allowing the use of user-defined > functions in generation expressions for now. The attached patch > implements this. This assumes that all built-in functions are > trustworthy, for this purpose, which seems

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-05 Thread Pavel Stehule
čt 5. 6. 2025 v 12:49 odesílatel Peter Eisentraut napsal: > On 23.05.25 10:43, Feike Steenbergen wrote: > > Attached is a sample exploit, that achieves this, key components: > > > > - the GENERATED column uses a user defined immutable function > > - this immutable function cannot ALTER ROLE (need

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-05 Thread Peter Eisentraut
ever. From 0a6fd83bc1b96a0a6a465d64ef06bec4f0a3e824 Mon Sep 17 00:00:00 2001 From: Peter Eisentraut Date: Thu, 5 Jun 2025 12:35:38 +0200 Subject: [PATCH] Restrict virtual columns to use built-in functions Just like selecting from a view is exploitable (CVE-2024-7348), selecting from a table with virtual generat

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-03 Thread Jeff Davis
On Tue, 2025-06-03 at 11:27 -0400, Robert Haas wrote: > That's true, but search_path manipulation is still enough to cause > quite a few problems. +1. The only defense is to declare the function with "SET search_path", but until recently, that was a major performance penalty for cheap functions. A

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-03 Thread Robert Haas
On Tue, Jun 3, 2025 at 10:11 AM Bruce Momjian wrote: > I think the two cases are slightly different. Our existing system has > users running triggers on tables that don't own as themselves, so the > table owner has full control over what is in the triggers. If we were > to switch it so users run

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-03 Thread Bruce Momjian
On Tue, Jun 3, 2025 at 08:58:58AM -0400, Robert Haas wrote: > On Mon, Jun 2, 2025 at 11:30 PM Tom Lane wrote: > > > That being said I would like to see it corrected everywhere. > > > > Yeah, one approach we could take here is to try to move the goalposts > > for this whole topic, understanding th

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-03 Thread Isaac Morland
On Mon, 2 Jun 2025 at 23:30, Tom Lane wrote: > Isaac Morland writes: > > My fix would > > be for check constraints, triggers, and view definitions to run as the > > owner of the object in question (constraint, trigger, or view or > > materialized view), essentially using the same facility as us

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-03 Thread Robert Haas
On Mon, Jun 2, 2025 at 11:30 PM Tom Lane wrote: > > That being said I would like to see it corrected everywhere. > > Yeah, one approach we could take here is to try to move the goalposts > for this whole topic, understanding that that will mean incompatible > changes as well as some performance lo

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-02 Thread Tom Lane
Isaac Morland writes: > On Mon, 2 Jun 2025 at 22:52, jian he wrote: > Do we consider INSERT associated with user defined function a security >> bug? > A very old issue for INSERT/UPDATE/DELETE, but until this patch not an > issue for SELECT from a table (although if I understand correctly earlie

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-02 Thread Isaac Morland
On Mon, 2 Jun 2025 at 22:52, jian he wrote: Do we consider INSERT associated with user defined function a security > bug? for > example, the following, INSERT with a check constraint. > [] If so, then it's a very old issue... > A very old issue for INSERT/UPDATE/DELETE, but until this pat

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-02 Thread jian he
ture-freeze. > > I'm leaning more and more to the position that we ought to revert > virtual generated columns for v18 and give ourselves breathing > room to design a proper fix for the security hazard. > Do we consider INSERT associated with user defined function a security

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-02 Thread Jeff Davis
he expression is obviously safe. > > I'm leaning more and more to the position that we ought to revert > virtual generated columns for v18 and give ourselves breathing > room to design a proper fix for the security hazard. Unfortunate, but I think I agree. Even if we do come up

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-02 Thread Tom Lane
sn't feel like something to be defining and implementing post-beta1. Even if it were not security-critical, the amount of complication involved is well past our standards for what can go in post-feature-freeze. I'm leaning more and more to the position that we ought to revert virtual gener

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-06-02 Thread Jeff Davis
On Thu, 2025-05-29 at 11:12 -0400, Tom Lane wrote: > Perhaps a compromise is to invent RunAsUser but only apply it to > virtual columns for now, leaving the view case as a research > project.  Then we aren't destroying the performance of any > existing queries. Could we instead check that the expr

Re: Virtual generated columns

2025-06-01 Thread jian he
ChangeRequiresRewrite sees the expanded > version of the expression — your proposed change fails to do that. > > Additionally, I think we also need to ensure that the virtual > generated columns are expanded before the expression is fed through > expression_planner, to ensure it can be successfully transformed into > an executable form. > > Hence, the attached patch. looks good to me.

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-29 Thread Bruce Momjian
On Thu, May 29, 2025 at 02:15:22PM -0400, Tom Lane wrote: > Feike Steenbergen writes: > > pg_restore may have issues though, as it will run these functions > > for GENERATED STORED columns? > > pg_restore is already fairly exposed, as it will run tables' CHECK > constraints, index expressions, et

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-29 Thread Matthias van de Meent
On Thu, 29 May 2025 at 20:30, Tom Lane wrote: > > Matthias van de Meent writes: > > On Thu, 29 May 2025 at 15:44, Robert Haas wrote: > >> But so far - apart from this feature - we > >> have managed to avoid making it categorically unsafe for the superuser > >> to run "SELECT * FROM table" > > >

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-29 Thread Matthias van de Meent
* FROM table. This also includes outputting alternative data sources, or e.g. calling a user-defined SECURITY INVOKER function. PG18 still seems to have support for CREATE RULE, so virtual generated columns don't create a completely new security issue (blind SELECT * FROM user_defined_table w

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-29 Thread Tom Lane
Matthias van de Meent writes: > On Thu, 29 May 2025 at 15:44, Robert Haas wrote: >> But so far - apart from this feature - we >> have managed to avoid making it categorically unsafe for the superuser >> to run "SELECT * FROM table" > With CREATE RULE [0], a table owner can redefine what happens

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-29 Thread Tom Lane
Feike Steenbergen writes: > pg_restore may have issues though, as it will run these functions > for GENERATED STORED columns? pg_restore is already fairly exposed, as it will run tables' CHECK constraints, index expressions, etc. I don't think GENERATED STORED makes that picture much worse. As

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-29 Thread Feike Steenbergen
On Thu, 29 May 2025 at 15:43, Robert Haas wrote: > that would also imply, > for example, that there's no way to run a pg_dump without letting any > user on the system obtain superuser privileges. I checked, pg_dump seems safe, it doesn't extract the values, even when using --column-inserts. pg_r

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-29 Thread Tom Lane
"David G. Johnston" writes: > Just to make sure we are on the same page as to who IS supposed to be > "current_user" within these functions - it should be the table owner, right? If we could make that happen (ie, run the generated-column expressions as the table owner), it would likely be a suffi

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-29 Thread David G. Johnston
On Thu, May 29, 2025 at 6:43 AM Robert Haas wrote: > > Point being: this > feature will need to be fixed in some way that avoids further > expanding the set of things that a superuser must not ever do for fear > of giving away their privileges accidentally, or else it will need to > be reverted.

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-29 Thread Robert Haas
On Mon, May 26, 2025 at 10:52 AM Feike Steenbergen wrote: > On Mon, 26 May 2025 at 16:17, jian he wrote: > > calling exploit_generated.exploit by normal user or superuser the > > effects are different, > > that by definition is not IMMUTABLE. > > Yeah, i know this is *wrong* usage of IMMUTABLE, t

Re: Virtual generated columns

2025-05-28 Thread Richard Guo
e also need to ensure that the virtual generated columns are expanded before the expression is fed through expression_planner, to ensure it can be successfully transformed into an executable form. Hence, the attached patch. Thanks Richard v1-0001-Expand-virtual-generated-columns-for-ALTER-COLUMN.patch Description: Binary data

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-26 Thread Feike Steenbergen
On Mon, 26 May 2025 at 16:17, jian he wrote: > calling exploit_generated.exploit by normal user or superuser the > effects are different, > that by definition is not IMMUTABLE. Yeah, i know this is *wrong* usage of IMMUTABLE, the point is that a rogue regular user *can* use this pattern to become

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-26 Thread jian he
On Mon, May 26, 2025 at 4:56 PM Feike Steenbergen wrote: > > > > On Sat, 24 May 2025 at 15:43, jian he wrote: > > sorry, I am not fully sure what this means. a minimum sql reproducer would > > be > > great. > > The initial email contains a fully self-contained example of a regular user > becomin

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-26 Thread Feike Steenbergen
ch wrote: > SELECT is fairly unsafe. We ended up with commit 66e9444 (CVE-2024-7348) to > make secure use of SELECT feasible in released branches. It sounds like this > v18 feature may need changes like commit 66e9444. In other words, virtual > generated columns make a table into a hy

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-24 Thread David G. Johnston
On Saturday, May 24, 2025, jian he wrote: > On Sat, May 24, 2025 at 2:39 PM Feike Steenbergen > wrote: > > > > The loophole is this: > > > > - the generated virtual column can use a user-defined function > > - when running SELECT against that column by a superuser > > the function is called wi

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-24 Thread jian he
On Sat, May 24, 2025 at 2:39 PM Feike Steenbergen wrote: > > The loophole is this: > > - the generated virtual column can use a user-defined function > - when running SELECT against that column by a superuser > the function is called within the context of a superuser > - this in turn allows the

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-23 Thread Feike Steenbergen
On Fri, 23 May 2025 at 14:48, jian he wrote: > when you mark it as IMMUTABLE, postgres think it's IMMUTABLE, but in this case > exploit_generated.exploit(i int) clearly is not an IMMUTABLE function. > > Only IMMUTABLE functions are allowed in generated expressions, > but you can still misuse it by

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-23 Thread jian he
On Fri, May 23, 2025 at 4:43 PM Feike Steenbergen wrote: > > > Hi, > > While evaluating the PostgreSQL 18 beta, I had a thought experiment where I > thought it might be possible to use the new virtual generated columns to gain > superuser privileges for a regular user. >

pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-23 Thread Feike Steenbergen
Hi, While evaluating the PostgreSQL 18 beta, I had a thought experiment where I thought it might be possible to use the new virtual generated columns to gain superuser privileges for a regular user. Attached is a sample exploit, that achieves this, key components: - the GENERATED column uses a

Re: Virtual generated columns

2025-05-16 Thread jian he
(1); > > alter table vt alter column c type bigint using b + c; > > > > ERROR: XX000: unexpected virtual generated column reference > > LOCATION: CheckVarSlotCompatibility, execExprInterp.c:2410 > > Thank you for the report. It seems that we fail to expand refer

Re: Virtual generated columns

2025-05-16 Thread Richard Guo
ROR: XX000: unexpected virtual generated column reference > LOCATION: CheckVarSlotCompatibility, execExprInterp.c:2410 Thank you for the report. It seems that we fail to expand references to virtual generated columns in the NewColumnValues expression when altering tables. We might be able

Re: Virtual generated columns

2025-05-15 Thread Alexander Lakhin
Hello Richard, 22.02.2025 16:55, Richard Guo wrote: create table t (a int, b int); create table vt (a int, b int generated always as (a * 2)); insert into t values (1, 1); insert into vt values (1); # select 1 from t t1 where exists (select 1 from vt where exists (select t1.a from t t

Allow to collect statistics on virtual generated columns

2025-04-22 Thread Yugo Nagata
statistics on virtual generated columns ANALYZE expands virtual generated columns' expression, and collects statistics on evaluated values. In this approach, the statistics on virtual generated columns are collected in default, but ANALYZE on the table would become a bit expensive. (2) All

Re: Virtual generated columns

2025-02-24 Thread Richard Guo
On Mon, Feb 24, 2025 at 9:07 PM Dean Rasheed wrote: > On Mon, 24 Feb 2025 at 09:21, Richard Guo wrote: > > Here are the updated patches with revised comments and some tweaks to > > the commit messages. I plan to push them in one or two days. > LGTM. Pushed. Thanks all for working on this issu

Re: Virtual generated columns

2025-02-24 Thread Dean Rasheed
On Mon, 24 Feb 2025 at 09:21, Richard Guo wrote: > > Here are the updated patches with revised comments and some tweaks to > the commit messages. I plan to push them in one or two days. > LGTM. Regards, Dean

Re: Virtual generated columns

2025-02-24 Thread Richard Guo
though, so maybe we're good > > here. Still, I'll fix this later. > > > i also noticed this issue... > > some minor comments about v7. Thanks for reviewing. Here are the updated patches with revised comments and some tweaks to the commit messages. I plan to push

Re: Virtual generated columns

2025-02-23 Thread jian he
On Sat, Feb 22, 2025 at 11:12 PM Richard Guo wrote: > > On Sat, Feb 22, 2025 at 11:55 PM Richard Guo wrote: > > Attached are the updated patches to fix all the mentioned issues. I > > plan to push them early next week after staring at the code for a bit > > longer, barring any objections. > > Si

Re: Virtual generated columns

2025-02-22 Thread Richard Guo
On Sat, Feb 22, 2025 at 11:55 PM Richard Guo wrote: > Attached are the updated patches to fix all the mentioned issues. I > plan to push them early next week after staring at the code for a bit > longer, barring any objections. Sign... I neglected to make the change in 0001 that a Var newnode co

Re: Virtual generated columns

2025-02-22 Thread Richard Guo
On Sat, Feb 22, 2025 at 2:35 AM Dean Rasheed wrote: > On Fri, 21 Feb 2025 at 06:16, Richard Guo wrote: > > * The expansion of virtual generated columns occurs after subquery > > pullup, which can lead to issues. This was an oversight on my part. > > Initially, I believed i

Re: Virtual generated columns

2025-02-21 Thread Dean Rasheed
an Vars, considering dropped > columns. I think we need to check for this when generating the fields > for a RowExpr. Yes, good point. > * The expansion of virtual generated columns occurs after subquery > pullup, which can lead to issues. This was an oversight on my part. > Initial

Re: Virtual generated columns

2025-02-20 Thread Richard Guo
ces, it is possible that some fields are expanded as Consts rather than Vars, considering dropped columns. I think we need to check for this when generating the fields for a RowExpr. * The expansion of virtual generated columns occurs after subquery pullup, which can lead to issues. This was an

Re: Virtual generated columns

2025-02-20 Thread jian he
On Wed, Feb 19, 2025 at 11:25 PM Dean Rasheed wrote: > > One of the new regression tests fails, which actually appears to be a > pre-existing grouping sets bug, due to the fact that only non-Vars are > wrapped in PHVs. This bug can be triggered without virtual generated > colu

Re: Virtual generated columns

2025-02-19 Thread jian he
On Wed, Feb 19, 2025 at 11:25 PM Dean Rasheed wrote: > > On Wed, 19 Feb 2025 at 01:42, Dean Rasheed wrote: > > > > One thing I don't like about this is that it's introducing more code > > duplication between pullup_replace_vars() and > > ReplaceVarsFromTargetList(). Those already had a lot of cod

Re: Virtual generated columns

2025-02-19 Thread Dean Rasheed
wrapped in PHVs. This bug can be triggered without virtual generated columns: CREATE TABLE t (a int, b int); INSERT INTO t VALUES (1, 1); SELECT * FROM (SELECT a, a AS b FROM t) AS vt GROUP BY GROUPING SETS (a, b) HAVING b = 1; a | b ---+--- 1 | (1 row) whereas the result should be a | b

Re: Virtual generated columns

2025-02-18 Thread Dean Rasheed
On Tue, 18 Feb 2025 at 13:12, Richard Guo wrote: > > > It seems to me that, for a relation in the rangetable that has virtual > > generated columns, we can consider it a subquery to some extent. For > > instance, suppose we have a query: > > > > select ... fr

Re: Virtual generated columns

2025-02-18 Thread Richard Guo
On Tue, Feb 18, 2025 at 7:09 PM Richard Guo wrote: > It seems to me that, for a relation in the rangetable that has virtual > generated columns, we can consider it a subquery to some extent. For > instance, suppose we have a query: > > select ... from ... join t on ...; > >

Re: Virtual generated columns

2025-02-18 Thread Richard Guo
ed is a rough patch that moves the expansion of virtual > generated columns to the planner. It needs a lot more testing (and > some regression tests), but it does seem to fix all the issues > mentioned in this thread. Yeah, I believe this is the right way to go: virtual generated columns should be

Re: Virtual generated columns

2025-02-16 Thread jian he
ted column expansion in > > > the rewrite stage as is, > > > and wrap the expressions in PHVs if the virtual generated > > > columns come from the nullable side of an outer join. > > > > PlaceHolderVar looks like a fitting mechanism for this. But it's so far

Re: Virtual generated columns

2025-02-15 Thread Dean Rasheed
x27;s used. It seems pretty baked into how PHVs work that they should only be added by the planner, so I think that I agree with Richard -- virtual generated columns probably have to be expanded in the planner rather than the rewriter. > Maybe a short-term fix would be to error out if we find ou

Re: Virtual generated columns

2025-02-14 Thread Peter Eisentraut
On 13.02.25 14:06, jian he wrote: I didn't solve the out join semantic issue. i am wondering, can we do the virtual generated column expansion in the rewrite stage as is, and wrap the expressions in PHVs if the virtual generated columns come from the nullable side of an outer

Re: Virtual generated columns

2025-02-13 Thread jian he
; > NULL | NULL > > NULL | NULL > > (2 rows) > > Yeah, I also feel that the virtual generated columns should adhere to > outer join semantics, rather than being unconditionally replaced by > the generation expressions. But maybe I'm wrong. > > If that's the

Re: Virtual generated columns

2025-02-11 Thread Richard Guo
On Tue, Feb 11, 2025 at 10:34 AM Richard Guo wrote: > Yeah, I also feel that the virtual generated columns should adhere to > outer join semantics, rather than being unconditionally replaced by > the generation expressions. But maybe I'm wrong. > > If that's the case, th

Re: Virtual generated columns

2025-02-10 Thread Richard Guo
s. > I agree with Richard, the result seems incorrect. The right outcome should be: > gpadmin=# SELECT t2.a, t2.b FROM t t1 LEFT JOIN t t2 ON FALSE; > a | b > --+-- > NULL | NULL > NULL | NULL > (2 rows) Yeah, I also feel that the virtual generated columns should adher

Re: Virtual generated columns

2025-02-09 Thread Zhang Mingli
On Feb 10, 2025 at 12:53 +0800, jian he , wrote: > > please check attached. > > > BTW, I was curious about what happens if the replacement expression is > > constant, so I tried running the query below. > > > > CREATE TABLE t (a int, b int GENERATED ALWAYS AS (1 + 1)); > > INSERT INTO t VALUES (1);

Re: Virtual generated columns

2025-02-09 Thread jian he
On Mon, Feb 10, 2025 at 11:54 AM Richard Guo wrote: > > On Sun, Feb 9, 2025 at 7:02 PM Zhang Mingli wrote: > > On Feb 9, 2025 at 16:00 +0800, Alexander Lakhin , > > wrote: > > Please look at a planner error with a virtual generated column triggered > > by the following script: > > CREATE TABLE t

Re: Virtual generated columns

2025-02-09 Thread Richard Guo
On Sun, Feb 9, 2025 at 7:02 PM Zhang Mingli wrote: > On Feb 9, 2025 at 16:00 +0800, Alexander Lakhin , wrote: > Please look at a planner error with a virtual generated column triggered > by the following script: > CREATE TABLE t(a int, b int GENERATED ALWAYS AS (a * 1)); > > SELECT SUM(CASE WHEN t

Re: Virtual generated columns

2025-02-09 Thread Zhang Mingli
On Feb 9, 2025 at 16:00 +0800, Alexander Lakhin , wrote: > > Please look at a planner error with a virtual generated column triggered > by the following script: > CREATE TABLE t(a int, b int GENERATED ALWAYS AS (a * 1)); > > SELECT SUM(CASE WHEN t.b = 1 THEN 1 ELSE 1 END) OVER (PARTITION BY t.a) >

Re: Virtual generated columns

2025-02-09 Thread Alexander Lakhin
Hello Peter, 07.02.2025 14:34, Peter Eisentraut wrote: I've committed it.  Thanks. Please look at a planner error with a virtual generated column triggered by the following script: CREATE TABLE t(a int, b int GENERATED ALWAYS AS (a * 1)); SELECT SUM(CASE WHEN t.b = 1 THEN 1 ELSE 1 END) OVER (

Re: Virtual generated columns

2025-02-07 Thread Peter Eisentraut
On 06.02.25 14:03, vignesh C wrote: One suggestion: for the option where the user specifies publish_generated_columns as virtual (as shown below), could we change the error indicating that virtual generated columns are not currently supported? CREATE PUBLICATION pub1 FOR TABLE t1 WITH

Re: Virtual generated columns

2025-02-07 Thread Peter Eisentraut
On 06.02.25 00:25, Dean Rasheed wrote: On Tue, 4 Feb 2025 at 22:36, Peter Eisentraut wrote: Yeah, this is quite contorted. I have renamed it like you suggested. I looked over this again and I think the patch is in good shape to be committed. I've committed it. Thanks. One thought that

Re: Virtual generated columns

2025-02-06 Thread vignesh C
ri_extraUpdatedCols is populated. > > Yeah, this is quite contorted. I have renamed it like you suggested. One suggestion: for the option where the user specifies publish_generated_columns as virtual (as shown below), could we change the error indicating that virtual generated columns are

Re: Virtual generated columns

2025-02-05 Thread Dean Rasheed
On Tue, 4 Feb 2025 at 22:36, Peter Eisentraut wrote: > > Yeah, this is quite contorted. I have renamed it like you suggested. I looked over this again and I think the patch is in good shape to be committed. One thought that occurred to me was whether it would be better for the psql describe out

Re: Virtual generated columns

2025-02-04 Thread Peter Eisentraut
On 28.01.25 10:40, Shlok Kyal wrote: Test 5: Update publication on non virtual gen with no column list specified CREATE TABLE t1 (a int, b int GENERATED ALWAYS AS (a * 2) VIRTUAL); create publication pub1 for table t1; alter table t1 replica identity full; update t1 set a = 10; No error is thro

Re: Virtual generated columns

2025-01-28 Thread Shlok Kyal
th that fixed and also a few > >>> tweaks suggested by Jian. > >>> > >> > >> I'm hoping to push my RETURNING OLD/NEW patch [1] soon, so I thought > >> that I would check how it works together with this patch. The good > >> news is

Re: Virtual generated columns

2025-01-27 Thread Dean Rasheed
On Mon, 27 Jan 2025 at 09:59, Peter Eisentraut wrote: > > Here is an updated patch that integrates the above changes and also > makes some adjustments now that the logical replication configuration > questions are resolved. I think this is complete now. > In struct ResultRelInfo, the following f

Re: Virtual generated columns

2025-01-23 Thread Shlok Kyal
fixed. Here is a new patch with that fixed and also a few > tweaks suggested by Jian. > > I've also added a patch that addresses logical replication. It > basically adds back some of the prohibitions against including generated > columns in publications that have been lifted,

Re: Virtual generated columns

2025-01-16 Thread vignesh C
basically adds back some of the prohibitions against including generated > >> columns in publications that have been lifted, but this time only for > >> virtual generated columns, and amends the documentation. It doesn't > >> rename the publication option "publi

Re: Virtual generated columns

2025-01-15 Thread Peter Eisentraut
only for virtual generated columns, and amends the documentation. It doesn't rename the publication option "publish_generated_columns", but maybe that should be done. There are two potential approaches we could take to address the "publish_generated_columns" option

Re: Virtual generated columns

2025-01-15 Thread Peter Eisentraut
this patch. The good news is that AFAICS everything just works, and it's possible to return old/new virtual generated columns in DML queries as expected. It did require a minor update, because my patch adds a new "result_relation" argument to ReplaceVarsFromTargetList() -- needed in

Re: Virtual generated columns

2025-01-15 Thread Dean Rasheed
ws is that AFAICS everything just works, and it's possible to return old/new virtual generated columns in DML queries as expected. It did require a minor update, because my patch adds a new "result_relation" argument to ReplaceVarsFromTargetList() -- needed in DML queries because,

Re: Virtual generated columns

2025-01-14 Thread vignesh C
On Tue, 14 Jan 2025 at 19:08, Peter Eisentraut wrote: > > > I've also added a patch that addresses logical replication. It > basically adds back some of the prohibitions against including generated > columns in publications that have been lifted, but this time only for

Re: Virtual generated columns

2025-01-14 Thread Peter Eisentraut
On 09.01.25 13:41, jian he wrote: we can not ALTER COLUMN DROP EXPRESSION for virtual for now. so the following comments in generated_virtual.sql conflict with the output. ``` -- check that dependencies between columns have also been removed ALTER TABLE gtest29 DROP COLUMN a; -- should not drop

Re: Virtual generated columns

2025-01-14 Thread Peter Eisentraut
On 09.01.25 09:38, jian he wrote: create user foo; create user bar; grant create on schema public to foo; \c - foo create table t1 (id int, ccnum text, ccredacted text generated always as (repeat('*', 12) || substr(ccnum, 13, 4)) virtual); grant select (id, ccredacted) on table t1 to bar; insert

Re: Virtual generated columns

2025-01-13 Thread Dean Rasheed
On Wed, 8 Jan 2025 at 16:14, Peter Eisentraut wrote: > > Here is a new patch version In expand_generated_columns_in_expr(): + RangeTblEntry *rte; + + rte = makeNode(RangeTblEntry); + rte->relid = RelationGetRelid(rel); + + node = expand_generated_columns_internal(node, re

Re: Virtual generated columns

2025-01-09 Thread Dean Rasheed
On Wed, 8 Jan 2025 at 16:14, Peter Eisentraut wrote: > > One thing I could use some review on is the access control handling and > security in general. You can create virtual generated columns that have > their own access privileges but which can read columns that the user > does

Re: Virtual generated columns

2025-01-09 Thread jian he
a int, > check (a * 2 > 0) > > in terms of the constraint execution. > > The current patch does not support not-null constraints, but that's > mostly because it's not implemented yet. Maybe that's what Jian was > thinking about. > yes. we have 4 fo

Re: Virtual generated columns

2025-01-09 Thread jian he
ctly to give my response to each item. > > One thing I could use some review on is the access control handling and > security in general. You can create virtual generated columns that have > their own access privileges but which can read columns that the user > does not have access t

Re: Virtual generated columns

2025-01-08 Thread Marcos Pegoraro
Em qua., 8 de jan. de 2025 às 16:23, Vik Fearing escreveu: > This is lying to the planner, and you get to enjoy whatever breaks > because of it. A function that accesses external data is not immutable; > it is stable at best. > I understand that, but it's not documented, so users can think that

Re: Virtual generated columns

2025-01-08 Thread Peter Eisentraut
On 08.01.25 17:38, Tom Lane wrote: Peter Eisentraut writes: On 03.12.24 15:15, jian he wrote: SELECT attrelid, attname, attgenerated FROM pg_attribute WHERE attgenerated IN ('v') and (attnotnull or not atthasdef); I don't understand what the purpose of testing attnotnull is. That is indepe

Re: Virtual generated columns

2025-01-08 Thread Vik Fearing
On 08/01/2025 20:19, Marcos Pegoraro wrote: Em qua., 8 de jan. de 2025 às 13:14, Peter Eisentraut escreveu: Here is a new patch version where I have gathered various pieces of feedback and improvement suggestions that are scattered over this thread.  I hope I got them all.  I wil

Re: Virtual generated columns

2025-01-08 Thread Marcos Pegoraro
Em qua., 8 de jan. de 2025 às 13:14, Peter Eisentraut escreveu: > Here is a new patch version where I have gathered various pieces of > feedback and improvement suggestions that are scattered over this > thread. I hope I got them all. I will respond to the respective > messages directly to give

Re: Virtual generated columns

2025-01-08 Thread Tom Lane
Peter Eisentraut writes: > On 03.12.24 15:15, jian he wrote: >> SELECT attrelid, attname, attgenerated FROM pg_attribute WHERE >> attgenerated IN ('v') and (attnotnull or not atthasdef); > I don't understand what the purpose of testing attnotnull is. That is > independent of attgenerated, I thi

Re: Virtual generated columns

2025-01-08 Thread Peter Eisentraut
On 03.12.24 15:15, jian he wrote: -- check constraints CREATE TABLE gtest20 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) VIRTUAL CHECK (b < 50)); INSERT INTO gtest20 (a) VALUES (10); -- ok INSERT INTO gtest20 (a) VALUES (30); -- violates constraint ALTER TABLE gtest20 ALTER COLUMN b S

Re: Virtual generated columns

2025-01-08 Thread Peter Eisentraut
On 16.12.24 15:34, jian he wrote: hi. some minor issues... SET EXPRESSION AS This form replaces the expression of a generated column. Existing data in the column is rewritten and all the future changes will apply the new generation expression.

Re: Virtual generated columns

2025-01-08 Thread Peter Eisentraut
ssary. please check the attached patch. * remove check_modified_virtual_generated. * using heap_modify_tuple_by_cols in ExecBRInsertTriggers, ExecBRUpdateTriggers to overwrite virtual generated columns value to null. and it's not complicated. so that trigger behavior for stored and virtual wi

Re: Virtual generated columns

2025-01-08 Thread Peter Eisentraut
On 04.12.24 05:55, jian he wrote: On Fri, Nov 29, 2024 at 6:13 PM Peter Eisentraut wrote: - Added support for virtual columns in trigger column lists. (For that, I renamed ExecInitStoredGenerated() to ExecInitGenerated(), which handles the computation of ri_extraUpdatedCols.) why not dupli

Re: Virtual generated columns

2025-01-08 Thread Peter Eisentraut
On 08.01.25 09:22, Richard Guo wrote: - Added support for ALTER TABLE ... SET EXPRESSION. When using ALTER TABLE to set expression for virtual generated columns, we don't enforce a rewrite, which means we don't have the opportunity to check whether the new values for these columns c

Re: Virtual generated columns

2025-01-08 Thread Richard Guo
On Fri, Nov 29, 2024 at 7:14 PM Peter Eisentraut wrote: > Here is a new patch version, with several updates. > - Added support for ALTER TABLE ... SET EXPRESSION. When using ALTER TABLE to set expression for virtual generated columns, we don't enforce a rewrite, which means we don

Re: Virtual generated columns

2024-12-16 Thread jian he
hi. some minor issues... SET EXPRESSION AS This form replaces the expression of a generated column. Existing data in the column is rewritten and all the future changes will apply the new generation expression. the second sentence seems not to ap

  1   2   >