Which hook to use when overriding utility commands (COPY ...)

2022-03-19 Thread Hannu Krosing
Hi Pgsql-Hackers Which hook should I use when overriding the COPY command in an extension? I am working on adding new functionalities to COPY (compression, index management, various other transports in addition to stdin and file, other data formats, etc...) and while the aim is to contribute this

We should stop telling users to "vacuum that database in single-user mode"

2021-03-01 Thread Hannu Krosing
g flags. After this it should complete in 15-30 min, after which the database is available for writes again. Cheers, Hannu Krosing

Re: We should stop telling users to "vacuum that database in single-user mode"

2021-03-03 Thread Hannu Krosing
On Wed, Mar 3, 2021 at 11:33 AM David Rowley wrote: > > On Wed, 3 Mar 2021 at 21:44, Magnus Hagander wrote: ... > > I think we misunderstand each other. I meant this only as a comment > > about the idea of ignoring the cost limit in single user mode -- that > > is, it's a reason to *want* vacuum

Re: Extensibility of the PostgreSQL wire protocol

2021-03-03 Thread Hannu Krosing
I have not looked at the actual patch, but does it allow you to set up its own channels to listen to ? For example if I'd want to set up a server to listen to incoming connections over QUIC [1] - a protocol which create a connection over UDP and allows clients to move to new IP addresses (among ot

Re: PROXY protocol support

2021-03-04 Thread Hannu Krosing
The current proposal seems to miss the case of transaction pooling (and statement pooling) where the same established connection multiplexes transactions / statements from multiple remote clients. What we would need for that case would be a functionl pg_set_remote_client_address( be_key, remote_i

Re: Extensibility of the PostgreSQL wire protocol

2021-03-04 Thread Hannu Krosing
On Thu, Mar 4, 2021 at 9:55 PM Jan Wieck wrote: > > Another possibility, and this is what is being used by the AWS team > implementing the TDS protocol for Babelfish, is to completely replace > the entire TCOP mainloop function PostgresMain(). I suspect this is the only reasonable way to do it fo

Re: Boundary value check in lazy_tid_reaped()

2021-03-16 Thread Hannu Krosing
ration for each value. Of course it is possible that this has a very bad RAM access behaviour and is no win at all even if it otherways works. -- Hannu Krosing On Tue, Mar 16, 2021 at 10:08 PM Peter Geoghegan wrote: > On Sun, Mar 14, 2021 at 4:22 PM Thomas Munro > wrote: > > BTW

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-17 Thread Hannu Krosing
why are you using yet another special syntax for this ? would it not be better to do something like this: CREATE FUNCTION a_reall_long_and_winding_function_name(i int, out o int) LANGUAGE plpgsql AS $plpgsql$ DECLARE args function_name_alias BEGIN args.o = 2 * args.i END; $plpgsql$; or at le

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-18 Thread Hannu Krosing
On Thu, Mar 18, 2021 at 5:27 AM Pavel Stehule wrote: > > > There are few main reasons: > > a) compile options are available already - so I don't need invent new syntax > - #OPTION DUMP, #PRINT_STRICT ON, #VARIABLE_CONFLICT ERROR Are these documented anywhere ? At least a quick search for pl/pgs

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-18 Thread Hannu Krosing
, 2021 at 3:19 PM Hannu Krosing wrote: > > On Thu, Mar 18, 2021 at 5:27 AM Pavel Stehule wrote: > > > > > > There are few main reasons: > > > > a) compile options are available already - so I don't need invent new > > syntax - #OPTION DUMP, #PRINT_ST

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-18 Thread Hannu Krosing
On Thu, Mar 18, 2021 at 3:45 PM Pavel Stehule wrote: > > > > čt 18. 3. 2021 v 15:19 odesílatel Hannu Krosing napsal: ... >> Variation could be >> >> DECLARE >>fnarg ALIAS FOR FUNCTION a_function_with_an_inconveniently_long_name; >> >> so it is

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-19 Thread Hannu Krosing
On Thu, Mar 18, 2021 at 4:23 PM Pavel Stehule wrote: > But we don't support this feature. We are changing just a top scope's label. > So syntax "ALIAS FOR FUNCTION is not good. The user can have false hopes In this case it looks like it should go together with other labels and have << label_her

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-19 Thread Hannu Krosing
On Fri, Mar 19, 2021 at 2:29 PM Bruce Momjian wrote: > > OK, that makes perfect sense. I think the best solution is to document > that compute_query_id just controls the built-in computation of the > query id, and that extensions can also compute it if this is off, and > pg_stat_activity and log_

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-19 Thread Hannu Krosing
On Sat, Mar 20, 2021 at 1:21 AM Andres Freund wrote: > > Hi, > > On 2021-03-20 01:16:31 +0100, Hannu Krosing wrote: > > > But now we could instead schedule stats to be removed at commit > > time. That's not trivial of course, as we'd need to handle cases

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-20 Thread Hannu Krosing
> But now we could instead schedule stats to be removed at commit time. That's not trivial of course, as we'd need to handle cases where the commit fails after the commit record, but before processing the dropped stats. We likely can not remove them at commit time, but only after the oldest open s

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-20 Thread Hannu Krosing
It would be really convenient if user-visible serialisations of the query id had something that identifies the computation method. maybe prefix 'N' for internal, 'S' for pg_stat_statements etc. This would immediately show in logs at what point the id calculator was changed On Fri, Mar 19, 2021 a

Re: Proposal: http2 wire format

2018-03-29 Thread Hannu Krosing
> > > * room for other resultset formats later. Like Damir, I really want to > add > > protobuf or json serializations of result sets at some point, mainly so > we > > can return "entity graphs" in graph representation rather than left-join > > projection. > > -1. I don't think this belongs in post

A Generic Question about Generic type subscripting

2018-01-28 Thread Hannu Krosing
Sorry for being late to the party I started looking at the thread about "Generic type subscripting" and am wondering, why does it take the approach of modifying pg_type and modifying lots of internal functions, when instead it could be defined in a much lighter and less intrusive way as an operato

Re: pgbench - adding pl/pgsql versions of tests

2024-02-02 Thread Hannu Krosing
Thanks for the update. I will give it another go over the weekend Cheers, Hannu On Thu, Feb 1, 2024 at 7:33 PM vignesh C wrote: > On Fri, 18 Aug 2023 at 23:04, Hannu Krosing wrote: > > > > I will address the comments here over this coming weekend. > > The patch which y

Re: pgbench - adding pl/pgsql versions of tests

2024-02-02 Thread Hannu Krosing
My justification for adding pl/pgsql tests as part of the immediately available tests is that pl/pgsql itself is always enabled, so having a no-effort way to test its performance benefits would be really helpful. We also should have "tps-b-like as SQL function" to round up the "test what's availabl

Re: Function and Procedure with same signature?

2024-03-07 Thread Hannu Krosing
Hi Tom On Sat, Feb 10, 2024 at 12:38 AM Tom Lane wrote: > > "David G. Johnston" writes: > > On Fri, Feb 9, 2024, 12:05 Deepak M wrote: > >> Folks, When tried to create a function with the same signature as > >> procedure it fails. > > > That seems like a good hint you cannot do it. Specificall

CREATE TABLE creates a composite type corresponding to the table row, which is and is not there

2024-03-07 Thread Hannu Krosing
I could not find any explanation of the following behaviour in docs - Our documentation for CREATE TABLE says: CREATE TABLE also automatically creates a data type that represents the composite type corresponding to one row of the table. Therefore, tables cannot have the same name as any existing

Re: Function and Procedure with same signature?

2024-03-11 Thread Hannu Krosing
On Thu, Mar 7, 2024 at 5:46 PM Tom Lane wrote: > > Hannu Krosing writes: > > On Sat, Feb 10, 2024 at 12:38 AM Tom Lane wrote: > >> Worth noting perhaps that this is actually required by the SQL > >> standard: per spec, functions and procedures are both &quo

pgbench - adding pl/pgsql versions of tests

2023-01-04 Thread Hannu Krosing
viour, but can be disabled fia long flag --no-functions ) I selected Yy as they were unused and can be thought of as "inverted lambda symbol" :) If there are no strong objections, I'll add it to the commitfest as well - Hannu Krosing Google Cloud - We have a long list of planne

Time to move pg_test_timing to measure in nanoseconds

2023-03-26 Thread Hannu Krosing
Currently pg_test_timing utility measures its timing overhead in microseconds, giving results like this ~$ /usr/lib/postgresql/15/bin/pg_test_timing Testing timing overhead for 3 seconds. Per loop time including overhead: 18.97 ns Histogram of timing durations: < us % of total count

Re: Disable vacuuming to provide data history

2023-03-26 Thread Hannu Krosing
There is also another blocker - our timestamp resolution is 1 microsecond and we are dangerously close to speeds where one could update a row twice in the same microsecond . I have been thinking about this, and what is needed is 1. a nanosecond-resolution "abstime" type - not absolutely necessary

Re: Time to move pg_test_timing to measure in nanoseconds

2023-03-26 Thread Hannu Krosing
Sure, will do. On Sun, Mar 26, 2023 at 11:40 PM Andres Freund wrote: > > Hi, > > On 2023-03-26 16:43:21 +0200, Hannu Krosing wrote: > > Currently pg_test_timing utility measures its timing overhead in > > microseconds, giving results like this > > I have a patch

Allowing TRUNCATE of FK target when session_replication_role=replica

2023-10-31 Thread Hannu Krosing
Hi Currently we do not allow TRUNCATE of a table when any Foreign Keys point to that table. At the same time we do allow one to delete all rows when session_replication_role=replica This causes all kinds of pain when trying to copy in large amounts of data, especially at the start of logical rep

Re: Allowing TRUNCATE of FK target when session_replication_role=replica

2023-10-31 Thread Hannu Krosing
eira wrote: > > On Tue, Oct 31, 2023, at 5:09 AM, Hannu Krosing wrote: > > Currently we do not allow TRUNCATE of a table when any Foreign Keys > point to that table. > > > It is allowed iif you *also* truncate all tables referencing it. > > At the same tim

Why are wal_keep_size, max_slot_wal_keep_size requiring server restart?

2023-12-09 Thread Hannu Krosing
Hello fellow Hackers, Does anyone know why we have decided that the wal_keep_size, max_slot_wal_keep_size GUCs "can only be set in the postgresql.conf file or on the server command line." [1]? It does not seem fundamentally needed , as they are "kind of guidance", especially the second one. The

Re: Emitting JSON to file using COPY TO

2023-12-09 Thread Hannu Krosing
> On Sat, Dec 2, 2023 at 4:11 PM Tom Lane wrote: > > Joe Conway writes: > >> I noticed that, with the PoC patch, "json" is the only format that must be > >> quoted. Without quotes, I see a syntax error. In longer term we should move any specific COPY flag names and values out of grammar and t

Re: Initdb-time block size specification

2023-09-05 Thread Hannu Krosing
Something I also asked at this years Unconference - Do we currently have Build Farm animals testing with different page sizes ? I'd say that testing all sizes from 4KB up (so 4, 8, 16, 32) should be done at least before each release if not continuously. -- Cheers Hannu On Tue, Sep 5, 2023 at 4

Re: Initdb-time block size specification

2023-09-05 Thread Hannu Krosing
und wrote: > > Hi, > > On 2023-09-05 21:52:18 +0200, Hannu Krosing wrote: > > Something I also asked at this years Unconference - Do we currently > > have Build Farm animals testing with different page sizes ? > > You can check that yourself as easily as anybody else. > > Greetings, > > Andres Freund

Is there a way to use exported snapshots in autocommit mode ?

2022-12-09 Thread Hannu Krosing
Hello hackers, Is there a way to use exported snapshots in autocommit mode ? Either something similar to defaults in default_transaction_deferrable, default_transaction_isolation, default_transaction_read_only Or something that could be set in the function's SET part. Context: I am working on s

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Hannu Krosing
likely should be moved to access/heap/ Doing it this way would leave us with some manageable complexity in mapping from TID to 48-bit integer and/or 3 wanted positions in 2^32 Hannu Krosing On Wed, May 5, 2021 at 8:40 PM Peter Geoghegan wrote: > > On Wed, May 5, 2021 at 11

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Hannu Krosing
plicit goal. But would this not have the downside that all the secondary indexes will blow up as they now need to have the full table row as the TID ? - Hannu Krosing

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Hannu Krosing
- Hannu Krosing On Thu, May 6, 2021 at 4:53 AM Jeff Davis wrote: > > On Thu, 2021-05-06 at 03:26 +0200, Hannu Krosing wrote: > > How hard would it be to declare TID as current ItemPointerData with > > some values prohibited (NULL, SpecTokenOff

Re: .ready and .done files considered harmful

2021-05-06 Thread Hannu Krosing
How are you envisioning the shared-memory signaling should work in the original sample case, where the archiver had been failing for half a year ? Or should we perhaps have a system table for ready-to-archive WAL files to get around limitation sof file system to return just the needed files with O

Is anybody planning to release pglogical for v15 ?

2022-10-13 Thread Hannu Krosing
So, is anybody planning to release pglogical for v15 ? There are still a few things that one can do in pglogical but not in native / built0in replication ... Best Regards Hannu

Re: [PoC] Improve dead tuple storage for lazy vacuum

2021-07-08 Thread Hannu Krosing
dead tids into a file makes it trivial (well, almost :) ) to parallelize the initial heap scan, so more resources can be thrown at it if available. Cheers - Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested. On Thu, Jul 8

Re: pgbench logging broken by time logic changes

2021-07-08 Thread Hannu Krosing
ms than it solves :) Cheers ----- Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested.

Re: [PoC] Improve dead tuple storage for lazy vacuum

2021-07-08 Thread Hannu Krosing
ems_delete_check() to do this. It does the same > basic thing already. Also a lot of testing would be needed to figure out which strategy fits best for which distribution of dead tuples, and possibly their relation to the order of tuples to check from indexes . Cheers -- Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested.

Re: [PoC] Improve dead tuple storage for lazy vacuum

2021-07-08 Thread Hannu Krosing
in order to batch could end up looking > like cheap insurance that we should be glad to pay for. If the most expensive operation is sorting a few hundred of tids, then this should be fast enough. My worries were more that after the sorting we can not to dsimple index lookups for them, but

Hardening PostgreSQL via (optional) ban on local file system access

2022-06-24 Thread Hannu Krosing
Hi Pgsql-Hackers As part of ongoing work on PostgreSQL security hardening we have added a capability to disable all file system access (COPY TO/FROM [PROGRAM] , pg_*file*() functions, lo_*() functions accessing files, etc) in a way that can not be re-enabled without already having access to the f

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-24 Thread Hannu Krosing
On Sat, Jun 25, 2022 at 1:13 AM Andres Freund wrote: > > Hi, > > On 2022-06-25 00:08:13 +0200, Hannu Krosing wrote: > > Currently the file system access is controlled via being a SUPREUSER > > or having the pg_read_server_files, pg_write_server_files and > > pg_exe

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-24 Thread Hannu Krosing
On Sat, Jun 25, 2022 at 1:23 AM Hannu Krosing wrote: > My impression was that this was largely fixed via disabling the old > direct file calling convention, but then again I did not pay much > attention at that time :) I meant of course direct FUNCTION calling convention (Version

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-24 Thread Hannu Krosing
info in not on that page). On Sat, Jun 25, 2022 at 1:46 AM Gurjeet Singh wrote: > > On Fri, Jun 24, 2022 at 4:13 PM Andres Freund wrote: > > On 2022-06-25 00:08:13 +0200, Hannu Krosing wrote: > > > > 3) should this be back-patched (we can provide batches for all > &

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-24 Thread Hannu Krosing
n 25, 2022 at 2:08 AM David G. Johnston wrote: > > > > On Friday, June 24, 2022, Gurjeet Singh wrote: >> >> On Fri, Jun 24, 2022 at 4:13 PM Andres Freund wrote: >> > On 2022-06-25 00:08:13 +0200, Hannu Krosing wrote: >> >> > > 3) should this be

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-25 Thread Hannu Krosing
What are your ideas of applying a change similar to above to actually being a superuser ? That is adding a check for "superuser being currently available" to function superuser() in ./src/backend/utils/misc/superuser.c ? It could be as simple as a flag that can be set only at startup for maximum

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-25 Thread Hannu Krosing
ching user superuser to fix whatever needs fixing as superuser. Cheers Hannu On Sat, Jun 25, 2022 at 10:54 PM Hannu Krosing wrote: > > What are your ideas of applying a change similar to above to actually > being a superuser ? > > That is adding a check for "superuser being curr

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-06-27 Thread Hannu Krosing
> Another thought: for non-x86 platforms, the SIMD nodes degenerate to > "simple loop", and looping over up to 32 elements is not great > (although possibly okay). We could do binary search, but that has bad > branch prediction. I am not sure that for relevant non-x86 platforms SIMD / vector instr

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-27 Thread Hannu Krosing
25 at 00:08 +0200, Hannu Krosing wrote: > > Hi Pgsql-Hackers > > > > As part of ongoing work on PostgreSQL security hardening we have > > added a capability to disable all file system access (COPY TO/FROM > > [PROGRAM] , pg_*file*() functions, lo_*() functions > >

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-28 Thread Hannu Krosing
start to figuring out the set of roles into which one can decompose superuser access in longer run -- Hannu On Tue, Jun 28, 2022 at 8:30 PM Robert Haas wrote: > > On Mon, Jun 27, 2022 at 5:37 PM Hannu Krosing wrote: > > My current thinking is (based on more insights from Andres) t

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-29 Thread Hannu Krosing
025: $ CREATE EXTENSION ... .. and then clean up the sentinel file after, or just make it valid for N minutes from creation Cheers, Hannu Krosing On Wed, Jun 29, 2022 at 8:51 AM Laurenz Albe wrote: > > On Tue, 2022-06-28 at 16:27 -0700, Andres Freund wrote: > > > Experience shows that 9

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-29 Thread Hannu Krosing
og in as a user with superuser attribute 2) you must present proof that you can access the underlying file system Cheers, Hannu Krosing On Wed, Jun 29, 2022 at 12:48 PM Laurenz Albe wrote: > > On Wed, 2022-06-29 at 00:05 -0700, Andres Freund wrote: > > On 2022-06-29 08:51:10 +02

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-07-01 Thread Hannu Krosing
On Thu, Jun 30, 2022 at 7:25 PM Bruce Momjian wrote: > > On Thu, Jun 30, 2022 at 11:52:20AM -0400, Robert Haas wrote: > > I don't think this would be very convenient in most scenarios, This is the eternal problem with security - more security always includes more inconvenience. Unlocking your do

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-07-01 Thread Hannu Krosing
And thanks to Robert and Bruce for bringing up good points about potential pitfalls! I think we do have a good discussion going on here :) --- Hannu On Fri, Jul 1, 2022 at 11:14 AM Hannu Krosing wrote: > > On Thu, Jun 30, 2022 at 7:25 PM Bruce Momjian wrote: > > > > On Thu,

Re: making relfilenodes 56 bits

2022-07-12 Thread Hannu Krosing
Re: staticAssertStmt(MAX_FORKNUM <= INT8_MAX); Have you really thought through making the ForkNum 8-bit ? For example this would limit a columnar storage with each column stored in it's own fork (which I'd say is not entirely unreasonable) to having just about ~250 columns. And there can easily

Re: The Free Space Map: Problems and Opportunities

2021-09-06 Thread Hannu Krosing
will not improve our feeder mechanism to have back-to-back incoming commands, which can already be done today, but which I have seen seldom used. Cheers, - Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested. On Mon, Sep 6, 20

Re: The Free Space Map: Problems and Opportunities

2021-09-07 Thread Hannu Krosing
On Tue, Sep 7, 2021 at 2:29 AM Peter Geoghegan wrote: > > On Mon, Sep 6, 2021 at 4:33 PM Hannu Krosing wrote: > > When I have been thinking of this type of problem it seems that the > > latest -- and correct :) -- place which should do all kinds of > > cleanup like

Re: The Free Space Map: Problems and Opportunities

2021-09-08 Thread Hannu Krosing
On Wed, Sep 8, 2021 at 6:52 AM Peter Geoghegan wrote: > > On Tue, Sep 7, 2021 at 5:25 AM Hannu Krosing wrote: > > Are you speaking of just heap pages here or also index pages ? > > Mostly heap pages, but FWIW I think it could work for index tuples > too, with retail index tu

Re: WIP: System Versioned Temporal Table

2021-09-19 Thread Hannu Krosing
And we may even want to partition history tables for speed, storage cost or just to drop very ancient history - Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested. On Sun, Sep 19, 2021 at 8:32 PM Simon Riggs wrote: > >

Re: WIP: System Versioned Temporal Table

2021-09-20 Thread Hannu Krosing
On Mon, Sep 20, 2021 at 7:09 AM Corey Huinker wrote: > > On Sun, Sep 19, 2021 at 3:12 PM Hannu Krosing wrote: >> >> A side table has the nice additional benefit that we can very easily >> version the *table structure* so when we ALTER TABLE and the table >> struct

Re: prevent immature WAL streaming

2021-09-24 Thread Hannu Krosing
al Durability, for example 2-out-of-3 where primary is one of the 3 and not necessarily the most durable one? - Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested. On Fri, Sep 24, 2021 at 4:33 AM Alvaro Herrera wrote: >

Re: logical replication restrictions

2021-09-25 Thread Hannu Krosing
rts with " These might be addressed in future releases." so there is no exclusivity of being either a restriction or TODO. > [1] - https://wiki.postgresql.org/wiki/Todo > [2] - > https://www.postgresql.org/docs/devel/logical-replication-restrictions.html - Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested.

Re: logical decoding and replication of sequences

2021-09-25 Thread Hannu Krosing
ces are not transactional - so rollbacks should not matter" . Or we may get away with most in-detail sequence tracking on the source if we just keep track of the xmin of the sequence and send the sequence info over at commit if it == current_transaction_id ? ----- Hannu Krosing Google Cloud - We

Is there now an official way to pass column projection to heap AM

2021-08-18 Thread Hannu Krosing
ods As this was already almost a year ago, have there been any developments in the core PostgreSQL code for supporting this ? Or is projection still supported only in FDWs ? -- [*] https://github.com/greenplum-db/postgres/blob/zedstore/src/backend/access/zedstore/README - Hannu Krosing

Re: NAMEDATALEN increase because of non-latin languages

2021-08-18 Thread Hannu Krosing
least the truncation does not cut utf-8 characters in half ? - Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested. On Wed, Aug 18, 2021 at 1:33 PM Julien Rouhaud wrote: > > On Wed, Aug 18, 2021 at 7:27 PM John Naylor &g

pgbench functions as extension

2021-08-18 Thread Hannu Krosing
ed to have exact same behaviour as the ones in bundled pgbench would enable more work to be pushed to database and generally resu;lt in better interoperability for tests. -- [*] https://www.postgresql.org/docs/13/pgbench.html#PGBENCH-BUILTIN-FUNCTIONS ----- Hannu Krosing Google Cloud - We have a

Re: Middleware Messages for FE/BE

2021-08-19 Thread Hannu Krosing
der moving some load away" or "Planning to switch over to replica x.x.x.x, please follow" - Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested. On Thu, Aug 19, 2021 at 10:33 AM Simon Riggs wrote: > > The cur

Re: Middleware Messages for FE/BE

2021-08-19 Thread Hannu Krosing
One more set of "standard middleware messages" clients/middleware could turn on could be reporting LSNs * various local LSNs for progress of WAL persisting * reporting replication state of some or all replicas ----- Hannu Krosing Google Cloud - We have a long list of planned contrib

Re: Middleware Messages for FE/BE

2021-08-19 Thread Hannu Krosing
writing Of course there are (and should be) ways to still use the WALs normally for cases where replica x.x.x.x does not exists, like PITR And making this play nicely with Logical Decoding is another can of worms needing to be solved, but it is a start to becoming "Cloud Native" :) -

Re: Middleware Messages for FE/BE

2021-08-19 Thread Hannu Krosing
e than a single middleware component at which point they should be standardised . - Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested. On Thu, Aug 19, 2021 at 8:29 PM Simon Riggs wrote: > > On Thu, 19 Aug 2021 at 1

Re: question regarding policy for patches to out-of-support branches

2024-06-06 Thread Hannu Krosing
On Wed, Jun 5, 2024 at 8:29 PM Tom Lane wrote: > > Joe Conway writes: > > I was having a discussion regarding out-of-support branches and effort > > to keep them building, but could not for the life of me find any actual > > documented policy (although I distinctly remember that we do something..

Re: Function and Procedure with same signature?

2024-06-06 Thread Hannu Krosing
to support full "method call syntax" in general, up from one-argument case so that SELECT function(a_thing, arg2, arg 2, ...) could also be called as SELECT a_thing.function(arg2, arg 2, ...) -- Hannu On Mon, Mar 11, 2024 at 12:55 PM Hannu Krosing wrote: > > On Thu, Mar 7,

Re: What is a typical precision of gettimeofday()?

2024-06-18 Thread Hannu Krosing
I plan to send patch to pg_test_timing in a day or two the underlying time precision on modern linux seems to be 2 ns for some Intel CPUs 10 ns for Zen4 40 ns for ARM (Ampere) --- Hannu | On Tue, Jun 18, 2024 at 7:48 AM Andrey M. Borodin wrote: > > > > On 19 Mar 2024, at 13:28, Peter Ei

Re: What is a typical precision of gettimeofday()?

2024-06-19 Thread Hannu Krosing
is exactly 40 ns and I'd expect it to be the same on M1. On Tue, Jun 18, 2024 at 5:08 PM Hannu Krosing wrote: > > I plan to send patch to pg_test_timing in a day or two > > the underlying time precision on modern linux seems to be > > 2 ns for some Intel CPUs > 10 ns for

Re: DROP OWNED BY fails to clean out pg_init_privs grants

2024-06-20 Thread Hannu Krosing
Hi Tom, Is there anything that could be back-patched with reasonable effort ? -- Hannu On Mon, Jun 17, 2024 at 6:37 PM Daniel Gustafsson wrote: > > > On 17 Jun 2024, at 16:56, Tom Lane wrote: > > Daniel Gustafsson writes: > > >> I wonder if this will break any tools/scripts in prod which reli

Re: What is a typical precision of gettimeofday()?

2024-06-20 Thread Hannu Krosing
(resending to list and other CC:s ) Hi Tom This is my current patch which also adds running % and optionally uses faster way to count leading zeros, though I did not see a change from that. It also bucketizes first 128 ns to get better overview of exact behaviour. We may want to put reporting

Re: What is a typical precision of gettimeofday()?

2024-06-20 Thread Hannu Krosing
Another thing I changed in reporting was to report <= ns instead of < ns This was inspired by not wanting to report "zero ns" as "< 1 ns" and easiest was to change them all to <= On Thu, Jun 20, 2024 at 12:41 PM Hannu Krosing wrote: > > (resending to list a

Re: DROP OWNED BY fails to clean out pg_init_privs grants

2024-06-20 Thread Hannu Krosing
role to replace it with the owner of the object if we figure out a correct place to publish it. --- Hannu On Thu, Jun 20, 2024 at 5:35 PM Tom Lane wrote: > > Hannu Krosing writes: > > Is there anything that could be back-patched with reasonable effort ? > > Afraid not.

Re: DROP OWNED BY fails to clean out pg_init_privs grants

2024-06-20 Thread Hannu Krosing
Or perhaps we should still also patch pg_dump to ignore the aclentries which refer to roles that do not exist in the database ? On Thu, Jun 20, 2024 at 7:41 PM Hannu Krosing wrote: > > Then maybe we should put a query / function in the release notes to > clean up the existing mess. >

Re: DROP OWNED BY fails to clean out pg_init_privs grants

2024-06-20 Thread Hannu Krosing
databases as well On Thu, Jun 20, 2024 at 8:25 PM Tom Lane wrote: > > Hannu Krosing writes: > > Or perhaps we should still also patch pg_dump to ignore the aclentries > > which refer to roles that do not exist in the database ? > > I didn't want to do that before, and I

Re: Adminpack removal

2024-06-26 Thread Hannu Krosing
I agree that removing adminpack was a bit of a surprise for me as well. At first I assumed that it was just moved into the core to accompany the file and directory *reading* functions, until I found the release notes mentioning that now one of the users of adminpack does not need it and so it is dr

Re: What is a typical precision of gettimeofday()?

2024-07-02 Thread Hannu Krosing
Hi Tom, On various Intel CPUs I got either steps close to single nanosecond or sometimes a little more on older ones One specific CPU moved in in 2 tick increments while the ration to ns was 2,1/1 or 2100 ticks per microsecond. On Zen4 AMD the step seems to be 10 ns, even though the tick-to-ns

Re: What is a typical precision of gettimeofday()?

2024-07-02 Thread Hannu Krosing
4 ticks -- 0 ( 0.0) 74.5082 74.5082 217288944 1 ( 40.0) 25.4886 99.9968 74332699 2 ( 80.0) 0. 99.9968 4 3 ( 120.0) 0. 99.9968 1 4 ( 160.0) 0. 99.9968 3 On Tue, Jul 2, 2024 at 7:31 PM Hannu Krosing wrote: > > Hi Tom, > > On various Intel CPUs I got either steps close

Re: What is a typical precision of gettimeofday()?

2024-07-02 Thread Hannu Krosing
On Tue, Jul 2, 2024 at 7:50 PM Tom Lane wrote: > > > Do we also need tests for this one ? > > Yeah, it was annoying me that we are eating the overhead of a TAP test > for pg_test_timing and yet it covers barely a third of the code [1]. > We obviously can't expect any specific numbers out of a tes

Re: What is a typical precision of gettimeofday()?

2024-07-03 Thread Hannu Krosing
On Wed, Jul 3, 2024 at 10:03 AM Tom Lane wrote: Keep in mind also that instr_time.h does not pretend to provide > real time --- the clock origin is arbitrary. But these results > do give me additional confidence that gettimeofday() should be > good to the microsecond on any remotely-modern platfo

Re: What is a typical precision of gettimeofday()?

2024-07-03 Thread Hannu Krosing
We currently do something similar with OIDs where we just keep generating them and then testing for conflicts. I don't think this is the best way to do it but it mostly works when you can actually test for uniqueness, like for example in TOAST or system tables. Not sure this works even reasonably

Re: incremental-checkopints

2023-07-26 Thread Hannu Krosing
Starting from increments checkpoint is approaching the problem from the wrong end. What you actually want is Atomic Disk Writes which will allow turning off full_page_writes . Without this you really can not do incremental checkpoints efficiently as checkpoints are currently what is used to deter

Re: incremental-checkopints

2023-07-26 Thread Hannu Krosing
On Wed, Jul 26, 2023 at 9:54 PM Matthias van de Meent wrote: > > Then you ignore the max_wal_size GUC as PostgreSQL so often already > does. At least, it doesn't do what I expect it to do at face value - > limit the size of the WAL directory to the given size. That would require stopping any new

Re: How to build a new grammer for pg?

2023-08-08 Thread Hannu Krosing
I would look at how Babelfish DB did it when adding SQL Server compatibility https://babelfishpg.org/ and https://github.com/babelfish-for-postgresql/ another source to inspect could be https://github.com/IvorySQL/IvorySQL for "oracle compatible PostgreSQL" On Tue, Aug 1, 2023 at 10:07 PM Jonah

Re: pgbench - adding pl/pgsql versions of tests

2023-08-18 Thread Hannu Krosing
I will address the comments here over this coming weekend. I think that in addition to current "tpc-b like" test we could also have more modern "tpc-c like" and "tpc-h like" tests And why not any other "* -like" from the rest of TPC-*, YCSP, sysbench, ... :) though maybe not as part of pg_bench

Re: DROP OWNED BY fails to clean out pg_init_privs grants

2024-05-23 Thread Hannu Krosing
While the 'DROP OWNED BY fails to clean out pg_init_privs grants' issue is now fixed,we have a similar issue with REASSIGN OWNED BY that is still there: Tested on fresh git checkout om May 20th test=# create user privtestuser superuser; CREATE ROLE test=# set role privtestuser; SET test=# create

Re: DROP OWNED BY fails to clean out pg_init_privs grants

2024-05-25 Thread Hannu Krosing
On Fri, May 24, 2024 at 10:00 PM Tom Lane wrote: > > Robert Haas writes: > > On Fri, May 24, 2024 at 2:57 PM Tom Lane wrote: > >> Doesn't seem right to me. That will give pg_dump the wrong idea > >> of what the initial privileges actually were, and I don't see how > >> it can construct correct

Re: DROP OWNED BY fails to clean out pg_init_privs grants

2024-05-25 Thread Hannu Krosing
On Sat, May 25, 2024 at 4:48 PM Tom Lane wrote: > > Hannu Krosing writes: > > Having an pg_init_privs entry referencing a non-existing user is > > certainly of no practical use. > > Sure, that's not up for debate. What I think we're discussing > right now

Re: DROP OWNED BY fails to clean out pg_init_privs grants

2024-05-26 Thread Hannu Krosing
rivileges | Policies ++---++---+-- public | revoketest | table | hannuk=arwdDxtm/hannuk | | public | vacwatch | table || | (2 rows) On Sun, May 26, 2024 at 12:05 AM Hannu Krosing wrote: > &g

Re: DROP OWNED BY fails to clean out pg_init_privs grants

2024-05-28 Thread Hannu Krosing
nnu On Sun, May 26, 2024 at 11:27 PM Daniel Gustafsson wrote: > > > On 26 May 2024, at 23:25, Tom Lane wrote: > > > > Hannu Krosing writes: > >> Attached is a minimal patch to allow missing roles in REVOKE command > > > > FTR, I think this is a very b

Will there be https://wiki.postgresql.org/wiki/PgCon_2024_Developer_Unconference ?

2024-06-03 Thread Hannu Krosing
Hello Everybody! For at least last two years we have had Developers Conference Unconference notes in PostgreSQL Wiki https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference https://wiki.postgresql.org/wiki/PgCon_2023_Developer_Unconference And I know that people took notes at least at

Re: Let's make PostgreSQL multi-threaded

2023-06-08 Thread Hannu Krosing
On Wed, Jun 7, 2023 at 11:37 PM Andres Freund wrote: > > Hi, > > On 2023-06-05 13:40:13 -0400, Jonathan S. Katz wrote: > > 2. While I wouldn't want to necessarily discourage a moonshot effort, I > > would ask if developer time could be better spent on tackling some of the > > other problems around

  1   2   >