Re: Cast INTEGER to BIT confusion

2023-08-15 Thread Greg Sabino Mullane
On Tue, Aug 15, 2023 at 4:49 AM [Quipsy] Markus Karg wrote: > > CREATE TABLE T (c BIT); > > INSERT INTO T VALUES (1); > > -- I MUST NOT change both lines in any way as these are part of a > third-party application! > That is quite the requirement! I hope that the rest of the application is more

Re: vacuum freeze wait_event BufferPin

2024-02-02 Thread Greg Sabino Mullane
On Wed, Jan 31, 2024 at 2:05 PM abrahim abrahao wrote: > There is a three-day vacuum freeze on a partition table with wait_event = > BufferPin, no transactions (active or idle) older than it that are not > blocked, but at least one select query is executing at all times related of > this partitio

Re: vacuum freeze wait_event BufferPin

2024-02-02 Thread Greg Sabino Mullane
On Fri, Feb 2, 2024 at 3:25 PM abrahim abrahao wrote: > SHARE UPDATE EXCLUSIVE (ShareUpdateExclusiveLock) should not be blocked > by ACCESS SHARE (AccessShareLock). Am I wrong about it? If I am not wrong > why it still locking it? > Those locks with no other context are somewhat of a red herri

Re: How to do faster DML

2024-02-03 Thread Greg Sabino Mullane
As a general rule, avoid heavy subselects like that. You don't need to build a full list of duplicates before starting. Another approach: create table mytable2 (like mytable1); alter table mytable2 add primary key (id); insert into mytable2 select * from mytable1 on conflict do nothing; Given t

Re: How to do faster DML

2024-02-04 Thread Greg Sabino Mullane
Partitioning will depend a lot on how you access the data. Is it always using that transaction_date column in the where clause? Can you share the table definition via \d? We've talked about this table quite a bit, but not actually seen what it looks like. We are flying blind a little bit. You menti

Re: Unused indexes

2024-02-05 Thread Greg Sabino Mullane
> > The pg_stat_*_tables tables idx_* columns has accumulated usage since the > last time you started the postmaster. > Actually, those persist at restart - you can use select datname, stats_reset from pg_stat_database; to see when/if they were reset. You can look for zero/low entries in pg_sta

Re: How to do faster DML

2024-02-05 Thread Greg Sabino Mullane
On Sun, Feb 4, 2024 at 3:52 PM Lok P wrote: > This table will always be queried on the transaction_date column as one of > the filters criteria. But the querying/search criteria can span from a day > to a month worth of transaction date data. > In that case, and based on the numbers you provided

Re: How to do faster DML

2024-02-05 Thread Greg Sabino Mullane
> So when you said *"In that case, and based on the numbers you provided, > daily partitioning seems a decent solution."* > , does it mean that size of the partition (which will be ~100-150GB per > partition here) is not an issue for the postgres to serve read latency > sensitive queries? > Yes, t

Re: How to do faster DML

2024-02-08 Thread Greg Sabino Mullane
On Thu, Feb 8, 2024 at 12:12 AM Lok P wrote: > Rearranging the table columns by typlen desc, didn't give much storage > space reduction. > It's not so much "biggest to smallest" as making sure you don't have any gaps when you move from one data type to another. You may have not had any "holes" o

Re: Partitioning options

2024-02-08 Thread Greg Sabino Mullane
On Thu, Feb 8, 2024 at 12:42 AM sud wrote: ... > The key transaction table is going to have ~450 Million transactions per > day and the data querying/filtering will always happen based on the > "transaction date" column. > ... > Should we go for simple daily range partitioning on the transaction

Re: Partitioning options

2024-02-08 Thread Greg Sabino Mullane
> > Out of curiosity, As OP mentioned that there will be Joins and also > filters on column Customer_id column , so why don't you think that > subpartition by customer_id will be a good option? I understand List > subpartition may not be an option considering the new customer_ids gets > added slowl

Re: Clarification regarding managing advisory locks in postgresql

2024-02-08 Thread Greg Sabino Mullane
On Thu, Jan 25, 2024 at 5:44 AM Sasmit Utkarsh wrote: > Therefore, locks acquired in one process (or transaction) do not directly > affect locks in another process (or transaction). > Ummaybe if you describe the exact problem you are trying to solve with advisory locks, we can find the best

Re: How should we design our tables and indexes

2024-02-10 Thread Greg Sabino Mullane
There is a lot to unpack here. I'm going to take a quick pass, but you ought to consider getting some custom expert help. On Sat, Feb 10, 2024 at 2:39 PM veem v wrote: > ... These will be pagination queries. These read queries will be on big > transaction tables (will have ~500+ attributes appro

Re: How to do faster DML

2024-02-12 Thread Greg Sabino Mullane
On Tue, Feb 6, 2024 at 12:15 AM Lok P wrote: > Another thing I noticed the shared_buffer parameters set as 2029684 in > this instance, which comes to ~21MB and that seems very small for a > database operating in large scale. And I see we have RAM in the instance > showing as ~256GB. So thinking o

Re: How to do faster DML

2024-02-12 Thread Greg Sabino Mullane
On Mon, Feb 12, 2024 at 1:50 AM veem v wrote: > So we were thinking, adding many column to a table should be fine in > postgres (as here we have a use case in which total number of columns may > go till ~500+). But then, considering the access of columns towards the > end of a row is going to ad

Re: How should we design our tables and indexes

2024-02-12 Thread Greg Sabino Mullane
> > When the user clicks to the second page , it will see the next set of rows > i.e 100 to 200 and next will see 200 to 300 and so on till the result set > finishes. > As others have pointed out, that still makes no sense. You will either fail to show certain rows completely, or have a stale view

Re: Safest pgupgrade jump distance

2024-02-12 Thread Greg Sabino Mullane
On Mon, Feb 12, 2024 at 5:08 AM Dávid Suchan wrote: > Hi, I was wondering what is the safest pg_upgrade version upgrade distance > going from 9.6 version. Do I need to go version by version or I can go from > 9.6 to 15? You can go direct. You really should go to 16 though. If nothing else, you

Re: How should we design our tables and indexes

2024-02-12 Thread Greg Sabino Mullane
> Is there any way to track those historical executions and be able to find the exact root cause of the slow executions confidently? https://www.postgresql.org/docs/current/auto-explain.html auto_explain.log_min_duration = '5s' ## or large enough to capture your quickest one Do NOT enable auto_e

Re: Using a Conversion Table

2024-02-14 Thread Greg Sabino Mullane
> > "Fiscal year" double precision, > This column is an INTEGER in your other table, so your schema is not even internally consistent! Try to use TEXT, INT, DATE and TIMESTAMPTZ whenever possible, with NUMERIC as needed for things with a precision. See: https://www.postgresql.org/docs/current

Re: Postgres pg_cron extension

2024-02-14 Thread Greg Sabino Mullane
Yes, definitely the wrong forum. RDS is not Postgres, and "parameter groups" is an Amazon thing. A quick web search would seem to indicate that the default group is replaced by the custom one, rather than enhancing it. But really, this is a simple "try it and see" sort of task. https://docs.aws.am

Re: How should we design our tables and indexes

2024-02-14 Thread Greg Sabino Mullane
On Tue, Feb 13, 2024 at 2:26 PM veem v wrote: > Can the optimizer, only scan the TABLE1 using ACCESS criteria " > TABLE1.MID in ()" or "TABLE1.CID in ()" which will be catered by > two different index i.e one index on column "MID" and other on column "CID"? > Yes: greg=# create table t

Re: PITR

2024-02-14 Thread Greg Sabino Mullane
On Wed, Feb 14, 2024 at 1:39 PM Yongye Serkfem wrote: > I checked the Postgresql.conf file and can't find the > appropriate parameter to set the target time. > It is set inside the postgresql.conf file. Unless you have modified it, there is a section in there called "Recovery Target" which has

Re: How to do faster DML

2024-02-15 Thread Greg Sabino Mullane
> > So as I also tested the same as you posted, there has been no change in > "ctid" , when I altered the column data type from 'int' to 'bigint' in the > table, so that means full table rewriting won't happen in such a scenario. No it was definitely rewritten - do not depend on the ctid to verif

Re: How to do faster DML

2024-02-15 Thread Greg Sabino Mullane
On Thu, Feb 15, 2024 at 11:43 AM Adrian Klaver wrote: > That is a mixed bag: > Ha! Good point. Our contrived example table does suffer from that, so perhaps the test should be: create table int_test(c1 int, c2 int); Cheers, Greg

Re: How to do faster DML

2024-02-15 Thread Greg Sabino Mullane
I really worry you are overthinking this. The only real concern is going from INT to BIGINT, jumping from 4 to 8 bytes or storage. That really covers 99% of real world cases, and the canonical advice is to start with BIGINT if you ever think your rows are going to be numbered in the billions. Also,

Re: Encryption Options

2024-02-16 Thread Greg Sabino Mullane
You need to clearly define your threat model. What exactly are you defending against? What scenario do you want to avoid? Also, your decision of on-premise or Aurora is extremely relevant to your range of options. Cheers, Greg

Re: Encryption Options

2024-02-16 Thread Greg Sabino Mullane
On Fri, Feb 16, 2024 at 4:04 PM sud wrote: > > Yes these are Account number/PCI data and "data at rest" encryption is > something management is asking to have irrespective of whether we encrypt > those before storing in the database or not. And this system needs to > adhere to PCI 4.0 standards ,

Re: "reverse" (?) UPSERT -- how to ?

2024-02-17 Thread Greg Sabino Mullane
On Sat, Feb 17, 2024 at 10:24 AM Karsten Hilbert wrote: > There will be a view giving rows for > each detail row enriched with master table data > UNION ALL > rows for each master row that does not have any detail row with > detail table columns NULLed > A better

Re: Partitioning, Identity and Uniqueness (given pg 16 changes)

2024-02-19 Thread Greg Sabino Mullane
On Sun, Feb 18, 2024 at 10:32 PM Darryl Green wrote: > 1) Is my attempt to improve performance of insert AND (as the number of > sources goes up, as it has) querying which is invariably by id but it is > easy to include "and src = x" in the majority of usage (the other common > case is to query a

Re: User pooler error: unsupported startup parameter: extra_float_digits

2024-02-20 Thread Greg Sabino Mullane
Add this to your pgbouncer config file: ignore_startup_parameters = extra_float_digits A web search of your subject line finds this solution quite easily. Providing more details would also be a good idea for future emails. Cheers, Greg

Re: Postgresql assistance needed

2024-02-22 Thread Greg Sabino Mullane
On the surface, it looks as though it *should* work if the only thing changing is the username. Those other more serious errors should get fixed, but putting those aside for now... We don't know what your program does. Write a smaller one that just does a PQexec and calls nextval, then returns a pr

Re: Partitioning, Identity and Uniqueness (given pg 16 changes)

2024-02-22 Thread Greg Sabino Mullane
> > I have a table that is capturing what is, basically, time series data Time series data usually is concerned with "recent" data, and has a subsequent drop off date. This is ideal for partitioning by timestamp - not only do your queries only need to hit a few of the total tables, but you can si

Re: PostgreSQL Guard

2024-02-27 Thread Greg Sabino Mullane
You may need to expand on / rephrase question 2.

Re: PostgreSQL Guard

2024-02-28 Thread Greg Sabino Mullane
On Tue, Feb 27, 2024 at 3:48 PM Jason Long wrote: > Suppose you want to transfer the database of a website like Amazon.com to > a new server and delete the old one. Many users are buying and selling on > this website at the same time and it is not possible to turn off the > server. What do you do

Re: Orphan table files at data/base/

2024-02-28 Thread Greg Sabino Mullane
No, I would not remove those files without making 100% sure they do not belong to that database or any other. Are you sure you are inside database 16384 when you ran those commands? Does a 'stat' on those files line up with the time of the crash? If so, I would stop pg, move the files someplace els

Re: When manual analyze is needed

2024-03-04 Thread Greg Sabino Mullane
On Mon, Mar 4, 2024 at 12:23 AM veem v wrote: > Additionally if a query was working fine but suddenly takes a > suboptimal plan because of missing stats , do we have any hash value column > on any performance view associated with the queryid which we can refer to > see past vs current plans diffe

Re: When manual analyze is needed

2024-03-05 Thread Greg Sabino Mullane
> > We were planning to have the auto_explain extension added and set the > log_min_duration to ~5 seconds and log_analyze to true. So that all the > queries going above that time period will be logged and provide detailed > information on the exact point of bottleneck. Will it be a good idea to se

Re: pgBadger: Cannot find any log entries from systemd-journald

2024-03-06 Thread Greg Sabino Mullane
On Tue, Mar 5, 2024 at 3:14 AM Frank Lanitz wrote: > $ pgbadger --journalctl "journalctl -u postgresql.service" > You could try adding --verbose to see if it gives more clues. > Having a look into the journal there is a lot of > None of the snippets from journald you showed were actually thin

Re: Feature Request: log_line_prefix %h Counterpart

2024-03-06 Thread Greg Sabino Mullane
On Fri, Jan 26, 2024 at 1:44 AM Jawala wrote: > Looking for an escape sequence for log_line_prefix for the IP address that > traffic was destined to, i.e., the *local* address of the TCP connection, > natural counterpart to "remote IP address". > I made a quick proof of concept of this, will pos

Re: Feature Request: log_line_prefix %h Counterpart

2024-03-06 Thread Greg Sabino Mullane
Actually, will just post it here in case someone finds this thread and wants to follow along: https://www.postgresql.org/message-id/flat/CAKAnmmK-U%2BUicE-qbNU23K--Q5XTLdM6bj%2BgbkZBZkjyjrd3Ow%40mail.gmail.com

Re: update to 16.2

2024-03-08 Thread Greg Sabino Mullane
On Fri, Mar 8, 2024 at 5:01 AM Matthias Apitz wrote: > The other option (pg_upgrade) we never used. > You really should give this a shot. Much easier, and orders of magnitude faster with the --link option. It should work fine even with a custom-compiled postgres (really, as long as pg_dump can s

Re: Windows service randomly stops with no indication why

2024-03-08 Thread Greg Sabino Mullane
Go to "Services", find Postgres, and try a manual restart, see what happens. Then check the recovery tab and see what it is supposed to do on failures - you are probably at the "Do nothing" count limit, hence the no auto restart. If you can manually duplicate the failure to restart, try increasing

Re: Seeing high query planning time on Azure Postgres Single Server version 11.

2024-03-10 Thread Greg Sabino Mullane
On Sat, Mar 9, 2024 at 1:57 PM hassan rafi wrote: > Would upgrading to the latest version of Postgres potentially solve the > issue? > Potentially, yes, but the only one who can answer that for sure is you. Upgrade to 11.22 and re-run the query. Worst case scenario, it runs the same speed but yo

Re: alter table xxx alter column yyy set (n_distinct=nnnn );

2024-03-11 Thread Greg Sabino Mullane
Works for me on Postgres 14 as well. Maybe you are looking at the wrong table? Might help to show the exact steps you typed in.

Re: operator is only a shell - Error

2024-03-15 Thread Greg Sabino Mullane
On Fri, Mar 15, 2024 at 6:26 AM Rajesh S wrote: > I wanted to implement a new "=" (equal) operator with LEFTARG as numeric > and RIGHTARG as varchar. But after creating the function and operator, > psql shows the error "operator is only a shell: character varying = > numeric > Your operator has

Re: select results on pg_class incomplete

2024-03-15 Thread Greg Sabino Mullane
On Fri, Mar 15, 2024 at 6:43 AM Thiemo Kellner wrote: > I am not sure, we are taking about the same problem, but would be > surprised to be the only one having experienced filling disks. ... > So, if I have a disk getting filled up, I would like to get easily > information on the problematic >

Re: Query on Postgres SQL transaction

2024-03-15 Thread Greg Sabino Mullane
That's a very vague question, but you can trace exactly what is happening by issuing SET log_statement = 'all'; Ideally at the session level by your application, but can also set it at the database and user level. If all else fails, set it globally (i.e. postgresql.conf). Turn it off again as soo

Re: operator is only a shell - Error

2024-03-18 Thread Greg Sabino Mullane
The order of the arguments matter. You need an operator that expects a varchar on the left hand side of the operator, and numeric on the right side. For example: create database cast_test; \c cast_test You are now connected to database "cast_test" as user "greg". cast_test=# create table deposit_

Re: Postgres database encryption

2024-03-19 Thread Greg Sabino Mullane
What is your threat model? That will determine the best answer. The easiest thing is to use OS-level encryption. Cheers, Greg

Re: Query on Postgres SQL transaction

2024-03-30 Thread Greg Sabino Mullane
That log snippet shows two different PIDs. Check the logs to see the complete path that process 1876 took after it did the insert at 2024-02-19 15:21:54.850 +08 Make sure you are not using prepared transactions. This should return 0: select current_setting('max_prepared_transactions'); Cheers,

Re: Timestamp conversion Error in dynamic sql script

2024-04-02 Thread Greg Sabino Mullane
> > 1. Declare start_date as DATE when you want to add days with date + int > 2. Keep TIMESTAMP and use start_date + make_interval(days => i) > Also 0. Use TIMESTAMPTZ not TIMESTAMP Cheers, Greg

Re: Moving delta data faster

2024-04-04 Thread Greg Sabino Mullane
So you want advice from this open source community about moving things from a proprietary database to another proprietary database, going through a proprietary storage system along the way? :) To answer the original question, use MERGE. That's it's job, and your table size is very tiny, so I doubt

Re: Storing and comparing columns of cryptographic hashes?

2024-04-08 Thread Greg Sabino Mullane
On Mon, Apr 8, 2024 at 10:08 AM Josh Triplett wrote: > - Is there a way to tell postgresql "this column contains cryptographic > hashes, so you can do hash joins using any subset of the bits, > without having to hash them again"? If not, should there be? > No, and no. (if I understand your quest

Re: Tracing libpq client: Only with PQtrace()?

2024-04-09 Thread Greg Sabino Mullane
On Tue, Apr 9, 2024 at 8:51 AM Sebastien Flaesch wrote: > Is the PQtrace() API the only way to enable libpq client tracing? > > I thought about some environment variable of client configuration > setting... > That's generally the job of the client, or more specifically, the driver providing the

Re: mystery with postgresql.auto.conf

2024-04-10 Thread Greg Sabino Mullane
On Wed, Apr 10, 2024 at 8:40 AM Matthias Apitz wrote: > Maybe later I did it accidently against the 15.1 server from the psql > history. Yes, as shown by the timestamps from your first post: -rw--- 1 postgres postgres 124 Mar 28 11:35 > postgresql151/data/postgresql.auto.conf > This is al

Re: Forcing INTERVAL days display, even if the interval is less than one day

2024-05-07 Thread Greg Sabino Mullane
Could just use a CASE for that particular case: CASE WHEN now()-latest_vacuum < '1 day' THEN '0 days ' ELSE '' END || DATE_TRUNC('second', now()-latest_vacuum) AS vacuumed_ago Cheers, Greg

Re: Postgresql active-active nodes in cluster

2024-05-09 Thread Greg Sabino Mullane
> https://wiki.postgresql.org/wiki/Multimaster > That is a pretty old page. There are other solutions, such as pgedge, but the three most important questions when it comes to active-active replication are: * Do you really, really need it? (hint: the answer is no) * What problem are you trying to

Re: Postgresql active-active nodes in cluster

2024-05-13 Thread Greg Sabino Mullane
On Mon, May 13, 2024 at 2:53 AM Vidyashree H S < shreevidya...@exaleapsemi.com> wrote: > >- Is that multi master replication is not recommended to do ? > > No, it is not recommended, as it has a very, very narrow use case, and comes with a lot of drawbacks. Sure, you COULD configure your car t

Re: Execution history of a single query

2024-05-17 Thread Greg Sabino Mullane
Not directly through pg_stat_statements, no. But you can: * Use log_min_duration_statement to spy on individual runs * Periodically copy pg_stat_statements to another table, reset the stats, and then you have some time-based buckets to play with. Cheers, Greg

Re: Finding "most recent" using daterange

2024-05-22 Thread Greg Sabino Mullane
This is a good candidate for a window function. Also note that nulls already get sorted correctly by the DESC so no need to get 'infinity' involved, although you could write 'DESC NULLS FIRST' to be explicit about it. with x as (select *, row_number() over (partition by id order by upper(dates) d

Re: Finding "most recent" using daterange

2024-05-22 Thread Greg Sabino Mullane
Oh, you are right - NULLS LAST, got my -infinity crossed with my infinity. :) Cheers, Greg

Re: Timeout gets unset on a syntax error.

2024-05-29 Thread Greg Sabino Mullane
On Wed, May 29, 2024 at 6:29 AM ISHAN CHHANGANI . < f20200...@hyderabad.bits-pilani.ac.in> wrote: > Is there some code within Postgres that resets/removes the signals in case > a query hits any error? > See the comment and call to disable_all_timeouts() in postgres.c, part of the cleanup-after-ex

Re: Postgresql 16.3 Out Of Memory

2024-06-03 Thread Greg Sabino Mullane
On Mon, Jun 3, 2024 at 6:19 AM Radu Radutiu wrote: > Do you have any idea how to further debug the problem? > Putting aside the issue of non-reclaimed memory for now, can you show us the actual query? The explain analyze you provided shows it doing an awful lot of joins and then returning 14+ mi

Re: Password complexity/history - credcheck?

2024-06-23 Thread Greg Sabino Mullane
On Sun, Jun 23, 2024 at 5:30 AM Martin Goodson wrote: > I believe that our security team is getting most of this from our > auditors, who seem convinced that minimal complexity, password history > etc are the way to go despite the fact that, as you say, server-side > password checks can't really

Planet Postgres and the curse of AI

2024-07-17 Thread Greg Sabino Mullane
I've been noticing a growing trend of blog posts written mostly, if not entirely, with AI (aka LLMs, ChatGPT, etc.). I'm not sure where to raise this issue. I considered a blog post, but this mailing list seemed a better forum to generate a discussion. The problem is two-fold as I see it. First,

Re: Planet Postgres and the curse of AI

2024-07-18 Thread Greg Sabino Mullane
> But to what degree exactly should that be allowed? Somewhat ironically, here's a distinction chatgpt and I came up with: LLM-generated content: Content where the substantial part of the text is directly created by LLMs without significant human alteration or editing. Human-edited or reviewed c

Re: Planet Postgres and the curse of AI

2024-07-23 Thread Greg Sabino Mullane
On Fri, Jul 19, 2024 at 3:22 AM Laurenz Albe wrote: > I have no problem with that definition, but it is useless as a policy: > Even in a blog with glaring AI nonsense in it, how can you prove that the > author did not actually edit and improve other significant parts of the > text? > Well, we ca

Re: data checksums

2024-08-06 Thread Greg Sabino Mullane
The penalty is not "considerable", but it is not completely trivial either. But it's more on the trivial side. Part of the problem is that it is hard to measure, as it is very workload dependent. As to why it is not the default, Postgres is very careful and conservative by default, and not everybod

Re: Column type modification in big tables

2024-08-08 Thread Greg Sabino Mullane
On Thu, Aug 8, 2024 at 2:39 PM Lok P wrote: > Can anybody suggest any other possible way here. > Sure - how about not changing the column type at all? > one of the columns from varchar(20) to varchar(2) ALTER TABLE foobar ADD CONSTRAINT twocharplease CHECK (length(mycol) <= 2) NOT VALID; > on

Re: Getting specific partition from the partition name

2024-08-08 Thread Greg Sabino Mullane
_MM_DD is already setup for sorting, so just do: SELECT table_name FROM information_schema.tables WHERE table_name ~ 'table_part_p' ORDER BY 1 DESC; If you need to grab the numbers: SELECT substring('table_part_p2022_03_04' from '([\d_]+)$'); Cheers, Greg

Re: Column type modification in big tables

2024-08-09 Thread Greg Sabino Mullane
On Fri, Aug 9, 2024 at 6:39 AM Lok P wrote: > Thank you so much. Will definitely try to evaluate this approach. The Only > concern I have is , as this data is moving downstream with exactly the same > data type and length , so will it cause the downstream code to break while > using this column i

Re: Insert works but fails for merge

2024-08-11 Thread Greg Sabino Mullane
So it looks like the OP does not mind updating more than one row. If you want to keep it simple and not do a lot of casting, consider using a CTE to do a reverse-upsert and use a prepared statement. Prepare and cast once, and have your app send the raw uncasted strings many, many times: prepare fo

Re: Insert works but fails for merge

2024-08-13 Thread Greg Sabino Mullane
I just remembered that one of the complaints was not wanting to worry about looking up the data types. In my previous example, you can also leave out the types and Postgres will do the right thing. I prefer the explicit data type version for clarity, but though I would provide this one for complete

Re: Column type modification in big tables

2024-08-13 Thread Greg Sabino Mullane
On Sat, Aug 10, 2024 at 5:06 PM Lok P wrote: > Can someone through some light , in case we get 5-6hrs downtime for this > change , then what method should we choose for this Alter operation? > We can't really answer that. Only you know what resources you have, what risk/reward you are willing t

Re: Column type modification in big tables

2024-08-17 Thread Greg Sabino Mullane
On Thu, Aug 15, 2024 at 4:41 PM Lok P wrote: > Additionally , if we are okay with the 7.5hrs of down time , is my > calculation/extrapolation of total time consumption based on a sample > table, for direct alter, accurate? Because, in that case , I was thinking > it's less complex and also less

Re: Planet Postgres and the curse of AI

2024-08-20 Thread Greg Sabino Mullane
On Tue, Jul 23, 2024 at 12:45 PM Avinash Vallarapu < avinash.vallar...@gmail.com> wrote: > However, I do agree with Lawrence that it is impossible to prove whether > it is written by AI or a human. > AI can make mistakes and it might mistakenly point out that a blog is > written by AI (which I kno

Re: Planet Postgres and the curse of AI

2024-08-20 Thread Greg Sabino Mullane
On Fri, Jul 19, 2024 at 3:22 AM Laurenz Albe wrote: > Why not say that authors who repeatedly post grossly counterfactual or > misleading content can be banned? > I like this, and feel we are getting closer. How about: "Posts should be technically and factually correct. Use of AI should be used

Re: How to validate restore of backup?

2024-08-22 Thread Greg Sabino Mullane
On Thu, Aug 22, 2024 at 8:49 AM o1bigtenor wrote: > > > On Thu, Aug 22, 2024 at 6:24 AM Ron Johnson > wrote: > >> That's great on small databases. Not so practical when they're big. >> >> So - - - - what is the recommended procedure for 'large' databases? > Use a real backup system like pgBack

Re: Using PQsocketPoll() for PIPELINE mode

2024-08-27 Thread Greg Sabino Mullane
On Tue, Aug 27, 2024 at 9:20 AM Dominique Devienne wrote: > Once again, this is late, although my original questions are now 2 weeks > old. > After all, PQsocketPoll() has not been released yet officially. Thanks, > --DD > As this is so new, you might have better luck on -hackers than here. I've

Re: Pgbackrest specifying the default DB necessary/correct way ?

2024-08-28 Thread Greg Sabino Mullane
On Wed, Aug 28, 2024 at 1:39 AM KK CHN wrote: > In this DB serverI have other databases than the default "edb" > database. Specifying the above line aspg1-database=edb // I am > not sure this line is necessary or not ? > The pgbackrest process needs to connect to the database, w

Re: Ghost data from failed FDW transactions?

2024-08-28 Thread Greg Sabino Mullane
On Tue, Aug 27, 2024 at 9:03 PM Jacob Biesinger wrote: > I'm scratching my head at a few rows in the root DB, where it seems the > corresponding tenant transaction rolled back, but the root DB transaction > committed > ... > Before I jump into particulars, does this sound like expected behavior?

Re: PgBackRest Full backup and N/W reliability

2024-08-29 Thread Greg Sabino Mullane
On Thu, Aug 29, 2024 at 2:21 AM KK CHN wrote: > I am doing a full backup using PgBackRest from a production server to > Reposerver. > ... > If so, does the backup process start again from scratch ? or it > resumes from where the backup process is stopped ? > It resumes. You will see a m

Re: PgBackRest client_loop: send disconnect: Connection reset

2024-08-29 Thread Greg Sabino Mullane
On Thu, Aug 29, 2024 at 9:31 AM KK CHN wrote: > "Unable to acquire lock on file '/tmp/pgbackrest/Repo-backup.lock' > When this happens, take a look inside this file. If there is another pgbackrest process running, the pid will be inside that file. Kill that process before trying to run another b

Re: PgBackRest full backup first time : Verification

2024-08-30 Thread Greg Sabino Mullane
> > database size: 146.9GB, database backup size: 146.9GB > repo1: backup size: 20.6GB It looks to me as though everything is working as expected. You took a full backup of your system, which was around 147GB - most of which is in a tablespace. It got compressed down to 20GB. You then took two in

Re: Using left joins instead of inner joins as an optimization

2024-09-06 Thread Greg Sabino Mullane
On Fri, Sep 6, 2024 at 7:05 AM Xavier Solomon wrote: > > explain select b_id from b natural left join a; > results in a `Seq Scan on b`. Whereas the query > > explain select b_id from b natural join a; > results in a join with sequential scans on both a and b. > I think your example is a little

Re: How effectively do the indexing in postgres in such cases

2024-09-09 Thread Greg Sabino Mullane
Your questions are a little too vague to answer well, but let me try a bit. 1)In the query below , if the optimizer chooses tab1 as the driving table, > the index on just col1 should be enough or it should be (col1, tab1_id)? > No way to tell without trying it yourself. We need information on how

Re: infinite loop in an update statement

2024-09-11 Thread Greg Sabino Mullane
On Wed, Sep 11, 2024 at 6:14 AM Fabrice Chapuis wrote: > status = active >> wait event = NULL >> wait event type = NULL >> > That seems quite unlikely. Perhaps you are seeing the pg_stat_activity query itself? Try this: select state, now()-state_change, wait_event_type, wait_event, query from pg

Re: Ghost data from failed FDW transactions?

2024-09-11 Thread Greg Sabino Mullane
Any updates on this? A few replies from me inline: On Wed, Aug 28, 2024 at 12:18 PM Jacob Biesinger wrote: > There aren't many details in the docs around failure modes... is there > anything there that could cause this issue? > Nothing that I know of, but it's possible there is some sort of we

Re: Logical replication without direct link between publisher and subscriber?

2024-09-11 Thread Greg Sabino Mullane
> > Dumping changes periodically, sending them directly or uploading to cloud > storage and then downloading and applying them on the subscriber side. > But maybe there's a simpler option someone here knows about? How about using WAL shipping to populate a replica, and either query that directly

Re: Database schema for "custom fields"

2024-09-11 Thread Greg Sabino Mullane
I'd go with option 2 (jsonb), as it's likely already well supported by your applications, while the other approaches will require a good bit of customization. JSONB can be indexed, so performance should be on par with "traditional" tables. Cheers, Greg

Re: Performance degrade on insert on conflict do nothing

2024-09-11 Thread Greg Sabino Mullane
On Wed, Sep 11, 2024 at 1:02 AM Durgamahesh Manne wrote: > Hi > createdat | timestamp with time zone | | not null | now() >| plain| | | > modified | timestamp with time zone | | not null | now() >| plain| |

Re: Effects of REVOKE SELECT ON ALL TABLES IN SCHEMA pg_catalog FROM PUBLIC

2024-09-12 Thread Greg Sabino Mullane
On Thu, Sep 12, 2024 at 12:52 AM Andreas Joseph Krogh wrote: > I know PG is not designed for this, but I have this requirement > nonetheless… > I think preventing “most users and tools" from seeing/presenting this > information is “good enough”. > As pointed out, there are very many workarounds.

Re: Effects of REVOKE SELECT ON ALL TABLES IN SCHEMA pg_catalog FROM PUBLIC

2024-09-12 Thread Greg Sabino Mullane
On Thu, Sep 12, 2024 at 9:12 AM Dominique Devienne wrote: > On Thu, Sep 12, 2024 at 3:06 PM Greg Sabino Mullane > wrote: > > (Also note that determining if a database or user exists does not even > require a successful login to the cluster.) > > Hi. How so? I was not aware

Re: Effects of REVOKE SELECT ON ALL TABLES IN SCHEMA pg_catalog FROM PUBLIC

2024-09-12 Thread Greg Sabino Mullane
On Thu, Sep 12, 2024 at 9:21 AM Andreas Joseph Krogh wrote: > Yes, it *is* theater, but that doesn't prevent “compliance people” to > care about it. We have to take measures to prevent “information leaks”. > *shrug* Then the compliance people are not good at their jobs, frankly. But if it works

Re: Will hundred of thousands of this type of query cause Parsing issue

2024-09-13 Thread Greg Sabino Mullane
On Fri, Sep 13, 2024 at 11:35 AM Wong, Kam Fook (TR Technology) < kamfook.w...@thomsonreuters.com> wrote: > 1) Where does query parsing occur? > > Always on the server side, although your driver may do something as well. 2) Will this cause extra parsing to the posgress DB? > Yes > Any pg sys

Re: IO related waits

2024-09-17 Thread Greg Sabino Mullane
On Mon, Sep 16, 2024 at 11:56 PM veem v wrote: > So what can be the caveats in this approach, considering transactions > meant to be ACID compliant as financial transactions. > Financial transactions need to be handled with care. Only you know your business requirements, but as Christophe pointe

Re: question on plain pg_dump file usage

2024-09-17 Thread Greg Sabino Mullane
On Tue, Sep 17, 2024 at 8:22 AM Zwettler Markus (OIZ) < markus.zwett...@zuerich.ch> wrote: > pg_dump -F p -f dump.sql … > > sed -i "s/old_name/new_name/g" > > psql -f dump.sql … > Why not rename afterwards? Just "pg_dump mydb | psql -h newhost -f -" and rename things via ALTER. Certainly much saf

Re: Question about when PostgreSQL 11.0 was released

2021-03-09 Thread Greg Sabino Mullane
do.org/postgres_all_versions - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iF0EARECAB0WIQQlKd9quPeUB+lERbS8m5BnFJZKyAUCYEd1hgAKCRC8m5

Re: Disk is filling up with large files. How can I clean?

2024-10-09 Thread Greg Sabino Mullane
On Wed, Oct 9, 2024 at 4:10 AM Mikael Petterson wrote: > Hi, > > I find our disk is filling up. > > sudo find /var/lib -type f -size +100M -exec ls -lh {} \; | awk '{ print > $9 ": " $5 }' > ... Those files only add up to about 30GB. That's pretty small these days : time for a bigger disk? Or pe

Re: Question on indexes

2024-10-11 Thread Greg Sabino Mullane
(please start a new thread in the future rather than replying to an existing one) You cannot query on b and use an index on (a,b) as you observed. However, you can have two indexes: index1(a) index2(b) Postgres will be able to combine those when needed in the case where your WHERE clause needs t

  1   2   >