VACUUM FULL, power failure results in unrecoverable space

2024-12-02 Thread Pierre Barre
Hello, I encountered an issue while attempting to reclaim space from a heavily bloated table: Initial analysis using https://github.com/ioguix/pgsql-bloat-estimation/blob/master/table/table_bloat.sql indicated approximately 600GB of bloat in the table. I initiated a VACUUM FULL operation to r

Re: Preallocation changes in Postgresql 16

2024-12-26 Thread Pierre Barre
Hello, It seems that I am running into this issue as well. Is it likely that this would ever be a config option? Best, Pierre Barre On Fri, May 3, 2024, at 05:11, Riku Iki wrote: > I did the testing and confirmed that this was the issue. > > I run following query: > > cr

Re: PostgreSQL on S3-backed Block Storage with Near-Local Performance

2025-07-18 Thread Pierre Barre
NFS directly. Best, Pierre On Fri, Jul 18, 2025, at 06:40, Laurenz Albe wrote: > On Fri, 2025-07-18 at 00:57 +0200, Pierre Barre wrote: >> Looking forward to your feedback and questions! > > I think the biggest hurdle you will have to overcome is to > convince notorio

PostgreSQL on S3-backed Block Storage with Near-Local Performance

2025-07-17 Thread Pierre Barre
Hi everyone, I wanted to share a project I've been working on that enables PostgreSQL to run on S3 storage while maintaining performance comparable to local NVMe. The approach uses block-level access rather than trying to map filesystem operations to S3 objects. ZeroFS: https://github.com/Barr

Re: PostgreSQL on S3-backed Block Storage with Near-Local Performance

2025-07-18 Thread Pierre Barre
ormance in that case :) > > Cheers, > Seref > > > On Fri, Jul 18, 2025 at 11:58 AM Pierre Barre wrote: >> __ >> Hi Seref, >> >> For the benchmarks, I used Hetzner's cloud service with the following setup: >> >> - A Hetzner s3 bucket in th

Re: PostgreSQL on S3-backed Block Storage with Near-Local Performance

2025-07-18 Thread Pierre Barre
fy how exactly you're running > postgres in your tests? A specific AWS service? What's the test > infrastructure that sits above the file system? > > On Thu, Jul 17, 2025 at 11:59 PM Pierre Barre wrote: >> Hi everyone, >> >> I wanted to share a project I

Re: PostgreSQL on S3-backed Block Storage with Near-Local Performance

2025-07-18 Thread Pierre Barre
ri, Jul 18, 2025, at 12:57, Pierre Barre wrote: > Hi Seref, > > For the benchmarks, I used Hetzner's cloud service with the following setup: > > - A Hetzner s3 bucket in the FSN1 region > - A virtual machine of type ccx63 48 vCPU 192 GB memory > - 3 ZeroFS nbd devices (sam

Re: PostgreSQL on S3-backed Block Storage with Near-Local Performance

2025-07-24 Thread Pierre Barre
! I applaud you. > > On Thu, Jul 17, 2025, 4:59 PM Pierre Barre wrote: >> Hi everyone, >> >> I wanted to share a project I've been working on that enables PostgreSQL to >> run on S3 storage while maintaining performance comparable to local NVMe. >> The ap

Re: PostgreSQL on S3-backed Block Storage with Near-Local Performance

2025-07-24 Thread Pierre Barre
ierre On Fri, Jul 18, 2025, at 06:40, Laurenz Albe wrote: > On Fri, 2025-07-18 at 00:57 +0200, Pierre Barre wrote: >> Looking forward to your feedback and questions! > > I think the biggest hurdle you will have to overcome is to > convince notoriously paranoid DBAs that this

Re: PostgreSQL on S3-backed Block Storage with Near-Local Performance

2025-07-26 Thread Pierre Barre
8a3186fb749ea3 On Sat, Jul 26, 2025, at 09:51, Pierre Barre wrote: > Ah, by "shared storage" I mean that each node can acquire exclusivity, not > that they can both R/W to it at the same time. > > > Some pretty well-known cases of storage / compute separation (Aurora

Re: PostgreSQL on S3-backed Block Storage with Near-Local Performance

2025-07-26 Thread Pierre Barre
s of storage / compute separation (Aurora, Neon) > also share the storage between instances, > that's why I'm a bit confused by your reply. I thought you're thinking about > this approach too, that's why I mentioned what kind of challenges one may > have on that p

Re: PostgreSQL on S3-backed Block Storage with Near-Local Performance

2025-07-26 Thread Pierre Barre
e > on any write operation. > That alone is not that simple. You will have to modify some locking logic. > Most likely do a lot of other changes in a lot of places, Postgres was not > just built with the assumption that the storage can be shared. > > -Vladimir > > On Fri, Ju

Re: PostgreSQL on S3-backed Block Storage with Near-Local Performance

2025-07-25 Thread Pierre Barre
7.723 ms initial connection time = 46.089 ms tps = 252.878721 (without initial connection time) Not great barebones with with synchronous_commit, but still usable! Best, Pierre On Fri, Jul 25, 2025, at 00:44, Pierre Barre wrote: >> This then begs the obvious question of how fast i

Re: PostgreSQL on S3-backed Block Storage with Near-Local Performance

2025-07-24 Thread Pierre Barre
. Best, Pierre On Thu, Jul 24, 2025, at 21:44, Nico Williams wrote: > On Fri, Jul 18, 2025 at 12:57:39PM +0200, Pierre Barre wrote: >> - Postgres configured accordingly memory-wise as well as with >> synchronous_commit = off, wal_init_zero = off and wal_recycle = off. > > Bi

Re: PostgreSQL on S3-backed Block Storage with Near-Local Performance

2025-07-24 Thread Pierre Barre
> This then begs the obvious question of how fast is this with > synchronous_commit = on? Probably not awful, especially with commit_delay. I'll try that and report back. Best, Pierre On Fri, Jul 25, 2025, at 00:03, Jeff Ross wrote: > On 7/24/25 13:50, Pierre Barre wrote: >

Re: PostgreSQL on S3-backed Block Storage with Near-Local Performance

2025-07-25 Thread Pierre Barre
er of transactions per client: 1000 number of transactions actually processed: 10/10 number of failed transactions: 0 (0.000%) latency average = 337.762 ms initial connection time = 43.969 ms tps = 296.066616 (without initial connection time) Best, Pierre On Fri, Jul 25, 2025, at 11:25, Pierre

Re: PostgreSQL on S3-backed Block Storage with Near-Local Performance

2025-07-25 Thread Pierre Barre
, Pierre On Sat, Jul 26, 2025, at 03:16, Pierre Barre wrote: > I built postgres (same version, 16.9) but --with-block-size=32 (I'd > really love if this would be a initdb time flag!) and did some more > testing: > > synchronous_commit = off > > postgres@zerofs:~$ pgbench