Re: Incremental aggregate/rollup strategy advice

2019-07-08 Thread Tatsuo Ishii
> Thank you for your response, I have followed the discussion on Hackers with > interest. I hope that your efforts are a great success! In my case, I need > to find a solution available in shipping versions of Postgres. But, since > you've joined in, I'm curious: What is the advantage of a material

Re: Incremental aggregate/rollup strategy advice

2019-07-08 Thread Morris de Oryx
Thanks Steven, nice suggestions. I should have mentioned that the deployment setup is on RDS on PG 11.x, which rules out those extensions. I've looked at TimescaleDB several times, and it looks pretty great. I've now read through some of the archives from years back when pg_xact_commit_timestamp w

RE: Incremental aggregate/rollup strategy advice

2019-07-08 Thread Steven Winfield
There are a couple of extensions that might help you: PipelineDB[1]: Their “Continuous Views” could be useful. A big caveat here is that PipelineDB’s future is uncertain[2], though. I haven’t used it myself, but it looks like you could put triggers onto your existing tables to insert data in Pip

Re: Incremental aggregate/rollup strategy advice

2019-07-08 Thread Morris de Oryx
Tatsuo, Thank you for your response, I have followed the discussion on Hackers with interest. I hope that your efforts are a great success! In my case, I need to find a solution available in shipping versions of Postgres. But, since you've joined in, I'm curious: What is the advantage of a materia

Re: Incremental aggregate/rollup strategy advice

2019-07-08 Thread Tatsuo Ishii
> * Full queries will take too long, and will scale poorly. So, MATERIALIZED > VIEW is unappealing. So, rollup tables as it's possible to update them > incrementally. F.Y.I. There is a proposal to implemnt incremental updation against MATERIALIZED VIEW. It is still in WIP patch but currently it su

Incremental aggregate/rollup strategy advice

2019-07-08 Thread Morris de Oryx
I'm researching strategies for incrementally updating aggregate/rollup tables. The problem is how to do so without double-counting changes, and not skipping changes. I know enough about concurrency issues to ask the question, but do *not* know enough about the features and details of Postgres' conc