Re: [GENERAL] Vacuum Full - Questions

2016-08-31 Thread Gary Evans
Hi Patrick, I believe Vacuum full rebuilds the indexes automatically by default, as a new copy of the table is created. Because the indexes are new, no stats are available to the optimiser to make an informed decision about whether to utilise it or not, so it doesn't. Once the analyze is perform

Re: [GENERAL] UPDATE OR REPLACE?

2016-08-31 Thread Michael Paquier
On Thu, Sep 1, 2016 at 12:10 PM, dandl wrote: > Sqlite has options to handle an update that causes a duplicate key. Is there > anything similar in Postgres? > This is not an UPSERT. The scenario is an UPDATE that changes some key field > so that there is now a duplicate key. In Sqlite this handled

Re: [GENERAL] Vacuum Full - Questions

2016-08-31 Thread Venkata B Nagothi
On Thu, Sep 1, 2016 at 10:32 AM, Patrick B wrote: > > > 2016-09-01 11:53 GMT+12:00 Venkata B Nagothi : > >> >> On Thu, Sep 1, 2016 at 8:41 AM, Patrick B >> wrote: >> >>> Hi guys, >>> >>> A dev has ran a VACUUM FULL command into our test database running >>> PostgreSQL 9.5 (I know... goddamn)

[GENERAL] UPDATE OR REPLACE?

2016-08-31 Thread dandl
Sqlite has options to handle an update that causes a duplicate key. Is there anything similar in Postgres? This is not an UPSERT. The scenario is an UPDATE that changes some key field so that there is now a duplicate key. In Sqlite this handled as: UPDATE OR IGNORE table SET UPDATE OR R

Re: [GENERAL] Vacuum Full - Questions

2016-08-31 Thread Patrick B
2016-09-01 11:53 GMT+12:00 Venkata B Nagothi : > > On Thu, Sep 1, 2016 at 8:41 AM, Patrick B > wrote: > >> Hi guys, >> >> A dev has ran a VACUUM FULL command into our test database running >> PostgreSQL 9.5 (I know... goddamn)... >> >> ... after the Vacuum Full, some queries start using SEQ s

Re: [GENERAL] Vacuum Full - Questions

2016-08-31 Thread Venkata B Nagothi
On Thu, Sep 1, 2016 at 8:41 AM, Patrick B wrote: > Hi guys, > > A dev has ran a VACUUM FULL command into our test database running > PostgreSQL 9.5 (I know... goddamn)... > > ... after the Vacuum Full, some queries start using SEQ scans instead of > indexes... > > Does that happen because of

Re: [GENERAL] Vacuum Full - Questions

2016-08-31 Thread Adam Brusselback
Yes that very well could happen because the size of the table changed, as well as stats being more accurate now. Just because you have a seq scan doesn't mean the planer is making a bad choice.

Re: [GENERAL] Clustered index to preserve data locality in a multitenant application?

2016-08-31 Thread Nicolas Grilly
On Thu, Sep 1, 2016 at 12:31 AM, Nicolas Grilly wrote: > In DB2, it seems possible to define a "clustering index" that determines > how rows are physically ordered in the "table space" (the heap). > > The documentation says: "When a table has a clustering index, an INSERT > statement causes DB2 t

[GENERAL] Rackspace to RDS using DMS (Postgres 9.2)

2016-08-31 Thread Patrick B
Hi guys, I posted this question on the ADMIN list but will post here as well so more people can comment... https://www.postgresql.org/message-id/CAJNY3it_AfxJhmwMHtpiAbHG47GS5rJOAUgfHw%2BGm5OXCbUm7w%40mail.gmail.com I've got a 2.3TB Database running at Rackspace... We'll be migrating it to RDS Pos

[GENERAL] Vacuum Full - Questions

2016-08-31 Thread Patrick B
Hi guys, A dev has ran a VACUUM FULL command into our test database running PostgreSQL 9.5 (I know... goddamn)... ... after the Vacuum Full, some queries start using SEQ scans instead of indexes... Does that happen because of the size of the table? The table that I'm referring to is 150MB bi

Re: [GENERAL] Clustered index to preserve data locality in a multitenant application?

2016-08-31 Thread Nicolas Grilly
On Tue, Aug 30, 2016 at 8:17 PM, Kenneth Marshall wrote: > We have been using the extension pg_repack to keep a table groomed into > cluster order. With an appropriate FILLFACTOR to keep updates on the same > page, it works well. The issue is that it needs space to rebuild the new > index/table.

Re: [GENERAL] Clustered index to preserve data locality in a multitenant application?

2016-08-31 Thread Nicolas Grilly
On Thu, Sep 1, 2016 at 12:05 AM, Ben Chobot wrote: > If what they did 3 years ago is similar to what you are trying to do > today, who cares what they are doing today? (Besides using pg_repack > instead of pg_reorg, of course.) > I'm curious because, in the meantime, Instagram could have stopped

Re: [GENERAL] Clustered index to preserve data locality in a multitenant application?

2016-08-31 Thread Ben Chobot
On Aug 31, 2016, at 2:55 PM, Nicolas Grilly wrote: > > It looks like Instagram has been using pg_reorg (the ancestor of pg_repack) > to keep all likes from the same user contiguous on disk, in order to minimize > disk seeks. > > http://instagram-engineering.tumblr.com/post/40781627982/handling

Re: [GENERAL] Clustered index to preserve data locality in a multitenant application?

2016-08-31 Thread Nicolas Grilly
On Tue, Aug 30, 2016 at 8:17 PM, Kenneth Marshall wrote: > We have been using the extension pg_repack to keep a table groomed into > cluster order. With an appropriate FILLFACTOR to keep updates on the same > page, it works well. The issue is that it needs space to rebuild the new > index/table.

Re: [GENERAL] Clustered index to preserve data locality in a multitenant application?

2016-08-31 Thread Nicolas Grilly
On Wed, Aug 31, 2016 at 6:05 PM, Kenneth Marshall wrote: > We just run it via cron. In our case, we run it once a day, but depending > on > your churn, it could be run once a week or more. > Could you provide some numbers: what is the size of the tables or tables that are repacked? how long does

Re: [GENERAL] Clustered index to preserve data locality in a multitenant application?

2016-08-31 Thread Kenneth Marshall
On Wed, Aug 31, 2016 at 06:06:54PM +0200, Nicolas Grilly wrote: > On Wed, Aug 31, 2016 at 6:05 PM, Kenneth Marshall wrote: > > > We just run it via cron. In our case, we run it once a day, but depending > > on > > your churn, it could be run once a week or more. > > > > Could you provide some nu

Re: [GENERAL] Clustered index to preserve data locality in a multitenant application?

2016-08-31 Thread Kenneth Marshall
On Wed, Aug 31, 2016 at 05:23:50PM +0200, Nicolas Grilly wrote: > On Tue, Aug 30, 2016 at 8:17 PM, Kenneth Marshall wrote: > > > We have been using the extension pg_repack to keep a table groomed into > > cluster order. With an appropriate FILLFACTOR to keep updates on the same > > page, it works

Re: [GENERAL] Clustered index to preserve data locality in a multitenant application?

2016-08-31 Thread Nicolas Grilly
Mike Sofen wrote: > For Nicolas’s situation, that would require 10,000 partitions – not very > useful, and each partition would be very small. > This is exactly my conclusion about using partitions in my situation. > In Postgres, as you mentioned, clustering is a “one time” operation but > only

Re: [GENERAL] Clustered index to preserve data locality in a multitenant application?

2016-08-31 Thread Nicolas Grilly
Eduardo Morras wrote: > Check BRIN indexs, they are "designed for handling very large tables in > which certain columns have some natural correlation with their physical > location within the table", I think they fit your needs. Yes, a BRIN index on the tenant ID would be very useful if the row

Re: [GENERAL] Clustered index to preserve data locality in a multitenant application?

2016-08-31 Thread Nicolas Grilly
On Tue, Aug 30, 2016 at 8:17 PM, Kenneth Marshall wrote: > We have been using the extension pg_repack to keep a table groomed into > cluster order. With an appropriate FILLFACTOR to keep updates on the same > page, it works well. The issue is that it needs space to rebuild the new > index/table.

[GENERAL] postgresql.conf RH comment, and a systemd RH note

2016-08-31 Thread Karl O. Pinc
Hi, FYI, the RH rpm contains the following comment in postgresql.conf, which is not in the postgresql.org rpm. I found it helpful. @@ -61,11 +61,7 @@ # defaults to 'localhost'; use '*' for all # (change requires restart) #port = 5432