Re: [GENERAL] corruption issue after server crash - ERROR: unexpected chunk number 0

2013-11-22 Thread Kevin Grittner
Mike Broers wrote: > vacuumb avz, pg_dumpall, and vacuum freeze analyze on the former > standby database that received the corruption via replication all > came back without errors.  Is the vacuum freeze intended to > potentially fix problems or just reveal if other tables may have > corruption,

Re: [GENERAL] corruption issue after server crash - ERROR: unexpected chunk number 0

2013-11-22 Thread Mike Broers
vacuumb avz, pg_dumpall, and vacuum freeze analyze on the former standby database that received the corruption via replication all came back without errors. Is the vacuum freeze intended to potentially fix problems or just reveal if other tables may have corruption, Im trying to decide if this nee

Re: [GENERAL] corruption issue after server crash - ERROR: unexpected chunk number 0

2013-11-21 Thread Scott Marlowe
On Thu, Nov 21, 2013 at 4:14 PM, John R Pierce wrote: > On 11/21/2013 2:51 PM, Kevin Grittner wrote: >> >> That leaves three possibilities: >>(1) fsync doesn't actually guarantee persistence in your stack. > > > I'll put my $5 on (1) virtualization stacks add way too much ooga-booga > to

Re: [GENERAL] corruption issue after server crash - ERROR: unexpected chunk number 0

2013-11-21 Thread John R Pierce
On 11/21/2013 2:51 PM, Kevin Grittner wrote: That leaves three possibilities: (1) fsync doesn't actually guarantee persistence in your stack. I'll put my $5 on (1) virtualization stacks add way too much ooga-booga to the storage stack, and tend to play fast and loose with write buffer

Re: [GENERAL] corruption issue after server crash - ERROR: unexpected chunk number 0

2013-11-21 Thread Kevin Grittner
Mike Broers wrote: > Is there anything I should look out for with vacuum freeze? Just check the logs and the vacuum output for errors and warnings. -- Kevin Grittner EDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-general mailing list (pgsql-general@postg

Re: [GENERAL] corruption issue after server crash - ERROR: unexpected chunk number 0

2013-11-21 Thread Mike Broers
Thanks, after this pg_dumpall I am going to see what kind of impact I can expect from running VACUUM FREEZE ANALYZE (normally I just run vacuumdb -avz nightly via a cron job) and schedule time to run this in production against all the tables in the database. Is there anything I should look out for

Re: [GENERAL] corruption issue after server crash - ERROR: unexpected chunk number 0

2013-11-21 Thread Kevin Grittner
Mike Broers wrote: > Thanks for the response.  fsync and full_page_writes are both on. > [ corruption appeared following power loss on the machine hosing > the VM running PostgreSQL ] That leaves three possibilities:   (1)  fsync doesn't actually guarantee persistence in your stack.   (2)  Ther

Re: [GENERAL] corruption issue after server crash - ERROR: unexpected chunk number 0

2013-11-21 Thread Mike Broers
Thanks for the response. fsync and full_page_writes are both on. Our database runs on a managed hosting provider's vmhost server/san, I can possibly request for them to provide some hardware test results - do you have any specifics diagnostics in mind? The crash was apparently due to our vmhost

Re: [GENERAL] corruption issue after server crash - ERROR: unexpected chunk number 0

2013-11-21 Thread Kevin Grittner
Mike Broers wrote: > Hello we are running postgres 9.2.5 on RHEL6, our production > server crashed hard and when it came back up our logs were > flooded with: > ERROR:  unexpected chunk number 0 (expected 1) for toast value 117927127 in > pg_toast_19122 Your database is corrupted.  Unless you

[GENERAL] corruption issue after server crash - ERROR: unexpected chunk number 0

2013-11-21 Thread Mike Broers
Hello we are running postgres 9.2.5 on RHEL6, our production server crashed hard and when it came back up our logs were flooded with: STATEMENT: SELECT "session_session"."session_key", "session_session"."session_data", "session_session"."expire_date", "session_session"."nonce" FROM "session_sessi