Thank you for the advice. Shortly after submitting this bug I found Tom Lane's suggestion here
http://www.postgresql.org/message-id/201109082233.p88mxbge026...@wwwmaster.postgresql.org Once we had the core dump we were quickly able to figure out that the problem was in one our custom functions. Thank you for the help. Kim Applegate kapplegate(at)apsalar(dot)com writes: > One of our queries has started failing randomly on our slaves with > invalid memory alloc request size 18446744073709537559 It's hard to say much with that amount of information. Is it always the exact same number? The root cause is probably either corrupted data (that is, a trashed length word for some variable-width field) or some internal logic bug that's causing the server to miscompute how much memory it needs for some transient allocation. You could confirm or refute the corrupt-data hypothesis by seeing if you can pg_dump each of the tables referenced by the failing procedure. If pg_dump fails with the same error then it's corrupt data, else not. If it's a bug, though, we'd still be needing more info to track it down. Don't suppose you'd want to change that specific ERROR to a PANIC so we could get a stack trace :-( regards, tom lane On Fri, Jun 21, 2013 at 7:05 AM, <kappleg...@apsalar.com> wrote: > The following bug has been logged on the website: > > Bug reference: 8245 > Logged by: Kim Applegate > Email address: kappleg...@apsalar.com > PostgreSQL version: 9.2.4 > Operating system: OpenIndiana > Description: > > > > > > One of our queries has started failing randomly on our slaves with > > > invalid memory alloc request size 18446744073709537559 > > > This is eventually repeatable on any slave that has failed. Of the 64 > slaves > it will fail on 1-5 of them for any given run. The stored procedure that is > failing queries on a partition with 64 children and only returns rows from > 4 > of the child tables. > > > Turning Debug on in the logs only gives this > > > Jun 20 06:50:53 * postgres[14825]: [ID 748848 local2.warning] [27-1] ERROR: > invalid memory alloc request size 18446744073709532101 > Jun 20 06:50:53 * postgres[14825]: [ID 748848 local2.warning] [27-2] > CONTEXT: SQL function "events_args_top500_week" statement 1 > Jun 20 06:50:53 * postgres[14825]: [ID 748848 local2.debug] [28-1] DEBUG: > shmem_exit(0): 7 callbacks to make > Jun 20 06:50:53 * postgres[14825]: [ID 748848 local2.debug] [29-1] DEBUG: > proc_exit(0): 3 callbacks to make > Jun 20 06:50:53 * postgres[14825]: [ID 748848 local2.debug] [30-1] DEBUG: > exit(0) > Jun 20 06:50:53 * postgres[14825]: [ID 748848 local2.debug] [31-1] DEBUG: > shmem_exit(-1): 0 callbacks to make > Jun 20 06:50:53 * postgres[14825]: [ID 748848 local2.debug] [32-1] DEBUG: > proc_exit(-1): 0 callbacks to make > > > apsalar=# select version(); > > > -------------------------------------------------------------------------------- > PostgreSQL 9.2.4 on x86_64-pc-solaris2.11, compiled by gcc (GCC) 4.7.2, > 64-bit > (1 row) > > > > > >
