Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-05 Thread Jeff Davis
On Fri, 2013-07-05 at 08:21 -0500, Jon Nelson wrote: > Wonderful. Is removing the GUC something that I should do or should > that be done by somebody that knows more about what they are doing? (I > am happy to give it a go!) I'll take care of that. > Should the small test program that I made also

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-05 Thread Jon Nelson
On Fri, Jul 5, 2013 at 2:23 AM, Greg Smith wrote: > On 7/5/13 2:50 AM, Jeff Davis wrote: >> >> So, my simple conclusion is that glibc emulation should be about the >> same as what we're doing now, so there's no reason to avoid it. That >> means, if posix_fallocate() is present, we should use it, b

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-05 Thread Greg Smith
On 7/5/13 2:50 AM, Jeff Davis wrote: So, my simple conclusion is that glibc emulation should be about the same as what we're doing now, so there's no reason to avoid it. That means, if posix_fallocate() is present, we should use it, because it's either the same (if emulated in glibc) or significa

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-04 Thread Jeff Davis
On Mon, 2013-07-01 at 00:52 -0400, Greg Smith wrote: > On 6/30/13 9:28 PM, Jon Nelson wrote: > > The performance of the latter (new) test sometimes seems to perform > > worse and sometimes seems to perform better (usually worse) than > > either of the other two. In all cases, posix_fallocate perfor

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-01 Thread Greg Smith
On 7/1/13 3:44 PM, Robert Haas wrote: Yeah. If the patch isn't going to be a win on RHEL 5, I'd consider that a good reason to scrap it for now and revisit it in 3 years. There are still a LOT of people running RHEL 5, and the win isn't big enough to engineer a more complex solution. I'm still

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-01 Thread Peter Eisentraut
On 7/1/13 12:34 PM, Jeff Davis wrote: > On Sun, 2013-06-30 at 16:09 -0400, Andrew Dunstan wrote: >> It was originally generated. Since then it's been maintained by hand. > > What is the procedure for maintaining it by hand? Edit away. > Why are > HAVE_POSIX_SIGNALS and HAVE_SYNC_FILE_RANGE in th

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-01 Thread Robert Haas
On Mon, Jul 1, 2013 at 2:17 PM, Jeff Davis wrote: > On Tue, 2013-07-02 at 02:13 +0900, Fujii Masao wrote: >> Even in that case, if a user can easily know which platform posix_fallocate >> should be used in, we can commit the patch with the configurable GUC >> parameter. > > I disagree here. We're

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-01 Thread Jeff Davis
On Tue, 2013-07-02 at 02:13 +0900, Fujii Masao wrote: > Even in that case, if a user can easily know which platform posix_fallocate > should be used in, we can commit the patch with the configurable GUC > parameter. I disagree here. We're not talking about a huge win; this speedup may not even be

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-01 Thread Fujii Masao
On Tue, Jul 2, 2013 at 1:55 AM, Jeff Davis wrote: > On Sun, 2013-06-30 at 18:55 -0400, Greg Smith wrote: >> This makes platform level testing a lot easier, thanks. Attached is an >> updated copy of that program with some error checking. If the files it >> creates already existed, the code didn't

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-01 Thread Jeff Davis
On Sun, 2013-06-30 at 18:55 -0400, Greg Smith wrote: > This makes platform level testing a lot easier, thanks. Attached is an > updated copy of that program with some error checking. If the files it > creates already existed, the code didn't notice, and a series of write > errors happened. If

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-01 Thread Jeff Davis
On Sun, 2013-06-30 at 16:09 -0400, Andrew Dunstan wrote: > It was originally generated. Since then it's been maintained by hand. What is the procedure for maintaining it by hand? Why are HAVE_POSIX_SIGNALS and HAVE_SYNC_FILE_RANGE in there (though commented out), but not HAVE_POSIX_FADVISE? Regar

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-07-01 Thread Jon Nelson
On Sun, Jun 30, 2013 at 11:52 PM, Greg Smith wrote: > On 6/30/13 9:28 PM, Jon Nelson wrote: >> >> The performance of the latter (new) test sometimes seems to perform >> worse and sometimes seems to perform better (usually worse) than >> either of the other two. In all cases, posix_fallocate perfor

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Greg Smith
On 6/30/13 9:28 PM, Jon Nelson wrote: The performance of the latter (new) test sometimes seems to perform worse and sometimes seems to perform better (usually worse) than either of the other two. In all cases, posix_fallocate performs better, but I don't have a sufficiently old kernel to test wit

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Jon Nelson
On Sun, Jun 30, 2013 at 6:49 PM, Greg Smith wrote: > On 5/28/13 10:00 PM, Jon Nelson wrote: > >> A note: The attached test program uses *fsync* instead of *fdatasync* >> after calling fallocate (or writing out 16MB of zeroes), per an >> earlier suggestion. > > > I tried this out on the RHEL5 platf

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Greg Smith
On 5/28/13 10:00 PM, Jon Nelson wrote: A note: The attached test program uses *fsync* instead of *fdatasync* after calling fallocate (or writing out 16MB of zeroes), per an earlier suggestion. I tried this out on the RHEL5 platform I'm worried about now. There's something weird about the tes

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Jon Nelson
On Sun, Jun 30, 2013 at 5:55 PM, Greg Smith wrote: > > > pwrite(4, "\0", 1, 16769023)= 1 > pwrite(4, "\0", 1, 16773119)= 1 > pwrite(4, "\0", 1, 16777215)= 1 > > That's glibc helpfully converting your call to posix_fallocate into small > writes, because the OS do

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Greg Smith
On 6/30/13 2:01 PM, Jeff Davis wrote: Simple test program attached, which creates two files and fills them: one by 2048 8KB writes; and another by 1 posix_fallocate of 16MB. Then, I just cmp the resulting files (and also "ls" them, to make sure they are 16MB). This makes platform level testing

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Andrew Dunstan
On 06/30/2013 03:50 PM, Jeff Davis wrote: On Sun, 2013-06-30 at 11:11 -0700, Jeff Davis wrote: Unless something surprising comes up, or someone thinks and objection has been missed, I am going to commit this soon. Quick question to anyone who happens to know: What is the standard procedure fo

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Jeff Davis
On Sun, 2013-06-30 at 11:11 -0700, Jeff Davis wrote: > Unless something surprising comes up, or someone thinks and objection > has been missed, I am going to commit this soon. Quick question to anyone who happens to know: What is the standard procedure for changes to pg_config.h.win32? I looked a

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Jeff Davis
On Fri, 2013-06-28 at 11:38 -0700, Josh Berkus wrote: > Since Greg seems to be busy, what needs to be done to test this? As I understand it, he was mainly asking if posix_fallocate works at all. I tried to address that question with a simple test, which behaves as I expected it to: http://www.pos

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-30 Thread Jeff Davis
On Tue, 2013-05-28 at 22:10 -0400, Greg Smith wrote: > I was just thinking of something to run in your test program, not > another build time check. Just run the new allocation sequence, and > then check the resulting WAL file for a) correct length, and b) 16K of > zero bytes. I would like to

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-28 Thread Josh Berkus
On 06/20/2013 07:19 PM, Jeff Davis wrote: > On Sun, 2013-06-16 at 23:53 -0500, Jon Nelson wrote: >> Please find attached v5 of the patch, with the above correction in place. >> The only change from the v4 patch is wrapping the if >> (wal_use_fallocate) block in an #ifdef USE_POSIX_FALLOCATE. >> Tha

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-20 Thread Jeff Davis
On Sun, 2013-06-16 at 23:53 -0500, Jon Nelson wrote: > Please find attached v5 of the patch, with the above correction in place. > The only change from the v4 patch is wrapping the if > (wal_use_fallocate) block in an #ifdef USE_POSIX_FALLOCATE. > Thanks! Thank you. Greg, are you still working on

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-16 Thread Jon Nelson
On Sun, Jun 16, 2013 at 9:59 PM, Jon Nelson wrote: > On Fri, Jun 14, 2013 at 12:06 PM, Jeff Davis wrote: >> On Sat, 2013-05-25 at 13:55 -0500, Jon Nelson wrote: .. >> * You check for the presence of posix_fallocate at configure time, but >> don't #ifdef the call site. It looks like you removed th

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-16 Thread Jon Nelson
On Fri, Jun 14, 2013 at 12:06 PM, Jeff Davis wrote: > On Sat, 2013-05-25 at 13:55 -0500, Jon Nelson wrote: >> Ack. I've revised the patch to always have the GUC (for now), default >> to false, and if configure can't find posix_fallocate (or the user >> disables it by way of pg_config_manual.h) th

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-14 Thread Jeff Davis
On Fri, 2013-06-14 at 13:21 -0400, Greg Smith wrote: > I'm planning to duplicate Jon's test program on a few machines here, and > then see if that turns into a useful latency improvement for clients. > I'm trying to get this pgbench rate limit stuff working first though, > because one of the tes

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-14 Thread Jeff Davis
On Tue, 2013-06-11 at 12:58 -0400, Stephen Frost wrote: > My main question is really- would this be useful for extending > *relations*? Apologies if it's already been discussed; I do plan to go > back and read the threads about this more fully, but I wanted to voice > my support for using posix_fa

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-14 Thread Greg Smith
On 6/14/13 1:06 PM, Jeff Davis wrote: Why have a GUC here at all? Perhaps this was already discussed, and I missed it? Is it just for testing purposes, or did you intend for it to be in the final version? You have guessed correctly! I suggested it stay in there only to make review benchmarkin

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-14 Thread Jeff Davis
On Sat, 2013-05-25 at 13:55 -0500, Jon Nelson wrote: > Ack. I've revised the patch to always have the GUC (for now), default > to false, and if configure can't find posix_fallocate (or the user > disables it by way of pg_config_manual.h) then it remains a GUC that > simply can't be changed. Why h

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-12 Thread Robert Haas
On Tue, Jun 11, 2013 at 12:58 PM, Stephen Frost wrote: > * Merlin Moncure (mmonc...@gmail.com) wrote: >> It's understood that posix_fallocate is faster at this -- the question >> on the table is 'does this matter in context of postgres?'. >> Personally I think this patch should go in regardless --

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-11 Thread Merlin Moncure
On Tue, Jun 11, 2013 at 12:45 PM, Greg Smith wrote: > On 6/11/13 12:22 PM, Merlin Moncure wrote: > >> Personally I think this patch should go in regardless -- the concerns >> made IMNSHO are specious. > > That's nice, but we have this process for validating whether features go in > or not that rel

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-11 Thread Jon Nelson
On Tue, Jun 11, 2013 at 12:49 PM, Stefan Drees wrote: > On 2013-06-11 19:45 CEST, Greg Smith wrote: >> >> On 6/11/13 12:22 PM, Merlin Moncure wrote: >> >>> Personally I think this patch should go in regardless -- the concerns >>> made IMNSHO are specious. >> >> >> That's nice, but we have this pro

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-11 Thread Stefan Drees
On 2013-06-11 19:45 CEST, Greg Smith wrote: On 6/11/13 12:22 PM, Merlin Moncure wrote: Personally I think this patch should go in regardless -- the concerns made IMNSHO are specious. That's nice, but we have this process for validating whether features go in or not that relies on review inste

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-11 Thread Greg Smith
On 6/11/13 12:22 PM, Merlin Moncure wrote: Personally I think this patch should go in regardless -- the concerns made IMNSHO are specious. That's nice, but we have this process for validating whether features go in or not that relies on review instead of opinions. -- Greg Smith 2ndQuadran

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-11 Thread Stephen Frost
* Merlin Moncure (mmonc...@gmail.com) wrote: > It's understood that posix_fallocate is faster at this -- the question > on the table is 'does this matter in context of postgres?'. > Personally I think this patch should go in regardless -- the concerns > made IMNSHO are specious. I've not had a cha

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-11 Thread Merlin Moncure
On Tue, Jun 11, 2013 at 11:08 AM, Stefan Drees wrote: > On 2013-11.06 17:28, Jon Nelson wrote: >> >> There hasn't been much activity here recently. I'm curious, then, if >> there are questions that I can answer. >> It may be useful to summarize some things here: >> >> - the purpose of the patch i

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-11 Thread Stefan Drees
On 2013-11.06 17:28, Jon Nelson wrote: There hasn't been much activity here recently. I'm curious, then, if there are questions that I can answer. It may be useful to summarize some things here: - the purpose of the patch is to use posix_fallocate when creating new WAL files, because it's (usua

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-11 Thread Jon Nelson
There hasn't been much activity here recently. I'm curious, then, if there are questions that I can answer. It may be useful to summarize some things here: - the purpose of the patch is to use posix_fallocate when creating new WAL files, because it's (usually) much quicker - using posix_fallocate

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-31 Thread Noah Misch
On Thu, May 30, 2013 at 02:58:26PM +0200, Andres Freund wrote: > > * Andres Freund (and...@2ndquadrant.com) wrote: > > > But really, I am not at all concerned about some obscure values being > > > returned, but about a read() not being successful.. > After a bit of standard perusing writing a sing

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Greg Smith
On 5/30/13 11:21 AM, Alvaro Herrera wrote: Greg Smith escribió: The messy part of extending relations in larger chunks is how to communicate that back into the buffer manager usefully. The extension path causing trouble is RelationGetBufferForTuple calling ReadBufferBI. All of that is passing

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Alvaro Herrera
Greg Smith escribió: > The messy part of extending relations in larger chunks > is how to communicate that back into the buffer manager usefully. > The extension path causing trouble is RelationGetBufferForTuple > calling ReadBufferBI. All of that is passing a single buffer > around. There's no

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Robert Haas
On Thu, May 30, 2013 at 8:14 AM, Andres Freund wrote: > I don't think there's much danger of getting uninitialized data or > such. That clearly would be insane. I think somebody might interpret it > as read(2) returning an error until the page has been written to which > isn't completely crazy. I

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Greg Smith
On 5/30/13 8:50 AM, Stephen Frost wrote: I don't think this solves the locking issue around the relation extention lock, but it might help some and it may make it easier to tweak that logic to allocate larger chunks in the future. It might make it a bit faster, but it doesn't make it any easier

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > After a bit of standard perusing writing a single byte to the end of the > file after the fallocate ought to make at least the reading guaranteed > to be defined. If we did seek(last_byte); write(); posix_fallocate() we > should even always have def

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Andres Freund
On 2013-05-30 08:53:37 -0400, Stephen Frost wrote: > * Andres Freund (and...@2ndquadrant.com) wrote: > > But really, I am not at all concerned about some obscure values being > > returned, but about a read() not being successful.. > > Alright, so what do we need to do to test this? We really just

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > But really, I am not at all concerned about some obscure values being > returned, but about a read() not being successful.. Alright, so what do we need to do to test this? We really just need a short C program written up and then a bunch of folks

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > On 2013-05-30 08:19:17 -0400, Peter Eisentraut wrote: > > On 5/30/13 8:02 AM, Robert Haas wrote: > > > If there's some OS out > > > there that chooses to fill the pre-extended pages with 0x55 or cat > > > /dev/urandom instead of 0x00, they probably

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Stephen Frost
* Peter Eisentraut (pete...@gmx.net) wrote: > On 5/30/13 7:13 AM, Andres Freund wrote: > > Why? The spec doesn't specify that case and that very well allows other > > behaviour. Glibc sure does behave sensibly and zeroes the data > > (sysdeps/posix/posix_fallocate64.c for the generic implementation

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Andres Freund
On 2013-05-30 08:17:28 -0400, Peter Eisentraut wrote: > On 5/30/13 7:13 AM, Andres Freund wrote: > > Why? The spec doesn't specify that case and that very well allows other > > behaviour. Glibc sure does behave sensibly and zeroes the data > > (sysdeps/posix/posix_fallocate64.c for the generic impl

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Andres Freund
On 2013-05-30 08:19:17 -0400, Peter Eisentraut wrote: > On 5/30/13 8:02 AM, Robert Haas wrote: > > If there's some OS out > > there that chooses to fill the pre-extended pages with 0x55 or cat > > /dev/urandom instead of 0x00, they probably deserve what they get. > > Even that wouldn't be a proble

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Peter Eisentraut
On 5/30/13 8:02 AM, Robert Haas wrote: > If there's some OS out > there that chooses to fill the pre-extended pages with 0x55 or cat > /dev/urandom instead of 0x00, they probably deserve what they get. Even that wouldn't be a problem for our purpose. The only problem would be if you can't read fr

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Peter Eisentraut
On 5/30/13 7:13 AM, Andres Freund wrote: > Why? The spec doesn't specify that case and that very well allows other > behaviour. Glibc sure does behave sensibly and zeroes the data > (sysdeps/posix/posix_fallocate64.c for the generic implementation) and > so does linux' fallocate() syscall, but that

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Andres Freund
On 2013-05-30 08:02:56 -0400, Robert Haas wrote: > On Thu, May 30, 2013 at 7:13 AM, Andres Freund wrote: > >> Surely this is undue pessimism. > > > > Why? The spec doesn't specify that case and that very well allows other > > behaviour. Glibc sure does behave sensibly and zeroes the data > > (sysd

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Robert Haas
On Thu, May 30, 2013 at 7:13 AM, Andres Freund wrote: >> Surely this is undue pessimism. > > Why? The spec doesn't specify that case and that very well allows other > behaviour. Glibc sure does behave sensibly and zeroes the data > (sysdeps/posix/posix_fallocate64.c for the generic implementation)

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Greg Smith
On 5/30/13 7:52 AM, Andres Freund wrote: fadvise is a hint which is pretty different from a fallocate where ignoring would have way much more severe consequences. Yes, it will. That's why I want to see it tested. There is more than enough past examples of bad behavior here to be skeptical th

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Andres Freund
On 2013-05-30 07:48:51 -0400, Greg Smith wrote: > On 5/30/13 7:17 AM, Andres Freund wrote: > >That argument in contrast I find not very convincing though. What was > >the last incidence of such a system call that did not just error out > >with ENOTSUPP or such? > > http://linux.die.net/man/2/posix

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Greg Smith
On 5/30/13 7:17 AM, Andres Freund wrote: That argument in contrast I find not very convincing though. What was the last incidence of such a system call that did not just error out with ENOTSUPP or such? http://linux.die.net/man/2/posix_fadvise talks about POSIX_FADV_NOREUSE and POSIX_FADV_WIL

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Andres Freund
On 2013-05-30 06:55:16 -0400, Greg Smith wrote: > On 5/30/13 6:49 AM, Robert Haas wrote: > >On Wed, May 29, 2013 at 10:42 AM, Andres Freund > >wrote: > >>So we don't even know whether we can read. I think that means we need to > >>zero the file anyway... > > > >Surely this is undue pessimism. >

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Andres Freund
On 2013-05-30 06:49:42 -0400, Robert Haas wrote: > On Wed, May 29, 2013 at 10:42 AM, Andres Freund > wrote: > > FWIW, posix' description about posix_fallocate() doesn't actually say > > *anything* about reading. The guarantee it makes is: > > "If posix_fallocate() returns successfully, subsequent

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Greg Smith
On 5/30/13 6:49 AM, Robert Haas wrote: On Wed, May 29, 2013 at 10:42 AM, Andres Freund wrote: So we don't even know whether we can read. I think that means we need to zero the file anyway... Surely this is undue pessimism. There have been many occasions where I've found the Linux kernel de

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-30 Thread Robert Haas
On Wed, May 29, 2013 at 10:42 AM, Andres Freund wrote: > FWIW, posix' description about posix_fallocate() doesn't actually say > *anything* about reading. The guarantee it makes is: > "If posix_fallocate() returns successfully, subsequent writes to the > specified file data shall not fail due to t

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-29 Thread Peter Eisentraut
On 5/29/13 10:42 AM, Andres Freund wrote: > On 2013-05-29 10:36:07 -0400, Stephen Frost wrote: >> I *really* hope that the Linux kernel, and other, folks are smart enough >> to realize that they can't just re-use random blocks from an I/O device >> without cleaning it first. > > FWIW, posix' descr

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-29 Thread Andres Freund
On 2013-05-29 10:36:07 -0400, Stephen Frost wrote: > * Peter Eisentraut (pete...@gmx.net) wrote: > > On 5/28/13 11:36 AM, Greg Smith wrote: > > > Outside of the run for performance testing, I think it would be good at > > > this point to validate that there is really a 16MB file full of zeroes > >

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-29 Thread Stephen Frost
* Peter Eisentraut (pete...@gmx.net) wrote: > On 5/28/13 11:36 AM, Greg Smith wrote: > > Outside of the run for performance testing, I think it would be good at > > this point to validate that there is really a 16MB file full of zeroes > > resulting from these operations. I am not really concerned

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-29 Thread Peter Eisentraut
On 5/28/13 11:36 AM, Greg Smith wrote: > Outside of the run for performance testing, I think it would be good at > this point to validate that there is really a 16MB file full of zeroes > resulting from these operations. I am not really concerned that > posix_fallocate might be slower in some case

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-28 Thread Greg Smith
On 5/28/13 10:00 PM, Jon Nelson wrote: On Tue, May 28, 2013 at 10:36 AM, Greg Smith wrote: On 5/28/13 11:12 AM, Jon Nelson wrote: It opens a new file, fallocates 16MB, calls fdatasync. Outside of the run for performance testing, I think it would be good at this point to validate that there

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-28 Thread Jon Nelson
On Tue, May 28, 2013 at 10:36 AM, Greg Smith wrote: > On 5/28/13 11:12 AM, Jon Nelson wrote: >> >> It opens a new file, fallocates 16MB, calls fdatasync. > > > Outside of the run for performance testing, I think it would be good at this > point to validate that there is really a 16MB file full of

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-28 Thread Greg Smith
On 5/28/13 11:12 AM, Jon Nelson wrote: It opens a new file, fallocates 16MB, calls fdatasync. Outside of the run for performance testing, I think it would be good at this point to validate that there is really a 16MB file full of zeroes resulting from these operations. I am not really concer

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-28 Thread Andres Freund
On 2013-05-28 10:12:05 -0500, Jon Nelson wrote: > On Tue, May 28, 2013 at 9:19 AM, Robert Haas wrote: > > On Tue, May 28, 2013 at 10:15 AM, Andres Freund > > wrote: > >> On 2013-05-28 10:03:58 -0400, Robert Haas wrote: > >>> On Sat, May 25, 2013 at 2:55 PM, Jon Nelson > >>> wrote: > >>> >> The

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-28 Thread Jon Nelson
On Tue, May 28, 2013 at 9:19 AM, Robert Haas wrote: > On Tue, May 28, 2013 at 10:15 AM, Andres Freund > wrote: >> On 2013-05-28 10:03:58 -0400, Robert Haas wrote: >>> On Sat, May 25, 2013 at 2:55 PM, Jon Nelson >>> wrote: >>> >> The biggest thing missing from this submission is information abo

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-28 Thread Robert Haas
On Tue, May 28, 2013 at 10:15 AM, Andres Freund wrote: > On 2013-05-28 10:03:58 -0400, Robert Haas wrote: >> On Sat, May 25, 2013 at 2:55 PM, Jon Nelson >> wrote: >> >> The biggest thing missing from this submission is information about what >> >> performance testing you did. Ideally performanc

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-28 Thread Andres Freund
On 2013-05-28 10:03:58 -0400, Robert Haas wrote: > On Sat, May 25, 2013 at 2:55 PM, Jon Nelson wrote: > >> The biggest thing missing from this submission is information about what > >> performance testing you did. Ideally performance patches are submitted > >> with > >> enough information for a

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-28 Thread Robert Haas
On Sat, May 25, 2013 at 2:55 PM, Jon Nelson wrote: >> The biggest thing missing from this submission is information about what >> performance testing you did. Ideally performance patches are submitted with >> enough information for a reviewer to duplicate the same test the author did, >> as well

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-25 Thread Jon Nelson
On Thu, May 16, 2013 at 7:05 PM, Greg Smith wrote: > On 5/16/13 9:16 AM, Jon Nelson wrote: >> >> Am I doing this the right way? Should I be posting the full patch each >> time, or incremental patches? > > > There are guidelines for getting your patch in the right format at > https://wiki.postgresq

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-17 Thread Merlin Moncure
On Fri, May 17, 2013 at 4:18 PM, Andres Freund wrote: > On 2013-05-17 15:48:38 -0500, Merlin Moncure wrote: >> On Fri, May 17, 2013 at 8:29 AM, Merlin Moncure wrote: >> > On Fri, May 17, 2013 at 4:47 AM, Andres Freund >> > wrote: >> >> On 2013-05-15 16:46:33 -0500, Jon Nelson wrote: >> >>> > *

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-17 Thread Andres Freund
On 2013-05-17 15:48:38 -0500, Merlin Moncure wrote: > On Fri, May 17, 2013 at 8:29 AM, Merlin Moncure wrote: > > On Fri, May 17, 2013 at 4:47 AM, Andres Freund > > wrote: > >> On 2013-05-15 16:46:33 -0500, Jon Nelson wrote: > >>> > * Is wal file creation performance actually relevant? Is the per

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-17 Thread Merlin Moncure
On Fri, May 17, 2013 at 8:29 AM, Merlin Moncure wrote: > On Fri, May 17, 2013 at 4:47 AM, Andres Freund wrote: >> On 2013-05-15 16:46:33 -0500, Jon Nelson wrote: >>> > * Is wal file creation performance actually relevant? Is the performance >>> > of a system running on fallocate()d wal files an

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-17 Thread Merlin Moncure
On Fri, May 17, 2013 at 4:47 AM, Andres Freund wrote: > On 2013-05-15 16:46:33 -0500, Jon Nelson wrote: >> > * Is wal file creation performance actually relevant? Is the performance >> > of a system running on fallocate()d wal files any different? >> >> In my limited testing, I noticed a drop of

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-17 Thread Andres Freund
On 2013-05-15 16:46:33 -0500, Jon Nelson wrote: > > * Is wal file creation performance actually relevant? Is the performance > > of a system running on fallocate()d wal files any different? > > In my limited testing, I noticed a drop of approx. 100ms per WAL file. > I do not have a good idea for

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-16 Thread Greg Smith
On 5/16/13 9:16 AM, Jon Nelson wrote: Am I doing this the right way? Should I be posting the full patch each time, or incremental patches? There are guidelines for getting your patch in the right format at https://wiki.postgresql.org/wiki/Working_with_Git#Context_diffs_with_Git that would imp

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-16 Thread Alvaro Herrera
Jon Nelson escribió: > Am I doing this the right way? Should I be posting the full patch each > time, or incremental patches? Full patch each time is okay. Context-format patch is even better. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Tr

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-16 Thread Jon Nelson
On Wed, May 15, 2013 at 10:36 PM, Jon Nelson wrote: > On Wed, May 15, 2013 at 10:17 PM, Alvaro Herrera > wrote: >> Jon Nelson escribió: >>> On Wed, May 15, 2013 at 4:46 PM, Jon Nelson >>> wrote: >> >>> > That's true. I originally wrote the patch using fallocate(2). What >>> > would be appropria

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-15 Thread Jon Nelson
On Wed, May 15, 2013 at 10:17 PM, Alvaro Herrera wrote: > Jon Nelson escribió: >> On Wed, May 15, 2013 at 4:46 PM, Jon Nelson >> wrote: > >> > That's true. I originally wrote the patch using fallocate(2). What >> > would be appropriate here? Should I switch on the return value and the >> > six (

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-15 Thread Alvaro Herrera
Jon Nelson escribió: > On Wed, May 15, 2013 at 4:46 PM, Jon Nelson wrote: > > That's true. I originally wrote the patch using fallocate(2). What > > would be appropriate here? Should I switch on the return value and the > > six (6) or so relevant error codes? > > I addressed this, hopefully in a

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-15 Thread Jon Nelson
On Wed, May 15, 2013 at 4:46 PM, Jon Nelson wrote: > On Wed, May 15, 2013 at 4:34 PM, Andres Freund wrote: .. >> Some where quick comments, without thinking about this: > > Thank you for the kind feedback. > >> * needs a configure check for posix_fallocate. The current version will >> e.g. fail

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-15 Thread Jon Nelson
On Wed, May 15, 2013 at 4:34 PM, Andres Freund wrote: > Hi, > > On 2013-05-15 16:26:15 -0500, Jon Nelson wrote: >> >> I have written up a patch to use posix_fallocate in new WAL file >> >> creation, including configuration by way of a GUC variable, but I've >> >> not contributed to the PostgreSQL

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-15 Thread Andres Freund
Hi, On 2013-05-15 16:26:15 -0500, Jon Nelson wrote: > >> I have written up a patch to use posix_fallocate in new WAL file > >> creation, including configuration by way of a GUC variable, but I've > >> not contributed to the PostgreSQL project before. Therefore, I'm > >> fairly certain the patch is

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-15 Thread Jon Nelson
On Tue, May 14, 2013 at 9:43 PM, Robert Haas wrote: > On Mon, May 13, 2013 at 9:54 PM, Jon Nelson wrote: >> Pertinent to another thread titled >> [HACKERS] corrupt pages detected by enabling checksums >> I hope to explore the possibility of using fallocate (or >> posix_fallocate) for new WAL file

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-14 Thread Robert Haas
On Mon, May 13, 2013 at 9:54 PM, Jon Nelson wrote: > Pertinent to another thread titled > [HACKERS] corrupt pages detected by enabling checksums > I hope to explore the possibility of using fallocate (or > posix_fallocate) for new WAL file creation. > > Most modern Linux filesystems support fast f

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-13 Thread David Fetter
On Mon, May 13, 2013 at 08:54:39PM -0500, Jon Nelson wrote: > Pertinent to another thread titled > [HACKERS] corrupt pages detected by enabling checksums > I hope to explore the possibility of using fallocate (or > posix_fallocate) for new WAL file creation. > > Most modern Linux filesystems suppo

[HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-13 Thread Jon Nelson
Pertinent to another thread titled [HACKERS] corrupt pages detected by enabling checksums I hope to explore the possibility of using fallocate (or posix_fallocate) for new WAL file creation. Most modern Linux filesystems support fast fallocate/posix_fallocate, reducing extent fragmentation (where