Re: [HACKERS] Improve lseek scalability v3

2011-09-21 Thread Marco Stornelli
2011/9/19 Matthew Wilcox : > On Mon, Sep 19, 2011 at 08:31:00AM -0400, Stephen Frost wrote: >> * Benjamin LaHaise (b...@kvack.org) wrote: >> > For such tables, can't Postgres track the size of the file internally?  I'm >> > assuming it's keeping file descriptors open on the tables it manages, in >>

Re: [HACKERS] Improve lseek scalability v3

2011-09-19 Thread Matthew Wilcox
On Mon, Sep 19, 2011 at 08:31:00AM -0400, Stephen Frost wrote: > * Benjamin LaHaise (b...@kvack.org) wrote: > > For such tables, can't Postgres track the size of the file internally? I'm > > assuming it's keeping file descriptors open on the tables it manages, in > > which case when it writes to

Re: [HACKERS] Improve lseek scalability v3

2011-09-19 Thread Robert Haas
On Mon, Sep 19, 2011 at 8:31 AM, Stephen Frost wrote: > * Benjamin LaHaise (b...@kvack.org) wrote: >> For such tables, can't Postgres track the size of the file internally?  I'm >> assuming it's keeping file descriptors open on the tables it manages, in >> which case when it writes to a file to ex

Re: [HACKERS] Improve lseek scalability v3

2011-09-19 Thread Stephen Frost
* Benjamin LaHaise (b...@kvack.org) wrote: > For such tables, can't Postgres track the size of the file internally? I'm > assuming it's keeping file descriptors open on the tables it manages, in > which case when it writes to a file to extend it, the internally stored size > could be updated.

Re: [HACKERS] Improve lseek scalability v3

2011-09-18 Thread Benjamin LaHaise
On Fri, Sep 16, 2011 at 07:27:33PM +0200, Andres Freund wrote: > many tuples does the table have. Those statistics are only updated every now > and then though. > So it uses those old stats to check how many tuples are normally stored on a > page and then uses that to extrapolate the number of tu

Re: [HACKERS] Improve lseek scalability v3

2011-09-16 Thread Matthew Wilcox
On Fri, Sep 16, 2011 at 04:16:49PM +0200, Andres Freund wrote: > I sent an email containing benchmarks from Robert Haas regarding the Subject. > Looking at lkml.org I can't see it right now, Will recheck when I am at home. > > He replaced lseek(SEEK_END) with fstat() and got speedups up to 8.7 ti

Re: [HACKERS] Improve lseek scalability v3

2011-09-16 Thread Andi Kleen
> One other thing we're interested in is portability. I mean, even if > Linux were to introduce a new hypothetical syscall that was able to > return the file size at a ridiculously low cost, we probably wouldn't > use it because it'd be Linux-specific. So an improvement of lseek() > seems to be t

Re: [HACKERS] Improve lseek scalability v3

2011-09-16 Thread Greg Stark
On Fri, Sep 16, 2011 at 9:08 PM, Benjamin LaHaise wrote: > For such tables, can't Postgres track the size of the file internally?  I'm > assuming it's keeping file descriptors open on the tables it manages, in > which case when it writes to a file to extend it, the internally stored size > could b

Re: [HACKERS] Improve lseek scalability v3

2011-09-16 Thread Andres Freund
On Friday, September 16, 2011 11:02:38 PM Andres Freund wrote: > Also with fstat() instead of lseek() there was no bottleneck anymore, so I > don't think the benefits would warrant that. At least thats what I observed on a 4 x 6 machine without the patch applied (can't reboot it). That shouldn't

Re: [HACKERS] Improve lseek scalability v3

2011-09-16 Thread Andres Freund
On Friday, September 16, 2011 10:08:17 PM Benjamin LaHaise wrote: > On Fri, Sep 16, 2011 at 07:27:33PM +0200, Andres Freund wrote: > > many tuples does the table have. Those statistics are only updated every > > now and then though. > > So it uses those old stats to check how many tuples are normal

Re: [HACKERS] Improve lseek scalability v3

2011-09-16 Thread Alvaro Herrera
Excerpts from Andres Freund's message of vie sep 16 14:27:33 -0300 2011: > Hi, > On Friday 16 Sep 2011 17:36:20 Matthew Wilcox wrote: > > Does the query planner need to know the exact number of bytes in the file, > > or is it after an order-of-magnitude? Or to-the-nearest-gigabyte? > It depends

Re: [HACKERS] Improve lseek scalability v3

2011-09-16 Thread Andres Freund
Hi, On Friday 16 Sep 2011 17:36:20 Matthew Wilcox wrote: > On Fri, Sep 16, 2011 at 04:16:49PM +0200, Andres Freund wrote: > > I sent an email containing benchmarks from Robert Haas regarding the > > Subject. Looking at lkml.org I can't see it right now, Will recheck when > > I am at home. > > > >