Qingqing Zhou wrote:
[also with reply to Gavin] look up dictionary for "gut-rot", got it ... Uh,
this behavior is intended - I try to push enough requests shortly to kernel
so that it understands that I am doing sequential scan, so it would pull the
data from disk to file system cache more eff
Yes. The O_DIRECT issue is my biggest concern about Linux at the moment.
That being said, the plan is to only pre-fetch the next N blocks, where N
< 32, and to read them into the local buffer cache. In a situation where
space in the cache low (and prefetched pages might be pushed out before we
e
lto:[EMAIL PROTECTED]
Sent: Tuesday, November 29, 2005 5:23 PM
To: Pollard, Mike
Cc: Qingqing Zhou; pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] ice-broker scan thread
On Tue, 2005-11-29 at 09:45 -0500, Pollard, Mike wrote:
> I've implemented this on another database product
You
Jeff,
On 11/29/05 9:35 AM, "Jeffrey W. Baker" <[EMAIL PROTECTED]> wrote:
> On Tue, 2005-11-29 at 09:45 -0500, Pollard, Mike wrote:
>
>> Anyway, what I did was the following. When doing a sequential scan, we
>> were starting at the beginning of the table and scanning forward. If I
>> threw up
On Tue, 2005-11-29 at 09:45 -0500, Pollard, Mike wrote:
> I've implemented this on another database product
You're scaring me. Is the information you describe in the public domain
or is it intellectual property of any particular company? Are you sure?
We just recovered from one patent scare.
Goo
On Wed, 2005-11-30 at 08:30 +1100, Gavin Sherry wrote:
> On Tue, 29 Nov 2005, David Boreham wrote:
>
> >
> > >By default when you use aio you get the version in libc (-lrt IIRC)
> > >which has the issue I mentioned, probably because it's probably
> > >optimised for the lots-of-network-connections
On Tue, 29 Nov 2005, David Boreham wrote:
>
> >By default when you use aio you get the version in libc (-lrt IIRC)
> >which has the issue I mentioned, probably because it's probably
> >optimised for the lots-of-network-connections type program where
> >multiple outstanding requests on a single fd
On Tue, 29 Nov 2005, Andrew Piskorski wrote:
> On Tue, Nov 29, 2005 at 03:14:38PM +1100, Gavin Sherry wrote:
> > On Mon, 28 Nov 2005, David Boreham wrote:
> > > Gavin Sherry wrote:
> > > > MySQL, Oracle and others implement read-ahead threads to simulate async
> > > > IO
> > >
> > > I always beli
"Qingqing Zhou" <[EMAIL PROTECTED]> wrote
>
> I wrote a program to simulate the sequential scan in PostgreSQL
> with/without ice-broker.
>
> We need more tests
>
If anybody has a test results then I'd love to see it ...
Thanks,
Qingqing
---(end of broadcast)---
On Tue, Nov 29, 2005 at 10:28:57AM -0700, David Boreham wrote:
> Actually, after reading up on the current state of things, I'm not sure you
> can even get POSIX aio on top of kernel aio in Linux. There are also a
> few limitations in the 2.6 aio implementation that might prove troublesome:
> for e
On Tue, 2005-11-29 at 09:45 -0500, Pollard, Mike wrote:
> Anyway, what I did was the following. When doing a sequential scan, we
> were starting at the beginning of the table and scanning forward. If I
> threw up some threads to read ahead, then my user thread and my read
> ahead threads would t
By default when you use aio you get the version in libc (-lrt IIRC)
which has the issue I mentioned, probably because it's probably
optimised for the lots-of-network-connections type program where
multiple outstanding requests on a single fd are not meaningful. You
can however link in some other
On Tue, Nov 29, 2005 at 08:42:18AM -0700, David Boreham wrote:
>
> >Unfortunatly I can't really test it at it's full potential because it
> >uses glibc's default POSIX AIO which is *lame*. No more than one
> >outstanding request per fd which for PostgreSQL is crappy. There was
> >
> I had the imp
On Tue, Nov 29, 2005 at 03:14:38PM +1100, Gavin Sherry wrote:
> On Mon, 28 Nov 2005, David Boreham wrote:
> > Gavin Sherry wrote:
> > > MySQL, Oracle and others implement read-ahead threads to simulate async IO
> >
> > I always believed that Oracle used async file I/O. Not that I've seen their
> T
Unfortunatly I can't really test it at it's full potential because it
uses glibc's default POSIX AIO which is *lame*. No more than one
outstanding request per fd which for PostgreSQL is crappy. There was
I had the impression from the kernel aio mailing list a while back that
post- linux, the
lto:[EMAIL PROTECTED]
Sent: Tuesday, November 29, 2005 10:06 AM
To: Pollard, Mike
Cc: Qingqing Zhou; pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] ice-broker scan thread
On Tue, Nov 29, 2005 at 09:45:30AM -0500, Pollard, Mike wrote:
> Anyway, what I did was the following. When doing a se
On Tue, Nov 29, 2005 at 09:45:30AM -0500, Pollard, Mike wrote:
> Anyway, what I did was the following. When doing a sequential scan, we
> were starting at the beginning of the table and scanning forward. If I
> threw up some threads to read ahead, then my user thread and my read
> ahead threads w
threw up some threads to read ahead, then my user thread and my read
ahead threads would thrash on trying to lock the buffer slots. So, I
had the read ahead threads start at some distance into the table, and
work toward the beginning. The user thread would do its own I/O until
Ah. The ligh
alf Of Qingqing Zhou
Sent: Tuesday, November 29, 2005 12:56 AM
To: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] ice-broker scan thread
"David Boreham" <[EMAIL PROTECTED]> wrote
>>
> I don't think your NT overlapped I/O code is quite right. At least
> I think
On Tue, Nov 29, 2005 at 02:53:36PM +1100, Gavin Sherry wrote:
> The second idea is using posix async IO at key points within the system
> to better parallelise CPU and IO work. There areas I think we could use
> async IO are: during sequential scans, use async IO to do pre-fetching of
> blocks; ins
"David Boreham" <[EMAIL PROTECTED]> wrote
>>
> I don't think your NT overlapped I/O code is quite right. At least
> I think it will issue reads at a high rate without waiting for any of them
> to complete. Beyond some point that has to give the kernel gut-rot.
>
[also with reply to Gavin] look up
On Mon, 28 Nov 2005, Qingqing Zhou wrote:
>
>
> On Mon, 28 Nov 2005, Gavin Sherry wrote:
> >
> > MySQL, Oracle and others implement read-ahead threads to simulate async IO
> > 'pre-fetching'.
>
> Due to my tests on Windows (using the attached program and change
> enable_aio=true), seems aio doesn'
Qingqing Zhou wrote:
On Mon, 28 Nov 2005, Gavin Sherry wrote:
MySQL, Oracle and others implement read-ahead threads to simulate async IO
'pre-fetching'.
Due to my tests on Windows (using the attached program and change
enable_aio=true), seems aio doesn't help as a separ
"David Boreham" <[EMAIL PROTECTED]> wrote
>
> BTW, I heard a long time ago that NTFS has quite fancy read-ahead, where
> it attempts to detect the application's access pattern including if it is
> reading sequentially and even if there is a 'stride' to the accesses when
> they're not contiguous.
On Mon, 28 Nov 2005, Gavin Sherry wrote:
>
> I didn't want to jump on list and waive my hands until I had something to
> show, but since Qingqing is looking at the issue I thought I better raise
> it.
>
Don't worry :-) I separate the logic into a standalone program in order to
let more people ca
On Mon, 28 Nov 2005, Gavin Sherry wrote:
>
> MySQL, Oracle and others implement read-ahead threads to simulate async IO
> 'pre-fetching'.
Due to my tests on Windows (using the attached program and change
enable_aio=true), seems aio doesn't help as a separate thread - but maybe
because my usage i
FYI, I've personally used Oracle 9.2.0.4's async IO on Linux and have seen several installations which make use of it also.
On 11/28/05, Gavin Sherry <[EMAIL PROTECTED]> wrote:
On Mon, 28 Nov 2005, Tom Lane wrote:> Gavin Sherry <[EMAIL PROTECTED]
> writes:> > I haven't had time to prototype wheth
On Mon, 28 Nov 2005, Mark Kirkwood wrote:
>
> Do these ideas require threads in principle? ISTM that there could be
> (additional) process(es) waiting to perform pre-fetching or async io,
> and we could use the usual IPC machinary to talk between them...
>
Right. I use threads because it is easy
Gavin Sherry wrote:
The paper I linked to seemed to suggest that they weren't using async IO
in 9.2 -- which is fairly old. I'm not sure why the authors didn't test
10g.
There have been async io type parameters in Oracle's init.ora files from
(at least) 8i (disk_async_io=true IIRC) - on Sol
On Mon, 28 Nov 2005, Tom Lane wrote:
> Gavin Sherry <[EMAIL PROTECTED]> writes:
> > I haven't had time to prototype whether we can easily implement async IO
>
> Just as with any suggestion to depend on threads, you are going to have
> to show results that border on astounding to have any chance of
The paper I linked to seemed to suggest that they weren't using async IO
in 9.2 -- which is fairly old. I'm not sure why the authors didn't test
10g.
.. ok, interesting. Did they say that Oracle
isn't using aio ?
I can't see that. They that Oracle has no more than one outstanding I
Tom Lane wrote:
Gavin Sherry <[EMAIL PROTECTED]> writes:
I haven't had time to prototype whether we can easily implement async IO
Just as with any suggestion to depend on threads, you are going to have
to show results that border on astounding to have any chance of getting
this in. Otherwis
On Mon, 28 Nov 2005, David Boreham wrote:
> Gavin Sherry wrote:
>
> > MySQL, Oracle and others implement read-ahead threads to simulate async IO
>
> I always believed that Oracle used async file I/O. Not that I've seen their
> code, but I'm fairly sure they funded the addition of kernel aio to Lin
Gavin Sherry wrote:
MySQL, Oracle and others implement read-ahead threads to simulate async IO
I always believed that Oracle used async file I/O. Not that I've seen their
code, but I'm fairly sure they funded the addition of kernel aio to Linux
a few years back.
ButOracle comes from a tim
Gavin Sherry <[EMAIL PROTECTED]> writes:
> I haven't had time to prototype whether we can easily implement async IO
Just as with any suggestion to depend on threads, you are going to have
to show results that border on astounding to have any chance of getting
this in. Otherwise the portability is
Qingqing,
I am considering add an "ice-broker scan thread" to accelerate PostgreSQL
sequential scan IO speed. The basic idea of this thread is just like the
"read-ahead" method, but the difference is this one does not read the
data
into shared buffer pool directly, instead, it reads the data i
On Mon, 28 Nov 2005, Qingqing Zhou wrote:
>
> I am considering add an "ice-broker scan thread" to accelerate PostgreSQL
> sequential scan IO speed. The basic idea of this thread is just like the
> "read-ahead" method, but the difference is this one does not read the data
> into shared buffer pool
Qingqing Zhou wrote:
I am considering add an "ice-broker scan thread" to accelerate PostgreSQL
sequential scan IO speed. The basic idea of this thread is just like the
"read-ahead" method, but the difference is this one does not read the data
into shared buffer pool directly, instead, it reads t
38 matches
Mail list logo
