Jim,
On 9/22/06 7:01 AM, "Jim C. Nasby" <[EMAIL PROTECTED]> wrote:
> There's been talk of adding code that would have a seqscan detect if
> another seqscan is happening on the table at the same time, and if it
> is, to start it's seqscan wherever the other seqscan is currently
> running. That wou
Luke Lonergan wrote:
>
> I think the topic is similar to "cache bypass", used in cache capable vector
> processors (Cray, Convex, Multiflow, etc) in the 90's. When you are
> scanning through something larger than the cache, it should be marked
> "non-cacheable" and bypass caching altogether. Thi
On Thu, Sep 21, 2006 at 11:05:39PM -0400, Bruce Momjian wrote:
> We tried posix_fadvise() during the 8.2 development cycle, but had
> problems as outlined in a comment in xlog.c:
>
> /*
> * posix_fadvise is problematic on many platforms: on older x86 Linux
> * it just dumps core, and
On Thu, Sep 21, 2006 at 08:46:41PM -0700, Luke Lonergan wrote:
> Mark,
>
> On 9/21/06 8:40 PM, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote:
>
> > I'd advise against using this call unless it can be shown that the page
> > will not be used in the future, or at least, that the page is less useful
Hi, Guy,
Guy Thornley wrote:
> Of course you could argue the OS should be able to detect this, and prevent
> it occuring anyway. I don't know anything about linux's behaviour in this
> area.
Yes, one can argue that way.
But a generic Algorithm in the OS can never be as smart as the
application
Mark,
On 9/21/06 8:40 PM, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote:
> I'd advise against using this call unless it can be shown that the page
> will not be used in the future, or at least, that the page is less useful
> than all other pages currently in memory. This is what the call really m
On Fri, Sep 22, 2006 at 02:52:09PM +1200, Guy Thornley wrote:
> > >> I thought that posix_fadvise() with POSIX_FADV_WILLNEED was exactly
> > >> meant for this purpose?
> > > This is a good idea - I wasn't aware that this was possible.
> > This possibility was the reason for me to propose it. :-)
>
Guy Thornley wrote:
> > >> I thought that posix_fadvise() with POSIX_FADV_WILLNEED was exactly
> > >> meant for this purpose?
> > >
> > > This is a good idea - I wasn't aware that this was possible.
> >
> > This possibility was the reason for me to propose it. :-)
>
> posix_fadvise() features in
> >> I thought that posix_fadvise() with POSIX_FADV_WILLNEED was exactly
> >> meant for this purpose?
> >
> > This is a good idea - I wasn't aware that this was possible.
>
> This possibility was the reason for me to propose it. :-)
posix_fadvise() features in the TODO list already; I'm not sure
Bucky,
On 9/21/06 2:16 PM, "Bucky Jordan" <[EMAIL PROTECTED]> wrote:
> Does this have anything to do with postgres indexes not storing data, as
> some previous posts to this list have mentioned? (In otherwords, having
> the index in memory doesn't help? Or are we talking about indexes that
> are
Hi, Bucky,
Bucky Jordan wrote:
> Each postgres process also uses shared memory (aka the buffer cache) so
> as to not fetch data that another process has already requested,
> correct?
Yes.
Additinally, the OS caches disk blocks. Most unixoid ones like Linux use
(nearly) all unused memory for thi
Markus,
First, thanks- your email was very enlightining. But, it does bring up a
few additional questions, so thanks for your patience also- I've listed
them below.
> It applies per active backend. When connecting, the Postmaster forks a
> new backend process. Each backend process has its own sca
Hi, Bucky,
Bucky Jordan wrote:
>> We can implement multiple scanners (already present in MPP), or we
> could
>> implement AIO and fire off a number of simultaneous I/O requests for
>> fulfillment.
>
> So this might be a dumb question, but the above statements apply to the
> cluster (e.g. postmas
> So this might be a dumb question, but the above statements apply to the
> cluster (e.g. postmaster) as a whole, not per postgres
> process/transaction correct? So each transaction is blocked waiting for
> the main postmaster to retrieve the data in the order it was requested
> (i.e. not multiple
> > Do you think that adding some posix_fadvise() calls to the backend
to
> > pre-fetch some blocks into the OS cache asynchroneously could
improve
> > that situation?
>
> Nope - this requires true multi-threading of the I/O, there need to be
> multiple seek operations running simultaneously. The
Hi, Luke,
Luke Lonergan wrote:
>> I thought that posix_fadvise() with POSIX_FADV_WILLNEED was exactly
>> meant for this purpose?
>
> This is a good idea - I wasn't aware that this was possible.
This possibility was the reason for me to propose it. :-)
> We'll do some testing and see if it work
Markus,
On 9/20/06 11:02 AM, "Markus Schaber" <[EMAIL PROTECTED]> wrote:
> I thought that posix_fadvise() with POSIX_FADV_WILLNEED was exactly
> meant for this purpose?
This is a good idea - I wasn't aware that this was possible.
We'll do some testing and see if it works as advertised on Linux
IMHO, AIO is the architecturally cleaner and more elegant solution.
We in fact have a project on the boards to do this but funding (as
yet) has not been found.
My $.02,
Ron
At 02:02 PM 9/20/2006, Markus Schaber wrote:
Hi, Luke,
Luke Lonergan wrote:
>> Do you think that adding some posix_f
Hi, Luke,
Luke Lonergan wrote:
>> Do you think that adding some posix_fadvise() calls to the backend to
>> pre-fetch some blocks into the OS cache asynchroneously could improve
>> that situation?
>
> Nope - this requires true multi-threading of the I/O, there need to be
> multiple seek operation
Markus,
On 9/20/06 1:09 AM, "Markus Schaber" <[EMAIL PROTECTED]> wrote:
> Do you think that adding some posix_fadvise() calls to the backend to
> pre-fetch some blocks into the OS cache asynchroneously could improve
> that situation?
Nope - this requires true multi-threading of the I/O, there ne
Hi, Luke,
Luke Lonergan wrote:
> Since PG's heap scan is single threaded, the seek rate is equivalent to a
> single disk (even though RAID arrays may have many spindles), the typical
> random seek rates are around 100-200 seeks per second from within the
> backend. That means that as sequential
Mike,
> On Mon, Sep 18, 2006 at 07:14:56PM -0400, Alex Turner wrote:
> >If you have a table with 100million records, each of which is
200bytes
> long,
> >that gives you roughtly 20 gig of data (assuming it was all written
> neatly
> >and hasn't been updated much).
>
I'll keep that in mind (minimi
Mark,
On 9/18/06 8:45 PM, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote:
> Does a tool exist yet to time this for a particular configuration?
We're considering building this into ANALYZE on a per-table basis. The
basic approach times sequential access in page rate, then random seeks as
page rat
On Mon, Sep 18, 2006 at 06:10:13PM -0700, Luke Lonergan wrote:
> Also be sure to set the random_page_cost parameter in
> postgresql.conf to 100 or even higher when you use indexes, as the actual
> seek rate for random access ranges between 50 and 300 for modern disk
> hardware. If this parameter i
Yep, Solaris ZFS kicks butt. It does RAID10/5/6, etc and implements most of
the high end features available on high end SANs...
- Luke
On 9/18/06 8:40 PM, "Alex Turner" <[EMAIL PROTECTED]> wrote:
> Sweet - thats good - RAID 10 support seems like an odd thing to leave out.
>
> Alex
>
> On 9/1
Sweet - thats good - RAID 10 support seems like an odd thing to leave out.AlexOn 9/18/06, Luke Lonergan <
[EMAIL PROTECTED]> wrote:Alex,On 9/18/06 4:14 PM, "Alex Turner" <
[EMAIL PROTECTED]> wrote:> Be warned, the tech specs page:> http://www.sun.com/servers/x64/x4500/specs.xml#anchor3
> doesn't me
Alex,
On 9/18/06 4:14 PM, "Alex Turner" <[EMAIL PROTECTED]> wrote:
> Be warned, the tech specs page:
> http://www.sun.com/servers/x64/x4500/specs.xml#anchor3
> doesn't mention RAID 10 as a possible, and this is probably what most would
> recommend for fast data access if you are doing both read a
Bucky,
On 9/18/06 7:37 AM, "Bucky Jordan" <[EMAIL PROTECTED]> wrote:
> My question is at what point do I have to get fancy with those big
> tables? From your presentation, it looks like PG can handle 1.2 billion
> records or so as long as you write intelligent queries. (And normal PG
> should be
On Mon, Sep 18, 2006 at 07:14:56PM -0400, Alex Turner wrote:
If you have a table with 100million records, each of which is 200bytes long,
that gives you roughtly 20 gig of data (assuming it was all written neatly
and hasn't been updated much).
If you're in that range it doesn't even count as
Do the basic math:If you have a table with 100million records, each of which is 200bytes long, that gives you roughtly 20 gig of data (assuming it was all written neatly and hasn't been updated much). If you have to do a full table scan, then it will take roughly 400 seconds with a single 10k RPM
> good normalization skills are really important for large databases,
> along with materialization strategies for 'denormalized sets'.
Good points- thanks. I'm especially curious what others have done for
the materialization. The matview project on gborg appears dead, and I've
only found a smatter
On Monday 18 September 2006 13:56, "Merlin Moncure" <[EMAIL PROTECTED]>
wrote:
> just another fyi, if you have a really big database, you can forget
> about doing pg_dump for backups (unless you really don't care about
> being x day or days behind)...you simply have to due some type of
> replicati
On 9/18/06, Bucky Jordan <[EMAIL PROTECTED]> wrote:
My question is at what point do I have to get fancy with those big
tables? From your presentation, it looks like PG can handle 1.2 billion
records or so as long as you write intelligent queries. (And normal PG
should be able to handle that, corr
>Yes. What's pretty large? We've had to redefine large recently, now
we're
>talking about systems with between 100TB and 1,000TB.
>
>- Luke
Well, I said large, not gargantuan :) - Largest would probably be around
a few TB, but the problem I'm having to deal with at the moment is large
numbers (p
34 matches
Mail list logo
