On Tue, Jan 11, 2011 at 11:39:20AM -0500, Cédric Villemain wrote: > 2011/1/11 Garick Hamlin <gham...@isc.upenn.edu>: > > On Mon, Jan 10, 2011 at 09:09:28AM -0500, Magnus Hagander wrote: > >> On Sun, Jan 9, 2011 at 23:33, Cédric Villemain > >> <cedric.villemain.deb...@gmail.com> wrote: > >> > 2011/1/7 Magnus Hagander <mag...@hagander.net>: > >> >> On Fri, Jan 7, 2011 at 01:47, Cédric Villemain > >> >> <cedric.villemain.deb...@gmail.com> wrote: > >> >>> 2011/1/5 Magnus Hagander <mag...@hagander.net>: > >> >>>> On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine > >> >>>> <dimi...@2ndquadrant.fr> wrote: > >> >>>>> Magnus Hagander <mag...@hagander.net> writes: > >> >>>>>> * Stefan mentiond it might be useful to put some > >> >>>>>> posix_fadvise(POSIX_FADV_DONTNEED) > >> >>>>>> in the process that streams all the files out. Seems useful, as > >> >>>>>> long as that > >> >>>>>> doesn't kick them out of the cache *completely*, for other > >> >>>>>> backends as well. > >> >>>>>> Do we know if that is the case? > >> >>>>> > >> >>>>> Maybe have a look at pgfincore to only tag DONTNEED for blocks that > >> >>>>> are > >> >>>>> not already in SHM? > >> >>>> > >> >>>> I think that's way more complex than we want to go here. > >> >>>> > >> >>> > >> >>> DONTNEED will remove the block from OS buffer everytime. > >> >> > >> >> Then we definitely don't want to use it - because some other backend > >> >> might well want the file. Better leave it up to the standard logic in > >> >> the kernel. > >> > > >> > Looking at the patch, it is (very) easy to add the support for that in > >> > basebackup.c > >> > That supposed allowing mincore(), so mmap(), and so probably switch > >> > the fopen() to an open() (or add an open() just for mmap > >> > requirement...) > >> > > >> > Let's go ? > >> > >> Per above, I still don't think we *should* do this. We don't want to > >> kick things out of the cache underneath other backends, and since we > >> can't control that. Either way, it shouldn't happen in the beginning, > >> and if it does, should be backed with proper benchmarks. > > > > Another option that occurs to me is an option to use direct IO (or another > > means as needed) to bypass the cache. So rather than kicking it out of > > the cache, we attempt just not to pollute the cache by bypassing it for cold > > pages and use either normal io for 'hot pages', or use a 'read()' to "heat" > > the cache afterward. > > AFAIR, even Linus is rejecting the idea to use it seriously, except if > I shuffle in my memory.
Direct IO is generally a pain. POSIX_FADV_NOREUSE is an alternative (I think). Realistically I wasn't sure which way(s) actually worked. My gut was that direct io would likely work right on Linux and Solaris, at least. If POSIX_FADV_NOREUSE works than maybe that is the answer instead, but I haven't tested either. Garick > > > > > Garick > > > >> > >> I've committed the backend side of this, without that. Still working > >> on the client, and on cleaning up Heikki's patch for grammar/parser > >> support. > >> > >> -- > >> Magnus Hagander > >> Me: http://www.hagander.net/ > >> Work: http://www.redpill-linpro.com/ > >> > >> -- > >> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > >> To make changes to your subscription: > >> http://www.postgresql.org/mailpref/pgsql-hackers > > > > > > -- > Cédric Villemain 2ndQuadrant > http://2ndQuadrant.fr/ PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
