Bruce Momjian wrote:
> I have talked to Jan over the phone, and he has convinced me that UDP is
> the proper way to communicate stats to the collector, rather than my
> shared memory idea.
>
> The advantages of his UDP approach is that the collector can sleep on
> the UDP socket rather than having
I have talked to Jan over the phone, and he has convinced me that UDP is
the proper way to communicate stats to the collector, rather than my
shared memory idea.
The advantages of his UDP approach is that the collector can sleep on
the UDP socket rather than having the collector poll the shared m
> > I figured it could just wake up every few seconds and check. It will
> > remember the loop counter and current pointer, and read any new
> > information. I was thinking of a 20k buffer, which could cover about 4k
> > events.
>
> Here I wonder what your EVENT is. With an Oid as identif
Bruce Momjian wrote:
> > Bruce Momjian <[EMAIL PROTECTED]> writes:
> > > Only shared memory gives us near-zero cost for write/read. 99% of
> > > backends will not be using stats, so it has to be cheap.
> >
> > Not with a circular buffer it's not cheap, because you need interlocking
> > on writes.
I have a new statistics collection proposal.
I suggest three shared memory areas:
One per backend to hold the query string and other per-backend stats
One global area to hold accumulated stats for all backends
One global circular buffer to hold per-table/object stats
The
> Bruce Momjian <[EMAIL PROTECTED]> writes:
> > Only shared memory gives us near-zero cost for write/read. 99% of
> > backends will not be using stats, so it has to be cheap.
>
> Not with a circular buffer it's not cheap, because you need interlocking
> on writes. Your claim that you can get aw
Bruce Momjian <[EMAIL PROTECTED]> writes:
> Only shared memory gives us near-zero cost for write/read. 99% of
> backends will not be using stats, so it has to be cheap.
Not with a circular buffer it's not cheap, because you need interlocking
on writes. Your claim that you can get away without t
Jan Wieck <[EMAIL PROTECTED]> writes:
> Just to get some evidence at hand - could some owners of
> different platforms compile and run the attached little C
> source please?
> (The program tests how much data can be stuffed into a pipe
> or a Sys-V message queue befo
On Fri, Mar 16, 2001 at 05:25:24PM -0500, Jan Wieck wrote:
> Jan Wieck wrote:
...
> Just to get some evidence at hand - could some owners of
> different platforms compile and run the attached little C
> source please?
...
> Seems Tom is (unfortunately) right. The pipe blo
Tom Lane wrote:
> Samuel Sieb <[EMAIL PROTECTED]> writes:
> > Just as another suggestion, what about sending the data to a different
> > computer, so instead of tying up the database server with processing the
> > statistics, you have another computer that has some free time to do the
> > processi
> > But per-table stats aren't something that people will look at often,
> > right? They can sit in the collector's memory for quite a while. See
> > people wanting to look at per-backend stuff frequently, and that is why
> > I thought share memory should be good, and a global area for aggregate
Bruce Momjian <[EMAIL PROTECTED]> writes:
> Even better, have an SQL table updated with the per-table stats
> periodically.
>>
>> That will be horribly expensive, if it's a real table.
> But per-table stats aren't something that people will look at often,
> right? They can sit in the collector'
> Bruce Momjian <[EMAIL PROTECTED]> writes:
> > The only open issue is per-table stuff, and I would like to see some
> > circular buffer implemented to handle that, with a collection process
> > that has access to shared memory.
>
> That will get us into locking/contention issues. OTOH, frequent
Bruce Momjian <[EMAIL PROTECTED]> writes:
> The only open issue is per-table stuff, and I would like to see some
> circular buffer implemented to handle that, with a collection process
> that has access to shared memory.
That will get us into locking/contention issues. OTOH, frequent trips
to th
> ... and a lot more load on the CPU. Same-machine "network" connections
> are much cheaper (on most kernels, anyway) than real network
> connections.
>
> I think all of this discussion is vast overkill. No one has yet
> demonstrated that it's not sufficient to have *one* collector process
> an
Samuel Sieb <[EMAIL PROTECTED]> writes:
> Just as another suggestion, what about sending the data to a different
> computer, so instead of tying up the database server with processing the
> statistics, you have another computer that has some free time to do the
> processing.
> Some drawbacks are
On Sat, Mar 17, 2001 at 09:33:03AM -0500, Jan Wieck wrote:
>
> The general problem remains. We only have one central
> collector with a limited receive capacity. The more load is
> on the machine, the smaller it's capacity gets. The more
> complex the DB schemas get
Philip Warner wrote:
> At 13:49 16/03/01 -0500, Jan Wieck wrote:
> >
> >Similar problem as with shared memory - size. If a long
> >running backend of a multithousand table database needs to
> >send access stats per table - and had accessed them all up to
> >now - it'll be
At 13:49 16/03/01 -0500, Jan Wieck wrote:
>
>Similar problem as with shared memory - size. If a long
>running backend of a multithousand table database needs to
>send access stats per table - and had accessed them all up to
>now - it'll be alot of wasted bandwidth.
Not if
* Larry Rosenman <[EMAIL PROTECTED]> [010316 20:47]:
> * Jan Wieck <[EMAIL PROTECTED]> [010316 16:35]:
> $ ./queuetest
> Pipe buffer is 32768 bytes
> Sys-V message queue buffer is 4096 bytes
> $ uname -a
> UnixWare lerami 5 7.1.1 i386 x86at SCO UNIX_SVR5
> $
>
> I think some of these are configu
* Jan Wieck <[EMAIL PROTECTED]> [010316 16:35]:
> Jan Wieck wrote:
> > Tom Lane wrote:
> > > Now this would put a pretty tight time constraint on the collector:
> > > fall more than 4K behind, you start losing data. I am not sure if
> > > a UDP socket would provide more buffering or not; anyone k
> Just to get some evidence at hand - could some owners of
> different platforms compile and run the attached little C
> source please?
$ uname -srm
FreeBSD 4.1.1-STABLE
$ ./jan
Pipe buffer is 16384 bytes
Sys-V message queue buffer is 2048 bytes
$ uname -srm
NetBSD 1.5 alph
Jan Wieck <[EMAIL PROTECTED]> writes:
> Just to get some evidence at hand - could some owners of
> different platforms compile and run the attached little C
> source please?
HPUX 10.20:
Pipe buffer is 8192 bytes
Sys-V message queue buffer is 16384 bytes
Jan Wieck wrote:
> Tom Lane wrote:
> > Now this would put a pretty tight time constraint on the collector:
> > fall more than 4K behind, you start losing data. I am not sure if
> > a UDP socket would provide more buffering or not; anyone know?
>
> Looks like Linux has something around 16-32K
Tom Lane wrote:
> Now this would put a pretty tight time constraint on the collector:
> fall more than 4K behind, you start losing data. I am not sure if
> a UDP socket would provide more buffering or not; anyone know?
Looks like Linux has something around 16-32K of buffer space
for UDP
Tom Lane wrote:
> Jan Wieck <[EMAIL PROTECTED]> writes:
> > Does a pipe guarantee that a buffer, written with one atomic
> > write(2), never can get intermixed with other data on the
> > readers end?
>
> Yes. The HPUX man page for write(2) sez:
>
> o Write requests of
Jan Wieck <[EMAIL PROTECTED]> writes:
> Does a pipe guarantee that a buffer, written with one atomic
> write(2), never can get intermixed with other data on the
> readers end?
Yes. The HPUX man page for write(2) sez:
o Write requests of {PIPE_BUF} bytes or less will
* Tom Lane <[EMAIL PROTECTED]> [010316 10:06] wrote:
> Jan Wieck <[EMAIL PROTECTED]> writes:
> > Uh - not much time to spend if the statistics should at least
> > be half accurate. And it would become worse in SMP systems.
> > So that was a nifty idea, but I think it'd cause much
Tom Lane wrote:
> Jan Wieck <[EMAIL PROTECTED]> writes:
> > Uh - not much time to spend if the statistics should at least
> > be half accurate. And it would become worse in SMP systems.
> > So that was a nifty idea, but I think it'd cause much more
> > statistic losses than I
Alfred Perlstein wrote:
> * Jan Wieck <[EMAIL PROTECTED]> [010316 08:08] wrote:
> > Philip Warner wrote:
> > >
> > > But I prefer the UDP/Collector model anyway; it gives use greater
> > > flexibility + the ability to keep stats past backend termination, and,as
> > > you say, removes any possible
At 17:10 15/03/01 -0800, Alfred Perlstein wrote:
>>
>> Which is why the backends should not do anything other than maintain the
>> raw data. If there is atomic data than can cause inconsistency, then a
>> dropped UDP packet will do the same.
>
>The UDP packet (a COPY) can contain a consistant sna
Jan Wieck <[EMAIL PROTECTED]> writes:
> Uh - not much time to spend if the statistics should at least
> be half accurate. And it would become worse in SMP systems.
> So that was a nifty idea, but I think it'd cause much more
> statistic losses than I assumed at first.
> B
* Jan Wieck <[EMAIL PROTECTED]> [010316 08:08] wrote:
> Philip Warner wrote:
> >
> > But I prefer the UDP/Collector model anyway; it gives use greater
> > flexibility + the ability to keep stats past backend termination, and,as
> > you say, removes any possible locking requirements from the backen
Philip Warner wrote:
>
> But I prefer the UDP/Collector model anyway; it gives use greater
> flexibility + the ability to keep stats past backend termination, and,as
> you say, removes any possible locking requirements from the backends.
OK, did some tests...
The postmaster can create a
At 16:55 15/03/01 -0800, Alfred Perlstein wrote:
>* Philip Warner <[EMAIL PROTECTED]> [010315 16:46] wrote:
>> At 16:17 15/03/01 -0800, Alfred Perlstein wrote:
>> >
>> >Lost data is probably better than incorrect data. Either use locks
>> >or a copying mechanism. People will depend on the data r
* Philip Warner <[EMAIL PROTECTED]> [010315 17:08] wrote:
> At 16:55 15/03/01 -0800, Alfred Perlstein wrote:
> >* Philip Warner <[EMAIL PROTECTED]> [010315 16:46] wrote:
> >> At 16:17 15/03/01 -0800, Alfred Perlstein wrote:
> >> >
> >> >Lost data is probably better than incorrect data. Either use
* Philip Warner <[EMAIL PROTECTED]> [010315 16:46] wrote:
> At 16:17 15/03/01 -0800, Alfred Perlstein wrote:
> >
> >Lost data is probably better than incorrect data. Either use locks
> >or a copying mechanism. People will depend on the data returned
> >making sense.
> >
>
> But with per-backend
At 16:17 15/03/01 -0800, Alfred Perlstein wrote:
>
>Lost data is probably better than incorrect data. Either use locks
>or a copying mechanism. People will depend on the data returned
>making sense.
>
But with per-backend data, there is only ever *one* writer to a given set
of counters. Everyon
At 06:57 15/03/01 -0500, Jan Wieck wrote:
>
>And shared memory has all the interlocking problems we want
>to avoid.
I suspect that if we keep per-backend data in a separate area, then we
don;t need locking since there is only one writer. It does not matter if a
reader gets an inconsiste
* Philip Warner <[EMAIL PROTECTED]> [010315 16:14] wrote:
> At 06:57 15/03/01 -0500, Jan Wieck wrote:
> >
> >And shared memory has all the interlocking problems we want
> >to avoid.
>
> I suspect that if we keep per-backend data in a separate area, then we
> don;t need locking since the
Tom Lane wrote:
> Jan Wieck <[EMAIL PROTECTED]> writes:
> > What about a collector deamon, fired up by the postmaster and
> > receiving UDP packets from the backends. Under heavy load, it
> > might miss some statistic messages, well, but that's not as
> > bad as having locks caus
Tom Lane wrote:
> Jan Wieck <[EMAIL PROTECTED]> writes:
> > What about a collector deamon, fired up by the postmaster and
> > receiving UDP packets from the backends. Under heavy load, it
> > might miss some statistic messages, well, but that's not as
> > bad as having locks caus
Jan Wieck <[EMAIL PROTECTED]> writes:
> What about a collector deamon, fired up by the postmaster and
> receiving UDP packets from the backends. Under heavy load, it
> might miss some statistic messages, well, but that's not as
> bad as having locks causing backends to loose perf
Bruce Momjian wrote:
>
> Yes, it seems storing info in shared memory and having a system table to
> access it is the way to go.
Depends,
first of all we need to know WHAT we want to collect. If we
talk about block read/write statistics and such on a per
table base, which
> At 13:34 12/03/01 -0800, Alfred Perlstein wrote:
> >Is it possible
> >to have a spinlock over it so that an external utility can take a snapshot
> >of it with the spinlock held?
>
> I'd suggest that locking the stats area might be a bad idea; there is only
> one writer for each backend-specific
* Philip Warner <[EMAIL PROTECTED]> [010313 06:42] wrote:
> >
> >This ought to always give a consistant snapshot of the file to
> >whomever opens it.
> >
>
> I think Tom has previously stated that there are technical reasons not to
> do IO in signal handlers, and I have philosophical problems wit
>
>This ought to always give a consistant snapshot of the file to
>whomever opens it.
>
I think Tom has previously stated that there are technical reasons not to
do IO in signal handlers, and I have philosophical problems with
performance monitors that ask 50 backends to do file IO. I really do t
> >
> >This ought to always give a consistant snapshot of the file to
> >whomever opens it.
> >
>
> I think Tom has previously stated that there are technical reasons not to
> do IO in signal handlers, and I have philosophical problems with
> performance monitors that ask 50 backends to do file I
> > I think Tom has previously stated that there are technical reasons not to
> > do IO in signal handlers, and I have philosophical problems with
> > performance monitors that ask 50 backends to do file IO. I really do think
> > shared memory is TWTG.
>
> I wasn't really suggesting any of those
* Philip Warner <[EMAIL PROTECTED]> [010312 18:56] wrote:
> At 13:34 12/03/01 -0800, Alfred Perlstein wrote:
> >Is it possible
> >to have a spinlock over it so that an external utility can take a snapshot
> >of it with the spinlock held?
>
> I'd suggest that locking the stats area might be a bad
At 13:34 12/03/01 -0800, Alfred Perlstein wrote:
>Is it possible
>to have a spinlock over it so that an external utility can take a snapshot
>of it with the spinlock held?
I'd suggest that locking the stats area might be a bad idea; there is only
one writer for each backend-specific chunk, and it
* Bruce Momjian <[EMAIL PROTECTED]> [010312 12:12] wrote:
> I was going to implement the signal handler like we do with Cancel,
> where the signal sets a flag and we check the status of the flag in
> various _safe_ places.
>
> Can anyone think of a better way to get information out of a backend?
I was going to implement the signal handler like we do with Cancel,
where the signal sets a flag and we check the status of the flag in
various _safe_ places.
Can anyone think of a better way to get information out of a backend?
--
Bruce Momjian| http://candle.pha.pa.
53 matches
Mail list logo
