-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
> old method scaled (badly) on volume of notifications and your stuff
> seems to scale based on # of client's sending simultaneous
> notifications. Well, you're better all day long, but it shows that
> your fears regarding locking were not com
On Mon, Nov 16, 2009 at 4:41 PM, Joachim Wieland wrote:
> On Sat, Nov 14, 2009 at 11:06 PM, Merlin Moncure wrote:
>> The old method (measured on a 4 core high performance server) has
>> severe scaling issues due to table bloat (we knew that):
>> ./pgbench -c 10 -t 1000 -n -b listen.sql -f notify.
On Sat, Nov 14, 2009 at 11:06 PM, Merlin Moncure wrote:
> The old method (measured on a 4 core high performance server) has
> severe scaling issues due to table bloat (we knew that):
> ./pgbench -c 10 -t 1000 -n -b listen.sql -f notify.sql
> run #1 tps = 1364.948079 (including connections establis
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
>> You misunderstand the requirements. LISTEN notifications are *not*
>> meant to survive a database crash, and never have been. However,
>> so long as both client and server stay up, they must be reliable.
>> If the client has to poll databas
Simon Riggs writes:
> On Sun, 2009-11-15 at 16:48 -0500, Tom Lane wrote:
>> You misunderstand the requirements. LISTEN notifications are *not*
>> meant to survive a database crash, and never have been. However,
>> so long as both client and server stay up, they must be reliable.
>> If the client
On Sun, 2009-11-15 at 16:48 -0500, Tom Lane wrote:
> Simon Riggs writes:
> > On Wed, 2009-11-11 at 22:25 +0100, Joachim Wieland wrote:
> >> 3. Every distinct notification is delivered.
> >> Regarding performance, the slru-queue is not fsync-ed to disk
>
> > These two statements seem to be in oppo
Simon Riggs writes:
> On Wed, 2009-11-11 at 22:25 +0100, Joachim Wieland wrote:
>> 3. Every distinct notification is delivered.
>> Regarding performance, the slru-queue is not fsync-ed to disk
> These two statements seem to be in opposition. How do you know a
> notification will be delivered if t
On Wed, 2009-11-11 at 22:25 +0100, Joachim Wieland wrote:
> 3. Every distinct notification is delivered.
> Regarding performance, the slru-queue is not fsync-ed to disk
These two statements seem to be in opposition. How do you know a
notification will be delivered if the queue is non-recoverabl
On Thu, 12 Nov 2009 11:22:32 -0500
Andrew Chernow wrote:
>
> > However I share Greg's concerns that people are trying to use NOTIFY
> > as a message queue which it is not designed to be.
>
> When you have an established libpq connection waiting for notifies it
> is not unreasonable to expect/de
On Fri, Nov 13, 2009 at 11:08 AM, Tom Lane wrote:
(By the way, has anyone yet tried to
> compare the speed of this implementation to the old code?)
I quickly hacked pgbench to take a custom script on connection (for
listen), and make pgbench'd 'notify x'; with all clients doing 'listen
x'.
The
Josh Berkus wrote:
Payloads are also quite useful even in a lossy environment, where you
understand that LISTEN is not a queue. For example, I'd like to be
using LISTEN/NOTIFY for cache invalidation for some applications; if it
misses a few, or double-counts them, it's not an issue. However, I'
My original intention was to have the queue as a circular buffer where
the size of the entries was variable, but possibly bounded. Certainly
using fixed length records of large size seems somewhat wasteful.
Maybe we should do away with 'spill to disk' all together and either
hard-code an o
Tom Lane wrote:
"Greg Sabino Mullane" writes:
Talk of efficiency also seems silly here - using
shared memory is already way more efficient than our current listen/notify
system.
Except that the proposed implementation spills to disk. Particularly if
it has to have support for large payloads,
"Greg Sabino Mullane" writes:
> Talk of efficiency also seems silly here - using
> shared memory is already way more efficient than our current listen/notify
> system.
Except that the proposed implementation spills to disk. Particularly if
it has to have support for large payloads, it could very
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
> This is BS. The problem is not that someone might do something stupid
> with this feature. The problem is that we're making these other use
> cases into r
Merlin Moncure wrote:
On Fri, Nov 13, 2009 at 10:00 AM, Andrew Chernow wrote:
I think the original OP was close. The structure can still be fixed length
but maybe we can bump it to 8k (BLCKSZ)?
The problem with this (which I basically agree with) is that this will
greatly increase
Merlin Moncure writes:
> The problem with this (which I basically agree with) is that this will
> greatly increase the size of the queue for all participants of this
> feature if they use the payload or not. I think it boils down to
> this: is there a reasonably effective way of making the payloa
On Fri, Nov 13, 2009 at 10:00 AM, Andrew Chernow wrote:
> I think the original OP was close. The structure can still be fixed length
> but maybe we can bump it to 8k (BLCKSZ)?
The problem with this (which I basically agree with) is that this will
greatly increase the size of the queue for all pa
Calling this a creeping feature is quite a leap.
It's true that the real creep is having the payload at all rather than
not having it.
Not having the payload at all is like santa showing up without his bag
of toys. Instead, you have to drive/fly to the north pole where he just
came from to
On Fri, Nov 13, 2009 at 1:47 PM, Andrew Chernow wrote:
> This is what I think the people's real problem is, the implementation
> becomes a more complex when including payloads (larger ones even more so).
> I think its a side-track to discuss queue vs condition variables. Whether
> a notify is 20
spill to disk and need an efficient storage mechanism. The natural
implementation of this in Postgres would be a table, not the slru. If
This is what I think the people's real problem is, the implementation becomes a
more complex when including payloads (larger ones even more so). I think it
Joachim Wieland wrote:
> 1. Instead of placing the queue into shared memory only I propose to create a
> new subdirectory pg_notify/ and make the queue slru-based, such that we do not
> risk blocking. Several people here have pointed out that blocking is a true
> no-go for a new listen/notify impl
On Fri, Nov 13, 2009 at 5:35 AM, Greg Stark wrote:
> On Fri, Nov 13, 2009 at 1:57 AM, Robert Haas wrote:
>> I agree. We frequently reject features on the basis that someone
>> might do something stupid with them. It's lame and counterproductive,
>> and we should stop. The world contains infini
On Fri, Nov 13, 2009 at 1:57 AM, Robert Haas wrote:
> I agree. We frequently reject features on the basis that someone
> might do something stupid with them. It's lame and counterproductive,
> and we should stop. The world contains infinite amounts of lameness,
> but that's the world's problem,
Unfortunately with all that payload-length discussion, the other part
of my email regarding ACID compliant behavior got completely lost. I
would appreciate some input on that part also...
Thanks,
Joachim
On Thu, Nov 12, 2009 at 12:17 PM, Joachim Wieland wrote:
> On Thu, Nov 12, 2009 at 4:25 AM,
On Nov 12, 2009, at 5:57 PM, Robert Haas wrote:
> On Thu, Nov 12, 2009 at 8:44 PM, Josh Berkus wrote:
>> On 11/12/09 8:30 AM, Tom Lane wrote:
>>> So while a payload string for NOTIFY has been on the to-do list since
>>> forever, I have to think that Greg's got a good point questioning
>>> whethe
and we should stop. The world contains infinite amounts of lameness,
but that's the world's problem, not ours. There is zero evidence that
+1
this feature is only useful for stupid purposes, and some evidence
(namely, the opinions of esteemed community members) that it is useful
for at lea
On Thu, Nov 12, 2009 at 8:44 PM, Josh Berkus wrote:
> On 11/12/09 8:30 AM, Tom Lane wrote:
>> So while a payload string for NOTIFY has been on the to-do list since
>> forever, I have to think that Greg's got a good point questioning
>> whether it is actually a good idea.
>
> Sure, people will abus
On 11/12/09 8:30 AM, Tom Lane wrote:
> So while a payload string for NOTIFY has been on the to-do list since
> forever, I have to think that Greg's got a good point questioning
> whether it is actually a good idea.
Sure, people will abuse it as a queue. But people abuse arrays when
they should be
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
Tom Lane writes:
> Yes. Particularly those complaining that they want to have very large
> payload strings --- that's pretty much a dead giveaway that it's not
> being used as a
On Thu, Nov 12, 2009 at 11:40 AM, Merlin Moncure wrote:
> I'm sorry, the 128 character limit is simply lame (other than for
> unsolvable implementation/performance complexity which I doubt is the
> case here), and if that constraint is put in by the implementation,
> than the implementation is bus
Now you might say that yeah, that's the point, we're trying to enable
using NOTIFY in a different style. The problem is that if you are
trying to use NOTIFY as a queue, you will soon realize that it has
the wrong semantics for that --- in particular, losing notifies across
a system crash or cli
On Thu, Nov 12, 2009 at 11:39 AM, Robert Haas wrote:
> On Thu, Nov 12, 2009 at 11:30 AM, Tom Lane wrote:
>> Joachim Wieland writes:
>>> However I share Greg's concerns that people are trying to use NOTIFY
>>> as a message queue which it is not designed to be.
>>
>> Yes. Particularly those compl
On Thu, Nov 12, 2009 at 11:30 AM, Tom Lane wrote:
> Joachim Wieland writes:
>> However I share Greg's concerns that people are trying to use NOTIFY
>> as a message queue which it is not designed to be.
>
> Yes. Particularly those complaining that they want to have very large
> payload strings --
On Thu, Nov 12, 2009 at 11:30 AM, Tom Lane wrote:
> Joachim Wieland writes:
>> However I share Greg's concerns that people are trying to use NOTIFY
>> as a message queue which it is not designed to be.
>
> Yes. Particularly those complaining that they want to have very large
> payload strings --
On Thu, Nov 12, 2009 at 4:30 PM, Tom Lane wrote:
> So while a payload string for NOTIFY has been on the to-do list since
> forever, I have to think that Greg's got a good point questioning
> whether it is actually a good idea.
Here's an example of why I'd like a payload (and not a queue in an
ad
Joachim Wieland writes:
> However I share Greg's concerns that people are trying to use NOTIFY
> as a message queue which it is not designed to be.
Yes. Particularly those complaining that they want to have very large
payload strings --- that's pretty much a dead giveaway that it's not
being use
However I share Greg's concerns that people are trying to use NOTIFY
as a message queue which it is not designed to be.
When you have an established libpq connection waiting for notifies it is
not unreasonable to expect/desire a payload. ISTM, the problem is that
the initial design was half
--On 12. November 2009 15:48:44 + Greg Stark wrote:
I'm beginning to think the right solution is to reject the idea of
adding a payload to the NOTIFY mechanism and instead provide a queue
contrib module which provides the interface people want.
Isn't PgQ already the solution you have in
On Thu, Nov 12, 2009 at 3:30 PM, Merlin Moncure wrote:
> Couple of questions:
>
> *) is BLCKSZ a hard requirement, that is, coming from the slru
> implementation, or can QUEUE_PAGESIZE be bumped independently of block
> size.
It's the size of slru's pages.
> *) why not make the AsyncQueueEntry d
On Thu, Nov 12, 2009 at 3:09 PM, Andrew Chernow wrote:
>>
>> What advantage is there in limiting it to a tiny size? This is a
>> 'payload' after all...an arbitrary data block. Looking at the patch I
>> noticed the payload structure (AsyncQueueEntry) is fixed length and
>> designed to lay into QUE
What advantage is there in limiting it to a tiny size? This is a
'payload' after all...an arbitrary data block. Looking at the patch I
noticed the payload structure (AsyncQueueEntry) is fixed length and
designed to lay into QUEUE_PAGESIZE (set to) BLCKSZ sized pages.
H. Looks like the
On Thu, Nov 12, 2009 at 8:25 AM, Andrew Chernow wrote:
>
2. The payload parameter is optional. A notifying client can either call
"NOTIFY foo;" or "NOTIFY foo 'payload';". The length of the payload is
currently limited to 128 characters... Not sure if we should allow
longer
>>>
2. The payload parameter is optional. A notifying client can either call
"NOTIFY foo;" or "NOTIFY foo 'payload';". The length of the payload is
currently limited to 128 characters... Not sure if we should allow longer
payload strings...
Might be a good idea to make the max the same as the max l
On Thu, Nov 12, 2009 at 4:25 AM, Tom Lane wrote:
>> One possible solution would be to write to the queue before committing
>> and adding the TransactionID. Then other backends can check if our
>> TransactionID has successfully committed or not. Not sure if this is
>> worth the overhead however...
On Thu, Nov 12, 2009 at 2:12 AM, Andrew Gierth
wrote:
> Does it cope with the case where a trigger is doing NOTIFY, and you do
> a whole-table update, therefore dumping potentially millions of
> notifications in at once?
>
> (for example a rare maintenance operation on a table which has a
> listen
On Nov 11, 2009, at 10:43 PM, Tom Lane wrote:
Andrew Chernow writes:
I thought of a compromise: add the number of times a notification
was
generated (coalesced count+1) to the callback data. That would
satisfy
any backwards compatibility concerns and my use case too!
If you are suggesti
2. The payload parameter is optional. A notifying client can either call
"NOTIFY foo;" or "NOTIFY foo 'payload';". The length of the payload is
currently limited to 128 characters... Not sure if we should allow longer
payload strings...
Might be a good idea to make the max the same as the max le
Andrew Chernow writes:
>> I thought of a compromise: add the number of times a notification was
>> generated (coalesced count+1) to the callback data. That would satisfy
>> any backwards compatibility concerns and my use case too!
> If you are suggesting that the server poke data into the notif
I thought of a compromise: add the number of times a notification was
generated (coalesced count+1) to the callback data. That would satisfy
any backwards compatibility concerns and my use case too!
If you are suggesting that the server poke data into the notifier's opaque
payload, I vot
Joachim Wieland writes:
> However, if for some reason we cannot write to the slru files in the
> pg_notify/
> directory we might want to roll back the current transaction but with the
> proposed patch we cannot because we have already committed...
I think this is a deal-breaker, and arguing abou
On Nov 11, 2009, at 9:28 PM, Merlin Moncure wrote:
On Wed, Nov 11, 2009 at 5:48 PM, A.M.
wrote:
At least with this new payload, I can set the payload to the
transaction ID
and be certain that all the notifications I sent are processed
(and in order
even!) but could you explain why the co
On Wed, Nov 11, 2009 at 5:48 PM, A.M. wrote:
> At least with this new payload, I can set the payload to the transaction ID
> and be certain that all the notifications I sent are processed (and in order
> even!) but could you explain why the coalescing is still necessary?
Christmas comes early thi
Premature optimization is the root of all evil ;-). Unless you've done
some profiling and can show that this is a hot spot, making it more
complicated isn't the thing to be doing now.
I'm thinking of how our system uses/abuses notifies, and began wondering
if several thousand backends liste
> "Martijn" == Martijn van Oosterhout writes:
>> Hi,
>>
>> Attached is a patch for a new listen/notify implementation.
>>
>> In a few words, the patch reimplements listen/notify as an
>> slru-based queue which works similar to the sinval
>> structure. Essentially it is a ring buffer
Andrew Chernow writes:
> + * This function is executed for every notification found in the queue in
> order
> + * to check if the current backend is listening on that channel. Not sure
> if we
> + * should further optimize this, for example convert to a sorted array and
> + * allow binary se
Joachim Wieland wrote:
On Thu, Nov 12, 2009 at 1:04 AM, Andrew Chernow wrote:
I think a bsearch would be needed. Very busy servers that make heavy use of
notifies would be quite a penalty.
In such an environment, how many relations/channels is a backend
typically listening to?
Do you have av
On Thu, Nov 12, 2009 at 1:04 AM, Andrew Chernow wrote:
> I think a bsearch would be needed. Very busy servers that make heavy use of
> notifies would be quite a penalty.
In such an environment, how many relations/channels is a backend
typically listening to?
Do you have average / maximal numbers
/*
+ * This function is executed for every notification found in the queue in
order
+ * to check if the current backend is listening on that channel. Not sure if
we
+ * should further optimize this, for example convert to a sorted array and
+ * allow binary search on it...
+ */
+ static b
2. adds the possibility to specify a payload parameter, i.e. executing
in SQL
"NOTIFY foo 'payload';" and 'payload' will be delivered to any
listening
backend.
Thank you for implementing this- LISTEN/NOTIFY without a payload has
been a major problem to work around for me.
+1
--
Andre
On Nov 11, 2009, at 4:25 PM, Joachim Wieland wrote:
Hi,
Attached is a patch for a new listen/notify implementation.
In a few words, the patch reimplements listen/notify as an slru-
based queue
which works similar to the sinval structure. Essentially it is a
ring buffer on
disk with pages
On Wed, Nov 11, 2009 at 10:25:05PM +0100, Joachim Wieland wrote:
> Hi,
>
> Attached is a patch for a new listen/notify implementation.
>
> In a few words, the patch reimplements listen/notify as an slru-based queue
> which works similar to the sinval structure. Essentially it is a ring buffer
>
62 matches
Mail list logo
