There is a problem trying to make Postgres do these things in Parallel.
The backend code isn't thread-safe, so doing a multi-thread
implementation requires quite a bit of work.
Using multiple processes has its own problems: The whole way locking
works equates one process with one transact
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Mon, Oct 20, 2008 at 05:49:04PM -0300, Alvaro Herrera wrote:
> Hi,
>
> My Debian system (now running Linux 2.6.26) is no longer dumping core
> files, and I can't figure out why :-(
FWIW, same happens here, out-of-the-box 2.6.26-1 vanilla Debian. B
On Mon, Oct 20, 2008 at 12:05 PM, Julius Stroffek
<[EMAIL PROTECTED]>wrote:
> Topics that seem to be of interest and most of them were already
> discussed at developers meeting in Ottawa are
> 1.) parallel sorts
> 2.) parallel query execution
> 3.) asynchronous I/O
> 4.) parallel COPY
> 5.) parall
I wrote:
> Now I'm working on storing statistics into disks on server
> shutdown. If it is impossible unless the module is in core,
> I would change my policy...
I reconsidered this part and found that pg_stat_statements needs to be
in core to write stats in file on server shutdown because:
1.
2008/10/21 Heikki Linnakangas <[EMAIL PROTECTED]>:
> Hitoshi Harada wrote:
>>
>> The real problem is not how to cut off preceding rows, but how to read
>> ahead after the current row. I intend to avoid reading ahead until end
>> of the partition for only row_number() that doesn't need any following
Alvaro Herrera wrote:
> So this discussion died with no solution arising to the
> hint-bit-setting-invalidates-the-CRC problem.
Is there no point at which a page is logically committed to
storage, past which no mutating access may be performed?
--
Sent via pgsql-hackers mailing list (pgsql-hac
On Oct 20, 2008, at 3:00 PM, Joshua Tolley wrote:
One of the Java-as-a-procedural-language options uses RMI to get the
server talking to a separate JVM, where the actual function processing
gets done. Could a PL/Lisp work similarly (and would it be anything
approaching a good idea...)?
I thin
We propose a patch that improves hybrid hash join's performance for
large multi-batch joins where the probe relation has skew.
Project name: Histojoin
Patch file: histojoin_v1.patch
This patch implements the Histojoin join algorithm as an optional
feature added to the standard Hybrid Hash
Alvaro Herrera <[EMAIL PROTECTED]> writes:
> Hmm, I just noticed this comment in bufpage.h (which was also in Jonah's
> patch) :-(
> typedef struct PageHeaderData
> {
> /* XXX LSN is member of *any* block, not only page-organized ones */
> ...
We don't have any non-page-organized blocks ;
Simon Riggs <[EMAIL PROTECTED]> writes:
> On Mon, 2008-10-20 at 17:44 -0300, Alvaro Herrera wrote:
>> That's been "extended with an epoch counter" per the docs; I don't think
>> that's appropriate for the new functions, is it?
> I assumed it was, so you can subtract them easily.
> It can be done
On Mon, 20 Oct 2008, Alvaro Herrera wrote:
My Debian system (now running Linux 2.6.26) is no longer dumping core
files, and I can't figure out why :-(
My guess is that you're being nailed by one of the changes related to
implementing the improved capabilities interface made in 2.6.25 or 2.6.2
On Mon, Oct 20, 2008 at 5:23 PM, Alvaro Herrera
<[EMAIL PROTECTED]> wrote:
> Hmm, I just noticed this comment in bufpage.h (which was also in Jonah's
> patch) :-(
>
> typedef struct PageHeaderData
> {
>/* XXX LSN is member of *any* block, not only page-organized ones */
Passed regressions and
Alvaro Herrera wrote:
> Log Message:
> ---
> Properly access a buffer's LSN using existing access macros instead of abusing
> knowledge of page layout.
Hmm, I just noticed this comment in bufpage.h (which was also in Jonah's
patch) :-(
typedef struct PageHeaderData
{
/* XXX LSN is mem
On Mon, 2008-10-20 at 17:44 -0300, Alvaro Herrera wrote:
> Simon Riggs escribió:
> >
> > On Mon, 2008-10-20 at 16:22 -0400, Robert Haas wrote:
> > > > * pg_last_recovered_xact_xid()
> > > > Will throw an ERROR if *not* executed in recovery mode.
> > > > returns bigint
> > > >
> > > > * pg_last_co
Robert Haas wrote:
>>> How can you make that the default? Won't it immediately break every
>>> installation without certificates?
>> *all* SSL installations have certificate on the server side. You cannot
>> run without it.
>
> s/without certificates/with self-signed certificates/
>
> which I wo
Hi,
My Debian system (now running Linux 2.6.26) is no longer dumping core
files, and I can't figure out why :-(
Of course, I've set ulimit -c to unlimited, and I'm running the
postmaster directly in the same shell (no pg_ctl or init scripts), but
it's still not working. I'm not sure where else t
>> How can you make that the default? Won't it immediately break every
>> installation without certificates?
>
> *all* SSL installations have certificate on the server side. You cannot
> run without it.
s/without certificates/with self-signed certificates/
which I would guess to be a common conf
Simon Riggs escribió:
>
> On Mon, 2008-10-20 at 16:22 -0400, Robert Haas wrote:
> > > * pg_last_recovered_xact_xid()
> > > Will throw an ERROR if *not* executed in recovery mode.
> > > returns bigint
> > >
> > > * pg_last_completed_xact_xid()
> > > Will throw an ERROR *if* executed in recovery mod
Zdenek Kotala wrote:
> Last commit broken transam.c function definitions:
>
>
> "transam.c", line 268: void function cannot return value
> "transam.c", line 280: void function cannot return value
> cc: acomp failed for transam.c
Interesting ... my compiler did not even complain. Fixed, thanks for
Per earlier discussion,
http://archives.postgresql.org/pgsql-hackers/2008-10/msg00853.php
I've been fooling around with a patch to let the planner evaluate
some expressions at lower join levels and bubble the results up
like Vars. I've got it passing the regression tests now, and though
there is m
On Mon, 2008-10-20 at 16:22 -0400, Robert Haas wrote:
> > * pg_last_recovered_xact_xid()
> > Will throw an ERROR if *not* executed in recovery mode.
> > returns bigint
> >
> > * pg_last_completed_xact_xid()
> > Will throw an ERROR *if* executed in recovery mode.
> > returns bigint
>
> Should thes
"Mark Cave-Ayland" <[EMAIL PROTECTED]> writes:
> I'm just following up on the patch created here by Tom to aid with
> repeated de-TOASTing attempts:
> http://archives.postgresql.org/pgsql-hackers/2008-06/msg01096.php
> Given the performance report from Jeff
> (http://archives.postgresql.org/pgsql-
Last commit broken transam.c function definitions:
"transam.c", line 268: void function cannot return value
"transam.c", line 280: void function cannot return value
cc: acomp failed for transam.c
Detail
http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=gothic_moth&dt=2008-10-20%2020:06:01
> * pg_last_recovered_xact_xid()
> Will throw an ERROR if *not* executed in recovery mode.
> returns bigint
>
> * pg_last_completed_xact_xid()
> Will throw an ERROR *if* executed in recovery mode.
> returns bigint
Should these return xid?
...Robert
--
Sent via pgsql-hackers mailing list (pgsql-
On Mon, 2008-10-20 at 16:23 -0300, Alvaro Herrera wrote:
> Simon Riggs wrote:
>
> > Renamed to set_status_by_pages because we never use this on the whole
> > tree. Added comments to say that.
> >
> > Overall, cleaner and more readable now. Thanks.
>
> Committed, thanks.
Cheers.
--
Simon Rig
On Mon, 2008-10-20 at 21:11 +0300, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > OK, I think I've found a problem.
> >
> > In _bt_insertonpg(), if we split we do _bt_split() then do
> > _bt_insert_parent(), which then does _bt_insertonpg() recursively.
> >
> > _bt_split() writes a WAL recor
Hitoshi Harada wrote:
The real problem is not how to cut off preceding rows, but how to read
ahead after the current row. I intend to avoid reading ahead until end
of the partition for only row_number() that doesn't need any following
rows. Sometimes we have to store whole the partition before re
Simon Riggs wrote:
> Renamed to set_status_by_pages because we never use this on the whole
> tree. Added comments to say that.
>
> Overall, cleaner and more readable now. Thanks.
Committed, thanks.
--
Alvaro Herrerahttp://www.CommandPrompt.com/
PostgreSQL Replic
Hi All,
we would like to start some work on improving the performance of
PostgreSQL in a multi-CPU environment. Dano Vojtek is student at the
Faculty of Mathematics and Physics of Charles university in Prague
(http://www.mff.cuni.cz) and he is going to cover this topic in his
master thesis. He is
Hi everyone,
I'm just following up on the patch created here by Tom to aid with
repeated de-TOASTing attempts:
http://archives.postgresql.org/pgsql-hackers/2008-06/msg01096.php
Given the performance report from Jeff
(http://archives.postgresql.org/pgsql-hackers/2008-08/msg01178.php), is
there sti
Markus Wanner wrote:
Alvaro Herrera wrote:
So this discussion died with no solution arising to the
hint-bit-setting-invalidates-the-CRC problem.
Isn't double-buffering solving this issue? Has somebody checked if it
even helps performance due to being able to release the lock on the
buffer *bef
On Mon, Oct 20, 2008 at 12:56 PM, John DeSoi <[EMAIL PROTECTED]> wrote:
>
> On Oct 19, 2008, at 1:27 PM, Douglas McNaught wrote:
>
>> SBCL is a big and very sophisticated program. It's designed to be a
>> self-contained Lisp system and has (AFAIK) no concessions to
>> "embeddability". It uses thr
On Oct 19, 2008, at 1:27 PM, Douglas McNaught wrote:
SBCL is a big and very sophisticated program. It's designed to be a
self-contained Lisp system and has (AFAIK) no concessions to
"embeddability". It uses threads internally, and plays games with the
memory map to make GC more efficient. On
Tom Lane wrote:
> Magnus Hagander <[EMAIL PROTECTED]> writes:
>> Attached patch cleans up the certificate verification in libpq, and adds
>> a configuration paraqmeter to control it. The new parameter is
>> "sslverify", and can be set to:
>
>> * cn = default = will validate that the certificate ch
Simon Riggs wrote:
OK, I think I've found a problem.
In _bt_insertonpg(), if we split we do _bt_split() then do
_bt_insert_parent(), which then does _bt_insertonpg() recursively.
_bt_split() writes a WAL record but continues holding write locks.
btree_xlog_split() reads WAL record and does *not
But does holding cleanup lock on root prevent an in-progress Insert from
changing non-root pages? I assume so, just not sure how.
Yes, because insertion process doesn't unpin root page until insert will done.
--
Teodor Sigaev E-mail: [EMAIL PROTECTED]
On Mon, 2008-10-20 at 20:12 +0400, Teodor Sigaev wrote:
> > I don't understand why in ginVacuumPostingTreeLeaves() we lock only the
> > root page for Cleanup and no others. Why do we need to hold Cleanup lock
> > on the root? If the index is concurrent safe for existing scans, why is
> > it not sa
On Mon, 2008-10-20 at 16:11 +0100, Simon Riggs wrote:
> On Mon, 2008-10-20 at 18:24 +0400, Teodor Sigaev wrote:
> > > 3. Implement an extra indexAM API call that allows indexAM to decide
> > > when/if index is valid during recovery. This would also cover the second
> > > concern neatly in a single
Peter Eisentraut <[EMAIL PROTECTED]> writes:
> Tom Lane wrote:
>> BTW, I think it's a bad idea to assign made-up parse locations, as
>> you did here:
> Hmm, @$ is the location of the complete rule, so it should point to the
> "empty" spot in theory. Or am I misunderstanding something?
Well, yeah
I don't understand why in ginVacuumPostingTreeLeaves() we lock only the
root page for Cleanup and no others. Why do we need to hold Cleanup lock
on the root? If the index is concurrent safe for existing scans, why is
it not safe for new scans also? And the converse: if it is not safe for
new scans
One more thing about GiST - when database is switched from recovery mode to the
normal mode then it's needed to complete insertion in GiST and, possibly, vacuum
index. Dig around GistBulkDeleteResult->needFullVacuum and gistContinueInsert()
OK, will look at those. (Problems++).
Not a very big i
2008/10/20 Heikki Linnakangas <[EMAIL PROTECTED]>:
> Hitoshi Harada wrote:
>>
>> Hi,
>>
>> 2008/10/20 Simon Riggs <[EMAIL PROTECTED]>:
>>>
>>> On Mon, 2008-10-20 at 10:32 +0900, Hitoshi Harada wrote:
>>>
So I propose three Window node buffering strategies,
row/frame/partition buffering so
ITAGAKI Takahiro wrote:
> I'd like to submit pg_stat_statements contrib module, that counts up
> incoming statements in shared memory and summarizes the result as a view.
> It is just a statements-version of pg_stat_user_functions.
>
Nice work! There is one feature I'd like to request -- we need t
On Mon, 2008-10-20 at 18:24 +0400, Teodor Sigaev wrote:
> > 3. Implement an extra indexAM API call that allows indexAM to decide
> > when/if index is valid during recovery. This would also cover the second
> > concern neatly in a single API call.
> >
> > wait until after deadline to implement (2)
Tom Lane wrote:
Peter Eisentraut <[EMAIL PROTECTED]> writes:
SQL:2008 specifies the following syntax for what we have so far called
LIMIT and OFFSET
SELECT ... [ ORDER BY ... ]
OFFSET num {ROW|ROWS} FETCH {FIRST|NEXT} [num] {ROW|ROWS} ONLY
What does the "NEXT" option mean?
FIRST and NE
On Mon, 2008-10-20 at 18:28 +0400, Teodor Sigaev wrote:
> > * For GIN indexes, we appear to not hold a Cleanup lock during
> > vacuuming, except on root page. That stops new scans from starting, but
> > it doesn't prevent progress of concurrent scans. Doesn't look correct to
> > me... so not sure
Magnus Hagander wrote:
Andrew Dunstan wrote:
Tom Lane wrote:
Andrew Dunstan <[EMAIL PROTECTED]> writes:
OK. Where would be a good place to put the code? Maybe a new file
src/backend/utils/adt/trigger_utils.c ?
I thought the plan was to make it a contrib module
Hitoshi Harada wrote:
Hi,
2008/10/20 Simon Riggs <[EMAIL PROTECTED]>:
On Mon, 2008-10-20 at 10:32 +0900, Hitoshi Harada wrote:
So I propose three Window node buffering strategies,
row/frame/partition buffering so as to avoid unnecessary row
buffering.
Sounds good from here. Can I suggest you
Tom Lane wrote:
Peter Eisentraut <[EMAIL PROTECTED]> writes:
AFAICT, removing an authentication method requires a protocol version
bump.
Why would it require that? There would just be some auth method codes
that remain reserved but aren't used anymore.
Yeah, I was mistaken. AuthenticationC
Peter Eisentraut <[EMAIL PROTECTED]> writes:
> AFAICT, removing an authentication method requires a protocol version
> bump.
Why would it require that? There would just be some auth method codes
that remain reserved but aren't used anymore.
regards, tom lane
--
Sent vi
* For GIN indexes, we appear to not hold a Cleanup lock during
vacuuming, except on root page. That stops new scans from starting, but
it doesn't prevent progress of concurrent scans. Doesn't look correct to
me... so not sure what strength lock to acquire in each case. Probably
Why do you think s
3. Implement an extra indexAM API call that allows indexAM to decide
when/if index is valid during recovery. This would also cover the second
concern neatly in a single API call.
wait until after deadline to implement (2) or (3), in case somebody
fixes this up in the next few weeks.
IMHO, With
Hi,
2008/10/20 Simon Riggs <[EMAIL PROTECTED]>:
>
> On Mon, 2008-10-20 at 10:32 +0900, Hitoshi Harada wrote:
>
>> So I propose three Window node buffering strategies,
>> row/frame/partition buffering so as to avoid unnecessary row
>> buffering.
>
> Sounds good from here. Can I suggest you release
Magnus Hagander wrote:
I notice our docs have:
If you are at all concerned about password
sniffing attacks then md5 is preferred, with
crypt to be used only if you must support pre-7.2
clients. Plain password should be avoided especially for
At what point do we just remove the
On Mon, 2008-10-20 at 14:23 +0100, Gregory Stark wrote:
> Simon Riggs <[EMAIL PROTECTED]> writes:
>
> > * re-create the Cleanup lock on blocks, when the original operation was
> > performed while a Cleanup lock was held.
> >
> > So the plan is to introduce a new XLogLockBufferForCleanup() functio
Simon Riggs <[EMAIL PROTECTED]> writes:
> * re-create the Cleanup lock on blocks, when the original operation was
> performed while a Cleanup lock was held.
>
> So the plan is to introduce a new XLogLockBufferForCleanup() function
> and then use it in all places where a cleanup lock was held duri
Magnus Hagander <[EMAIL PROTECTED]> writes:
> Attached patch cleans up the certificate verification in libpq, and adds
> a configuration paraqmeter to control it. The new parameter is
> "sslverify", and can be set to:
> * cn = default = will validate that the certificate chains to a trusted
> root
Magnus Hagander wrote:
ITAGAKI Takahiro wrote:
Magnus Hagander <[EMAIL PROTECTED]> wrote:
I'm not sure what should be in the main and what should not.
Why is pg_freespacemap still in contrib?
I don't know, why is it? :-)
I guess that was a joke, given the smiley, but I'll bite:
1. pg_freesp
Tom Lane wrote:
> Magnus Hagander <[EMAIL PROTECTED]> writes:
>> Bruce Momjian wrote:
>>> This is missing 'do' or something:
>>>
>>> + #define MANDATORY_AUTH_ARG(argvar, argname, authname) \
>>> + if (argvar == NULL) {\
>>> + ereport(LOG, \
>>> + (errcode(ERRCODE_CONFIG_FILE_ERROR), \
>
I'm looking at how to make queries safe during recovery, in the presence
of concurrent changes to blocks. In particular, concurrent removal of
rows that might be read by queries.
My thinking is
* we ignore LockRelationForExtension(). Normal queries never request it.
All new blocks were created wi
Magnus Hagander <[EMAIL PROTECTED]> writes:
> Bruce Momjian wrote:
>> This is missing 'do' or something:
>>
>> + #define MANDATORY_AUTH_ARG(argvar, argname, authname) \
>> + if (argvar == NULL) {\
>> + ereport(LOG, \
>> + (errcode(ERRCODE_CONFIG_FILE_ERROR), \
>> +errmsg("a
Attached patch cleans up the certificate verification in libpq, and adds
a configuration paraqmeter to control it. The new parameter is
"sslverify", and can be set to:
* cn = default = will validate that the certificate chains to a trusted
root, *and* that the cn on the certificate matches the hos
Peter Eisentraut <[EMAIL PROTECTED]> writes:
> SQL:2008 specifies the following syntax for what we have so far called
> LIMIT and OFFSET
> SELECT ... [ ORDER BY ... ]
> OFFSET num {ROW|ROWS} FETCH {FIRST|NEXT} [num] {ROW|ROWS} ONLY
What does the "NEXT" option mean? I'm a bit worried that th
ITAGAKI Takahiro wrote:
> Magnus Hagander <[EMAIL PROTECTED]> wrote:
>
>>> I'd like to submit pg_stat_statements contrib module
>> Sounds very good, but why contrib and not along with the rest of the
>> stats stuff in the main backend (with an on/off switch if the overhead
>> is high)?
>
> That's
I notice our docs have:
If you are at all concerned about password
sniffing attacks then md5 is preferred, with
crypt to be used only if you must support pre-7.2
clients. Plain password should be avoided especially for
At what point do we just remove the support and say that peop
For Hot Standby I need to mark which indexes are usable or not.
There are three aspects to index use during recovery:
* Certain index types may not correctly implement fully concurrent
locking order to allow that index type to be used during recovery.
* Other indexes might become unusable as index
Magnus Hagander <[EMAIL PROTECTED]> wrote:
> > I'd like to submit pg_stat_statements contrib module
>
> Sounds very good, but why contrib and not along with the rest of the
> stats stuff in the main backend (with an on/off switch if the overhead
> is high)?
That's because it could be done as a
Tom Lane wrote:
Another objection to this design is that it's completely unclear that
functions from text to text should necessarily yield the same collation
that went into them, but if you treat collation as a hard-wired part of
the expression syntax tree you aren't going to be able to do anythi
Hi,
Alvaro Herrera wrote:
> So this discussion died with no solution arising to the
> hint-bit-setting-invalidates-the-CRC problem.
Isn't double-buffering solving this issue? Has somebody checked if it
even helps performance due to being able to release the lock on the
buffer *before* the syscall
I'm looking to implement the following functions for Hot Standby, to
allow those with administrative tools or management applications to have
more control during recovery. Please let me know if other functions are
required.
What else do we need?
* pg_is_in_recovery()
returns bool (true if in rec
SQL:2008 specifies the following syntax for what we have so far called
LIMIT and OFFSET
SELECT ... [ ORDER BY ... ]
OFFSET num {ROW|ROWS} FETCH {FIRST|NEXT} [num] {ROW|ROWS} ONLY
For example,
SELECT id, name FROM tab1 ORDER BY id OFFSET 20 ROWS FETCH NEXT 10 ROWS
ONLY;
(I understand thi
Andrew Dunstan wrote:
>
>
> Tom Lane wrote:
>> Andrew Dunstan <[EMAIL PROTECTED]> writes:
>>
>>> OK. Where would be a good place to put the code? Maybe a new file
>>> src/backend/utils/adt/trigger_utils.c ?
>>>
>>
>> I thought the plan was to make it a contrib module.
>>
>>
>>
ITAGAKI Takahiro wrote:
> Hello,
>
> I'd like to submit pg_stat_statements contrib module, that counts up
> incoming statements in shared memory and summarizes the result as a view.
> It is just a statements-version of pg_stat_user_functions.
Sounds very good, but why contrib and not along with t
> Okay. I'll go fix the core code, and you can take out
> whatever you want in GiST/GIN.
Which PostgreSQL versions will contain the fix?
Regards,
Martin Schaefer
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.
Tom Lane wrote:
> Magnus Hagander <[EMAIL PROTECTED]> writes:
>> Here's an ugly attempt towards this. Though I'm unsure if we can change
>> the "const" on the PQerrorMessage parameter without messing with library
>> versions and such?
>
> That's a bad idea in any case --- PQerrorMessage shouldn't
Bruce Momjian wrote:
> OK, a few comments:
>
> This should have spaces:
>
> !port->hba->ldapprefix?port->hba->ldapprefix:"",
Fixed. Though I guess pgindent would've fixed it eventually otherwise.
> This is missing 'do' or something:
>
> + #define MANDATORY_AUTH_ARG(arg
76 matches
Mail list logo
