Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

I tried another experiment, which was "make -j100 all" on my relatively new Linux box (2 dual-core CPUs). It blew up real good, as per attached stderr output, which shows evidence of more missing dependencies as well as some additional cases of concurrent attempts to build the same target. It's c

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

Robert Haas writes: > Very odd, but this completely blew up the first time I tried it. > In file included from path.c:34: > pg_config_paths.h:2:11: error: missing terminating " character FWIW, I didn't replicate that, but I did get this during one attempt with -j4: /usr/bin/ranlib: archive memb

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

On Mon, Nov 15, 2010 at 4:10 PM, Peter Eisentraut wrote: >> ld: file not found: ../../../../../../src/backend/postgres >> collect2: ld returned 1 exit status >> make[3]: *** [ascii_and_mic.so] Error 1 >> make[2]: *** [all-ascii_and_mic-recurse] Error 2 >> make[1]: *** [all-backend/utils/mb/convers

Re: [HACKERS] Count backend self-sync calls

Jeff Janes wrote: Do you know where this competition is happening? Is it on the platters, or is it in the hard drive write cache (I thought high-end hardware had tagged writes to avoid that), or in the kernel? Kernel. Linux systems with lots of memory will happily queue up gigabytes of me

Re: [HACKERS] Explain analyze getrusage tracking

On Sun, Nov 14, 2010 at 9:33 PM, Greg Stark wrote: > This is an update to my earlier patch to add getrusage resource > tracking to EXPLAIN ANALYZE. > > With this patch you get something like: > >                                                  QUERY PLAN >

Re: [HACKERS] Spread checkpoint sync

On Sun, Nov 14, 2010 at 6:48 PM, Greg Smith wrote: > The second issue is that the delay between sync calls is currently > hard-coded, at 3 seconds.  I believe the right path here is to consider the > current checkpoint_completion_target to still be valid, then work back from > there.  That raises

[HACKERS] unlogged tables

I am attempting to test this https://commitfest.postgresql.org/action/patch_view?id=424 but I'm not sure which version of PG this should be applied to. (it would be really neat, on here: https://commitfest.postgresql.org/action/commitfest_view?id=8 if there was a note that said, this test this

Re: [HACKERS] libpq changes for synchronous replication

On Mon, Nov 15, 2010 at 7:26 PM, Tom Lane wrote: > Robert Haas writes: >> On Mon, Sep 20, 2010 at 12:17 PM, Tom Lane wrote: >>> Personally I think this demonstrates that piggybacking replication >>> data transfer on the COPY protocol was a bad design to start with. >>> It's probably time to spli

[HACKERS] Isn't HANDLE 64 bits on Win64?

... and if so, isn't postmaster.c's code to transfer a HANDLE value to a child process all wet? sprintf(paramHandleStr, "%lu", (DWORD) paramHandle); ... paramHandle = (HANDLE) atol(id); BTW, it seems like it'd be a good thing if we had a Win64 machine in the buildfarm.

Re: [HACKERS] libpq changes for synchronous replication

Robert Haas writes: > On Mon, Sep 20, 2010 at 12:17 PM, Tom Lane wrote: >> Personally I think this demonstrates that piggybacking replication >> data transfer on the COPY protocol was a bad design to start with. >> It's probably time to split them apart. > This appears to be the only obvious unr

Re: [HACKERS] Fix for seg picksplit function

On Mon, Nov 15, 2010 at 4:06 AM, Alexander Korotkov wrote: > With help of Oleg I found, that line "*left = *right = FirstOffsetNumber;" > was needed only for 7.X compatibility, and it isn't needed any more. > Also, I've replaced "i - 1" by "i - FirstOffsetNumber" in array filling. > I believe it's

Re: [HACKERS] libpq changes for synchronous replication

On Mon, Sep 20, 2010 at 12:17 PM, Tom Lane wrote: > Personally I think this demonstrates that piggybacking replication > data transfer on the COPY protocol was a bad design to start with. > It's probably time to split them apart. This appears to be the only obvious unresolved issue regarding this

Re: [HACKERS] How to rename each field in ROW expression?

On Tue, Nov 16, 2010 at 00:18, Tom Lane wrote: > Why?  It's an anonymous type, you shouldn't care about names. > If you do, make a real named rowtype. If so, we cannot extract any fields in an anonymous type, right? We cannot lookup fields with (an anonymous type).name because the fields have no

Re: [HACKERS] Per-column collation

Hello 2010/11/15 Peter Eisentraut : > On mån, 2010-11-15 at 11:34 +0100, Pavel Stehule wrote: >> I am checking a patch. I found a problem with initdb > > Ah, late night brain farts, it appears.  Here is a corrected version. > > yes, it's ok now. I see still a few issues: a) default encoding for

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

On mån, 2010-11-15 at 11:13 +0100, Bernd Helmle wrote: > > --On 14. November 2010 11:08:13 -0500 Robert Haas > wrote: > > > +1. The current master branch fails to build on my (rather new) Mac with > > make -j2. I could upgrade my toolchain but it seems like more trouble > > than it's worth, n

Re: [HACKERS] Count backend self-sync calls

On Sun, Nov 14, 2010 at 2:07 PM, Greg Smith wrote: > The attached patch adds a new field to pg_stat_bgwriter, counting the number > of times backends execute their own fsync calls.  Normally, when a backend > needs to fsync data, it passes a request to the background writer, which > then absorbs t

Re: [HACKERS] Instrument checkpoint sync calls

On Mon, Nov 15, 2010 at 2:48 PM, Tom Lane wrote: > Robert Haas writes: >> I would be very surprised if we can find a system where gettimeofday() >> takes a significant amount of time compared with fsync().  It might be >> (probably is) too expensive to stick into code paths that are heavily >> CP

Re: [HACKERS] Instrument checkpoint sync calls

So my task list is: 0) Rebase against the HEAD that just code related to this touched today 1) Assume that log_checkpoints is sufficient control over whether the timing overhead added is worth collecting, and therefore remove the half-baked idea of also wrapping with a compile-time option. 2

Re: [HACKERS] Instrument checkpoint sync calls

Robert Haas writes: > I would be very surprised if we can find a system where gettimeofday() > takes a significant amount of time compared with fsync(). It might be > (probably is) too expensive to stick into code paths that are heavily > CPU-bounded, but surely the cost here is going to be dwarf

Re: [HACKERS] Re: Rethinking hint bits WAS: Protecting against unexpected zero-pages: proposal

Robert Haas writes: > On Mon, Nov 15, 2010 at 2:06 PM, Josh Berkus wrote: >>> I'm not ignoring them; I just choose to work on other issues, since >>> there is no viable proposal for fixing them.  I don't intend to put >>> my time into dead ends. >> So, that's a "show me a patch and we'll talk"?

Re: [HACKERS] MULTISET and additional functions for ARRAY

David Fetter writes: > On Mon, Nov 15, 2010 at 10:13:40AM -0500, Tom Lane wrote: >> Itagaki Takahiro writes: >>> Another issue for sorting is that we have 4 kinds of sorting: ASC/DESC >>> and NULLS FIRST/LAST. >> We have a lot more kinds than that. See USING. > USING pretty much gives us no ch

Re: [HACKERS] Instrument checkpoint sync calls

On Sun, Nov 14, 2010 at 7:04 PM, Greg Smith wrote: > It might.  One trade-off is that if you're looking at the sync write detail, > the summary comes out in a similar form.  And it was easy to put in > here--I'd have to return some new data out of the sync phase call in order > for that to show up

Re: [HACKERS] Bug in plpython's Python Generators

=?UTF-8?B?SmFuIFVyYmHFhHNraQ==?= writes: > On 24/10/10 00:32, Jan Urbański wrote: >> On 21/10/10 20:48, Alvaro Herrera wrote: >>> ... and presumably somebody can fix the real bug that Jean-Baptiste hit, >>> too. >> AFAICS the error comes from PLy_function_handler disconnecting from SPI >> after

Re: [HACKERS] Re: Rethinking hint bits WAS: Protecting against unexpected zero-pages: proposal

On Mon, Nov 15, 2010 at 2:06 PM, Josh Berkus wrote: >> I'm not ignoring them; I just choose to work on other issues, since >> there is no viable proposal for fixing them.  I don't intend to put >> my time into dead ends. > > So, that's a "show me a patch and we'll talk"?  Understood, then. Or eve

Re: [HACKERS] MULTISET and additional functions for ARRAY

On Mon, Nov 15, 2010 at 10:13:40AM -0500, Tom Lane wrote: > Itagaki Takahiro writes: > > Another issue for sorting is that we have 4 kinds of sorting: ASC/DESC > > and NULLS FIRST/LAST. > > We have a lot more kinds than that. See USING. USING pretty much gives us no chance of optimizing at all.

Re: [HACKERS] Re: Rethinking hint bits WAS: Protecting against unexpected zero-pages: proposal

> I'm not ignoring them; I just choose to work on other issues, since > there is no viable proposal for fixing them. I don't intend to put > my time into dead ends. So, that's a "show me a patch and we'll talk"? Understood, then. -- -- Josh Berkus

Re: [HACKERS] Re: Rethinking hint bits WAS: Protecting against unexpected zero-pages: proposal

Josh Berkus writes: > No, I'm not. I'm pointing out that it doesn't unblock the other 4 > features/improvements I mentioned, *all* of which would be unblocked by > not storing the hint bits in the table, whatever means we use to do so. > You, for your part, are consistently ignoring these other

Re: [HACKERS] Re: Rethinking hint bits WAS: Protecting against unexpected zero-pages: proposal

Greg, Tom, > We *already* have separate bitmap outside the table for transaction > commit bits. It's the clog. You didn't read my whole e-mail. I talk about the CLOG further down. > Josh is ignoring the proposal that is on the table and seems actually > workable, which is to consult the visibil

Re: [HACKERS] Count backend self-sync calls

On Sun, Nov 14, 2010 at 8:31 PM, Robert Haas wrote: > On Sun, Nov 14, 2010 at 7:19 PM, Greg Smith wrote: >>> But if this is generating a lot of log data or adding a lot of >>> overhead, then you have bigger problems anyway: >>> >>> +               elog(DEBUG1, "Unable to forward fsync request, ex

Re: [HACKERS] unlogged tables

On Mon, Nov 15, 2010 at 11:22 AM, Robert Haas wrote: > Yeah, this infrastructure doesn't really allow that.  The truncate > happens way too early on in startup to execute any user-provided code. But you could use the very feature of unlogged tables to know if you've "initialized" some unlogged t

Re: [HACKERS] unlogged tables

On Mon, Nov 15, 2010 at 12:02 PM, Marti Raudsepp wrote: > On Mon, Nov 15, 2010 at 18:25, Tom Lane wrote: >> Marti Raudsepp writes: >>> Would there be a problem with invoking this trigger from the session >>> that first touches the table? >> >> Other than security? > > Right, I guess that would o

Re: [HACKERS] unlogged tables

On Mon, Nov 15, 2010 at 18:25, Tom Lane wrote: > Marti Raudsepp writes: >> Would there be a problem with invoking this trigger from the session >> that first touches the table? > > Other than security? Right, I guess that would only make sense with SECURITY DEFINER. On Mon, Nov 15, 2010 at 18:2

Re: [HACKERS] SQL/MED estimated time of arrival?

With Informix VTI, indexing is the same for native tables as for virtual tables, except the interpretation of the 32 bit rowid is left up to the developer. When you define the VTI class, you optionally supply a method that can fetch data based on a 32 bit rowid, and it's the responsibility of y

Re: [HACKERS] unlogged tables

Marti Raudsepp writes: > Would there be a problem with invoking this trigger from the session > that first touches the table? Other than security? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscriptio

Re: [HACKERS] unlogged tables

On Mon, Nov 15, 2010 at 10:54 AM, Marti Raudsepp wrote: > On Sun, Nov 14, 2010 at 02:16, Robert Haas wrote: >> Here is a series of three patches related to unlogged tables. > > Just wondering, have you thought of any mechanisms how application > code might detect that an unlogged table was trunca

Re: [HACKERS] [PATCH] Custom code int(32|64) => text conversions out of performance reasons

On Monday 15 November 2010 17:12:25 Robert Haas wrote: > It would speed things up for me if you or someone else could take a > quick pass over what remains here and fix the formatting and > whitespace to be consistent with our general project style, and make > the comment headers more consistent am

Re: [HACKERS] SSI update

Alvaro Herrera wrote: > I remind you, though, that the most painful point that caused the > commitfest process to come into existance was the HOT patch, which > was very large and intrusive and didn't get any review until very > late in the cycle. So getting some review earlier than at the > la

Re: [HACKERS] Amazon now supporting GPU focused EC2 instances

On 2010-11-15 18:49, Greg Stark wrote: I've seen papers on doing relational joins using GPUs and I'm sure there are other algorithms we wonderful stuff we could do. But if it comes at the cost of being able to handle arbitrary join clauses it'll be a tough sacrifice to make. Perhaps the cooles

Re: [HACKERS] SSI update

On Mon, Nov 15, 2010 at 10:56 AM, Kevin Grittner wrote: > Greg Smith wrote: > >> I could use a brief reminder of how this bit fits into the >> "serializable lock consistency" patch that's already sitting into >> the CF queue as "Ready for Committer" though. > > [explanation] The short version is

Re: [HACKERS] [PATCH] Custom code int(32|64) => text conversions out of performance reasons

On Sun, Oct 31, 2010 at 5:41 PM, Andres Freund wrote: > While at it: These words always make me a bit frightened when reviewing a patch, since it's generally simpler if a single patch only does one thing. However, in this case... > * I remove the outdated > -- NOTE: int[24] operators never check

Re: Latches with weak memory ordering (Re: [HACKERS] max_wal_senders must die)

Heikki Linnakangas writes: > In SetLatch, is it enough to add the SpinLockAcquire() call *after* > checking that is_set is not already set? Ie. still do the quick exit > without holding a lock. Or do we need a memory barrier operation before > the fetch, to ensure that we see if the other proce

[HACKERS] Extensions

Hi, I've been trying to summarise the last month of work into a single wiki page, to ease approaching the review of this now quite large patch. It touches lots of things but the design and goal are simple enough to fit in my mind, that should be no problem for you guys here :) http://wiki.postg

Re: [HACKERS] SSI update

Greg Smith wrote: > I could use a brief reminder of how this bit fits into the > "serializable lock consistency" patch that's already sitting into > the CF queue as "Ready for Committer" though. Florian's patch prevents an "integrity inversion" that PostgreSQL has had for ages, where attemptin

Re: [HACKERS] unlogged tables

On Sun, Nov 14, 2010 at 02:16, Robert Haas wrote: > Here is a series of three patches related to unlogged tables. Just wondering, have you thought of any mechanisms how application code might detect that an unlogged table was truncated due to restart? While polling with something like "SELECT 1 F

Re: [HACKERS] SSI update

Excerpts from Greg Smith's message of lun nov 15 12:32:25 -0300 2010: > I could use a brief reminder of how this bit fits into the "serializable > lock consistency" patch that's already sitting into the CF queue as > "Ready for Committer" though. I remind you, though, that the most painful poin

Re: [HACKERS] SSI update

Kevin Grittner wrote: That went in an hour and a half before the CF deadline, but I just didn't feel right putting it into the CF in that shape. Then keep on working on it and we can revisit its state when you're happy with it. The purpose of the CommitFest cut-off is not to block work on lo

Re: Latches with weak memory ordering (Re: [HACKERS] max_wal_senders must die)

On 15.11.2010 16:51, Tom Lane wrote: Heikki Linnakangas writes: I believe it's safe to assume that two operations using a volatile pointer will not be rearranged wrt. each other. This is entirely wrong, so far as cross-processor visibility of changes is concerned. Ok. In SetLatch, is it en

Re: [HACKERS] How to rename each field in ROW expression?

Itagaki Takahiro writes: > Is it possible to change names fields in ROW? > We can use CREATE TYPE AS on ahead, but I'd like to > change names of ROW expression in ad-hoc queries. Why? It's an anonymous type, you shouldn't care about names. If you do, make a real named rowtype.

Re: [HACKERS] MULTISET and additional functions for ARRAY

Itagaki Takahiro writes: > Another issue for sorting is that we have 4 kinds of sorting: ASC/DESC > and NULLS FIRST/LAST. We have a lot more kinds than that. See USING. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make chan

Re: Latches with weak memory ordering (Re: [HACKERS] max_wal_senders must die)

On Mon, Nov 15, 2010 at 9:51 AM, Tom Lane wrote: > Heikki Linnakangas writes: Hmm, SetLatch only sets one flag, so I don't see how it could malfunction all by itself. And I would've thought that declaring the Latch variable "volatile" prevents rearrangements. >>> >>> It's not a que

Re: [HACKERS] SSI update

David Fetter wrote: > That it's not ready for commit this minute does not mean that it > shouldn't be in the CF this month. Delaying the first review of > the patch until the next CF pretty much ensures that we'll miss > 9.1 with it, so please add to the current CF :) Well, I had a version wh

Re: Latches with weak memory ordering (Re: [HACKERS] max_wal_senders must die)

Heikki Linnakangas writes: >>> Hmm, SetLatch only sets one flag, so I don't see how it could malfunction >>> all by itself. And I would've thought that declaring the Latch variable >>> "volatile" prevents rearrangements. >> >> It's not a question of code rearrangement. Precisely. "volatile" pre

Re: [HACKERS] duplicate connection failure messages

Excerpts from Bruce Momjian's message of sáb nov 13 22:36:31 -0300 2010: > OK, I found out how to get the IP address with the attached patch. The > problem is that only pghost is set, never pghostaddr. I am not even > sure how that would get set for this code because my tests show it is > not:

Re: [HACKERS] 8.4-vintage problem in postmaster.c

Excerpts from Tom Lane's message of sáb nov 13 19:07:50 -0300 2010: > Stefan Kaltenbrunner writes: > > On 11/13/2010 06:58 PM, Tom Lane wrote: > >> Just looking at it, I think that the logic in canAcceptConnections got > >> broken by somebody in 8.4, and then broken some more in 9.0: in some > >>

Re: Latches with weak memory ordering (Re: [HACKERS] max_wal_senders must die)

On Mon, Nov 15, 2010 at 8:45 AM, Heikki Linnakangas wrote: >> It's not a question of code rearrangement. > > Rearrangement of code, rearrangement of CPU instructions, or rearrangement > of the order the changes in the memory become visible to other processes. > The end result is the same. I'll le

Re: Latches with weak memory ordering (Re: [HACKERS] max_wal_senders must die)

On 15.11.2010 15:22, Robert Haas wrote: On Mon, Nov 15, 2010 at 2:15 AM, Heikki Linnakangas wrote: Can you elaborate? Weak memory ordering means that stores into shared memory initiated by one processor are not guaranteed to be observed to occur in the same sequence by another processor. Th

Re: Latches with weak memory ordering (Re: [HACKERS] max_wal_senders must die)

On Mon, Nov 15, 2010 at 2:15 AM, Heikki Linnakangas wrote: >>> Can you elaborate? >> >> Weak memory ordering means that stores into shared memory initiated by >> one processor are not guaranteed to be observed to occur in the same >> sequence by another processor.  This implies first that the latc

Re: [HACKERS] changing MyDatabaseId

On Mon, Nov 15, 2010 at 12:41 AM, Tom Lane wrote: > Robert Haas writes: >> Looking through the code, it appears to me that we'd need to do the >> following (not necessarily in this order): > > Don't forget > > 9. Unload loadable modules that do not exist according to the new > database's catalogs

Re: [HACKERS] changing MyDatabaseId

Excerpts from Tom Lane's message of lun nov 15 02:41:40 -0300 2010: > I believe also that there are probably race conditions in several of > the steps you listed; in particular there is certainly a risk involved > in changing the database-we-advertise-being-connected-to versus a > concurrent DROP

Re: [HACKERS] Amazon now supporting GPU focused EC2 instances

On 2010-11-15 12:37, Thom Brown wrote: On 15 November 2010 11:26, Greg Stark > wrote: I keep wondering if there's a role for GPUs in Postgres and haven't figure out how to integrate them yet but the day when we'll be expected to exploit them seems to be getting

Re: [HACKERS] Amazon now supporting GPU focused EC2 instances

On Mon, Nov 15, 2010 at 11:37 AM, Thom Brown wrote: > Is this somewhere OpenCL is an option? > Sure. Personally I wonder whether the context switching is fast enough to handle a multi-user system. In the past graphics cards have always been targeted at the idea of a single-user workstation playi

Re: [HACKERS] Amazon now supporting GPU focused EC2 instances

On 15 November 2010 11:26, Greg Stark wrote: > I keep wondering if there's a role for GPUs in Postgres and haven't > figure out how to integrate them yet but the day when we'll be > expected to exploit them seems to be getting nearer: > > > http://aws.typepad.com/aws/2010/11/new-ec2-instance-type

[HACKERS] Amazon now supporting GPU focused EC2 instances

I keep wondering if there's a role for GPUs in Postgres and haven't figure out how to integrate them yet but the day when we'll be expected to exploit them seems to be getting nearer: http://aws.typepad.com/aws/2010/11/new-ec2-instance-type-the-cluster-gpu-instance.html -- greg -- Sent via pgs

Re: [HACKERS] Per-column collation

Hello I am checking a patch. I found a problem with initdb [postg...@pavel-stehule postgresql]$ /usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data/ could not change directory to "/home/pavel/src/postgresql" The files belonging to this database system will be owned by user "postgres". This user

[HACKERS] How to rename each field in ROW expression?

We can rename each field in VALUES clause: =# SELECT * FROM (VALUES(123, 'ABC', NULL)) AS t(a,b,c); a | b | c -+-+--- 123 | ABC | (1 row) But I cannot find ways for ROW expression to do the same thing: =# SELECT ROW(123, 'ABC', NULL) AS (a,b,c); ERROR: syntax error at or near "("

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

--On 14. November 2010 11:08:13 -0500 Robert Haas wrote: +1. The current master branch fails to build on my (rather new) Mac with make -j2. I could upgrade my toolchain but it seems like more trouble than it's worth, not to mention a possible obstacle to new users and developers. The sa

Re: [HACKERS] MULTISET and additional functions for ARRAY

On Monday 15 November 2010 09:47:15 Itagaki Takahiro wrote: > On Mon, Nov 15, 2010 at 14:37, Pavel Stehule wrote: > > Maybe can be useful to implement a searching on sorted array. > > You can hold a flag if multiset is sorted or not. > > Are you suggesting to add an IS_SORTED bit to for each Arr

Re: [HACKERS] MULTISET and additional functions for ARRAY

2010/11/15 Itagaki Takahiro : > On Mon, Nov 15, 2010 at 14:37, Pavel Stehule wrote: >> Maybe can be useful to implement a searching on sorted array. >> You can hold a flag if multiset is sorted or not. > > Are you suggesting to add an IS_SORTED bit to for each ArrayType value? > It might be possib

Re: [HACKERS] Fix for seg picksplit function

With help of Oleg I found, that line "*left = *right = FirstOffsetNumber;" was needed only for 7.X compatibility, and it isn't needed any more. Also, I've replaced "i - 1" by "i - FirstOffsetNumber" in array filling. I believe it's more correct way, because it'll work correctly in the case when Fir

Re: [HACKERS] MULTISET and additional functions for ARRAY

On Mon, Nov 15, 2010 at 14:37, Pavel Stehule wrote: > Maybe can be useful to implement a searching on sorted array. > You can hold a flag if multiset is sorted or not. Are you suggesting to add an IS_SORTED bit to for each ArrayType value? It might be possible, but I'm not sure how much it is use

[HACKERS] some notes about implementation of SQL/PSM in PostgreSQL

Hello I try to implement SQL/PSM in PostgreSQL. I experimented with rewriting a plpgsql - you can check this implementation http://postgres.cz/index.php/SQL/PSM_Manual. I though so we can have some common PL stack. But I don't think it now. It means a some more uglification of current code or ugl