Re: Toast issues with OldestXmin going backwards

2018-04-28 Thread Andrew Gierth
> "Amit" == Amit Kapila writes: >>> (Or do we need to track it across restarts? maybe we do, to deal with >>> replication slaves without slots, or changes in parameters) >> Yeah, I'm worried that it might need to be persistent across restarts. >> >> One idea that occurred to me is to

Re: Toast issues with OldestXmin going backwards

2018-04-28 Thread Amit Kapila
On Fri, Apr 27, 2018 at 8:33 PM, Robert Haas wrote: > On Thu, Apr 26, 2018 at 9:03 PM, Andrew Gierth > wrote: >>(Or do we need to track it across restarts? maybe we do, to deal with >>replication slaves without slots, or changes in parameters) > > Yeah, I'm worried that it might need to b

GSoC 2018: thrift encoding format

2018-04-28 Thread Charles Cui
Hi Aleksander, Start to study the format of thrift encoding format (binary protocol) and found this document( https://erikvanoosten.github.io/thrift-missing-specification/#_struct_encoding). Having one question. Can I assume the data in thrift is always send inside a struct? Otherwise, it needs

Re: [RFC] Add an until-0 loop in psql

2018-04-28 Thread Pavel Stehule
2018-04-28 21:25 GMT+02:00 Corey Huinker : > >> I did it too. It is not too hard - there was not a agreement on syntax. >> >> can be nice some like CTE >> >> WITH PROCEDURE x(a int, b int) AS ... $$ SELECT x(10); >> > > I've seen "WITH function" syntax in Oracle (here's an example: > https://o

Re: Postgres, fsync, and OSs (specifically linux)

2018-04-28 Thread Craig Ringer
On 28 April 2018 at 23:25, Simon Riggs wrote: > On 27 April 2018 at 15:28, Andres Freund wrote: > >> - Add a pre-checkpoint hook that checks for filesystem errors *after* >> fsyncing all the files, but *before* logging the checkpoint completion >> record. Operating systems, filesystems, etc.

Re: Postgres, fsync, and OSs (specifically linux)

2018-04-28 Thread Craig Ringer
On 29 April 2018 at 00:15, Andres Freund wrote: > Hi, > > On 2018-04-28 08:25:53 -0700, Simon Riggs wrote: >> > - Use direct IO. Due to architectural performance issues in PG and the >> > fact that it'd not be applicable for all installations I don't think >> > this is a reasonable fix for the

Re: Toast issues with OldestXmin going backwards

2018-04-28 Thread Andrew Gierth
> "Stephen" == Stephen Frost writes: Stephen> I tend to agree.. However, this isn't something that's been Stephen> happening a lot, from what I gather, Right now I can't prove it happens in the wild at all - I think I've ruled it out as an explanation for the original problem report - but

Re: community bonding

2018-04-28 Thread Charles Cui
Thanks for your comments, Craig! 2018-04-26 21:03 GMT-07:00 Craig Ringer : > On 24 April 2018 at 12:22, Charles Cui wrote: > > Hi PostgreSQL community and mentors: > > > > Thanks for selecting my project as one of GSoC student projects! Pretty > > exciting and honor to join the development for P

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-28 Thread Peter Geoghegan
On Sat, Apr 28, 2018 at 7:39 AM, Stephen Frost wrote: > Perhaps I'm misunderstanding the case here, but with the ring buffer we > use for sequential access, aren't we already discouraging keeping heap > pages around, particularly when they're coming from a sequential scan? > > I'm not suggesting t

Re: jitflags in _outPlannedStmt and _readPlannedStmt treated as bool type

2018-04-28 Thread Tom Lane
Stephen Frost writes: > On Sat, Apr 28, 2018 at 18:23 Tom Lane wrote: >> Actually, I was about to ask whether you guys had changed anything about >> dory's configuration recently. It's now had eight consecutive passes >> since I put in 63ca350 last night. Given its record before that, just >> a

RE: GSoC 2018: Sorting Algorithm and Benchmarking

2018-04-28 Thread Kefan Yang
Hey, I’ve joined the workspace. The username is Kefan Yang I agree we can create a channel for this project From: Andrey Borodin Sent: April 27, 2018 9:22 PM To: Kefan Yang Cc: PostgreSQL Hackers; Atri Sharma Subject: Re: GSoC 2018: Sorting Algorithm and Benchmarking > 27 апр. 2018 г., в 23:23

Re: jitflags in _outPlannedStmt and _readPlannedStmt treated as bool type

2018-04-28 Thread Stephen Frost
Greetings, On Sat, Apr 28, 2018 at 18:23 Tom Lane wrote: > Stephen Frost writes: > > On Sat, Apr 28, 2018 at 17:55 Andres Freund wrote: > >> Perhaps Steven could change dory's config so it rebuilds when idle? > >> There's a setting for that... > > > Not at a sufficiently capable keyboard at th

Re: jitflags in _outPlannedStmt and _readPlannedStmt treated as bool type

2018-04-28 Thread Tom Lane
Stephen Frost writes: > On Sat, Apr 28, 2018 at 17:55 Andres Freund wrote: >> Perhaps Steven could change dory's config so it rebuilds when idle? >> There's a setting for that... > Not at a sufficiently capable keyboard at the moment to do that, so I’ve > passed the ask on to someone else who ca

Re: jitflags in _outPlannedStmt and _readPlannedStmt treated as bool type

2018-04-28 Thread Stephen Frost
Greetings, On Sat, Apr 28, 2018 at 17:55 Andres Freund wrote: > > > On April 28, 2018 1:47:56 PM PDT, Tom Lane wrote: > >Andres Freund writes: > >> On 2018-04-26 13:34:44 +1000, Haribabu Kommi wrote: > >>> I am thinking of it is a copy paste mistake as the other members > >around the > >>> ini

Re: jitflags in _outPlannedStmt and _readPlannedStmt treated as bool type

2018-04-28 Thread Andres Freund
On April 28, 2018 1:47:56 PM PDT, Tom Lane wrote: >Andres Freund writes: >> On 2018-04-26 13:34:44 +1000, Haribabu Kommi wrote: >>> I am thinking of it is a copy paste mistake as the other members >around the >>> initflags are of "bool" type or is there any specific reason to >treat as >>> "boo

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-28 Thread Thomas Munro
On Sun, Apr 29, 2018 at 2:39 AM, Stephen Frost wrote: > * Peter Geoghegan (p...@bowt.ie) wrote: >> On Thu, Apr 26, 2018 at 10:39 AM, Robert Haas wrote: >> > I'm personally not very excited about making rules like "index pages >> > are more valuable than heap pages". Such rules will in many cases

Re: jitflags in _outPlannedStmt and _readPlannedStmt treated as bool type

2018-04-28 Thread Tom Lane
Andres Freund writes: > On 2018-04-26 13:34:44 +1000, Haribabu Kommi wrote: >> I am thinking of it is a copy paste mistake as the other members around the >> initflags are of "bool" type or is there any specific reason to treat as >> "bool" type? > No, it's an oversight. It was a bool first and t

Re: WIP: a way forward on bootstrap data

2018-04-28 Thread Tom Lane
John Naylor writes: > On 4/26/18, Tom Lane wrote: >> (The prosrc-from-proname case, in isolation, could be handled about >> as well by adding a hardwired rule like we have for pronargs.) > If we think we might go in the direction of more special-case > behavior, the attached patch more fully doc

Re: [RFC] Add an until-0 loop in psql

2018-04-28 Thread Corey Huinker
> > > I did it too. It is not too hard - there was not a agreement on syntax. > > can be nice some like CTE > > WITH PROCEDURE x(a int, b int) AS ... $$ SELECT x(10); > I've seen "WITH function" syntax in Oracle (here's an example: https://oracle-base.com/articles/12c/with-clause-enhancements-

Re: Is a modern build system acceptable for older platforms

2018-04-28 Thread Hartmut Holzgraefe
On 28.04.2018 05:27, Yuriy Zhuravlev wrote: "make distcheck" CMake have no this bad concept, in my opinion, if you want to make the project you should have a full build environment. (but I don't want to argue about it here) this is not about having a working build environment, it is abou

Re: Cold welcoming message when installing anything because of LLVM bitcode stuff

2018-04-28 Thread Tom Lane
Andres Freund writes: > On April 28, 2018 11:05:15 AM PDT, Tom Lane wrote: >> Close inspection of the >> "make install" output revealed a few other issues, which I hope >> I fixed rather than making things worse. > Broke the llvm enabled part of the bf. Will have a look once I'm home (1.5h), >

Re: Cold welcoming message when installing anything because of LLVM bitcode stuff

2018-04-28 Thread Andres Freund
On April 28, 2018 11:05:15 AM PDT, Tom Lane wrote: >Michael Paquier writes: >> JIT is introducing the installation of LLVM bitcode modules, and any >> installation done results in the following comment in output, which >> repeats basically for all the extensions installed: >> # Then install fil

Re: Cold welcoming message when installing anything because of LLVM bitcode stuff

2018-04-28 Thread Tom Lane
Michael Paquier writes: > JIT is introducing the installation of LLVM bitcode modules, and any > installation done results in the following comment in output, which > repeats basically for all the extensions installed: > # Then install files > # > # The many INSTALL_DATA invocations aren't particu

Re: [HACKERS] GSoC 2017: weekly progress reports (week 6)

2018-04-28 Thread Teodor Sigaev
I don't very happy with rootBuffer added everywhere. ginInsertItemPointers() and  ginPrepareDataScan() now take both args, rootBlkno and rootBuffer, second could be invalid. As I can see, you did it to call CheckForSerializableConflictIn() in ginEntryInsert(). Seems, it could be reverted and C

Re: Verbosity of genbki.pl

2018-04-28 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Peter Eisentraut writes: > > Maybe we can take another look at this. Right now, make -s world > > produces *only* the following output: > > > $ make -s world > > Generating BKI files and symbol definition headers... > > Generating fmgrtab.c, f

Re: inconsistency and inefficiency in setup_conversion()

2018-04-28 Thread Tom Lane
John Naylor writes: > Taking a close look at the result of setup_conversion(), wrong or at > least confusing comments are applied to the functions. Ugh. Between that and the large chunk of initdb runtime eaten by setup_conversion(), that seems like plenty of reason to redo it. > Solution #1 - A

Re: GCC 8 warnings

2018-04-28 Thread Andres Freund
Hi, On 2018-04-28 12:16:39 -0400, Tom Lane wrote: > -SharedFilePath(char *path, SharedFileSet *fileset, const char *name) > +SharedFilePath(char *path, size_t size, SharedFileSet *fileset, const char > *name) > { > - chardirpath[MAXPGPATH]; > + chardirpath[MAXPGPA

Re: Postgres, fsync, and OSs (specifically linux)

2018-04-28 Thread Andres Freund
Hi, On 2018-04-28 20:00:25 +0800, Craig Ringer wrote: > On 28 April 2018 at 06:28, Andres Freund wrote: > > The second major type of proposal was using direct-IO. That'd generally > > be a desirable feature, but a) would require some significant changes to > > postgres to be performant, b) isn't

Re: [HACKERS] GSoC 2017: weekly progress reports (week 6)

2018-04-28 Thread Teodor Sigaev
I took a stab at fixing those issues, as well as the bug when fastupdate is turned on concurrently. Does the attached patch look sane to you? That's look sane, and I believe it should be applied but I see some issue in your patch: I don't very happy with rootBuffer added everywhere. ginInsertIt

Re: Verbosity of genbki.pl

2018-04-28 Thread Tom Lane
Peter Eisentraut writes: > Maybe we can take another look at this. Right now, make -s world > produces *only* the following output: > $ make -s world > Generating BKI files and symbol definition headers... > Generating fmgrtab.c, fmgroids.h, and fmgrprotos.h... > PostgreSQL, contrib, and documen

Re: GCC 8 warnings

2018-04-28 Thread Tom Lane
Peter Eisentraut writes: > On 4/24/18 05:57, Devrim Gündüz wrote: >> While building stable releases and v11 on Fedora 28, I am seeing some >> warnings. > Attached is a patch to fix these warnings in master. These are very > similar to the class of warnings we fixed last time around for GCC 7.

Re: Postgres, fsync, and OSs (specifically linux)

2018-04-28 Thread Andres Freund
Hi, On 2018-04-28 08:25:53 -0700, Simon Riggs wrote: > > - Use direct IO. Due to architectural performance issues in PG and the > > fact that it'd not be applicable for all installations I don't think > > this is a reasonable fix for the issue presented here. Although it's > > independently

Re: Postgres, fsync, and OSs (specifically linux)

2018-04-28 Thread Andres Freund
Hi, On 2018-04-28 17:35:48 +0200, Michael Banck wrote: > This dmesg-checking has been mentioned several times now, but IME > enterprise distributions (or server ops teams?) seem to tighten access > to dmesg and /var/log to non-root users, including postgres. > > Well, or just vanilla Debian stabl

Re: Postgres, fsync, and OSs (specifically linux)

2018-04-28 Thread Andres Freund
Hi, On 2018-04-28 11:10:54 -0400, Stephen Frost wrote: > When we crash-restart, we also go through and clean things up some, no? > Seems like that gives us the potential to end up fixing things ourselves > and allowing the crash-restart to succeed. Sure, there's the potential for that. But it's q

Re: [GENERAL] huge RAM use in multi-command ALTER of table heirarchy

2018-04-28 Thread Justin Pryzby
On Tue, Jul 18, 2017 at 07:26:30PM -0400, Tom Lane wrote: > > It's probably a bit late in the v10 cycle to be taking any risks in > > this area, but I'd vote for ripping out RememberToFreeTupleDescAtEOX > > as soon as the v11 cycle opens, unless someone can show an example > > of non-broken coding

inconsistency and inefficiency in setup_conversion()

2018-04-28 Thread John Naylor
Taking a close look at the result of setup_conversion(), wrong or at least confusing comments are applied to the functions. Consider this family of conversions: select conproc, conname from pg_conversion where conproc = 'utf8_to_win'::regproc order by oid; conproc | conname

Re: Fix some trivial issues of the document/comment

2018-04-28 Thread Tom Lane
"ZHUO QL (KDr2)" writes: > I'm learning PGSQL and reading the source, and find few trivial issues in the > docs: LGTM, pushed. Thanks for the patch! regards, tom lane

Re: Postgres, fsync, and OSs (specifically linux)

2018-04-28 Thread Michael Banck
Hi, On Sat, Apr 28, 2018 at 11:21:20AM -0400, Stephen Frost wrote: > * Craig Ringer (cr...@2ndquadrant.com) wrote: > > On 28 April 2018 at 06:28, Andres Freund wrote: > > > - Add a pre-checkpoint hook that checks for filesystem errors *after* > > > fsyncing all the files, but *before* logging t

Re: Verbosity of genbki.pl

2018-04-28 Thread Peter Eisentraut
On 4/9/18 15:02, Tom Lane wrote: > Andres Freund writes: >> On 2018-04-08 13:33:42 -0400, Tom Lane wrote: >>> Traditionally genbki.pl has printed "Writing foo" for every file >>> it writes out. > >>> 2. Print just one message like "Generating postgres.bki and related >>> files", and I guess a sec

Re: Postgres, fsync, and OSs (specifically linux)

2018-04-28 Thread Simon Riggs
On 27 April 2018 at 15:28, Andres Freund wrote: > - Add a pre-checkpoint hook that checks for filesystem errors *after* > fsyncing all the files, but *before* logging the checkpoint completion > record. Operating systems, filesystems, etc. all log the error format > differently, but for lar

Re: Postgres, fsync, and OSs (specifically linux)

2018-04-28 Thread Stephen Frost
Greetings, * Craig Ringer (cr...@2ndquadrant.com) wrote: > On 28 April 2018 at 06:28, Andres Freund wrote: > > - Add a pre-checkpoint hook that checks for filesystem errors *after* > > fsyncing all the files, but *before* logging the checkpoint completion > > record. Operating systems, filesy

Re: Postgres, fsync, and OSs (specifically linux)

2018-04-28 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2018-04-27 19:38:30 -0400, Bruce Momjian wrote: > > On Fri, Apr 27, 2018 at 04:10:43PM -0700, Andres Freund wrote: > > > On 2018-04-27 19:04:47 -0400, Bruce Momjian wrote: > > > > On Fri, Apr 27, 2018 at 03:28:42PM -0700, Andres Freund wr

Re: Patch missing from back branches

2018-04-28 Thread Stephen Frost
Greetings, * Michael Paquier (mich...@paquier.xyz) wrote: > There is dory in the buildfarm which compiles using VS 2015, but it runs > only 9.6 and newer versions. That would be specifically because compiling 9.5 didn't work.. We'd be happy to have dory running on older major versions, if someon

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-28 Thread Stephen Frost
Greetings, * Peter Geoghegan (p...@bowt.ie) wrote: > On Thu, Apr 26, 2018 at 10:39 AM, Robert Haas wrote: > > I'm personally not very excited about making rules like "index pages > > are more valuable than heap pages". Such rules will in many cases be > > true, but it's easy to come up with case

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-28 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Noah Misch writes: > > On Tue, Apr 24, 2018 at 11:37:33AM +1200, Thomas Munro wrote: > >> Maybe try asking what's mapped there with VirtualQueryEx() on failure? > > > +1. An implementation of that: > > https://www.postgresql.org/message-id/201

Re: Toast issues with OldestXmin going backwards

2018-04-28 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Fri, Apr 27, 2018 at 11:35 AM, Andrew Gierth > wrote: > >> "Robert" == Robert Haas writes: > > Robert> One idea that occurred to me is to somehow record -- I guess in > > Robert> pg_class using non-transactional updates -- the la

Re: Is a modern build system acceptable for older platforms

2018-04-28 Thread Craig Ringer
On 27 April 2018 at 15:10, Yuriy Zhuravlev wrote: > 1. You can remove tools/msvc folder because all your build rules will be > universal. (cmake build now have much fewer lines of code) Which is nice, but not actually a major day to day impact. > 2. You can forget about terminal in Windows (for

Re: Postgres, fsync, and OSs (specifically linux)

2018-04-28 Thread Craig Ringer
On 28 April 2018 at 06:28, Andres Freund wrote: > Hi, > > I thought I'd send this separately from [0] as the issue has become more > general than what was mentioned in that thread, and it went off into > various weeds. Thanks very much for going and for the great summary. > - Actually marking fi

Re: WIP: a way forward on bootstrap data

2018-04-28 Thread John Naylor
On 4/26/18, Tom Lane wrote: > (The prosrc-from-proname case, in isolation, could be handled about > as well by adding a hardwired rule like we have for pronargs.) If we think we might go in the direction of more special-case behavior, the attached patch more fully documents what we already do, an

Re: [RFC] Add an until-0 loop in psql

2018-04-28 Thread Pavel Stehule
2018-04-28 7:36 GMT+02:00 Corey Huinker : > > > >>> What you don't see here is that you're using your psql process's >>> available open file handles as a stack, and when you hit that limit psql >>> will fail. If you remove that limit, then you get a bit further before psql >>> segfaults on you. I

Cold welcoming message when installing anything because of LLVM bitcode stuff

2018-04-28 Thread Michael Paquier
Hi all, JIT is introducing the installation of LLVM bitcode modules, and any installation done results in the following comment in output, which repeats basically for all the extensions installed: # Then install files # # The many INSTALL_DATA invocations aren't particularly fast, it'd be # good i

Re: Built-in connection pooling

2018-04-28 Thread Konstantin Knizhnik
On 27.04.2018 23:43, Merlin Moncure wrote: On Fri, Apr 27, 2018 at 11:44 AM, Konstantin Knizhnik wrote: On 27.04.2018 18:33, Merlin Moncure wrote: On Fri, Apr 27, 2018 at 10:05 AM, Konstantin Knizhnik wrote: On 27.04.2018 16:49, Merlin Moncure wrote: I'm confused here...could be language