Re: Internal key management system

2020-06-11 Thread Fabien COELHO
Hello Bruce, Sorry for the length (yet again) of this answer, I'm trying to make my point as clear as possible. Obviously it requires some more thinking and design, but my point is that postgres should not hold a KEK, ever, nor presume how DEK are to be managed by a DMS, and that is not ver

Re: Internal key management system

2020-06-11 Thread Fabien COELHO
Hello Masahiko-san, If the KEK is ever present in pg process, it presumes that the threat model being addressed allows its loss if the process is compromised, i.e. all (past, present, future) security properties are void once the process is compromised. Why we should not put KEK in pg proces

Re: Internal key management system

2020-06-12 Thread Fabien COELHO
Hello Masahiko-san, I'm not sure I understood your concern. I try to answer below. If I understand your idea correctly we put both DEK and KEK "elsewhere", and a postgres process gets only DEK from it. Yes, that is one of the option. It seems to me this idea assumes that the place storing

Re: Internal key management system

2020-06-12 Thread Fabien COELHO
Hello Masahiko-san, Summarizing the discussed points so far, I think that the major advantage points of your idea comparing to the current patch's architecture are: * More secure. Because it never loads KEK in postgres processes we can lower the likelihood of KEK leakage. Yes. * More exten

Re: Internal key management system

2020-06-12 Thread Fabien COELHO
Hello Bruce. The question is what should be put in the protocol, and I would tend to think that some careful design time should be put in it. I still don't see the value of this vs. its complexity. Dunno. I'm looking for the value of having such a thing in core. ISTM that there are no cle

Re: extensible options syntax for replication parser?

2020-06-14 Thread Fabien COELHO
Hello Robert, My 0.02 €: It seems to me that we're making the same mistake with the replication parser that we've made in various placesin the regular parser: using a syntax for options that requires that every potential option be a keyword, and every potential option requires modification of

Re: Recording test runtimes with the buildfarm

2020-06-14 Thread Fabien COELHO
Hello Tom, I have in the past scraped the latter results and tried to make sense of them. They are *mighty* noisy, even when considering just one animal that I know to be running on a machine with little else to do. Maybe averaging across the whole buildfarm could reduce the noise level, b

Re: Internal key management system

2020-06-14 Thread Fabien COELHO
Hello Masahiko-san, * The external place needs to manage more encryption keys than the current patch does. Why? If the external place is just a separate process on the same host, probably it would manage the very same amount as what your patch. In the current patch, the external place need

Re: [PATCH] Missing links between system catalog documentation pages

2020-06-15 Thread Fabien COELHO
Hello Dagfinn, The attached patch makes the first mention of another system catalog or view (as well as pg_hba.conf in pg_hba_file_lines) a link, for easier navigation. Why only the first mention? It seems unlikely that I would ever read such chapter linearly, and even so that I would want

Re: Internal key management system

2020-06-18 Thread Fabien COELHO
Hello Masahiko-san, What I referred to "only one key" is KEK. Ok, sorry, I misunderstood. Yeah, it depends on KMS, meaning we need different extensions for different KMS. A KMS might support an interface that creates key if not exist during GET but some KMS might support CREATE and GET sepa

Re: update substring pattern matching syntax

2020-06-20 Thread Fabien COELHO
Hello Peter, whereas the current standard says SUBSTRING(text SIMILAR pattern ESCAPE escapechar) The former was in SQL99, but the latter has been there since SQL:2003. It's pretty easy to implement the second form also, so here is a patch that does that. Patches apply cleanly, compile

Re: [PATCH] Missing links between system catalog documentation pages

2020-06-21 Thread Fabien COELHO
Hello Dagfinn, The attached patch applies cleanly, doc generation is ok. I'm ok with adding such links systematically. makes the first mention of another system catalog or view (as well as pg_hba.conf in pg_hba_file_lines) a link, for easier navigation. Why only the first mention? It se

tag typos in "catalog.sgml"

2020-06-21 Thread Fabien COELHO
Hello, While reviewing a documentation patch, I noticed that a few tags where wrong in "catalog.sgml". Attached patch fixes them. -- Fabien.diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 700271fd40..5a66115df1 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/s

Missing "Up" navigation link between parts and doc root?

2020-06-21 Thread Fabien COELHO
Hello devs, I've been annoyed that the documentation navigation does not always has an "Up" link. It has them inside parts, but the link disappears and you have to go for the "Home" link which is far on the right when on the root page of a part? Is there a good reason not to have the "Up"

Re: [PATCH] Missing links between system catalog documentation pages

2020-06-21 Thread Fabien COELHO
Hello Tom, I didn't think there was much point in linkifying both in that case, and other similar situations. The point is that the user reads a sentence, attempts to jump but sometimes can't, because the is not the first occurrence. I'd go for all mentions of another relation should be li

Re: tag typos in "catalog.sgml"

2020-06-22 Thread Fabien COELHO
Good catches, thanks Fabien. I will fix that tomorrow or so. And applied to HEAD. Ok. Should it be backpatched? I'm not sure what the usual practice is wrt to small fixes in the doc. -- Fabien.

Re: tag typos in "catalog.sgml"

2020-06-22 Thread Fabien COELHO
Bonjour Michaël, Should it be backpatched? I'm not sure what the usual practice is wrt to small fixes in the doc. The text is right, and this impacts only the appearance of the text, so I did not see that this was enough for a backpatch. Ok. It would mean that possible other doc patches on

Why forbid "INSERT INTO t () VALUES ();"

2020-06-24 Thread Fabien COELHO
Hello devs, I would like to create an "all defaults" row, i.e. a row composed of the default values for all attributes, so I wrote: INSERT INTO t() VALUES (); This is forbidden by postgres, and also sqlite. Is there any good reason why this should be the case? -- Fabien.

Re: Why forbid "INSERT INTO t () VALUES ();"

2020-06-24 Thread Fabien COELHO
Hallo Thomas,   INSERT INTO t() VALUES (); This is forbidden by postgres, and also sqlite. Is there any good reason why this should be the case? Maybe because insert into t default values; exists (and is standard SQL if I'm not mistaken) That's a nice alternative I did not notice. Wel

Re: Why forbid "INSERT INTO t () VALUES ();"

2020-06-24 Thread Fabien COELHO
Hello Tom, INSERT INTO t() VALUES (); I'm still unclear why it would be forbidden though, it seems logical to try that, whereas the working one is quite away from the usual syntax. It's forbidden because the SQL standard forbids it. Ok, that is definitely a reason. I'm not sure it is

Re: some more pg_dump refactoring

2020-06-24 Thread Fabien COELHO
Hallo Peter, My 0.02 €: Patch applies cleanly, compiles, make check and pg_dump tap tests ok. The refactoring is a definite improvements. You changed the query strings to use "\n" instead of " ". I would not have changed that, because it departs from the style around, and I do not think it

Re: Why forbid "INSERT INTO t () VALUES ();"

2020-06-25 Thread Fabien COELHO
Bonjour Vik, It's forbidden because the SQL standard forbids it. Ok, that is definitely a reason. I'm not sure it is a good reason, though. It's a very good reason. It might not be good *enough*, but it is a good reason. Ok for good, although paradoxically not "good enough":-) Why wou

Re: Why forbid "INSERT INTO t () VALUES ();"

2020-06-26 Thread Fabien COELHO
Hello Isaac, This is not the only area where empty tuples are not supported. Consider: PRIMARY KEY () This should mean the table may only contain a single row, but is not supported. Yep. This is exactly the kind of case about which I was trying the command, after reading Bruce Momjian blo

Re: update substring pattern matching syntax

2020-06-27 Thread Fabien COELHO
Hallo Peter, v2 patches apply cleanly, compile, global check ok, citext check ok, doc gen ok. No further comments. As I did not find an entry in the CF, so I did nothing about tagging it "ready". -- Fabien.

Re: pgbench and timestamps (bounced)

2020-06-29 Thread Fabien COELHO
[Resent on hackers for CF registration, sorry for the noise] Hello Tom, The attached patch fixes some of the underlying problems reported by delaying the :var to $1 substitution to the last possible moments, so that what variables are actually defined is known. PREPARE-ing is also delayed to

Re: some more pg_dump refactoring

2020-06-30 Thread Fabien COELHO
Hello, You changed the query strings to use "\n" instead of " ". I would not have changed that, because it departs from the style around, and I do not think it improves readability at the C code level. This was the style that was introduced by daa9fe8a5264a3f192efa5ddee8fb011ad9da365. Yep

Re: pgbench: option delaying queries till connections establishment?

2020-07-03 Thread Fabien COELHO
* First patch reworks time measurements in pgbench. * Second patch adds a barrier before starting the bench It applies on top of the previous one. The initial imbalance due to thread creation times is smoothed. The usecs patch fails to apply to HEAD, can you please submit a rebased version o

Re: Missing "Up" navigation link between parts and doc root?

2020-07-03 Thread Fabien COELHO
The original stylesheets explicitly go out of their way to do it that way. Can we find any evidence of the reasoning? As you say, that clearly was an intentional choice. Given the code, my guess would be the well-intentioned but misplaced desire to avoid a redundancy, i.e. two links side-

Re: Missing "Up" navigation link between parts and doc root?

2020-07-03 Thread Fabien COELHO
Hello Peter, The original stylesheets explicitly go out of their way to do it that way. We can easily fix that by removing that special case. See attached patch. That patch only fixes it for the header. To fix it for the footer as well, we'd first need to import the navfooter template to

Re: Allow continuations in "pg_hba.conf" files

2020-07-06 Thread Fabien COELHO
In the attached v3, I've tried to clarify comments and doc about tokenization rules relating to comments, strings and continuations. Attached v4 improves comments & doc as suggested by Justin. -- Fabien.diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index 5cd88b462

Re: some more pg_dump refactoring

2020-07-07 Thread Fabien COELHO
Hallo Peter, Would it make sense to accumulate in the other direction, older to newer, so that new attributes are added at the end of the select? I think that could make sense, but the current style was introduced by daa9fe8a5264a3f192efa5ddee8fb011ad9da365. Should we revise that? It seems

Re: Proposal: Automatic partition creation

2020-07-07 Thread Fabien COELHO
Hello Anastasia, My 0.02 €: The patch implements following syntax: CREATE TABLE ... PARTITION BY partition_method (list_of_columns) partition_auto_create_clause where partition_auto_create_clause is CONFIGURATION [IMMEDIATE| DEFERRED] USING partition_bound_spec and partition_bound_spec is:

Re: pgbench and timestamps (bounced)

2020-07-08 Thread Fabien COELHO
Attached v2 fixes some errors, per cfbot. -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml index 9f3bb5fce6..9894ae9c04 100644 --- a/doc/src/sgml/ref/pgbench.sgml +++ b/doc/src/sgml/ref/pgbench.sgml @@ -998,15 +998,14 @@ pgbench options d There is a

Re: pgbench - add pseudo-random permutation function

2020-07-08 Thread Fabien COELHO
I don't think we should boot this patch. I don't think I would be able to get this over the commit line in this CF, but let's not discard it. Understood. I have moved the patch to the 2020-07 CF in Needs Review state. Attached v19 is a rebase, per cfbot. -- Fabien.diff --git a/doc/src/sgml

Re: pgbench - refactor init functions with buffers

2020-07-09 Thread Fabien COELHO
in favor of *PQExpBuffer(). Attached v7 is rebased v5 which uses PQExpBuffer, per cfbot. -- Fabien.diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c index 08a5947a9e..3abc41954a 100644 --- a/src/bin/pgbench/pgbench.c +++ b/src/bin/pgbench/pgbench.c @@ -602,7 +602,9 @@ static

Re: pgbench - add pseudo-random permutation function

2020-07-10 Thread Fabien COELHO
Attached v19 is a rebase, per cfbot. Attached v20 fixes a doc xml typo, per cfbot again. -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml index 9f3bb5fce6..d4a604c6fa 100644 --- a/doc/src/sgml/ref/pgbench.sgml +++ b/doc/src/sgml/ref/pgbench.sgml @@ -1033,7

Re: psql - add SHOW_ALL_RESULTS option

2020-07-19 Thread Fabien COELHO
In the meantime, here is a v9 which also fixes the behavior when using \watch, so that now one can issue several \;-separated queries and have their progress shown. I just needed that a few days ago and was disappointed but unsurprised that it did not work. This seems to break the 013_crash_r

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2020-04-12 Thread Fabien COELHO
Hello Justin, About v15, seen as one patch. Patches serie applies cleanly, compiles, "make check" ok. Documentation: - indent documentation text around 80 cols, as done around? - indent SQL example for readability and capitalize keywords (pg_ls_dir_metadata) - "For each file in a direct

Re: pgbench - test whether a variable exists

2020-04-13 Thread Fabien COELHO
Bonjour Michaël, Patch v3 is also a rebase. This has rotten for half a year, so I am marking it as returned with feedback. There have been comments from Alvaro and Andres as well... Attached a v4. I'm resurrecting this small patch, after "\aset" has been added to pgbench (9d8ef988). Alv

Re: Poll: are people okay with function/operator table redesign?

2020-04-13 Thread Fabien COELHO
Hello Tom, Before I spend more time on this, I want to make sure that people are happy with this line of attack. +1 I like it this way, because the structure is quite readable, which is the point. My 0.02€: Maybe column heander "Example Result" should be simply "Result", because it is a

Re: pgbench - test whether a variable exists

2020-04-13 Thread Fabien COELHO
Bonjour Michaël, Hmm. It seems to me that this stuff needs to be more careful with the function handling? For example, all those cases fail but they directly pass down a variable that may not be defined, so shouldn't those results be undefined as well instead of failing: \set g double(:{

Re: Proposing WITH ITERATIVE

2020-04-28 Thread Fabien COELHO
Hello Jonah, Nice. -- No ORDER/LIMIT is required with ITERATIVE as only a single tuple is present EXPLAIN ANALYZE WITH ITERATIVE fib_sum (iteration, previous_number, new_number) AS (SELECT 1, 0::numeric, 1::numeric UNION ALL SELECT (iteration + 1), new_number, (previous_number + ne

Re: Proposing WITH ITERATIVE

2020-04-29 Thread Fabien COELHO
Hello Corey, Hello Peter, My 0.02 € about the alternative syntaxes: Peter: I think a syntax that would fit better within the existing framework would be something like WITH RECURSIVE t AS ( SELECT base case REPLACE ALL -- instead of UNION ALL SELECT recursive case ) A good po

Re: Proposing WITH ITERATIVE

2020-04-29 Thread Fabien COELHO
Hello, more random thoughts about syntax, semantics, and keeping it relational. While I'm not a huge fan of it, one of the other databases implementing this functionality does so using the syntax: WITH ITERATIVE R AS '(' R0 ITERATE Ri UNTIL N (ITERATIONS | UPDATES) ')' Qf Where N in ITERATI

Re: PG 13 release notes, first draft

2020-05-04 Thread Fabien COELHO
CREATE DATABASE LOCALE option (Fabien COELHO) * Add function gen_random_uuid to generate version 4 UUIDs (Fabien COELHO) I'm not responsible for these, I just reviewed them. ISTM that the author for both is the committer, Peter Eisentraut. Maybe there is something amiss in the commit-l

Re: Another modest proposal for docs formatting: catalog descriptions

2020-05-04 Thread Fabien COELHO
Hello Tom, oid oid Row identifier adrelid oid (references pg_class.oid) The table this column belongs to adnum int2 (references pg_attribute.attnum) The number of the column adbin pg_node_tree The column default value, in nodeToString() representation. Use

Re: PG 13 release notes, first draft

2020-05-05 Thread Fabien COELHO
Hello Bruce, * "DOCUMENT THE DEFAULT GENERATION METHOD" => The default is still to generate data client-side. My point is that the docs are not clear about this. Indeed. Can you fix it? Sure. Attached patch adds an explicit sentence about it, as it was only hinted about in the defaul

Re: Another modest proposal for docs formatting: catalog descriptions

2020-05-05 Thread Fabien COELHO
Hello Tom, oid OID Meh. I'm not a fan of overuse of upper case --- it's well established that that's harder to read than lower or mixed case. And it's definitely project policy that type names are generally treated as identifiers not keywords, even if some of them happen to be keywords

Re: PG 13 release notes, first draft

2020-05-06 Thread Fabien COELHO
about in the default initialization command string, and removes a spurious empty paragraph found nearby. Thanks, patch applied. Ok. You might remove the "DOCUMENT THE DEFAULT…" in the release note. I'm wondering about the commit comment: "Reported-by: Fabien COELHO", act

Re: PG 13 release notes, first draft

2020-05-08 Thread Fabien COELHO
Hello Tom, Uh, can someone else give an opinion on this? I am not sure how hard or un-fun an item is should be used as criteria. Historically we don't document documentation changes at all, do we? ISTM that the "we did not do it previously" is as weak an argument as un-fun-ness:-) It

Why no "array_sort" function?

2020-05-08 Thread Fabien COELHO
Hello devs, although having arrays is an anathema in a relational world, pg has them, and I find it useful for some queries, mostly in an aggregation to show in a compact way what items were grouped together. There are a few functions available to deal with arrays. Among these functions, t

Re: Why no "array_sort" function?

2020-05-08 Thread Fabien COELHO
Hello Sergei, Aggregate functions have syntax for ordering: just "select array_agg(i order by i) from " Described here: https://www.postgresql.org/docs/current/sql-expressions.html#SYNTAX-AGGREGATES Great, that's indeed enough for my usage, thanks for the tip! The questions remains, w

Re: Another modest proposal for docs formatting: catalog descriptions

2020-05-08 Thread Fabien COELHO
Hello Tom, Here's a more fully fleshed out draft for this, with stylesheet markup to get extra space around the column type names. I find this added spacing awkward, espacially as attribute names are always one word anyway. I prefer the non spaced approach. If spacing is discussed, should

Re: Another modest proposal for docs formatting: catalog descriptions

2020-05-09 Thread Fabien COELHO
Hello Tom, Here's a more fully fleshed out draft for this, with stylesheet markup to get extra space around the column type names. I find this added spacing awkward, espacially as attribute names are always one word anyway. I prefer the non spaced approach. It's certainly arguable that tha

Re: PG 13 release notes, first draft

2020-05-12 Thread Fabien COELHO
Hello Bruce, OK, section and item added, patch attached, Thanks! Some items that might be considered for the added documentation section: * e1ff780485 * 34a0a81bfb * e829337d42 * "Document color support (Peter Eisentraut)" THIS WAS NOT DOCUMENTED BEFORE? Not as such, AFAICR it was

Re: PG 13 release notes, first draft

2020-05-12 Thread Fabien COELHO
Hello Bruce, * e1ff780485 I was told in this email thread to not include that one. Ok. * 34a0a81bfb We already have: Reformat tables containing function information for better clarity (Tom Lane) so it seems it is covered as part of this. AFAICR this one is not by

Re: PG 13 release notes, first draft

2020-05-13 Thread Fabien COELHO
Hello Bruce, * 34a0a81bfb We already have: Reformat tables containing function information for better clarity (Tom Lane) so it seems it is covered as part of this. AFAICR this one is not by the same author, and although the point was about better clarity, it was not abou

Re: pgbench: option delaying queries till connections establishment?

2020-05-17 Thread Fabien COELHO
Hello, I've merged all time-related stuff (time_t, instr_time, int64) to use a unique type (pg_time_usec_t) and set of functions/macros, which simplifies the code somehow. Hm. I'm not convinced it's a good idea for pgbench to do its own thing here. I really think that the refactoring part

Re: [Proposal] Page Compression for OLTP

2020-05-21 Thread Fabien COELHO
Hello, My 0.02€, some of which may just show some misunderstanding on my part: - you have clearly given quite a few thoughts about the what and how… which makes your message an interesting read. - Could this be proposed as some kind of extension, provided that enough hooks are availabl

Re: Default gucs for EXPLAIN

2020-05-24 Thread Fabien COELHO
Bonjour Vik, Do we really want default_explain_analyze ? It sounds like bad news that EXPLAIN DELETE might or might not remove rows depending on the state of a variable. I have had sessions where not using ANALYZE was the exception, not the rule. I would much prefer to type EXPLAIN (ANALYZE

Re: Default gucs for EXPLAIN

2020-05-24 Thread Fabien COELHO
The safe option seems not allowing to change ANALYZE option default. EXPLAIN [ ANALYZE ] [ VERBOSE ] statement Some more thoughts: An argument for keeping it that way is that there is already a special syntax to enable ANALYSE explicitely, which as far as I am concerned I only ever atte

Re: Default gucs for EXPLAIN

2020-05-24 Thread Fabien COELHO
Why not allowing the following:  EXPLAIN [ ANALYZE ] [ VERBOSE ] [ ( option [, ...] ) ] statement That has nothing to do with this patch. Sure, it was just in passing, I was surprised by this restriction. -- Fabien.

future pg+llvm compilation is broken

2020-05-27 Thread Fabien COELHO
Hello devs, commit 2c24051bacd2d0eb7141fc4adb870281aec4e714 Author: Craig Topper Date: Fri Apr 24 22:12:21 2020 -0700 [CallSite removal] Rename CallSite.h to AbstractCallSite.h. NFC The CallSite and ImmutableCallSite were removed in a previous commit. So rename the file to m

Re: future pg+llvm compilation is broken

2020-05-27 Thread Fabien COELHO
Hello Justin, llvmjit_inline.cpp:59:10: fatal error: llvm/IR/CallSite.h: No such file or directory Seems to be the same as here: https://www.postgresql.org/message-id/flat/CAGf%2BfX4sDP5%2B43HBz_3fjchawO6boqwgbYVfuFc1D4gbA6qQxw%40mail.gmail.com#540c3746c79c0f13360b35c9c369a887 Definitely

Re: Internal key management system

2020-05-31 Thread Fabien COELHO
Hello Masahiko-san, I am sharing here a document patch based on top of kms_v10 that was shared awhile back. This document patch aims to cover more design details of the current KMS design and to help people understand KMS better. Please let me know if you have any more comments. A few quest

Re: Internal key management system

2020-06-03 Thread Fabien COELHO
Hello Masahiko-san, This key manager is aimed to manage cryptographic keys used for transparent data encryption. As a result of the discussion, we concluded it's safer to use multiple keys to encrypt database data rather than using one key to encrypt the whole thing, for example, in order to ma

Re: Alternative to \copy in psql modelled after \g

2019-01-19 Thread Fabien COELHO
Bonjour Daniel, I took a quick look at this patch. PFA an updated patch addressing your comments and Fabien's. About this v2. Applies cleanly, compiles cleanly, "make check" ok. No tests, but Tom suggests this does not belong here: the committer is right:-) I tested the feature manual

Re: PSA: we lack TAP test coverage on NetBSD and OpenBSD

2019-01-20 Thread Fabien COELHO
Hello Tom, Maybe on OpenBSD pg should switch srandom to srandom_deterministic? Dunno. I'm fairly annoyed by their idea that they're smarter than POSIX. Hmmm. I'm afraid that is not that hard. However, for most of our uses of srandom, this behavior isn't awful; it's only pgbench that has

Re: Alternative to \copy in psql modelled after \g

2019-01-20 Thread Fabien COELHO
Bonjour Daniel, :ROW_COUNT is incorrect after COPY TO STDOUT but this is a PG11-only bug, :ROW_COUNT being a recent addition, so maybe we should deal with this separately, since the current patch is supposed to address all versions? ISTM that the patch is considered a bug fix, so it shoud be

Re: Alternative to \copy in psql modelled after \g

2019-01-20 Thread Fabien COELHO
I understand from the code that the COPY is really executed, so the ERROR and so ROW_COUNT about the SQL should reflect that. Basically the change makes the client believe that there is an SQL error whereas the error is on the client. Right, but wether COPY fails because psql can't write the

Re: PSA: we lack TAP test coverage on NetBSD and OpenBSD

2019-01-20 Thread Fabien COELHO
Hello Tom, Here is a POC which defines an internal interface for a PRNG, and use it within pgbench, with several possible implementations which default to rand48. I seriously dislike this patch. pgbench's random support is quite overengineered already IMO, and this proposes to add a whole b

Re: [PATCH] pgbench tap tests fail if the path contains a perl special character

2019-01-21 Thread Fabien COELHO
Hello Tom, Hm, so bowerbird (a Windows machine) has been failing the pgbench tests since this went in, cf https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=bowerbird&dt=2019-01-20%2004%3A57%3A01 I'm just guessing, but I suspect that bowerbird is using a path spec that includes a backsl

Re: [PATCH] pgbench tap tests fail if the path contains a perl special character

2019-01-21 Thread Fabien COELHO
Hello Andrew, About windows-specific issues, from File::Glob man page: """ On DOSISH systems, backslash is a valid directory separator character. In this case, use of backslash as a quoting character (via GLOB_QUOTE) interferes with the use of backslash as a directory separator. ... """ It se

Re: [PATCH] pgbench tap tests fail if the path contains a perl special character

2019-01-21 Thread Fabien COELHO
Hello Tom, Well, glob does have some metacharacters, so not doing any quoting at all still leaves us with failure modes. I did a little bit of googling on this subject last night, and it seems that at least some people believe that the answer is to not use glob, period, but read the directory

Re: Alternative to \copy in psql modelled after \g

2019-01-21 Thread Fabien COELHO
Bonsoir Daniel, sql> SELECT 1 \g /BAD /BAD: Permission denied sql> \echo :ERROR false That seems broken, because it's pointless to leave out a class of errors from ERROR. Yep. That is my point, but fixing it requires to decide whether it is okay to change ERROR documentation and

Re: [PATCH] pgbench tap tests fail if the path contains a perl special character

2019-01-22 Thread Fabien COELHO
Hello Tom, I did a little bit of googling on this subject last night, and it seems that at least some people believe that the answer is to not use glob, period, but read the directory for yourself. As a short-term move to un-break the buildfarm, I'm just going to revert that patch altogether.

Re: PSA: we lack TAP test coverage on NetBSD and OpenBSD

2019-01-22 Thread Fabien COELHO
Hello Tom, BTW, did you look at the question of the range of zipfian? Yep. I confirmed here that as used in the test case, it's generating a range way smaller than the other ones: repeating the insertion snippet 1000x produces stats like this: [...] I have no idea whether that indicates

Re: [PATCH] pgbench tap tests fail if the path contains a perl special character

2019-01-22 Thread Fabien COELHO
- works around pgbench command splitting on spaces, if postgres sources are in a strangely named directory… I tested within a directory named "pg .* dir". I've also tested it with the initial case (including a +) and it works. Good, thanks for checking! -- Fabien.

Re: Alternative to \copy in psql modelled after \g

2019-01-22 Thread Fabien COELHO
BEGIN; UPDATE pgbench_branches SET bbalance = bbalance + 1 RETURNING * \g /bad // the update is performed, the transaction is not rollbacked, // *but* the output file was not written, "COMMIT" save changes. if PQexec() could not store the results for lack of memory, it would yield a P

Re: pg_dump multi VALUES INSERT

2019-01-22 Thread Fabien COELHO
Hello Surafel, okay .thank you for explaining. i attach a patch corrected as such About this v9: applies cleanly, compiles, global and local "make check" ok. The option is not exercise in the TAP tests. I'd suggest that it should be tested on a small table with zero, 1, more than the val

Re: PSA: we lack TAP test coverage on NetBSD and OpenBSD

2019-01-22 Thread Fabien COELHO
Hello Tom, Given these results, I do not think that it is useful to change random_zipfian TAP test parameter from 2.5 to something else. I'm not following this argument. The test case is basically useless for its intended purpose with that parameter, because it's highly likely that the failu

Re: PSA: we lack TAP test coverage on NetBSD and OpenBSD

2019-01-22 Thread Fabien COELHO
The first value is taken about 75% of the time for N=1000 and s=2.5, which means that a non deterministic implementation would succeed about 0.75² ~ 56% of the time on that one. Right, that's about what we've been seeing on OpenBSD. Also, the drawing procedure is less efficient when the para

Re: PSA: we lack TAP test coverage on NetBSD and OpenBSD

2019-01-22 Thread Fabien COELHO
It's not demonstrably slower than 2.5 either. (1.1 is measurably slower; probably not by enough for anyone to care, but 1.5 is good enough for me.) Good if it fails quick enough for you. Attached a patch with the zipf doc update & the TAP test parameter change. -- Fabien.diff --git a/doc/s

Re: PSA: we lack TAP test coverage on NetBSD and OpenBSD

2019-01-22 Thread Fabien COELHO
Hello Tom, Here is a POC which defines an internal interface for a PRNG, and use it within pgbench, with several possible implementations which default to rand48. I seriously dislike this patch. pgbench's random support is quite overengineered already IMO, and this proposes to add a whole b

yet another comment typo patch

2019-01-23 Thread Fabien COELHO
An assorted set of command typos is fixed in the attached patch. -- Fabien.diff --git a/contrib/pgcrypto/pgp-decrypt.c b/contrib/pgcrypto/pgp-decrypt.c index 7d31e5354b..96c344c30b 100644 --- a/contrib/pgcrypto/pgp-decrypt.c +++ b/contrib/pgcrypto/pgp-decrypt.c @@ -132,7 +132,7 @@ pgp_parse_pkt_

Re: pg_dump multi VALUES INSERT

2019-01-23 Thread Fabien COELHO
Hello David & Surafel, About this v10: Patch applies and compiles cleanly, local & global "make check" ok. A few comments, possibly redundant with some already in the thread. Out of abc-order rows-per-inserts option in getopt struct. help string does not specify the expected argument. I st

Re: pg_dump multi VALUES INSERT

2019-01-23 Thread Fabien COELHO
Hello David, I thought about this and looked into it, but I decided it didn't look like a smart thing to do. The reason is that if --inserts sets dump_inserts to 1 then --rows-per-insert sets it to something else that's likely fine, but if that happens in the opposite order then the --rows-pe

Re: Alternative to \copy in psql modelled after \g

2019-01-25 Thread Fabien COELHO
Bonsoir Daniel, Sure. As there are several bugs (doubtful features) uncovered, a first patch could fix "COPY ...TO STDOUT \g file", but probably replicate ERROR current behavior however debatable it is (i.e. your patch without the ERROR change, which is unrelated to the bug being fixed), and t

Re: Alternative to \copy in psql modelled after \g

2019-01-25 Thread Fabien COELHO
Hello Tom, Fixing the problem envolves deciding what is the right behavior, and update the documentation and the implementation accordingly. Currently the documentation suggests that :ERROR is about SQL error (subject to interpretation), and the implementation is more or less consistent with t

Re: pg_dump multi VALUES INSERT

2019-02-02 Thread Fabien COELHO
Hello David, Wondering if you have anything else here? I'm happy for the v13 version to be marked as ready for committer. I still have a few comments. Patch applies cleanly, compiles, global & local make check are ok. Typos and style in the doc: "However, since, by default this op

Re: Removing \cset from pgbench

2019-02-02 Thread Fabien COELHO
Hola Alvaro, In reply to https://postgr.es/m/alpine.DEB.2.21.1901102211350.27692@lancre wherein Fabien wrote: I'm not very happy with the resulting syntax, but IMO the feature is useful. My initial design was to copy PL/pgSQL "into" with some "\into" orthogonal to \; and ;, but the implementa

Re: pg_dump multi VALUES INSERT

2019-02-03 Thread Fabien COELHO
Hello David, I do not understand why dump_inserts declaration has left the "flags for options" section. I moved that because it's no longer just a flag. It now stores an int value. Hmmm. Indeed, all th "int"s of this section should be "bool" instead. Now, some "flags" do not appear althou

Re: Online verification of checksums

2019-02-04 Thread Fabien COELHO
Hallo Michael, I'm wondering (possibly again) about the existing early exit if one block cannot be read on retry: the command should count this as a kind of bad block, proceed on checking other files, and obviously fail in the end, but having checked everything else and generated a report. I d

Re: pgbench - add pseudo-random permutation function

2019-02-14 Thread Fabien COELHO
Hello Andres, +# PGAC_C_BUILTIN_CLZLL I think this has been partially superceded by commit 711bab1e4d19b5c9967328315a542d93386b1ac5 Author: Alvaro Herrera Date: 2019-02-13 16:10:06 -0300 Indeed, the patch needs a rebase & conflit resolution. I'll do it. Later. +Function pr_pe

Re: libpq host/hostaddr/conninfo inconsistencies

2019-02-14 Thread Fabien COELHO
On 2018-10-26 09:21:51 +0200, Fabien COELHO wrote: (1) you are somehow against changing the current implementation, eg erroring out on possibly misleading configurations, because you do not think it is really useful to help users in those cases. I find this formulation somewhat passive

Re: libpq host/hostaddr/conninfo inconsistencies

2019-02-16 Thread Fabien COELHO
Hello Kyotaro-san, On such standpoint, the first hunk in the patch attracted my eyes. host -Name of host to connect to.host name -If a host name begins with a slash, it specifies Unix-domain -communication rather than TCP/IP communication; the val

Re: CPU costs of random_zipfian in pgbench

2019-02-17 Thread Fabien COELHO
Hello Tomas, I'm trying to use random_zipfian() for benchmarking of skewed data sets, and I ran head-first into an issue with rather excessive CPU costs. [...] This happens because generalizedHarmonicNumber() does this: for (i = n; i > 1; i--) ans += pow(i, -s); wher

Re: libpq host/hostaddr/conninfo inconsistencies

2019-02-17 Thread Fabien COELHO
I am getting error while applying patch.I think the patch needs to be redone on the latest code in master as there are some commits in master after the patch is created Possibly. Here is a v5 with a simplified text. The key point is to tell that "host" expects names, ips or directories, and

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2019-02-19 Thread Fabien COELHO
Hello Tom, Unfortunately, there was no activity over the last few commitfests and the proposed patch pgbench-tap-progress-6 can't be applied anymore without conflicts. Fabien, what are your plans about it, could you please post a rebased version? Here it is. I'm confused about the intende

Re: CPU costs of random_zipfian in pgbench

2019-02-19 Thread Fabien COELHO
Hello Peter, My 0.02€: I'm not quite interested in maintaining a tool for *one* benchmark, whatever the benchmark, its standardness or quality. What I like in "pgbench" is that it is both versatile and simple so that people can benchmark their own data with their own load and their own quer

<    5   6   7   8   9   10   11   12   13   14   >