Re: storing an explicit nonce

2021-10-13 Thread Bruce Momjian
On Wed, Oct 13, 2021 at 09:16:37AM -0400, Stephen Frost wrote: > Greetings, > > * Ants Aasma (a...@cybertec.at) wrote: > > On Wed, 13 Oct 2021 at 02:20, Bruce Momjian wrote: > > > On Wed, Oct 13, 2021 at 12:48:51AM +0300, Ants Aasma wrote: > > > > On Wed,

XTS cipher mode for cluster file encryption

2021-10-13 Thread Bruce Momjian
review it since there doesn't seem to be enough documentation for us to decide ourselves. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion.

Re: [PATCH] Proposal for HIDDEN/INVISIBLE column

2021-10-15 Thread Bruce Momjian
ly can't think of a single case where output is specified at the DDL level. Why is this not better addressed by creating a view on the original table, even perhaps renaming the original table and create a view using the old table name. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion.

Re: XTS cipher mode for cluster file encryption

2021-10-16 Thread Bruce Momjian
t detect changes to pg_xact, fsm, vm, etc, which could also affect the integrity of the data. Someone could also restore and old copy of a patch to revert a change, and that would not be detected even by GCM. I consider this a checkbox feature and making it too complex will cause it to be rightly

Re: XTS cipher mode for cluster file encryption

2021-10-16 Thread Bruce Momjian
On Sat, Oct 16, 2021 at 09:15:05AM -0700, Andres Freund wrote: > Hi, > > On 2021-10-16 10:16:25 -0400, Bruce Momjian wrote: > > As a final comment to Andres's email, adding a GCM has the problems > > above, plus it wouldn't detect changes to pg_xact, fsm, vm, etc,

Re: XTS cipher mode for cluster file encryption

2021-10-22 Thread Bruce Momjian
either XTS or AES-GCM-SIV. There are threat models > which dm-crypt would address, of course, such as data-at-rest (hard > drive theft, improper disposal of storage media, backups which don't > have their own encryption, etc), but, again, dm-crypt isn't always an > option that

Re: XTS cipher mode for cluster file encryption

2021-10-22 Thread Bruce Momjian
at gets reduced to 50% because we don't protect against write attacks --- we are still 100% read-protected and 0% write protected. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion.

Re: XTS cipher mode for cluster file encryption

2021-10-22 Thread Bruce Momjian
ommitted feature > here than it does today - even if somehow that were to happen for v15. > I expect there are people out there trying to break it even as I write > these words, and it seems likely that they will eventually succeed, > but as to when, who can say? Yes, we should start on

Re: XTS cipher mode for cluster file encryption

2021-10-22 Thread Bruce Momjian
different WAL records at the > > same LSN and so using the LSN there should be alright." > > I don't like the idea of "CTR-like". What's wrong with using CTR for > WAL encryption? Based on the available information, that seems like the > exact use-case f

Re: XTS cipher mode for cluster file encryption

2021-10-23 Thread Bruce Momjian
pt does. That doing so would mean that That is an excellent point, and something we should mention in our documentation --- the fact that a change of 8k granularity will be visible, and in certain specified cases, 16-byte change granularity will also be visible. -- Bruce Momjian https://m

Re: XTS cipher mode for cluster file encryption

2021-10-23 Thread Bruce Momjian
ng non-security Postgres features, cool features can be more easily implemented because they are built on the sold foundation of Postgres. For security features, you have to assume that attacks can come from anywhere, so the foundation is unclear and caution is wise. -- Bruce Momjian

Re: XTS cipher mode for cluster file encryption

2021-10-23 Thread Bruce Momjian
cryption was a requirement. Would a CTR-steaming-encryption API for temporary tables be easier to implement? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion.

Re: XTS cipher mode for cluster file encryption

2021-10-25 Thread Bruce Momjian
eing > created (potentially concurrently). Probably not that hard to do but > just something to make sure we do. Of course, if we arrange for > block-based access then we could use XTS or perhaps GCM/GCM-SIV if we > wanted to. Agreed on all points. -- Bruce Momjian

Re: XTS cipher mode for cluster file encryption

2021-10-26 Thread Bruce Momjian
cheme, with two > options. That's generally a good engineering practice, as it ensures things > are not coupled too much. And it's not like the encryption methods are > expected to be super difficult. I am not in favor of adding additional options to this feature unless we can explain why users should choose one over the other. There is also the problem of OpenSSL not supporting Adiantum. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion.

Re: [PATCH] Proposal for HIDDEN/INVISIBLE column

2021-10-28 Thread Bruce Momjian
verify that ALTER ... SET UNEXPANDED > statements are well generated in the dump. I want to state I still think this feature is not generally desired, and is better implemented at the query level. -- Bruce Momjian https://momjian.us EDB https://e

Re: Width of SubTransactionId (hello Postgres PRO)

2021-10-28 Thread Bruce Momjian
of. I know of no plans to implement 64-bit transaction ids in community Postgres because of the longer tuple header and file format changes. It is discussed occasionally though. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com If only

Re: Width of SubTransactionId (hello Postgres PRO)

2021-10-28 Thread Bruce Momjian
On Thu, Oct 28, 2021 at 12:02:30PM -0400, Chapman Flack wrote: > On 10/28/21 10:37, Bruce Momjian wrote: > > > I know of no plans to implement 64-bit transaction ids in community > > Postgres because of the longer tuple header and file format changes. > > It is discu

Re: XTS cipher mode for cluster file encryption

2021-11-02 Thread Bruce Momjian
27;ll > do from the start, I would think providing the options of AES128 and > AES256 would be good to ensure that we have the bits covered to support > multiple methods and I don't think that would put us into a situation of My patch supports AES128, AES192, and AES256. -- Bruce Mo

Re: should we enable log_checkpoints out of the box?

2021-11-02 Thread Bruce Momjian
red out with log_min_message. I think if we want to offer a more verbose set of analytic output, by default or not, we might want to relabel them as something other than LOG messages so they can be filtered out with log_min_messages. -- Bruce Momjian https://momjian.us EDB

Re: should we enable log_checkpoints out of the box?

2021-11-03 Thread Bruce Momjian
with enabling that by default. I wonder if we need to follow the Unix model on this by having kernel messages logged to a file via syslog (and potentially filtered), and then have all recent activity logged into a ring buffer and visible via something like dmesg. -- Bruce Momjian https:/

Re: should we enable log_checkpoints out of the box?

2021-11-03 Thread Bruce Momjian
the default install * changing the labels of some of the normal-operation log messages * changing the way some of these log messages are controlled * perhaps using a ring buffer for common log messages -- Bruce Momjian https://momjian.us EDB htt

Re: should we enable log_checkpoints out of the box?

2021-11-03 Thread Bruce Momjian
ving with the > gigantic volume of ERROR and FATAL messages that typically end up in > the logs today, they can certainly live with the absolutely tiny > volume of additional information that would be generated by > log_checkpoints=on. See my later em

Re: should we enable log_checkpoints out of the box?

2021-11-05 Thread Bruce Momjian
er for admins to see the > really bad things that are happening? Agreed. We have three levels of non-error message for the client (info, notice, warning), but only one level for error (query is stopped), ERROR. Seems we need another level that stops the current query and indicates it contains

Re: Commitfest 2021-11 Patch Triage - Part 2

2021-11-09 Thread Bruce Momjian
mbination. Yeah, I am leaning this diretion too. On the one hand, the number of ways to exploit compression-then-encryption seem to continue to grow, but it is the complex behavior of HTTPS and connecting to multiple sites which seems to always be required, and we don't do that with the Postg

Re: add recovery, backup, archive, streaming etc. activity messages to server logs along with ps display

2021-11-11 Thread Bruce Momjian
accessible in a running server, but they would not be written to the > > server log. > > That's an interesting idea. It'd be nice to have a way to see more > debugging information without exploding the server log. Uh, that's what dmesg does, which I mentioned ea

Re: storing an explicit nonce

2021-11-11 Thread Bruce Momjian
mean I don't have anything against that appendix, but I think we > need to understand - with confidence - what the expectations are > specifically around XTS, and that appendix seems much more general > than that. Since there has not been activity on this thread for one month

Re: XTS cipher mode for cluster file encryption

2021-11-12 Thread Bruce Momjian
y on this thread, I have updated the TDE wiki to reflect the conclusions and discussions: https://wiki.postgresql.org/wiki/Transparent_Data_Encryption -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion.

Re: pg_upgrade parallelism

2021-11-18 Thread Bruce Momjian
gain, would there be a benefit? > I'm trying to add more parallelism by copying individual segments > of a relfilenode in different processes. Does anyone one see a big > problem in trying to do that? I'm asking because no one did it before, &g

Re: Reduce function call costs on ELF platforms

2021-11-22 Thread Bruce Momjian
symbols bound eagerly. I found this really interesting, and I am surprised how things got so suboptimal. Has it always been this way? Is it the use of C++ that is causing this by default? -- Bruce Momjian https://momjian.us EDB https://enterprise

Re: Correct handling of blank/commented lines in PSQL interactive-mode history

2021-11-30 Thread Bruce Momjian
determined. > (I dislike using "application/octet-stream" for this, because > the archives won't show that as text, they only let you download > the attachment. Maybe that's more Safari's fault than the > archives per se, not sure.) I wou

Re: Correct handling of blank/commented lines in PSQL interactive-mode history

2021-11-30 Thread Bruce Momjian
On Tue, Nov 30, 2021 at 04:35:13PM -0500, Tom Lane wrote: > Bruce Momjian writes: > > On Mon, Nov 29, 2021 at 11:12:35PM -0500, Tom Lane wrote: > >> Interesting. I can probably adjust my MUA to send "text/x-patch", > >> but I'll have to look around to s

Re: file cloning in pg_upgrade and CREATE DATABASE

2018-03-23 Thread Bruce Momjian
opying user relation files to Copying (copy-on-write) user relation files -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: Fix a typo in walsender.c

2018-03-28 Thread Bruce Momjian
alSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, > TransactionId xid) Thanks, patch applied. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: 2018-03 Commitfest Summary (Andres #1)

2018-03-29 Thread Bruce Momjian
a separate project where a separate team could maintain and improve it? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: [HACKERS] [POC] Faster processing at Gather node

2018-03-29 Thread Bruce Momjian
Pairs with the full barrier in shm_mq_send_bytes(). > We > + * only need a read barrier here because the increment of mq_bytes_read > is > + * actually a read followed by a dependent write. > + */ > > (" Pairs ..." vs. &

Re: Typo in src/backend/access/hash/README

2018-03-29 Thread Bruce Momjian
and > > |check whether the bucket count stored in hasho_prevblkno is greater than > > |OUR the number of buckets stored in our cached copy of the metapage. If > > > > remove "our"? > > > > Both your and Thomas's proposed change looks good to me. Fixe

Re: Incorrect use of "an" and "a" in code comments and docs

2018-03-29 Thread Bruce Momjian
12 12:58:35 2018 -0300 docs: Fix typo: a -> an David Rowley -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: Incorrect use of "an" and "a" in code comments and docs

2018-03-29 Thread Bruce Momjian
d is a proposal of patch to fix all those things. Attached patch applied. It includes a suggested fix by Abhijit Menon-Sen. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + +

Re: Incorrect use of "an" and "a" in code comments and docs

2018-03-29 Thread Bruce Momjian
gt; >> > correct, not "an". > > > > I guess it depends on whether you read it as "sin-val" or "ess-inval". > > $ git grep ' a SQL ' | wc -l > 642 > $ git grep ' an SQL ' | wc -l > 219 > > /me

Re: cannot drop replication slot if server is running in single-user mode

2018-03-29 Thread Bruce Momjian
slot    (typeid = 2278, len = 4, typmod = -1, > byval = t) >     > 2018-03-06 13:20:24.391 GMT [14869] ERROR:  epoll_ctl() failed: Bad file > descriptor > 2018-03-06 13:20:24.391 GMT [14869] STATEMENT:  select > pg_drop_replication_slot('p'); I can confirm this bug e

Re: cannot drop replication slot if server is running in single-user mode

2018-03-29 Thread Bruce Momjian
On Thu, Mar 29, 2018 at 02:51:24PM -0700, Andres Freund wrote: > On 2018-03-29 17:42:57 -0400, Bruce Momjian wrote: > > On Tue, Mar 6, 2018 at 06:59:33PM +0530, tushar wrote: > > > Hi, > > > > > > I found that if server is running in single-us

Re: cannot drop replication slot if server is running in single-user mode

2018-03-29 Thread Bruce Momjian
On Thu, Mar 29, 2018 at 06:09:28PM -0700, Andres Freund wrote: > On 2018-03-29 21:07:36 -0400, Bruce Momjian wrote: > > I think the question is whether this is exposing a bug or is expected > > behavior. > > It's unsurprising. Acquiring a slot uses a condition variabl

Re: Fix missing spaces in docs

2018-03-29 Thread Bruce Momjian
sgml > @@ -21,7 +21,7 @@ PostgreSQL documentation > > > > -CREATE USER MAPPING [IF NOT EXISTS] FOR { class="parameter">user_name | USER | CURRENT_USER | PUBLIC } > +CREATE USER MAPPING [ IF NOT EXISTS ] FOR { class="parameter">user_name | USER | C

lo_import() of an empty file

2018-03-30 Thread Bruce Momjian
consume no space in pg_largeobject, but it does seem odd. I am asking hackers in case someone can see an error here, or something that should be documented. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you a

Re: lo_import() of an empty file

2018-03-30 Thread Bruce Momjian
On Fri, Mar 30, 2018 at 02:22:49PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > I have just discovered if you try to lo_import() an empty file, and oid > > is returned, but nothing is added to pg_largeobject: > > Why would you expect there to be? There's no

Re: BUG #15112: Unable to run pg_upgrade with earthdistance extension

2018-04-02 Thread Bruce Momjian
On Fri, Mar 30, 2018 at 04:52:29PM -0400, Bruce Momjian wrote: > On Wed, Mar 14, 2018 at 11:12:26PM +, PG Bug reporting form wrote: > > The following bug has been logged on the website: > > > > Bug reference: 15112 > > Logged by: Keiko Oda &g

Re: 2018-03 Commitfest Summary (Andres #1)

2018-04-02 Thread Bruce Momjian
rking use > in mind, about which core committers do not seem much interested. Agreed. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient

Re: jsonb nesting level edge case

2018-04-03 Thread Bruce Momjian
some > other databases there is just a limit for number of nested levels for a > document (e.g. in MongoDB Bson, MySQL binary json it's exactly 100). I think our current behavior is the best we can do. We are limited only by configured stack memory, which people can increase. I don&#x

Re: BUG #15122: can't import data if table has a constraint with a function calling another function

2018-04-03 Thread Bruce Momjian
moval of public was trying to avoid. So, the crux of the problem is how do you install a function in an non-predefined schema, e.g. public, and reference other extension objects in a safe and transparent way. This is true of non-extension objects as well if they are assuming they

Re: Comment update in BuildTupleFromCStrings()

2018-04-03 Thread Bruce Momjian
_result_row() which does the same thing and explain why > we need to invoke in() function even on the NULL values. I thought, > the same comment applies here. Here's patch to update the comment in > BuildTupleFromCStrings(). Patch applied. -- Bruce Momjian h

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-03 Thread Bruce Momjian
but I have to ask since our fsync() assumptions have been wrong for so long. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-03 Thread Bruce Momjian
On Wed, Apr 4, 2018 at 01:54:50PM +1200, Thomas Munro wrote: > On Wed, Apr 4, 2018 at 12:56 PM, Bruce Momjian wrote: > > There has been a lot of focus in this thread on the workflow: > > > > write() -> blocks remain in kernel memory -> fsync() -> panic? &g

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-03 Thread Bruce Momjian
On Tue, Apr 3, 2018 at 10:05:19PM -0400, Bruce Momjian wrote: > On Wed, Apr 4, 2018 at 01:54:50PM +1200, Thomas Munro wrote: > > I believe there were some problems of that nature (with various > > twists, based on other concurrent activity and possibly different > > fds), an

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-04 Thread Bruce Momjian
o storage. We knew the operating system wrote the data in 8k chunks to storage but: o the 8k pages are written as separate 512-byte sectors o the 8k might be contiguous logically on the drive but not physically o even 512-byte sectors are not written atomically This is why we added pre-page im

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-04 Thread Bruce Momjian
there is precedent for this approach, though I am not advocating it. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-04 Thread Bruce Momjian
> either (so what does it do?). Anthony Iliopoulos reported in this thread that errors=remount-ro is only affected by metadata writes. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-04 Thread Bruce Momjian
On Wed, Apr 4, 2018 at 11:23:51PM +0800, Craig Ringer wrote: > On 4 April 2018 at 21:49, Bruce Momjian wrote: > I can understand why kernel developers don't want to keep failed sync > buffers in memory, and once they are gone we lose reporting of their > failure.  Al

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-05 Thread Bruce Momjian
o ENOSPC we'll > definitely see the same problems. This makes ENOSPC on NFS a potentially data > corrupting condition since NFS doesn't preallocate space before returning from > write(). This does explain why NFS has a reputation for unreliability for Postgres. -- Bruce M

Re: pgsql: New files for MERGE

2018-04-05 Thread Bruce Momjian
Specific changes requested have now been implemented by Pavan and > committed by me. > > My understanding is that he is working on a patch for Tom's requested > parser changes, will post on other thread. Simon, you have three committers in this thread suggesting this patch be rever

Re: pgsql: New files for MERGE

2018-04-06 Thread Bruce Momjian
On Fri, Apr 6, 2018 at 09:21:54AM +0100, Simon Riggs wrote: > On 5 April 2018 at 21:02, Bruce Momjian wrote: > > Simon, you have three committers in this thread suggesting this patch be > > reverted. Are you just going to barrel ahead with the fixes without > > ad

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-04-07 Thread Bruce Momjian
nk it will solve 80-90% of optimizer complaints, and I don't see any other way to fix them than this. This seems like the right way to fix optimizer problems, instead of optimizer hints. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.c

Re: pgsql: New files for MERGE

2018-04-07 Thread Bruce Momjian
mmitfest. I think one of the missing rules we have is that when we say no new large patches in the final commitfest, do we mean that all _three_ stages should be solidified before the final commitfest? I have never been clear on that point. -- Bruce Momjian http://momjian

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-04-07 Thread Bruce Momjian
On Sat, Apr 7, 2018 at 06:52:42PM +0100, Dean Rasheed wrote: > On 7 April 2018 at 15:12, Bruce Momjian wrote: > > Uh, where are we on this patch? It isn't going to make it into PG 11? > > Feature development for this has been going on for years. > > Unfortunately, I

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-07 Thread Bruce Momjian
g systems including keeping > buffers dirty and marking the whole filesystem broken/read-only). > > Idea 3: > > Give up on buffered IO and develop an O_SYNC | O_DIRECT based system ASAP. Idea 4 would be for people to assume their database is corrupt if their server l

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-08 Thread Bruce Momjian
Postgres when such errors happen, at least the administrator could fix the problem of fail-over to a standby. An crazy idea would be to have a daemon that checks the logs and stops Postgres when it seems something wrong. -- Bruce Momjian http://momjian.us EnterpriseDB

'make check' fails

2018-04-11 Thread Bruce Momjian
t; ^ compilation terminated. : recipe for target 'relpath.o' failed make[3]: *** [relpath.o] Error 1 Oddly, this works: make clean; make; make check I found this because I have some scripts that do the form

Re: 'make check' fails

2018-04-11 Thread Bruce Momjian
On Wed, Apr 11, 2018 at 10:59:45AM -0400, Tom Lane wrote: > Bruce Momjian writes: > > I have discovered that: > > make clean; make check > > fails with: > > No doubt this is related to the generated-headers changes I've been > making, but I find your rec

Re: 'make check' fails

2018-04-11 Thread Bruce Momjian
clear, I can reproduce this failure without parallel mode. FYI, the reason I used this shortcut is so I can just run one command and check one error code to know if they all worked; see src/tools/pgtest for an example. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: 'make check' fails

2018-04-11 Thread Bruce Momjian
On Wed, Apr 11, 2018 at 01:04:56PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Wed, Apr 11, 2018 at 12:35:41PM -0400, Tom Lane wrote: > >> We can fix this by making submake-generated-headers be a recursive > >> prerequisite for "check" as w

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-17 Thread Bruce Momjian
ght get the write request until after we return write success to the process. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-17 Thread Bruce Momjian
On Mon, Apr 9, 2018 at 03:42:35PM +0200, Tomas Vondra wrote: > On 04/09/2018 12:29 AM, Bruce Momjian wrote: > > > > An crazy idea would be to have a daemon that checks the logs and > > stops Postgres when it seems something wrong. > > > > That doesn'

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-17 Thread Bruce Momjian
On Mon, Apr 9, 2018 at 03:42:35PM +0200, Tomas Vondra wrote: > On 04/09/2018 12:29 AM, Bruce Momjian wrote: > > > > An crazy idea would be to have a daemon that checks the logs and > > stops Postgres when it seems something wrong. > > > > That doesn'

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-17 Thread Bruce Momjian
seems NAS/NFS and any thin provisioned storage will have this problem, and again, not always reported. So, our initial action might just be to educate users that write errors can cause silent corruption, and out-of-space errors on NAS/NFS and any thin provisioned storage can cause corruptio

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-18 Thread Bruce Momjian
On Tue, Apr 17, 2018 at 02:34:53PM -0700, Andres Freund wrote: > On 2018-04-17 17:29:17 -0400, Bruce Momjian wrote: > > Also, if we are relying on WAL, we have to make sure WAL is actually > > safe with fsync, and I am betting only the O_DIRECT methods actual

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-18 Thread Bruce Momjian
On Wed, Apr 18, 2018 at 06:04:30PM +0800, Craig Ringer wrote: > On 18 April 2018 at 05:19, Bruce Momjian wrote: > > On Tue, Apr 10, 2018 at 05:54:40PM +0100, Greg Stark wrote: > >> On 10 April 2018 at 02:59, Craig Ringer wrote: > >> > >> > Nitpick: In mo

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-18 Thread Bruce Momjian
On Tue, Apr 17, 2018 at 02:41:42PM -0700, Andres Freund wrote: > On 2018-04-17 17:32:45 -0400, Bruce Momjian wrote: > > On Mon, Apr 9, 2018 at 03:42:35PM +0200, Tomas Vondra wrote: > > > That doesn't seem like a very practical way. It's better than nothing, > > &

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-20 Thread Bruce Momjian
On Wed, Apr 18, 2018 at 08:45:53PM +0800, Craig Ringer wrote: > wrOn 18 April 2018 at 19:46, Bruce Momjian wrote: > > > So, if sync mode passes the write to NFS, and NFS pre-reserves write > > space, and throws an error on reservation failure, that means that NFS > > wi

Re: documentation is now XML

2018-04-23 Thread Bruce Momjian
t. Well, I will say that backpatching doc changes is more complex having SGML and XML in supported versions, particularly because of the different tag endings between them. Therefore, having all the docs be XML would be nice from my perspective. -- Bruce Momjian http://momjian.us Enterpris

Re: Built-in connection pooling

2018-04-23 Thread Bruce Momjian
e procArray to reduce the number of _assigned_ PGPROC entries we have to scan. Why can't we create another array that only contains _active_ sessions, i.e. those not in a transaction. In what places can procArray scans be changed to use this new array? -- Bruce Momjian http://

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-23 Thread Bruce Momjian
mber that NFS does not reserve space for writes like local file systems like ext4/xfs do. For that reason, we might be able to capture the out-of-space error on close and exit sooner for NFS. -- Bruce Momjian http://momjian.us EnterpriseDB

Re: Built-in connection pooling

2018-04-23 Thread Bruce Momjian
On Mon, Apr 23, 2018 at 09:47:07PM -0400, Robert Haas wrote: > On Mon, Apr 23, 2018 at 7:59 PM, Bruce Momjian wrote: > > So, instead of trying to multiplex multiple sessions in a single > > operating system process, why don't we try to reduce the overhead of > > idle

Re: Built-in connection pooling

2018-04-23 Thread Bruce Momjian
On Mon, Apr 23, 2018 at 09:53:37PM -0400, Bruce Momjian wrote: > On Mon, Apr 23, 2018 at 09:47:07PM -0400, Robert Haas wrote: > > On Mon, Apr 23, 2018 at 7:59 PM, Bruce Momjian wrote: > > > So, instead of trying to multiplex multiple sessions in a single > > > operatin

Re: Bugs in TOAST handling, OID assignment and redo recovery

2018-04-24 Thread Bruce Momjian
rts. > > Also, "toast value" is same as "chunk_id". Should we clean up something there > too? "chunk_seq number" -- should that be just "chunk_seq"? We can put a comment next to the error message C string if we want to keep historical knowledge.

Re: description of root_tuple_slot missing

2018-04-26 Thread Bruce Momjian
partitions in > the root partitioned table's > + * rowtype > *--- > */ > typedef struct PartitionTupleRouting Patch applied. Thanks. -- Bruce Momjian http://momjian.us EnterpriseD

Re: Fwd: [BUGS] pg_trgm word_similarity inconsistencies or bug

2018-04-26 Thread Bruce Momjian
trigrams is > - {" t"," tw",two,"wo "," w"," wo","wor","ord","rds", ds > "}. > + {" t"," tw","two","wo "," w"," wo","wor","ord","rds","ds > "}. > The most similar extent of an ordered set of trigrams in the second string > is {" w"," wo","wor","ord"}, and the similarity is > 0.8. > @@ -172,7 +172,7 @@ > At the same time, strict_word_similarity(text, text) > has to select an extent that matches word boundaries. In the example > above, > strict_word_similarity(text, text) would select the > - extent {" w"," wo","wor","ord","rds", ds "}, which > + extent {" w"," wo","wor","ord","rds","ds "}, which > corresponds to the whole word 'words'. > > -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: pg_upgrade fails with non-standard ACL

2019-09-26 Thread Bruce Momjian
On Wed, Sep 11, 2019 at 06:25:38PM -0300, Álvaro Herrera wrote: > On 2019-Aug-21, Bruce Momjian wrote: > > > > 1) How exactly should we report this incompatibility to a user? > > > I think it's fine to leave the warnings and also write some hint for the > >

Re: pg_upgrade fails with non-standard ACL

2019-09-26 Thread Bruce Momjian
On Thu, Sep 26, 2019 at 05:16:19PM -0300, Alvaro Herrera wrote: > On 2019-Sep-26, Bruce Momjian wrote: > > Well, right now, pg_upgrade --check succeeds, but the upgrade fails. I > > am proposing, at a minimum, that pg_upgrade --check fails in such cases, > > Agreed, that sh

Re: pg_upgrade fails with non-standard ACL

2019-09-27 Thread Bruce Momjian
On Fri, Sep 27, 2019 at 04:22:15PM +0900, Michael Paquier wrote: > On Thu, Sep 26, 2019 at 04:19:38PM -0400, Bruce Momjian wrote: > > On Thu, Sep 26, 2019 at 05:16:19PM -0300, Alvaro Herrera wrote: > >> On 2019-Sep-26, Bruce Momjian wrote: > >>> Well, right now, pg

Re: fix "Success" error messages

2019-09-27 Thread Bruce Momjian
e that's worth a back-patch though. Those parts could > > find their way to v12 easily. > > Committed to master and PG12 with your suggested changes. This "Success" has happened so many times I think we should tell people to report any such error message as a bug by emi

Re: Append with naive multiplexing of FDWs

2019-09-27 Thread Bruce Momjian
ow to integrate with that so I did a quick "sledgehammer" rebase that > disables itself if parallelism is in the picture. Yes, sharding has been waiting on parallel FDW scans. Would this work for parallel partition scans if the partitions were FDWs? -- Bruce Momjian http://mom

Re: patch: psql - enforce constant width of last column

2019-09-27 Thread Bruce Momjian
th a little more work: $ export PSQL_PAGER='pspg --vertical-cursor' $ psql test \pset border 1 \pset linestyle ascii \pset pager always select * from generate_series(1,3); Line '1' has highlighted trailing space.

Re: pg_upgrade issues

2019-09-27 Thread Bruce Momjian
upgrade just restores the pg_description descriptions and doesn't modify them. Do you get an error on restore because of the duplicate pg_description oids? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you

Re: A problem presentaion about ECPG, DECLARE STATEMENT

2019-09-27 Thread Bruce Momjian
y confused by this also when researching this for the PG 12 release notes. I am glad we are going to redo it for PG 13. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be.

Re: Modest proposal for making bpchar less inconsistent

2019-09-28 Thread Bruce Momjian
s and let such calls devolve > into implicit casts to text. > > This might annoy people who are actually writing trailing spaces > in their patterns to make such cases work.  But I think there > are probably not too many such people, and having real consistency

Re: [DOC] Document concurrent index builds waiting on each other

2019-09-28 Thread Bruce Momjian
not return on any one table until all have completed. > + > + > > Concurrent builds for indexes on partitioned tables are currently not > supported. However, you may concurrently build the index on each -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: Usage of the system truststore for SSL certificate validation

2019-09-28 Thread Bruce Momjian
ient certificate requires sending the public key to the ssh server to be added to ~/.ssh/authorized_keys. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: [DOC] Document concurrent index builds waiting on each other

2019-09-28 Thread Bruce Momjian
On Sat, Sep 28, 2019 at 09:54:48PM -0400, James Coleman wrote: > On Sat, Sep 28, 2019 at 9:22 PM Alvaro Herrera > wrote: > > > > On 2019-Sep-28, Bruce Momjian wrote: > > > > > The CREATE INDEX docs already say: > > > > > > In a concurr

Re: Online checksums patch - once again

2019-09-30 Thread Bruce Momjian
a standby that has encryption enabled, then switchover to it. Is that a method we support now for adding checksums to Postgres? Do we need the ability to do it in-place too? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: Online checksums patch - once again

2019-09-30 Thread Bruce Momjian
On Mon, Sep 30, 2019 at 04:57:41PM +0200, Magnus Hagander wrote: > > > On Mon, Sep 30, 2019 at 4:53 PM Bruce Momjian wrote: > > On Mon, Sep 30, 2019 at 02:49:44PM +0200, Tomas Vondra wrote: > > On Mon, Sep 30, 2019 at 01:03:20PM +0200, Magnus Hagander wrote: >

Transparent Data Encryption (TDE) and encrypted files

2019-09-30 Thread Bruce Momjian
ue to encrypting vm, fsm, pg_xact, pg_multixact, or other files. Is that correct? Do any other PGDATA files contain user data? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you wi

<    1   2   3   4   5   6   7   8   9   10   >