Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-10-01 Thread Tomas Vondra
On Tue, Oct 01, 2019 at 06:55:52PM +0530, Amit Kapila wrote: On Sun, Sep 29, 2019 at 11:24 AM Amit Kapila wrote: On Sun, Sep 29, 2019 at 12:39 AM Tomas Vondra wrote: > Yeah, it is better to deal it separately as I am also not entirely convinced at this stage about this parameter. I h

Re: Optimize partial TOAST decompression

2019-10-01 Thread Tomas Vondra
On Tue, Oct 01, 2019 at 10:10:37AM -0400, Tom Lane wrote: Tomas Vondra writes: Hmmm, this seems to trigger a failure on thorntail, which is a sparc64 machine (and it seems to pass on all x86 machines, so far). gharial's not happy either, and I bet if you wait a bit longer you'll se

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-10-02 Thread Tomas Vondra
On Wed, Oct 02, 2019 at 04:27:30AM +0530, Amit Kapila wrote: On Tue, Oct 1, 2019 at 7:21 PM Tomas Vondra wrote: On Tue, Oct 01, 2019 at 06:55:52PM +0530, Amit Kapila wrote: > >On further testing, I found that the patch seems to have problems with >toast. Consider below scenario: &

Re: Value of Transparent Data Encryption (TDE)

2019-10-03 Thread Tomas Vondra
but this just seems like a really strange idea. It's not new and it's how TDE works in all of the other database systems which support it. Yep. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-03 Thread Tomas Vondra
#x27;t. I know it's not an easy problem to solve, but it may contain user data (which is what we manage). We may allow disabling that, at which point it becomes someone else's problem. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Regarding extension

2019-10-03 Thread Tomas Vondra
r will be invoked or not? " I'm not sure I understand the question. Are you asking if the event trigger will be invoked to notify you about creation of the extension containing it? I'm pretty sure that won't happen - it will be executed only for future CREATE EXTENSION commands. reg

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-03 Thread Tomas Vondra
On Thu, Oct 03, 2019 at 11:51:41AM -0400, Stephen Frost wrote: Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: On Thu, Oct 03, 2019 at 10:40:40AM -0400, Stephen Frost wrote: >People who are looking for 'encrypt all the things' should and will be >looking at

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-03 Thread Tomas Vondra
the idea of just encrypting everything (including vm, fsm etc.). The only case that seems to be an exception is the column-level encryption in Oracle, all the other options (especially the database-level ones) seem to be consistent with this principle. regards -- Tomas Vondra

Re: Memory Accounting

2019-10-04 Thread Tomas Vondra
ient, but I haven't analyzed it. I think so too, but I'll take a closer look in the afternoon, unless you beat me to it. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: HashTable KeySize

2019-10-04 Thread Tomas Vondra
uctId -- 8 bytes See https://en.wikipedia.org/wiki/Data_structure_alignment and there's also a tool to show the memory layout: https://linux.die.net/man/1/pahole regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Memory Accounting

2019-10-04 Thread Tomas Vondra
On Fri, Oct 04, 2019 at 10:26:44AM +0200, Tomas Vondra wrote: On Fri, Oct 04, 2019 at 12:36:01AM -0400, Tom Lane wrote: I haven't chased down exactly what else would need to change. It might be that s/int64/Size/g throughout the patch is sufficient, but I haven't analyzed it. I th

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-04 Thread Tomas Vondra
ltimately it's a trade-off between complexity of implementation and severity of the side-channel. But without at least trying to quantify the severity of the side-channel we can't really have a discussion whether it's OK not to encrypt clog, whether it can be omitted from v1 etc.

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-04 Thread Tomas Vondra
On Thu, Oct 03, 2019 at 01:26:55PM -0400, Stephen Frost wrote: Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: On Thu, Oct 03, 2019 at 11:51:41AM -0400, Stephen Frost wrote: >* Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: >>On Thu, Oct 03, 2019 at 10:40:4

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-04 Thread Tomas Vondra
bout that, and even issue a server log message if encryption is enabled and syslog is not being used. (I don't know how to test if syslog is being shipped to a remote server.) Not sure. I wonder if it's possible to setup syslog so that it encrypts the data on storage, and if that would be

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-04 Thread Tomas Vondra
On Fri, Oct 04, 2019 at 04:58:14PM -0400, Bruce Momjian wrote: On Fri, Oct 4, 2019 at 10:46:57PM +0200, Tomas Vondra wrote: Oracle also has a handy "TDE best practices" document [2], which says when to use column-level encryption - let me quote a couple of points: * Location of

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-04 Thread Tomas Vondra
On Fri, Oct 04, 2019 at 06:06:10PM -0400, Bruce Momjian wrote: On Fri, Oct 4, 2019 at 11:48:19PM +0200, Tomas Vondra wrote: On Fri, Oct 04, 2019 at 04:58:14PM -0400, Bruce Momjian wrote: > On Fri, Oct 4, 2019 at 10:46:57PM +0200, Tomas Vondra wrote: > > Oracle also has a handy

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-05 Thread Tomas Vondra
On Fri, Oct 04, 2019 at 08:14:44PM -0400, Bruce Momjian wrote: On Sat, Oct 5, 2019 at 12:54:35AM +0200, Tomas Vondra wrote: On Fri, Oct 04, 2019 at 06:06:10PM -0400, Bruce Momjian wrote: > For full-cluster TDE with AES-NI-enabled, the performance impact is > usually ~4%, so doing anythin

Re: How to retain lesser paths at add_path()?

2019-10-06 Thread Tomas Vondra
t it, and it's not clear to me why would it be OK to call the hook for remove_old=true but not also for accept_new=false? How do we know whether the "better" path arrives first? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-07 Thread Tomas Vondra
On Mon, Oct 07, 2019 at 10:22:22AM -0400, Bruce Momjian wrote: On Sat, Oct 5, 2019 at 09:13:59PM +0200, Tomas Vondra wrote: On Fri, Oct 04, 2019 at 08:14:44PM -0400, Bruce Momjian wrote: > On Sat, Oct 5, 2019 at 12:54:35AM +0200, Tomas Vondra wrote: > > On Fri, Oct 04, 2019 at 0

Re: pg_init

2019-10-08 Thread Tomas Vondra
ea to explain what you're trying to achieve. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Remove size limitations of vacuums dead_tuples array

2019-10-10 Thread Tomas Vondra
VugVht6Q%40mail.gmail.com -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: BRIN index which is much faster never chosen by planner

2019-10-10 Thread Tomas Vondra
xable" thing here, I think. The other issue is that the estimation of pages fetched using bitmap heap scan is rather crude - but that's simply hard, and I don't think we can fundamentally improve this. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: BRIN index which is much faster never chosen by planner

2019-10-11 Thread Tomas Vondra
'2019-10-01 08:20:38.115471-05'::timestamp with time zone) Let me know if this rings any bells! I will respond to other comments with other replies. My guess - it's (at least partially) due to cpu_operator_cost, associated with the now() call. When replaced with a literal, this cost disappears and so the total query cost decreases. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

CREATE TEXT SEARCH DICTIONARY segfaulting on 9.6+

2019-10-12 Thread Tomas Vondra
e=ispell, DictFile=hunspell_sample_num, AffFile=hunspell_sample_long); But when using the "same" group for both dictfile and afffile, it seems to work just fine. [1] https://www.postgresql.org/message-id/flat/16050-024ae722464ab604%40postgresql.org regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: CREATE TEXT SEARCH DICTIONARY segfaulting on 9.6+

2019-10-14 Thread Tomas Vondra
LAG num while with FLAG long it works just fine. But I'm not sure that's actually possible, because I don't see anything in hunspell_sample_num.dict that would allow us to decide that it expects "FLAG num" and not "FLAG long". Furthermore, we certainly can't r

Re: Add Change Badges to documentation

2019-10-18 Thread Tomas Vondra
pages to see how badges would work in those contexts. Haven't looked yet, but I agree the colors might need a change - that's a rather minor detail, though. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Bug about drop index concurrently

2019-10-18 Thread Tomas Vondra
s the root cause is pretty simple - we close/unlock the indexes after completing the query, but then EXPLAIN tries to open it again when producing the explain plan. I don't have a very good idea how to fix this, as explain has no idea which indexes will be used by the query, etc. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Backport "WITH ... AS MATERIALIZED" syntax to <12?

2019-10-19 Thread Tomas Vondra
avior on 12, and then they'll get confused on older releases (e.g. if you don't specify AS MATERIALIZED you'd expect the CTE to get inlined, but that won't happen). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Backport "WITH ... AS MATERIALIZED" syntax to <12?

2019-10-19 Thread Tomas Vondra
st, which essentially just allowed users to postpone the fix indefinitely, and increased our maintenance burden. I wonder if an extension could do something like that, though. It can install a hook after parse analysis, so I guess it could walk the CTEs and mark them as materialized. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Compressed pluggable storage experiments

2019-10-19 Thread Tomas Vondra
e (WOS) and read-optimized store (ROS), where the WOS is mostly just an uncompressed append-only buffer, and ROS is compressed etc. ISTM the WOS would benefit from a more elaborate WAL logging, but ROS should be mostly fine with the generic WAL logging. But yeah, we should test and measure how benefici

Re: jsonb_set() strictness considered harmful to data

2019-10-19 Thread Tomas Vondra
ot;...", NULL) should do the same thing as jsonb_set(..., "...", 'null':jsonb) I'm not entirely surprised it's what MySQL does ;-) but I'd say treating it as a deletion of the key (just like MSSQL) is somewhat more sensible. But I admit it's quit

Re: jsonb_set() strictness considered harmful to data

2019-10-19 Thread Tomas Vondra
On Sat, Oct 19, 2019 at 12:47:39PM -0400, Andrew Dunstan wrote: On 10/19/19 12:32 PM, David G. Johnston wrote: On Sat, Oct 19, 2019 at 9:19 AM Tomas Vondra mailto:tomas.von...@2ndquadrant.com>> wrote: > >We invented jsonb_set() (credit to Dmitry Dolgov). And we've

Re: jsonb_set() strictness considered harmful to data

2019-10-20 Thread Tomas Vondra
onb_set and NULL in the archives), but I'm not sure that's enough to justify any changes in backbranches. I'd say no, but I have no idea how many people are affected by this but don't know about it ... regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: jsonb_set() strictness considered harmful to data

2019-10-20 Thread Tomas Vondra
On Sun, Oct 20, 2019 at 06:51:05PM -0400, Andrew Dunstan wrote: On 10/20/19 4:18 PM, Tomas Vondra wrote: On Sun, Oct 20, 2019 at 03:48:05PM -0400, Andrew Dunstan wrote: On 10/20/19 1:14 PM, David G. Johnston wrote: On Sun, Oct 20, 2019 at 5:31 AM Andrew Dunstan mailto:andrew.duns

Re: jsonb_set() strictness considered harmful to data

2019-10-21 Thread Tomas Vondra
On Mon, Oct 21, 2019 at 08:06:46AM -0700, Adrian Klaver wrote: On 10/20/19 11:07 PM, Tomas Vondra wrote: On Sun, Oct 20, 2019 at 06:51:05PM -0400, Andrew Dunstan wrote: True. And AFAIK catching exceptions is not really possible in some code, e.g. in stored procedures (because we can'

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-10-22 Thread Tomas Vondra
On Tue, Oct 22, 2019 at 10:30:16AM +0530, Dilip Kumar wrote: On Fri, Oct 18, 2019 at 5:32 PM Amit Kapila wrote: On Mon, Oct 14, 2019 at 3:09 PM Dilip Kumar wrote: > > On Thu, Oct 3, 2019 at 4:03 AM Tomas Vondra > wrote: > > > > > > Sure, I wasn't really pr

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-10-22 Thread Tomas Vondra
5ce80e-f536-78c8-d5c8-a5df3e230785%40postgrespro.ru I think the patch should do the simplest thing possible, i.e. what it does today. Otherwise we'll never get it committed. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: 回复:Bug about drop index concurrently

2019-10-22 Thread Tomas Vondra
so we gave up that. I don't understand? What method? If you have a better solution in the future, please push it to the new version, or email it, thank you very much. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: 回复:Bug about drop index concurrently

2019-10-22 Thread Tomas Vondra
does seem like a bug, i.e. something we need to fix. Not sure what/how we could ban? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Bug about drop index concurrently

2019-10-22 Thread Tomas Vondra
On Fri, Oct 18, 2019 at 05:00:54PM +0200, Tomas Vondra wrote: Hi, I can trivially reproduce this - it's enough to create a master-standby setup, and then do this on the master CREATE TABLE t (a int, b int); INSERT INTO t SELECT i, i FROM generate_series(1,1) s(i); and run pgbench

Re: 回复:回复:Bug about drop index concurrently

2019-10-23 Thread Tomas Vondra
r if we could invalidate the relcache for the relation at some point. Or maybe we could add additional information to the WAL to make the redo wait for all lock waiters, just like on the master. But that might be tricky because of deadlocks, and because the redo could easily get "stuck&q

Re: Creating foreign key on partitioned table is too slow

2019-10-24 Thread Tomas Vondra
005 0.22% 0.00% postgres [unknown] [.] 0.18% 0.18% postgres postgres[.] AllocSetCheck ... Haven't looked into the details yet. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development,

Re: Creating foreign key on partitioned table is too slow

2019-10-24 Thread Tomas Vondra
On Fri, Oct 25, 2019 at 12:17:58AM +0200, Tomas Vondra wrote: ... FWIW, even with this fix it still takes an awful lot to create the foreign key, because the CPU is stuck doing this 60.78%60.78% postgres postgres[.] bms_equal 32.58%32.58% postgres postgres

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-02-28 Thread Tomas Vondra
gs(), but then it switched to heap_form_tuple() without building a valid array. I've decided to simply revert back to BuildTupleFromCStrings(). It's not going to be used very frequently, so the small performance difference is not important. I've also fixed the formatting issues, point

Re: FETCH FIRST clause PERCENT option

2019-02-28 Thread Tomas Vondra
On 2/28/19 12:26 PM, Kyotaro HORIGUCHI wrote: > Hello. > > At Sat, 23 Feb 2019 22:27:44 +0100, Tomas Vondra > wrote in > <81a5c0e9-c17d-28f3-4647-8a4659cdf...@2ndquadrant.com> >> >> >> On 2/23/19 8:53 AM, Surafel Temesgen wrote: >>> >

Re: Index Skip Scan

2019-02-28 Thread Tomas Vondra
stupid joke that occurred to me when I saw that > DB2 had gone for "jump scan". I think "skip scan" is a perfectly good > name and it's pretty widely used by now (for example, by our friends > over at SQLite to blow us away at these kinds of queries). > +1 to "hop scan" regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Why don't we have a small reserved OID range for patch revisions?

2019-02-28 Thread Tomas Vondra
check? That is, a list of stuff that is expected to be done by a committer before a commit? I do recall we have [1], but perhaps we have something else. https://wiki.postgresql.org/wiki/Committing_checklist regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: FETCH FIRST clause PERCENT option

2019-03-01 Thread Tomas Vondra
On 3/1/19 2:31 AM, Kyotaro HORIGUCHI wrote: > Hello. > > At Thu, 28 Feb 2019 21:16:25 +0100, Tomas Vondra > wrote in > >>> One biggest issue seems to be we don't know the total number of > > # One *of* the biggest *issues*? > >>> outer tuple

Re: WIP: BRIN multi-range indexes

2019-03-02 Thread Tomas Vondra
On 3/2/19 10:05 AM, Alexander Korotkov wrote: > On Sun, Mar 4, 2018 at 3:15 AM Tomas Vondra > wrote: >> I've been thinking about this after looking at 0a459cec96, and I don't >> think this patch has the same issues. One reason is that just like the >> original

Re: WIP: BRIN multi-range indexes

2019-03-02 Thread Tomas Vondra
On 3/2/19 10:00 AM, Alexander Korotkov wrote: > Hi! > > I'm starting to look at this patchset. In the general, I think it's > very cool! We definitely need this. > > On Tue, Apr 3, 2018 at 10:51 PM Tomas Vondra > wrote: >> 1) index parameters >>

Re: Online verification of checksums

2019-03-02 Thread Tomas Vondra
w data. This is partial write, most likely because we read the blocks one by one, and when we hit the last page while the table is being extended, we may only see the fist 4kB. And if we retry very fast, we may still see only the first 4kB. regards -- Tomas Vondra http://www.2n

Re: Online verification of checksums

2019-03-02 Thread Tomas Vondra
se it's a data corruption), or an extension in progress? I wonder if we can simply ignore those errors entirely, if it's the last page in the segment? We can't really check the file is "complete" anyway, e.g. if you have multiple segments for a table, and the "middle" one is a page shorter, we'll happily ignore that during verification. Also, what if we're reading a file and it gets truncated (e.g. after vacuum notices the last few pages are empty)? Doesn't that have the same issue? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Online verification of checksums

2019-03-02 Thread Tomas Vondra
ou and Andres may be right that trying to verify checksums online without close interaction with the server is ultimately futile (or at least overly complex). But I'm not sure those issues (torn pages and partial reads) are very good arguments, considering basebackup has to deal with them too. Not s

Re: jsonpath

2019-03-03 Thread Tomas Vondra
at we don't do this already? Or is it not needed for some reason? jsonpath.c == I suppose this should say "jsonpath version number" instead? elog(ERROR, "unsupported jsonb version number %d", version); regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Online verification of checksums

2019-03-04 Thread Tomas Vondra
any differently than direct access (i.e. what the tool does now). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Online verification of checksums

2019-03-04 Thread Tomas Vondra
On 3/4/19 2:00 AM, Michael Paquier wrote: > On Sun, Mar 03, 2019 at 03:12:51AM +0100, Tomas Vondra wrote: >> You and Andres may be right that trying to verify checksums online >> without close interaction with the server is ultimately futile (or at >> least overly compl

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-03-04 Thread Tomas Vondra
default, then allow for specific cases. It's much easier to reason about, and also validate such solutions. It's pretty much the same reason why firewall rules generally prohibit everything by default, and then only allow access for specific ports, from specific IP ranges, etc. Doing i

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-03-04 Thread Tomas Vondra
7;d allow us to (a) avoid making the same mistakes and (b) build a solution the users are already somewhat familiar with. May I suggest creating a page on the PostgreSQL wiki, explaining the design and updating it as the discussion develops? It's rather difficult to follow all the different sub-thre

Re: jsonpath

2019-03-04 Thread Tomas Vondra
ing of some of the error messages in the execute methods seems a bit odd. For example executeNumericItemMethod may complain that it ... is applied to not a numeric value but perhaps a more natural wording would be ... is applied to a non-numeric value And similarly for the other e

Re: Should we increase the default vacuum_cost_limit?

2019-03-05 Thread Tomas Vondra
no intention to block it (because in most cases I do actually increase the value anyway). I wonder if those with small systems will be happy about it, though. But on the other hand it feels a bit weird that we increase this one value and leave all the other (also very conservative) defaults alone.

Re: Online verification of checksums

2019-03-05 Thread Tomas Vondra
On 3/5/19 4:12 AM, Michael Paquier wrote: > On Mon, Mar 04, 2019 at 03:08:09PM +0100, Tomas Vondra wrote: >> I still don't understand what issue you see in how basebackup verifies >> checksums. Can you point me to the explanation you've sent after 11 was >> released

Re: Delay locking partitions during query execution

2019-03-05 Thread Tomas Vondra
hallow (I'd say 2-3 levels will cover like 95% of cases, and 4 levels would be 100% in practice). And the second lookup should be fairly cheap thanks to syscache and the fact that the hierarchies do not change very often. I can't judge how invasive this patch would be, but I agree it's more complex than the originally proposed patch. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Make drop database safer

2019-03-06 Thread Tomas Vondra
directory 7. Commit Transaction I don't see how that actually fixes any of the issues? Can you explain? Not to mention we might end up doing quite a bit of I/O to checkpoint buffers from the database that is going to disappear shortly ... regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: performance issue in remove_from_unowned_list()

2019-03-06 Thread Tomas Vondra
On 3/6/19 7:52 PM, Alvaro Herrera wrote: > On 2019-Feb-08, Tomas Vondra wrote: > >> I'm wondering if we should just get rid of all such optimizations, and >> make the unowned list doubly-linked (WIP patch attached, needs fixing >> the comments etc.). > >

Re: Online verification of checksums

2019-03-06 Thread Tomas Vondra
On 3/6/19 6:26 PM, Robert Haas wrote: > On Sat, Mar 2, 2019 at 4:38 PM Tomas Vondra > wrote: >> FWIW I don't think this qualifies as torn page - i.e. it's not a full >> read with a mix of old and new data. This is partial write, most likely >> because we read

Re: Online verification of checksums

2019-03-06 Thread Tomas Vondra
seems to be to IO lock the page via PG after such a > failure, and then retry. Which should be relatively easily doable for > the basebackup case, but obviously harder for the pg_verify_checksums > case. > Yes, if we could ensure the retry happens after completing the current I/O

Re: performance issue in remove_from_unowned_list()

2019-03-06 Thread Tomas Vondra
On 3/6/19 8:04 PM, Robert Haas wrote: > On Wed, Mar 6, 2019 at 1:53 PM Alvaro Herrera > wrote: >> On 2019-Feb-08, Tomas Vondra wrote: >>> I'm wondering if we should just get rid of all such optimizations, and >>> make the unowned list doubly-linked (WIP p

Re: Online verification of checksums

2019-03-06 Thread Tomas Vondra
On 3/6/19 8:41 PM, Andres Freund wrote: > Hi, > > On 2019-03-06 20:37:39 +0100, Tomas Vondra wrote: >> Not sure how to integrate it into the CLI tool, though. Perhaps we it >> could require connection info so that it can execute a function, when >> executed in online

Re: Should we increase the default vacuum_cost_limit?

2019-03-06 Thread Tomas Vondra
On 3/6/19 12:10 AM, David Rowley wrote: > Thanks for chipping in on this. > > On Wed, 6 Mar 2019 at 01:53, Tomas Vondra > wrote: >> But on the other hand it feels a bit weird that we increase this one >> value and leave all the other (also very conservative) defaults

Re: Protect syscache from bloating with negative cache entries

2019-03-06 Thread Tomas Vondra
eed with that. But I haven't seen any further discussion about that. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Online verification of checksums

2019-03-07 Thread Tomas Vondra
, and then just evict it right away (not to spoil sb) Or did you have something else in mind? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Protect syscache from bloating with negative cache entries

2019-03-07 Thread Tomas Vondra
On 3/7/19 3:34 PM, Robert Haas wrote: > On Wed, Mar 6, 2019 at 6:18 PM Tomas Vondra > wrote: >> I agree clock sweep might be sufficient, although the benchmarks done in >> this thread so far do not suggest the LRU approach is very expensive. > > I'm not sure ho

Re: Protect syscache from bloating with negative cache entries

2019-03-07 Thread Tomas Vondra
On 3/7/19 4:01 PM, Robert Haas wrote: > On Thu, Mar 7, 2019 at 9:49 AM Tomas Vondra > wrote: >> I don't think this shows any regression, but perhaps we should do a >> microbenchmark isolating the syscache entirely? > > Well, if we need the LRU list, then yeah I thin

Re: Online verification of checksums

2019-03-08 Thread Tomas Vondra
On 3/8/19 4:19 PM, Julien Rouhaud wrote: > On Thu, Mar 7, 2019 at 7:00 PM Andres Freund wrote: >> >> On 2019-03-07 12:53:30 +0100, Tomas Vondra wrote: >>> >>> But then again, we could just >>> hack a special version of ReadBuffer_common() which would ju

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-03-08 Thread Tomas Vondra
key. So sessions performing logical decoding (which are regular user sessions) would know the WAL key, which gives them the ability to decode everything. So if the threat model includes insider thread (someone with access to a subset of data, gaining unauthorized access to everything), then thi

Re: Hash index initial size is too large given NULLs or partial indexes

2019-03-08 Thread Tomas Vondra
nd selectivity of the index predicate. When those two are redundant (i.e. when there's IS NOT NULL condition on indexed column), this will result in under-estimate. That means the index build will do a an extra split, but that's probably better than having permanently bloated index. regards -

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-10 Thread Tomas Vondra
ave the problem pointed out before where it > can't apply both MCV and histogram stats if they're on different > STATISTICS objects. I agree clauselist_selectivity() shouldn't care about various types of extended statistics (MCV vs. functional dependencies). But I'

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-10 Thread Tomas Vondra
On 3/10/19 11:27 PM, David Rowley wrote: > On Mon, 11 Mar 2019 at 06:36, Tomas Vondra > wrote: >> >> On 3/9/19 7:33 PM, Dean Rasheed wrote: >>> I wonder if it's possible to write smaller, more targeted tests. >>> Currently "stats_ext" is by

Re: WIP: BRIN multi-range indexes

2019-03-12 Thread Tomas Vondra
NULLs processing according to patch 0003. I also noticed that the > following functions contain a lot of duplicated code, which needs to be > extracted into common subroutine: > inclusion_get_procinfo() > bloom_get_procinfo() > minmax_multi_get_procinfo() > Yes. The reason for the

Re: performance issue in remove_from_unowned_list()

2019-03-12 Thread Tomas Vondra
On 3/10/19 9:09 PM, Alvaro Herrera wrote: > On 2019-Feb-07, Tomas Vondra wrote: > >> Attached is a WIP patch removing the optimization from DropRelationFiles >> and adding it to smgrDoPendingDeletes. This resolves the issue, at least >> in the cases I've been ab

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-12 Thread Tomas Vondra
ap in statext_is_compatible_clause_internal when processing AND/OR/NOT clauses. I plan to look into those items next, but I don't want block review of other parts of the patch unnecessarily. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support,

Re: WIP: BRIN multi-range indexes

2019-03-13 Thread Tomas Vondra
On 3/13/19 9:15 AM, Alexander Korotkov wrote: > On Tue, Mar 12, 2019 at 8:15 PM Tomas Vondra > wrote: >>> 0001. Pass all keys to BRIN consistent function at once. >>> >>> I think that changing the signature of consistent function is bad, because >>

Re: Compressed TOAST Slicing

2019-03-13 Thread Tomas Vondra
erting me of places that may require attention. And I'm with Andres here about the complexity being rather unwarranted here - I don't think we've changed pglz API in years (if ever), so what is the chance we'd actually benefit from the extensibility soon? regards -- Tomas Vondra

Re: performance issue in remove_from_unowned_list()

2019-03-13 Thread Tomas Vondra
On 3/13/19 1:12 PM, Robert Haas wrote: > On Tue, Mar 12, 2019 at 6:54 PM Tomas Vondra > wrote: >> Attached is a patch adopting the dlist approach - it seems to be working >> quite fine, and is a bit cleaner than just slapping another pointer into >> the SMgrRelationData st

seems like a bug in pgbench -R

2019-03-14 Thread Tomas Vondra
eduled = 699536519281, sleep_until = 0, txn_begin = {tv_sec = 699536, tv_nsec = 518478603}, stmt_begin = {tv_sec = 0, tv_nsec = 0}, prepared = {false }, cnt = 132, ecnt = 0} So I guess this is a bug in 12788ae49e1933f463bc59a6efe46c4a01701b76, or one of the other commits touching this part

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-14 Thread Tomas Vondra
ot;: 1.00, "2 => 1": 1.00} (1 row) That is, we don't remove the statistics, but the estimate still changes. But that's because the ALTER TABLE also resets reltuples/relpages: select relpages, reltuples from pg_class where relname = 't'; relpages | relt

Re: seems like a bug in pgbench -R

2019-03-15 Thread Tomas Vondra
t;> >> Indeed. I'll look at it over the weekend. >> >>> So I guess this is a bug in 12788ae49e1933f463bc59a6efe46c4a01701b76, or >>> one of the other commits touching this part of the code. > > I could not reproduce this issue on head, but I confirm on 11.2. &

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-16 Thread Tomas Vondra
On 3/16/19 11:55 AM, Dean Rasheed wrote: > On Fri, 15 Mar 2019 at 00:06, Tomas Vondra > wrote: >> I've noticed an annoying thing when modifying type of column not >> included in a statistics... >> >> That is, we don't remove the statistics, but the estim

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-16 Thread Tomas Vondra
On 3/16/19 10:26 PM, Dean Rasheed wrote: > On Fri, 15 Mar 2019 at 00:06, Tomas Vondra > wrote: >> ... attached patch ... > > Some more review comments, carrying on from where I left off: > > 16). This regression test fails for me: > > @@ -654,11 +654,11 @@ &

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-17 Thread Tomas Vondra
Hi, On 3/17/19 12:47 PM, Dean Rasheed wrote: > On Sat, 16 Mar 2019 at 23:44, Tomas Vondra > wrote: >> >>> 28). I just spotted the 1MB limit on the serialised MCV list size. I >>> think this is going to be too limiting. For example, if the stats >>> targe

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-17 Thread Tomas Vondra
On 3/17/19 1:14 PM, Dean Rasheed wrote: > On Sat, 16 Mar 2019 at 23:44, Tomas Vondra > wrote: >>> >>> 16). This regression test fails for me: >>> >>> @@ -654,11 +654,11 @@ >>> -- check change of unrelated column type does not reset the MCV s

Re: [HACKERS] Custom compression methods

2019-03-18 Thread Tomas Vondra
b. Which is not great, and so people tend to pick other patches. Now, I understand there's a lot of potential benefits further down the line, like column-level compression (which I think is the main goal here). But that's not included in the patch, so the gains are somewhat far in t

Re: [HACKERS] Custom compression methods

2019-03-19 Thread Tomas Vondra
On 3/19/19 10:59 AM, Chris Travers wrote: > > > On Mon, Mar 18, 2019 at 11:09 PM Tomas Vondra > mailto:tomas.von...@2ndquadrant.com>> wrote: > > > > On 3/15/19 12:52 PM, Ildus Kurbangaliev wrote: > > On Fri, 15 Mar 2019 14:07:14 +0400 > &g

Re: PostgreSQL pollutes the file system

2019-03-20 Thread Tomas Vondra
o they come from. But it's not really an issue, because we have tools to do that 1) man 2) -h/--help 3) rpm -qf $file (and similarly for other packagers) 4) set --prefix to install binaries so separate directory (which some distros already do anyway) So to me this seems like a fairly inva

Re: performance issue in remove_from_unowned_list()

2019-03-20 Thread Tomas Vondra
On 3/12/19 11:54 PM, Tomas Vondra wrote: > > > On 3/10/19 9:09 PM, Alvaro Herrera wrote: >> On 2019-Feb-07, Tomas Vondra wrote: >> >>> Attached is a WIP patch removing the optimization from DropRelationFiles >>> and adding it to smgrDoPendingDeletes. This

Re: PostgreSQL pollutes the file system

2019-03-20 Thread Tomas Vondra
gt; (I have 1600 entries in /usr/bin on a Debian installation.) > Maybe. Do we actually know about such cases? Also, isn't setting --prefix a suitable solution? I mean, it's what we/packagers do to support installing multiple Pg versions (in which case it'll conflict no matter

Re: [HACKERS] Custom compression methods

2019-03-21 Thread Tomas Vondra
On 3/19/19 4:44 PM, Chris Travers wrote: > > > On Tue, Mar 19, 2019 at 12:19 PM Tomas Vondra > mailto:tomas.von...@2ndquadrant.com>> wrote: > > > On 3/19/19 10:59 AM, Chris Travers wrote: > > > > > > Not discussing whether any p

Re: Enable data checksums by default

2019-03-22 Thread Tomas Vondra
gt; and neither basebackup nor backend checks detect zeroed out files/file > ranges. > Yep :-( The pg_basebackup vulnerability to random garbage in a page header is unfortunate, we better improve that. It's not clear to me what can checksums do about zeroed pages (and/or truncated files) though. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Enable data checksums by default

2019-03-22 Thread Tomas Vondra
On 3/22/19 5:41 PM, Andres Freund wrote: > Hi, > > On 2019-03-22 17:32:10 +0100, Tomas Vondra wrote: >> On 3/22/19 5:10 PM, Andres Freund wrote: >>> IDK, being able to verify in some form that backups aren't corrupted on >>> an IO level is mighty nice. Tha

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-23 Thread Tomas Vondra
On 3/21/19 4:05 PM, David Rowley wrote: > On Mon, 18 Mar 2019 at 02:18, Tomas Vondra > wrote: >> Yes, it was using the toasted value directly. The attached patch >> detoasts the value explicitly, similarly to the per-column stats, and it >> also removes the 1MB limit.

<    3   4   5   6   7   8   9   10   11   12   >