Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Tomas Vondra
harding is entirely independent to the fact that these solutions seem to work quite well. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_dump / copy bugs with "big lines" ?

2016-03-02 Thread Tomas Vondra
mpty file, so there are more problems to solve to go beyond 2GB text per row. My guess is this is a problem at the protocol level - the 'd' message is CopyData, and all the messages use int32 to define length. So if there's a 2GB row, it's likely to overflow. regards -- T

Re: [HACKERS] pg_dump / copy bugs with "big lines" ?

2016-03-02 Thread Tomas Vondra
On 03/02/2016 04:23 PM, Tom Lane wrote: Tomas Vondra writes: On 03/02/2016 03:18 PM, Daniel Verite wrote: However, getting it to the client with \copy big2 to 'file' still produces the error in psql: lost synchronization with server: got message type "d" and leaves an

Re: [HACKERS] multivariate statistics v10

2016-03-02 Thread Tomas Vondra
{ column_name } [, ... ]) [ WITH ( statistics_parameter [= value] [, ... ] ) ] Yeah, will fix. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Re: PATCH: Split stats file per database WAS: autovacuum stress-testing our system

2016-03-02 Thread Tomas Vondra
Hi, On 02/03/2016 06:46 AM, Noah Misch wrote: On Mon, Feb 01, 2016 at 07:03:45PM +0100, Tomas Vondra wrote: On 12/22/2015 03:49 PM, Noah Misch wrote: On Mon, Feb 18, 2013 at 06:19:12PM -0300, Alvaro Herrera wrote: I have pushed it now. Further testing, of course, is always welcome. While

Re: [HACKERS] improving GROUP BY estimation

2016-03-03 Thread Tomas Vondra
verage_rows_per_value = (3 / 2) = 1.5; min_estimate = ceil(2 / 1.5) = 2; Of course, it's just a heuristics, so this may fail in some other cases. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Service

Re: [HACKERS] improving GROUP BY estimation

2016-03-03 Thread Tomas Vondra
Hi, On 03/03/2016 06:40 PM, Mark Dilger wrote: On Mar 3, 2016, at 8:35 AM, Tomas Vondra wrote: Hi, On 03/03/2016 12:53 PM, Alexander Korotkov wrote: Hi, Tomas! I've assigned to review this patch. I've checked version estimate-num-groups-v2.txt by Mark Dilger. It applies to he

Re: [HACKERS] WIP: Upper planner pathification

2016-03-04 Thread Tomas Vondra
a diagonal (no change) or above it (speedup), but the points are actually below. See the chart attached. But I do agree this is mostly at the noise level, pretty good for a first cut that intentionally does not include any hacks. It's definitely way below the benefits of this patch, +1

Re: [HACKERS] improving GROUP BY estimation

2016-03-04 Thread Tomas Vondra
On Thu, 2016-03-03 at 11:42 -0800, Mark Dilger wrote: > > On Mar 3, 2016, at 11:27 AM, Alexander Korotkov > > wrote: > > > > On Thu, Mar 3, 2016 at 10:16 PM, Tomas Vondra > > wrote: > > So yes, each estimator works great for exactly the opposite cases.

Re: [HACKERS] Static code checker research worth investigating (Communications of the ACM, 03/2016, Vol. 59, No. 03, p. 99)

2016-03-05 Thread Tomas Vondra
lished a few years back (2013?) along with a paper that also mentioned a few issues in PostgreSQL. And it was dealt with, see for example this thread http://www.postgresql.org/message-id/flat/20130715215950.ga4...@eldon.alvh.no-ip.org Or is this something new? regards -- Tomas Vondra

Re: [HACKERS] More stable query plans via more predictable column statistics

2016-03-07 Thread Tomas Vondra
the fixes won't change the plans, we kinda try to minimize such cases. That being said, I don't have a clear opinion whether to backpatch this. I think that it's clearly a bug (especially the first part dealing with NULL values), and it'd be good to backpatch that. OTOH I can'

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-03-08 Thread Tomas Vondra
t I've split the patch into two. One for the infrastructure, and > one for the callsites. I've repeated the power-loss testing today. With the patches applied I'm not longer able to reproduce the issue (despite trying about 10x), while without them I've hit it on the fir

Re: [HACKERS] More stable query plans via more predictable column statistics

2016-03-09 Thread Tomas Vondra
he planner to use default selectivity estimates (~5% for equality expressions, IIRC), not the per-column statistics. Combined with the independence assumption for multiple AND clauses, this may easily confuse the planner. I'd expect that would affect both indexes equally, but perhaps not. Would be useful to see explain analyze - it's not clear what you mean by "the number of rows matching WHERE were tiny", whether an estimate or the actual number. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More stable query plans via more predictable column statistics

2016-03-09 Thread Tomas Vondra
y seems to be (nonnull_cnt==toowide_cnt). Why not to use this condition instead? FWIW while looking at the code I noticed that we skip wide varlena values but not cstrings. Seems a bit suspicious. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Su

Re: [HACKERS] More stable query plans via more predictable column statistics

2016-03-09 Thread Tomas Vondra
On Wed, 2016-03-09 at 12:02 -0300, Alvaro Herrera wrote: > Tomas Vondra wrote: > > > FWIW while looking at the code I noticed that we skip wide varlena > > values but not cstrings. Seems a bit suspicious. > > Uh, can you actually have columns of cstring type? I don

Re: [HACKERS] multivariate statistics v14

2016-03-09 Thread Tomas Vondra
Hi, On Wed, 2016-03-09 at 08:45 -0800, Jeff Janes wrote: > On Wed, Mar 9, 2016 at 7:02 AM, Tomas Vondra > wrote: > > Hi, > > > > thanks for the feedback. Attached is v14 of the patch series, fixing > > most of the points you've raised. > > > Hi Toma

Re: [HACKERS] multivariate statistics v14

2016-03-09 Thread Tomas Vondra
On Wed, 2016-03-09 at 18:21 +0100, Tomas Vondra wrote: > Hi, > > On Wed, 2016-03-09 at 08:45 -0800, Jeff Janes wrote: > > On Wed, Mar 9, 2016 at 7:02 AM, Tomas Vondra > > wrote: > > > Hi, > > > > > > thanks for the feedback. Attached is v14 of t

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2016-03-09 Thread Tomas Vondra
++) nulls[i+3] = true; Now let's say PARAM=10, which means COLS=13. The last index accessed by the loop will be i=10, which means we'll do this: nulls[13] = true; which is above bounds. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL

Re: [HACKERS] POC, WIP: OR-clause support for indexes

2016-03-10 Thread Tomas Vondra
;::integer[])) (2 rows) So the OR-clause is estimated to match 0 rows, less than each of the clauses independently. Needless to say that without the patch this works just fine. 10) Also, this already breaks some regression tests, apparently because it changes how 'width' is computed. So

Re: [HACKERS] Using quicksort for every external sort run

2016-03-10 Thread Tomas Vondra
. > > Attached is a revision that significantly overhauls the memory patch, > with several smaller changes. I was thinking about running some benchmarks on this patch, but the thread is pretty huge so I want to make sure I'm not missing something and this is indeed the most recent

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-10 Thread Tomas Vondra
few minor comments: 1) This should probably say "one block per level": /* * A B-Tree cannot possibly have this many levels, since there must be * one block per page, and that is bound by the range of BlockNumber: */ 2) comment before bt_check_every_level() says: ... assumed to prevent any k

Re: [HACKERS] auto_explain sample rate

2016-03-11 Thread Tomas Vondra
ple_ratio) as that's what pgbench uses for the same thing. That'd be more consistent. regards -- Tomas Vondra  http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Tomas Vondra
checksum, but broken page. Or perhaps issues coming from outside of PostgreSQL, like a subtle change in the locales. Simulating those issues requires a tool that allows minor tweaks to the page, dd is a bit too dull for that. regards -- Tomas Vondra  http://www.2ndQuadrant.com P

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Tomas Vondra
e as much as possible, etc. Currently this is tedious because we essentially find/fix the pages one by one. It'd be very useful to list all broken pages in one go and then fix all of them. Obviously, that's about heapam checks, but perhaps it would be useful for an index too? regard -

Re: [HACKERS] Refectoring of receivelog.c

2016-03-12 Thread Tomas Vondra
e setting all the fields explicitly, but gcc is nagging about it. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make ch

Re: [HACKERS] improving GROUP BY estimation

2016-03-13 Thread Tomas Vondra
Hi, On Sun, 2016-03-13 at 15:24 +, Dean Rasheed wrote: > On 4 March 2016 at 13:10, Tomas Vondra > wrote: > > > > The risk associated with over-estimation is that we may switch from > > HashAggregate to GroupAggregate, and generally select paths better > > s

Re: [HACKERS] Re: PATCH: Split stats file per database WAS: autovacuum stress-testing our system

2016-03-13 Thread Tomas Vondra
On Sun, 2016-03-13 at 18:46 -0400, Noah Misch wrote: > On Thu, Mar 03, 2016 at 06:08:07AM +0100, Tomas Vondra wrote: > > > > On 02/03/2016 06:46 AM, Noah Misch wrote: > > > > > > On Mon, Feb 01, 2016 at 07:03:45PM +0100, Tomas Vondra wrote: > > > >

Re: [HACKERS] Re: PATCH: Split stats file per database WAS: autovacuum stress-testing our system

2016-03-14 Thread Tomas Vondra
y the line wrapping algorithm is broken. I've switched back to Thunderbird, so hopefully that'll fix it. On Mon, Mar 14, 2016 at 02:00:03AM +0100, Tomas Vondra wrote: On Sun, 2016-03-13 at 18:46 -0400, Noah Misch wrote: I've not attempted to study the behavior on slow hardware. Ins

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2016-03-14 Thread Tomas Vondra
in a message from 24/2/2015, while this is definitely an issue worth fixing, I believe it needs to be done in some other way, not by foreign keys. Attached is v3 of the patch, and also three SQL scripts demonstrating the impact of the patch on simple examples. regards -- Tomas Vondra

Re: [HACKERS] Parallel Aggregate

2016-03-14 Thread Tomas Vondra
mPartialGroups = numWorkers * workerGroups It's probably better to see Dean's message from 13/3. 4) Is clauses.h the right place for PartialAggType? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Combining Aggregates

2016-03-15 Thread Tomas Vondra
it's related to the type of the internal state and not to the input types. An example of that is the SUM(bigint) example mentioned above. That's actually the one thing I think the patch is missing. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Develop

[HACKERS] incorrect docs for pgbench / skipped transactions

2016-03-18 Thread Tomas Vondra
will contain "skipped" for transactions exceeding the limit latency (it's only mentioned in the example output). So I think we should apply the attached patch (and also backpatch it to 9.5, where the --latency-limit got introduced). regards -- Tomas Vondra

Re: [HACKERS] Re: PATCH: Split stats file per database WAS: autovacuum stress-testing our system

2016-03-18 Thread Tomas Vondra
Hi, On 03/15/2016 03:04 AM, Noah Misch wrote: On Mon, Mar 14, 2016 at 01:33:08PM +0100, Tomas Vondra wrote: On 03/14/2016 07:14 AM, Noah Misch wrote: On Mon, Mar 14, 2016 at 02:00:03AM +0100, Tomas Vondra wrote: +* XXX Maybe this should also care about the clock skew, just like the

Re: [HACKERS] oldest xmin is far in the past

2016-03-18 Thread Tomas Vondra
t change. That probably means there's an old transaction somewhere - either a regular one (check pg_stat_activity) or a prepared one (pg_prepared_xacts). The meaning of "old" depends on autovacuum_freeze_max_age - what value is set in the session running the VACUUM FREEZE? regard

Re: [HACKERS] multivariate statistics v14

2016-03-18 Thread Tomas Vondra
bsets columns (currently we need a perfect match). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Proposal: Generic WAL logical messages

2016-03-19 Thread Tomas Vondra
omment. Not a big deal I guess, the method is simple enough, but why to break the rule when all the other methods around have at least a short one? 4) ReorderBufferChange The new struct in the 'union' would probably deserve at least a short comment explaining the purpose (just like

Re: [HACKERS] checkpointer continuous flushing

2016-03-19 Thread Tomas Vondra
1 second (new) about 99% of the time. So I don't think this would make any measurable difference in practice. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers ma

Re: [HACKERS] checkpointer continuous flushing

2016-03-19 Thread Tomas Vondra
), but then it starts winning for higher values. And that's what counts when it comes to consistency. Again, notice that the x-axis is log scale, so the differences for large values are actually way more significant than it might look. So, good work I guess! regards -- Tomas Vondra

Re: [HACKERS] Combining Aggregates

2016-03-19 Thread Tomas Vondra
n a minor version anyway). I'm not sure the sharding is particularly useful argument at this point, considering we don't really know if the current format is actually a good match for that. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 S

Re: [HACKERS] improving GROUP BY estimation

2016-03-19 Thread Tomas Vondra
On 03/13/2016 11:09 PM, Tomas Vondra wrote: Hi, On Sun, 2016-03-13 at 15:24 +, Dean Rasheed wrote: On 4 March 2016 at 13:10, Tomas Vondra wrote: ... >>> I think that a better formula to use would be reldistinct *= (1 - powl(1 - rel-rows / rel->tuples, rel->tuple

Re: [HACKERS] incorrect docs for pgbench / skipped transactions

2016-03-19 Thread Tomas Vondra
t the timestamp should be made explicit milliseconds, i.e. "123.004567" instead of "123 4567", but that is another question not relevant to this patch. Agreed. -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Trai

Re: [HACKERS] Weighted Stats

2016-03-19 Thread Tomas Vondra
s, unless we can come up with proper definition or at least good examples demonstrating the usefulness. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-ha

Re: [HACKERS] oldest xmin is far in the past

2016-03-19 Thread Tomas Vondra
fault value After looking at the code a bit more, I see it uses some additional configuration options: * freeze_min_age * vacuum_freeze_min_age * autovacuum_freeze_max_age (we already know this one) What values are set for those? regards -- Tomas Vondra http://www.2ndQuadran

Re: [HACKERS] checkpointer continuous flushing

2016-03-19 Thread Tomas Vondra
average, so the tps achieved when discarding late transactions would be under 4000 tps. You mean the 'throttled-tps' chart? Yes, that one shows that without the patches, there's a lot of intervals where the tps was much lower - presumably due to a lot of slow transaction

Re: [HACKERS] FIX : teach expression walker about RestrictInfo

2016-03-19 Thread Tomas Vondra
On 03/18/2016 08:53 PM, Robert Haas wrote: On Fri, Mar 18, 2016 at 3:29 PM, Tom Lane wrote: Robert Haas writes: On Wed, Apr 29, 2015 at 12:33 PM, Tomas Vondra wrote: On 04/29/15 18:26, Tom Lane wrote: But there are basic reasons why expression_tree_walker should not try to deal with

Re: [HACKERS] Combining Aggregates

2016-03-19 Thread Tomas Vondra
have. I've been sticking to *_combine() for these, so maybe float8_combine() and float8_regr_combine() are better names. +1 to the _combine naming convention. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training &am

Re: [HACKERS] POC, WIP: OR-clause support for indexes

2016-03-19 Thread Tomas Vondra
exscan=off will be included multiple times. ... and it does not address this at all. I really doubt a costing derived from the bitmap index scan nodes will make much sense - you essentially need to revert unknown parts of the costing to only include building the bitmap once, etc. regards --

Re: [HACKERS] [PATCH] fix DROP OPERATOR to reset links to itself on commutator and negator

2016-03-20 Thread Tomas Vondra
(OidIsValid(op->oprnegate) && operOid != op->oprnegate)) OperatorUpd(operOid, operOid == op->oprcom ? InvalidOid : op->oprcom, operOid == op->oprnegate ? InvalidOid : op->oprnegate, true); regards -- Tomas Vondra http://www.2ndQu

Re: [HACKERS] PATCH: index-only scans with partial indexes

2016-03-20 Thread Tomas Vondra
hor and myself as a secondary one. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services partial-index-only-scan-v9.patch Description: binary/octet-stream -- Sent via pgsql-hackers mailing list (pgsql-hackers@po

Re: [HACKERS] Parallel Aggregate

2016-03-20 Thread Tomas Vondra
ularly clear comment, I think the old one was way much informative despite being a single line: /* Consider reasons to disable hashing, but only if we can sort instead */ BTW create_grouping_paths probably grew to a size when splitting it into smaller pieces would be helpful? regards -- Tomas

Re: [HACKERS] Combining Aggregates

2016-03-20 Thread Tomas Vondra
Hi, On 03/20/2016 04:48 AM, David Rowley wrote: On 17 March 2016 at 14:25, Tomas Vondra wrote: On 03/16/2016 12:03 PM, David Rowley wrote: Patch 2: This adds the serial/deserial aggregate infrastructure, pg_dump support, CREATE AGGREGATE changes, and nodeAgg.c changes to have it serialise

Re: [HACKERS] Parallel Aggregate

2016-03-20 Thread Tomas Vondra
Hi, On 03/21/2016 12:30 AM, David Rowley wrote: On 21 March 2016 at 09:47, Tomas Vondra wrote: ... I'm not sure changing the meaning of enable_hashagg like this is a good idea. It worked as a hard switch before, while with this change that would not be the case. Or more accurately - it

Re: [HACKERS] multivariate statistics v14

2016-03-21 Thread Tomas Vondra
stics is built on a single table. I don't have a clear idea what should we do in the future with multiple tables (e.g. when the statistics is built on 3 tables, the query is on 2 of them and the user does not have access to the remaining one). But maybe we need to support ACLs

Re: [HACKERS] multivariate statistics v14

2016-03-21 Thread Tomas Vondra
hat and move it to src/include/. In mvstats.h, please mark function prototypes as "extern". Many files need a pgindent pass. OK. -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgs

Re: [HACKERS] checkpointer continuous flushing

2016-03-21 Thread Tomas Vondra
the previous chart, showing how much later the transactions were scheduled. Again, the new code is winning. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pg

Re: [HACKERS] checkpointer continuous flushing

2016-03-22 Thread Tomas Vondra
cheduled. Again, the new code is winning. No brainer again. I infer from this figure that with the initial version 60% of transactions have trouble being processed on time, while this is maybe about 35% with the new version. Yep. -- Tomas Vondra http://www.2ndQuadrant.com PostgreS

Re: [HACKERS] multivariate statistics v14

2016-03-22 Thread Tomas Vondra
Hi, On 03/22/2016 06:53 AM, Jeff Janes wrote: On Sun, Mar 20, 2016 at 4:34 PM, Tomas Vondra wrote: D'oh. Thanks for reporting. Attached is v16, hopefully fixing the few remaining whitespace issues. Hi Tomas, I'm trying out v16 against a common problem, where postgresql th

Re: [HACKERS] multivariate statistics v14

2016-03-22 Thread Tomas Vondra
discussion, but it's mostly about the question whether combining the statistics by using the overlapping clauses as "conditions" is the right thing to do (or whether a more expensive approach is needed). None of that however invalidates the preceding patches. regards -- Tomas Von

Re: [HACKERS] checkpointer continuous flushing

2016-03-22 Thread Tomas Vondra
ng that in this case less transactions where processed. Huh, that seems weird. That's not how percentiles or CDFs work, and I don't quite understand what would that tell us. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Trai

Re: [HACKERS] multivariate statistics v14

2016-03-22 Thread Tomas Vondra
There are other ways to combine the statistics, but I do expect them to be considerably more expensive. Is this a sufficient explanation? Of course, there's a fair amount of additional complexity that I have not mentioned here (e.g. selecting the right combination of stats). regards --

Re: [HACKERS] multivariate statistics v14

2016-03-22 Thread Tomas Vondra
preceding parts, who mentioned he might look at the patch this week. So I'm not sure splitting 0006 into smaller pieces makes sense at this point. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- S

Re: [HACKERS] Using quicksort for every external sort run

2016-03-22 Thread Tomas Vondra
Hi, On 03/22/2016 11:07 PM, Peter Geoghegan wrote: On Tue, Mar 22, 2016 at 2:27 PM, Tomas Vondra wrote: Each query was executed 5x for each work_mem value (between 8MB and 1GB), and then a median of the runs was computed and that's what's on the "comparison". This compares

Re: [HACKERS] [PATCH] fix DROP OPERATOR to reset links to itself on commutator and negator

2016-03-22 Thread Tomas Vondra
perhaps don't remove the empty line at the end of the function (before simple_heap_update). I think the comments will need rewording, but I'll leave that to a native speaker. regards Tomas -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support

Re: [HACKERS] multivariate statistics v14

2016-03-22 Thread Tomas Vondra
de:ref/drop_statistics.sgml:86:26:X: reference to non-existent ID "SQL-ALTERSTATISTICS" I believe this is because reference.sgml is missing a call to &alterStatistic (per report by Alvaro Herrera). thanks -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Developme

Re: [HACKERS] multivariate statistics v14

2016-03-23 Thread Tomas Vondra
are shown in the manual. Am I missing something? No, thanks for spotting those mistakes. I'll fix them and submit a new version of the patch - either later today or perhaps tomorrow. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Re

Re: [HACKERS] Using quicksort for every external sort run

2016-03-23 Thread Tomas Vondra
Hi, On 03/24/2016 03:00 AM, Peter Geoghegan wrote: On Tue, Mar 22, 2016 at 3:35 PM, Tomas Vondra wrote: I've tested the patch you've sent on 2016/3/11, which I believe is the last version. I haven't tuned the replacement_sort_mem at all. because my understanding was that

Re: [HACKERS] multivariate statistics v14

2016-03-25 Thread Tomas Vondra
On 03/24/2016 06:45 PM, Alvaro Herrera wrote: Tomas Vondra wrote: +values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data); Why the double space (that's actually in several places in several of the patches). To align the whole block like this:

Re: [HACKERS] multivariate statistics v14

2016-03-28 Thread Tomas Vondra
er" section. Yes, those are valid omissions. I plan to address them, and I'd also considering adding a section to 65.1 (How the Planner Uses Statistics), explaining more thoroughly how the planner uses multivariate stats. regards -- Tomas Vondra http://www.2ndQuadra

Re: [HACKERS] multivariate statistics v14

2016-03-28 Thread Tomas Vondra
On 03/26/2016 08:09 PM, Alvaro Herrera wrote: Tomas Vondra wrote: There are a few places where I reverted the pgindent formatting, because it seemed a bit too weird - the first one are the lists of function prototypes in common.h/mvstat.h, the second one are function calls to _greedy

Re: [HACKERS] Using quicksort for every external sort run

2016-03-29 Thread Tomas Vondra
7;t done any analysis of how much better that made the problem *across the board* yet, but for int4 cases I could make 1MB work_mem queries faster with gigabytes of data on my laptop. I believe I tested various datum sort cases there, like "select count(distinct(foo)) from bar"; those are

Re: [HACKERS] improving GROUP BY estimation

2016-03-30 Thread Tomas Vondra
used the comment as proposed by Dean. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services estimate-num-groups-v4.patch Description: binary/octet-stream -- Sent via pgsql-hackers mailing list

Re: [HACKERS] PATCH: index-only scans with partial indexes

2016-03-30 Thread Tomas Vondra
condition to decline ripping index predicates from base restrictinfo without understanding the reason to do so. U, I'm a bit confused. Which condition? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Servi

Re: [HACKERS] PATCH: postpone building buckets to the end of Hash (in HashJoin)

2016-01-18 Thread Tomas Vondra
Hi, On 12/17/2015 10:28 PM, Tomas Vondra wrote: Hi, On 12/17/2015 07:20 PM, Robert Haas wrote: ... If this doesn't regress performance in the case where the number of buckets is estimated accurately to begin with, then I think this is a great idea. Can you supply some performance

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread Tomas Vondra
would help the costing problem, but it would also cause hashagg to be a less chosen option during planning. I'm not quite sure I understand - the current code ends up using 8192 for the transition space (per count_agg_clauses_walker). Are you suggesting lowering the value, despite the dange

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread Tomas Vondra
On 01/19/2016 06:04 AM, Tomas Vondra wrote: Hi, On 01/19/2016 05:00 AM, David Rowley wrote: Good question. I looked at this and found my VM was swapping like crazy. Upon investigation it appears that's because, since the patch creates a memory context per aggregated group, and in this

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread Tomas Vondra
ate state (I don't see why would be the case), it's a dead end. Not so long ago we've fixed exactly this issue in array_agg(), which addressed a bunch of reported OOM issues and improved array_agg() performance quite a bit. It'd be rather crazy introducing the same problem t

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-01-18 Thread Tomas Vondra
On 01/19/2016 07:44 AM, Michael Paquier wrote: On Wed, Dec 2, 2015 at 3:24 PM, Michael Paquier wrote: On Wed, Dec 2, 2015 at 3:23 PM, Michael Paquier wrote: On Wed, Dec 2, 2015 at 7:05 AM, Tomas Vondra wrote: Attached is v2 of the patch, that (a) adds explicit fsync on the parent

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-01-18 Thread Tomas Vondra
On 01/19/2016 08:03 AM, Michael Paquier wrote: On Tue, Jan 19, 2016 at 3:58 PM, Tomas Vondra wrote: ... Tomas, I am planning to have a look at that, because it seems to be important. In case it becomes lost on my radar, do you mind if I add it to the 2016-03 CF? Well, what else can I

Re: [HACKERS] PATCH: postpone building buckets to the end of Hash (in HashJoin)

2016-01-19 Thread Tomas Vondra
On 01/19/2016 08:34 PM, Robert Haas wrote: On Mon, Jan 18, 2016 at 10:57 PM, Tomas Vondra wrote: If this doesn't regress performance in the case where the number of buckets is estimated accurately to begin with, then I think this is a great idea. Can you supply some performance tests re

Re: [HACKERS] PATCH: Extending the HyperLogLog API a bit

2016-01-19 Thread Tomas Vondra
unctions, and that's exactly what merge does. So why not just fix the bug? -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To m

Re: [HACKERS] multivariate statistics v8

2016-01-20 Thread Tomas Vondra
On 01/20/2016 10:54 PM, Alvaro Herrera wrote: Bruce Momjian wrote: On Wed, Jan 20, 2016 at 02:20:38PM -0500, Robert Haas wrote: On Wed, Dec 23, 2015 at 2:07 PM, Tomas Vondra wrote: The remaining question is how unique the statistics name should be. My initial plan was to make it

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-01-22 Thread Tomas Vondra
switching back to recovery needlessly). I'd vote for the latter, as I think it makes the database easier to operate (less manual interventions) and the performance impact is 0 (as those fsyncs are really rare). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL D

Re: [HACKERS] Declarative partitioning

2016-01-22 Thread Tomas Vondra
at a proper abstraction (thanks to good syntax) may be a valuable hint how to define the catalogs and such. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgs

Re: [HACKERS] Tsvector editing functions

2016-01-22 Thread Tomas Vondra
On 12/30/2015 06:49 PM, Stas Kelvich wrote: Hi, Tomáš! Thanks for comprehensive review. On 15 Dec 2015, at 06:07, Tomas Vondra wrote: 1) It's a bit difficult to judge the usefulness of the API, as I've always been a mere user of full-text search, and I never had a need (

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-01-22 Thread Tomas Vondra
Hi, On 12/17/2015 02:17 PM, David Rowley wrote: On 17 December 2015 at 19:11, Simon Riggs mailto:si...@2ndquadrant.com>> wrote: On 17 December 2015 at 00:17, Tomas Vondra mailto:tomas.von...@2ndquadrant.com>> wrote: I'd go with match_first_tup

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-01-22 Thread Tomas Vondra
On 01/23/2016 02:35 AM, Michael Paquier wrote: On Fri, Jan 22, 2016 at 9:41 PM, Greg Stark wrote: On Fri, Jan 22, 2016 at 8:26 AM, Tomas Vondra wrote: On 01/22/2016 06:45 AM, Michael Paquier wrote: So, I have been playing with a Linux VM with VMware Fusion and on ext4 with data=ordered the

Re: [HACKERS] More stable query plans via more predictable column statistics

2016-01-23 Thread Tomas Vondra
1000 | 6022 | 6791 So this seems to significantly improve the ndistinct estimate (patch attached). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/src/backend/commands/

Re: [HACKERS] PoC: Partial sort

2016-01-23 Thread Tomas Vondra
p only the "current group" in a hash table, making it much more memory efficient / faster. What do you think? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers m

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-01-25 Thread Tomas Vondra
ments! Tomas -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Add generate_series(date,date) and generate_series(date,date,integer)

2016-01-25 Thread Tomas Vondra
e for example a script that in some rare cases passes happens to pass infinity into generate_series() - in that case I'd much rather error out than wait till the end of the universe. So +1 from me to checking for infinity. regard -- Tomas Vondra http://www.2ndQuadrant.com

Re: [HACKERS] easy way of copying regex_t

2016-01-25 Thread Tomas Vondra
ory, possibly allocating one segment per dictionary as needed, or something like that. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Template for commit messages

2016-01-28 Thread Tomas Vondra
Backpatch through Any reason why not to adapt the git message conventions from kernel? https://git.wiki.kernel.org/index.php/CommitMessageConventions I'd expect there are tools already working with that format, making the life easier for us. regards -- Tomas Vondra

Re: [HACKERS] Template for commit messages

2016-01-28 Thread Tomas Vondra
Dne 28. 1. 2016 11:57 napsal uživatel "Alvaro Herrera" < alvhe...@2ndquadrant.com>: > > Magnus Hagander wrote: > > > They also had tested-by, it might be an idea to include that as well? > > How about > User-Interface-Bikeshedded-By: > ? +1 > > -- > Álvaro Herrerahttp://www.2ndQua

Re: [HACKERS] Template for commit messages

2016-01-28 Thread Tomas Vondra
ee on that he's not prepared to endorse is not going to have much value.) I can't speak for Tom, but I'm sitting fairly close to him and I haven't heard any complains or even groaning. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Develop

Re: [HACKERS] Template for commit messages

2016-01-28 Thread Tomas Vondra
On 01/28/2016 03:37 PM, Robert Haas wrote: On Thu, Jan 28, 2016 at 9:34 AM, Bruce Momjian wrote: On Thu, Jan 28, 2016 at 09:31:46AM -0500, Robert Haas wrote: On Thu, Jan 28, 2016 at 6:16 AM, Tomas Vondra wrote: How about User-Interface-Bikeshedded-By: ? +1 That's sort of impli

Re: [HACKERS] PATCH: track last known XLOG segment in control file

2016-01-31 Thread Tomas Vondra
s "rejected" but that's likely to eliminate any further feedback / discussion about the protection in general. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-h

Re: [HACKERS] Re: PATCH: Split stats file per database WAS: autovacuum stress-testing our system

2016-02-01 Thread Tomas Vondra
iry is X seconds old. I don't have a more-specific design in mind, though. That's a nice idea, but I agree that binding the coalescing to buffer like this seems like a pretty bad idea exactly because of the starving. What might work though is if we could look at how much dat

Re: [HACKERS] Re: PATCH: Split stats file per database WAS: autovacuum stress-testing our system

2016-02-03 Thread Tomas Vondra
On 02/03/2016 06:46 AM, Noah Misch wrote: On Mon, Feb 01, 2016 at 07:03:45PM +0100, Tomas Vondra wrote: On 12/22/2015 03:49 PM, Noah Misch wrote: ... If the collector entered pgstat_write_statsfile() with more inquiries waiting in its socket receive buffer, it would ignore them as being too

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-02-05 Thread Tomas Vondra
ven notice that. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tomas Vondra
e code changes than the fix), or we'd have to revert all the hashjoin goodies. FWIW I'm willing to put some time into fixing this over the weekend. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Servic

  1   2   3   4   5   6   7   8   9   10   >