Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

> On 28 Jul 2017, at 16:46, Robert Haas wrote: > > On Fri, Jul 28, 2017 at 12:39 AM, Pavan Deolasee > wrote: >> I see your point. But I would like to think this way: does the technology >> significantly help many common use cases, that are currently not addressed >> by HOT? It probably won't hel

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Jul 28, 2017 at 12:39 AM, Pavan Deolasee wrote: > I see your point. But I would like to think this way: does the technology > significantly help many common use cases, that are currently not addressed > by HOT? It probably won't help all workloads, that's given. Also, we don't > have any c

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Jul 28, 2017 at 5:57 AM, Peter Geoghegan wrote: > Pavan Deolasee wrote: > > I'll be happy if someone wants to continue hacking the patch further and > > get it in a committable shape. I can stay actively involved. But TBH the > > amount of time I can invest is far as compared to what I c

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Pavan Deolasee wrote: > I'll be happy if someone wants to continue hacking the patch further and > get it in a committable shape. I can stay actively involved. But TBH the > amount of time I can invest is far as compared to what I could during the > last cycle. That's disappointing. I personally

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Jul 26, 2017 at 6:26 PM, Robert Haas wrote: > On Tue, Apr 18, 2017 at 4:25 AM, Pavan Deolasee > wrote: > > I'll include the fix in the next set of patches. > > I haven't see a new set of patches. Are you intending to continue > working on this? > > Looks like I'll be short on bandwidth

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Apr 18, 2017 at 4:25 AM, Pavan Deolasee wrote: > I'll include the fix in the next set of patches. I haven't see a new set of patches. Are you intending to continue working on this? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Apr 14, 2017 at 9:21 PM, Jaime Casanova wrote: > > > Hi Pavan, > > I run a test on current warm patchset, i used pgbench with a scale of > 20 and a fillfactor of 90 and then start the pgbench run with 6 > clients in parallel i also run sqlsmith on it. > > And i got a core dump after somet

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On 5 April 2017 at 13:32, Pavan Deolasee wrote: > > Ok. I've extensively updated the README to match the current state of > affairs. Updated patch set attached. Hi Pavan, I run a test on current warm patchset, i used pgbench with a scale of 20 and a fillfactor of 90 and then start the pgbench ru

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 12, 2017 at 10:42 PM, Robert Haas wrote: > On Tue, Apr 11, 2017 at 1:20 PM, Pavan Deolasee > > > 5. Added code to set a CLEAR pointer to a WARM pointer when we know that > the > > entire chain is WARM. This should address the workload Dilip ran and > found > > regression (I don't th

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Apr 13, 2017 at 2:04 AM, Peter Geoghegan wrote: > On Wed, Apr 12, 2017 at 10:12 AM, Robert Haas > wrote: > >> I may have missed something, but there is no intention to ignore known > >> regressions/reviews. Of course, I don't think that every regression > will be > >> solvable, like if y

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 12, 2017 at 10:12 AM, Robert Haas wrote: >> I may have missed something, but there is no intention to ignore known >> regressions/reviews. Of course, I don't think that every regression will be >> solvable, like if you run a CPU-bound workload, setting it up in a way such >> that you r

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Apr 11, 2017 at 1:20 PM, Pavan Deolasee wrote: > I don't know why you say that regressions are not addressed. Here are a few > things I did to address the regressions/reviews/concerns, apart from fixing > all the bugs discovered, but please let me know if there are things I've not > addres

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 12, 2017 at 9:23 AM, Amit Kapila wrote: > On Tue, Apr 11, 2017 at 10:50 PM, Pavan Deolasee > > > > > And I fixed them as quickly as humanly possible. > > > > Yes, you have responded to them quickly, but I didn't get a chance to > re-verify all of those. However, I think the main poin

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Apr 11, 2017 at 10:50 PM, Pavan Deolasee wrote: > > > On Tue, Apr 11, 2017 at 7:10 PM, Robert Haas wrote: >> >> >> >> Yes, and as Andres says, you don't help with those, and then you're >> upset when your own patch doesn't get attention. > > > I am not upset, I was obviously a bit disappo

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Mon, Apr 10, 2017 at 04:34:50PM -0700, Andres Freund wrote: > Hi, > > > On 2017-04-08 23:36:13 +0530, Pavan Deolasee wrote: > > On Wed, Apr 5, 2017 at 11:57 PM, Andres Freund wrote: > > > > > On 2017-04-05 09:36:47 -0400, Robert Haas wrote: > > > > By the way, the "Converting WARM chains bac

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Apr 11, 2017 at 7:10 PM, Robert Haas wrote: > > > Yes, and as Andres says, you don't help with those, and then you're > upset when your own patch doesn't get attention. I am not upset, I was obviously a bit disappointed which I think is a very natural emotion after spending weeks on it.

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Sat, Apr 8, 2017 at 2:06 PM, Pavan Deolasee wrote: > Thank you all for the reviews, feedback, tests, criticism. And apologies > for keep pushing it till the last minute even though it was clear to me > quite some time back the patch is not going to make it. But if I'd given up, > it would have

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Hi, On 2017-04-08 23:36:13 +0530, Pavan Deolasee wrote: > On Wed, Apr 5, 2017 at 11:57 PM, Andres Freund wrote: > > > On 2017-04-05 09:36:47 -0400, Robert Haas wrote: > > > By the way, the "Converting WARM chains back to HOT chains" section of > > > README.WARM seems to be out of date. Any cha

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Sat, Apr 8, 2017 at 11:36:13PM +0530, Pavan Deolasee wrote: > Thank you all for the  reviews, feedback, tests, criticism. And apologies for > keep pushing it till the last minute even though it was clear to me quite some > time back the patch is not going to make it. But if I'd given up, it wou

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 5, 2017 at 11:57 PM, Andres Freund wrote: > On 2017-04-05 09:36:47 -0400, Robert Haas wrote: > > By the way, the "Converting WARM chains back to HOT chains" section of > > README.WARM seems to be out of date. Any chance you could update that > > to reflect the current state and think

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Apr 6, 2017 at 12:20 AM, Peter Geoghegan wrote: > On Wed, Apr 5, 2017 at 11:27 AM, Andres Freund wrote: > > I propose we move this patch to the next CF. > > I agree. I think it's too late to be working out fine details around > TOAST like this. This is a patch that touches the storage fo

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Apr 6, 2017 at 1:06 AM, Robert Haas wrote: > On Wed, Apr 5, 2017 at 2:32 PM, Pavan Deolasee > wrote: > > > The other way is to pass old tuple values along with the new tuple > values to > > amwarminsert, build index tuples and then do a comparison. For duplicate > > index tuples, skip WA

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 5, 2017 at 2:32 PM, Pavan Deolasee wrote: >> The only other idea that I have for a really clean solution here is to >> support this only for index types that are amcanreturn, and actually >> compare the value stored in the index tuple with the one stored in the >> heap tuple, ensuring

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 5, 2017 at 11:27 AM, Andres Freund wrote: > I propose we move this patch to the next CF. I agree. I think it's too late to be working out fine details around TOAST like this. This is a patch that touches the storage format in a fairly fundamental way. The idea of turning WARM on or o

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On 2017-04-05 09:36:47 -0400, Robert Haas wrote: > By the way, the "Converting WARM chains back to HOT chains" section of > README.WARM seems to be out of date. Any chance you could update that > to reflect the current state and thinking of the patch? I propose we move this patch to the next CF.

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Apr 4, 2017 at 11:43 PM, Pavan Deolasee wrote: > Well, better than causing a deadlock ;-) Yep. > Lets see if we want to go down the path of blocking WARM when tuples have > toasted attributes. I submitted a patch yesterday, but having slept over it, > I think I made mistakes there. It mi

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 5, 2017 at 8:42 AM, Robert Haas wrote: > On Tue, Apr 4, 2017 at 10:21 PM, Pavan Deolasee > wrote: > > On Thu, Mar 30, 2017 at 7:55 PM, Robert Haas > wrote: > >> but > >> try to access the TOAST table would be fatal; that probably would have > >> deadlock hazards among other problem

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Apr 4, 2017 at 10:21 PM, Pavan Deolasee wrote: > On Thu, Mar 30, 2017 at 7:55 PM, Robert Haas wrote: >> but >> try to access the TOAST table would be fatal; that probably would have >> deadlock hazards among other problems. > > Hmm. I think you're right. We could make a copy of the heap

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 7:55 PM, Robert Haas wrote: > but > try to access the TOAST table would be fatal; that probably would have > deadlock hazards among other problems. Hmm. I think you're right. We could make a copy of the heap tuple, drop the lock and then access TOAST to handle that. Wou

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 11:17 PM, Bruce Momjian wrote: > On Tue, Mar 21, 2017 at 04:04:58PM -0400, Bruce Momjian wrote: > > On Tue, Mar 21, 2017 at 04:56:16PM -0300, Alvaro Herrera wrote: > > > Bruce Momjian wrote: > > > > On Tue, Mar 21, 2017 at 04:43:58PM -0300, Alvaro Herrera wrote: > > > > >

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 31, 2017 at 12:31 PM, Dilip Kumar wrote: > On Thu, Mar 30, 2017 at 5:27 PM, Amit Kapila > wrote: > > I am not sure if we can consider it as completely synthetic because we > > might see some similar cases for json datatypes. Can we once try to > > see the impact when the same test r

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 31, 2017 at 11:54 PM, Pavan Deolasee wrote: > > On Fri, Mar 31, 2017 at 11:16 PM, Robert Haas wrote: >> >> Now, I understand you to be suggesting a flag at >> table-creation time that would, maybe, be immutable after that, but >> even then - are we going to run completely unmodified

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 4:13 AM, Pavan Deolasee wrote: > > > On Thu, Mar 30, 2017 at 3:29 PM, Dilip Kumar > wrote: > >> On Wed, Mar 29, 2017 at 11:51 AM, Pavan Deolasee >> wrote: >> > Thanks. I think your patch of tracking interesting attributes seems ok >> too >> > after the performance issue

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 31, 2017 at 11:16 PM, Robert Haas wrote: > On Fri, Mar 31, 2017 at 10:24 AM, Pavan Deolasee > wrote: > > Having worked on it for some time now, I can say that WARM uses pretty > much > > the same infrastructure that HOT uses for cleanup/pruning tuples from the > > heap. So the risk o

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 31, 2017 at 10:24 AM, Pavan Deolasee wrote: > Having worked on it for some time now, I can say that WARM uses pretty much > the same infrastructure that HOT uses for cleanup/pruning tuples from the > heap. So the risk of having a bug which can eat your data from the heap is > very low.

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 10:49 AM, Petr Jelinek wrote: > While reading this thread I am thinking if we could just not do WARM on > TOAST and compressed values if we know there might be regressions there. > I mean I've seen the problem WARM tries to solve mostly on timestamp or > boolean values and

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 31, 2017 at 6:47 PM, Robert Haas wrote: > > > 2. WARM is a non-optional feature which touches the on-disk format. > There is nothing more dangerous than that. If hash indexes have bugs, > people can avoid those bugs by not using them; there are good reasons > to suppose that hash inde

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 31, 2017 at 7:53 AM, Simon Riggs wrote: > So Andres says defer this, but Robert says "more review", which is > more than just deferral. > > We have some risky things in this release such as Hash Indexes, > function changes. I perfectly understand that perception of risk is > affected s

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On 30 March 2017 at 16:50, Robert Haas wrote: > On Thu, Mar 30, 2017 at 11:41 AM, Andres Freund wrote: >> On 2017-03-30 16:43:41 +0530, Pavan Deolasee wrote: >>> Looks like OID conflict to me.. Please try rebased set. >> >> Pavan, Alvaro, everyone: I know you guys are working very hard on this, >

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 5:27 PM, Amit Kapila wrote: > I am not sure if we can consider it as completely synthetic because we > might see some similar cases for json datatypes. Can we once try to > see the impact when the same test runs from multiple clients? For > your information, I am also try

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 21, 2017 at 04:04:58PM -0400, Bruce Momjian wrote: > On Tue, Mar 21, 2017 at 04:56:16PM -0300, Alvaro Herrera wrote: > > Bruce Momjian wrote: > > > On Tue, Mar 21, 2017 at 04:43:58PM -0300, Alvaro Herrera wrote: > > > > Bruce Momjian wrote: > > > > > > > > > I don't think it makes sens

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 11:41 AM, Andres Freund wrote: > On 2017-03-30 16:43:41 +0530, Pavan Deolasee wrote: >> Looks like OID conflict to me.. Please try rebased set. > > Pavan, Alvaro, everyone: I know you guys are working very hard on this, > but I think at this point it's too late to commit th

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Hi, On 2017-03-30 16:43:41 +0530, Pavan Deolasee wrote: > Looks like OID conflict to me.. Please try rebased set. Pavan, Alvaro, everyone: I know you guys are working very hard on this, but I think at this point it's too late to commit this for v10. This is patch that's affecting the on-disk for

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On 30/03/17 16:04, Pavan Deolasee wrote: > > > On Thu, Mar 30, 2017 at 7:27 PM, Robert Haas > wrote: > > On Thu, Mar 30, 2017 at 6:37 AM, Pavan Deolasee > mailto:pavan.deola...@gmail.com>> wrote: > > I think we can fix that by comparing compressed value

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 10:08 AM, Amit Kapila wrote: > I think we should not consider doing compression and decompression as > free at this point in code, because we hold a buffer lock during > recheck. Buffer locks are meant for short-term locks (it is even > mentioned in storage/buffer/README),

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 5:55 PM, Pavan Deolasee wrote: > > On Thu, Mar 30, 2017 at 5:27 PM, Amit Kapila > wrote: >> >> >> >> How have you verified that? Have you checked that in >> heap_prepare_insert it has called toast_insert_or_update() and then >> returned a tuple different from what the inp

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 7:27 PM, Robert Haas wrote: > On Thu, Mar 30, 2017 at 6:37 AM, Pavan Deolasee > wrote: > > I think we can fix that by comparing compressed values. I know you had > > raised concerns, but Robert confirmed that (IIUC) it's not a problem > today. > > I'm not sure that's an

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 6:37 AM, Pavan Deolasee wrote: > I think we can fix that by comparing compressed values. I know you had > raised concerns, but Robert confirmed that (IIUC) it's not a problem today. I'm not sure that's an entirely fair interpretation of what I said. My point was that, whi

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 5:27 PM, Amit Kapila wrote: > > > How have you verified that? Have you checked that in > heap_prepare_insert it has called toast_insert_or_update() and then > returned a tuple different from what the input tup is? Basically, I > am easily able to see it and even the reas

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 4:07 PM, Pavan Deolasee wrote: > > On Wed, Mar 29, 2017 at 4:42 PM, Amit Kapila > wrote: >> >> On Wed, Mar 29, 2017 at 1:10 PM, Pavan Deolasee >> wrote: >> > >> > On Wed, Mar 29, 2017 at 12:02 PM, Amit Kapila >> > wrote: >> >> >> >> On Wed, Mar 29, 2017 at 11:52 AM, Amit

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 29, 2017 at 4:42 PM, Amit Kapila wrote: > On Wed, Mar 29, 2017 at 1:10 PM, Pavan Deolasee > wrote: > > > > On Wed, Mar 29, 2017 at 12:02 PM, Amit Kapila > > wrote: > >> > >> On Wed, Mar 29, 2017 at 11:52 AM, Amit Kapila > >> wrote: > > > > Then during recheck, we pass already compr

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 29, 2017 at 11:51 AM, Pavan Deolasee wrote: > Thanks. I think your patch of tracking interesting attributes seems ok too > after the performance issue was addressed. Even though we can still improve > that further, at least Mithun confirmed that there is no significant > regression any

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Pavan Deolasee wrote: > On Wed, Mar 29, 2017 at 3:42 AM, Alvaro Herrera > wrote: > > > I pushed 0002 after some makeup, since it's just cosmetic and not > > controversial. > > Thanks. I think your patch of tracking interesting attributes seems ok too > after the performance issue was addressed.

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 29, 2017 at 7:12 AM, Amit Kapila wrote: > No as I agreed above, it won't double-compress, but still looks > slightly risky to rely on different set of values passed to > index_form_tuple and then compare them. It assumes that the compressor is completely deterministic, which I'm fairl

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 29, 2017 at 1:10 PM, Pavan Deolasee wrote: > > On Wed, Mar 29, 2017 at 12:02 PM, Amit Kapila > wrote: >> >> On Wed, Mar 29, 2017 at 11:52 AM, Amit Kapila >> wrote: > > Then during recheck, we pass already compressed values to > index_form_tuple(). But my point is, the following code

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 29, 2017 at 12:02 PM, Amit Kapila wrote: > On Wed, Mar 29, 2017 at 11:52 AM, Amit Kapila > wrote: > > On Tue, Mar 28, 2017 at 10:35 PM, Pavan Deolasee > > wrote: > >> > >> On Tue, Mar 28, 2017 at 7:04 PM, Amit Kapila > >> wrote: > >>> > >>> For such an heap insert, we will pass >

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 10:31 PM, Pavan Deolasee wrote: > > On Tue, Mar 28, 2017 at 4:05 PM, Amit Kapila > wrote: >> >> As asked previously, can you explain me on what basis are you >> considering it robust? The comments on top of datumIsEqual() clearly >> indicates the danger of using it for to

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 29, 2017 at 11:52 AM, Amit Kapila wrote: > On Tue, Mar 28, 2017 at 10:35 PM, Pavan Deolasee > wrote: >> >> On Tue, Mar 28, 2017 at 7:04 PM, Amit Kapila >> wrote: >>> >>> For such an heap insert, we will pass >>> the actual value of column to index_form_tuple during index insert. >>

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 10:35 PM, Pavan Deolasee wrote: > > On Tue, Mar 28, 2017 at 7:04 PM, Amit Kapila > wrote: >> >> >> >> For such an heap insert, we will pass >> the actual value of column to index_form_tuple during index insert. >> However during recheck when we fetch the value of c2 from

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

I pushed 0002 after some makeup, since it's just cosmetic and not controversial. Here's 0003 rebased on top of it. (Also, I took out the gin and gist changes: it would be wrong to change that unconditionally, because the 0x pattern appears in indexes that would be pg_upgraded. We need a diff

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 8:34 PM, Robert Haas wrote: > On Mon, Mar 27, 2017 at 10:25 PM, Pavan Deolasee > wrote: > > On Tue, Mar 28, 2017 at 1:59 AM, Robert Haas > wrote: > >> On Thu, Mar 23, 2017 at 2:47 PM, Pavan Deolasee > >> wrote: > >> > It's quite hard to say that until we see many more b

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 7:04 PM, Amit Kapila wrote: > > > For such an heap insert, we will pass > the actual value of column to index_form_tuple during index insert. > However during recheck when we fetch the value of c2 from heap tuple > and pass it index tuple, the value is already in compres

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 4:07 PM, Amit Kapila wrote: > > Noted few cosmetic issues in 0005_warm_updates_v21: > > 1. > pruneheap.c(939): warning C4098: 'heap_get_root_tuples' : 'void' > function returning a value > Thanks. Will fix. > > 2. > + * HCWC_WARM_UPDATED_TUPLE - a tuple with HEAP_WARM_

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 4:05 PM, Amit Kapila wrote: > > > As asked previously, can you explain me on what basis are you > considering it robust? The comments on top of datumIsEqual() clearly > indicates the danger of using it for toasted values (Also, it will > probably not give the answer you w

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Hi Pavan, On 3/28/17 11:04 AM, Robert Haas wrote: On Mon, Mar 27, 2017 at 10:25 PM, Pavan Deolasee wrote: On Tue, Mar 28, 2017 at 1:59 AM, Robert Haas wrote: On Thu, Mar 23, 2017 at 2:47 PM, Pavan Deolasee wrote: It's quite hard to say that until we see many more benchmarks. As author of t

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Mon, Mar 27, 2017 at 10:25 PM, Pavan Deolasee wrote: > On Tue, Mar 28, 2017 at 1:59 AM, Robert Haas wrote: >> On Thu, Mar 23, 2017 at 2:47 PM, Pavan Deolasee >> wrote: >> > It's quite hard to say that until we see many more benchmarks. As author >> > of >> > the patch, I might have got repeti

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 4:05 PM, Amit Kapila wrote: > On Mon, Mar 27, 2017 at 2:19 PM, Pavan Deolasee > wrote: >> On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee >> wrote: >>> >>> >>> >>> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila >>> wrote: I was worried for the case if

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Mon, Mar 27, 2017 at 2:19 PM, Pavan Deolasee wrote: > > > On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee > wrote: >> >> >> >> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila >> wrote: >>> >>> >>> >>> I was worried for the case if the index is created non-default >>> collation, will the datumIs

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Mon, Mar 27, 2017 at 2:19 PM, Pavan Deolasee wrote: > On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee > wrote: >> >> >> >> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila >> wrote: >>> >>> >>> >>> I was worried for the case if the index is created non-default >>> collation, will the datumIsEqua

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 1:32 AM, Alvaro Herrera wrote: > Is the WARM tap test suite supposed to work when applied without all the > other patches? I just tried applied that one and running "make check -C > src/test/modules", and it seems to hang after giving "ok 5" for > t/002_warm_stress.pl. (

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Mon, Mar 27, 2017 at 4:45 PM, Amit Kapila wrote: > On Sat, Mar 25, 2017 at 1:24 PM, Amit Kapila > wrote: > > On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee > > wrote: > >> > >> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila > >> wrote: > >>> > > > >> While looking at this problem, it occurr

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 08:04:34AM +0530, Pavan Deolasee wrote: > And I've answered it so many times by now :-)  LOL > Just to add more to what I just said in another email, note that HOT/WARM > chains are created when a new root line pointer is created in the heap (a line > pointer that has an i

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 7:49 AM, Bruce Momjian wrote: > On Mon, Mar 27, 2017 at 04:29:56PM -0400, Robert Haas wrote: > > On Thu, Mar 23, 2017 at 2:47 PM, Pavan Deolasee > > wrote: > > > It's quite hard to say that until we see many more benchmarks. As > author of > > > the patch, I might have go

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 1:59 AM, Robert Haas wrote: > On Thu, Mar 23, 2017 at 2:47 PM, Pavan Deolasee > wrote: > > It's quite hard to say that until we see many more benchmarks. As author > of > > the patch, I might have got repetitive with my benchmarks. But I've seen > > over 50% improvement i

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Mon, Mar 27, 2017 at 04:29:56PM -0400, Robert Haas wrote: > On Thu, Mar 23, 2017 at 2:47 PM, Pavan Deolasee > wrote: > > It's quite hard to say that until we see many more benchmarks. As author of > > the patch, I might have got repetitive with my benchmarks. But I've seen > > over 50% improvem

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 1:32 AM, Alvaro Herrera wrote: > Is the WARM tap test suite supposed to work when applied without all the > other patches? I just tried applied that one and running "make check -C > src/test/modules", and it seems to hang after giving "ok 5" for > t/002_warm_stress.pl. (

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 2:47 PM, Pavan Deolasee wrote: > It's quite hard to say that until we see many more benchmarks. As author of > the patch, I might have got repetitive with my benchmarks. But I've seen > over 50% improvement in TPS even without chain conversion (6 indexes on a 12 > column ta

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Is the WARM tap test suite supposed to work when applied without all the other patches? I just tried applied that one and running "make check -C src/test/modules", and it seems to hang after giving "ok 5" for t/002_warm_stress.pl. (I had to add a Makefile too, attached.) -- Álvaro Herrera

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Sat, Mar 25, 2017 at 1:24 PM, Amit Kapila wrote: > On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee > wrote: >> >> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila >> wrote: >>> > >> While looking at this problem, it occurred to me that the assumptions made >> for hash indexes are also wrong :-(

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Sat, Mar 25, 2017 at 11:24 PM, Pavan Deolasee wrote: > > On Sat, 25 Mar 2017 at 11:03 PM, Peter Geoghegan wrote: >> >> On Sat, Mar 25, 2017 at 12:54 AM, Amit Kapila >> wrote: >> > I am not sure how do you want to binary compare two datums, if you are >> > thinking datumIsEqual(), that won't w

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Sat, 25 Mar 2017 at 11:03 PM, Peter Geoghegan wrote: > On Sat, Mar 25, 2017 at 12:54 AM, Amit Kapila > wrote: > > I am not sure how do you want to binary compare two datums, if you are > > thinking datumIsEqual(), that won't work. I think you need to use > > datatype specific compare functio

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Sat, Mar 25, 2017 at 12:54 AM, Amit Kapila wrote: > I am not sure how do you want to binary compare two datums, if you are > thinking datumIsEqual(), that won't work. I think you need to use > datatype specific compare function something like what we do in > _bt_compare(). How will that inter

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee wrote: > > > On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila > wrote: >> >> >> >> I was worried for the case if the index is created non-default >> collation, will the datumIsEqual() suffice the need. Now again >> thinking about it, I think it will b

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila wrote: > > > I was worried for the case if the index is created non-default > collation, will the datumIsEqual() suffice the need. Now again > thinking about it, I think it will because in the index tuple we are > storing the value as in heap tuple.

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 3:54 PM, Pavan Deolasee wrote: > Thanks Amit. v19 addresses some of the comments below. > > On Thu, Mar 23, 2017 at 10:28 AM, Amit Kapila > wrote: >> >> On Wed, Mar 22, 2017 at 4:06 PM, Amit Kapila >> wrote: >> > On Tue, Mar 21, 2017 at 6:47 PM, Pavan Deolasee >> > wrote

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 24, 2017 at 4:04 PM, Amit Kapila wrote: > On Fri, Mar 24, 2017 at 12:25 AM, Pavan Deolasee > wrote: > > > > > > On Thu, Mar 23, 2017 at 7:53 PM, Amit Kapila > > > > The general sense I've got > > here is that we're ok to push some work in background if it helps the > > real-time que

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 24, 2017 at 12:25 AM, Pavan Deolasee wrote: > > > On Thu, Mar 23, 2017 at 7:53 PM, Amit Kapila > wrote: >> >> > >> >> I am not sure on what basis user can set such parameters, it will be >> quite difficult to tune those parameters. I think the point is >> whatever threshold we keep,

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 7:53 PM, Amit Kapila wrote: > On Thu, Mar 23, 2017 at 3:44 PM, Pavan Deolasee > > > > > Yes, this is a very fair point. The way I proposed to address this > upthread > > is by introducing a set of threshold/scale GUCs specific to WARM. So > users > > can control when to in

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 11:44 PM, Mithun Cy wrote: > Hi Pavan, > On Thu, Mar 23, 2017 at 12:19 AM, Pavan Deolasee > wrote: > > Ok, no problem. I did some tests on AWS i2.xlarge instance (4 vCPU, 30GB > > RAM, attached SSD) and results are shown below. But I think it is > important > > to get ind

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Hi Pavan, On Thu, Mar 23, 2017 at 12:19 AM, Pavan Deolasee wrote: > Ok, no problem. I did some tests on AWS i2.xlarge instance (4 vCPU, 30GB > RAM, attached SSD) and results are shown below. But I think it is important > to get independent validation from your side too, just to ensure I am not > m

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 3:44 PM, Pavan Deolasee wrote: > On Wed, Mar 22, 2017 at 4:06 PM, Amit Kapila > wrote: >> > >> 3. >> + /* >> + * HASH indexes compute a hash value of the key and store that in the >> + * index. So >> we must first obtain the hash of the value obtained from the >> + * heap

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 4:08 PM, Pavan Deolasee wrote: > > > On Thu, Mar 23, 2017 at 3:02 PM, Amit Kapila > wrote: > >> >> >> That sounds like you are dodging the actual problem. I mean you can >> put that same PageIsFull() check in master code as well and then you >> will most probably again s

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 3:02 PM, Amit Kapila wrote: > > > That sounds like you are dodging the actual problem. I mean you can > put that same PageIsFull() check in master code as well and then you > will most probably again see the same regression. Well I don't see it that way. There was a spe

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Thanks Amit. v19 addresses some of the comments below. On Thu, Mar 23, 2017 at 10:28 AM, Amit Kapila wrote: > On Wed, Mar 22, 2017 at 4:06 PM, Amit Kapila > wrote: > > On Tue, Mar 21, 2017 at 6:47 PM, Pavan Deolasee > > wrote: > >> > >>> > >> > >> Please find attached rebased patches. > >> > >

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 12:19 AM, Pavan Deolasee wrote: > > Ok, no problem. I did some tests on AWS i2.xlarge instance (4 vCPU, 30GB > RAM, attached SSD) and results are shown below. But I think it is important > to get independent validation from your side too, just to ensure I am not > making an

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 22, 2017 at 4:06 PM, Amit Kapila wrote: > On Tue, Mar 21, 2017 at 6:47 PM, Pavan Deolasee > wrote: >> >>> >> >> Please find attached rebased patches. >> > > Few comments on 0005_warm_updates_v18.patch: > Few more comments on 0005_warm_updates_v18.patch: 1. @@ -234,6 +241,25 @@ index_

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 12:19 AM, Pavan Deolasee wrote: > > > On Wed, Mar 22, 2017 at 4:53 PM, Mithun Cy > wrote: > Ok, no problem. I did some tests on AWS i2.xlarge instance (4 vCPU, 30GB > RAM, attached SSD) and results are shown below. But I think it is important > to get independent validatio

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 22, 2017 at 4:53 PM, Mithun Cy wrote: > On Wed, Mar 22, 2017 at 3:44 PM, Pavan Deolasee > wrote: > > > > This looks quite weird to me. Obviously these numbers are completely > > non-comparable. Even the time for VACUUM FULL goes up with every run. > > > > May be we can blame it on AW

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 22, 2017 at 8:43 AM, Pavan Deolasee wrote: > Sorry, I did not mean to suggest that you set it up wrongly, I was just > trying to point out that the test case itself may not be very practical. That is cool np!, I was just trying to explain why those tests were made if others wondered ab

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On 21 March 2017 at 20:04, Bruce Momjian wrote: > Yes, but once it is written it will take years before those bits can be > used on most installations. Well the problem isn't most installations. On most installations it should be pretty straightforward to check the oldest database xid and compare

  1   2   3   >